hrynco-notification-service/docker/kubernetes/shared-notification/README.md

Shared Notification Service on MicroK8s

For the current six-migration production release, use the production operations and recovery runbook. The original bootstrap/deploy schema below predates push channels; do not run its full deploy against the current database. The current additive Google HTTPS policy is google-api-network.json; refresh it with refresh-google-api-network.py and review/server-dry-run it before applying. Existing SMTP and broker policies remain.

This package deploys one shared Web/Worker, PostgreSQL 17, RabbitMQ 4 and a temporary Mailpit sandbox in shared-notification, on mini-pc-2. It does not expose public Ingress, NodePort or unauthenticated administration. Application administration, RabbitMQ management and the sandbox inbox are available through explicit local kubectl port-forward sessions over SSH.

The initial sandbox channel is Invemory.Staging.Sandbox, template TestEmail, exact language en. Its SMTP egress uses Mailpit:1025, which has no relay configuration. The production add-on also permits the Worker to reach external SMTP on TCP 587. Only its temporary inbox is ephemeral. Channels, templates, SMTP settings and usage counters are persisted in PostgreSQL; durable broker queues, messages, users and permissions are retained on the RabbitMQ volume. Never put real SMTP credentials into this sandbox package.

Mailpit runs as UID/GID 1000:1000 with dropped capabilities and writes its temporary SQLite inbox under /tmp on the writable container filesystem. Exact manifest flags and /livez were verified with the pinned image in an isolated local container. Adding readOnlyRootFilesystem requires a separate writable /tmp mount; the image exits immediately without it. The current manifest keeps that filesystem writable.

Verified code contract

The current Worker consumes the fixed durable queue notification.send-email using the default exchange. ServiceName selects channels and exact-language templates; CorrelationContext.ReplyTo selects a result queue. These are logical routing boundaries, not authorization of the message's ServiceName. Separate credentials and queue permissions cannot restrict the JSON service name inside that shared queue. There is one consumer and at-least-once processing; a redelivery can produce duplicates. See the consumer contract.

This initial vhost is /notification. Separate broker users are generated for administration, the Notification Worker and the future staging publisher. Worker read permission covers only notification.send-email; staging reads only its result queue. Both require write access to RabbitMQ's authorization name amq.default for default-exchange publication. Resource ACLs constrain exchange/queue names, not the routing key of a publish operation: access to the default exchange permits routing to other existing queues within the same vhost. This MVP shared-queue design therefore does not provide security isolation between message ServiceNames or routing keys. The production publisher credential, result queue, network policies and SMTP egress are provisioned separately below. Additional environments need separately reviewed routing, users, permissions, exact templates, provider settings and egress before they can join the shared service.

Invemory production publisher credential

The additive activate-production-publisher.py provisions the dedicated invemory_production user in /notification, durable result queue invemory.production.notification-results, and the Notification Worker permissions for that result queue. It preserves the staging user and queue. Run it on mini-pc-2 as devops from its reviewed copy; repeated runs verify the existing password and ACLs. The generated password is retained only in /home/devops/shared-notification-private/invemory-production-rabbitmq-password.txt with mode 0600. It is never printed by the provisioning script.

On mini-pc-2, the owner can display the password in their own terminal with:

sudo -u devops cat /home/devops/shared-notification-private/invemory-production-rabbitmq-password.txt

Use the invemory_production user, that password, broker rabbitmq.shared-notification.svc.cluster.local:5672, vhost /notification, send queue notification.send-email, result queue invemory.production.notification-results, and the existing technical service name StoreMate-Prod in the Invemory production notification settings. The visible sender and TestEmail text still contain legacy branding; IT-1510 tracks that copy change.

Live activation — 2026-10-03

production-network.yaml adds only production API/Outbox egress and RabbitMQ ingress on TCP 5672, plus shared Worker egress to public SMTP on TCP 587 (private and cluster address ranges excluded). The existing sandbox Mailpit allowance stays intact. The file passed Kubernetes server dry-run and was applied on mini-pc-2. The API result consumer and both Workers are running at one replica; RabbitMQ has one consumer on notification.send-email and one on invemory.production.notification-results. Two owned production test messages were published, accepted by the Gmail SMTP channel, acknowledged as delivery successes in Invemory Outbox, and confirmed in the recipient's Gmail conversation. Today's channel usage increased by two. Both queues ended with zero ready and unacknowledged messages.

Administrative SMTP test from Web — 2026-10-18

The channel editor's Send Test Email action sends directly from Web through SMTP; it does not use the Worker or RabbitMQ. Its TCP 587 connection timed out while the production network policy allowed external SMTP only from Worker. The additive web-to-external-smtp policy in production-network.yaml grants Web the same public IPv4 TCP 587 egress as Worker, with the same private and cluster address exclusions. Apply this production overlay separately from the sandbox-only notification.py render. No SMTP credentials, channel settings, templates, or workload restart are required for this network change.

The activation helper also persists the production user, queue and expanded Worker ACL in the rabbit-definitions Secret. The private password and matching RabbitMQ hash are saved with mode 0600; updated notification.py reproduces that Secret during bootstrap without rotating passwords. The reviewed source is installed at /home/devops/notification-package/notification.py, with its pre-activation copy retained beside it. A broker restart should therefore reload production access; verify the two consumers and an owned delivery afterward. The SMTP password remains outside the repository. Apply production-network.yaml separately when reconciling network policies. Do not use the initial sandbox-only render as a complete production delivery configuration.

Web/API administration has no built-in authentication in this source revision. Its Service stays ClusterIP and default-deny ingress; access through SSH/port-forward is restricted to operators. The worker exposes no health HTTP endpoint. Pod rollout is a process-start check; the explicit consumer gate also verifies one broker consumer. An actual owned smoke message and Seq event remain required after startup.

Storage preparation

Owner decision on 2026-10-02: Notification PostgreSQL and RabbitMQ use the external XFS disk. Invemory staging PostgreSQL remains on its existing internal disk.

Service	Data directory	Owner	Mode	Retain capacity
PostgreSQL	/mnt/shared/notification-service/postgres (pgdata child)	999:999	0700	10Gi
RabbitMQ	/mnt/shared/notification-service/rabbitmq	100:101	0700	5Gi

/mnt/shared is the bind mount of /mnt/files/shared; both reach the same original data. They are not backups. The required filesystem UUID is c2c31640-a247-4988-bc7b-9ee9789464d1. Static local PVs use Retain, fixed node affinity and prebound PVCs. Capacity declarations are scheduling/accounting budgets; local PVs do not enforce filesystem byte quotas.

Transfer the package to /home/devops/notification-package, then the owner runs one command on mini-pc-2:

sudo python3 /home/devops/notification-package/prepare-storage.py

The script verifies exact /mnt/files and /mnt/shared XFS mount targets and UUID before creating anything. It accepts the already provisioned exact empty directories, or creates missing directories under a root-owned 0755 parent. It writes root-owned 0644 .invemory-storage.json markers with the UUID and roles shared-notification/postgres / shared-notification/rabbitmq. Repeated execution verifies markers without replacing them. It refuses symlinks, incorrect permissions, unexpected parent contents and unmarked nonempty directories. It never deletes data.

The deployment host preflight runs findmnt for each PV directory and checks actual UUID, filesystem and ownership. Each Pod has a nonroot startup guard that checks XFS statfs and the exact role/UUID marker on its mounted PVC. The marker is an owner-provisioned assertion of UUID, not a live block-device query; the host preflight supplies that check. A missing external mount falls onto ext4 and fails the guard. These checks and Retain do not protect against direct manual deletion.

Storage Pods deliberately omit fsGroup so kubelet does not recursively change the owner-provisioned directory and marker permissions. PostgreSQL uses the pinned image's writable /var/run/postgresql directory on its container filesystem; RabbitMQ writes its cookie/state in its UID-owned retained data directory. Both run with explicit nonroot UID/GID, dropped capabilities and restricted Pod security.

RabbitMQ diagnostics probes allow 15 seconds per call: a healthy call on the resource-limited node took seven seconds. Startup keeps a three-minute failure budget (nine attempts, every 20 seconds); readiness runs every 20 seconds and liveness every 30 seconds. Startup must succeed before the other probes run.

Never remove files directly from these data directories. Storage path/UUID/node/ capacity changes and database/broker image changes are rejected by repeated deploys. A future storage move requires a separate controlled procedure: stop publishers and both service writers, verify database and broker backups, stop storage processes, copy and verify offline data, provision target markers/ownership, bind new retained claims, test restoration and perform rollback checks before retiring old storage. This package does not perform that migration.

Artifacts and private state

Copy config.example.json to config.local.json and verify actual Seq/publisher Pod selectors. The example reply queue invemory.staging.notification-results is isolated from the old client fallback. Its exact name must be agreed and configured in the Invemory publisher/result consumer before that later cutover. The current publisher remains LogOnly, and its result consumer remains disabled.

Build Web, Worker and Migrator from the same verified source archive, using each existing Dockerfile. Record the checked commit and archive SHA256, including any explicitly recorded uncommitted delta. Mirror PostgreSQL, RabbitMQ, Mailpit and a Python 3 Linux image to registry.hrynco.com. Copy artifacts.example.json to artifacts.local.json and record all seven immutable image digests, source hashes and the three reviewed migration IDs, ending with 20261017072410_EmailTemplateLocalizations. The script rejects placeholder/tag references, other registries, mismatched application/migrator archives and unsupported DB/broker major versions. The archive hash fields are an operator provenance record; independently verify the build inputs and registry digests before live application.

Provision a dedicated Seq ingestion key for Notification Service, with its own identity/permissions, in a private file. Do not reuse the staging application's stamped key. Store registry authentication as a private Docker config.json with only registry.hrynco.com in auths. Never print key values, full Secrets or credentials.

Run the dedicated helper as devops on mini-pc-2 after package review:

python3 /home/devops/notification-package/prepare-seq-key.py

It reads the existing private /home/devops/seq-first/credentials.json silently and uses the checked session/CSRF API pattern at the trusted Seq LAN endpoint. It creates only HrynCo Notification Service, granting only Ingest and stamping Environment=SharedNotification, Service=HrynCo.NotificationService. Application remains supplied by each application. It stores the token only in /home/devops/notification-private/seq-key.txt (0600, parent 0700) before the create request, so an unknown POST outcome can be retried with the same token. Repeats verify existing permissions/properties and preserve the token; mismatches, duplicate keys or a missing local token for an existing key fail without rotation. The installed Seq does not expose the original token after creation, and its EventEntity has no ApiKeyId. The helper therefore adds the server-applied property NotificationIngestKeyId=<dedicated key ID> through a guarded PUT of only the dedicated key (apikey-79 at the initial checkpoint). It preserves its token, permissions, all other input settings and entity fields. It verifies that no other key has the same identity marker before and after updating using explicit api/apikeys?shared=true lists (including shared keys), and verifies exact readback. Repeats with the correct marker perform no PUT. An unknown PUT outcome is recovered by reading the existing marker on the next run, without token rotation or another key. A marker mismatch/collision stops without changing another key. The official Seq API update contract states that updating settings cannot change the API key token. HTTP redirects are denied before forwarding credentials; environment proxies are disabled for this fixed LAN operation. Each run verifies token ingestion using one technical canary event without user payload or credentials, then reads that exact ProvisioningCheckId through the admin session with bounded polling. It verifies the actual server-stamped NotificationIngestKeyId, Environment and Service. The canary body omits all three stamps, so a valid staging/foreign token cannot supply the dedicated identity marker. If ApiKeyId is exposed by a server version, it must also agree. Missing or conflicting identity proof fails closed without rotation. No event payload/token is printed. Back up the private file.

The key's stamped properties take precedence over application fields. Verify the first Web/Worker event has Environment=SharedNotification before accepting delivery. The helper never changes staging keys, signals, retention or Seq deployment. The existing shared Seq retention/storage policy remains in force; changing retention is a separate observability decision. Dedicated key and event verification remain mandatory live gates if the helper has not yet been run.

bootstrap generates random passwords once in /home/devops/shared-notification-private/credentials.json (0600, parent 0700) and reconciles Kubernetes Secrets. Keep and back up this private file outside Git. Repeats reuse it, compare live Secret data and refuse implicit password/key/routing rotation. If the file is missing while a live Secret exists, recover the original file; generating new credentials would not change an existing PostgreSQL role. Registry authentication can be refreshed independently. .local.json, rendered files and private/ are ignored in this package, but prefer keeping secrets outside the repo.

Ordered installation and repeat deployment

All mutating commands run as devops on mini-pc-2. prepare-storage.py is the only owner-run root operation. Scripts use Python's standard library, MicroK8s kubectl, /home/devops/.kube/cluster-admin.conf and bounded timeouts. No command prints Secrets.

1. Review source changes, artifacts and manifests independently. Measure live node requests/limits and usage; inspect namespace/node pressure. The namespace budget is 8 Pods, 5 Deployments, 3 Jobs, 2 PVCs, 15Gi storage, requests1600m/2Gi and limits4CPU/4Gi. This is a separate budget; existing invemory-staging quotas are not expanded.
2. Run owner storage preparation and privately supply registry/Seq files.
3. Bootstrap the namespace and preserved Secrets:

python3 notification.py bootstrap --config config.local.json --artifacts artifacts.local.json --seq-key-file /home/devops/notification-private/seq-key.txt --registry-auth-file /home/devops/notification-private/registry-auth.json

4. Render for review and run server-side dry-run. check also rejects resource adoption, wrong mounted disk, Pod selector mismatches and storage/image changes:

python3 notification.py render --config config.local.json --artifacts artifacts.local.json --output /home/devops/notification-manifests.json
python3 notification.py check --config config.local.json --artifacts artifacts.local.json

5. First install or a schema upgrade applies storage/isolated foundation and runs migrations with Web and Worker stopped. The migration target is exactly InitialCreate + PendingChanges + EmailTemplateLocalizations. After migration success, the script verifies history, grants a separate notification_runtime role only DML on the application tables, including the service/language/definition catalog, and idempotently seeds the Mailpit channel, sandbox service/catalog and neutral English TestEmail. Existing matching-service channels/templates and shared variable contracts are left unchanged. It resumes Web after these gates; Worker remains stopped for inspection:

python3 notification.py deploy --config config.local.json --artifacts artifacts.local.json --migrate

6. Inspect channel/template, queue backlog/permissions, DB role and Seq identity. Start the Worker only after those checks. The script verifies the sandbox SMTP settings, required template, empty incoming queue and exactly one consumer:

python3 notification.py deploy --config config.local.json --artifacts artifacts.local.json --enable-worker --sandbox-verified

7. Send one owned neutral test message to Mailpit; inspect its matching result, usage counter, ACK and redacted technical logs. Test malformed metadata and missing template/variable failure with owned safe fixtures. Remove only owned test data through application/broker APIs. No arbitrary live email or client backlog is authorized by this smoke test.

Ordinary repeat deploy verifies target schema and preserves the current Worker replica count. It reapplies canonical manifests with the same persistent claims and Secret references. When Worker remains enabled, it waits for its rollout and verifies exactly one broker consumer before reporting completion. Both explicit enable and ordinary redeploy allow up to 60 seconds after rollout for a zero consumer count to become one, polling with a two-second pause and bounded broker commands. Extra or unreadable consumer counts fail immediately. Rollout failure, verification failure or timeout scales Worker to zero and stops the deployment without reporting success. It does not recreate Secrets or reseed provider settings. A later --migrate stops Web/Worker, takes a verified local PostgreSQL custom-format dump, runs the Migrator and history/grant/seed gates, then resumes Web. Worker stays stopped unless --enable-worker --sandbox-verified is supplied and all gates pass. Once writers have been stopped, backup, migration, history or grants failure leaves them stopped. An initial history mismatch on an already ready database is rejected before scaling writers, without changing their state. Failed Jobs remain for private inspection; the next run replaces only its own named Job. Migration locking uses a private host file and is local to this deployment host.

An interrupted first install can leave the PostgreSQL Deployment present with no ready Pod (for example, before owner storage markers exist). A retry with --migrate first verifies ownership/storage/images, stops both writers and reconciles only the foundation/storage workloads. It waits for PostgreSQL, RabbitMQ and Mailpit readiness before querying actual migration history. Every migration cycle backs up the actual initialized database, including an empty first database: Deployment absence/existence never establishes that the retained volume has no data. Backup/history/storage failure keeps writers stopped and prevents migration. An ordinary deploy against an unready database requires this explicit recovery path and performs no mutation. Stopping an already empty Deployment is supported: collection reads require a JSON PodList, and an empty items list needs no Pod-termination wait. Named resource lookups alone use --ignore-not-found.

Backups are retained under the private directory with a SHA256/byte-count/history manifest. pg_restore --list validates dump structure; a full restore test and offsite copy are separate gates. This backup covers the Notification database only. It is not a snapshot of RabbitMQ's queue files. Migration does not upgrade/recreate the broker volume; durable queues remain on that retained volume. Broker disk backup/recovery must be verified before a broker upgrade or storage migration.

Before upgrading a live flat database to EmailTemplateLocalizations, inspect the exact source columns/indexes/history and audit every template's identities and JSON variable shape. Versions of one service/key must have the same variable contract after sorting by variable name. Rehearse the current Migrator and sandbox/grants SQL on a disposable restored PostgreSQL 17 copy, then verify preserved content and a repeat no-op migration. The live deploy --migrate creates its own final backup after stopping Web and Worker. Do not use the old Web/Worker images with the new schema. Rollback requires restoring a verified pre-upgrade database backup with the matching old application images; migration Down is not a general rollback path. The production network add-on and production broker access remain separate, preserved resources; verify them and the existing production queues before re-enabling Worker.

Access and networking

Use SSH to the deployment host, then forward one local port when needed:

/snap/microk8s/current/kubectl --kubeconfig=/home/devops/.kube/cluster-admin.conf -n shared-notification port-forward service/web 5200:8080 --address=127.0.0.1

Apply equivalent explicit forwards for service/rabbitmq15672 and service/mailpit8025. Keep them bound to loopback. ClusterIP services expose no LAN/public ports.

Default-deny policies allow namespace-scoped DB5432, shared Worker/bounded staging publisher AMQP5672, sandbox SMTP1025, cluster DNS53 and Web/Worker Seq8080 only. The package adds its own notification-seq-ingress policy in observability and notification-broker-egress in invemory-staging; existing policies stay owned by their original packages. Verify these exact selectors in the rendered diff. No SMTP internet egress or broad namespace allowance is included.

The Invemory publisher/result-consumer cutover remains a separate operation after Notification acceptance, exact staged settings and backlog inspection. The package does not change Invemory configuration, production routes or CI deployment.

Local validation

python -m unittest discover -s docker/kubernetes/shared-notification/tests -v
dotnet build HrynCo.NotificationService.slnx --no-restore
dotnet test HrynCo.NotificationService.slnx --no-build --no-restore

Unit tests cover immutable artifacts, Secret preservation structure, isolated network and storage contracts, failure ordering and no automatic Worker activation. Passing local tests does not establish live storage, image startup, Seq access, mail delivery, backups/restoration or repeat-deploy acceptance; record these after actual testing.

Verified mini-pc-2 sandbox checkpoint (2026-10-02)

The initial installation from source commit a44b98292f9001f28fed425be131a2c993a85847 completed on mini-pc-2. The external XFS storage guards passed; PostgreSQL, RabbitMQ, Mailpit, Web and one Worker are Ready with zero restarts. The namespace has no Ingress or NodePort. InitialCreate and PendingChanges, runtime grants and sandbox seed completed. Two private local PostgreSQL backups were created and validated by the deployment script before the initial and repeated migration runs. Neither backup is an offsite copy or a RabbitMQ backup; full restoration was not tested.

The dedicated Seq key was verified on live ingestion and Web/Worker events carry Environment=SharedNotification. A single owned TestEmail message reached only Mailpit, was ACKed, produced a matching successful result and incremented the usage counter to one. An unsupported message type was rejected without an email; a missing template produced a matching negative result without an email. The owned messages and results were removed afterward. Both queues are empty and the incoming queue has one consumer. Ordinary repeat deploy with the Worker running and a repeated no-op migration passed; the Worker was enabled again afterward. The external volume data measured about 47MiB for PostgreSQL and 292KiB for RabbitMQ at this checkpoint.

Invemory staging still publishes Outbox entries in LogOnly mode and does not consume results. Connecting its publisher/result consumer and proving a real staging application flow remain separate gated work. Other environments and production SMTP are not part of this sandbox checkpoint.