hrynco-notification-service/docker/kubernetes/shared-notification/README.md

# Shared Notification Service on MicroK8s

This package deploys one shared Web/Worker, PostgreSQL 17, RabbitMQ 4 and a temporary
Mailpit sandbox in `shared-notification`, on `mini-pc-2`. It does not expose public
Ingress, NodePort or unauthenticated administration. Application administration,
RabbitMQ management and the sandbox inbox are available through explicit local
`kubectl port-forward` sessions over SSH.

The initial sandbox channel is `Invemory.Staging.Sandbox`, template `TestEmail`, exact
language `en`. Its SMTP egress uses Mailpit:1025, which has no relay configuration.
The production add-on also permits the Worker to reach external SMTP on TCP 587.
Only its temporary inbox is ephemeral. Channels, templates, SMTP settings and usage
counters are persisted in PostgreSQL; durable broker queues, messages, users and
permissions are retained on the RabbitMQ volume. Never put real SMTP credentials
into this sandbox package.

Mailpit runs as UID/GID `1000:1000` with dropped capabilities and writes its temporary
SQLite inbox under `/tmp` on the writable container filesystem. Exact manifest flags
and `/livez` were verified with the pinned image in an isolated local container.
Adding `readOnlyRootFilesystem` requires a separate writable `/tmp` mount; the image
exits immediately without it. The current manifest keeps that filesystem writable.

## Verified code contract

The current Worker consumes the fixed durable queue `notification.send-email` using
the default exchange. `ServiceName` selects channels and exact-language templates;
`CorrelationContext.ReplyTo` selects a result queue. These are logical routing
boundaries, not authorization of the message's `ServiceName`. Separate credentials
and queue permissions cannot restrict the JSON service name inside that shared queue.
There is one consumer and at-least-once processing; a redelivery can produce duplicates.
See [the consumer contract](../../../docs/itemtracker-outbox-email-consumer.md).

This initial vhost is `/notification`. Separate broker users are generated for
administration, the Notification Worker and the future staging publisher. Worker
read permission covers only `notification.send-email`; staging reads only its result
queue. Both require write access to RabbitMQ's authorization name `amq.default` for
default-exchange publication. Resource ACLs constrain exchange/queue names, not the
routing key of a publish operation: access to the default exchange permits routing
to other existing queues within the same vhost. This MVP shared-queue design therefore
does not provide security isolation between message ServiceNames or routing keys.
The production publisher credential, result queue, network policies and SMTP egress
are provisioned separately below.
Additional environments need separately reviewed routing, users, permissions, exact
templates, provider settings and egress before they can join the shared service.

## Invemory production publisher credential

The additive `activate-production-publisher.py` provisions the dedicated
`invemory_production` user in `/notification`, durable result queue
`invemory.production.notification-results`, and the Notification Worker permissions
for that result queue. It preserves the staging user and queue. Run it on mini-pc-2
as `devops` from its reviewed copy; repeated runs verify the existing password and
ACLs. The generated password is retained only in
`/home/devops/shared-notification-private/invemory-production-rabbitmq-password.txt`
with mode `0600`. It is never printed by the provisioning script.

On mini-pc-2, the owner can display the password in their own terminal with:

```sh
sudo -u devops cat /home/devops/shared-notification-private/invemory-production-rabbitmq-password.txt
```

Use the `invemory_production` user, that password, broker
`rabbitmq.shared-notification.svc.cluster.local:5672`, vhost `/notification`, send
queue `notification.send-email`, result queue
`invemory.production.notification-results`, and the existing technical service name
`StoreMate-Prod` in the Invemory production notification settings. The visible sender
and `TestEmail` text still contain legacy branding; IT-1510 tracks that copy change.

### Live activation — 2026-10-03

`production-network.yaml` adds only production API/Outbox egress and RabbitMQ ingress
on TCP 5672, plus shared Worker egress to public SMTP on TCP 587 (private and cluster
address ranges excluded). The existing sandbox Mailpit allowance stays intact. The
file passed Kubernetes server dry-run and was applied on mini-pc-2. The API result
consumer and both Workers are running at one replica; RabbitMQ has one consumer on
`notification.send-email` and one on
`invemory.production.notification-results`. Two owned production test messages were
published, accepted by the Gmail SMTP channel, acknowledged as delivery successes in
Invemory Outbox, and confirmed in the recipient's Gmail conversation. Today's channel
usage increased by two. Both queues ended with zero ready and unacknowledged messages.

The activation helper also persists the production user, queue and expanded Worker ACL
in the `rabbit-definitions` Secret. The private password and matching RabbitMQ hash
are saved with mode `0600`; updated `notification.py` reproduces that Secret during
bootstrap without rotating passwords. The reviewed source is installed at
`/home/devops/notification-package/notification.py`, with its pre-activation copy
retained beside it. A broker restart should therefore reload production access;
verify the two consumers and an owned delivery afterward. The SMTP password remains
outside the repository. Apply `production-network.yaml` separately when reconciling
network policies. Do not use the initial sandbox-only render as a complete production
delivery configuration.

Web/API administration has no built-in authentication in this source revision. Its
Service stays ClusterIP and default-deny ingress; access through SSH/port-forward is
restricted to operators. The worker exposes no health HTTP endpoint. Pod rollout is
a process-start check; the explicit consumer gate also verifies one broker consumer.
An actual owned smoke message and Seq event remain required after startup.

## Storage preparation

Owner decision on 2026-10-02: Notification PostgreSQL and RabbitMQ use the external
XFS disk. Invemory staging PostgreSQL remains on its existing internal disk.

| Service | Data directory | Owner | Mode | Retain capacity |
| --- | --- | --- | --- | --- |
| PostgreSQL | `/mnt/shared/notification-service/postgres` (`pgdata` child) | `999:999` | `0700` | `10Gi` |
| RabbitMQ | `/mnt/shared/notification-service/rabbitmq` | `100:101` | `0700` | `5Gi` |

`/mnt/shared` is the bind mount of `/mnt/files/shared`; both reach the same original
data. They are not backups. The required filesystem UUID is
`c2c31640-a247-4988-bc7b-9ee9789464d1`. Static local PVs use `Retain`, fixed node
affinity and prebound PVCs. Capacity declarations are scheduling/accounting budgets;
local PVs do not enforce filesystem byte quotas.

Transfer the package to `/home/devops/notification-package`, then the owner runs
one command on mini-pc-2:

```sh
sudo python3 /home/devops/notification-package/prepare-storage.py
```

The script verifies exact `/mnt/files` and `/mnt/shared` XFS mount targets and UUID
before creating anything. It accepts the already provisioned exact empty directories,
or creates missing directories under a root-owned `0755` parent. It writes root-owned
`0644` `.invemory-storage.json` markers with the UUID and roles
`shared-notification/postgres` / `shared-notification/rabbitmq`. Repeated execution
verifies markers without replacing them. It refuses symlinks, incorrect permissions,
unexpected parent contents and unmarked nonempty directories. It never deletes data.

The deployment host preflight runs `findmnt` for each PV directory and checks actual
UUID, filesystem and ownership. Each Pod has a nonroot startup guard that checks
XFS `statfs` and the exact role/UUID marker on its mounted PVC. The marker is an
owner-provisioned assertion of UUID, not a live block-device query; the host preflight
supplies that check. A missing external mount falls onto ext4 and fails the guard.
These checks and `Retain` do not protect against direct manual deletion.

Storage Pods deliberately omit `fsGroup` so kubelet does not recursively change
the owner-provisioned directory and marker permissions. PostgreSQL uses the pinned
image's writable `/var/run/postgresql` directory on its container filesystem;
RabbitMQ writes its cookie/state in its UID-owned retained data directory. Both run
with explicit nonroot UID/GID, dropped capabilities and restricted Pod security.

RabbitMQ diagnostics probes allow 15 seconds per call: a healthy call on the
resource-limited node took seven seconds. Startup keeps a three-minute failure
budget (nine attempts, every 20 seconds); readiness runs every 20 seconds and
liveness every 30 seconds. Startup must succeed before the other probes run.

Never remove files directly from these data directories. Storage path/UUID/node/
capacity changes and database/broker image changes are rejected by repeated deploys.
A future storage move requires a separate controlled procedure: stop publishers and
both service writers, verify database and broker backups, stop storage processes,
copy and verify offline data, provision target markers/ownership, bind new retained
claims, test restoration and perform rollback checks before retiring old storage.
This package does not perform that migration.

## Artifacts and private state

Copy `config.example.json` to `config.local.json` and verify actual Seq/publisher Pod
selectors. The example reply queue `invemory.staging.notification-results` is isolated
from the old client fallback. Its exact name must be agreed and configured in the
Invemory publisher/result consumer before that later cutover. The current publisher
remains `LogOnly`, and its result consumer remains disabled.

Build Web, Worker and Migrator from the same verified source archive, using each
existing Dockerfile. Record the checked commit and archive SHA256, including any
explicitly recorded uncommitted delta. Mirror PostgreSQL, RabbitMQ, Mailpit and a
Python 3 Linux image to `registry.hrynco.com`. Copy `artifacts.example.json` to
`artifacts.local.json` and record all seven immutable image digests, source hashes
and the three reviewed migration IDs, ending with `20261017072410_EmailTemplateLocalizations`.
The script rejects placeholder/tag references,
other registries, mismatched application/migrator archives and unsupported DB/broker
major versions. The archive hash fields are an operator provenance record; independently
verify the build inputs and registry digests before live application.

Provision a **dedicated** Seq ingestion key for Notification Service, with its own
identity/permissions, in a private file. Do not reuse the staging application's stamped
key. Store registry authentication as a private Docker `config.json` with only
`registry.hrynco.com` in `auths`. Never print key values, full Secrets or credentials.

Run the dedicated helper as devops on mini-pc-2 after package review:

```sh
python3 /home/devops/notification-package/prepare-seq-key.py
```

It reads the existing private `/home/devops/seq-first/credentials.json` silently and
uses the checked session/CSRF API pattern at the trusted Seq LAN endpoint. It creates
only `HrynCo Notification Service`, granting only `Ingest` and stamping
`Environment=SharedNotification`, `Service=HrynCo.NotificationService`. `Application`
remains supplied by each application. It stores the token only in
`/home/devops/notification-private/seq-key.txt` (`0600`, parent `0700`) before the
create request, so an unknown POST outcome can be retried with the same token.
Repeats verify existing permissions/properties and preserve the token; mismatches,
duplicate keys or a missing local token for an existing key fail without rotation.
The installed Seq does not expose the original token after creation, and its
EventEntity has no `ApiKeyId`. The helper therefore adds the server-applied property
`NotificationIngestKeyId=<dedicated key ID>` through a guarded PUT of **only** the
dedicated key (`apikey-79` at the initial checkpoint). It preserves its token,
permissions, all other input settings and entity fields. It verifies that no other
key has the same identity marker before and after updating using explicit
`api/apikeys?shared=true` lists (including shared keys), and verifies exact
readback. Repeats with the correct marker perform no PUT. An unknown PUT outcome is
recovered by reading the existing marker on the next run, without token rotation or
another key. A marker mismatch/collision stops without changing another key.
The [official Seq API update contract](https://github.com/datalust/seq-api/blob/dev/src/Seq.Api/ResourceGroups/ApiKeysResourceGroup.cs)
states that updating settings cannot change the API key token.
HTTP redirects are denied before forwarding credentials; environment proxies are
disabled for this fixed LAN operation. Each run verifies token ingestion using one
technical canary event without user payload or credentials, then reads that exact
`ProvisioningCheckId` through the admin session with bounded polling. It verifies
the actual server-stamped `NotificationIngestKeyId`, Environment and Service. The
canary body omits all three stamps, so a valid staging/foreign token cannot supply
the dedicated identity marker. If `ApiKeyId` is exposed by a server version, it must
also agree. Missing or conflicting identity proof fails closed without rotation.
No event payload/token is printed. Back up the private file.

The key's stamped properties take precedence over application fields. Verify the first
Web/Worker event has `Environment=SharedNotification` before accepting delivery.
The helper never changes staging keys, signals, retention or Seq deployment.
The existing shared Seq retention/storage policy remains in force; changing retention
is a separate observability decision. Dedicated key and event verification remain
mandatory live gates if the helper has not yet been run.

`bootstrap` generates random passwords once in
`/home/devops/shared-notification-private/credentials.json` (`0600`, parent `0700`)
and reconciles Kubernetes Secrets. Keep and back up this private file outside Git.
Repeats reuse it, compare live Secret data and refuse implicit password/key/routing
rotation. If the file is missing while a live Secret exists, recover the original
file; generating new credentials would not change an existing PostgreSQL role.
Registry authentication can be refreshed independently. `.local.json`, rendered files
and `private/` are ignored in this package, but prefer keeping secrets outside the repo.

## Ordered installation and repeat deployment

All mutating commands run as devops on mini-pc-2. `prepare-storage.py` is the only
owner-run root operation. Scripts use Python's standard library, MicroK8s kubectl,
`/home/devops/.kube/cluster-admin.conf` and bounded timeouts. No command prints Secrets.

1. Review source changes, artifacts and manifests independently. Measure live node
   requests/limits and usage; inspect namespace/node pressure. The namespace budget is
   8 Pods, 5 Deployments, 3 Jobs, 2 PVCs, 15Gi storage, requests1600m/2Gi and limits4CPU/4Gi.
   This is a separate budget; existing `invemory-staging` quotas are not expanded.
2. Run owner storage preparation and privately supply registry/Seq files.
3. Bootstrap the namespace and preserved Secrets:

```sh
python3 notification.py bootstrap --config config.local.json --artifacts artifacts.local.json --seq-key-file /home/devops/notification-private/seq-key.txt --registry-auth-file /home/devops/notification-private/registry-auth.json
```

4. Render for review and run server-side dry-run. `check` also rejects resource
   adoption, wrong mounted disk, Pod selector mismatches and storage/image changes:

```sh
python3 notification.py render --config config.local.json --artifacts artifacts.local.json --output /home/devops/notification-manifests.json
python3 notification.py check --config config.local.json --artifacts artifacts.local.json
```

5. First install or a schema upgrade applies storage/isolated foundation and runs
   migrations with Web and Worker stopped. The migration target is exactly
   `InitialCreate` + `PendingChanges` + `EmailTemplateLocalizations`.
   After migration success, the script verifies history, grants a separate
   `notification_runtime` role only DML on the application tables, including the
   service/language/definition catalog, and idempotently seeds the Mailpit channel,
   sandbox service/catalog and neutral English `TestEmail`. Existing matching-service
   channels/templates and shared variable contracts are left unchanged. It resumes
   Web after these gates; Worker remains stopped for inspection:

```sh
python3 notification.py deploy --config config.local.json --artifacts artifacts.local.json --migrate
```

6. Inspect channel/template, queue backlog/permissions, DB role and Seq identity.
   Start the Worker only after those checks. The script verifies the sandbox SMTP
   settings, required template, empty incoming queue and exactly one consumer:

```sh
python3 notification.py deploy --config config.local.json --artifacts artifacts.local.json --enable-worker --sandbox-verified
```

7. Send one owned neutral test message to Mailpit; inspect its matching result,
   usage counter, ACK and redacted technical logs. Test malformed metadata and
   missing template/variable failure with owned safe fixtures. Remove only owned
   test data through application/broker APIs. No arbitrary live email or client backlog
   is authorized by this smoke test.

Ordinary repeat `deploy` verifies target schema and preserves the current Worker
replica count. It reapplies canonical manifests with the same persistent claims and
Secret references. When Worker remains enabled, it waits for its rollout and verifies
exactly one broker consumer before reporting completion. Both explicit enable and
ordinary redeploy allow up to 60 seconds after rollout for a zero consumer count to
become one, polling with a two-second pause and bounded broker commands. Extra or
unreadable consumer counts fail immediately. Rollout failure, verification failure
or timeout scales Worker to zero and stops the deployment without reporting success.
It does not recreate Secrets
or reseed provider settings. A later
`--migrate` stops Web/Worker, takes a verified local PostgreSQL custom-format dump,
runs the Migrator and history/grant/seed gates, then resumes Web. Worker stays stopped
unless `--enable-worker --sandbox-verified` is supplied and all gates pass. Once
writers have been stopped, backup, migration, history or grants failure leaves them
stopped. An initial history mismatch on an already ready database is rejected before
scaling writers, without changing their state. Failed Jobs remain for
private inspection; the next run replaces only its own named Job. Migration locking
uses a private host file and is local to this deployment host.

An interrupted first install can leave the PostgreSQL Deployment present with no
ready Pod (for example, before owner storage markers exist). A retry with `--migrate`
first verifies ownership/storage/images, stops both writers and reconciles only the
foundation/storage workloads. It waits for PostgreSQL, RabbitMQ and Mailpit readiness
before querying actual migration history. Every migration cycle backs up the actual
initialized database, including an empty first database: Deployment absence/existence
never establishes that the retained volume has no data. Backup/history/storage
failure keeps writers stopped and prevents migration. An ordinary deploy against an
unready database requires this explicit recovery path and performs no mutation.
Stopping an already empty Deployment is supported: collection reads require a JSON
PodList, and an empty `items` list needs no Pod-termination wait. Named resource
lookups alone use `--ignore-not-found`.

Backups are retained under the private directory with a SHA256/byte-count/history
manifest. `pg_restore --list` validates dump structure; a full restore test and offsite
copy are separate gates. This backup covers the Notification database only. It is not
a snapshot of RabbitMQ's queue files. Migration does not upgrade/recreate the broker
volume; durable queues remain on that retained volume. Broker disk backup/recovery
must be verified before a broker upgrade or storage migration.

Before upgrading a live flat database to `EmailTemplateLocalizations`, inspect the
exact source columns/indexes/history and audit every template's identities and JSON
variable shape. Versions of one service/key must have the same variable contract
after sorting by variable name. Rehearse the current Migrator and sandbox/grants SQL
on a disposable restored PostgreSQL 17 copy, then verify preserved content and a
repeat no-op migration. The live `deploy --migrate` creates its own final backup after
stopping Web and Worker. Do not use the old Web/Worker images with the new schema.
Rollback requires restoring a verified pre-upgrade database backup with the matching
old application images; migration `Down` is not a general rollback path. The
production network add-on and production broker access remain separate, preserved
resources; verify them and the existing production queues before re-enabling Worker.

## Access and networking

Use SSH to the deployment host, then forward one local port when needed:

```sh
/snap/microk8s/current/kubectl --kubeconfig=/home/devops/.kube/cluster-admin.conf -n shared-notification port-forward service/web 5200:8080 --address=127.0.0.1
```

Apply equivalent explicit forwards for `service/rabbitmq`15672 and `service/mailpit`8025.
Keep them bound to loopback. ClusterIP services expose no LAN/public ports.

Default-deny policies allow namespace-scoped DB5432, shared Worker/bounded staging
publisher AMQP5672, sandbox SMTP1025, cluster DNS53 and Web/Worker Seq8080 only.
The package adds its own `notification-seq-ingress` policy in `observability` and
`notification-broker-egress` in `invemory-staging`; existing policies stay owned by
their original packages. Verify these exact selectors in the rendered diff.
No SMTP internet egress or broad namespace allowance is included.

The Invemory publisher/result-consumer cutover remains a separate operation after
Notification acceptance, exact staged settings and backlog inspection. The package
does not change Invemory configuration, production routes or CI deployment.

## Local validation

```powershell
python -m unittest discover -s docker/kubernetes/shared-notification/tests -v
dotnet build HrynCo.NotificationService.slnx --no-restore
dotnet test HrynCo.NotificationService.slnx --no-build --no-restore
```

Unit tests cover immutable artifacts, Secret preservation structure, isolated network
and storage contracts, failure ordering and no automatic Worker activation. Passing
local tests does not establish live storage, image startup, Seq access, mail delivery,
backups/restoration or repeat-deploy acceptance; record these after actual testing.

## Verified mini-pc-2 sandbox checkpoint (2026-10-02)

The initial installation from source commit `a44b98292f9001f28fed425be131a2c993a85847`
completed on mini-pc-2. The external XFS storage guards passed; PostgreSQL, RabbitMQ,
Mailpit, Web and one Worker are Ready with zero restarts. The namespace has no Ingress
or NodePort. `InitialCreate` and `PendingChanges`, runtime grants and sandbox seed
completed. Two private local PostgreSQL backups were created and validated by the
deployment script before the initial and repeated migration runs. Neither backup is
an offsite copy or a RabbitMQ backup; full restoration was not tested.

The dedicated Seq key was verified on live ingestion and Web/Worker events carry
`Environment=SharedNotification`. A single owned `TestEmail` message reached only
Mailpit, was ACKed, produced a matching successful result and incremented the usage
counter to one. An unsupported message type was rejected without an email; a missing
template produced a matching negative result without an email. The owned messages and
results were removed afterward. Both queues are empty and the incoming queue has one
consumer. Ordinary repeat deploy with the Worker running and a repeated no-op
migration passed; the Worker was enabled again afterward. The external volume data
measured about 47MiB for PostgreSQL and 292KiB for RabbitMQ at this checkpoint.

Invemory staging still publishes Outbox entries in `LogOnly` mode and does not
consume results. Connecting its publisher/result consumer and proving a real staging
application flow remain separate gated work. Other environments and production SMTP
are not part of this sandbox checkpoint.