chore: preserve current workspace changes
Save pending configuration, deployment, and documentation work.
This commit is contained in:
@@ -0,0 +1,331 @@
|
||||
# Shared Notification Service on MicroK8s
|
||||
|
||||
This package deploys one shared Web/Worker, PostgreSQL 17, RabbitMQ 4 and a temporary
|
||||
Mailpit sandbox in `shared-notification`, on `mini-pc-2`. It does not expose public
|
||||
Ingress, NodePort or unauthenticated administration. Application administration,
|
||||
RabbitMQ management and the sandbox inbox are available through explicit local
|
||||
`kubectl port-forward` sessions over SSH.
|
||||
|
||||
The initial channel is `Invemory.Staging.Sandbox`, template `TestEmail`, exact language
|
||||
`en`. SMTP egress permits only Mailpit:1025. Mailpit has no relay configuration.
|
||||
Only its temporary inbox is ephemeral. Channels, templates, SMTP settings and usage
|
||||
counters are persisted in PostgreSQL; durable broker queues, messages, users and
|
||||
permissions are retained on the RabbitMQ volume. Never put real SMTP credentials
|
||||
into this sandbox package.
|
||||
|
||||
Mailpit runs as UID/GID `1000:1000` with dropped capabilities and writes its temporary
|
||||
SQLite inbox under `/tmp` on the writable container filesystem. Exact manifest flags
|
||||
and `/livez` were verified with the pinned image in an isolated local container.
|
||||
Adding `readOnlyRootFilesystem` requires a separate writable `/tmp` mount; the image
|
||||
exits immediately without it. The current manifest keeps that filesystem writable.
|
||||
|
||||
## Verified code contract
|
||||
|
||||
The current Worker consumes the fixed durable queue `notification.send-email` using
|
||||
the default exchange. `ServiceName` selects channels and exact-language templates;
|
||||
`CorrelationContext.ReplyTo` selects a result queue. These are logical routing
|
||||
boundaries, not authorization of the message's `ServiceName`. Separate credentials
|
||||
and queue permissions cannot restrict the JSON service name inside that shared queue.
|
||||
There is one consumer and at-least-once processing; a redelivery can produce duplicates.
|
||||
See [the consumer contract](../../../docs/itemtracker-outbox-email-consumer.md).
|
||||
|
||||
This initial vhost is `/notification`. Separate broker users are generated for
|
||||
administration, the Notification Worker and the future staging publisher. Worker
|
||||
read permission covers only `notification.send-email`; staging reads only its result
|
||||
queue. Both require write access to RabbitMQ's authorization name `amq.default` for
|
||||
default-exchange publication. Resource ACLs constrain exchange/queue names, not the
|
||||
routing key of a publish operation: access to the default exchange permits routing
|
||||
to other existing queues within the same vhost. This MVP shared-queue design therefore
|
||||
does not provide security isolation between message ServiceNames or routing keys.
|
||||
No production publisher or SMTP egress is configured.
|
||||
Additional environments need separately reviewed routing, users, permissions, exact
|
||||
templates, provider settings and egress before they can join the shared service.
|
||||
|
||||
Web/API administration has no built-in authentication in this source revision. Its
|
||||
Service stays ClusterIP and default-deny ingress; access through SSH/port-forward is
|
||||
restricted to operators. The worker exposes no health HTTP endpoint. Pod rollout is
|
||||
a process-start check; the explicit consumer gate also verifies one broker consumer.
|
||||
An actual owned smoke message and Seq event remain required after startup.
|
||||
|
||||
## Storage preparation
|
||||
|
||||
Owner decision on 2026-10-02: Notification PostgreSQL and RabbitMQ use the external
|
||||
XFS disk. Invemory staging PostgreSQL remains on its existing internal disk.
|
||||
|
||||
| Service | Data directory | Owner | Mode | Retain capacity |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| PostgreSQL | `/mnt/shared/notification-service/postgres` (`pgdata` child) | `999:999` | `0700` | `10Gi` |
|
||||
| RabbitMQ | `/mnt/shared/notification-service/rabbitmq` | `100:101` | `0700` | `5Gi` |
|
||||
|
||||
`/mnt/shared` is the bind mount of `/mnt/files/shared`; both reach the same original
|
||||
data. They are not backups. The required filesystem UUID is
|
||||
`c2c31640-a247-4988-bc7b-9ee9789464d1`. Static local PVs use `Retain`, fixed node
|
||||
affinity and prebound PVCs. Capacity declarations are scheduling/accounting budgets;
|
||||
local PVs do not enforce filesystem byte quotas.
|
||||
|
||||
Transfer the package to `/home/devops/notification-package`, then the owner runs
|
||||
one command on mini-pc-2:
|
||||
|
||||
```sh
|
||||
sudo python3 /home/devops/notification-package/prepare-storage.py
|
||||
```
|
||||
|
||||
The script verifies exact `/mnt/files` and `/mnt/shared` XFS mount targets and UUID
|
||||
before creating anything. It accepts the already provisioned exact empty directories,
|
||||
or creates missing directories under a root-owned `0755` parent. It writes root-owned
|
||||
`0644` `.invemory-storage.json` markers with the UUID and roles
|
||||
`shared-notification/postgres` / `shared-notification/rabbitmq`. Repeated execution
|
||||
verifies markers without replacing them. It refuses symlinks, incorrect permissions,
|
||||
unexpected parent contents and unmarked nonempty directories. It never deletes data.
|
||||
|
||||
The deployment host preflight runs `findmnt` for each PV directory and checks actual
|
||||
UUID, filesystem and ownership. Each Pod has a nonroot startup guard that checks
|
||||
XFS `statfs` and the exact role/UUID marker on its mounted PVC. The marker is an
|
||||
owner-provisioned assertion of UUID, not a live block-device query; the host preflight
|
||||
supplies that check. A missing external mount falls onto ext4 and fails the guard.
|
||||
These checks and `Retain` do not protect against direct manual deletion.
|
||||
|
||||
Storage Pods deliberately omit `fsGroup` so kubelet does not recursively change
|
||||
the owner-provisioned directory and marker permissions. PostgreSQL uses the pinned
|
||||
image's writable `/var/run/postgresql` directory on its container filesystem;
|
||||
RabbitMQ writes its cookie/state in its UID-owned retained data directory. Both run
|
||||
with explicit nonroot UID/GID, dropped capabilities and restricted Pod security.
|
||||
|
||||
RabbitMQ diagnostics probes allow 15 seconds per call: a healthy call on the
|
||||
resource-limited node took seven seconds. Startup keeps a three-minute failure
|
||||
budget (nine attempts, every 20 seconds); readiness runs every 20 seconds and
|
||||
liveness every 30 seconds. Startup must succeed before the other probes run.
|
||||
|
||||
Never remove files directly from these data directories. Storage path/UUID/node/
|
||||
capacity changes and database/broker image changes are rejected by repeated deploys.
|
||||
A future storage move requires a separate controlled procedure: stop publishers and
|
||||
both service writers, verify database and broker backups, stop storage processes,
|
||||
copy and verify offline data, provision target markers/ownership, bind new retained
|
||||
claims, test restoration and perform rollback checks before retiring old storage.
|
||||
This package does not perform that migration.
|
||||
|
||||
## Artifacts and private state
|
||||
|
||||
Copy `config.example.json` to `config.local.json` and verify actual Seq/publisher Pod
|
||||
selectors. The example reply queue `invemory.staging.notification-results` is isolated
|
||||
from the old client fallback. Its exact name must be agreed and configured in the
|
||||
Invemory publisher/result consumer before that later cutover. The current publisher
|
||||
remains `LogOnly`, and its result consumer remains disabled.
|
||||
|
||||
Build Web, Worker and Migrator from the same verified source archive, using each
|
||||
existing Dockerfile. Record the checked commit and archive SHA256, including any
|
||||
explicitly recorded uncommitted delta. Mirror PostgreSQL, RabbitMQ, Mailpit and a
|
||||
Python 3 Linux image to `registry.hrynco.com`. Copy `artifacts.example.json` to
|
||||
`artifacts.local.json` and record all seven immutable image digests, source hashes
|
||||
and the two reviewed migration IDs. The script rejects placeholder/tag references,
|
||||
other registries, mismatched application/migrator archives and unsupported DB/broker
|
||||
major versions. The archive hash fields are an operator provenance record; independently
|
||||
verify the build inputs and registry digests before live application.
|
||||
|
||||
Provision a **dedicated** Seq ingestion key for Notification Service, with its own
|
||||
identity/permissions, in a private file. Do not reuse the staging application's stamped
|
||||
key. Store registry authentication as a private Docker `config.json` with only
|
||||
`registry.hrynco.com` in `auths`. Never print key values, full Secrets or credentials.
|
||||
|
||||
Run the dedicated helper as devops on mini-pc-2 after package review:
|
||||
|
||||
```sh
|
||||
python3 /home/devops/notification-package/prepare-seq-key.py
|
||||
```
|
||||
|
||||
It reads the existing private `/home/devops/seq-first/credentials.json` silently and
|
||||
uses the checked session/CSRF API pattern at the trusted Seq LAN endpoint. It creates
|
||||
only `HrynCo Notification Service`, granting only `Ingest` and stamping
|
||||
`Environment=SharedNotification`, `Service=HrynCo.NotificationService`. `Application`
|
||||
remains supplied by each application. It stores the token only in
|
||||
`/home/devops/notification-private/seq-key.txt` (`0600`, parent `0700`) before the
|
||||
create request, so an unknown POST outcome can be retried with the same token.
|
||||
Repeats verify existing permissions/properties and preserve the token; mismatches,
|
||||
duplicate keys or a missing local token for an existing key fail without rotation.
|
||||
The installed Seq does not expose the original token after creation, and its
|
||||
EventEntity has no `ApiKeyId`. The helper therefore adds the server-applied property
|
||||
`NotificationIngestKeyId=<dedicated key ID>` through a guarded PUT of **only** the
|
||||
dedicated key (`apikey-79` at the initial checkpoint). It preserves its token,
|
||||
permissions, all other input settings and entity fields. It verifies that no other
|
||||
key has the same identity marker before and after updating using explicit
|
||||
`api/apikeys?shared=true` lists (including shared keys), and verifies exact
|
||||
readback. Repeats with the correct marker perform no PUT. An unknown PUT outcome is
|
||||
recovered by reading the existing marker on the next run, without token rotation or
|
||||
another key. A marker mismatch/collision stops without changing another key.
|
||||
The [official Seq API update contract](https://github.com/datalust/seq-api/blob/dev/src/Seq.Api/ResourceGroups/ApiKeysResourceGroup.cs)
|
||||
states that updating settings cannot change the API key token.
|
||||
HTTP redirects are denied before forwarding credentials; environment proxies are
|
||||
disabled for this fixed LAN operation. Each run verifies token ingestion using one
|
||||
technical canary event without user payload or credentials, then reads that exact
|
||||
`ProvisioningCheckId` through the admin session with bounded polling. It verifies
|
||||
the actual server-stamped `NotificationIngestKeyId`, Environment and Service. The
|
||||
canary body omits all three stamps, so a valid staging/foreign token cannot supply
|
||||
the dedicated identity marker. If `ApiKeyId` is exposed by a server version, it must
|
||||
also agree. Missing or conflicting identity proof fails closed without rotation.
|
||||
No event payload/token is printed. Back up the private file.
|
||||
|
||||
The key's stamped properties take precedence over application fields. Verify the first
|
||||
Web/Worker event has `Environment=SharedNotification` before accepting delivery.
|
||||
The helper never changes staging keys, signals, retention or Seq deployment.
|
||||
The existing shared Seq retention/storage policy remains in force; changing retention
|
||||
is a separate observability decision. Dedicated key and event verification remain
|
||||
mandatory live gates if the helper has not yet been run.
|
||||
|
||||
`bootstrap` generates random passwords once in
|
||||
`/home/devops/shared-notification-private/credentials.json` (`0600`, parent `0700`)
|
||||
and reconciles Kubernetes Secrets. Keep and back up this private file outside Git.
|
||||
Repeats reuse it, compare live Secret data and refuse implicit password/key/routing
|
||||
rotation. If the file is missing while a live Secret exists, recover the original
|
||||
file; generating new credentials would not change an existing PostgreSQL role.
|
||||
Registry authentication can be refreshed independently. `.local.json`, rendered files
|
||||
and `private/` are ignored in this package, but prefer keeping secrets outside the repo.
|
||||
|
||||
## Ordered installation and repeat deployment
|
||||
|
||||
All mutating commands run as devops on mini-pc-2. `prepare-storage.py` is the only
|
||||
owner-run root operation. Scripts use Python's standard library, MicroK8s kubectl,
|
||||
`/home/devops/.kube/cluster-admin.conf` and bounded timeouts. No command prints Secrets.
|
||||
|
||||
1. Review source changes, artifacts and manifests independently. Measure live node
|
||||
requests/limits and usage; inspect namespace/node pressure. The namespace budget is
|
||||
8 Pods, 5 Deployments, 3 Jobs, 2 PVCs, 15Gi storage, requests1600m/2Gi and limits4CPU/4Gi.
|
||||
This is a separate budget; existing `invemory-staging` quotas are not expanded.
|
||||
2. Run owner storage preparation and privately supply registry/Seq files.
|
||||
3. Bootstrap the namespace and preserved Secrets:
|
||||
|
||||
```sh
|
||||
python3 notification.py bootstrap --config config.local.json --artifacts artifacts.local.json --seq-key-file /home/devops/notification-private/seq-key.txt --registry-auth-file /home/devops/notification-private/registry-auth.json
|
||||
```
|
||||
|
||||
4. Render for review and run server-side dry-run. `check` also rejects resource
|
||||
adoption, wrong mounted disk, Pod selector mismatches and storage/image changes:
|
||||
|
||||
```sh
|
||||
python3 notification.py render --config config.local.json --artifacts artifacts.local.json --output /home/devops/notification-manifests.json
|
||||
python3 notification.py check --config config.local.json --artifacts artifacts.local.json
|
||||
```
|
||||
|
||||
5. First install applies storage/isolated foundation and runs migrations with Web and
|
||||
Worker stopped. The migration target is exactly `InitialCreate` + `PendingChanges`.
|
||||
After migration success, the script verifies history, grants a separate
|
||||
`notification_runtime` role only DML on the three application tables, and idempotently
|
||||
seeds the Mailpit channel and neutral English `TestEmail`. Existing matching-service
|
||||
channels/templates are left unchanged. It resumes Web after these gates; Worker
|
||||
remains stopped for inspection:
|
||||
|
||||
```sh
|
||||
python3 notification.py deploy --config config.local.json --artifacts artifacts.local.json --migrate
|
||||
```
|
||||
|
||||
6. Inspect channel/template, queue backlog/permissions, DB role and Seq identity.
|
||||
Start the Worker only after those checks. The script verifies the sandbox SMTP
|
||||
settings, required template, empty incoming queue and exactly one consumer:
|
||||
|
||||
```sh
|
||||
python3 notification.py deploy --config config.local.json --artifacts artifacts.local.json --enable-worker --sandbox-verified
|
||||
```
|
||||
|
||||
7. Send one owned neutral test message to Mailpit; inspect its matching result,
|
||||
usage counter, ACK and redacted technical logs. Test malformed metadata and
|
||||
missing template/variable failure with owned safe fixtures. Remove only owned
|
||||
test data through application/broker APIs. No arbitrary live email or client backlog
|
||||
is authorized by this smoke test.
|
||||
|
||||
Ordinary repeat `deploy` verifies target schema and preserves the current Worker
|
||||
replica count. It reapplies canonical manifests with the same persistent claims and
|
||||
Secret references. When Worker remains enabled, it waits for its rollout and verifies
|
||||
exactly one broker consumer before reporting completion. Both explicit enable and
|
||||
ordinary redeploy allow up to 60 seconds after rollout for a zero consumer count to
|
||||
become one, polling with a two-second pause and bounded broker commands. Extra or
|
||||
unreadable consumer counts fail immediately. Rollout failure, verification failure
|
||||
or timeout scales Worker to zero and stops the deployment without reporting success.
|
||||
It does not recreate Secrets
|
||||
or reseed provider settings. A later
|
||||
`--migrate` stops Web/Worker, takes a verified local PostgreSQL custom-format dump,
|
||||
runs the Migrator and history/grant/seed gates, then resumes Web. Worker stays stopped
|
||||
unless `--enable-worker --sandbox-verified` is supplied and all gates pass. Once
|
||||
writers have been stopped, backup, migration, history or grants failure leaves them
|
||||
stopped. An initial history mismatch on an already ready database is rejected before
|
||||
scaling writers, without changing their state. Failed Jobs remain for
|
||||
private inspection; the next run replaces only its own named Job. Migration locking
|
||||
uses a private host file and is local to this deployment host.
|
||||
|
||||
An interrupted first install can leave the PostgreSQL Deployment present with no
|
||||
ready Pod (for example, before owner storage markers exist). A retry with `--migrate`
|
||||
first verifies ownership/storage/images, stops both writers and reconciles only the
|
||||
foundation/storage workloads. It waits for PostgreSQL, RabbitMQ and Mailpit readiness
|
||||
before querying actual migration history. Every migration cycle backs up the actual
|
||||
initialized database, including an empty first database: Deployment absence/existence
|
||||
never establishes that the retained volume has no data. Backup/history/storage
|
||||
failure keeps writers stopped and prevents migration. An ordinary deploy against an
|
||||
unready database requires this explicit recovery path and performs no mutation.
|
||||
Stopping an already empty Deployment is supported: collection reads require a JSON
|
||||
PodList, and an empty `items` list needs no Pod-termination wait. Named resource
|
||||
lookups alone use `--ignore-not-found`.
|
||||
|
||||
Backups are retained under the private directory with a SHA256/byte-count/history
|
||||
manifest. `pg_restore --list` validates dump structure; a full restore test and offsite
|
||||
copy are separate gates. This backup covers the Notification database only. It is not
|
||||
a snapshot of RabbitMQ's queue files. Migration does not upgrade/recreate the broker
|
||||
volume; durable queues remain on that retained volume. Broker disk backup/recovery
|
||||
must be verified before a broker upgrade or storage migration.
|
||||
|
||||
## Access and networking
|
||||
|
||||
Use SSH to the deployment host, then forward one local port when needed:
|
||||
|
||||
```sh
|
||||
/snap/microk8s/current/kubectl --kubeconfig=/home/devops/.kube/cluster-admin.conf -n shared-notification port-forward service/web 5200:8080 --address=127.0.0.1
|
||||
```
|
||||
|
||||
Apply equivalent explicit forwards for `service/rabbitmq`15672 and `service/mailpit`8025.
|
||||
Keep them bound to loopback. ClusterIP services expose no LAN/public ports.
|
||||
|
||||
Default-deny policies allow namespace-scoped DB5432, shared Worker/bounded staging
|
||||
publisher AMQP5672, sandbox SMTP1025, cluster DNS53 and Web/Worker Seq8080 only.
|
||||
The package adds its own `notification-seq-ingress` policy in `observability` and
|
||||
`notification-broker-egress` in `invemory-staging`; existing policies stay owned by
|
||||
their original packages. Verify these exact selectors in the rendered diff.
|
||||
No SMTP internet egress or broad namespace allowance is included.
|
||||
|
||||
The Invemory publisher/result-consumer cutover remains a separate operation after
|
||||
Notification acceptance, exact staged settings and backlog inspection. The package
|
||||
does not change Invemory configuration, production routes or CI deployment.
|
||||
|
||||
## Local validation
|
||||
|
||||
```powershell
|
||||
python -m unittest discover -s docker/kubernetes/shared-notification/tests -v
|
||||
dotnet build HrynCo.NotificationService.slnx --no-restore
|
||||
dotnet test HrynCo.NotificationService.slnx --no-build --no-restore
|
||||
```
|
||||
|
||||
Unit tests cover immutable artifacts, Secret preservation structure, isolated network
|
||||
and storage contracts, failure ordering and no automatic Worker activation. Passing
|
||||
local tests does not establish live storage, image startup, Seq access, mail delivery,
|
||||
backups/restoration or repeat-deploy acceptance; record these after actual testing.
|
||||
|
||||
## Verified mini-pc-2 sandbox checkpoint (2026-10-02)
|
||||
|
||||
The initial installation from source commit `a44b98292f9001f28fed425be131a2c993a85847`
|
||||
completed on mini-pc-2. The external XFS storage guards passed; PostgreSQL, RabbitMQ,
|
||||
Mailpit, Web and one Worker are Ready with zero restarts. The namespace has no Ingress
|
||||
or NodePort. `InitialCreate` and `PendingChanges`, runtime grants and sandbox seed
|
||||
completed. Two private local PostgreSQL backups were created and validated by the
|
||||
deployment script before the initial and repeated migration runs. Neither backup is
|
||||
an offsite copy or a RabbitMQ backup; full restoration was not tested.
|
||||
|
||||
The dedicated Seq key was verified on live ingestion and Web/Worker events carry
|
||||
`Environment=SharedNotification`. A single owned `TestEmail` message reached only
|
||||
Mailpit, was ACKed, produced a matching successful result and incremented the usage
|
||||
counter to one. An unsupported message type was rejected without an email; a missing
|
||||
template produced a matching negative result without an email. The owned messages and
|
||||
results were removed afterward. Both queues are empty and the incoming queue has one
|
||||
consumer. Ordinary repeat deploy with the Worker running and a repeated no-op
|
||||
migration passed; the Worker was enabled again afterward. The external volume data
|
||||
measured about 47MiB for PostgreSQL and 292KiB for RabbitMQ at this checkpoint.
|
||||
|
||||
Invemory staging still publishes Outbox entries in `LogOnly` mode and does not
|
||||
consume results. Connecting its publisher/result consumer and proving a real staging
|
||||
application flow remain separate gated work. Other environments and production SMTP
|
||||
are not part of this sandbox checkpoint.
|
||||
Reference in New Issue
Block a user