# Notification Service: production operations and recovery Verified against the running Kubernetes service on **2026-10-24**. Commands below distinguish read-only checks, service interruptions and destructive recovery. Do not paste passwords, service-account JSON, tokens, Secret YAML or database dumps into tickets, chat, Git or CI logs. The administrative UI has no application authentication; keep it behind the existing loopback SSH tunnel. ## 1. Actual environment and source of truth | Item | Actual value | | --- | --- | | Windows source checkout | `C:\projects\HrynCo\hrynco-notification-service` | | SSH target | `devops@192.168.2.44` (`mini-pc-2`; Linux username is lowercase) | | Windows SSH key | `C:\Users\agryn\.ssh\id_ed25519_devops`, unlocked in the owner's ssh-agent | | Kubernetes CLI | `/snap/microk8s/current/kubectl` | | Kubeconfig / context | `/home/devops/.kube/cluster-admin.conf` / `microk8s` | | Namespace / CNI | `shared-notification` / Calico `v3.29.3` | | Server package | `/home/devops/notification-package` | | Private files / local backups | `/home/devops/shared-notification-private` | | External storage | `/mnt/shared/notification-service`, XFS, UUID `c2c31640-a247-4988-bc7b-9ee9789464d1` | | Local admin URL | `http://localhost:15200/admin/services` | | Web service | `web:8080`, ClusterIP; no public Ingress or NodePort | | PostgreSQL | `postgres:5432`, database `notification_service` | | RabbitMQ | `rabbitmq:5672`, vhost `/notification`, management port `15672` | | Seq | `http://seq.observability.svc.cluster.local:8080` | | Sandbox SMTP / UI | `mailpit:1025` / `mailpit:8025`; no external relay | The repository's `docker/kubernetes/shared-notification/` owns the package. The server's `config.local.json` supplies the verified host/storage and namespace values; `artifacts.local.json` still describes the older bootstrap application images. `notification.py` owns the original foundation and credential bootstrap. Its `MIGRATIONS`, sandbox SQL and application artifact file are **not yet reconciled with the six-migration release below**. Do not run its full `bootstrap` or `deploy` command over this production release. It can overwrite current image/configuration intent or fail on removed `email_channels.service_name`. Use the targeted procedures in this document for the existing production resources. They do not depend on the historical one-off `it1074-deploy.py`. The original deployment audit is retained privately at `/home/devops/shared-notification-private/it1074/{before,result}.json`; `before.json` contains deployment specifications and fingerprints, not a restorable database backup. The one-off helper's initial GRANT used the wrong table case; do not rerun it. Network source of truth is additive: base policies from `notification.py`, existing `production-network.yaml` for SMTP and production broker access, and `google-api-network.json` for Google APIs. Never reconcile the sandbox render as the complete production setup. Reconcile and review all overlays together. ## 2. Connect and inspect before changing anything Run in **Windows PowerShell** (no SSH alias is required or assumed): ```powershell ssh -o StrictHostKeyChecking=yes -o HostKeyAlgorithms=ssh-ed25519 -o ConnectTimeout=10 -i C:/Users/agryn/.ssh/id_ed25519_devops devops@192.168.2.44 ``` Inside that **Linux SSH session**, define these session-local helpers. All subsequent `k` and `kn` commands in this document run there, not in Windows PowerShell: ```sh k() { /snap/microk8s/current/kubectl --kubeconfig=/home/devops/.kube/cluster-admin.conf --request-timeout=20s "$@"; } kn() { k --namespace=shared-notification "$@"; } cd /home/devops/notification-package k config current-context k get node mini-pc-2 kn get deployment,pod,service,pvc,job k get pv shared-notification-postgres shared-notification-rabbitmq findmnt -T /mnt/shared/notification-service -o TARGET,SOURCE,FSTYPE,UUID kn get resourcequota ``` Expected: context `microk8s`, node Ready, the exact XFS UUID above, both PVCs Bound. Stop if the disk is not mounted: do not let an empty directory on the root disk stand in for persistent storage. PVs use `Retain` and node affinity to `mini-pc-2`: | PVC | PV | Path | Capacity | | --- | --- | --- | --- | | `postgres-data` | `shared-notification-postgres` | `/mnt/shared/notification-service/postgres` | 10Gi | | `rabbitmq-data` | `shared-notification-rabbitmq` | `/mnt/shared/notification-service/rabbitmq` | 5Gi | Do not delete PVCs, clear PV claimRefs, reformat storage or regenerate credentials to repair an application timeout. These are singleton workloads, not an HA database/broker. The quota allows five Deployments and three Jobs; the completed migrator/grants/sandbox Jobs can occupy all three Job slots. A new fourth Job can fail admission. ## 3. Current application release and infrastructure images Application source: `d4ee8fc0c8036b0b48e1a481ef1a12a6a1c1c269`, merged PR 28. Image build: TeamCity `141783`, tag `main-34`. Deploy immutable digests: ```sh WEB_IMAGE='registry.hrynco.com/hrynco.notification-service.web@sha256:69c51ac1ba7c1101472a0b7d45153544a95565cc544bba49908b1e4079195911' WORKER_IMAGE='registry.hrynco.com/hrynco.notification-service.worker@sha256:cc93cbfd82e1d7ff4d88845dfa90c8a1188c51d35775432d385fa5f23795a6a3' MIGRATOR_IMAGE='registry.hrynco.com/hrynco.notification-service.migrator@sha256:9975e05cb075f45a6a629233af4c56194731378f710feb719b205a68bebdb035' ``` The repositories include `notification-service` with a hyphen. Old bootstrap images used `notificationservice`; do not accidentally choose those. These digests are OCI indexes; the matching amd64 manifests are web `d69a6d7cdbc8c08d5a1234933a88579fd176a72b8c64fa02167c42d988eed0be`, worker `92d454a4139de55f5619070ed6b070c94db05749bbd3aca5d0e91d5a11ae8875`, migrator `a94766802609f4348adba4bdef4bc528275f317c3015b10bb97c8d530749adbd`. Registry manifest inspection must accept OCI indexes; a Docker-only Accept header can return a misleading 404. Retained infrastructure digests (do not upgrade during application recovery): | Image repository | Digest | | --- | --- | | `registry.hrynco.com/mirror/postgres` | `sha256:e31e3d5327d1806f6177827c9710643e4f35f7ab3f14d26d05332753d3e95ee0` | | `registry.hrynco.com/mirror/rabbitmq` | `sha256:ad995f859fce23a0ccd91c24cfac89d8f8cc57de4d708b58dcecea2697ad0d8c` | | `registry.hrynco.com/mirror/mailpit` | `sha256:13de4ffbd28f3089be6dff32afbc8210edb5a593d738fa877ae7272cc73aeb67` | Read-only image and readiness verification: ```sh kn get deployment web worker -o custom-columns='NAME:.metadata.name,DESIRED:.spec.replicas,READY:.status.readyReplicas,IMAGE:.spec.template.spec.containers[*].image' kn get pods -o custom-columns='NAME:.metadata.name,READY:.status.containerStatuses[*].ready,RESTARTS:.status.containerStatuses[*].restartCount,IMAGEID:.status.containerStatuses[*].imageID' kn rollout status deployment/web --timeout=240s kn rollout status deployment/worker --timeout=240s ``` Web readiness queries `/api/v1/email-channels?serviceName=Invemory.Staging.Sandbox` on 8080, every 10 seconds with a one-second timeout. Cold start under the 250m CPU limit can initially time out. Worker has no HTTP/readiness probe: **Ready is not proof of consumer readiness**. Verify RabbitMQ separately. ## 4. Configuration and credentials Both Web and Worker use `notification-config` through `envFrom`: `DOTNET_ENVIRONMENT=Production`, `ASPNETCORE_ENVIRONMENT=Production`, `ASPNETCORE_HTTP_PORTS=8080`, RabbitMQ host `rabbitmq`, port `5672`, vhost `/notification`, Seq URL above, Serilog service `HrynCo.NotificationService` and environment `Shared`. Inspect nonsecret settings with: ```sh kn get configmap notification-config -o yaml kn get secret ``` `notification-private` keys: `ownerConnectionString`, `runtimeConnectionString`, `postgresPassword`, `runtimePassword`, `rabbitAdminPassword`, `rabbitWorkerPassword`, `rabbitPublisherPassword`, `seqApiKey`. Web/Worker use the runtime connection; migrations use owner connection. Worker username is `notification_worker`. `registry-pull` contains `.dockerconfigjson`; `rabbit-definitions` contains `definitions.json` with sensitive password hashes. Do not print their values. The server private directory retains `credentials.json` and production publisher password/hash files. Registry authentication and Seq source files are under `/home/devops/notification-private/registry-auth.json` and `seq-key.txt`. Recover original credentials from a protected backup if lost; changing Kubernetes Secrets does not automatically change passwords in existing database/broker volumes. Do not run bootstrap to generate replacements against retained data. Firebase service-account JSON is stored in PostgreSQL `push_channels.credentials` (JSONB), without application encryption, by explicit owner decision. Database backups therefore contain private keys. Edit forms never return the saved JSON; leaving the field blank retains it. A project change requires matching new credentials. Current channel is attached to `Invemory-Prod`, Firebase project technical ID `storemate-hrynco-com`. Do not copy local development keys into production. `Check Firebase authorization` obtains an OAuth token using saved credentials when the project is unchanged. It does not send a push, prove FCM send permission, or prove device display. `Send test push` is a separate explicitly authorized action. The standalone test sends a Firebase `notification` payload without Invemory binding data. On the currently installed Invemory Android build it displays through the FCM SDK in background. In foreground, `InvemoryMessagingService.onMessageReceived` rejects the empty data map because `bindingId`, `version`, `messageId`, route and expiry are required; it does not call the base Capacitor handler. Thus accepted test messages can be invisible while Invemory is open. Samsung logs on 2026-10-24 show the FCM fallback notification channel and removal at 13:04:28 Europe/Kiev; the receiver does not log its rejection, so no individual foreground callback was proven. Do not diagnose this as egress failure or rotate tokens solely because foreground display is absent. The Worker push consumer is **not enabled** by this rollout: configuration and broker ACLs still cover email only. A saved active channel and successful OAuth check do not activate end-to-end production push processing. Do not enable it incidentally during an email service recovery. ## 5. Database history and runtime grants Read-only history check: ```sh kn exec deployment/postgres -- psql -U notification_owner -d notification_service -At -c 'SELECT "MigrationId" FROM "__EFMigrationsHistory" ORDER BY "MigrationId"' ``` Expected, in order: ```text 20260501214629_InitialCreate 20260502154249_PendingChanges 20261017072410_EmailTemplateLocalizations 20261023223753_AddPushDeliveries 20261024073202_LinkEmailChannelsToServices 20261024080457_AddPushChannels ``` `notification_owner` is currently the PostgreSQL bootstrap superuser; keep it out of application connections. `notification_runtime` is a login with no superuser, createdb or createrole privileges. There are **no default ACL entries**: future tables do not inherit runtime CRUD automatically. For each new migration, grant only the tables needed by the application and persist the same SQL in `runtime-grants`. Do not add blanket default privileges as a workaround. The exact current push grants are below. `"PushDeliveries"` is case-sensitive; `push_deliveries` does not exist. This command is idempotent and was verified: ```sh kn exec deployment/postgres -- psql -U notification_owner -d notification_service -v ON_ERROR_STOP=1 -c 'GRANT SELECT, INSERT, UPDATE, DELETE ON push_channels, "PushDeliveries" TO notification_runtime;' kn exec deployment/postgres -- psql -U notification_owner -d notification_service -At -c "SELECT t,has_table_privilege('notification_runtime',t,'SELECT,INSERT,UPDATE,DELETE') FROM unnest(ARRAY['push_channels','\"PushDeliveries\"']) t" ``` Both checks must return `t`. Existing runtime CRUD covers `email_channels`, `email_templates`, `email_channel_usage`, `services`, `service_languages`, `email_template_definitions`; schema USAGE and database CONNECT are granted, schema CREATE is revoked from PUBLIC. The live `runtime-grants` ConfigMap already includes the correct push grant. To persist it again without rotating passwords: ```sh python3 - <<'PY' import json, subprocess k = ['/snap/microk8s/current/kubectl', '--kubeconfig=/home/devops/.kube/cluster-admin.conf', '-n', 'shared-notification'] value = json.loads(subprocess.check_output(k + ['get','configmap','runtime-grants','-o','json'])) grant = 'GRANT SELECT, INSERT, UPDATE, DELETE ON push_channels, "PushDeliveries" TO notification_runtime;' sql = value['data']['grants.sql'] if grant not in sql: subprocess.run(k + ['patch','configmap','runtime-grants','--type=merge','-p',json.dumps({'data':{'grants.sql':sql+'\n'+grant+'\n'}})], check=True) PY ``` Do not execute the entire grants ConfigMap manually without its intended `RUNTIME_PASSWORD` secret environment: its bootstrap SQL also manages the runtime role password. Do not rerun the old sandbox job; channel ownership now uses `service_id`. The migration maps the legacy `StoreMate-Prod` channel to existing `Invemory-Prod`. The old identifier is historical migration input, not a new channel/service name. ## 6. Backup before migration or restore **Service interruption:** stop Web and Worker writers first. This does not stop external publishers; coordinate a publisher pause if a consistent broker/database recovery point is needed. Never purge queues as a substitute. ```sh kn scale deployment/worker deployment/web --replicas=0 kn wait --for=delete pod -l 'app.kubernetes.io/component in (web,worker)' --timeout=120s ``` The canonical backup implementation is safe independently of its old deploy schema. This maintained source command creates a custom-format dump with mode 0600, checks the `PGDMP` signature and `pg_restore --list`, and writes SHA-256, bytes and migration history to `manifest.json`. Run from the server package; only these existing functions are used, not `deploy` or sandbox SQL: ```sh python3 - <<'PY' import types import notification as n c = n.Cluster(types.SimpleNamespace(kubectl='/snap/microk8s/current/kubectl', kubeconfig='/home/devops/.kube/cluster-admin.conf')) with n.lock('/home/devops/shared-notification-private'): assert all(c.get('deployment', x)['spec']['replicas'] == 0 for x in ('web','worker')) print(n.backup(c, '/home/devops/shared-notification-private')) PY ``` Record the returned folder. Pre-migration backup from this rollout: `/home/devops/shared-notification-private/backup-20261024T094457816933Z`. It has only the **first three migrations**, predates the owner's production push channel, and is not a backup of today's six-migration state. Take a fresh backup before any recovery. A local dump does not protect against loss of this host or disk. No offsite copy or full disaster restore has been proven. Verify a selected backup without restoring it (set the exact existing folder): ```sh BACKUP_DIR=/home/devops/shared-notification-private/backup-20261024T094457816933Z export BACKUP_DIR python3 - <<'PY' import hashlib,json,os,pathlib p=pathlib.Path(os.environ['BACKUP_DIR']) m=json.loads((p/'manifest.json').read_text()); data=(p/'database.dump').read_bytes() assert len(data)==m['bytes'] and hashlib.sha256(data).hexdigest()==m['sha256'] assert data.startswith(b'PGDMP') print('Checksum and length verified; migration history:', m['history']) PY kn exec -i deployment/postgres -- pg_restore --list < "$BACKUP_DIR/database.dump" >/dev/null ``` Protect an additional recovery export with restrictive permissions. The following commands contain secrets in their **files**, never print or commit them. They are operator recovery instructions, not proof that an offsite backup exists: ```sh umask 077 RECOVERY_DIR=/home/devops/shared-notification-private/recovery-$(date -u +%Y%m%dT%H%M%SZ) mkdir "$RECOVERY_DIR" kn get secret -o json > "$RECOVERY_DIR/secrets.json" kn get deployment,service,configmap,networkpolicy,resourcequota,limitrange,pvc -o json > "$RECOVERY_DIR/namespace-resources.json" k get pv shared-notification-postgres shared-notification-rabbitmq -o json > "$RECOVERY_DIR/pvs.json" ``` These are inventories with server-assigned fields, not manifests to blindly apply to a new cluster. Keep protected copies of the package config, kubeconfig and private credential files with your backup system. RabbitMQ definitions do not back up queued messages. A broker volume backup requires a coordinated stopped broker and publisher maintenance window; no live-volume tar or restore drill was performed here. ## 7. Recover/redeploy this release on retained infrastructure Prerequisites: correct disk/PVCs, original Secrets, PostgreSQL and RabbitMQ Ready, no concurrent deployment, verified fresh backup, exact pinned image variables from section 3. Web/Worker remain stopped from section 6. If history already contains exactly the six entries above, skip migration. If it is exactly the original three-entry prefix, use the following maintained recipe from the package directory. It creates the same bounded migrator Job used for the rollout, references the existing owner Secret, and replaces only the completed owned migrator Job to fit the quota. Other histories require inspection; do not guess or edit EF history. ```sh export MIGRATOR_IMAGE python3 - <<'PY' import os,types import notification as n c=n.Cluster(types.SimpleNamespace(kubectl='/snap/microk8s/current/kubectl',kubeconfig='/home/devops/.kube/cluster-admin.conf')) prefix=['20260501214629_InitialCreate','20260502154249_PendingChanges','20261017072410_EmailTemplateLocalizations'] target=prefix+['20261023223753_AddPushDeliveries','20261024073202_LinkEmailChannelsToServices','20261024080457_AddPushChannels'] with n.lock('/home/devops/shared-notification-private'): assert all(c.get('deployment',x)['spec']['replicas']==0 for x in ('web','worker')) assert c.history()==prefix, 'Unexpected history; stop and inspect' old=c.get('job','notification-migrator') assert old and old['metadata']['labels']['app.kubernetes.io/managed-by']==n.MANAGED and old['status'].get('succeeded')==1 job=n.job('migrator',os.environ['MIGRATOR_IMAGE'],environment=[n.env('DOTNET_ENVIRONMENT','Production'),n.env('App__ConnectionString',secret='notification-private',key='ownerConnectionString')]) n.run_job(c,job) assert c.history()==target print('All six migrations verified; apply runtime grants before starting writers.') PY ``` Apply and verify the grants in section 5. Inspect migrated channel ownership without reading credentials: ```sh kn exec deployment/postgres -- psql -U notification_owner -d notification_service -At -c 'SELECT s.name,count(*) FROM email_channels c JOIN services s ON s.id=c.service_id GROUP BY s.name ORDER BY s.name' ``` Set only application images, retaining Secrets, volumes, selectors and resources: ```sh kn set image deployment/web web="$WEB_IMAGE" kn set image deployment/worker worker="$WORKER_IMAGE" kn scale deployment/web --replicas=1 kn rollout status deployment/web --timeout=240s kn scale deployment/worker --replicas=1 kn rollout status deployment/worker --timeout=240s ``` Then verify actual pod imageIDs, six migrations, grants, consumers, UI and OAuth. If migration fails, keep writers stopped while inspecting the failed Job and database history. If rollout fails, inspect events/logs before changing anything else. Never leave replicas at zero without reporting the specific failure and recovery action. ## 8. Rollback and destructive database restore An image-only rollback is allowed only when the target image supports the **current** schema. `kubectl rollout undo` can select the old pre-ServiceId binary and is unsafe after this migration. Prefer reapplying the known six-migration image pair above for an application/configuration regression. Do not roll back the database merely because a port-forward or OAuth connection fails. **DESTRUCTIVE, NOT EXECUTED ON PRODUCTION:** the following replaces the database with the explicitly selected backup. It loses writes after that backup and can conflict with retained broker messages. Obtain a maintenance window, pause publishers, stop Web/Worker, verify a fresh safety backup, and select the intended recovery point first. PostgreSQL must be healthy, original roles/passwords retained, and the backup must pass section 6. Do not run this block for validation or as a routine redeploy. ```sh # Linux SSH session; BACKUP_DIR must identify the deliberately selected verified dump. test -s "$BACKUP_DIR/database.dump" || exit 1 kn exec deployment/postgres -- dropdb -U notification_owner --force notification_service kn exec deployment/postgres -- createdb -U notification_owner -O notification_owner notification_service kn exec -i deployment/postgres -- pg_restore -U notification_owner -d notification_service --exit-on-error --single-transaction < "$BACKUP_DIR/database.dump" ``` `dropdb --force` terminates database connections. If restoration fails, leave writers stopped, retain the failed database for diagnosis and do not replay arbitrary messages. Inspect restored migration history. With a three-migration backup, follow section 7 to migrate forward and reapply grants before starting the six-migration application. With six migrations, verify grants and restart the pinned pair. Inspect broker queues and externally recorded outcomes before resuming publishers: database restore does not rewind acknowledgements or external email/push delivery. For total node/PV loss, this document does **not** claim a tested bare-metal restore. Recover the original mounted disk and private files first where possible. Rebuilding a cluster from exports requires reviewing namespace ownership, PV binding and node paths, Secret consistency, and sanitized manifests. Do not invoke the stale bootstrap package blindly. A separately rehearsed full disaster-recovery procedure remains a gap. ## 9. RabbitMQ and Worker checks ```sh kn exec deployment/rabbitmq -- rabbitmq-diagnostics -q check_running kn exec deployment/rabbitmq -- rabbitmqctl list_queues -p /notification name consumers messages_ready messages_unacknowledged kn exec deployment/rabbitmq -- rabbitmqctl list_permissions -p /notification ``` Expected after recovery: `notification.send-email` has exactly one consumer; `invemory.production.notification-results` has one consumer from Invemory. Zero queued messages was the verified baseline, not a requirement to purge legitimate new work. Development/staging result queues can have zero consumers. Broker nodename is fixed `rabbit@rabbitmq`; do not change it against the retained data directory. Worker read ACL covers `notification.send-email` only. Its configure/write ACLs cover the existing email result routes, including production/staging/development. The production publisher is `invemory_production`, staging is `invemory_staging`; an additional development account exists. Preserve all current accounts and definitions; do not replay an older definitions Secret to fix a consumer. Imported definitions hold credentials, not a complete live message backup. The shared default exchange does not isolate application `ServiceName` values between publishers. ## 10. Google OAuth/FCM egress and timeout diagnosis On 2026-10-24 DNS resolved both Google endpoints, but all four Web/Worker TLS probes timed out. Base default-deny allowed only internal services and SMTP 587. Applying `google-api-network.json` fixed all four TLS handshakes immediately without a restart; the real production UI then returned **OAuth access token obtained**. The policy selects only managed Web/Worker pods in `shared-notification`, IPv4 TCP 443 to Google's published API/service ranges. It grants no ingress or arbitrary internet access. Existing DNS policy allows TCP/UDP 53 only to kube-system CoreDNS. Calico OSS standard NetworkPolicy does not provide an FQDN allowlist. Google documents that individual APIs have no stable dedicated IP list: this snapshot uses `goog.json` minus `cloud.json` (297 CIDRs at verification), excluding customer Cloud ranges but still including other Google-operated services. This is an IP-level provider boundary, not a hostname-level boundary. [Google's range method](https://docs.cloud.google.com/vpc/docs/configure-private-google-access#ip-addr-defaults). Read-only probes from each actual runtime pod (7-second bound, no credentials): ```sh for component in web worker; do for endpoint in oauth2.googleapis.com fcm.googleapis.com; do printf '%s %s\n' "$component" "$endpoint" kn exec deployment/"$component" -- getent ahostsv4 "$endpoint" kn exec deployment/"$component" -- sh -c 'timeout 7 openssl s_client -connect "$1:443" -servername "$1" -verify_hostname "$1" -verify_return_error -brief