README.md

@@ -9,6 +9,7 @@ Production image references now use `registry.hrynco.com`, preserving the existi
 - [Service and whole-template API for AI clients](docs/email-template-definition-api.md)
 - [ItemTracker outbox email consumer](docs/itemtracker-outbox-email-consumer.md)
 - [Shared Notification Service Kubernetes package](docker/kubernetes/shared-notification/README.md)
+- [Production operations and recovery runbook](docs/production-operations.md) — actual Kubernetes environment, backup/restore, grants, Google egress and owner access.
 
 ## Windows host development
 

docker/kubernetes/shared-notification/README.md

@@ -1,5 +1,12 @@
 # Shared Notification Service on MicroK8s
 
+For the current six-migration production release, use the
+[production operations and recovery runbook](../../../docs/production-operations.md).
+The original bootstrap/deploy schema below predates push channels; do not run its
+full deploy against the current database. The current additive Google HTTPS policy
+is `google-api-network.json`; refresh it with `refresh-google-api-network.py` and
+review/server-dry-run it before applying. Existing SMTP and broker policies remain.
+
 This package deploys one shared Web/Worker, PostgreSQL 17, RabbitMQ 4 and a temporary
 Mailpit sandbox in `shared-notification`, on `mini-pc-2`. It does not expose public
 Ingress, NodePort or unauthenticated administration. Application administration,

docker/kubernetes/shared-notification/refresh-google-api-network.py

@@ -0,0 +1,62 @@
+#!/usr/bin/env python3
+"""Render Google API IPv4 egress; never applies changes to Kubernetes."""
+
+import argparse
+import ipaddress
+import json
+from pathlib import Path
+import urllib.request
+
+
+def fetch(name):
+    with urllib.request.urlopen(f"https://www.gstatic.com/ipranges/{name}.json", timeout=20) as response:
+        return json.load(response)
+
+
+def api_networks(google, cloud):
+    networks = [ipaddress.ip_network(p["ipv4Prefix"]) for p in google["prefixes"] if "ipv4Prefix" in p]
+    for prefix in cloud["prefixes"]:
+        if "ipv4Prefix" not in prefix:
+            continue
+        excluded = ipaddress.ip_network(prefix["ipv4Prefix"])
+        remaining = []
+        for network in networks:
+            if not network.overlaps(excluded):
+                remaining.append(network)
+            elif not network.subnet_of(excluded):
+                remaining.extend(network.address_exclude(excluded))
+        networks = remaining
+    return list(ipaddress.collapse_addresses(networks))
+
+
+def render(google, cloud):
+    networks = api_networks(google, cloud)
+    if not networks or any(not network.is_global for network in networks):
+        raise ValueError("Expected nonempty public Google API IPv4 ranges")
+    labels = {"app.kubernetes.io/name": "hrynco-shared-notification",
+              "app.kubernetes.io/managed-by": "hrynco-shared-notification"}
+    return {
+        "apiVersion": "networking.k8s.io/v1", "kind": "NetworkPolicy",
+        "metadata": {"name": "web-worker-to-google-apis", "namespace": "shared-notification",
+                     "labels": labels, "annotations": {
+                         "hrynco.com/google-ranges-date": google["creationTime"],
+                         "hrynco.com/cloud-ranges-date": cloud["creationTime"]}},
+        "spec": {"podSelector": {"matchLabels": labels, "matchExpressions": [
+            {"key": "app.kubernetes.io/component", "operator": "In", "values": ["web", "worker"]}]},
+            "policyTypes": ["Egress"], "egress": [{
+                "to": [{"ipBlock": {"cidr": str(network)}} for network in networks],
+                "ports": [{"protocol": "TCP", "port": 443}]}]},
+    }
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    policy = render(fetch("goog"), fetch("cloud"))
+    args.output.write_text(json.dumps(policy, indent=2) + "\n", encoding="utf-8")
+    print(f"Rendered {len(policy['spec']['egress'][0]['to'])} IPv4 ranges; review before applying.")
+
+
+if __name__ == "__main__":
+    main()

docs/production-operations.md

@@ -0,0 +1,564 @@
+# Notification Service: production operations and recovery
+
+Verified against the running Kubernetes service on **2026-10-24**. Commands below
+distinguish read-only checks, service interruptions and destructive recovery. Do not
+paste passwords, service-account JSON, tokens, Secret YAML or database dumps into
+tickets, chat, Git or CI logs. The administrative UI has no application authentication;
+keep it behind the existing loopback SSH tunnel.
+
+## 1. Actual environment and source of truth
+
+| Item | Actual value |
+| --- | --- |
+| Windows source checkout | `C:\projects\HrynCo\hrynco-notification-service` |
+| SSH target | `devops@192.168.2.44` (`mini-pc-2`; Linux username is lowercase) |
+| Windows SSH key | `C:\Users\agryn\.ssh\id_ed25519_devops`, unlocked in the owner's ssh-agent |
+| Kubernetes CLI | `/snap/microk8s/current/kubectl` |
+| Kubeconfig / context | `/home/devops/.kube/cluster-admin.conf` / `microk8s` |
+| Namespace / CNI | `shared-notification` / Calico `v3.29.3` |
+| Server package | `/home/devops/notification-package` |
+| Private files / local backups | `/home/devops/shared-notification-private` |
+| External storage | `/mnt/shared/notification-service`, XFS, UUID `c2c31640-a247-4988-bc7b-9ee9789464d1` |
+| Local admin URL | `http://localhost:15200/admin/services` |
+| Web service | `web:8080`, ClusterIP; no public Ingress or NodePort |
+| PostgreSQL | `postgres:5432`, database `notification_service` |
+| RabbitMQ | `rabbitmq:5672`, vhost `/notification`, management port `15672` |
+| Seq | `http://seq.observability.svc.cluster.local:8080` |
+| Sandbox SMTP / UI | `mailpit:1025` / `mailpit:8025`; no external relay |
+
+The repository's `docker/kubernetes/shared-notification/` owns the package. The
+server's `config.local.json` supplies the verified host/storage and namespace values;
+`artifacts.local.json` still describes the older bootstrap application images.
+`notification.py` owns the original foundation and credential bootstrap. Its
+`MIGRATIONS`, sandbox SQL and application artifact file are **not yet reconciled with
+the six-migration release below**. Do not run its full `bootstrap` or `deploy` command
+over this production release. It can overwrite current image/configuration intent or
+fail on removed `email_channels.service_name`.
+
+Use the targeted procedures in this document for the existing production resources.
+They do not depend on the historical one-off `it1074-deploy.py`. The original
+deployment audit is retained privately at
+`/home/devops/shared-notification-private/it1074/{before,result}.json`; `before.json`
+contains deployment specifications and fingerprints, not a restorable database backup.
+The one-off helper's initial GRANT used the wrong table case; do not rerun it.
+
+Network source of truth is additive: base policies from `notification.py`, existing
+`production-network.yaml` for SMTP and production broker access, and
+`google-api-network.json` for Google APIs. Never reconcile the sandbox render as the
+complete production setup. Reconcile and review all overlays together.
+
+## 2. Connect and inspect before changing anything
+
+Run in **Windows PowerShell** (no SSH alias is required or assumed):
+
+```powershell
+ssh -o StrictHostKeyChecking=yes -o HostKeyAlgorithms=ssh-ed25519 -o ConnectTimeout=10 -i C:/Users/agryn/.ssh/id_ed25519_devops devops@192.168.2.44
+```
+
+Inside that **Linux SSH session**, define these session-local helpers. All subsequent
+`k` and `kn` commands in this document run there, not in Windows PowerShell:
+
+```sh
+k() { /snap/microk8s/current/kubectl --kubeconfig=/home/devops/.kube/cluster-admin.conf --request-timeout=20s "$@"; }
+kn() { k --namespace=shared-notification "$@"; }
+cd /home/devops/notification-package
+k config current-context
+k get node mini-pc-2
+kn get deployment,pod,service,pvc,job
+k get pv shared-notification-postgres shared-notification-rabbitmq
+findmnt -T /mnt/shared/notification-service -o TARGET,SOURCE,FSTYPE,UUID
+kn get resourcequota
+```
+
+Expected: context `microk8s`, node Ready, the exact XFS UUID above, both PVCs Bound.
+Stop if the disk is not mounted: do not let an empty directory on the root disk stand
+in for persistent storage. PVs use `Retain` and node affinity to `mini-pc-2`:
+
+| PVC | PV | Path | Capacity |
+| --- | --- | --- | --- |
+| `postgres-data` | `shared-notification-postgres` | `/mnt/shared/notification-service/postgres` | 10Gi |
+| `rabbitmq-data` | `shared-notification-rabbitmq` | `/mnt/shared/notification-service/rabbitmq` | 5Gi |
+
+Do not delete PVCs, clear PV claimRefs, reformat storage or regenerate credentials to
+repair an application timeout. These are singleton workloads, not an HA database/broker.
+The quota allows five Deployments and three Jobs; the completed migrator/grants/sandbox
+Jobs can occupy all three Job slots. A new fourth Job can fail admission.
+
+## 3. Current application release and infrastructure images
+
+Application source: `d4ee8fc0c8036b0b48e1a481ef1a12a6a1c1c269`, merged PR 28.
+Image build: TeamCity `141783`, tag `main-34`. Deploy immutable digests:
+
+```sh
+WEB_IMAGE='registry.hrynco.com/hrynco.notification-service.web@sha256:69c51ac1ba7c1101472a0b7d45153544a95565cc544bba49908b1e4079195911'
+WORKER_IMAGE='registry.hrynco.com/hrynco.notification-service.worker@sha256:cc93cbfd82e1d7ff4d88845dfa90c8a1188c51d35775432d385fa5f23795a6a3'
+MIGRATOR_IMAGE='registry.hrynco.com/hrynco.notification-service.migrator@sha256:9975e05cb075f45a6a629233af4c56194731378f710feb719b205a68bebdb035'
+```
+
+The repositories include `notification-service` with a hyphen. Old bootstrap images
+used `notificationservice`; do not accidentally choose those. These digests are OCI
+indexes; the matching amd64 manifests are web `d69a6d7cdbc8c08d5a1234933a88579fd176a72b8c64fa02167c42d988eed0be`,
+worker `92d454a4139de55f5619070ed6b070c94db05749bbd3aca5d0e91d5a11ae8875`,
+migrator `a94766802609f4348adba4bdef4bc528275f317c3015b10bb97c8d530749adbd`.
+Registry manifest inspection must accept OCI indexes; a Docker-only Accept header can
+return a misleading 404.
+
+Retained infrastructure digests (do not upgrade during application recovery):
+
+| Image repository | Digest |
+| --- | --- |
+| `registry.hrynco.com/mirror/postgres` | `sha256:e31e3d5327d1806f6177827c9710643e4f35f7ab3f14d26d05332753d3e95ee0` |
+| `registry.hrynco.com/mirror/rabbitmq` | `sha256:ad995f859fce23a0ccd91c24cfac89d8f8cc57de4d708b58dcecea2697ad0d8c` |
+| `registry.hrynco.com/mirror/mailpit` | `sha256:13de4ffbd28f3089be6dff32afbc8210edb5a593d738fa877ae7272cc73aeb67` |
+
+Read-only image and readiness verification:
+
+```sh
+kn get deployment web worker -o custom-columns='NAME:.metadata.name,DESIRED:.spec.replicas,READY:.status.readyReplicas,IMAGE:.spec.template.spec.containers[*].image'
+kn get pods -o custom-columns='NAME:.metadata.name,READY:.status.containerStatuses[*].ready,RESTARTS:.status.containerStatuses[*].restartCount,IMAGEID:.status.containerStatuses[*].imageID'
+kn rollout status deployment/web --timeout=240s
+kn rollout status deployment/worker --timeout=240s
+```
+
+Web readiness queries `/api/v1/email-channels?serviceName=Invemory.Staging.Sandbox` on
+8080, every 10 seconds with a one-second timeout. Cold start under the 250m CPU limit
+can initially time out. Worker has no HTTP/readiness probe: **Ready is not proof of
+consumer readiness**. Verify RabbitMQ separately.
+
+## 4. Configuration and credentials
+
+Both Web and Worker use `notification-config` through `envFrom`:
+`DOTNET_ENVIRONMENT=Production`, `ASPNETCORE_ENVIRONMENT=Production`,
+`ASPNETCORE_HTTP_PORTS=8080`, RabbitMQ host `rabbitmq`, port `5672`, vhost
+`/notification`, Seq URL above, Serilog service `HrynCo.NotificationService` and
+environment `Shared`. Inspect nonsecret settings with:
+
+```sh
+kn get configmap notification-config -o yaml
+kn get secret
+```
+
+`notification-private` keys: `ownerConnectionString`, `runtimeConnectionString`,
+`postgresPassword`, `runtimePassword`, `rabbitAdminPassword`, `rabbitWorkerPassword`,
+`rabbitPublisherPassword`, `seqApiKey`. Web/Worker use the runtime connection;
+migrations use owner connection. Worker username is `notification_worker`.
+`registry-pull` contains `.dockerconfigjson`; `rabbit-definitions` contains
+`definitions.json` with sensitive password hashes. Do not print their values.
+
+The server private directory retains `credentials.json` and production publisher
+password/hash files. Registry authentication and Seq source files are under
+`/home/devops/notification-private/registry-auth.json` and `seq-key.txt`.
+Recover original credentials from a protected backup if lost; changing Kubernetes
+Secrets does not automatically change passwords in existing database/broker volumes.
+Do not run bootstrap to generate replacements against retained data.
+
+Firebase service-account JSON is stored in PostgreSQL `push_channels.credentials`
+(JSONB), without application encryption, by explicit owner decision. Database backups
+therefore contain private keys. Edit forms never return the saved JSON; leaving the
+field blank retains it. A project change requires matching new credentials. Current
+channel is attached to `Invemory-Prod`, Firebase project technical ID
+`storemate-hrynco-com`. Do not copy local development keys into production.
+
+`Check Firebase authorization` obtains an OAuth token using saved credentials when
+the project is unchanged. It does not send a push, prove FCM send permission, or prove
+device display. `Send test push` is a separate explicitly authorized action.
+The standalone test sends a Firebase `notification` payload without Invemory binding
+data. On the currently installed Invemory Android build it displays through the FCM
+SDK in background. In foreground, `InvemoryMessagingService.onMessageReceived` rejects
+the empty data map because `bindingId`, `version`, `messageId`, route and expiry are
+required; it does not call the base Capacitor handler. Thus accepted test messages can
+be invisible while Invemory is open. Samsung logs on 2026-10-24 show the FCM fallback
+notification channel and removal at 13:04:28 Europe/Kiev; the receiver does not log
+its rejection, so no individual foreground callback was proven. Do not diagnose this
+as egress failure or rotate tokens solely because foreground display is absent.
+The Worker push consumer is **not enabled** by this rollout: configuration and broker
+ACLs still cover email only. A saved active channel and successful OAuth check do not
+activate end-to-end production push processing. Do not enable it incidentally during
+an email service recovery.
+
+## 5. Database history and runtime grants
+
+Read-only history check:
+
+```sh
+kn exec deployment/postgres -- psql -U notification_owner -d notification_service -At -c 'SELECT "MigrationId" FROM "__EFMigrationsHistory" ORDER BY "MigrationId"'
+```
+
+Expected, in order:
+
+```text
+20260501214629_InitialCreate
+20260502154249_PendingChanges
+20261017072410_EmailTemplateLocalizations
+20261023223753_AddPushDeliveries
+20261024073202_LinkEmailChannelsToServices
+20261024080457_AddPushChannels
+```
+
+`notification_owner` is currently the PostgreSQL bootstrap superuser; keep it out of
+application connections. `notification_runtime` is a login with no superuser,
+createdb or createrole privileges. There are **no default ACL entries**: future tables
+do not inherit runtime CRUD automatically. For each new migration, grant only the
+tables needed by the application and persist the same SQL in `runtime-grants`.
+Do not add blanket default privileges as a workaround.
+
+The exact current push grants are below. `"PushDeliveries"` is case-sensitive;
+`push_deliveries` does not exist. This command is idempotent and was verified:
+
+```sh
+kn exec deployment/postgres -- psql -U notification_owner -d notification_service -v ON_ERROR_STOP=1 -c 'GRANT SELECT, INSERT, UPDATE, DELETE ON push_channels, "PushDeliveries" TO notification_runtime;'
+kn exec deployment/postgres -- psql -U notification_owner -d notification_service -At -c "SELECT t,has_table_privilege('notification_runtime',t,'SELECT,INSERT,UPDATE,DELETE') FROM unnest(ARRAY['push_channels','\"PushDeliveries\"']) t"
+```
+
+Both checks must return `t`. Existing runtime CRUD covers `email_channels`,
+`email_templates`, `email_channel_usage`, `services`, `service_languages`,
+`email_template_definitions`; schema USAGE and database CONNECT are granted, schema
+CREATE is revoked from PUBLIC. The live `runtime-grants` ConfigMap already includes
+the correct push grant. To persist it again without rotating passwords:
+
+```sh
+python3 - <<'PY'
+import json, subprocess
+k = ['/snap/microk8s/current/kubectl', '--kubeconfig=/home/devops/.kube/cluster-admin.conf', '-n', 'shared-notification']
+value = json.loads(subprocess.check_output(k + ['get','configmap','runtime-grants','-o','json']))
+grant = 'GRANT SELECT, INSERT, UPDATE, DELETE ON push_channels, "PushDeliveries" TO notification_runtime;'
+sql = value['data']['grants.sql']
+if grant not in sql:
+    subprocess.run(k + ['patch','configmap','runtime-grants','--type=merge','-p',json.dumps({'data':{'grants.sql':sql+'\n'+grant+'\n'}})], check=True)
+PY
+```
+
+Do not execute the entire grants ConfigMap manually without its intended
+`RUNTIME_PASSWORD` secret environment: its bootstrap SQL also manages the runtime role
+password. Do not rerun the old sandbox job; channel ownership now uses `service_id`.
+The migration maps the legacy `StoreMate-Prod` channel to existing `Invemory-Prod`.
+The old identifier is historical migration input, not a new channel/service name.
+
+## 6. Backup before migration or restore
+
+**Service interruption:** stop Web and Worker writers first. This does not stop
+external publishers; coordinate a publisher pause if a consistent broker/database
+recovery point is needed. Never purge queues as a substitute.
+
+```sh
+kn scale deployment/worker deployment/web --replicas=0
+kn wait --for=delete pod -l 'app.kubernetes.io/component in (web,worker)' --timeout=120s
+```
+
+The canonical backup implementation is safe independently of its old deploy schema.
+This maintained source command creates a custom-format dump with mode 0600, checks
+the `PGDMP` signature and `pg_restore --list`, and writes SHA-256, bytes and migration
+history to `manifest.json`. Run from the server package; only these existing functions
+are used, not `deploy` or sandbox SQL:
+
+```sh
+python3 - <<'PY'
+import types
+import notification as n
+c = n.Cluster(types.SimpleNamespace(kubectl='/snap/microk8s/current/kubectl', kubeconfig='/home/devops/.kube/cluster-admin.conf'))
+with n.lock('/home/devops/shared-notification-private'):
+    assert all(c.get('deployment', x)['spec']['replicas'] == 0 for x in ('web','worker'))
+    print(n.backup(c, '/home/devops/shared-notification-private'))
+PY
+```
+
+Record the returned folder. Pre-migration backup from this rollout:
+`/home/devops/shared-notification-private/backup-20261024T094457816933Z`.
+It has only the **first three migrations**, predates the owner's production push
+channel, and is not a backup of today's six-migration state. Take a fresh backup before
+any recovery. A local dump does not protect against loss of this host or disk.
+No offsite copy or full disaster restore has been proven.
+
+Verify a selected backup without restoring it (set the exact existing folder):
+
+```sh
+BACKUP_DIR=/home/devops/shared-notification-private/backup-20261024T094457816933Z
+export BACKUP_DIR
+python3 - <<'PY'
+import hashlib,json,os,pathlib
+p=pathlib.Path(os.environ['BACKUP_DIR'])
+m=json.loads((p/'manifest.json').read_text()); data=(p/'database.dump').read_bytes()
+assert len(data)==m['bytes'] and hashlib.sha256(data).hexdigest()==m['sha256']
+assert data.startswith(b'PGDMP')
+print('Checksum and length verified; migration history:', m['history'])
+PY
+kn exec -i deployment/postgres -- pg_restore --list < "$BACKUP_DIR/database.dump" >/dev/null
+```
+
+Protect an additional recovery export with restrictive permissions. The following
+commands contain secrets in their **files**, never print or commit them. They are
+operator recovery instructions, not proof that an offsite backup exists:
+
+```sh
+umask 077
+RECOVERY_DIR=/home/devops/shared-notification-private/recovery-$(date -u +%Y%m%dT%H%M%SZ)
+mkdir "$RECOVERY_DIR"
+kn get secret -o json > "$RECOVERY_DIR/secrets.json"
+kn get deployment,service,configmap,networkpolicy,resourcequota,limitrange,pvc -o json > "$RECOVERY_DIR/namespace-resources.json"
+k get pv shared-notification-postgres shared-notification-rabbitmq -o json > "$RECOVERY_DIR/pvs.json"
+```
+
+These are inventories with server-assigned fields, not manifests to blindly apply
+to a new cluster. Keep protected copies of the package config, kubeconfig and private
+credential files with your backup system. RabbitMQ definitions do not back up queued
+messages. A broker volume backup requires a coordinated stopped broker and publisher
+maintenance window; no live-volume tar or restore drill was performed here.
+
+## 7. Recover/redeploy this release on retained infrastructure
+
+Prerequisites: correct disk/PVCs, original Secrets, PostgreSQL and RabbitMQ Ready,
+no concurrent deployment, verified fresh backup, exact pinned image variables from
+section 3. Web/Worker remain stopped from section 6.
+
+If history already contains exactly the six entries above, skip migration. If it is
+exactly the original three-entry prefix, use the following maintained recipe from
+the package directory. It creates the same bounded migrator Job used for the rollout,
+references the existing owner Secret, and replaces only the completed owned migrator
+Job to fit the quota. Other histories require inspection; do not guess or edit EF history.
+
+```sh
+export MIGRATOR_IMAGE
+python3 - <<'PY'
+import os,types
+import notification as n
+c=n.Cluster(types.SimpleNamespace(kubectl='/snap/microk8s/current/kubectl',kubeconfig='/home/devops/.kube/cluster-admin.conf'))
+prefix=['20260501214629_InitialCreate','20260502154249_PendingChanges','20261017072410_EmailTemplateLocalizations']
+target=prefix+['20261023223753_AddPushDeliveries','20261024073202_LinkEmailChannelsToServices','20261024080457_AddPushChannels']
+with n.lock('/home/devops/shared-notification-private'):
+    assert all(c.get('deployment',x)['spec']['replicas']==0 for x in ('web','worker'))
+    assert c.history()==prefix, 'Unexpected history; stop and inspect'
+    old=c.get('job','notification-migrator')
+    assert old and old['metadata']['labels']['app.kubernetes.io/managed-by']==n.MANAGED and old['status'].get('succeeded')==1
+    job=n.job('migrator',os.environ['MIGRATOR_IMAGE'],environment=[n.env('DOTNET_ENVIRONMENT','Production'),n.env('App__ConnectionString',secret='notification-private',key='ownerConnectionString')])
+    n.run_job(c,job)
+    assert c.history()==target
+    print('All six migrations verified; apply runtime grants before starting writers.')
+PY
+```
+
+Apply and verify the grants in section 5. Inspect migrated channel ownership without
+reading credentials:
+
+```sh
+kn exec deployment/postgres -- psql -U notification_owner -d notification_service -At -c 'SELECT s.name,count(*) FROM email_channels c JOIN services s ON s.id=c.service_id GROUP BY s.name ORDER BY s.name'
+```
+
+Set only application images, retaining Secrets, volumes, selectors and resources:
+
+```sh
+kn set image deployment/web web="$WEB_IMAGE"
+kn set image deployment/worker worker="$WORKER_IMAGE"
+kn scale deployment/web --replicas=1
+kn rollout status deployment/web --timeout=240s
+kn scale deployment/worker --replicas=1
+kn rollout status deployment/worker --timeout=240s
+```
+
+Then verify actual pod imageIDs, six migrations, grants, consumers, UI and OAuth.
+If migration fails, keep writers stopped while inspecting the failed Job and database
+history. If rollout fails, inspect events/logs before changing anything else. Never
+leave replicas at zero without reporting the specific failure and recovery action.
+
+## 8. Rollback and destructive database restore
+
+An image-only rollback is allowed only when the target image supports the **current**
+schema. `kubectl rollout undo` can select the old pre-ServiceId binary and is unsafe
+after this migration. Prefer reapplying the known six-migration image pair above for
+an application/configuration regression. Do not roll back the database merely because
+a port-forward or OAuth connection fails.
+
+**DESTRUCTIVE, NOT EXECUTED ON PRODUCTION:** the following replaces the database with
+the explicitly selected backup. It loses writes after that backup and can conflict
+with retained broker messages. Obtain a maintenance window, pause publishers, stop
+Web/Worker, verify a fresh safety backup, and select the intended recovery point first.
+PostgreSQL must be healthy, original roles/passwords retained, and the backup must
+pass section 6. Do not run this block for validation or as a routine redeploy.
+
+```sh
+# Linux SSH session; BACKUP_DIR must identify the deliberately selected verified dump.
+test -s "$BACKUP_DIR/database.dump" || exit 1
+kn exec deployment/postgres -- dropdb -U notification_owner --force notification_service
+kn exec deployment/postgres -- createdb -U notification_owner -O notification_owner notification_service
+kn exec -i deployment/postgres -- pg_restore -U notification_owner -d notification_service --exit-on-error --single-transaction < "$BACKUP_DIR/database.dump"
+```
+
+`dropdb --force` terminates database connections. If restoration fails, leave writers
+stopped, retain the failed database for diagnosis and do not replay arbitrary messages.
+Inspect restored migration history. With a three-migration backup, follow section 7
+to migrate forward and reapply grants before starting the six-migration application.
+With six migrations, verify grants and restart the pinned pair. Inspect broker queues
+and externally recorded outcomes before resuming publishers: database restore does
+not rewind acknowledgements or external email/push delivery.
+
+For total node/PV loss, this document does **not** claim a tested bare-metal restore.
+Recover the original mounted disk and private files first where possible. Rebuilding a
+cluster from exports requires reviewing namespace ownership, PV binding and node paths,
+Secret consistency, and sanitized manifests. Do not invoke the stale bootstrap package
+blindly. A separately rehearsed full disaster-recovery procedure remains a gap.
+
+## 9. RabbitMQ and Worker checks
+
+```sh
+kn exec deployment/rabbitmq -- rabbitmq-diagnostics -q check_running
+kn exec deployment/rabbitmq -- rabbitmqctl list_queues -p /notification name consumers messages_ready messages_unacknowledged
+kn exec deployment/rabbitmq -- rabbitmqctl list_permissions -p /notification
+```
+
+Expected after recovery: `notification.send-email` has exactly one consumer;
+`invemory.production.notification-results` has one consumer from Invemory. Zero queued
+messages was the verified baseline, not a requirement to purge legitimate new work.
+Development/staging result queues can have zero consumers. Broker nodename is fixed
+`rabbit@rabbitmq`; do not change it against the retained data directory.
+
+Worker read ACL covers `notification.send-email` only. Its configure/write ACLs cover
+the existing email result routes, including production/staging/development. The
+production publisher is `invemory_production`, staging is `invemory_staging`; an
+additional development account exists. Preserve all current accounts and definitions;
+do not replay an older definitions Secret to fix a consumer. Imported definitions
+hold credentials, not a complete live message backup. The shared default exchange
+does not isolate application `ServiceName` values between publishers.
+
+## 10. Google OAuth/FCM egress and timeout diagnosis
+
+On 2026-10-24 DNS resolved both Google endpoints, but all four Web/Worker TLS probes
+timed out. Base default-deny allowed only internal services and SMTP 587. Applying
+`google-api-network.json` fixed all four TLS handshakes immediately without a restart;
+the real production UI then returned **OAuth access token obtained**.
+
+The policy selects only managed Web/Worker pods in `shared-notification`, IPv4 TCP
+443 to Google's published API/service ranges. It grants no ingress or arbitrary
+internet access. Existing DNS policy allows TCP/UDP 53 only to kube-system CoreDNS.
+Calico OSS standard NetworkPolicy does not provide an FQDN allowlist. Google documents
+that individual APIs have no stable dedicated IP list: this snapshot uses `goog.json`
+minus `cloud.json` (297 CIDRs at verification), excluding customer Cloud ranges but
+still including other Google-operated services. This is an IP-level provider boundary,
+not a hostname-level boundary. [Google's range method](https://docs.cloud.google.com/vpc/docs/configure-private-google-access#ip-addr-defaults).
+
+Read-only probes from each actual runtime pod (7-second bound, no credentials):
+
+```sh
+for component in web worker; do
+  for endpoint in oauth2.googleapis.com fcm.googleapis.com; do
+    printf '%s %s\n' "$component" "$endpoint"
+    kn exec deployment/"$component" -- getent ahostsv4 "$endpoint"
+    kn exec deployment/"$component" -- sh -c 'timeout 7 openssl s_client -connect "$1:443" -servername "$1" -verify_hostname "$1" -verify_return_error -brief </dev/null' sh "$endpoint"
+  done
+done
+kn get networkpolicy
+```
+
+Expected TLS connection and certificate verification OK. Exit 124 means timeout;
+name-resolution errors indicate DNS rather than Firebase credentials. Do not disable
+certificate verification. A successful TLS check alone does not prove OAuth succeeds.
+Use the saved channel's **Check Firebase authorization** action afterwards; no key needs
+to be reentered when its project is unchanged. Its current application timeout is 30s.
+
+Refresh the range snapshot when Google updates its ranges; no automatic updater is
+installed. The renderer downloads only public feeds, generates a candidate, and never
+applies it. Keep the reviewed snapshot in Git. On Windows PowerShell, from this repo:
+
+```powershell
+python docker/kubernetes/shared-notification/refresh-google-api-network.py --output C:/projects/ItemTracker/tmp/IT-1074-push/google-api-network.candidate.json
+git diff --no-index -- docker/kubernetes/shared-notification/google-api-network.json C:/projects/ItemTracker/tmp/IT-1074-push/google-api-network.candidate.json
+```
+
+Review source dates and CIDR changes. A diff exit code of 1 means files differ. Confirm
+the current DNS answers are covered and all addresses remain public. Copy the reviewed
+candidate into the repository manifest, then transfer only that nonsecret file:
+
+```powershell
+Copy-Item -LiteralPath C:/projects/ItemTracker/tmp/IT-1074-push/google-api-network.candidate.json -Destination docker/kubernetes/shared-notification/google-api-network.json
+scp -i C:/Users/agryn/.ssh/id_ed25519_devops docker/kubernetes/shared-notification/google-api-network.json devops@192.168.2.44:/home/devops/notification-package/google-api-network.json
+```
+
+On the server:
+
+```sh
+kn apply --dry-run=server -f /home/devops/notification-package/google-api-network.json
+kn apply -f /home/devops/notification-package/google-api-network.json
+```
+
+Repeat all four TLS probes and OAuth. Keep the prior reviewed manifest for policy
+rollback. Do not replace a failed DNS answer with a permanent single /32 rule: Google
+addresses rotate. Apply the production SMTP/broker overlay separately only when those
+policies need reconciliation; Google recovery does not require modifying other namespaces.
+
+## 11. Owner's port-forward: exact address and restart
+
+Run in a dedicated **Windows PowerShell** terminal. Leave the process running:
+
+```powershell
+ssh -tt -o StrictHostKeyChecking=yes -o HostKeyAlgorithms=ssh-ed25519 -o ConnectTimeout=10 -o ExitOnForwardFailure=yes -o ServerAliveInterval=15 -o ServerAliveCountMax=2 -i C:/Users/agryn/.ssh/id_ed25519_devops -L 127.0.0.1:15200:127.0.0.1:15200 devops@192.168.2.44 'exec /snap/microk8s/current/kubectl --kubeconfig=/home/devops/.kube/cluster-admin.conf --namespace=shared-notification --request-timeout=15s port-forward --address=127.0.0.1 service/web 15200:8080'
+```
+
+Open `http://localhost:15200/admin/services`. A service port-forward binds to a selected
+pod; replacing that pod can break the existing tunnel even when its successor is Ready.
+After rollout, verify **this exact owner URL**, not only a temporary alternate port:
+
+```powershell
+Invoke-WebRequest -Uri http://localhost:15200/admin/services -UseBasicParsing -TimeoutSec 10 | Select-Object StatusCode
+Get-NetTCPConnection -LocalPort 15200 -ErrorAction SilentlyContinue | Select-Object LocalAddress,State,OwningProcess
+Get-CimInstance Win32_Process -Filter "Name = 'ssh.exe'" | Select-Object ProcessId,CommandLine
+```
+
+If broken, stop only the identified 15200 tunnel with Ctrl+C in its own terminal and
+rerun the exact command. Do not kill all SSH processes or a working tunnel belonging
+to another task. If the remote port is occupied, inspect its process/owner before
+stopping anything. For a background helper use `Start-Process -WindowStyle Hidden`,
+retain its PID and output paths; never leave an untracked visible helper window.
+
+For optional broker or Mailpit inspection, use separate loopback tunnels with the same
+SSH options and matching local/remote port pairs: RabbitMQ `15672:15672` forwarding
+`service/rabbitmq`, or Mailpit `18025:8025` forwarding `service/mailpit`. Do not bind
+administration to `0.0.0.0` or expose it publicly.
+
+## 12. Incident checklist and evidence
+
+Read logs locally; messages may contain recipient or provider details. Do not paste raw
+logs into public places. Narrow by time and component:
+
+```sh
+kn get events --sort-by=.metadata.creationTimestamp
+kn logs deployment/web --since=10m --tail=100
+kn logs deployment/worker --since=10m --tail=100
+kn logs job/notification-migrator --tail=100
+kn get configmap runtime-grants -o yaml
+```
+
+| Symptom | Next check |
+| --- | --- |
+| UI connection refused, pods Ready | Exact localhost:15200 tunnel; restart only stale forward |
+| OAuth times out after 30s | DNS and TLS from Web; Google policy and current feed coverage |
+| OAuth rejected immediately | Correct Firebase project/service account, key validity; no credential logging |
+| Permission denied for push table | Quoted `"PushDeliveries"`, runtime grants and connection role |
+| Web stays unready | Events, cold-start CPU, database query, schema/history; do not remove readiness |
+| Worker Ready but no deliveries | RabbitMQ consumer count/ACLs and runtime logs; process readiness is insufficient |
+| ImagePullBackOff | Exact digest, registry reachability, existing pull Secret; do not regenerate credentials |
+| Pod storage guard failure | Real filesystem UUID/mount/marker before any restart or data-directory repair |
+
+The platform may log a missing optional `libgssapi_krb5.so.2`; in this rollout Web still
+became ready and used its password connection. Diagnose the actual failing operation
+instead of assuming that warning caused an OAuth network timeout.
+
+## 13. CI distinction and verification limits
+
+TeamCity `HrynCo_NotificationService_BuildAndPush` build 141783 produced the verified
+images. `HrynCo_NotificationService_DeployProduction` build 141784 deployed the **old
+Docker Compose target on mini-pc-1**, not this Kubernetes namespace. A green job there
+is not Kubernetes rollout or owner UI proof. Kubernetes was manually deployed using
+the retained production objects and pinned digests described here.
+
+Contracts publishing is a separate NuGet operation. Build 141789 was blocked by
+incompatible agent requirements (`teamcity.agent.name=ItemTrackerDotNet10` and
+`system.agent.name=mini-pc-1`). Do not publish Contracts manually or alter CI as part of
+runtime recovery. Some current checks also return success despite missing dotnet or
+unavailable Docker test prerequisites; rely on actual test output, not status alone.
+
+Verified: pinned running images, six migration entries, runtime table privileges,
+preserved Secrets/PVC identities and existing email contents, custom-format backup
+checksum/listing, Web/Worker Ready with zero restarts, email/result consumers, owner
+URL UI, four OAuth/FCM TLS paths, and successful real production OAuth check. No push
+was sent. Destructive database restore, full host loss recovery, offsite backup,
+RabbitMQ message restore and end-to-end production push delivery were **not tested**.
+Publishing and merging these documentation/egress follow-ups require a separate
+delivery request; a local commit does not authorize those operations. IT-1074 remains open.