hrynco-notification-service/docs/production-operations.md

Notification Service: production operations and recovery

Verified against the running Kubernetes service on 2026-10-24. Commands below distinguish read-only checks, service interruptions and destructive recovery. Do not paste passwords, service-account JSON, tokens, Secret YAML or database dumps into tickets, chat, Git or CI logs. The administrative UI has no application authentication; keep it behind the existing loopback SSH tunnel.

1. Actual environment and source of truth

Item	Actual value
Windows source checkout	C:\projects\HrynCo\hrynco-notification-service
SSH target	devops@192.168.2.44 (mini-pc-2; Linux username is lowercase)
Windows SSH key	C:\Users\agryn\.ssh\id_ed25519_devops, unlocked in the owner's ssh-agent
Kubernetes CLI	/snap/microk8s/current/kubectl
Kubeconfig / context	/home/devops/.kube/cluster-admin.conf / microk8s
Namespace / CNI	shared-notification / Calico v3.29.3
Server package	/home/devops/notification-package
Private files / local backups	/home/devops/shared-notification-private
External storage	/mnt/shared/notification-service, XFS, UUID c2c31640-a247-4988-bc7b-9ee9789464d1
Local admin URL	http://localhost:15200/admin/services
Web service	web:8080, ClusterIP; no public Ingress or NodePort
PostgreSQL	postgres:5432, database notification_service
RabbitMQ	rabbitmq:5672, vhost /notification, management port 15672
Seq	http://seq.observability.svc.cluster.local:8080
Sandbox SMTP / UI	mailpit:1025 / mailpit:8025; no external relay

The repository's docker/kubernetes/shared-notification/ owns the package. The server's config.local.json supplies the verified host/storage and namespace values; artifacts.local.json still describes the older bootstrap application images. notification.py owns the original foundation and credential bootstrap. Its MIGRATIONS, sandbox SQL and application artifact file are not yet reconciled with the six-migration release below. Do not run its full bootstrap or deploy command over this production release. It can overwrite current image/configuration intent or fail on removed email_channels.service_name.

Use the targeted procedures in this document for the existing production resources. They do not depend on the historical one-off it1074-deploy.py. The original deployment audit is retained privately at /home/devops/shared-notification-private/it1074/{before,result}.json; before.json contains deployment specifications and fingerprints, not a restorable database backup. The one-off helper's initial GRANT used the wrong table case; do not rerun it.

Network source of truth is additive: base policies from notification.py, existing production-network.yaml for SMTP and production broker access, and google-api-network.json for Google APIs. Never reconcile the sandbox render as the complete production setup. Reconcile and review all overlays together.

2. Connect and inspect before changing anything

Run in Windows PowerShell (no SSH alias is required or assumed):

ssh -o StrictHostKeyChecking=yes -o HostKeyAlgorithms=ssh-ed25519 -o ConnectTimeout=10 -i C:/Users/agryn/.ssh/id_ed25519_devops devops@192.168.2.44

Inside that Linux SSH session, define these session-local helpers. All subsequent k and kn commands in this document run there, not in Windows PowerShell:

k() { /snap/microk8s/current/kubectl --kubeconfig=/home/devops/.kube/cluster-admin.conf --request-timeout=20s "$@"; }
kn() { k --namespace=shared-notification "$@"; }
cd /home/devops/notification-package
k config current-context
k get node mini-pc-2
kn get deployment,pod,service,pvc,job
k get pv shared-notification-postgres shared-notification-rabbitmq
findmnt -T /mnt/shared/notification-service -o TARGET,SOURCE,FSTYPE,UUID
kn get resourcequota

Expected: context microk8s, node Ready, the exact XFS UUID above, both PVCs Bound. Stop if the disk is not mounted: do not let an empty directory on the root disk stand in for persistent storage. PVs use Retain and node affinity to mini-pc-2:

PVC	PV	Path	Capacity
postgres-data	shared-notification-postgres	/mnt/shared/notification-service/postgres	10Gi
rabbitmq-data	shared-notification-rabbitmq	/mnt/shared/notification-service/rabbitmq	5Gi

Do not delete PVCs, clear PV claimRefs, reformat storage or regenerate credentials to repair an application timeout. These are singleton workloads, not an HA database/broker. The quota allows five Deployments and three Jobs; the completed migrator/grants/sandbox Jobs can occupy all three Job slots. A new fourth Job can fail admission.

3. Current application release and infrastructure images

Application source: d4ee8fc0c8036b0b48e1a481ef1a12a6a1c1c269, merged PR 28. Image build: TeamCity 141783, tag main-34. Deploy immutable digests:

WEB_IMAGE='registry.hrynco.com/hrynco.notification-service.web@sha256:69c51ac1ba7c1101472a0b7d45153544a95565cc544bba49908b1e4079195911'
WORKER_IMAGE='registry.hrynco.com/hrynco.notification-service.worker@sha256:cc93cbfd82e1d7ff4d88845dfa90c8a1188c51d35775432d385fa5f23795a6a3'
MIGRATOR_IMAGE='registry.hrynco.com/hrynco.notification-service.migrator@sha256:9975e05cb075f45a6a629233af4c56194731378f710feb719b205a68bebdb035'

The repositories include notification-service with a hyphen. Old bootstrap images used notificationservice; do not accidentally choose those. These digests are OCI indexes; the matching amd64 manifests are web d69a6d7cdbc8c08d5a1234933a88579fd176a72b8c64fa02167c42d988eed0be, worker 92d454a4139de55f5619070ed6b070c94db05749bbd3aca5d0e91d5a11ae8875, migrator a94766802609f4348adba4bdef4bc528275f317c3015b10bb97c8d530749adbd. Registry manifest inspection must accept OCI indexes; a Docker-only Accept header can return a misleading 404.

Retained infrastructure digests (do not upgrade during application recovery):

Image repository	Digest
registry.hrynco.com/mirror/postgres	sha256:e31e3d5327d1806f6177827c9710643e4f35f7ab3f14d26d05332753d3e95ee0
registry.hrynco.com/mirror/rabbitmq	sha256:ad995f859fce23a0ccd91c24cfac89d8f8cc57de4d708b58dcecea2697ad0d8c
registry.hrynco.com/mirror/mailpit	sha256:13de4ffbd28f3089be6dff32afbc8210edb5a593d738fa877ae7272cc73aeb67

Read-only image and readiness verification:

kn get deployment web worker -o custom-columns='NAME:.metadata.name,DESIRED:.spec.replicas,READY:.status.readyReplicas,IMAGE:.spec.template.spec.containers[*].image'
kn get pods -o custom-columns='NAME:.metadata.name,READY:.status.containerStatuses[*].ready,RESTARTS:.status.containerStatuses[*].restartCount,IMAGEID:.status.containerStatuses[*].imageID'
kn rollout status deployment/web --timeout=240s
kn rollout status deployment/worker --timeout=240s

Web readiness queries /api/v1/email-channels?serviceName=Invemory.Staging.Sandbox on 8080, every 10 seconds with a one-second timeout. Cold start under the 250m CPU limit can initially time out. Worker has no HTTP/readiness probe: Ready is not proof of consumer readiness. Verify RabbitMQ separately.

4. Configuration and credentials

Both Web and Worker use notification-config through envFrom: DOTNET_ENVIRONMENT=Production, ASPNETCORE_ENVIRONMENT=Production, ASPNETCORE_HTTP_PORTS=8080, RabbitMQ host rabbitmq, port 5672, vhost /notification, Seq URL above, Serilog service HrynCo.NotificationService and environment Shared. Inspect nonsecret settings with:

kn get configmap notification-config -o yaml
kn get secret

notification-private keys: ownerConnectionString, runtimeConnectionString, postgresPassword, runtimePassword, rabbitAdminPassword, rabbitWorkerPassword, rabbitPublisherPassword, seqApiKey. Web/Worker use the runtime connection; migrations use owner connection. Worker username is notification_worker. registry-pull contains .dockerconfigjson; rabbit-definitions contains definitions.json with sensitive password hashes. Do not print their values.

The server private directory retains credentials.json and production publisher password/hash files. Registry authentication and Seq source files are under /home/devops/notification-private/registry-auth.json and seq-key.txt. Recover original credentials from a protected backup if lost; changing Kubernetes Secrets does not automatically change passwords in existing database/broker volumes. Do not run bootstrap to generate replacements against retained data.

Firebase service-account JSON is stored in PostgreSQL push_channels.credentials (JSONB), without application encryption, by explicit owner decision. Database backups therefore contain private keys. Edit forms never return the saved JSON; leaving the field blank retains it. A project change requires matching new credentials. Current channel is attached to Invemory-Prod, Firebase project technical ID storemate-hrynco-com. Do not copy local development keys into production.

Check Firebase authorization obtains an OAuth token using saved credentials when the project is unchanged. It does not send a push, prove FCM send permission, or prove device display. Send test push is a separate explicitly authorized action. The standalone test sends a Firebase notification payload without Invemory binding data. On the currently installed Invemory Android build it displays through the FCM SDK in background. In foreground, InvemoryMessagingService.onMessageReceived rejects the empty data map because bindingId, version, messageId, route and expiry are required; it does not call the base Capacitor handler. Thus accepted test messages can be invisible while Invemory is open. Samsung logs on 2026-10-24 show the FCM fallback notification channel and removal at 13:04:28 Europe/Kiev; the receiver does not log its rejection, so no individual foreground callback was proven. Do not diagnose this as egress failure or rotate tokens solely because foreground display is absent. The Worker push consumer is not enabled by this rollout: configuration and broker ACLs still cover email only. A saved active channel and successful OAuth check do not activate end-to-end production push processing. Do not enable it incidentally during an email service recovery.

5. Database history and runtime grants

Read-only history check:

kn exec deployment/postgres -- psql -U notification_owner -d notification_service -At -c 'SELECT "MigrationId" FROM "__EFMigrationsHistory" ORDER BY "MigrationId"'

Expected, in order:

20260501214629_InitialCreate
20260502154249_PendingChanges
20261017072410_EmailTemplateLocalizations
20261023223753_AddPushDeliveries
20261024073202_LinkEmailChannelsToServices
20261024080457_AddPushChannels

notification_owner is currently the PostgreSQL bootstrap superuser; keep it out of application connections. notification_runtime is a login with no superuser, createdb or createrole privileges. There are no default ACL entries: future tables do not inherit runtime CRUD automatically. For each new migration, grant only the tables needed by the application and persist the same SQL in runtime-grants. Do not add blanket default privileges as a workaround.

The exact current push grants are below. "PushDeliveries" is case-sensitive; push_deliveries does not exist. This command is idempotent and was verified:

kn exec deployment/postgres -- psql -U notification_owner -d notification_service -v ON_ERROR_STOP=1 -c 'GRANT SELECT, INSERT, UPDATE, DELETE ON push_channels, "PushDeliveries" TO notification_runtime;'
kn exec deployment/postgres -- psql -U notification_owner -d notification_service -At -c "SELECT t,has_table_privilege('notification_runtime',t,'SELECT,INSERT,UPDATE,DELETE') FROM unnest(ARRAY['push_channels','\"PushDeliveries\"']) t"

Both checks must return t. Existing runtime CRUD covers email_channels, email_templates, email_channel_usage, services, service_languages, email_template_definitions; schema USAGE and database CONNECT are granted, schema CREATE is revoked from PUBLIC. The live runtime-grants ConfigMap already includes the correct push grant. To persist it again without rotating passwords:

python3 - <<'PY'
import json, subprocess
k = ['/snap/microk8s/current/kubectl', '--kubeconfig=/home/devops/.kube/cluster-admin.conf', '-n', 'shared-notification']
value = json.loads(subprocess.check_output(k + ['get','configmap','runtime-grants','-o','json']))
grant = 'GRANT SELECT, INSERT, UPDATE, DELETE ON push_channels, "PushDeliveries" TO notification_runtime;'
sql = value['data']['grants.sql']
if grant not in sql:
    subprocess.run(k + ['patch','configmap','runtime-grants','--type=merge','-p',json.dumps({'data':{'grants.sql':sql+'\n'+grant+'\n'}})], check=True)
PY

Do not execute the entire grants ConfigMap manually without its intended RUNTIME_PASSWORD secret environment: its bootstrap SQL also manages the runtime role password. Do not rerun the old sandbox job; channel ownership now uses service_id. The migration maps the legacy StoreMate-Prod channel to existing Invemory-Prod. The old identifier is historical migration input, not a new channel/service name.

6. Backup before migration or restore

Service interruption: stop Web and Worker writers first. This does not stop external publishers; coordinate a publisher pause if a consistent broker/database recovery point is needed. Never purge queues as a substitute.

kn scale deployment/worker deployment/web --replicas=0
kn wait --for=delete pod -l 'app.kubernetes.io/component in (web,worker)' --timeout=120s

The canonical backup implementation is safe independently of its old deploy schema. This maintained source command creates a custom-format dump with mode 0600, checks the PGDMP signature and pg_restore --list, and writes SHA-256, bytes and migration history to manifest.json. Run from the server package; only these existing functions are used, not deploy or sandbox SQL:

python3 - <<'PY'
import types
import notification as n
c = n.Cluster(types.SimpleNamespace(kubectl='/snap/microk8s/current/kubectl', kubeconfig='/home/devops/.kube/cluster-admin.conf'))
with n.lock('/home/devops/shared-notification-private'):
    assert all(c.get('deployment', x)['spec']['replicas'] == 0 for x in ('web','worker'))
    print(n.backup(c, '/home/devops/shared-notification-private'))
PY

Record the returned folder. Pre-migration backup from this rollout: /home/devops/shared-notification-private/backup-20261024T094457816933Z. It has only the first three migrations, predates the owner's production push channel, and is not a backup of today's six-migration state. Take a fresh backup before any recovery. A local dump does not protect against loss of this host or disk. No offsite copy or full disaster restore has been proven.

Verify a selected backup without restoring it (set the exact existing folder):

BACKUP_DIR=/home/devops/shared-notification-private/backup-20261024T094457816933Z
export BACKUP_DIR
python3 - <<'PY'
import hashlib,json,os,pathlib
p=pathlib.Path(os.environ['BACKUP_DIR'])
m=json.loads((p/'manifest.json').read_text()); data=(p/'database.dump').read_bytes()
assert len(data)==m['bytes'] and hashlib.sha256(data).hexdigest()==m['sha256']
assert data.startswith(b'PGDMP')
print('Checksum and length verified; migration history:', m['history'])
PY
kn exec -i deployment/postgres -- pg_restore --list < "$BACKUP_DIR/database.dump" >/dev/null

Protect an additional recovery export with restrictive permissions. The following commands contain secrets in their files, never print or commit them. They are operator recovery instructions, not proof that an offsite backup exists:

umask 077
RECOVERY_DIR=/home/devops/shared-notification-private/recovery-$(date -u +%Y%m%dT%H%M%SZ)
mkdir "$RECOVERY_DIR"
kn get secret -o json > "$RECOVERY_DIR/secrets.json"
kn get deployment,service,configmap,networkpolicy,resourcequota,limitrange,pvc -o json > "$RECOVERY_DIR/namespace-resources.json"
k get pv shared-notification-postgres shared-notification-rabbitmq -o json > "$RECOVERY_DIR/pvs.json"

These are inventories with server-assigned fields, not manifests to blindly apply to a new cluster. Keep protected copies of the package config, kubeconfig and private credential files with your backup system. RabbitMQ definitions do not back up queued messages. A broker volume backup requires a coordinated stopped broker and publisher maintenance window; no live-volume tar or restore drill was performed here.

7. Recover/redeploy this release on retained infrastructure

Prerequisites: correct disk/PVCs, original Secrets, PostgreSQL and RabbitMQ Ready, no concurrent deployment, verified fresh backup, exact pinned image variables from section 3. Web/Worker remain stopped from section 6.

If history already contains exactly the six entries above, skip migration. If it is exactly the original three-entry prefix, use the following maintained recipe from the package directory. It creates the same bounded migrator Job used for the rollout, references the existing owner Secret, and replaces only the completed owned migrator Job to fit the quota. Other histories require inspection; do not guess or edit EF history.

export MIGRATOR_IMAGE
python3 - <<'PY'
import os,types
import notification as n
c=n.Cluster(types.SimpleNamespace(kubectl='/snap/microk8s/current/kubectl',kubeconfig='/home/devops/.kube/cluster-admin.conf'))
prefix=['20260501214629_InitialCreate','20260502154249_PendingChanges','20261017072410_EmailTemplateLocalizations']
target=prefix+['20261023223753_AddPushDeliveries','20261024073202_LinkEmailChannelsToServices','20261024080457_AddPushChannels']
with n.lock('/home/devops/shared-notification-private'):
    assert all(c.get('deployment',x)['spec']['replicas']==0 for x in ('web','worker'))
    assert c.history()==prefix, 'Unexpected history; stop and inspect'
    old=c.get('job','notification-migrator')
    assert old and old['metadata']['labels']['app.kubernetes.io/managed-by']==n.MANAGED and old['status'].get('succeeded')==1
    job=n.job('migrator',os.environ['MIGRATOR_IMAGE'],environment=[n.env('DOTNET_ENVIRONMENT','Production'),n.env('App__ConnectionString',secret='notification-private',key='ownerConnectionString')])
    n.run_job(c,job)
    assert c.history()==target
    print('All six migrations verified; apply runtime grants before starting writers.')
PY

Apply and verify the grants in section 5. Inspect migrated channel ownership without reading credentials:

kn exec deployment/postgres -- psql -U notification_owner -d notification_service -At -c 'SELECT s.name,count(*) FROM email_channels c JOIN services s ON s.id=c.service_id GROUP BY s.name ORDER BY s.name'

Set only application images, retaining Secrets, volumes, selectors and resources:

kn set image deployment/web web="$WEB_IMAGE"
kn set image deployment/worker worker="$WORKER_IMAGE"
kn scale deployment/web --replicas=1
kn rollout status deployment/web --timeout=240s
kn scale deployment/worker --replicas=1
kn rollout status deployment/worker --timeout=240s

Then verify actual pod imageIDs, six migrations, grants, consumers, UI and OAuth. If migration fails, keep writers stopped while inspecting the failed Job and database history. If rollout fails, inspect events/logs before changing anything else. Never leave replicas at zero without reporting the specific failure and recovery action.

8. Rollback and destructive database restore

An image-only rollback is allowed only when the target image supports the current schema. kubectl rollout undo can select the old pre-ServiceId binary and is unsafe after this migration. Prefer reapplying the known six-migration image pair above for an application/configuration regression. Do not roll back the database merely because a port-forward or OAuth connection fails.

DESTRUCTIVE, NOT EXECUTED ON PRODUCTION: the following replaces the database with the explicitly selected backup. It loses writes after that backup and can conflict with retained broker messages. Obtain a maintenance window, pause publishers, stop Web/Worker, verify a fresh safety backup, and select the intended recovery point first. PostgreSQL must be healthy, original roles/passwords retained, and the backup must pass section 6. Do not run this block for validation or as a routine redeploy.

# Linux SSH session; BACKUP_DIR must identify the deliberately selected verified dump.
test -s "$BACKUP_DIR/database.dump" || exit 1
kn exec deployment/postgres -- dropdb -U notification_owner --force notification_service
kn exec deployment/postgres -- createdb -U notification_owner -O notification_owner notification_service
kn exec -i deployment/postgres -- pg_restore -U notification_owner -d notification_service --exit-on-error --single-transaction < "$BACKUP_DIR/database.dump"

dropdb --force terminates database connections. If restoration fails, leave writers stopped, retain the failed database for diagnosis and do not replay arbitrary messages. Inspect restored migration history. With a three-migration backup, follow section 7 to migrate forward and reapply grants before starting the six-migration application. With six migrations, verify grants and restart the pinned pair. Inspect broker queues and externally recorded outcomes before resuming publishers: database restore does not rewind acknowledgements or external email/push delivery.

For total node/PV loss, this document does not claim a tested bare-metal restore. Recover the original mounted disk and private files first where possible. Rebuilding a cluster from exports requires reviewing namespace ownership, PV binding and node paths, Secret consistency, and sanitized manifests. Do not invoke the stale bootstrap package blindly. A separately rehearsed full disaster-recovery procedure remains a gap.

9. RabbitMQ and Worker checks

kn exec deployment/rabbitmq -- rabbitmq-diagnostics -q check_running
kn exec deployment/rabbitmq -- rabbitmqctl list_queues -p /notification name consumers messages_ready messages_unacknowledged
kn exec deployment/rabbitmq -- rabbitmqctl list_permissions -p /notification

Expected after recovery: notification.send-email has exactly one consumer; invemory.production.notification-results has one consumer from Invemory. Zero queued messages was the verified baseline, not a requirement to purge legitimate new work. Development/staging result queues can have zero consumers. Broker nodename is fixed rabbit@rabbitmq; do not change it against the retained data directory.

Worker read ACL covers notification.send-email only. Its configure/write ACLs cover the existing email result routes, including production/staging/development. The production publisher is invemory_production, staging is invemory_staging; an additional development account exists. Preserve all current accounts and definitions; do not replay an older definitions Secret to fix a consumer. Imported definitions hold credentials, not a complete live message backup. The shared default exchange does not isolate application ServiceName values between publishers.

10. Google OAuth/FCM egress and timeout diagnosis

On 2026-10-24 DNS resolved both Google endpoints, but all four Web/Worker TLS probes timed out. Base default-deny allowed only internal services and SMTP 587. Applying google-api-network.json fixed all four TLS handshakes immediately without a restart; the real production UI then returned OAuth access token obtained.

The policy selects only managed Web/Worker pods in shared-notification, IPv4 TCP 443 to Google's published API/service ranges. It grants no ingress or arbitrary internet access. Existing DNS policy allows TCP/UDP 53 only to kube-system CoreDNS. Calico OSS standard NetworkPolicy does not provide an FQDN allowlist. Google documents that individual APIs have no stable dedicated IP list: this snapshot uses goog.json minus cloud.json (297 CIDRs at verification), excluding customer Cloud ranges but still including other Google-operated services. This is an IP-level provider boundary, not a hostname-level boundary. Google's range method.

Read-only probes from each actual runtime pod (7-second bound, no credentials):

for component in web worker; do
  for endpoint in oauth2.googleapis.com fcm.googleapis.com; do
    printf '%s %s\n' "$component" "$endpoint"
    kn exec deployment/"$component" -- getent ahostsv4 "$endpoint"
    kn exec deployment/"$component" -- sh -c 'timeout 7 openssl s_client -connect "$1:443" -servername "$1" -verify_hostname "$1" -verify_return_error -brief </dev/null' sh "$endpoint"
  done
done
kn get networkpolicy

Expected TLS connection and certificate verification OK. Exit 124 means timeout; name-resolution errors indicate DNS rather than Firebase credentials. Do not disable certificate verification. A successful TLS check alone does not prove OAuth succeeds. Use the saved channel's Check Firebase authorization action afterwards; no key needs to be reentered when its project is unchanged. Its current application timeout is 30s.

Refresh the range snapshot when Google updates its ranges; no automatic updater is installed. The renderer downloads only public feeds, generates a candidate, and never applies it. Keep the reviewed snapshot in Git. On Windows PowerShell, from this repo:

python docker/kubernetes/shared-notification/refresh-google-api-network.py --output C:/projects/ItemTracker/tmp/IT-1074-push/google-api-network.candidate.json
git diff --no-index -- docker/kubernetes/shared-notification/google-api-network.json C:/projects/ItemTracker/tmp/IT-1074-push/google-api-network.candidate.json

Review source dates and CIDR changes. A diff exit code of 1 means files differ. Confirm the current DNS answers are covered and all addresses remain public. Copy the reviewed candidate into the repository manifest, then transfer only that nonsecret file:

Copy-Item -LiteralPath C:/projects/ItemTracker/tmp/IT-1074-push/google-api-network.candidate.json -Destination docker/kubernetes/shared-notification/google-api-network.json
scp -i C:/Users/agryn/.ssh/id_ed25519_devops docker/kubernetes/shared-notification/google-api-network.json devops@192.168.2.44:/home/devops/notification-package/google-api-network.json

On the server:

kn apply --dry-run=server -f /home/devops/notification-package/google-api-network.json
kn apply -f /home/devops/notification-package/google-api-network.json

Repeat all four TLS probes and OAuth. Keep the prior reviewed manifest for policy rollback. Do not replace a failed DNS answer with a permanent single /32 rule: Google addresses rotate. Apply the production SMTP/broker overlay separately only when those policies need reconciliation; Google recovery does not require modifying other namespaces.

11. Owner's port-forward: exact address and restart

Run in a dedicated Windows PowerShell terminal. Leave the process running:

ssh -tt -o StrictHostKeyChecking=yes -o HostKeyAlgorithms=ssh-ed25519 -o ConnectTimeout=10 -o ExitOnForwardFailure=yes -o ServerAliveInterval=15 -o ServerAliveCountMax=2 -i C:/Users/agryn/.ssh/id_ed25519_devops -L 127.0.0.1:15200:127.0.0.1:15200 devops@192.168.2.44 'exec /snap/microk8s/current/kubectl --kubeconfig=/home/devops/.kube/cluster-admin.conf --namespace=shared-notification --request-timeout=15s port-forward --address=127.0.0.1 service/web 15200:8080'

Open http://localhost:15200/admin/services. A service port-forward binds to a selected pod; replacing that pod can break the existing tunnel even when its successor is Ready. After rollout, verify this exact owner URL, not only a temporary alternate port:

Invoke-WebRequest -Uri http://localhost:15200/admin/services -UseBasicParsing -TimeoutSec 10 | Select-Object StatusCode
Get-NetTCPConnection -LocalPort 15200 -ErrorAction SilentlyContinue | Select-Object LocalAddress,State,OwningProcess
Get-CimInstance Win32_Process -Filter "Name = 'ssh.exe'" | Select-Object ProcessId,CommandLine

If broken, stop only the identified 15200 tunnel with Ctrl+C in its own terminal and rerun the exact command. Do not kill all SSH processes or a working tunnel belonging to another task. If the remote port is occupied, inspect its process/owner before stopping anything. For a background helper use Start-Process -WindowStyle Hidden, retain its PID and output paths; never leave an untracked visible helper window.

For optional broker or Mailpit inspection, use separate loopback tunnels with the same SSH options and matching local/remote port pairs: RabbitMQ 15672:15672 forwarding service/rabbitmq, or Mailpit 18025:8025 forwarding service/mailpit. Do not bind administration to 0.0.0.0 or expose it publicly.

12. Incident checklist and evidence

Read logs locally; messages may contain recipient or provider details. Do not paste raw logs into public places. Narrow by time and component:

kn get events --sort-by=.metadata.creationTimestamp
kn logs deployment/web --since=10m --tail=100
kn logs deployment/worker --since=10m --tail=100
kn logs job/notification-migrator --tail=100
kn get configmap runtime-grants -o yaml

Symptom	Next check
UI connection refused, pods Ready	Exact localhost:15200 tunnel; restart only stale forward
OAuth times out after 30s	DNS and TLS from Web; Google policy and current feed coverage
OAuth rejected immediately	Correct Firebase project/service account, key validity; no credential logging
Permission denied for push table	Quoted "PushDeliveries", runtime grants and connection role
Web stays unready	Events, cold-start CPU, database query, schema/history; do not remove readiness
Worker Ready but no deliveries	RabbitMQ consumer count/ACLs and runtime logs; process readiness is insufficient
ImagePullBackOff	Exact digest, registry reachability, existing pull Secret; do not regenerate credentials
Pod storage guard failure	Real filesystem UUID/mount/marker before any restart or data-directory repair

The platform may log a missing optional libgssapi_krb5.so.2; in this rollout Web still became ready and used its password connection. Diagnose the actual failing operation instead of assuming that warning caused an OAuth network timeout.

13. CI distinction and verification limits

TeamCity HrynCo_NotificationService_BuildAndPush build 141783 produced the verified images. HrynCo_NotificationService_DeployProduction build 141784 deployed the old Docker Compose target on mini-pc-1, not this Kubernetes namespace. A green job there is not Kubernetes rollout or owner UI proof. Kubernetes was manually deployed using the retained production objects and pinned digests described here.

Contracts publishing is a separate NuGet operation. Build 141789 was blocked by incompatible agent requirements (teamcity.agent.name=ItemTrackerDotNet10 and system.agent.name=mini-pc-1). Do not publish Contracts manually or alter CI as part of runtime recovery. Some current checks also return success despite missing dotnet or unavailable Docker test prerequisites; rely on actual test output, not status alone.

Verified: pinned running images, six migration entries, runtime table privileges, preserved Secrets/PVC identities and existing email contents, custom-format backup checksum/listing, Web/Worker Ready with zero restarts, email/result consumers, owner URL UI, four OAuth/FCM TLS paths, and successful real production OAuth check. No push was sent. Destructive database restore, full host loss recovery, offsite backup, RabbitMQ message restore and end-to-end production push delivery were not tested. Publishing and merging these documentation/egress follow-ups require a separate delivery request; a local commit does not authorize those operations. IT-1074 remains open.