hrynco/hrynco-notification-service
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

fix(ops): allow google api egress and document recovery

Browse Source 
Add scoped google api network policy, its refresh command, and the verified production operations runbook.

Ref: IT-1074
This commit is contained in:
Anatolii Hrynchuk
2026-10-24 13:14:16 +03:00
parent 1631786ce2
commit 9e268b8ec3
5 changed files with 2168 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+1
View File
@@ -9,6 +9,7 @@ Production image references now use `registry.hrynco.com`, preserving the existi
- [Service and whole-template API for AI clients](docs/email-template-definition-api.md)
- [ItemTracker outbox email consumer](docs/itemtracker-outbox-email-consumer.md)
- [Shared Notification Service Kubernetes package](docker/kubernetes/shared-notification/README.md)
- [Production operations and recovery runbook](docs/production-operations.md) — actual Kubernetes environment, backup/restore, grants, Google egress and owner access.
## Windows host development
docker/kubernetes/shared-notification/README.md
+7
View File
@@ -1,5 +1,12 @@
# Shared Notification Service on MicroK8s
For the current six-migration production release, use the
[production operations and recovery runbook](../../../docs/production-operations.md).
The original bootstrap/deploy schema below predates push channels; do not run its
full deploy against the current database. The current additive Google HTTPS policy
is `google-api-network.json`; refresh it with `refresh-google-api-network.py` and
review/server-dry-run it before applying. Existing SMTP and broker policies remain.
This package deploys one shared Web/Worker, PostgreSQL 17, RabbitMQ 4 and a temporary
Mailpit sandbox in `shared-notification`, on `mini-pc-2`. It does not expose public
Ingress, NodePort or unauthenticated administration. Application administration,
docker/kubernetes/shared-notification/google-api-network.json
+1534
View File
File diff suppressed because it is too large Load Diff
docker/kubernetes/shared-notification/refresh-google-api-network.py
+62
View File
@@ -0,0 +1,62 @@
#!/usr/bin/env python3
"""Render Google API IPv4 egress; never applies changes to Kubernetes."""
import argparse
import ipaddress
import json
from pathlib import Path
import urllib.request
def fetch(name):
with urllib.request.urlopen(f"https://www.gstatic.com/ipranges/{name}.json", timeout=20) as response:
return json.load(response)
def api_networks(google, cloud):
networks = [ipaddress.ip_network(p["ipv4Prefix"]) for p in google["prefixes"] if "ipv4Prefix" in p]
for prefix in cloud["prefixes"]:
if "ipv4Prefix" not in prefix:
continue
excluded = ipaddress.ip_network(prefix["ipv4Prefix"])
remaining = []
for network in networks:
if not network.overlaps(excluded):
remaining.append(network)
elif not network.subnet_of(excluded):
remaining.extend(network.address_exclude(excluded))
networks = remaining
return list(ipaddress.collapse_addresses(networks))
def render(google, cloud):
networks = api_networks(google, cloud)
if not networks or any(not network.is_global for network in networks):
raise ValueError("Expected nonempty public Google API IPv4 ranges")
labels = {"app.kubernetes.io/name": "hrynco-shared-notification",
"app.kubernetes.io/managed-by": "hrynco-shared-notification"}
return {
"apiVersion": "networking.k8s.io/v1", "kind": "NetworkPolicy",
"metadata": {"name": "web-worker-to-google-apis", "namespace": "shared-notification",
"labels": labels, "annotations": {
"hrynco.com/google-ranges-date": google["creationTime"],
"hrynco.com/cloud-ranges-date": cloud["creationTime"]}},
"spec": {"podSelector": {"matchLabels": labels, "matchExpressions": [
{"key": "app.kubernetes.io/component", "operator": "In", "values": ["web", "worker"]}]},
"policyTypes": ["Egress"], "egress": [{
"to": [{"ipBlock": {"cidr": str(network)}} for network in networks],
"ports": [{"protocol": "TCP", "port": 443}]}]},
}
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--output", type=Path, required=True)
args = parser.parse_args()
policy = render(fetch("goog"), fetch("cloud"))
args.output.write_text(json.dumps(policy, indent=2) + "\n", encoding="utf-8")
print(f"Rendered {len(policy['spec']['egress'][0]['to'])} IPv4 ranges; review before applying.")
if __name__ == "__main__":
main()
docs/production-operations.md
+564
View File
@@ -0,0 +1,564 @@
# Notification Service: production operations and recovery
Verified against the running Kubernetes service on **2026-10-24**. Commands below
distinguish read-only checks, service interruptions and destructive recovery. Do not
paste passwords, service-account JSON, tokens, Secret YAML or database dumps into
tickets, chat, Git or CI logs. The administrative UI has no application authentication;
keep it behind the existing loopback SSH tunnel.
## 1. Actual environment and source of truth
| Item | Actual value |
| --- | --- |
| Windows source checkout | `C:\projects\HrynCo\hrynco-notification-service` |
| SSH target | `devops@192.168.2.44` (`mini-pc-2`; Linux username is lowercase) |
| Windows SSH key | `C:\Users\agryn\.ssh\id_ed25519_devops`, unlocked in the owner's ssh-agent |
| Kubernetes CLI | `/snap/microk8s/current/kubectl` |
| Kubeconfig / context | `/home/devops/.kube/cluster-admin.conf` / `microk8s` |
| Namespace / CNI | `shared-notification` / Calico `v3.29.3` |
| Server package | `/home/devops/notification-package` |
| Private files / local backups | `/home/devops/shared-notification-private` |
| External storage | `/mnt/shared/notification-service`, XFS, UUID `c2c31640-a247-4988-bc7b-9ee9789464d1` |
| Local admin URL | `http://localhost:15200/admin/services` |
| Web service | `web:8080`, ClusterIP; no public Ingress or NodePort |
| PostgreSQL | `postgres:5432`, database `notification_service` |
| RabbitMQ | `rabbitmq:5672`, vhost `/notification`, management port `15672` |
| Seq | `http://seq.observability.svc.cluster.local:8080` |
| Sandbox SMTP / UI | `mailpit:1025` / `mailpit:8025`; no external relay |
The repository's `docker/kubernetes/shared-notification/` owns the package. The
server's `config.local.json` supplies the verified host/storage and namespace values;
`artifacts.local.json` still describes the older bootstrap application images.
`notification.py` owns the original foundation and credential bootstrap. Its
`MIGRATIONS`, sandbox SQL and application artifact file are **not yet reconciled with
the six-migration release below**. Do not run its full `bootstrap` or `deploy` command
over this production release. It can overwrite current image/configuration intent or
fail on removed `email_channels.service_name`.
Use the targeted procedures in this document for the existing production resources.
They do not depend on the historical one-off `it1074-deploy.py`. The original
deployment audit is retained privately at
`/home/devops/shared-notification-private/it1074/{before,result}.json`; `before.json`
contains deployment specifications and fingerprints, not a restorable database backup.
The one-off helper's initial GRANT used the wrong table case; do not rerun it.
Network source of truth is additive: base policies from `notification.py`, existing
`production-network.yaml` for SMTP and production broker access, and
`google-api-network.json` for Google APIs. Never reconcile the sandbox render as the
complete production setup. Reconcile and review all overlays together.
## 2. Connect and inspect before changing anything
Run in **Windows PowerShell** (no SSH alias is required or assumed):
```powershell
ssh -o StrictHostKeyChecking=yes -o HostKeyAlgorithms=ssh-ed25519 -o ConnectTimeout=10 -i C:/Users/agryn/.ssh/id_ed25519_devops devops@192.168.2.44
```
Inside that **Linux SSH session**, define these session-local helpers. All subsequent
`k` and `kn` commands in this document run there, not in Windows PowerShell:
```sh
k() { /snap/microk8s/current/kubectl --kubeconfig=/home/devops/.kube/cluster-admin.conf --request-timeout=20s "$@"; }
kn() { k --namespace=shared-notification "$@"; }
cd /home/devops/notification-package
k config current-context
k get node mini-pc-2
kn get deployment,pod,service,pvc,job
k get pv shared-notification-postgres shared-notification-rabbitmq
findmnt -T /mnt/shared/notification-service -o TARGET,SOURCE,FSTYPE,UUID
kn get resourcequota
```
Expected: context `microk8s`, node Ready, the exact XFS UUID above, both PVCs Bound.
Stop if the disk is not mounted: do not let an empty directory on the root disk stand
in for persistent storage. PVs use `Retain` and node affinity to `mini-pc-2`:
| PVC | PV | Path | Capacity |
| --- | --- | --- | --- |
| `postgres-data` | `shared-notification-postgres` | `/mnt/shared/notification-service/postgres` | 10Gi |
| `rabbitmq-data` | `shared-notification-rabbitmq` | `/mnt/shared/notification-service/rabbitmq` | 5Gi |
Do not delete PVCs, clear PV claimRefs, reformat storage or regenerate credentials to
repair an application timeout. These are singleton workloads, not an HA database/broker.
The quota allows five Deployments and three Jobs; the completed migrator/grants/sandbox
Jobs can occupy all three Job slots. A new fourth Job can fail admission.
## 3. Current application release and infrastructure images
Application source: `d4ee8fc0c8036b0b48e1a481ef1a12a6a1c1c269`, merged PR 28.
Image build: TeamCity `141783`, tag `main-34`. Deploy immutable digests:
```sh
WEB_IMAGE='registry.hrynco.com/hrynco.notification-service.web@sha256:69c51ac1ba7c1101472a0b7d45153544a95565cc544bba49908b1e4079195911'
WORKER_IMAGE='registry.hrynco.com/hrynco.notification-service.worker@sha256:cc93cbfd82e1d7ff4d88845dfa90c8a1188c51d35775432d385fa5f23795a6a3'
MIGRATOR_IMAGE='registry.hrynco.com/hrynco.notification-service.migrator@sha256:9975e05cb075f45a6a629233af4c56194731378f710feb719b205a68bebdb035'
```
The repositories include `notification-service` with a hyphen. Old bootstrap images
used `notificationservice`; do not accidentally choose those. These digests are OCI
indexes; the matching amd64 manifests are web `d69a6d7cdbc8c08d5a1234933a88579fd176a72b8c64fa02167c42d988eed0be`,
worker `92d454a4139de55f5619070ed6b070c94db05749bbd3aca5d0e91d5a11ae8875`,
migrator `a94766802609f4348adba4bdef4bc528275f317c3015b10bb97c8d530749adbd`.
Registry manifest inspection must accept OCI indexes; a Docker-only Accept header can
return a misleading 404.
Retained infrastructure digests (do not upgrade during application recovery):
| Image repository | Digest |
| --- | --- |
| `registry.hrynco.com/mirror/postgres` | `sha256:e31e3d5327d1806f6177827c9710643e4f35f7ab3f14d26d05332753d3e95ee0` |
| `registry.hrynco.com/mirror/rabbitmq` | `sha256:ad995f859fce23a0ccd91c24cfac89d8f8cc57de4d708b58dcecea2697ad0d8c` |
| `registry.hrynco.com/mirror/mailpit` | `sha256:13de4ffbd28f3089be6dff32afbc8210edb5a593d738fa877ae7272cc73aeb67` |
Read-only image and readiness verification:
```sh
kn get deployment web worker -o custom-columns='NAME:.metadata.name,DESIRED:.spec.replicas,READY:.status.readyReplicas,IMAGE:.spec.template.spec.containers[*].image'
kn get pods -o custom-columns='NAME:.metadata.name,READY:.status.containerStatuses[*].ready,RESTARTS:.status.containerStatuses[*].restartCount,IMAGEID:.status.containerStatuses[*].imageID'
kn rollout status deployment/web --timeout=240s
kn rollout status deployment/worker --timeout=240s
```
Web readiness queries `/api/v1/email-channels?serviceName=Invemory.Staging.Sandbox` on
8080, every 10 seconds with a one-second timeout. Cold start under the 250m CPU limit
can initially time out. Worker has no HTTP/readiness probe: **Ready is not proof of
consumer readiness**. Verify RabbitMQ separately.
## 4. Configuration and credentials
Both Web and Worker use `notification-config` through `envFrom`:
`DOTNET_ENVIRONMENT=Production`, `ASPNETCORE_ENVIRONMENT=Production`,
`ASPNETCORE_HTTP_PORTS=8080`, RabbitMQ host `rabbitmq`, port `5672`, vhost
`/notification`, Seq URL above, Serilog service `HrynCo.NotificationService` and
environment `Shared`. Inspect nonsecret settings with:
```sh
kn get configmap notification-config -o yaml
kn get secret
```
`notification-private` keys: `ownerConnectionString`, `runtimeConnectionString`,
`postgresPassword`, `runtimePassword`, `rabbitAdminPassword`, `rabbitWorkerPassword`,
`rabbitPublisherPassword`, `seqApiKey`. Web/Worker use the runtime connection;
migrations use owner connection. Worker username is `notification_worker`.
`registry-pull` contains `.dockerconfigjson`; `rabbit-definitions` contains
`definitions.json` with sensitive password hashes. Do not print their values.
The server private directory retains `credentials.json` and production publisher
password/hash files. Registry authentication and Seq source files are under
`/home/devops/notification-private/registry-auth.json` and `seq-key.txt`.
Recover original credentials from a protected backup if lost; changing Kubernetes
Secrets does not automatically change passwords in existing database/broker volumes.
Do not run bootstrap to generate replacements against retained data.
Firebase service-account JSON is stored in PostgreSQL `push_channels.credentials`
(JSONB), without application encryption, by explicit owner decision. Database backups
therefore contain private keys. Edit forms never return the saved JSON; leaving the
field blank retains it. A project change requires matching new credentials. Current
channel is attached to `Invemory-Prod`, Firebase project technical ID
`storemate-hrynco-com`. Do not copy local development keys into production.
`Check Firebase authorization` obtains an OAuth token using saved credentials when
the project is unchanged. It does not send a push, prove FCM send permission, or prove
device display. `Send test push` is a separate explicitly authorized action.
The standalone test sends a Firebase `notification` payload without Invemory binding
data. On the currently installed Invemory Android build it displays through the FCM
SDK in background. In foreground, `InvemoryMessagingService.onMessageReceived` rejects
the empty data map because `bindingId`, `version`, `messageId`, route and expiry are
required; it does not call the base Capacitor handler. Thus accepted test messages can
be invisible while Invemory is open. Samsung logs on 2026-10-24 show the FCM fallback
notification channel and removal at 13:04:28 Europe/Kiev; the receiver does not log
its rejection, so no individual foreground callback was proven. Do not diagnose this
as egress failure or rotate tokens solely because foreground display is absent.
The Worker push consumer is **not enabled** by this rollout: configuration and broker
ACLs still cover email only. A saved active channel and successful OAuth check do not
activate end-to-end production push processing. Do not enable it incidentally during
an email service recovery.
## 5. Database history and runtime grants
Read-only history check:
```sh
kn exec deployment/postgres -- psql -U notification_owner -d notification_service -At -c 'SELECT "MigrationId" FROM "__EFMigrationsHistory" ORDER BY "MigrationId"'
```
Expected, in order:
```text
20260501214629_InitialCreate
20260502154249_PendingChanges
20261017072410_EmailTemplateLocalizations
20261023223753_AddPushDeliveries
20261024073202_LinkEmailChannelsToServices
20261024080457_AddPushChannels
```
`notification_owner` is currently the PostgreSQL bootstrap superuser; keep it out of
application connections. `notification_runtime` is a login with no superuser,
createdb or createrole privileges. There are **no default ACL entries**: future tables
do not inherit runtime CRUD automatically. For each new migration, grant only the
tables needed by the application and persist the same SQL in `runtime-grants`.
Do not add blanket default privileges as a workaround.
The exact current push grants are below. `"PushDeliveries"` is case-sensitive;
`push_deliveries` does not exist. This command is idempotent and was verified:
```sh
kn exec deployment/postgres -- psql -U notification_owner -d notification_service -v ON_ERROR_STOP=1 -c 'GRANT SELECT, INSERT, UPDATE, DELETE ON push_channels, "PushDeliveries" TO notification_runtime;'
kn exec deployment/postgres -- psql -U notification_owner -d notification_service -At -c "SELECT t,has_table_privilege('notification_runtime',t,'SELECT,INSERT,UPDATE,DELETE') FROM unnest(ARRAY['push_channels','\"PushDeliveries\"']) t"
```
Both checks must return `t`. Existing runtime CRUD covers `email_channels`,
`email_templates`, `email_channel_usage`, `services`, `service_languages`,
`email_template_definitions`; schema USAGE and database CONNECT are granted, schema
CREATE is revoked from PUBLIC. The live `runtime-grants` ConfigMap already includes
the correct push grant. To persist it again without rotating passwords:
```sh
python3 - <<'PY'
import json, subprocess
k = ['/snap/microk8s/current/kubectl', '--kubeconfig=/home/devops/.kube/cluster-admin.conf', '-n', 'shared-notification']
value = json.loads(subprocess.check_output(k + ['get','configmap','runtime-grants','-o','json']))
grant = 'GRANT SELECT, INSERT, UPDATE, DELETE ON push_channels, "PushDeliveries" TO notification_runtime;'
sql = value['data']['grants.sql']
if grant not in sql:
subprocess.run(k + ['patch','configmap','runtime-grants','--type=merge','-p',json.dumps({'data':{'grants.sql':sql+'\n'+grant+'\n'}})], check=True)
PY
```
Do not execute the entire grants ConfigMap manually without its intended
`RUNTIME_PASSWORD` secret environment: its bootstrap SQL also manages the runtime role
password. Do not rerun the old sandbox job; channel ownership now uses `service_id`.
The migration maps the legacy `StoreMate-Prod` channel to existing `Invemory-Prod`.
The old identifier is historical migration input, not a new channel/service name.
## 6. Backup before migration or restore
**Service interruption:** stop Web and Worker writers first. This does not stop
external publishers; coordinate a publisher pause if a consistent broker/database
recovery point is needed. Never purge queues as a substitute.
```sh
kn scale deployment/worker deployment/web --replicas=0
kn wait --for=delete pod -l 'app.kubernetes.io/component in (web,worker)' --timeout=120s
```
The canonical backup implementation is safe independently of its old deploy schema.
This maintained source command creates a custom-format dump with mode 0600, checks
the `PGDMP` signature and `pg_restore --list`, and writes SHA-256, bytes and migration
history to `manifest.json`. Run from the server package; only these existing functions
are used, not `deploy` or sandbox SQL:
```sh
python3 - <<'PY'
import types
import notification as n
c = n.Cluster(types.SimpleNamespace(kubectl='/snap/microk8s/current/kubectl', kubeconfig='/home/devops/.kube/cluster-admin.conf'))
with n.lock('/home/devops/shared-notification-private'):
assert all(c.get('deployment', x)['spec']['replicas'] == 0 for x in ('web','worker'))
print(n.backup(c, '/home/devops/shared-notification-private'))
PY
```
Record the returned folder. Pre-migration backup from this rollout:
`/home/devops/shared-notification-private/backup-20261024T094457816933Z`.
It has only the **first three migrations**, predates the owner's production push
channel, and is not a backup of today's six-migration state. Take a fresh backup before
any recovery. A local dump does not protect against loss of this host or disk.
No offsite copy or full disaster restore has been proven.
Verify a selected backup without restoring it (set the exact existing folder):
```sh
BACKUP_DIR=/home/devops/shared-notification-private/backup-20261024T094457816933Z
export BACKUP_DIR
python3 - <<'PY'
import hashlib,json,os,pathlib
p=pathlib.Path(os.environ['BACKUP_DIR'])
m=json.loads((p/'manifest.json').read_text()); data=(p/'database.dump').read_bytes()
assert len(data)==m['bytes'] and hashlib.sha256(data).hexdigest()==m['sha256']
assert data.startswith(b'PGDMP')
print('Checksum and length verified; migration history:', m['history'])
PY
kn exec -i deployment/postgres -- pg_restore --list < "$BACKUP_DIR/database.dump" >/dev/null
```
Protect an additional recovery export with restrictive permissions. The following
commands contain secrets in their **files**, never print or commit them. They are
operator recovery instructions, not proof that an offsite backup exists:
```sh
umask 077
RECOVERY_DIR=/home/devops/shared-notification-private/recovery-$(date -u +%Y%m%dT%H%M%SZ)
mkdir "$RECOVERY_DIR"
kn get secret -o json > "$RECOVERY_DIR/secrets.json"
kn get deployment,service,configmap,networkpolicy,resourcequota,limitrange,pvc -o json > "$RECOVERY_DIR/namespace-resources.json"
k get pv shared-notification-postgres shared-notification-rabbitmq -o json > "$RECOVERY_DIR/pvs.json"
```
These are inventories with server-assigned fields, not manifests to blindly apply
to a new cluster. Keep protected copies of the package config, kubeconfig and private
credential files with your backup system. RabbitMQ definitions do not back up queued
messages. A broker volume backup requires a coordinated stopped broker and publisher
maintenance window; no live-volume tar or restore drill was performed here.
## 7. Recover/redeploy this release on retained infrastructure
Prerequisites: correct disk/PVCs, original Secrets, PostgreSQL and RabbitMQ Ready,
no concurrent deployment, verified fresh backup, exact pinned image variables from
section 3. Web/Worker remain stopped from section 6.
If history already contains exactly the six entries above, skip migration. If it is
exactly the original three-entry prefix, use the following maintained recipe from
the package directory. It creates the same bounded migrator Job used for the rollout,
references the existing owner Secret, and replaces only the completed owned migrator
Job to fit the quota. Other histories require inspection; do not guess or edit EF history.
```sh
export MIGRATOR_IMAGE
python3 - <<'PY'
import os,types
import notification as n
c=n.Cluster(types.SimpleNamespace(kubectl='/snap/microk8s/current/kubectl',kubeconfig='/home/devops/.kube/cluster-admin.conf'))
prefix=['20260501214629_InitialCreate','20260502154249_PendingChanges','20261017072410_EmailTemplateLocalizations']
target=prefix+['20261023223753_AddPushDeliveries','20261024073202_LinkEmailChannelsToServices','20261024080457_AddPushChannels']
with n.lock('/home/devops/shared-notification-private'):
assert all(c.get('deployment',x)['spec']['replicas']==0 for x in ('web','worker'))
assert c.history()==prefix, 'Unexpected history; stop and inspect'
old=c.get('job','notification-migrator')
assert old and old['metadata']['labels']['app.kubernetes.io/managed-by']==n.MANAGED and old['status'].get('succeeded')==1
job=n.job('migrator',os.environ['MIGRATOR_IMAGE'],environment=[n.env('DOTNET_ENVIRONMENT','Production'),n.env('App__ConnectionString',secret='notification-private',key='ownerConnectionString')])
n.run_job(c,job)
assert c.history()==target
print('All six migrations verified; apply runtime grants before starting writers.')
PY
```
Apply and verify the grants in section 5. Inspect migrated channel ownership without
reading credentials:
```sh
kn exec deployment/postgres -- psql -U notification_owner -d notification_service -At -c 'SELECT s.name,count(*) FROM email_channels c JOIN services s ON s.id=c.service_id GROUP BY s.name ORDER BY s.name'
```
Set only application images, retaining Secrets, volumes, selectors and resources:
```sh
kn set image deployment/web web="$WEB_IMAGE"
kn set image deployment/worker worker="$WORKER_IMAGE"
kn scale deployment/web --replicas=1
kn rollout status deployment/web --timeout=240s
kn scale deployment/worker --replicas=1
kn rollout status deployment/worker --timeout=240s
```
Then verify actual pod imageIDs, six migrations, grants, consumers, UI and OAuth.
If migration fails, keep writers stopped while inspecting the failed Job and database
history. If rollout fails, inspect events/logs before changing anything else. Never
leave replicas at zero without reporting the specific failure and recovery action.
## 8. Rollback and destructive database restore
An image-only rollback is allowed only when the target image supports the **current**
schema. `kubectl rollout undo` can select the old pre-ServiceId binary and is unsafe
after this migration. Prefer reapplying the known six-migration image pair above for
an application/configuration regression. Do not roll back the database merely because
a port-forward or OAuth connection fails.
**DESTRUCTIVE, NOT EXECUTED ON PRODUCTION:** the following replaces the database with
the explicitly selected backup. It loses writes after that backup and can conflict
with retained broker messages. Obtain a maintenance window, pause publishers, stop
Web/Worker, verify a fresh safety backup, and select the intended recovery point first.
PostgreSQL must be healthy, original roles/passwords retained, and the backup must
pass section 6. Do not run this block for validation or as a routine redeploy.
```sh
# Linux SSH session; BACKUP_DIR must identify the deliberately selected verified dump.
test -s "$BACKUP_DIR/database.dump" || exit 1
kn exec deployment/postgres -- dropdb -U notification_owner --force notification_service
kn exec deployment/postgres -- createdb -U notification_owner -O notification_owner notification_service
kn exec -i deployment/postgres -- pg_restore -U notification_owner -d notification_service --exit-on-error --single-transaction < "$BACKUP_DIR/database.dump"
```
`dropdb --force` terminates database connections. If restoration fails, leave writers
stopped, retain the failed database for diagnosis and do not replay arbitrary messages.
Inspect restored migration history. With a three-migration backup, follow section 7
to migrate forward and reapply grants before starting the six-migration application.
With six migrations, verify grants and restart the pinned pair. Inspect broker queues
and externally recorded outcomes before resuming publishers: database restore does
not rewind acknowledgements or external email/push delivery.
For total node/PV loss, this document does **not** claim a tested bare-metal restore.
Recover the original mounted disk and private files first where possible. Rebuilding a
cluster from exports requires reviewing namespace ownership, PV binding and node paths,
Secret consistency, and sanitized manifests. Do not invoke the stale bootstrap package
blindly. A separately rehearsed full disaster-recovery procedure remains a gap.
## 9. RabbitMQ and Worker checks
```sh
kn exec deployment/rabbitmq -- rabbitmq-diagnostics -q check_running
kn exec deployment/rabbitmq -- rabbitmqctl list_queues -p /notification name consumers messages_ready messages_unacknowledged
kn exec deployment/rabbitmq -- rabbitmqctl list_permissions -p /notification
```
Expected after recovery: `notification.send-email` has exactly one consumer;
`invemory.production.notification-results` has one consumer from Invemory. Zero queued
messages was the verified baseline, not a requirement to purge legitimate new work.
Development/staging result queues can have zero consumers. Broker nodename is fixed
`rabbit@rabbitmq`; do not change it against the retained data directory.
Worker read ACL covers `notification.send-email` only. Its configure/write ACLs cover
the existing email result routes, including production/staging/development. The
production publisher is `invemory_production`, staging is `invemory_staging`; an
additional development account exists. Preserve all current accounts and definitions;
do not replay an older definitions Secret to fix a consumer. Imported definitions
hold credentials, not a complete live message backup. The shared default exchange
does not isolate application `ServiceName` values between publishers.
## 10. Google OAuth/FCM egress and timeout diagnosis
On 2026-10-24 DNS resolved both Google endpoints, but all four Web/Worker TLS probes
timed out. Base default-deny allowed only internal services and SMTP 587. Applying
`google-api-network.json` fixed all four TLS handshakes immediately without a restart;
the real production UI then returned **OAuth access token obtained**.
The policy selects only managed Web/Worker pods in `shared-notification`, IPv4 TCP
443 to Google's published API/service ranges. It grants no ingress or arbitrary
internet access. Existing DNS policy allows TCP/UDP 53 only to kube-system CoreDNS.
Calico OSS standard NetworkPolicy does not provide an FQDN allowlist. Google documents
that individual APIs have no stable dedicated IP list: this snapshot uses `goog.json`
minus `cloud.json` (297 CIDRs at verification), excluding customer Cloud ranges but
still including other Google-operated services. This is an IP-level provider boundary,
not a hostname-level boundary. [Google's range method](https://docs.cloud.google.com/vpc/docs/configure-private-google-access#ip-addr-defaults).
Read-only probes from each actual runtime pod (7-second bound, no credentials):
```sh
for component in web worker; do
for endpoint in oauth2.googleapis.com fcm.googleapis.com; do
printf '%s %s\n' "$component" "$endpoint"
kn exec deployment/"$component" -- getent ahostsv4 "$endpoint"
kn exec deployment/"$component" -- sh -c 'timeout 7 openssl s_client -connect "$1:443" -servername "$1" -verify_hostname "$1" -verify_return_error -brief </dev/null' sh "$endpoint"
done
done
kn get networkpolicy
```
Expected TLS connection and certificate verification OK. Exit 124 means timeout;
name-resolution errors indicate DNS rather than Firebase credentials. Do not disable
certificate verification. A successful TLS check alone does not prove OAuth succeeds.
Use the saved channel's **Check Firebase authorization** action afterwards; no key needs
to be reentered when its project is unchanged. Its current application timeout is 30s.
Refresh the range snapshot when Google updates its ranges; no automatic updater is
installed. The renderer downloads only public feeds, generates a candidate, and never
applies it. Keep the reviewed snapshot in Git. On Windows PowerShell, from this repo:
```powershell
python docker/kubernetes/shared-notification/refresh-google-api-network.py --output C:/projects/ItemTracker/tmp/IT-1074-push/google-api-network.candidate.json
git diff --no-index -- docker/kubernetes/shared-notification/google-api-network.json C:/projects/ItemTracker/tmp/IT-1074-push/google-api-network.candidate.json
```
Review source dates and CIDR changes. A diff exit code of 1 means files differ. Confirm
the current DNS answers are covered and all addresses remain public. Copy the reviewed
candidate into the repository manifest, then transfer only that nonsecret file:
```powershell
Copy-Item -LiteralPath C:/projects/ItemTracker/tmp/IT-1074-push/google-api-network.candidate.json -Destination docker/kubernetes/shared-notification/google-api-network.json
scp -i C:/Users/agryn/.ssh/id_ed25519_devops docker/kubernetes/shared-notification/google-api-network.json devops@192.168.2.44:/home/devops/notification-package/google-api-network.json
```
On the server:
```sh
kn apply --dry-run=server -f /home/devops/notification-package/google-api-network.json
kn apply -f /home/devops/notification-package/google-api-network.json
```
Repeat all four TLS probes and OAuth. Keep the prior reviewed manifest for policy
rollback. Do not replace a failed DNS answer with a permanent single /32 rule: Google
addresses rotate. Apply the production SMTP/broker overlay separately only when those
policies need reconciliation; Google recovery does not require modifying other namespaces.
## 11. Owner's port-forward: exact address and restart
Run in a dedicated **Windows PowerShell** terminal. Leave the process running:
```powershell
ssh -tt -o StrictHostKeyChecking=yes -o HostKeyAlgorithms=ssh-ed25519 -o ConnectTimeout=10 -o ExitOnForwardFailure=yes -o ServerAliveInterval=15 -o ServerAliveCountMax=2 -i C:/Users/agryn/.ssh/id_ed25519_devops -L 127.0.0.1:15200:127.0.0.1:15200 devops@192.168.2.44 'exec /snap/microk8s/current/kubectl --kubeconfig=/home/devops/.kube/cluster-admin.conf --namespace=shared-notification --request-timeout=15s port-forward --address=127.0.0.1 service/web 15200:8080'
```
Open `http://localhost:15200/admin/services`. A service port-forward binds to a selected
pod; replacing that pod can break the existing tunnel even when its successor is Ready.
After rollout, verify **this exact owner URL**, not only a temporary alternate port:
```powershell
Invoke-WebRequest -Uri http://localhost:15200/admin/services -UseBasicParsing -TimeoutSec 10 | Select-Object StatusCode
Get-NetTCPConnection -LocalPort 15200 -ErrorAction SilentlyContinue | Select-Object LocalAddress,State,OwningProcess
Get-CimInstance Win32_Process -Filter "Name = 'ssh.exe'" | Select-Object ProcessId,CommandLine
```
If broken, stop only the identified 15200 tunnel with Ctrl+C in its own terminal and
rerun the exact command. Do not kill all SSH processes or a working tunnel belonging
to another task. If the remote port is occupied, inspect its process/owner before
stopping anything. For a background helper use `Start-Process -WindowStyle Hidden`,
retain its PID and output paths; never leave an untracked visible helper window.
For optional broker or Mailpit inspection, use separate loopback tunnels with the same
SSH options and matching local/remote port pairs: RabbitMQ `15672:15672` forwarding
`service/rabbitmq`, or Mailpit `18025:8025` forwarding `service/mailpit`. Do not bind
administration to `0.0.0.0` or expose it publicly.
## 12. Incident checklist and evidence
Read logs locally; messages may contain recipient or provider details. Do not paste raw
logs into public places. Narrow by time and component:
```sh
kn get events --sort-by=.metadata.creationTimestamp
kn logs deployment/web --since=10m --tail=100
kn logs deployment/worker --since=10m --tail=100
kn logs job/notification-migrator --tail=100
kn get configmap runtime-grants -o yaml
```
| Symptom | Next check |
| --- | --- |
| UI connection refused, pods Ready | Exact localhost:15200 tunnel; restart only stale forward |
| OAuth times out after 30s | DNS and TLS from Web; Google policy and current feed coverage |
| OAuth rejected immediately | Correct Firebase project/service account, key validity; no credential logging |
| Permission denied for push table | Quoted `"PushDeliveries"`, runtime grants and connection role |
| Web stays unready | Events, cold-start CPU, database query, schema/history; do not remove readiness |
| Worker Ready but no deliveries | RabbitMQ consumer count/ACLs and runtime logs; process readiness is insufficient |
| ImagePullBackOff | Exact digest, registry reachability, existing pull Secret; do not regenerate credentials |
| Pod storage guard failure | Real filesystem UUID/mount/marker before any restart or data-directory repair |
The platform may log a missing optional `libgssapi_krb5.so.2`; in this rollout Web still
became ready and used its password connection. Diagnose the actual failing operation
instead of assuming that warning caused an OAuth network timeout.
## 13. CI distinction and verification limits
TeamCity `HrynCo_NotificationService_BuildAndPush` build 141783 produced the verified
images. `HrynCo_NotificationService_DeployProduction` build 141784 deployed the **old
Docker Compose target on mini-pc-1**, not this Kubernetes namespace. A green job there
is not Kubernetes rollout or owner UI proof. Kubernetes was manually deployed using
the retained production objects and pinned digests described here.
Contracts publishing is a separate NuGet operation. Build 141789 was blocked by
incompatible agent requirements (`teamcity.agent.name=ItemTrackerDotNet10` and
`system.agent.name=mini-pc-1`). Do not publish Contracts manually or alter CI as part of
runtime recovery. Some current checks also return success despite missing dotnet or
unavailable Docker test prerequisites; rely on actual test output, not status alone.
Verified: pinned running images, six migration entries, runtime table privileges,
preserved Secrets/PVC identities and existing email contents, custom-format backup
checksum/listing, Web/Worker Ready with zero restarts, email/result consumers, owner
URL UI, four OAuth/FCM TLS paths, and successful real production OAuth check. No push
was sent. Destructive database restore, full host loss recovery, offsite backup,
RabbitMQ message restore and end-to-end production push delivery were **not tested**.
Publishing and merging these documentation/egress follow-ups require a separate
delivery request; a local commit does not authorize those operations. IT-1074 remains open.