# Operational requirements

> What running Nauthera in production takes: PostgreSQL and Redis, network exposure, secrets, email, monitoring, audit forwarding, backups, upgrades and on-call.

Running Nauthera means running a tier-0 service: when it is down, nobody signs in to
anything that depends on it. This page lists what a production deployment needs, as a
checklist for the platform team and a basis for the service's runbook. Each item links to
the page with the detail.

```mermaid
flowchart TD
  U["Users and applications"] --> ING["Ingress or Gateway, TLS"]
  ING --> SRV["nauthera-server × 3<br/>across zones"]
  SS[("Secret store")] -->|"files"| SRV
  SRV --> PG[("PostgreSQL<br/>HA, point-in-time recovery")]
  SRV --> RD[("Redis<br/>Sentinel or Cluster")]
  SRV --> SMTP["SMTP relay"]
  SRV -->|"audit export"| SIEM["SIEM"]
  PROM["Prometheus"] -->|"scrape :9090"| SRV
  PROM --> AM["Alertmanager<br/>to on-call"]
  PG -.->|"backups"| BK[("Backups and the<br/>key recovery copy")]
  SS -.-> BK
```

## Platform

| Requirement | Detail |
|---|---|
| Kubernetes 1.27 or newer, Helm 3 or 4 | [Installation](https://www.nauthera.io/docs/v1/installation#requirements) |
| At least three server replicas across zones, with the chart's PodDisruptionBudget | The production profile's shape. Rollouts use `maxUnavailable: 0`. |
| cert-manager | For the ingress certificate, in-pod TLS, the gRPC control plane's mutual TLS, and the token signing keys and audit checkpoint key, whose Secrets belong in the recovery copy |
| An ingress or Gateway API route terminating TLS | `security.trusted_proxies` lists only your proxies' own addresses, so client addresses, rate limits and network zones are right ([Hardening](https://www.nauthera.io/docs/v1/security#the-client-address)) |
| The Prometheus Operator CRDs | For the chart's ServiceMonitor and alerting rules |
| A stable issuer hostname | Every token and application depends on it; see [Adopting Nauthera](https://www.nauthera.io/docs/v1/adoption#decisions-to-make-first) |

Keep the admin listener (`:9090`) and the gRPC listener (`:9091`) reachable only inside
the cluster. The production profile ships a NetworkPolicy; check that it covers both
listeners, and that your CNI enforces it.

## Data stores

**PostgreSQL** holds everything durable: users, clients, sessions, refresh tokens,
revocations, factors and the audit log ([where state lives](https://www.nauthera.io/docs/v1/operations#where-state-lives)).

- Run it highly available with point-in-time recovery, for example with CloudNativePG.
- Create a runtime role for the server and keep the schema owner for
  `nauthera-server maintain`, with the chart's `appendOnlyAudit`, so the server cannot
  rewrite its own audit log ([Audit Log](https://www.nauthera.io/docs/v1/audit-log#the-append-only-role)).
- Behind PgBouncer in transaction mode, set `database.pgbouncer: true` and run
  migrations over a direct connection.
- Size storage for the audit log, which is kept `audit.retention` (400 days) by default
  in monthly partitions.

**Redis** holds single-use state shared between replicas: authorization codes, pushed
requests, device codes, DPoP replay records and throttles. Any non-loopback issuer
needs it.

- `maxmemory-policy noeviction` (the server refuses to start otherwise), AOF persistence,
  and `cache.mode: sentinel` or `cluster` with a replica per primary.
- A Redis outage stops sign-ins at the code exchange, but signs nobody out and loses no
  refresh token. Treat it as a sign-in outage, not data loss.

## Secrets and their recovery

| Secret | If it is lost |
|---|---|
| Key-encryption key (`keys.encryption_key`) and every retired one | Other organizations' signing keys, every identity provider's client secret and every LDAP directory's bind credential become unreadable |
| MFA seed key (`mfa.encryption_key`) and every retired one | Every TOTP factor stops working |
| Main organization's signing keys | Every token already issued stops verifying |
| Audit checkpoint key | New checkpoints are signed under a new key |
| Database and Redis credentials, SMTP credentials, captcha secret | The matching feature stops until replaced |

- Keep them in your secret store and deliver them to the pods as files, the chart's
  default ([Configuration](https://www.nauthera.io/docs/v1/configuration#secrets-from-files)).
- Keep an offline recovery copy of the KEK, the MFA key, the signing keys and the audit
  checkpoint key (the last two, by default, in the Secrets cert-manager issued), held by
  someone other than the platform team, such as security. A database backup without them
  restores only the main organization ([Backups](https://www.nauthera.io/docs/v1/operations#backups)).
- If you used the bootstrap admin secret, read it, store it, and remove the file.

See the full [inventory](https://www.nauthera.io/docs/v1/keys-and-secrets#inventory).

## Email

Invitations, password reset, address verification, emailed second-factor codes,
new-sign-in alerts, and notices of a changed role, account status or credential are
emailed. Without a relay they are off.

- An SMTP relay over TLS for the deployment, and one per organization where a tenant
  sends from its own domain ([Email](https://www.nauthera.io/docs/v1/email)).
- SPF, DKIM and DMARC for each sending domain, so invitation waves land in inboxes.
- A relay quota above your largest planned invitation wave.

## Monitoring and alerting

- Turn on `metrics.serviceMonitor.enabled` and `metrics.prometheusRule.enabled`. The rule groups
  cover availability, the 99.9% error budget, latency, the database and cache, the audit
  pipeline, sign-in failure spikes, threat signals, back-channel logout, the control
  plane and certificate expiry ([Observability](https://www.nauthera.io/docs/v1/observability#alerting)).
- Route `availability`, `errors`, `dependencies` and `audit` alerts to the platform
  on-call, and `security` and `threats` to security.
- Load the Grafana dashboards from the repository, and ship the structured logs to your
  log platform.
- Turn on OpenTelemetry tracing if your applications already trace, so a slow sign-in can
  be followed across the hop.
- Watch the operator too: a resource stuck in `Error` or `Drifted`, or a
  `status.lastReconciledTime` that stops moving, means manifests are not reaching the
  server ([Operator](https://www.nauthera.io/docs/v1/operator)).

## Audit

- Forward the audit log to your SIEM with `audit.export`, de-duplicating on `id`, and
  alert on the exporter's lag ([Forwarding](https://www.nauthera.io/docs/v1/audit-log#forwarding)).
- Set `audit.checkpoint_key_path` so checkpoints are signed and the chain can be
  verified ([Tamper evidence](https://www.nauthera.io/docs/v1/audit-log#tamper-evidence)).
- Decide `audit.retention` and `audit.fail_closed` with compliance: fail-closed refuses
  changes the log cannot record.

## Backups and restore

There is no built-in backup.

- Back up PostgreSQL continuously, and the secrets above at the same time.
- Restore into a scratch environment at least once a quarter, with the recovered secrets,
  and sign in to each organization there. A backup that was never restored is a hope, not
  a backup.
- Write down the recovery point and recovery time you can meet, and agree them with the
  application owners.

## Upgrades

- Migrations are **forward only**: back up before every upgrade, and roll forward or
  restore, never back ([Upgrading](https://www.nauthera.io/docs/v1/operations#upgrading)).
- Run `nauthera-server maintain` as a pre-upgrade Job (the chart's `appendOnlyAudit` does)
  and on a schedule at least monthly, so audit partitions stay ahead of the writes.
- Rehearse each upgrade in staging first, and read the release notes for migrations that
  need downtime, such as 0049, or a long index build.
- Pin images by digest. A released chart does.

## Capacity

- Each password check runs argon2id with about 64 MiB of memory, and each replica runs at
  most `auth.brute_force.max_concurrent_verifies` (4) at once. A morning sign-in peak or a
  password-reset wave is bounded by replicas × 4 concurrent checks: add replicas rather
  than raising the limit without the memory to match
  ([Rate Limits & Lockout](https://www.nauthera.io/docs/v1/rate-limits)).
- With `rate_limit.backend: redis` (the production profile), the limit is shared by every
  replica: 3,000 requests a minute per client address by default
  (`rate_limit.rps` × `rate_limit.window`). Raise it before a bulk user import or a
  large wave.
- Size the database connection pool for replicas × pool size, below PostgreSQL's
  `max_connections`.

## On-call and runbook

The service needs an on-call rota before the first production wave. The runbook should
cover at least:

- the server not ready: check `/readyz`, the database, then Redis;
- sign-ins failing at the code exchange: Redis;
- sign-ins through one provider failing: its `IdentityProvider` status and the provider's
  client secret ([Errors & Troubleshooting](https://www.nauthera.io/docs/v1/troubleshooting));
- a user locked out, or who lost their second factor;
- a suspected compromised account: block it, which keeps the account for investigation
  ([Account status](https://www.nauthera.io/docs/v1/user-management#account-status)), or set a new password or
  delete the user; each ends every session and refresh token
  ([Ending a user's access](https://www.nauthera.io/docs/v1/user-management#ending-a-users-access));
- a suspected leaked signing key: rotate it ([Signing keys](https://www.nauthera.io/docs/v1/keys-and-secrets#rotating-the-main-orgs-signing-key));
- restoring from backup.

## Recurring tasks

| Task | How often |
|---|---|
| `nauthera-server maintain` for audit partitions | At least monthly (the chart's CronJob) |
| Restore drill | Quarterly |
| Review of `role.granted` and `role.revoked` events and of who holds platform roles | Quarterly |
| Upgrade to the latest chart | Each release, through staging |
| Rotate the main organization's signing keys | On a schedule you set, for example yearly; there is no automatic rotation |
| Rotate client secrets (delete the operator's `Secret`) and `private_key_jwt` keys | Per your secret policy |
| Check certificate renewal | Continuously, by the `certificates` alert group |

Before the first production wave, also work through the
[hardening checklist](https://www.nauthera.io/docs/v1/security#checklist).