# Backups & Upgrades

> Running nauthera-server over time: what lives in PostgreSQL and in Redis, what to back up, migrations, database maintenance, upgrades and automatic cleanup.

## Where state lives

| Store | Holds | If it is lost |
|---|---|---|
| **Database** (PostgreSQL) | Organizations and their encrypted signing keys, network zones and second-factor policies, users, clients, external identity providers (their client secrets encrypted) and linked upstream accounts, branding, roles, sign-in sessions and their history, consent approvals, refresh tokens and the records of spent ones, grant, session and subject revocations, MFA factors, passkeys and recovery codes, emailed links and invitations, the outgoing-email queue, the audit log and each audit exporter's position | Everything. Restore from backup. |
| **Redis** (`cache.*`) | Authorization codes, pushed authorization requests, device codes, DPoP nonces and replay records, spent captcha challenges, brute-force throttles, rate-limit windows (with `rate_limit.backend: redis`), cached discovery and branding | Codes and requests in flight, so an authorization code not yet redeemed, a pushed request or a device approval under way has to start again, and the throttles' counts. Refresh tokens and revocations survive. |
| **Secret store** | The KEK, the MFA seed key, signing keys, credentials | See [Keys & Secrets](https://www.nauthera.io/docs/v1/keys-and-secrets#inventory). |

Refresh tokens, the records that detect a reused one, and every revocation are kept in
the database, so they survive losing Redis. Redis still holds single-use state that must
be shared between replicas, and any issuer that is not loopback requires it. Run it so
that it keeps what it is given:

- Do not let Redis evict keys (`maxmemory-policy noeviction`, or no `maxmemory`). The
  server reads the policy from each primary at startup and refuses to start when it may
  evict. An evicted code or replay record is one the server can no longer hold to single
  use.
- Turn on persistence (AOF, `appendfsync everysec`). A restart without it drops every
  code and request in flight.
- Run a replicated topology: `cache.mode: sentinel` or `cluster`, with at least one replica
  per primary. Replication is asynchronous, so a failover can drop the last writes;
  `cache.replica_acks` makes each write wait for that many replicas (Redis `WAIT`). See
  [Configuration](https://www.nauthera.io/docs/v1/configuration#top-level-sections).
- Size it for your traffic and watch `nauthera_cache_operations_total` and
  `nauthera_cache_operation_seconds`.

With `rate_limit.backend: redis`, a Redis outage does not stop rate limiting: while Redis
cannot answer, each replica limits on its own `rps` and `burst` bucket, probes Redis again
every second, logs the switch both ways, and sets `nauthera_rate_limit_degraded` to 1. The
limits loosen by at most the replica count until Redis returns. See
[Rate Limits & Lockout](https://www.nauthera.io/docs/v1/rate-limits#rate-limiting).

## Backups

There is no built-in backup. Back up the database with your usual tooling (for example
CloudNativePG's backups or `pg_dump`), and back up the secrets that make it readable at
the same time:

- the key-encryption key and every retired one (`keys.encryption_key`,
  `keys.retired_keys`),
- the MFA seed key and every retired one (`mfa.encryption_key`, `mfa.retired_keys`),
- the main organization's signing keys.

A restore needs all of them. The database alone restores the main organization. With the
KEK it also restores the other organizations' signing keys, the external identity
providers' client secrets, the LDAP directories' bind passwords and the mail still
queued, and with the MFA key every user's TOTP factor.

## Schema migrations

The migrations are compiled into the binary.

- **Forward only.** There are no down migrations. Rolling back to an older binary after
  a migration has run is not supported; roll forward, or restore a backup.
- **Safe with several replicas.** The runner holds an advisory lock on one
  dedicated connection, so replicas migrate one after another instead of racing.
- **Atomic.** Each migration and its record in `schema_migrations` commit in the same
  transaction.
- **Tamper-evident.** A checksum of every applied migration is stored and checked on
  every start. If an applied migration was changed, startup fails.
- **PgBouncer.** The advisory lock needs a single session. Behind PgBouncer in
  transaction-pooling mode (`database.pgbouncer: true`), migrations should go over a
  direct or session-pooled connection; the server logs a warning when they would not.

There are two ways to apply them.

**The server migrates itself.** With `database.migrate: true`, each replica applies any
pending migrations when it starts, before it opens its listeners. Every replica also
creates the coming audit partitions, drops the expired ones and builds their indexes.
This needs the role that owns the tables, so the server's own credential could also
rewrite the audit log. With `database.migrate: false` and the default
`audit.partition_maintenance: internal`, the server does not check the schema version at
all, and a new binary on an old schema fails at the first query that needs a new table or
column.

**Database maintenance as a separate step.** `nauthera-server maintain` applies pending
migrations, creates the coming audit partitions, drops the expired ones, builds missing
partition indexes, and exits, non-zero if any step failed. It starts no listener. Run it
as a Job before every rollout, under the credential that owns the schema, and on a
schedule at least monthly so partitions stay ahead of the writes. Then set
`audit.partition_maintenance: external` (with `database.migrate` off) and
give the server its own role in `database.runtime_role`; `maintain` grants that role what
the server needs, with only `SELECT` and `INSERT` on the audit log. With `external`, the
server refuses to start while a migration it ships is unapplied, or while no audit
partition covers the current month.

The Helm chart does all of this with `appendOnlyAudit`: a `pre-install,pre-upgrade` hook
Job and a CronJob run `maintain` with `secrets.ownerDatabaseDsn`, and the server connects
as `appendOnlyAudit.runtimeRole`. See
[Installation](https://www.nauthera.io/docs/v1/installation#migrations-and-database-maintenance). Without
Helm, `deploy/k8s/migration-job.yaml` runs `maintain` as a one-shot Job.

## Upgrading

Signed releases are public on GHCR, and [Downloads](https://www.nauthera.io/downloads) lists every version; see
[Installation](https://www.nauthera.io/docs/v1/installation#from-the-published-chart). A released chart pins
both images by digest, so upgrading means `helm upgrade` to a newer chart version. The
procedure is:

1. Back up the database and the secrets listed above.
2. Apply the migrations: run `nauthera-server maintain` (with the chart's
   `appendOnlyAudit`, the pre-upgrade hook does it), or roll out with
   `database.migrate: true`.
3. Roll out the new image. The chart's Deployment and the raw manifests use
   `maxUnavailable: 0`, so old pods keep serving until new ones are ready.
4. During the rollout, old and new pods share one schema. The repository's rule for
   migrations is expand, then contract: add columns and tables before code uses them,
   and drop them only once no running version needs them.

The first version that keeps refresh tokens and revocations in the database also reads
the ones an earlier version wrote to Redis, for one refresh-token lifetime
(`issuer.refresh_token_ttl`, 30 days by default) after the new tables were created. Users
stay signed in across that upgrade, and a grant revoked before it stays revoked. Keep
Redis's data through that window.

Some migrations break that rule and need downtime. Migration 0049 moves
the time columns to `timestamptz` and three JSON columns to `jsonb`. A release from before
it writes values the new columns refuse, so when upgrading across it, stop every replica
of the earlier release first (scale the Deployment to zero, then upgrade). Each converted
table is locked while it is rewritten. The audit log is not converted.

A release whose migration declares a new audit index makes `maintain` build it on every
retained monthly partition, which takes far longer than usual on a large log. With the
chart, raise `helm upgrade --timeout` and `appendOnlyAudit.maintenance.activeDeadlineSeconds`
together for that release.

On `SIGTERM` a replica reports not ready and waits `server.drain_delay` (5s) for load
balancers to stop sending it traffic. It then drains its public HTTP and gRPC listeners
within `server.shutdown_timeout` (25s). Out of that budget, 5 seconds are reserved for
stopping the back-channel logout notifier and the background jobs, and
`audit.shutdown_timeout` (10s) for flushing the audit buffer, so the listeners get the
rest (10s at the defaults). Logout notifications still in flight are waited for; any not
delivered when the reserve runs out are cut short and recorded as failed. The server then
closes the rate limiter, cache and database. With tracing on, it flushes traces next, and
it stops the admin listener last, so probes and metrics answer until the end. A second
signal forces it to exit.

The trace flush and the admin listener's drain share one 5-second bound, so at the
defaults the worst case is about 35 seconds: the drain delay, the shutdown timeout and
that bound. The chart gives each pod 60 seconds to stop (`terminationGracePeriodSeconds`),
and the raw manifests in `deploy/k8s/` give it 40.

## What is cleaned up automatically

| Data | Cleanup |
|---|---|
| Sign-in sessions | Sessions past their absolute lifetime or idle past their timeout are deleted every hour by each replica. |
| Session history | Ended sessions are kept for `auth.session.history_retention` (2160h, 90 days), then deleted hourly. |
| Audit log | Whole months past `audit.retention` (400 days) are removed by dropping their monthly partitions. The server does this every `audit.maintenance_interval` (6h), or `nauthera-server maintain` does with `audit.partition_maintenance: external`. See [Audit Log](https://www.nauthera.io/docs/v1/audit-log#retention). |
| Codes, device codes, nonces | Expire in Redis with their own lifetime. |
| Refresh tokens and revocations | Deleted from the database every hour once they have lapsed. |
| Passkey challenges | Deleted every hour once expired. |
| Consent approvals | Never: they do not expire. |
| Expired pending sign-ins (for example, waiting for a second factor) | Deleted every hour. |
| Logout participation records (which applications each session signed in to) | Deleted after the user signs out and the logout notifications have been sent. Records whose session is gone are swept every hour. |
| Emailed links (password reset, email verification, invitations) | Deleted every hour once expired. |
| Known browsers and networks per user | Forgotten after 180 days without a sign-in from them, swept every 6 hours. |
| Sign-in failure records of identifiers no account holds | Deleted every hour once their window and lock have lapsed. |