Backups & Upgrades

Docs version v1 (main) · Nauthera is pre-1.0, in a design-partner program · Updated November 4, 2026

#Where state lives

Refresh tokens, the records that detect a reused one, and every revocation are kept in the database, so they survive losing Redis. Redis still holds single-use state that must be shared between replicas, and any issuer that is not loopback requires it. Run it so that it keeps what it is given:

• Do not let Redis evict keys (maxmemory-policy noeviction, or no maxmemory). The server reads the policy from each primary at startup and refuses to start when it may evict. An evicted code or replay record is one the server can no longer hold to single use.
• Turn on persistence (AOF, appendfsync everysec). A restart without it drops every code and request in flight.
• Run a replicated topology: cache.mode: sentinel or cluster, with at least one replica per primary. Replication is asynchronous, so a failover can drop the last writes; cache.replica_acks makes each write wait for that many replicas (Redis WAIT). See Configuration.
• Size it for your traffic and watch nauthera_cache_operations_total and nauthera_cache_operation_seconds.

With rate_limit.backend: redis, a Redis outage does not stop rate limiting: while Redis cannot answer, each replica limits on its own rps and burst bucket, probes Redis again every second, logs the switch both ways, and sets nauthera_rate_limit_degraded to 1. The limits loosen by at most the replica count until Redis returns. See Rate Limits & Lockout.

#Backups

There is no built-in backup. Back up the database with your usual tooling (for example CloudNativePG's backups or pg_dump), and back up the secrets that make it readable at the same time:

• the key-encryption key and every retired one (keys.encryption_key, keys.retired_keys),
• the MFA seed key and every retired one (mfa.encryption_key, mfa.retired_keys),
• the main organization's signing keys.

A restore needs all of them. The database alone restores the main organization. With the KEK it also restores the other organizations' signing keys, the external identity providers' client secrets, the LDAP directories' bind passwords and the mail still queued, and with the MFA key every user's TOTP factor.

#Schema migrations

The migrations are compiled into the binary.

• Forward only. There are no down migrations. Rolling back to an older binary after a migration has run is not supported; roll forward, or restore a backup.
• Safe with several replicas. The runner holds an advisory lock on one dedicated connection, so replicas migrate one after another instead of racing.
• Atomic. Each migration and its record in schema_migrations commit in the same transaction.
• Tamper-evident. A checksum of every applied migration is stored and checked on every start. If an applied migration was changed, startup fails.
• PgBouncer. The advisory lock needs a single session. Behind PgBouncer in transaction-pooling mode (database.pgbouncer: true), migrations should go over a direct or session-pooled connection; the server logs a warning when they would not.

There are two ways to apply them.

The server migrates itself. With database.migrate: true, each replica applies any pending migrations when it starts, before it opens its listeners. Every replica also creates the coming audit partitions, drops the expired ones and builds their indexes. This needs the role that owns the tables, so the server's own credential could also rewrite the audit log. With database.migrate: false and the default audit.partition_maintenance: internal, the server does not check the schema version at all, and a new binary on an old schema fails at the first query that needs a new table or column.

Database maintenance as a separate step. nauthera-server maintain applies pending migrations, creates the coming audit partitions, drops the expired ones, builds missing partition indexes, and exits, non-zero if any step failed. It starts no listener. Run it as a Job before every rollout, under the credential that owns the schema, and on a schedule at least monthly so partitions stay ahead of the writes. Then set audit.partition_maintenance: external (with database.migrate off) and give the server its own role in database.runtime_role; maintain grants that role what the server needs, with only SELECT and INSERT on the audit log. With external, the server refuses to start while a migration it ships is unapplied, or while no audit partition covers the current month.

The Helm chart does all of this with appendOnlyAudit: a pre-install,pre-upgrade hook Job and a CronJob run maintain with secrets.ownerDatabaseDsn, and the server connects as appendOnlyAudit.runtimeRole. See Installation. Without Helm, deploy/k8s/migration-job.yaml runs maintain as a one-shot Job.

#Upgrading

Signed releases are public on GHCR, and Downloads lists every version; see Installation. A released chart pins both images by digest, so upgrading means helm upgrade to a newer chart version. The procedure is:

1. Back up the database and the secrets listed above.
2. Apply the migrations: run nauthera-server maintain (with the chart's appendOnlyAudit, the pre-upgrade hook does it), or roll out with database.migrate: true.
3. Roll out the new image. The chart's Deployment and the raw manifests use maxUnavailable: 0, so old pods keep serving until new ones are ready.
4. During the rollout, old and new pods share one schema. The repository's rule for migrations is expand, then contract: add columns and tables before code uses them, and drop them only once no running version needs them.

The first version that keeps refresh tokens and revocations in the database also reads the ones an earlier version wrote to Redis, for one refresh-token lifetime (issuer.refresh_token_ttl, 30 days by default) after the new tables were created. Users stay signed in across that upgrade, and a grant revoked before it stays revoked. Keep Redis's data through that window.

Some migrations break that rule and need downtime. Migration 0049 moves the time columns to timestamptz and three JSON columns to jsonb. A release from before it writes values the new columns refuse, so when upgrading across it, stop every replica of the earlier release first (scale the Deployment to zero, then upgrade). Each converted table is locked while it is rewritten. The audit log is not converted.

A release whose migration declares a new audit index makes maintain build it on every retained monthly partition, which takes far longer than usual on a large log. With the chart, raise helm upgrade --timeout and appendOnlyAudit.maintenance.activeDeadlineSeconds together for that release.

On SIGTERM a replica reports not ready and waits server.drain_delay (5s) for load balancers to stop sending it traffic. It then drains its public HTTP and gRPC listeners within server.shutdown_timeout (25s). Out of that budget, 5 seconds are reserved for stopping the back-channel logout notifier and the background jobs, and audit.shutdown_timeout (10s) for flushing the audit buffer, so the listeners get the rest (10s at the defaults). Logout notifications still in flight are waited for; any not delivered when the reserve runs out are cut short and recorded as failed. The server then closes the rate limiter, cache and database. With tracing on, it flushes traces next, and it stops the admin listener last, so probes and metrics answer until the end. A second signal forces it to exit.

The trace flush and the admin listener's drain share one 5-second bound, so at the defaults the worst case is about 35 seconds: the drain delay, the shutdown timeout and that bound. The chart gives each pod 60 seconds to stop (terminationGracePeriodSeconds), and the raw manifests in deploy/k8s/ give it 40.

#What is cleaned up automatically