Skip to content

The v1 docs follow nauthera-server's main branch and its 0.x releases on the way to 1.0, so fields and APIs can still change.

v1

NautheraServerRestarting

Docs version v1 (main)Nauthera is pre-1.0, in a design-partner program

A nauthera-server replica restarted three or more times in 30 minutes.

Meaning

process_start_time_seconds of one replica changed at least three times (the default threshold) in the last 30 minutes: the process keeps exiting and Kubernetes keeps starting it again. The usual causes are an OOM kill, a panic, a configuration the server refuses at startup, a liveness probe (/healthz on :9090) that stops answering, or a boot that takes longer than the startup probe allows (60 seconds with the chart's probes.startup defaults).

Impact

While a replica is down, the others serve its traffic; requests in flight on it fail and users may have to retry a sign-in. If every replica restarts in step, sign-in and token requests fail intermittently for every relying party. A crash loop also means the replica's queued audit events and back-channel logout deliveries can be cut short.

Diagnosis

  1. Find the replica and why it last stopped: kubectl -n <ns> get pods -l app.kubernetes.io/name=nauthera-server and kubectl -n &lt;ns&gt; get pod &lt;pod&gt; -o jsonpath='\{.status.containerStatuses[0].lastState.terminated\}'. reason: OOMKilled is the memory limit; Error with an exit code is the process.
  2. Read the log of the container that stopped: kubectl -n <ns> logs <pod> --previous. A refused configuration is logged at startup before the exit; a panic ends in a stack trace.
  3. Probe failures and their timing: kubectl -n <ns> describe pod <pod> (events).
  4. Memory before an OOM kill: process_resident_memory_bytes{job="<name>"}. Each password verification (argon2id) uses about 64 MiB, and up to auth.brute_force.max_concurrent_verifies (4) run at once per replica.
  5. Did it start with a rollout? kubectl -n <ns> rollout history deploy/<name>.

Mitigation

  • OOM kill: raise resources.limits.memory (512Mi by default), or lower config.auth.brute_force.max_concurrent_verifies so fewer derivations run at once.
  • Refused configuration: fix the value the startup log names, or roll back with helm rollback <release>.
  • Startup probe too short for a slow boot (migrations with config.database.migrate): raise probes.startup.failureThreshold, or move migrations to nauthera-server maintain (appendOnlyAudit).
  • Liveness failures under load: check NautheraServerHighLatency and the database alerts first; a replica starved of CPU or database connections answers its probes late.

Escalation

A single restart after a node event needs no escalation. A crash loop that a rollback does not stop, or any panic, goes to the owner of the identity platform. Open an issue with the --previous log (remove anything your deployment considers sensitive), the server version (nauthera_build_info) and the configuration keys you changed.

Alert runbooks · Generated from internal/runbook/alerts.yaml and operator/internal/runbook/alerts.yaml in nauthera-server.