# NautheraServerRestarting

A nauthera-server replica restarted three or more times in 30 minutes.

## Meaning

`process_start_time_seconds` of one replica changed at least three times (the default
`threshold`) in the last 30 minutes: the process keeps exiting and Kubernetes keeps
starting it again. The usual causes are an OOM kill, a panic, a configuration the server
refuses at startup, a liveness probe (`/healthz` on `:9090`) that stops answering, or a
boot that takes longer than the startup probe allows (60 seconds with the chart's
`probes.startup` defaults).

## Impact

While a replica is down, the others serve its traffic; requests in flight on it fail and
users may have to retry a sign-in. If every replica restarts in step, sign-in and token
requests fail intermittently for every relying party. A crash loop also means the
replica's queued audit events and back-channel logout deliveries can be cut short.

## Diagnosis

1. Find the replica and why it last stopped:
   `kubectl -n <ns> get pods -l app.kubernetes.io/name=nauthera-server` and
   `kubectl -n &lt;ns&gt; get pod &lt;pod&gt;
   -o jsonpath='\{.status.containerStatuses[0].lastState.terminated\}'`.
   `reason: OOMKilled` is the memory limit; `Error` with an exit code is the process.
2. Read the log of the container that stopped: `kubectl -n <ns> logs <pod> --previous`.
   A refused configuration is logged at startup before the exit; a panic ends in a stack
   trace.
3. Probe failures and their timing: `kubectl -n <ns> describe pod <pod>` (events).
4. Memory before an OOM kill: `process_resident_memory_bytes{job="<name>"}`. Each
   password verification (argon2id) uses about 64 MiB, and up to
   `auth.brute_force.max_concurrent_verifies` (4) run at once per replica.
5. Did it start with a rollout? `kubectl -n <ns> rollout history deploy/<name>`.

## Mitigation

- OOM kill: raise `resources.limits.memory` (512Mi by default), or lower
  `config.auth.brute_force.max_concurrent_verifies` so fewer derivations run at once.
- Refused configuration: fix the value the startup log names, or roll back with
  `helm rollback <release>`.
- Startup probe too short for a slow boot (migrations with `config.database.migrate`):
  raise `probes.startup.failureThreshold`, or move migrations to `nauthera-server
  maintain` (`appendOnlyAudit`).
- Liveness failures under load: check NautheraServerHighLatency and the database alerts
  first; a replica starved of CPU or database connections answers its probes late.

## Escalation

A single restart after a node event needs no escalation. A crash loop that a rollback
does not stop, or any panic, goes to the owner of the identity platform. Open an issue
with the `--previous` log (remove anything your deployment considers sensitive), the
server version (`nauthera_build_info`) and the configuration keys you changed.

---

[Alert runbooks](https://www.nauthera.io/docs/v1/runbooks/alerts) · Generated from [`internal/runbook/alerts.yaml`](https://github.com/nauthera/nauthera-server/blob/main/internal/runbook/alerts.yaml) and [`operator/internal/runbook/alerts.yaml`](https://github.com/nauthera/nauthera-server/blob/main/operator/internal/runbook/alerts.yaml) in nauthera-server.