# NautheraDatabaseUnavailable

One or more nauthera-server replicas cannot reach the database.

## Meaning

`nauthera_dependency_up{dependency="database"}` has been 0 on at least one replica for a
minute: its outage window opened because the readiness ping or the queries failed. The
database check is critical, so `/readyz` fails and the replica leaves the Service. Causes
are a database that is down or failing over, a network path that broke (a
NetworkPolicy, DNS), a credential that was rotated or expired, a TLS setting the database
no longer accepts, or a database that refuses new connections (`max_connections`).

## Impact

Each affected replica stops serving: no sign-in, token issuance, refresh or admin
request. If every replica is affected, sign-in is down for every relying party and
NautheraServerNoReplicasAvailable follows. Tokens already issued keep working at resource
servers that validate them offline. Audit events are held in the replica's queues
meanwhile; see the audit alerts.

## Diagnosis

1. How many replicas are affected: `nauthera_dependency_up\{job="<name>",
   dependency="database"\}`.
2. The server's view: `kubectl -n <ns> logs <pod> | jq 'select(.msg=="dependency is
   failing" or .msg=="health check failed")'`. `error_class` and `error` say whether it
   is a timeout, a refused connection, authentication or TLS.
3. Readiness from the pod's admin listener: `kubectl -n <ns> port-forward pod/<pod>
   9090:9090` and `curl -s localhost:9090/readyz` (`checks.database`).
4. Check the database itself with its owner: is it up, did it fail over, is
   `max_connections` reached?
5. Changes on the path: NetworkPolicies, the database Service or DNS name, the Secret
   behind `secrets.databaseDsn`.

## Mitigation

- Database down or failing over: restore it; replicas return to the Service on their own
  once the ping succeeds, and record `system.dependency.recovered`.
- Rotated credential: update the Secret behind `secrets.databaseDsn` and restart the
  pods (`kubectl -n <ns> rollout restart deploy/<name>`).
- Connection limit: lower `config.database.max_open_conns` or the replica count so that
  replicas times `max_open_conns` fits within the database's `max_connections`.
- Network: restore the egress path from the pods to the database.

## Escalation

Page the database owner at once, and the identity platform owner when every replica is
affected: sign-in is down. If the database is healthy and reachable from other clients
but the server cannot connect, open an issue with the `dependency is failing` log lines,
the `database.*` settings you use (without the password) and the server version.

---

[Alert runbooks](https://www.nauthera.io/docs/v1/runbooks/alerts) · Generated from [`internal/runbook/alerts.yaml`](https://github.com/nauthera/nauthera-server/blob/main/internal/runbook/alerts.yaml) and [`operator/internal/runbook/alerts.yaml`](https://github.com/nauthera/nauthera-server/blob/main/operator/internal/runbook/alerts.yaml) in nauthera-server.