Skip to content

The v1 docs follow nauthera-server's main branch and its 0.x releases on the way to 1.0, so fields and APIs can still change.

v1

NautheraDatabaseUnavailable

Docs version v1 (main)Nauthera is pre-1.0, in a design-partner program

One or more nauthera-server replicas cannot reach the database.

Meaning

nauthera_dependency_up{dependency="database"} has been 0 on at least one replica for a minute: its outage window opened because the readiness ping or the queries failed. The database check is critical, so /readyz fails and the replica leaves the Service. Causes are a database that is down or failing over, a network path that broke (a NetworkPolicy, DNS), a credential that was rotated or expired, a TLS setting the database no longer accepts, or a database that refuses new connections (max_connections).

Impact

Each affected replica stops serving: no sign-in, token issuance, refresh or admin request. If every replica is affected, sign-in is down for every relying party and NautheraServerNoReplicasAvailable follows. Tokens already issued keep working at resource servers that validate them offline. Audit events are held in the replica's queues meanwhile; see the audit alerts.

Diagnosis

  1. How many replicas are affected: nauthera_dependency_up\{job="<name>", dependency="database"\}.
  2. The server's view: kubectl -n <ns> logs <pod> | jq 'select(.msg=="dependency is failing" or .msg=="health check failed")'. error_class and error say whether it is a timeout, a refused connection, authentication or TLS.
  3. Readiness from the pod's admin listener: kubectl -n <ns> port-forward pod/<pod> 9090:9090 and curl -s localhost:9090/readyz (checks.database).
  4. Check the database itself with its owner: is it up, did it fail over, is max_connections reached?
  5. Changes on the path: NetworkPolicies, the database Service or DNS name, the Secret behind secrets.databaseDsn.

Mitigation

  • Database down or failing over: restore it; replicas return to the Service on their own once the ping succeeds, and record system.dependency.recovered.
  • Rotated credential: update the Secret behind secrets.databaseDsn and restart the pods (kubectl -n <ns> rollout restart deploy/<name>).
  • Connection limit: lower config.database.max_open_conns or the replica count so that replicas times max_open_conns fits within the database's max_connections.
  • Network: restore the egress path from the pods to the database.

Escalation

Page the database owner at once, and the identity platform owner when every replica is affected: sign-in is down. If the database is healthy and reachable from other clients but the server cannot connect, open an issue with the dependency is failing log lines, the database.* settings you use (without the password) and the server version.

Alert runbooks · Generated from internal/runbook/alerts.yaml and operator/internal/runbook/alerts.yaml in nauthera-server.