# NautheraDatabasePoolExhausted

Requests on a nauthera-server replica are queuing for a database connection.

## Meaning

On one replica, more than 90% (the threshold) of `database.max_open_conns` connections
have been in use for ten minutes, and requests had to wait for one
(`go_sql_wait_count_total` keeps rising). Either the pool is too small for the traffic,
or queries hold their connections too long: slow queries, lock waits, or a database that
is overloaded.

## Impact

Requests on that replica wait before they touch the database, so sign-in, token and admin
requests slow down; past `database.statement_timeout` (30s) or a client's own timeout
they fail. Readiness pings may queue too.

## Diagnosis

1. Saturation and waits per replica: `go_sql_in_use_connections\{job="<name>",
   db_name="nauthera"\}`, `go_sql_max_open_connections\{...\}` and
   `rate(go_sql_wait_duration_seconds_total{job="<name>",db_name="nauthera"}[5m])`.
2. Is traffic higher than usual? `sum(rate(nauthera_http_requests_total{job="<name>"}[5m]))`.
3. Long-running queries and lock waits, with the database owner:
   `SELECT pid, state, wait_event_type, now() - query_start AS age, left(query, 80)
   FROM pg_stat_activity WHERE datname = current_database() ORDER BY age DESC;`
4. Whether the database itself is saturated (CPU, I/O, connections).

## Mitigation

- More traffic than the pool serves: raise `config.database.max_open_conns` (and
  `max_idle_conns`) within what the database allows across all replicas, or add replicas.
- Slow queries or lock waits: find the holder in `pg_stat_activity` with the database
  owner; a blocking session outside the server can be ended there.
- Overloaded database: scale it, or reduce load from other tenants of the same instance.

## Escalation

Escalate to the database owner when the database is slow or saturated. If one of the
server's own queries is consistently slow on a healthy database, open an issue with the
query text from `pg_stat_activity` (no parameter values), its plan if you can get one,
the table sizes involved and the server version.

---

[Alert runbooks](https://www.nauthera.io/docs/v1/runbooks/alerts) · Generated from [`internal/runbook/alerts.yaml`](https://github.com/nauthera/nauthera-server/blob/main/internal/runbook/alerts.yaml) and [`operator/internal/runbook/alerts.yaml`](https://github.com/nauthera/nauthera-server/blob/main/operator/internal/runbook/alerts.yaml) in nauthera-server.