# NautheraServerHighLatency

The 99th percentile latency of the protocol endpoints is above one second.

## Meaning

Over 15 minutes, the 99th percentile of `nauthera_http_request_duration_seconds` across
the protocol endpoints (authorize, PAR, token, device authorization, introspection,
revocation, UserInfo, end session, JWKS and discovery, by default) has stayed above the
threshold (1s). Database pool saturation, a slow cache, CPU starvation and a queue of
password verifications are the usual causes.

## Impact

Relying parties wait longer for tokens and redirects, and some time out: a client with a
short HTTP timeout on `/token` sees failed sign-ins although the server answers in the
end. Users see slow redirects. Discovery and JWKS fetches by resource servers slow down
too.

## Diagnosis

1. Which routes are slow: `histogram_quantile(0.99, sum by (route, le)
   (rate(nauthera_http_request_duration_seconds_bucket\{job="<name>"\}[5m])))`.
2. Database pool waits: `rate(go_sql_wait_duration_seconds_total\{job="<name>",
   db_name="nauthera"\}[5m])` and `go_sql_in_use_connections` against
   `go_sql_max_open_connections`.
3. Cache latency: `histogram_quantile(0.99, sum by (op, le)
   (rate(nauthera_cache_operation_seconds_bucket\{job="<name>"\}[5m])))`.
4. CPU: the chart sets no CPU limit by default; if you set one, look at throttling in your
   container metrics. Look at `nauthera_http_in_flight_requests` for a pile-up.
5. Slow sign-ins only: password verifications queue behind
   `auth.brute_force.max_concurrent_verifies`;
   `nauthera_auth_logins_total{result="throttled"}` rises when they wait longer than
   `verify_acquire_timeout` (2s).

## Mitigation

- Pool saturation: follow NautheraDatabasePoolExhausted.
- Slow cache: follow NautheraCacheErrors; check the cache's own CPU and memory.
- CPU or load: add replicas (`replicaCount`, or `autoscaling`), or raise
  `resources.requests.cpu` so the pods are scheduled where CPU is free.
- The threshold does not fit your environment: tune
  `metrics.prometheusRule.alerts.NautheraServerHighLatency.threshold`.

## Escalation

Escalate to the identity platform owner when latency keeps rising or relying parties
report timeouts. Involve the database or cache owner when their latency is the cause. If
the latency is inside the server with healthy dependencies, open an issue with the
per-route p99 graph, a few slow requests' `request_id`s (and trace IDs when tracing is on)
and the server version.

---

[Alert runbooks](https://www.nauthera.io/docs/v1/runbooks/alerts) · Generated from [`internal/runbook/alerts.yaml`](https://github.com/nauthera/nauthera-server/blob/main/internal/runbook/alerts.yaml) and [`operator/internal/runbook/alerts.yaml`](https://github.com/nauthera/nauthera-server/blob/main/operator/internal/runbook/alerts.yaml) in nauthera-server.