# NautheraControlPlaneErrors

More than 5% of gRPC AdminService calls fail on the server's side.

## Meaning

Over ten minutes, more than the threshold (5%) of calls counted in
`nauthera_grpc_handled_total` ended with `Unknown`, `Internal`, `Unavailable`, `DataLoss`
or `DeadlineExceeded`, for fifteen minutes. These are failures of the server, not
refusals of the caller: usually the database is unavailable or slow, or the server hit a
defect.

## Impact

The Kubernetes operator cannot register or update OIDC clients, brandings, identity
providers or organizations; new manifests stay pending and changes do not reach the
server. Clients already registered keep working, and users can sign in. `nauthera-cli`
calls fail too.

## Diagnosis

1. Which methods and codes: `sum by (method, code) (rate(nauthera_grpc_handled_total\{
   job="<name>"\}[10m]))`.
2. The failing calls: `kubectl -n <ns> logs deploy/<name> | jq 'select(.msg=="grpc_call"
   and .code!="OK")'` (`method`, `code`, `peer_identity`, `error`, `request_id`).
3. Database health: NautheraDatabaseUnavailable, NautheraDatabasePoolExhausted.
4. The operator's side: `nauthera_operator_server_calls_total` by `code`, and
   `kubectl get oidcclients -A -o wide` for resources left `Unreachable` or `Refused`.

## Mitigation

- Database problems: fix them; the operator retries on its own and sends waiting
  resources once the server answers (see
  [when the server does not answer](https://www.nauthera.io/docs/v1/operator#when-the-server-does-not-answer)).
- A failure since a rollout: `helm rollback <release>`.

## Escalation

Escalate to the identity platform owner if the operator's changes are needed urgently.
If calls fail with `Internal` or `Unknown` while the database is healthy, open an issue
with the `grpc_call` log lines, the method and the server and operator versions.

---

[Alert runbooks](https://www.nauthera.io/docs/v1/runbooks/alerts) · Generated from [`internal/runbook/alerts.yaml`](https://github.com/nauthera/nauthera-server/blob/main/internal/runbook/alerts.yaml) and [`operator/internal/runbook/alerts.yaml`](https://github.com/nauthera/nauthera-server/blob/main/operator/internal/runbook/alerts.yaml) in nauthera-server.