Skip to content

The v1 docs follow nauthera-server's main branch and its 0.x releases on the way to 1.0, so fields and APIs can still change.

v1

NautheraOperatorServerUnreachable

Docs version v1 (main)Nauthera is pre-1.0, in a design-partner program

Most of the operator's calls to the server's control plane get no answer.

Meaning

For fifteen minutes, more than half of the operator's calls to the gRPC AdminService (nauthera_operator_server_calls_total) ended Unavailable or DeadlineExceeded: nothing answered at the endpoint, the TLS handshake failed, or the server took too long. Once a call gets no answer the operator only probes until one does, so the calls counted are mostly its probes.

Impact

Application teams: nothing they declare reaches the server, and resources that need a write wait in Pending under Unreachable. Nothing is removed and no Secret is deleted while the server does not answer, and clients already registered keep working. If the server itself is down, its own alerts (NautheraServerDown) say so and users are affected too.

Diagnosis

  1. Is the server up? Check NautheraServerDown and kubectl -n <ns> get pods.
  2. Why the operator gets no answer: kubectl -n <release-ns> describe pod <operator-pod> shows the readiness probe's reason and message, for example Unreachable: no answer from <endpoint>: connection refused or certificate signed by unknown authority.
  3. Which calls fail: sum by (call, code) (rate(nauthera_operator_server_calls_total[10m])).
  4. The path: the operator dials port 9091 directly; a proxy that terminates TLS in between breaks the handshake.

Mitigation

  • Connection refused: check operator.server.endpoint against the server's gRPC Service, and that NetworkPolicies admit the operator's egress and the server's ingress on 9091.
  • Unknown authority: give the operator the CA that issued the server's gRPC certificate (ca.crt in its credential Secret, or operator.server.caSecret).
  • Certificate required or bad certificate: issue the operator's certificate from the CA the server's listener trusts, and dial past any TLS-terminating proxy.
  • A slow server (DeadlineExceeded): follow the server's latency and database alerts. Recovery needs no action on the resources: they are sent again within about a minute of the first answer.

Escalation

Escalate to the owner of the server's deployment when the server is down or slow, and to the platform team for certificates and network policy. Open an issue with the operator's and the server's versions if the handshake fails with credentials both sides accept.

Alert runbooks · Generated from internal/runbook/alerts.yaml and operator/internal/runbook/alerts.yaml in nauthera-server.