# NautheraOperatorDown

No nauthera-operator replica answers a Prometheus scrape.

## Meaning

Prometheus has had no successful scrape of the operator's metrics endpoint (`:8443`,
TLS, behind the `<release>-nauthera-server-operator-metrics` Service) for ten minutes.
No operator pod is running, or Prometheus can no longer scrape it: its ServiceAccount
is not among `operator.metrics.readers` (the scrape answers 401 or 403), the
ServiceMonitor changed, or a NetworkPolicy refuses it.

## Impact

If the operator is really gone, nothing that application teams declare reaches the
server: a new or changed OIDCClient, Organization, IdentityProvider or Branding is not
applied, a deleted one is not removed, and a client Secret a workload deleted to rotate
it is not written again. Clients already registered keep working, and users keep signing
in. If only scraping broke, nothing is affected but the other operator alerts are blind.

## Diagnosis

1. `kubectl -n <release-ns> get pods -l app.kubernetes.io/component=operator` and
   `kubectl -n <release-ns> describe pod <pod>` for crash loops, scheduling or image
   pull failures.
2. `kubectl -n <release-ns> logs <pod> --previous` for the last crash.
3. Pods running: open Prometheus' Targets page for the operator's job. A `401` or `403`
   means the scraper's ServiceAccount is not authorized for `GET /metrics`.

## Mitigation

- A crash or a failed rollout: roll back the release (`helm rollback <release>`) and
  read the crash in the previous container's log.
- A scrape refused with `401` or `403`: add Prometheus' ServiceAccount to
  `operator.metrics.readers`; the ServiceMonitor must present a ServiceAccount token.
- A NetworkPolicy refusing the scrape: admit Prometheus to the operator's port 8443.

## Escalation

Involve the platform team that owns the release. Application teams waiting on a client
registration can be told nothing they declare is lost: the operator applies it once it
runs again. If the operator crashes on a resource it accepted before, open an issue with
the crashed container's log and the operator's version.

---

[Alert runbooks](https://www.nauthera.io/docs/v1/runbooks/alerts) · Generated from [`internal/runbook/alerts.yaml`](https://github.com/nauthera/nauthera-server/blob/main/internal/runbook/alerts.yaml) and [`operator/internal/runbook/alerts.yaml`](https://github.com/nauthera/nauthera-server/blob/main/operator/internal/runbook/alerts.yaml) in nauthera-server.