# Observability: Metrics, Alerts and Tracing

> Metrics of nauthera-server and its operator, threat signals, the Helm chart's alerting rules, OpenTelemetry tracing, logs, health probes and Grafana dashboards.

## Metrics

Prometheus metrics are served on the **admin listener** (`:9090/metrics` by default;
see `metrics.enabled` and `metrics.path`). They are never served on the public port.
Besides its own metrics, the endpoint serves the Go runtime (`go_*`), process
(`process_*`) and database connection-pool (`go_sql_*`, with `db_name="nauthera"`)
metrics. The server's own metrics have the `nauthera_` prefix:

| Metric | Type | What it tells you |
|---|---|---|
| `nauthera_http_requests_total` | counter | Public HTTP requests by `route` (the operation), method and status |
| `nauthera_http_request_duration_seconds` | histogram | Public HTTP latency |
| `nauthera_http_in_flight_requests` | gauge | Requests being served now |
| `nauthera_http_cache_total` | counter | Per-operation response-cache hits and misses |
| `nauthera_grpc_handled_total` | counter | AdminService calls by method and code |
| `nauthera_grpc_handling_seconds` | histogram | AdminService latency |
| `nauthera_grpc_in_flight_requests` | gauge | AdminService calls in flight |
| `nauthera_cache_operations_total` | counter | Redis cache operations |
| `nauthera_cache_operation_seconds` | histogram | Redis cache latency |
| `nauthera_dependency_up` | gauge | Whether the `database` or `cache` is answering: `0` while its outage window is open |
| `nauthera_rate_limit_degraded` | gauge | `1` while the shared rate limiter cannot answer and this replica limits on its own counter instead, which loosens every limit by up to the replica count |
| `nauthera_auth_logins_total` | counter | Password sign-in attempts by `result` (`success`, `invalid`, `locked`, `throttled`, `unavailable`). A `success` means the password was right; a second factor may still follow. |
| `nauthera_auth_account_lockouts_total` | counter | Soft locks applied, re-locks included |
| `nauthera_auth_password_screens_total` | counter | New passwords screened, by `outcome` (`accepted`, `too_short`, `too_long`, `common`, `breached`, `unavailable`, `unchecked`) |
| `nauthera_audit_events_total` | counter | Audit events recorded, by category and outcome |
| `nauthera_audit_events_dropped_total` | counter | Audit events dropped because their queue was full, by `lane` |
| `nauthera_audit_buffer_depth`, `nauthera_audit_buffer_capacity` | gauge | Audit queue depth and size, by `lane` (`priority`, `buffered`) |
| `nauthera_audit_flushes_total`, `nauthera_audit_flush_duration_seconds` | counter, histogram | Audit batch writes |
| `nauthera_audit_partitions_missing` | gauge | Audit partitions not attached, for this month (`current`) and the coming ones (`upcoming`) |
| `nauthera_audit_export_lag_seconds` | gauge | How long the oldest record an audit exporter has yet to deliver has been stored |
| `nauthera_audit_exported_records_total` | counter | Records offered to an audit exporter, by `exporter` and `result` |
| `nauthera_backchannel_logout_deliveries_total` | counter | Back-channel logout deliveries by `outcome`; anything but `success` leaves a relying party's session alive |
| `nauthera_mail_deliveries_total` | counter | Outbound email delivery attempts by `result` (`sent`, `retried`, `failed`) |
| `nauthera_threat_signals_total` | counter | Audit records a [threat signal](#threat-signals) counts, by `signal`, `category` and `organization` |
| `nauthera_threat_threshold_per_hour` | gauge | The rate per hour at which each `signal` reads elevated or critical (`level`) |
| `nauthera_tls_certificate_expiration_timestamp_seconds` | gauge | When the certificate a listener (`http`, `grpc`) serves expires. The `http` listener's is read at startup; the `grpc` listener's is updated whenever a renewed certificate is loaded. |
| `nauthera_build_info` | gauge | Version, commit and Go version |

### Threat signals

Every attack signal of the dashboard's
[threat page](https://www.nauthera.io/docs/v1/admin-dashboard#threat-analytics) is exported too.
`nauthera_threat_signals_total` counts each audit record a signal counts as the replica
records it, once the audit policy has kept it, labelled with the signal, its category
and the organization's ID (empty for a platform record). Because it counts as records
are written, it sees what the [audit level](https://www.nauthera.io/docs/v1/audit-log#levels) records, as the
page does, and each replica counts its own records. `nauthera_threat_threshold_per_hour`
publishes the built-in elevated and critical rates for each signal, so an alert compares
against the same thresholds the page shows. Unlike `nauthera_audit_events_total`, the
threat counter carries the organization, so it can tell which tenant is under attack;
it never carries an actor or an address.

### The operator's metrics

The Kubernetes operator serves its metrics over TLS on `:8443`, only to callers the API
server authorizes for `GET /metrics` (`operator.metrics.readers` in the chart). Beside
controller-runtime's own, it publishes:

| Metric | Labels | What it tells you |
|---|---|---|
| `nauthera_operator_reconciles_total` | `kind`, `reason`, `cause` | Reconcile outcomes by the reason left on the resource (`Registered`, `Refused`, `Unreachable`, `Drifted`, …) and, for a refusal, the server's cause |
| `nauthera_operator_server_calls_total` | `call`, `code` | Calls on the server's control plane, by gRPC code |
| `nauthera_operator_server_call_duration_seconds` | `call` | Latency of those calls |
| `nauthera_operator_resources` | `kind`, `phase` | Resources managed, by phase |

See [Operator](https://www.nauthera.io/docs/v1/operator#metrics).

## Alerting

With the Helm chart, `metrics.serviceMonitor` creates a ServiceMonitor and
`metrics.prometheusRule.enabled` renders the server's alerting rules as a PrometheusRule;
both are off by default and need the Prometheus Operator CRDs. The rules come in groups
that can be switched off one by one under `metrics.prometheusRule.rules`:

| Group | What it alerts on |
|---|---|
| `availability` | No replica answers a scrape, restarts, file descriptors |
| `errors` | 5xx responses burning the error budget of `availabilityObjective` (99.9%), fast and slow |
| `latency` | The protocol endpoints' p99 above 1s |
| `dependencies` | A database or cache outage window, a saturated connection pool, failing cache operations |
| `audit` | Dropped audit events, failing audit writes, a backed-up queue, missing partitions, failing or lagging exporters |
| `security` | Sign-in failure and lockout spikes, a high share of `429`s |
| `threats` | One organization's rate per hour of a threat signal, over the last hour, reaching its critical rate (`NautheraThreatSignalCritical`) or its elevated rate (`NautheraThreatSignalElevated`) |
| `logout` | Failing back-channel logout deliveries |
| `controlPlane` | AdminService errors, and a caller it keeps refusing |
| `certificates` | A served certificate expiring within 7 or 2 days |
| `kubernetes` | No available replica (reads kube-state-metrics) |
| `operator` | No operator replica answers a scrape, resources in phase `Error`, the control plane not answering the operator. Rendered only with `operator.enabled` and the operator's ServiceMonitor (or `operatorSelector`). |

Leave out single alerts under `disabled`, and tune an alert's `severity`, `for` and
threshold under `alerts`. The threat alerts take their thresholds from
`nauthera_threat_threshold_per_hour` and have a `window` (`1h`) instead; by default the
critical one fires after `5m` and the elevated one, below the critical rate, after
`15m`. Every alert has a firing and a silent test case in the chart. See
[Installation](https://www.nauthera.io/docs/v1/installation#alerting); the chart README lists every alert
and its default.

Every alert carries a `runbook_url` annotation to its [runbook](https://www.nauthera.io/docs/v1/runbooks/alerts):
what it means, who is affected, how to diagnose it, what to do and when to escalate.
`metrics.prometheusRule.runbookUrl` (default `https://nauthera.io/runbooks/alerts`) sets
the base; point it at your own wiki to link `<runbookUrl>/<alert name>` there, or set it
empty to leave the annotation out. `nauthera-server runbook alerts <name>` prints the
runbook of the running version offline.

## Tracing

OpenTelemetry tracing is **opt-in** and exports over OTLP/gRPC:

```yaml
tracing:
  enabled: true
  endpoint: tempo:4317
  insecure: true          # no TLS to the collector
  sample_ratio: 1.0       # head-based sampling, 0.0–1.0
  environment: production # deployment.environment resource attribute
```

The public HTTP handler, the gRPC server, database queries, cache calls, SMTP delivery
and back-channel logout deliveries are instrumented.

## Logs

Structured logs come from `zap`, as `logging.format: json` (the default) or `console`.
Each request's own log line (`http_request`, or `grpc_call` for the AdminService) carries
its `request_id`, and, when tracing is on, `trace_id` and `span_id`, so you can go from a
request to its trace. An `http_request` line also carries `client_kind`, what kind of
software the User-Agent claims sent the request (`browser`, `ai_agent`, `crawler`,
`automation`, `library` or `unknown`), and, for anything but a browser, `client_agent`,
the product it names, such as `ClaudeBot` or `curl`. Both are a claim the sender makes;
the [audit log](https://www.nauthera.io/docs/v1/audit-log#what-a-record-says) labels its records the same way. Other lines logged while serving it carry them only when they are
logged through the request's logger.
DSNs, driver errors and secrets are logged on the server and never returned to clients.

## Health

| Probe | Path / mechanism |
|---|---|
| Liveness | `GET /healthz` on `:9090` |
| Readiness | `GET /readyz` on `:9090`. Fails while the database is unavailable and during shutdown. |
| Startup | `GET /startupz` on `:9090` |
| gRPC | `grpc.health.v1` shares the AdminService listener on `:9091`, which takes mTLS or, without TLS, listens on loopback only, so the kubelet's native gRPC probe cannot reach it. Probe the admin port's HTTP endpoints above instead. |
| Inside the image | `nauthera-server healthcheck --url http://127.0.0.1:9090/readyz` exits 0 or 1 |

The Redis cache and the audit pipeline are checked as soft dependencies: if either is
down, the server reports degraded, but the replica is not taken out of service.

## Dashboards

The Docker Compose stack in the repository provisions Grafana with:

- a **server overview** dashboard (`/d/nauthera-overview`) on Prometheus: HTTP and gRPC
  request rates and latency, requests in flight, the database pool and cache, goroutines
  and memory, and the build,
- an **audit-log explorer** (`/d/nauthera-audit`) that reads `audit_events` from
  PostgreSQL and opens a record's trace in Tempo when the request was sampled, and
- a **threats** dashboard (`/d/nauthera-threats`) on Prometheus: how many organization
  and signal pairs are elevated or critical now, every signal counted in the last hour
  against its thresholds, with a link from the organization to the audit-log explorer,
  and the signals over time by signal, category and organization.

Tempo is provisioned as a separate datasource: search traces in Explore. Nothing links a
metric to its trace. The server attaches `trace_id` exemplars to its HTTP and gRPC
latency histograms when the trace is sampled, but `/metrics` does not serve the
OpenMetrics format, so a Prometheus with default scrape settings, like this one,
receives none.

Prometheus in the same stack loads `deploy/compose/alerts.yml`, the Helm chart's alerting
rules rendered for the compose stack.