# NautheraAuditWritesFailing

nauthera-server cannot write audit batches to the database.

## Meaning

`nauthera_audit_flushes_total{result="error"}` has been increasing for five minutes:
audit batches fail to reach the database. Mandatory events are held and retried until it
takes them. A failed batch of other events is given up: those events are written to the
application log instead and counted in a `system.audit.write_failed` record. Causes are
a database outage, a role that lacks `INSERT` on the audit log (for example after a
migration whose grants `nauthera-server maintain` has not applied), statement timeouts,
or a full disk.

## Impact

Auditors and the SIEM: records are late (mandatory) or missing (the rest) until writes
recover. Once the database has refused mandatory events for a while, or their queue is
four fifths full, the pipeline is degraded: the `audit` readiness check reports it
(the replica stays in service), and with `audit.fail_closed: true` administrators'
user and organization changes, organization changes over the gRPC control plane and
second-factor changes are refused with a `503`. Sign-in and token requests are never
refused for the audit log's health.

## Diagnosis

1. Rate of failing batches per replica: `rate(nauthera_audit_flushes_total\{
   job="<name>",result="error"\}[5m])`.
2. The error: `kubectl -n <ns> logs <pod> | jq 'select(.msg=="audit store refused
   mandatory events; holding them for retry" or .msg=="audit events could not be
   persisted")'`; `error_class` and `error` name it.
3. Pipeline state: `kubectl -n <ns> port-forward pod/<pod> 9090:9090` and
   `curl -s localhost:9090/readyz` (`checks.audit`), and the log line
   `audit pipeline degraded`.
4. Database health: NautheraDatabaseUnavailable, disk usage, and with the database owner
   whether the server's role can insert into `audit_events`.
5. Mandatory queue: `nauthera_audit_buffer_depth{job="<name>",lane="priority"}`.

## Mitigation

- Database outage or full disk: restore it; held mandatory events are written on the
  next attempt, recorded as `system.audit.write_delayed`, and the pipeline records
  `system.audit.recovered`.
- Missing privileges with `appendOnlyAudit`: run the maintenance job, which grants the
  runtime role what it needs:
  `kubectl -n <ns> create job --from=cronjob/<name>-maintain <name>-maintain-manual`.
- A partition problem: follow NautheraAuditPartitionMissing.
- With `audit.fail_closed` refusing administrators while this is fixed, decide with the
  security owner whether to wait or switch it off temporarily.

## Escalation

Page the identity platform owner and the database owner; tell the security or compliance
owner, since records are being lost or delayed. If the database is healthy and the
server's role can insert, open an issue with the error lines (they carry no event
payload) and the server version.

---

[Alert runbooks](https://www.nauthera.io/docs/v1/runbooks/alerts) · Generated from [`internal/runbook/alerts.yaml`](https://github.com/nauthera/nauthera-server/blob/main/internal/runbook/alerts.yaml) and [`operator/internal/runbook/alerts.yaml`](https://github.com/nauthera/nauthera-server/blob/main/operator/internal/runbook/alerts.yaml) in nauthera-server.