Skip to content

The v1 docs follow nauthera-server's main branch and its 0.x releases on the way to 1.0, so fields and APIs can still change.

v1

NautheraAuditWritesFailing

Docs version v1 (main)Nauthera is pre-1.0, in a design-partner program

nauthera-server cannot write audit batches to the database.

Meaning

nauthera_audit_flushes_total{result="error"} has been increasing for five minutes: audit batches fail to reach the database. Mandatory events are held and retried until it takes them. A failed batch of other events is given up: those events are written to the application log instead and counted in a system.audit.write_failed record. Causes are a database outage, a role that lacks INSERT on the audit log (for example after a migration whose grants nauthera-server maintain has not applied), statement timeouts, or a full disk.

Impact

Auditors and the SIEM: records are late (mandatory) or missing (the rest) until writes recover. Once the database has refused mandatory events for a while, or their queue is four fifths full, the pipeline is degraded: the audit readiness check reports it (the replica stays in service), and with audit.fail_closed: true administrators' user and organization changes, organization changes over the gRPC control plane and second-factor changes are refused with a 503. Sign-in and token requests are never refused for the audit log's health.

Diagnosis

  1. Rate of failing batches per replica: rate(nauthera_audit_flushes_total\{ job="<name>",result="error"\}[5m]).
  2. The error: kubectl -n <ns> logs <pod> | jq 'select(.msg=="audit store refused mandatory events; holding them for retry" or .msg=="audit events could not be persisted")'; error_class and error name it.
  3. Pipeline state: kubectl -n <ns> port-forward pod/<pod> 9090:9090 and curl -s localhost:9090/readyz (checks.audit), and the log line audit pipeline degraded.
  4. Database health: NautheraDatabaseUnavailable, disk usage, and with the database owner whether the server's role can insert into audit_events.
  5. Mandatory queue: nauthera_audit_buffer_depth{job="<name>",lane="priority"}.

Mitigation

  • Database outage or full disk: restore it; held mandatory events are written on the next attempt, recorded as system.audit.write_delayed, and the pipeline records system.audit.recovered.
  • Missing privileges with appendOnlyAudit: run the maintenance job, which grants the runtime role what it needs: kubectl -n <ns> create job --from=cronjob/<name>-maintain <name>-maintain-manual.
  • A partition problem: follow NautheraAuditPartitionMissing.
  • With audit.fail_closed refusing administrators while this is fixed, decide with the security owner whether to wait or switch it off temporarily.

Escalation

Page the identity platform owner and the database owner; tell the security or compliance owner, since records are being lost or delayed. If the database is healthy and the server's role can insert, open an issue with the error lines (they carry no event payload) and the server version.

Alert runbooks · Generated from internal/runbook/alerts.yaml and operator/internal/runbook/alerts.yaml in nauthera-server.