Alert runbooks

Docs version v1 (main)Nauthera is pre-1.0, in a design-partner program

One runbook per alert of the Helm chart's PrometheusRule, the operator's group included. Each alert's runbook_url annotation links to its page here.

Alert	What it means
`NautheraServerDown`	No nauthera-server replica answers a Prometheus scrape.
`NautheraServerRestarting`	A nauthera-server replica restarted three or more times in 30 minutes.
`NautheraServerFileDescriptorsHigh`	A nauthera-server replica uses more than 80% of its file descriptor limit.
`NautheraServerErrorBudgetBurnFast`	Public requests fail with a 5xx fast enough to spend the error budget within days.
`NautheraServerErrorBudgetBurnSlow`	Public requests fail with a 5xx faster than the monthly error budget allows.
`NautheraServerHighLatency`	The 99th percentile latency of the protocol endpoints is above one second.
`NautheraDatabaseUnavailable`	One or more nauthera-server replicas cannot reach the database.
`NautheraDatabasePoolExhausted`	Requests on a nauthera-server replica are queuing for a database connection.
`NautheraCacheUnavailable`	One or more nauthera-server replicas cannot use the Redis cache.
`NautheraCacheErrors`	More than 1% of nauthera-server cache operations fail.
`NautheraAuditMandatoryEventsDropped`	Mandatory audit events were lost because their queue was full.
`NautheraAuditEventsDropped`	Non-mandatory audit events were lost because their queue was full.
`NautheraAuditWritesFailing`	nauthera-server cannot write audit batches to the database.
`NautheraAuditQueueBacklog`	An audit queue is more than 40% full.
`NautheraAuditPartitionMissing`	No audit log partition is attached for the current month.
`NautheraAuditPartitionsNotMaintained`	Audit log partitions for the coming months are missing.
`NautheraAuditExportFailing`	An audit exporter has failed every delivery for ten minutes.
`NautheraAuditExportLagging`	An audit exporter is more than 15 minutes behind the audit log.
`NautheraSignInFailureSpike`	Most password sign-ins are failing, at a rate that suggests an attack.
`NautheraAccountLockoutSpike`	More than ten brute-force locks were applied in 15 minutes.
`NautheraRateLimitingHigh`	More than 10% of public requests are refused with 429.
`NautheraThreatSignalCritical`	A threat signal for one organization is at or above its critical rate.
`NautheraThreatSignalElevated`	A threat signal for one organization is at or above its elevated rate.
`NautheraBackchannelLogoutFailing`	More than 10% of back-channel logout deliveries are failing.
`NautheraControlPlaneErrors`	More than 5% of gRPC AdminService calls fail on the server's side.
`NautheraControlPlaneRefusals`	The gRPC AdminService has refused calls for fifteen minutes.
`NautheraServerCertificateExpiringSoon`	A certificate a nauthera-server listener serves expires within seven days.
`NautheraServerCertificateExpiryImminent`	A certificate a nauthera-server listener serves expires within two days.
`NautheraServerNoReplicasAvailable`	The nauthera-server Deployment has no available replica.
`NautheraOperatorDown`	No nauthera-operator replica answers a Prometheus scrape.
`NautheraOperatorResourcesFailing`	Resources the operator manages are in phase Error.
`NautheraOperatorServerUnreachable`	Most of the operator's calls to the server's control plane get no answer.