# Keys & Secrets > Every key and secret nauthera-server holds, from signing keys to client secrets: how to supply each one, how to rotate it, and which ones to back up. ## Inventory | What | Where it lives | Supplied as | Lose it and… | |---|---|---|---| | Main org's signing keys (EC P-256 for ES256, RSA ≥ 2048 for RS256) | PEM files | `keys.signing_key_path`, `keys.rsa_signing_key_path` | Every token already issued stops verifying | | Other orgs' signing keys | The database, encrypted under the KEK | Generated when the org is created | That org's tokens stop verifying | | Key-encryption key (KEK) | Your secret store | `keys.encryption_key` (`_FILE`) | Every non-main org's signing keys, every identity provider's client secret, every LDAP directory's bind password and the mail not yet sent become unreadable | | MFA seed key | Your secret store | `mfa.encryption_key` (`_FILE`) | Every enrolled TOTP factor stops working | | Database credentials | Your secret store | `database.dsn` or `database.password` (`_FILE`) | | | Cache passwords | Your secret store | `cache.password`, `cache.sentinel_password` (`_FILE`) | | | Identity provider client secrets | The database, sealed under the KEK | An `IdentityProvider` resource's Secret, or `nauthera-cli create identity-provider --client-secret-file` | Sign-in through that provider fails until the secret is supplied again | | LDAP directory bind passwords | The database, sealed under the KEK | An `LDAPDirectory` resource's Secret | Sign-in through that directory fails until the password is supplied again | | Hosted captcha secret | Your secret store | `auth.captcha.secret` (`_FILE`), only for `turnstile`, `hcaptcha` or `recaptcha` | Requests that need a captcha answer `503` until it is replaced | | SMTP credentials | Your secret store | `mail.smtp.password` (`_FILE`); per organization, `mail.organizations[].smtp.username_file` and `password_file` | Mail stops until they are replaced | | Audit checkpoint key (Ed25519) | A PEM file | `audit.checkpoint_key_path` | New checkpoints are signed under a new key; keep the old public key to verify past months | | gRPC and HTTP TLS certificates | Files | `server.grpc.tls.*`, `server.http.tls.*` | | | Client secrets | The database, as SHA-256 digests | Generated by the server and shown at most once, or for an `OIDCClient` read from a Secret you hold (`spec.secret.secretRef`) | Nothing to recover: [issue a new one](#client-secrets) | To commit the Secrets that supply these to Git, see [Store Nauthera's secrets with Sealed Secrets](https://www.nauthera.io/docs/v1/guides/sealed-secrets). The `_FILE` convention is described in [Configuration](https://www.nauthera.io/docs/v1/configuration#secrets-from-files). The Helm chart mounts every secret as a file by default, except the two retired-key lists, which it passes as environment variables because the server has no file form for them (#340). ## Signing keys The main organization signs with the PEM keys you configure under `keys.*` when `keys.provider` is `file`. EC keys may be SEC1 (`EC PRIVATE KEY`) or PKCS#8, and RSA keys PKCS#1 or PKCS#8. Either key alone is allowed; with only the EC key the deployment is ES256-only and discovery says so. `keys.default_alg` picks the algorithm for clients that did not choose one. ```bash openssl ecparam -name prime256v1 -genkey -noout -out ec.key openssl genrsa -out rsa.key 3072 ``` The Helm chart issues both keys from cert-manager instead, with `rotationPolicy: Never`. Every other organization gets its own EC P-256 and RSA-2048 keys when it is created. They are sealed with AES-256-GCM under the KEK, bound to the organization's ID, and stored in the database. Every key's `kid` is its RFC 7638 thumbprint. The JWKS lists the current public keys and any extra public keys you publish for a rotation. The main org's discovery documents and JWKS are served with `Cache-Control: public, max-age=300`. The server also keeps a copy in the shared cache for 270 to 330 seconds, and a restart does not clear it. After a key change, the old set can be served for up to 330 seconds, and a client may keep its copy for 300 seconds more. A request sent with `Cache-Control: no-cache` skips the server's copy and refreshes it. Other organizations' discovery documents and JWKS are built on every request and carry no `Cache-Control` header. ### Rotating the main org's signing key There is no automatic rotation. Publish the new public keys before you sign with them: a replica that has not restarted yet refuses a token signed with a key it does not hold, and relying parties see a new key only once the cached copies described above have expired. To rotate by hand: 1. Generate the new keys, and export the public halves of the current keys and of the new ones: ```bash openssl ecparam -name prime256v1 -genkey -noout -out ec-new.key openssl genrsa -out rsa-new.key 3072 openssl ec -in ec.key -pubout -out ec-previous.pub openssl rsa -in rsa.key -pubout -out rsa-previous.pub openssl ec -in ec-new.key -pubout -out ec-new.pub openssl rsa -in rsa-new.key -pubout -out rsa-new.pub ``` 2. Publish the new public keys while the current keys still sign: ```yaml keys: signing_key_path: /etc/nauthera/keys/ec.key rsa_signing_key_path: /etc/nauthera/keys/rsa.key verification_key_paths: [/etc/nauthera/keys/ec-new.pub] rsa_verification_key_paths: [/etc/nauthera/keys/rsa-new.pub] ``` 3. Restart every replica. Then wait at least eleven minutes: the server's copy of the JWKS can take up to 330 seconds to list the new keys, and a client may keep its copy for 300 seconds more. 4. Sign with the new keys, and publish the previous public keys so tokens already issued still verify: ```yaml keys: signing_key_path: /etc/nauthera/keys/ec-new.key rsa_signing_key_path: /etc/nauthera/keys/rsa-new.key verification_key_paths: [/etc/nauthera/keys/ec-previous.pub] rsa_verification_key_paths: [/etc/nauthera/keys/rsa-previous.pub] ``` 5. Restart every replica. 6. Once the longest access and ID token lifetime has passed since that restart, remove the previous public keys and restart again. Relying parties that cache the JWKS must refetch it when they see an unknown `kid`. Most libraries do. Organizations other than the main one cannot rotate their keys today. Rotating them without a restart is tracked in #300. ## The key-encryption key `keys.encryption_key` wraps the signing keys of every organization except the main one. For every organization, the main one included, it also seals the client secrets of its identity providers and the bind passwords of its LDAP directories. Each is sealed with the organization and the slug bound to it, so a sealed value copied onto another provider or directory does not open. The KEK also seals every message in the [email outbox](https://www.nauthera.io/docs/v1/email#the-outbox) and the admin API's pagination cursors. It must be **32 bytes**, given as raw bytes, 64 hex characters or base64: ```bash openssl rand -hex 32 > kek ``` > **Set it before you create your second organization.** The KEK is not one of the > startup checks. Without it, each replica makes up its own key in memory and logs a > warning. An organization created meanwhile works only on the replica that created it, > and only until that replica restarts. After that, no replica can sign or verify its > tokens. The same goes for an identity provider's client secret, and every replica > refuses the others' pagination cursors. Refusing to start without the key on an issuer that is not loopback is > tracked in #345. **Rotating the KEK.** Each sealed key records which KEK sealed it, so a rotation needs no downtime for existing data: 1. Make the new key `keys.encryption_key`. 2. Put the old key in `keys.retired_keys`. Retired keys are used only to decrypt. They have no `_FILE` form (#340), so pass them through the environment variable `NAUTHERA_SERVER_KEYS_RETIRED_KEYS` (comma-separated). 3. Restart. New organizations and identity provider secrets are sealed under the new key. Existing ones are **not** re-sealed, so the old key has to stay in `retired_keys` for as long as those organizations and secrets exist. Re-sealing is tracked in #353. ## The MFA seed key `mfa.encryption_key` encrypts every TOTP seed with AES-256-GCM. It uses the same 32-byte format as the KEK, and `mfa.provider: file` requires it. Two more keys are derived from it, each under a label of its own: the key that names the sign-in failure record of an identifier no account holds, and the key ALTCHA challenges are signed with. Every replica therefore needs the same `mfa.encryption_key`, so a lock and a challenge hold across the deployment. Rotate it the same way as the KEK: the new key in `mfa.encryption_key`, the old one in `NAUTHERA_SERVER_MFA_RETIRED_KEYS`, then restart. Seeds stay sealed under the old key until the user enrolls again, so keep the retired key. A rotation restarts the failure counts of identifiers no account holds and lifts their locks, once; account counts and locks are kept, and ALTCHA challenges in flight at the restart are refused. See [Multi-Factor Authentication](https://www.nauthera.io/docs/v1/mfa#seed-storage). ## Client secrets The server generates client secrets: 256 random bits, stored only as a SHA-256 digest. An `OIDCClient` can supply its own instead, from a Secret named by `spec.secret.secretRef`; the server refuses one shorter than 43 characters, the length of a minted one, and stores it the same way. With that much entropy a slow password derivation adds nothing, so none is used; an argon2id hash stored for an older secret still verifies and is replaced by the digest the first time it does. A client registered through the AdminService or by dynamic registration gets its secret once, in the response that minted it. - **Rotation** of an [`OIDCClient`](https://www.nauthera.io/docs/v1/oidc-client) happens in place: delete the Secret the operator delivered, and the operator mints a new secret for the same client ID, writes it back and records a `SecretRotated` event. Workloads that read the Secret pick up the new value when they reload it. For a supplied secret, write the new value into the Secret `spec.secret.secretRef` names: it replaces the old one at once. - A client registered with `nauthera-cli` or by dynamic registration is rotated by deleting it and creating it again. `nauthera-cli` keeps its ID; dynamic registration assigns a new one. Rotation in place for these clients is tracked in #351. - Prefer `private_key_jwt` for confidential clients you control: the server stores only the client's `jwks_uri`, and rotation happens on the client's side. ## Admin client secrets Each organization's `{slug}-admin-m2m` machine client also has a generated secret. - For organizations created through the admin API, it is returned once as `admin_m2m_secret` in the response. - For organizations created through the gRPC `CreateOrganization` call, which `nauthera-cli create org` uses, it is not returned at all. The response has no field for it, so nobody holds the secret and the client cannot be used. - For the main organization, it is written once to `auth.admin.bootstrap_secret_path`. That path is **empty by default**, and then the secret is thrown away. Set the path before the very first start if you want this credential. The Helm chart leaves it off (`adminBootstrap.enabled: false`), because the file cannot be read out of the shell-less image without a sidecar. None of these secrets can be rotated. See [Admin API & Users](https://www.nauthera.io/docs/v1/user-management#getting-a-token) for other ways to get an admin token. ## SMTP credentials The deployment relay's password is `mail.smtp.password`, with a `_FILE` form. An organization's own relay never takes credentials from configuration: `smtp.username_file` and `smtp.password_file` name the files they are read from, and the server reads them again at every delivery, so a rotated Kubernetes Secret takes effect without a restart. The Helm chart mounts them from the Secret each `mailOrganizations` entry names. See [Installation](https://www.nauthera.io/docs/v1/installation#email). ## The audit checkpoint key `audit.checkpoint_key_path` names a PKCS#8 PEM Ed25519 key that signs the audit log's checkpoints. It is a path with no `_FILE` form; mount it from a Secret, and never reuse a token-signing key. Keep every public key you have signed with, current and retired, in the trusted-keys file `nauthera-server audit verify` reads, outside the database. Rotate by mounting a new key and restarting, and keep the old public key trusted for the whole of `audit.retention`. See [Audit Log](https://www.nauthera.io/docs/v1/audit-log#tamper-evidence). ## TLS certificates The server reads its gRPC certificate, key and client CA again at each handshake, and takes a renewed one into use without a restart. A replacement it cannot use is logged once, and the last good material keeps serving. The HTTP certificate (`server.http.tls.*`) is read once, at startup: after a renewal, restart the server before the old certificate expires. Serving a renewed HTTP certificate without a restart is tracked in #329. ## Backing up Back these up together, and restore them together: - the database, - the KEK and every retired KEK, - the MFA seed key and every retired one, - the main org's signing key files, - the trusted-keys file of audit checkpoint public keys. A database backup without its KEK cannot restore any organization except the main one. One without the MFA key restores every user without a working second factor. See [Backups & Upgrades](https://www.nauthera.io/docs/v1/operations). ## Planned - Fail on an empty secret file instead of letting it blank a value set in the YAML file or the environment (#339), and accept retired keys as files (#340). - Source the mounted secrets from OpenBao (#337), and hold both encryption keys there instead of in a cluster Secret (#338).