# The Nauthera Kubernetes Operator

> Nauthera's Kubernetes operator reconciles manifests into the server over mTLS, writes client secrets into Secrets, and handles deletion, drift and outages.

The Nauthera operator lets a team declare its identity setup the way it manages
everything else it owns: as manifests beside the application. The operator sends them to
nauthera-server through the gRPC `AdminService`, writes the server's answer to each
resource's `status`, and delivers a generated client secret into a `Secret` the workload
mounts.

| Resource | What the operator does with it |
|---|---|
| [`OIDCClient`](https://www.nauthera.io/docs/v1/oidc-client) | Registers and updates the client, delivers its secret into a Secret, notices drift, and removes the client under `deletionPolicy: Delete`. |
| [`Organization`](https://www.nauthera.io/docs/v1/organization-resource) | Creates the tenant or adopts the one holding its slug, keeps its display name, FAPI mode, verified-address requirement, second-factor policy, network zones and default branding in step, and claims its email domains with DNS verification. FIPS mode is fixed at creation. Cluster-scoped. |
| [`IdentityProvider`](https://www.nauthera.io/docs/v1/identity-provider-resource) | Registers an external OpenID Connect provider, or GitHub, for an organization, with the client secret read from a Secret, and sends the secret again when the Secret changes. |
| [`LDAPDirectory`](https://www.nauthera.io/docs/v1/ldap-directory-resource) | Registers an organization's LDAP directory or Active Directory, with the bind password read from a Secret and the CA bundle from a ConfigMap or Secret, and sends them again when either changes. |
| [`Branding`](https://www.nauthera.io/docs/v1/branding-resource) | Writes one client's sign-in branding, reports a document changed elsewhere as drifted, and removes it when the resource is deleted. |
| [`BrokerMembership`](https://www.nauthera.io/docs/v1/broker-membership-resource) | Declares one side of an [identity broker](https://www.nauthera.io/docs/v1/identity-broker) membership, the broker's invitation or the member's acceptance, and withdraws it when the resource is deleted. |

The operator is driven end to end against a real server on a kind cluster: clients under
both deletion policies, organizations, brandings, broker memberships declared from both
sides, the team roles, and a server that goes away and comes back.

Signed releases are public: the server and operator images at
`ghcr.io/nauthera/nauthera-server` and `ghcr.io/nauthera/nauthera-operator`, and the Helm
chart at `oci://ghcr.io/nauthera/charts/nauthera-server`, which anyone can pull without
a login. [Downloads](https://www.nauthera.io/downloads) lists every version. The source is still private.

Planned work: making the control plane safe for more than one operator, and configuring a
central server from downstream clusters. Which team may configure which organization is
decided in Kubernetes today; see [Who may change what](#who-may-change-what).

## Installing

The chart deploys the operator from the same release as the server. Pull the published
chart for its profiles, then install it:

```bash
VERSION=<version>   # the newest is at the top of /downloads
helm pull oci://ghcr.io/nauthera/charts/nauthera-server --version "$VERSION" --untar
helm upgrade --install nauthera oci://ghcr.io/nauthera/charts/nauthera-server --version "$VERSION" \
  --values nauthera-server/profiles/production.yaml \
  --set ingress.host=auth.example.com \
  --set certManager.grpcTLS.enabled=true \
  --set operator.enabled=true \
  --set 'operator.mainOrganizationNamespaces={nauthera}' \
  --set operator.networkPolicy.enabled=true \
  --set 'operator.networkPolicy.apiServer.cidrs={10.96.0.1/32}'
```

`certManager.grpcTLS` puts the control plane behind mutual TLS from a CA of its own and
issues the operator's client certificate. `operator.enabled` deploys the operator with
that credential, its permissions and the definitions of all six resource kinds.
`profiles/operator.yaml` does the same on a kind cluster.
`operator.mainOrganizationNamespaces` lets the manifests in the `nauthera` namespace, where
the examples on these pages live, configure the main organization.

| Value | Default | What it does |
|---|---|---|
| `operator.server.endpoint` | this release's gRPC Service | `host:port` of another server's control plane. |
| `operator.server.credentialSecret` | the chart's client certificate | A Secret with `tls.crt`, `tls.key` and `ca.crt` to use instead. |
| `operator.server.caSecret`, `caKey` | —, `ca.crt` | The trust anchor, when the credential Secret has no `ca.crt`. |
| `operator.namespaces` | `["*"]` | The namespaces whose manifests the operator sends to the server. A namespaced resource elsewhere is refused on the resource as `NamespaceNotCovered`. |
| `operator.mainOrganizationNamespaces` | `[]` | The namespaces whose manifests may configure the main organization, which no `Organization` declares. See [Which organization a resource configures](#which-organization-a-resource-configures). |
| `teamRoles.create` | `true` | Create the ClusterRoles a team binds to write these resources; see [Who may change what](#who-may-change-what). |
| `operator.replicaCount` | `1` | Replicas elect a leader; only the leader reconciles, and a standby takes over. |
| `operator.sweep.interval`, `maxRemovals` | `1h`, `10` | The orphan sweep (see [Deleting a manifest](#deleting-a-manifest)). |
| `operator.crds.install` | `true` | Render the resource definitions as templates, so `helm upgrade` applies a changed definition. They are kept on uninstall. |
| `operator.rbac.create` | `true` | Create the operator's ClusterRole and the Role for its leader Lease. |
| `operator.metrics.readers` | `[]` | Subjects (for example Prometheus's ServiceAccount) allowed to scrape metrics. |
| `operator.metrics.serviceMonitor.enabled` | `false` | Render a ServiceMonitor. |
| `operator.networkPolicy.enabled` | `false` | Admit only metrics from `metricsFrom`, health from `healthFrom`, and egress to DNS, the API server (`apiServer`) and the control plane (`serverTo`). |
| `operator.networkPolicy.cilium.enabled` | `false` | Render a CiliumNetworkPolicy instead, which names the API server and kubelet as entities and limits DNS to the control plane's name. |
| `operator.logLevel` | `info` | Log level. |

### What the operator may do on the server

The server decides what the operator may write, from the operator's entry in
`server.grpc.tls.allowed_clients`; in the chart that list is
`certManager.grpcTLS.allowedClients`. Each entry names the operations the identity may
call, the organizations it may act in, and the Kubernetes namespaces its resources may
come from. The chart's default entry is:

```yaml
certManager:
  grpcTLS:
    allowedClients:
      - identity: dns:nauthera-operator
        operations: [clients, brandings, identity_providers, ldap_directories]
        organizations: [default]
        namespaces: ["*"]
```

| Operation | Needed for |
|---|---|
| `clients` | `OIDCClient` |
| `brandings` | `Branding` |
| `identity_providers` | `IdentityProvider` |
| `ldap_directories` | [`LDAPDirectory`](https://www.nauthera.io/docs/v1/ldap-directory-resource) |
| `broker_memberships` | `BrokerMembership`, each side under the grant for its own organization |
| `email_domains` | An `Organization`'s `emailDomains` |
| `organizations` | `Organization`. Not in the default entry: add it, with the tenant's slug, if this cluster is to create and manage that tenant. |

To write clients, brandings or providers into a tenant, add its slug to
`organizations`. A call outside the grant is refused, and the resource reports `Refused`
with `OPERATION_NOT_GRANTED` or `OUTSIDE_GRANT`. The chart refuses to render an operator
whose identity (`dns:<certManager.grpcTLS.client.dnsName>`) is not listed with the
`clients` operation.

The main organization is configured with the deployment rather than by a resource. The
chart sets it under `config.orgs.main` (display name, FAPI mode, verified-address
requirement and default branding); see
[Organizations](https://www.nauthera.io/docs/v1/organizations#the-main-org).

Without the chart, run the image as `/nauthera-operator` with the flags below. The
resource definitions are in `deploy/crds/`, and the permissions it needs are in the
chart's `templates/operator-rbac.yaml`: the kinds above with their `status` and
`finalizers`, Secrets, ConfigMaps (read, for an LDAPDirectory's CA bundle), events,
token and access reviews for the metrics endpoint, and a Lease in its own namespace.

## Connecting to the server

Where the server is and how to reach it is operator configuration, not something a
manifest can change:

| Flag | Value |
|---|---|
| `--server-endpoint` | `host:port` of the server's gRPC listener. The operator dials it directly over mTLS, never through a TLS-terminating proxy. |
| `--server-ca-file` | CA bundle that the server's certificate chains to |
| `--client-cert-file`, `--client-key-file` | The operator's client certificate and key |
| `--main-organization-namespaces` | Comma-separated namespaces whose manifests may configure the main organization. A manifest in a namespace no organization admits is `NamespaceNotAdmitted`. The chart sets it from `operator.mainOrganizationNamespaces`. |
| `--namespaces` | Comma-separated namespaces whose manifests are sent, `*` for every one. With none set, every namespaced manifest is `NamespaceNotCovered`. The chart sets it from `operator.namespaces`. |

Mount the credential Secret whole, **not** with `subPath`, so a renewed certificate
reaches the pod. The operator re-reads the three files on every probe, so a renewed
certificate is used without a restart. The server reads its own certificate and client
CA at each handshake, so a renewal takes effect at that end without a restart too. The
pod is ready when the server answered, admitted the operator's identity and serves a
control-plane contract the operator understands. Otherwise `/readyz` fails with a
reason: `Unreachable`, `Refused`, `Unsupported`, `CredentialUnavailable` or
`TrustAnchorUnavailable`.

On the server, the certificate's SAN must be listed in `server.grpc.tls.allowed_clients`
as `dns:<name>` or `uri:<uri>`; the common name never matches. The server records that
identity as the manager of everything the operator writes, and the operator sees and
changes only its own clients.

The control plane is not served on the listener users reach. The chart puts it on the
in-cluster gRPC Service only; an operator in another cluster reaches it over one TLS
connection to port 9091, through a path that does not terminate TLS. The server reads no
client identity from a header.

Other flags: `--metrics-bind-address` (`:8443`, served over TLS to callers the API server
authorizes for `GET /metrics`), `--health-probe-bind-address` (`:8081`),
`--leader-elect` (the chart sets it) and `--leader-elect-namespace`,
`--graceful-shutdown-timeout` (`20s`), `--sweep-interval` and `--sweep-max-removals`,
`--version`, and the `--zap-*` logging flags.

Rotating the CA itself is not seamless. Distribute the old and new CAs together until
every certificate has been reissued.

### When the server does not answer

The operator waits out a server it cannot reach rather than acting on it:

- **Nothing is removed.** No client is removed from the server, no Secret is deleted, and
  every resource keeps the client ID, revision and Secret it had. The orphan sweep and the
  drift check skip a server that did not answer.
- **One probe, not a call per resource.** Once a probe or a call gets no answer, nothing
  else is sent until a probe gets one. The probe is retried after 15 seconds, doubling to
  two minutes, with jitter so many clusters do not return to a recovering server in step.
  Registered resources go to `Pending` under `Unreachable`, "the server stopped
  answering", and keep their state.
- **Recovery needs no one.** Once a probe gets an answer, waiting resources are sent
  within about a minute and return to `Ready`. A server restored from an older backup
  without clients it held reports them as `Drifted` rather than filling them in again.

## Upgrading

The control plane numbers its contract, and the operator supports a range of it, printed
by `/nauthera-operator --version`. A server outside that range is refused before anything is
written: the pod is not ready under `Unsupported`, every resource says so with both
versions, and the message says which side to upgrade. Both ends of one release support
the same contract.

## What the operator does with a manifest

The rest of this section describes `OIDCClient`. The other kinds have their own pages:
[Organization](https://www.nauthera.io/docs/v1/organization-resource),
[IdentityProvider](https://www.nauthera.io/docs/v1/identity-provider-resource),
[LDAPDirectory](https://www.nauthera.io/docs/v1/ldap-directory-resource),
[BrokerMembership](https://www.nauthera.io/docs/v1/broker-membership-resource) and
[Branding](https://www.nauthera.io/docs/v1/branding-resource).

- **Registers and updates** the client. It sends the spec whole for every generation the
  server does not hold yet, and nothing for a manifest that did not change.
- **Delivers the secret.** With `spec.secret.generate: true`, the server mints a secret
  and returns it once. The operator writes it into a Secret in the resource's namespace
  (`spec.secret.secretName`, or the resource's name) under the keys `client-id`,
  `client-secret` and `issuer`. It never writes over a Secret it does not own. Deleting
  that Secret rotates the credential: a new secret is minted, written, and recorded as a
  `SecretRotated` event. With `spec.secret.secretRef` instead, it reads a secret you
  supply from a Secret you hold, such as one a SealedSecret produces, sends it, and sends
  it again whenever that Secret changes; see
  [A secret you supply](https://www.nauthera.io/docs/v1/oidc-client#a-secret-you-supply).
- **Reports the outcome** on `status` and the `Ready` condition (`Registered`,
  `Refused`, `Unreachable`, `Drifted`, `SecretUndeliverable`, `NamespaceNotCovered` and
  others), with one event per outcome. `kubectl get oidcclients -o wide` shows the reason
  and message. The [OIDCClient reference](https://www.nauthera.io/docs/v1/oidc-client#status) lists every
  reason an `OIDCClient` reports, and the [operator runbooks](https://www.nauthera.io/docs/v1/runbooks/operator)
  say what each failure reason asks you to do.

### Deleting a manifest

`spec.deletionPolicy` decides, per resource. The default, `Retain`, keeps the client
registered and releases its Secret from the resource, so deleting a namespace cannot take
a live client with it. `Delete` removes the client from the server, and the Secret goes
with the resource. A finalizer (`nauthera.io/oidcclient`) holds the resource until that
is done; a removal the server does not take keeps it in phase `Deleting` and is retried.
`Organization`, `IdentityProvider` and `LDAPDirectory` have the same policy, with
`Retain` as the default. A `Branding` has none: deleting it removes the document.
Deleting either side of a `BrokerMembership` ends the membership.

A periodic sweep (`--sweep-interval`, hourly) removes clients registered under `Delete`,
and brandings, whose manifest disappeared without the finalizer running. It stops without
removing anything when it finds more than `--sweep-max-removals` (10), since that is more
likely a wrong listing than that many manifests gone.

### Changes made outside the operator

The operator notices when a client it registered was changed or removed by something
else, without reading clients one by one: it lists its clients only when the server
reports that its configuration moved. Such a resource goes to `Error` under `Drifted`,
and the change is **not** written over. Change the manifest, or annotate the resource
`nauthera.io/restore`, to write the declared state back (recorded as a `Restored`
event). The server's audit log says who made the change. A `Branding` is held to its
revision the same way, and reports a document changed elsewhere when it next writes.
[Keycloak with Terraform, or an operator](https://www.nauthera.io/compare/keycloak-terraform#3-drift) sets this
beside what Terraform, Crossplane and keycloak-config-cli do with the same change.

## How the resources fit together

An `Organization` is cluster-scoped and admits namespaces. Every other kind lives in one
of those namespaces and configures the organization that admits it. A `BrokerMembership`
in a namespace of each of two organizations joins them as broker and member.

```mermaid
flowchart TB
    subgraph cluster["Cluster-scoped: the platform team"]
        ORG["Organization acme<br/>namespaces: [acme, grafana]<br/>emailDomains: [acme.example]"]
    end
    subgraph acmens["Namespace acme"]
        IDP["IdentityProvider"]
        LDAP["LDAPDirectory"]
        BMM["BrokerMembership<br/>broker: default"]
        CL["OIDCClient"]
        BR["Branding"]
    end
    subgraph mainns["Namespace nauthera (main organization)"]
        BMB["BrokerMembership<br/>member: acme<br/>clients: [grafana]"]
    end
    ORG -. "admits" .-> acmens
    BMB <== "both sides: membership active" ==> BMM
    acmens --> OP["nauthera operator"]
    mainns --> OP
    OP -- "gRPC AdminService over mTLS,<br/>held to the operator's grant" --> SRV[("nauthera-server")]
```

## Which organization a resource configures

An `OIDCClient`, `IdentityProvider`, `LDAPDirectory`, `Branding` or `BrokerMembership`
configures an organization that admits its namespace. An `Organization` lists the
namespaces it admits in `spec.namespaces`; the main organization, which has no resource,
admits those in `operator.mainOrganizationNamespaces`. Only someone who may edit the
cluster-scoped `Organization`, or the release, can admit a namespace: labelling a
namespace grants nothing.

```mermaid
flowchart TD
    R["Namespaced resource"] --> C{"Namespace in the operator's<br/>--namespaces?"}
    C -- no --> NC["Error: NamespaceNotCovered"]
    C -- yes --> Q0{"spec.organization set?"}
    Q0 -- yes --> Q1{"That organization<br/>admits the namespace?"}
    Q1 -- no --> NA["Error: NamespaceNotAdmitted"]
    Q1 -- yes --> T["Its organization"]
    Q0 -- no --> Q2{"How many organizations<br/>admit the namespace?"}
    Q2 -- none --> NA
    Q2 -- "two or more" --> AMB["Error: OrganizationAmbiguous"]
    Q2 -- one --> T
    T --> Q3{"Server already holds the resource<br/>in another organization?"}
    Q3 -- yes --> CH["Error: OrganizationChanged"]
    Q3 -- no --> Q4{"Organization has an issuer yet?"}
    Q4 -- no --> NR["Pending: OrganizationNotReady"]
    Q4 -- yes --> SEND(["Sent to the server"])
```

A tenant's own namespace is admitted by one organization, and its resources name none. A
shared namespace, such as a Grafana holding clients of several organizations, is
admitted by each of them, and each resource names its own:

```yaml
apiVersion: nauthera.io/v1alpha1
kind: OIDCClient
metadata:
  name: grafana-acme
  namespace: grafana
spec:
  organization: acme       # needed only because several organizations admit grafana
  clientId: grafana-acme
```

- `spec.organization` is the organization's slug, `default` for the main organization. It
  may be set once on a resource that named none and is fixed after.
- A refused resource sends nothing, and one the server already holds is left as it is.
- `status.organization` records where the server holds a resource; it is never moved to
  another organization. Delete and recreate it to move it.
- Changing an `Organization`'s `namespaces`, or its issuer appearing, re-evaluates every
  resource the change may affect.

Each reason has a [runbook](https://www.nauthera.io/docs/v1/runbooks).

## Who may change what

Three layers decide whether a resource is applied, and each answers one question:

```mermaid
flowchart LR
    W["Who may write this kind<br/>in this namespace?"] -->|"Kubernetes RBAC,<br/>or CODEOWNERS in GitOps"| N["Which organizations<br/>admit this namespace?"]
    N -->|"Organization.spec.namespaces"| G["Which organizations may<br/>this operator touch?"]
    G -->|"the server's grant on the<br/>operator's identity"| OK(["Applied"])
```

The chart creates one ClusterRole per responsibility (`teamRoles.create`):

| Role | Bind with | For | May write |
|---|---|---|---|
| `nauthera-platform` | ClusterRoleBinding | the platform team | `Organization` |
| `nauthera-identity-admin` | RoleBinding in each namespace the identity team owns | security or IAM | `IdentityProvider`, `BrokerMembership`, and the namespace's Secrets |
| `nauthera-app-developer` | RoleBinding in each application namespace | application teams | `OIDCClient`, `Branding` |
| `nauthera-viewer` | nothing: folded into the built-in `view` role | anyone who may read the namespace | nothing |

```mermaid
flowchart TB
    P(["Platform team"]) -->|nauthera-platform| ORG["Organization"]
    S(["Identity team"]) -->|nauthera-identity-admin| IDP["IdentityProvider<br/>BrokerMembership<br/>Secrets"]
    D(["Application team"]) -->|nauthera-app-developer| APP["OIDCClient<br/>Branding"]
    E(["Everyone with view"]) -->|nauthera-viewer| RO["read Organization, OIDCClient,<br/>Branding, IdentityProvider,<br/>BrokerMembership"]
```

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: {name: identity-admins, namespace: acme-identity}
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: nauthera-identity-admin}
subjects: [{apiGroup: rbac.authorization.k8s.io, kind: Group, name: acme-security}]
```

- **Only the viewer is folded into a built-in role.** Folding the others into `edit`
  would let every namespace editor manage identity providers and memberships.
- **`LDAPDirectory` is in none of the team roles.** Cluster administrators manage it, or
  whoever you grant `ldapdirectories` with a Role of your own.
- **`nauthera-platform` is the most powerful.** It decides which namespaces configure which
  organization, so it belongs to the fewest people.
- **Secret references need no designation.** An `IdentityProvider` sends the Secret it
  names to the provider it names, so whoever may write one in a namespace can read any
  Secret in it. `nauthera-identity-admin` carries the namespace's Secrets for that reason,
  and no other role carries `IdentityProvider`. Keep provider credentials in a namespace
  of their own if other Secrets must stay out of the identity team's reach.
- **In a GitOps repository**, Argo CD or Flux applies every manifest under its own
  account. There the same split is made with protected branches and a code owner per
  kind's files. An Argo CD project allowed to apply `Organization` may apply any of them,
  so keep every `Organization` in a path and project the platform team owns. See
  [GitOps](https://www.nauthera.io/docs/v1/adoption/gitops).

## Metrics

Beside controller-runtime's own metrics, the operator publishes:

| Metric | Labels | What it says |
|---|---|---|
| `nauthera_operator_reconciles_total` | `kind`, `reason`, `cause` | Reconcile outcomes, by the reason left on the resource and, for a refusal, the server's cause. |
| `nauthera_operator_server_calls_total` | `call`, `code` | Every control-plane call, by call and gRPC code. |
| `nauthera_operator_server_call_duration_seconds` | `call` | Latency of those calls. |
| `nauthera_operator_resources` | `kind`, `phase` | Resources managed, by phase. |

The endpoint answers only callers the API server authorizes for `GET /metrics`: add
Prometheus's ServiceAccount to `operator.metrics.readers`.

## Resource references

- [OIDCClient](https://www.nauthera.io/docs/v1/oidc-client)
- [Organization](https://www.nauthera.io/docs/v1/organization-resource)
- [IdentityProvider](https://www.nauthera.io/docs/v1/identity-provider-resource)
- [LDAPDirectory](https://www.nauthera.io/docs/v1/ldap-directory-resource)
- [Branding](https://www.nauthera.io/docs/v1/branding-resource)
- [BrokerMembership](https://www.nauthera.io/docs/v1/broker-membership-resource)