# GitOps & Argo CD

> A suggested GitOps setup for Nauthera with Argo CD: repository layout, AppProjects, sync waves, External Secrets, health checks and environment promotion.

Nauthera is built to be declared: the server is a Helm chart, and organizations, identity
providers, clients and branding are Kubernetes resources the
[operator](https://www.nauthera.io/docs/v1/operator) reconciles. That makes it a natural fit for GitOps, where
Git is the only way changes reach the cluster and every change is a reviewed pull request.
This page proposes a setup with [Argo CD](https://argo-cd.readthedocs.io/). The same
split works with Flux; only the objects differ.

It builds on [Ownership](https://www.nauthera.io/docs/v1/adoption/ownership): the platform team owns the
server, organizations and identity providers; application teams own their `OIDCClient`
next to their code. Argo CD's AppProjects are where that split is enforced.

## What goes where

```mermaid
flowchart TD
  PG[("platform-gitops")] --> A["Argo CD"]
  TG[("Team repositories")] --> A
  V[("Secret store")] --> ES["External Secrets"]
  A -->|wave -2| CRD["CRDs"]
  A -->|wave -1| ES
  A -->|wave 0| H["Helm chart:<br/>server and operator"]
  A -->|wave 1| T["Organizations,<br/>identity providers"]
  A -->|wave 2| C["OIDCClient,<br/>Branding"]
  T --> OP["Operator"]
  C --> OP
  OP -->|"gRPC, mTLS"| SV["nauthera-server"]
  OP -->|"minted secret"| KS["Secret in the<br/>app's namespace"]
```

| What | Lives in | Synced by |
|---|---|---|
| The resource definitions (`deploy/crds/*-crd.yaml`, or the chart's `files/crds/`), pinned to the chart version | Platform repository | `nauthera-crds` |
| Secrets the chart reads (database DSN, Redis password, KEK, MFA key, SMTP password, and for `appendOnlyAudit` the schema owner's DSN) as `ExternalSecret`s | Platform repository | `nauthera-secrets` |
| The server and operator: the Helm chart and its values per environment | Platform repository | `nauthera-server` |
| `Organization` resources, whose `spec.namespaces` decides which team configures which organization | Platform repository | `nauthera-tenants` |
| `IdentityProvider` resources and their client secrets as `ExternalSecret`s | Platform repository | `nauthera-tenants` |
| `BrokerMembership` sides, each in its own organization's namespace, and the `DNSEndpoint`s that prove `emailDomains` | Platform repository, or each organization's identity team | `nauthera-tenants` |
| Admission policies limiting fields within a kind (see [Ownership](https://www.nauthera.io/docs/v1/adoption/ownership#4-the-namespaces-an-organization-admits)) | Platform repository | `nauthera-policies` |
| Each application's `OIDCClient` and `Branding`, beside its Deployment | The application's own repository | One Application per team and environment |
| Client secrets the operator mints | Nowhere in Git | The operator writes them into the application's namespace |

The client secrets are the point: with the operator, the only secrets in the GitOps flow
are the platform's own, and those come from your secret store, never from Git.

## Repository layout

```text
platform-gitops/
├── bootstrap/
│   ├── root-staging.yaml            # app of apps, one per cluster
│   └── root-production.yaml
├── projects/
│   ├── nauthera-platform.yaml       # AppProject for everything below platform/
│   └── teams.yaml                   # AppProject per team (or an ApplicationSet of them)
├── platform/
│   └── nauthera/
│       ├── apps/                    # the Application objects the root syncs
│       │   ├── crds.yaml
│       │   ├── secrets.yaml
│       │   ├── server.yaml
│       │   ├── tenants.yaml
│       │   ├── policies.yaml
│       │   └── teams.yaml           # ApplicationSet for team Applications
│       ├── crds/
│       │   └── 0.x.y/          # deploy/crds/*-crd.yaml (or files/crds/) of this chart version
│       ├── secrets/
│       │   ├── base/                # ExternalSecrets, SecretStore reference
│       │   └── overlays/{staging,production}/
│       ├── server/
│       │   ├── values-common.yaml   # production profile settings you share
│       │   ├── values-staging.yaml
│       │   └── values-production.yaml
│       ├── tenants/
│       │   ├── base/
│       │   │   ├── organizations/   # staff.yaml, acme.yaml (cluster-scoped)
│       │   │   ├── identity-providers/
│       │   │   │   ├── entra.yaml
│       │   │   │   └── entra-secret.yaml   # ExternalSecret for the client secret
│       │   │   └── namespaces/      # team namespaces, admitted by spec.namespaces
│       │   └── overlays/{staging,production}/   # issuers differ per environment
│       └── policies/
│           └── oidcclient-organization.yaml
└── teams/
    └── teams.yaml                   # team, repository, namespace, organization

billing-service/                     # an application team's own repository
└── deploy/
    ├── base/
    │   ├── deployment.yaml          # reads the operator's Secret
    │   ├── oidcclient.yaml
    │   └── branding.yaml
    └── overlays/{staging,production}/   # issuer and redirect URIs per environment
```

One root Application per cluster keeps environments separate: staging and production
have different issuers, keys and databases ([Rollout](https://www.nauthera.io/docs/v1/adoption/rollout#environments)),
so nothing should let a staging change reach production except a pull request to the
production overlay.

## Sync order

Each layer needs the one before it. Order them with sync waves on the Application objects
the root syncs:

| Wave | Application | Needs |
|---|---|---|
| `-2` | `nauthera-crds` | Nothing |
| `-1` | `nauthera-secrets` | The External Secrets operator and a `SecretStore` |
| `0` | `nauthera-server` | Secrets, cert-manager, PostgreSQL, Redis |
| `1` | `nauthera-tenants`, `nauthera-policies` | The CRDs, a `Ready` operator whose `allowedClients` entry grants `organizations`, `broker_memberships` and `email_domains` in the tenants' slugs |
| `2` | The team Applications | Their organization, the policy |

Argo CD waits for an Application to be healthy before the next wave only if it can assess
the health of `Application` resources, which it does not do by default since 1.8. Add
the health check from the
[Argo CD docs](https://argo-cd.readthedocs.io/en/stable/operator-manual/health/#argocd-app)
to `argocd-cm`, or the waves only order the creation of the Applications.

## The platform AppProject

```yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: nauthera-platform
  namespace: argocd
spec:
  description: The Nauthera server, operator, tenants and policies
  sourceRepos:
    - https://github.com/example/platform-gitops.git
    - ghcr.io/nauthera/charts                 # the OCI chart
  destinations:
    - server: https://kubernetes.default.svc
      namespace: nauthera
    - server: https://kubernetes.default.svc
      namespace: "*"                          # namespaces with team labels
  clusterResourceWhitelist:
    - { group: apiextensions.k8s.io, kind: CustomResourceDefinition }
    - { group: nauthera.io, kind: Organization }
    - { group: admissionregistration.k8s.io, kind: ValidatingAdmissionPolicy }
    - { group: admissionregistration.k8s.io, kind: ValidatingAdmissionPolicyBinding }
    - { group: rbac.authorization.k8s.io, kind: ClusterRole }
    - { group: rbac.authorization.k8s.io, kind: ClusterRoleBinding }
    - { group: "", kind: Namespace }
  roles:
    - name: platform
      groups: [alice@example.com, bob@example.com]
      policies:
        - p, proj:nauthera-platform:platform, applications, *, nauthera-platform/*, allow
```

Argo CD matches a role's `groups` against the claims that `scopes` in `argocd-rbac-cm`
names, `groups` by default. Nauthera's tokens carry no `groups` claim, so when Argo CD signs
in through Nauthera, list the team's email addresses and set `scopes: "[email]"`
([Argo CD sign-in](https://www.nauthera.io/integrations/argocd#4-roles-argocd-rbac-cm) has the client and both
ConfigMaps). With another provider in front of Argo CD, its group names work as usual.

Narrow `destinations` to the namespaces you actually use once they are known.

## The server Application

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: nauthera-server
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "0"
spec:
  project: nauthera-platform
  destination:
    server: https://kubernetes.default.svc
    namespace: nauthera
  sources:
    - repoURL: ghcr.io/nauthera/charts
      chart: nauthera-server
      targetRevision: 0.x.y            # the only line an upgrade changes
      helm:
        releaseName: nauthera
        valueFiles:
          - $values/platform/nauthera/server/values-common.yaml
          - $values/platform/nauthera/server/values-production.yaml
        values: |
          operator:
            enabled: true
            crds:
              install: false                # nauthera-crds owns them
          certManager:
            grpcTLS:
              enabled: true                 # the operator calls the AdminService over mTLS
              allowedClients:
                - identity: dns:nauthera-operator
                  operations: [clients, brandings, identity_providers, ldap_directories,
                               organizations, broker_memberships, email_domains]
                  organizations: [default, staff, acme]   # every tenant's slug
                  namespaces: ["*"]
          config:
            server:
              grpc:
                enabled: true
    - repoURL: https://github.com/example/platform-gitops.git
      targetRevision: main
      ref: values
  syncPolicy:
    # No automated sync in production: migrations are forward only, so an upgrade is
    # synced by a person, after a backup. Staging can use automated: { prune: true, selfHeal: true }.
    syncOptions:
      - CreateNamespace=false
```

The chart and both images are public, so Argo CD needs no credentials and the cluster
no image pull secret. Declare the registry once as an OCI Helm repository:

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: nauthera-charts
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  type: helm
  name: nauthera
  url: ghcr.io/nauthera/charts
  enableOCI: "true"
```

[Downloads](https://www.nauthera.io/downloads) lists every published version. The released chart pins both images by digest, so an upgrade is one change to
`targetRevision`. Let Renovate or Dependabot open that pull request.

Put `values-common.yaml` together from the production profile
(`profiles/production.yaml`) and your own settings. Argo CD reads value files from Git,
not from the chart, so copy the profile's content in rather than referencing it, and
review the profile's diff when you bump the chart.

### Helm hooks under Argo CD

Argo CD renders the chart with `helm template` and applies the result; it does not run
`helm install`. Two things change:

- **The migration Job still runs first.** Argo CD maps the chart's `pre-install` and
  `pre-upgrade` hooks to `PreSync`, so with `appendOnlyAudit` the `maintain` Job runs
  before the new pods, as with Helm. It runs on **every** sync, not only on upgrades;
  `maintain` is idempotent, so that is safe.
- **There is no `helm --timeout`.** A release whose migration builds an audit index on a
  large log is bounded by the Job's own
  `appendOnlyAudit.maintenance.activeDeadlineSeconds` alone. Raise it in the values for
  that release ([Upgrading](https://www.nauthera.io/docs/v1/operations#upgrading)).

### The resource definitions

The chart renders the CRDs as templates by default (`operator.crds.install`), and Helm
keeps them on uninstall. Under Argo CD they are ordinary resources of the Application,
and whether a prune or a deleted Application spares them depends on your Argo CD version
and settings, not on Helm. Deleting a CRD deletes every `OIDCClient` with it, so don't
leave that to chance: sync them from their own Application with pruning off, and turn
the chart's copy off as above:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: nauthera-crds
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "-2"
spec:
  project: nauthera-platform
  destination: { server: https://kubernetes.default.svc }
  source:
    repoURL: https://github.com/example/platform-gitops.git
    targetRevision: main
    path: platform/nauthera/crds/0.x.y
  syncPolicy:
    automated: { prune: false, selfHeal: true }
    syncOptions:
      - ServerSideApply=true      # large schemas exceed the client-side annotation limit
```

Bump the directory together with the chart's `targetRevision`, in the same pull request.

## Secrets

The chart takes each of its `secrets.*`, `ownerDatabaseDsn` and `auditCheckpointKey`
included, by `existingSecret`. Create those Secrets with the External Secrets operator
from your secret store (or [Sealed Secrets](https://www.nauthera.io/docs/v1/guides/sealed-secrets), or the Argo CD
Vault plugin), never as plain manifests in Git:

```yaml
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: nauthera-keys
  namespace: nauthera
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
spec:
  secretStoreRef: { kind: ClusterSecretStore, name: vault }
  target:
    name: nauthera-keys
    deletionPolicy: Retain        # losing the KEK makes tenant keys unreadable
  data:
    - secretKey: keys-encryption-key
      remoteRef: { key: nauthera/production, property: kek }
    - secretKey: mfa-encryption-key
      remoteRef: { key: nauthera/production, property: mfa-key }
```

Use the key names your chart version expects. cert-manager issues the token signing keys
(`certManager.signingKey`) and, unless `secrets.auditCheckpointKey` gives one, the audit
checkpoint key (`certManager.auditCheckpointKey`). To bring your own signing key, turn
`certManager.signingKey.enabled` off, mount the PEM with `extraVolumes` and
`extraVolumeMounts`, and set `config.keys.provider: file` and `config.keys.signing_key_path`
to it.

An `IdentityProvider`'s client secret is an `ExternalSecret` beside it, in the same
directory: the operator sends the new secret whenever the Secret changes, so rotating it at
the upstream provider and in the store is the whole procedure. Remember the operator
refuses a value with a trailing newline (`SecretPadded`), which some stores add.

The KEK, the MFA seed key and the signing keys must also exist outside the cluster, in
the recovery copy [Operational Requirements](https://www.nauthera.io/docs/v1/adoption/operational-requirements#secrets-and-their-recovery)
describes. A secret store is that copy only if it is backed up separately from the
cluster.

## Tenants and identity providers

`nauthera-tenants` holds the `Organization` and `IdentityProvider` resources and the team
namespaces, which each `Organization` admits in `spec.namespaces`. A sync must never be what removes
a tenant, so protect them from pruning and from cascade deletion:

```yaml
apiVersion: nauthera.io/v1alpha1
kind: Organization
metadata:
  name: acme
  annotations:
    argocd.argoproj.io/sync-options: Prune=false,Delete=false
spec:
  slug: acme
  displayName: Acme Inc.
  deletionPolicy: Retain
```

`deletionPolicy: Retain` keeps the tenant served even if the resource goes; the
annotations keep Argo CD from deleting the resource in the first place, on a pruned file
or a deleted Application. Do the same for `IdentityProvider`s. Removing a tenant becomes a
deliberate step: delete the resource by hand after the pull request that removed its file.

## Team applications

Each team gets an AppProject that allows its own repository, its own namespace, and only
the kinds it owns:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: team-billing
  namespace: argocd
spec:
  sourceRepos: [https://github.com/example/billing-service.git]
  destinations:
    - server: https://kubernetes.default.svc
      namespace: billing
  clusterResourceWhitelist: []          # nothing cluster-scoped
  namespaceResourceWhitelist:
    - { group: apps, kind: Deployment }
    - { group: "", kind: Service }
    - { group: "", kind: ConfigMap }
    - { group: networking.k8s.io, kind: Ingress }
    - { group: nauthera.io, kind: OIDCClient }
    - { group: nauthera.io, kind: Branding }
  roles:
    - name: developers
      groups: [team-billing]
      policies:
        - p, proj:team-billing:developers, applications, *, team-billing/*, allow
```

`IdentityProvider` and `Organization` are not on the list, so a team's repository cannot
declare them however it is written. The team's namespace configures only the organization
whose `spec.namespaces` lists it, which the platform repository owns, and RBAC on the
namespace stays as in
[Ownership](https://www.nauthera.io/docs/v1/adoption/ownership#3-kubernetes-rbac). An ApplicationSet with a list
or Git generator over `teams/teams.yaml` creates one AppProject and one Application per
team and environment, so onboarding a team is a pull request adding one entry.

Things to know about the team side:

- **The operator's Secret is not in Git, and that is right.** It carries an owner
  reference to the `OIDCClient` but not Argo CD's tracking label or annotation, so Argo CD
  shows it under the resource and never prunes it. Don't add a `Secret` with the same name to the
  repository; the operator does not write over a Secret it does not own
  (`SecretUndeliverable`).
- **Rotating a client secret** stays a delete of that Secret. Argo CD does not recreate
  it; the operator does.
- **Deleting the Application** deletes the `OIDCClient` with the rest. Its
  `deletionPolicy` decides what happens on the server, `Retain` by default. Leave it
  there for production clients.
- **Drift** shows up twice: Argo CD reports a manifest changed by hand in the cluster, and
  the operator reports a client changed in the server behind its back (`Drifted`).
  Self-heal fixes the first; `nauthera.io/restore` fixes the second
  ([OIDCClient](https://www.nauthera.io/docs/v1/oidc-client#status)).

## Health checks for the operator's resources

Argo CD has no health check of its own for the `nauthera.io` kinds, so it gives an
`OIDCClient` no health status and leaves it out of the Application's health, even when the
server refused it ([health code](https://github.com/argoproj/argo-cd/blob/v3.6.0/controller/health.go),
Argo CD 3.6.0). Teach it the operator's `status`:

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  resource.customizations.health.nauthera.io_OIDCClient: &nautheraHealth |
    hs = { status = "Progressing", message = "Waiting for the operator" }
    if obj.status == nil then return hs end
    if obj.status.observedGeneration ~= nil and obj.metadata.generation ~= nil
       and obj.status.observedGeneration < obj.metadata.generation then
      return hs
    end
    local msg = ""
    if obj.status.conditions ~= nil then
      for _, c in ipairs(obj.status.conditions) do
        if c.type == "Ready" then msg = (c.reason or "") .. ": " .. (c.message or "") end
      end
    end
    if obj.status.phase == "Ready" then
      hs.status = "Healthy"
    elseif obj.status.phase == "Error" then
      hs.status = "Degraded"
    end
    hs.message = msg
    return hs
  resource.customizations.health.nauthera.io_IdentityProvider: *nautheraHealth
  resource.customizations.health.nauthera.io_Organization: *nautheraHealth
  resource.customizations.health.nauthera.io_Branding: *nautheraHealth
  resource.customizations.health.nauthera.io_LDAPDirectory: *nautheraHealth
  resource.customizations.health.nauthera.io_BrokerMembership: *nautheraHealth
```

Now a `Refused` client turns its Application `Degraded`, with the server's objection as
the message, and the team sees it where they deploy. `Pending` reasons such as
`Unreachable` stay `Progressing`. Point Argo CD notifications for `Degraded` at the owning
team's channel.

## Promotion and upgrades


```mermaid
flowchart LR
  PR["Pull request"] --> ST["Staging overlay<br/>auto-sync"]
  ST -->|"Ready, tested"| PP["Production<br/>pull request"]
  PP --> BK["Backup"] --> SY["Manual sync<br/>in a sync window"]
```
- **Changes to tenants and clients** go through staging first: merge to the staging
  overlay, check the resource is `Ready` there, then open the production pull request.
  Overlays differ only in issuers, hosts and redirect URIs.
- **Upgrading Nauthera** is a pull request that bumps the chart's `targetRevision` and the
  CRD directory together, first for staging. In production, take the backup, then sync by
  hand, inside a [sync window](https://argo-cd.readthedocs.io/en/stable/user-guide/sync_windows/)
  that matches your change calendar. A deny window during rollout waves keeps platform
  changes out of them ([Rollout](https://www.nauthera.io/docs/v1/adoption/rollout#2-waves)).
- **Rolling back the server** is not a revert. Migrations are forward only, so reverting
  `targetRevision` after a migration ran starts an older binary on a newer schema. Roll
  forward, or restore the backup ([Upgrading](https://www.nauthera.io/docs/v1/operations#upgrading)). Reverting
  a tenant or client manifest is fine.
- **Check a sync is quiet.** After the first sync, a second one with no Git change should
  show nothing to apply. A resource that always differs, such as a field the API server
  defaults, needs an `ignoreDifferences` entry, not self-heal fighting it.

## Checklist

- [ ] One root Application per cluster; staging and production never share an overlay.
- [ ] CRDs in their own Application, with pruning off and server-side apply.
- [ ] No Secret in Git; the KEK and MFA key have a recovery copy outside the cluster.
- [ ] `Organization` and `IdentityProvider` carry `Prune=false,Delete=false`.
- [ ] Team AppProjects allow `OIDCClient` and `Branding` only, in their own namespace.
- [ ] Health checks for the six `nauthera.io` kinds, and for `Application` if you use waves.
- [ ] Production server syncs by hand, after a backup, inside a sync window.
- [ ] Notifications route `Degraded` resources to their owners.