GitOps & Argo CD
Docs version v1 (main)Nauthera is pre-1.0, in a design-partner programUpdated
Nauthera is built to be declared: the server is a Helm chart, and organizations, identity providers, clients and branding are Kubernetes resources the operator reconciles. That makes it a natural fit for GitOps, where Git is the only way changes reach the cluster and every change is a reviewed pull request. This page proposes a setup with Argo CD. The same split works with Flux; only the objects differ.
It builds on Ownership: the platform team owns the
server, organizations and identity providers; application teams own their OIDCClient
next to their code. Argo CD's AppProjects are where that split is enforced.
What goes where
| What | Lives in | Synced by |
|---|---|---|
The resource definitions (deploy/crds/*-crd.yaml, or the chart's files/crds/), pinned to the chart version | Platform repository | nauthera-crds |
Secrets the chart reads (database DSN, Redis password, KEK, MFA key, SMTP password, and for appendOnlyAudit the schema owner's DSN) as ExternalSecrets | Platform repository | nauthera-secrets |
| The server and operator: the Helm chart and its values per environment | Platform repository | nauthera-server |
Organization resources, whose spec.namespaces decides which team configures which organization | Platform repository | nauthera-tenants |
IdentityProvider resources and their client secrets as ExternalSecrets | Platform repository | nauthera-tenants |
BrokerMembership sides, each in its own organization's namespace, and the DNSEndpoints that prove emailDomains | Platform repository, or each organization's identity team | nauthera-tenants |
| Admission policies limiting fields within a kind (see Ownership) | Platform repository | nauthera-policies |
Each application's OIDCClient and Branding, beside its Deployment | The application's own repository | One Application per team and environment |
| Client secrets the operator mints | Nowhere in Git | The operator writes them into the application's namespace |
The client secrets are the point: with the operator, the only secrets in the GitOps flow are the platform's own, and those come from your secret store, never from Git.
Repository layout
platform-gitops/
├── bootstrap/
│ ├── root-staging.yaml # app of apps, one per cluster
│ └── root-production.yaml
├── projects/
│ ├── nauthera-platform.yaml # AppProject for everything below platform/
│ └── teams.yaml # AppProject per team (or an ApplicationSet of them)
├── platform/
│ └── nauthera/
│ ├── apps/ # the Application objects the root syncs
│ │ ├── crds.yaml
│ │ ├── secrets.yaml
│ │ ├── server.yaml
│ │ ├── tenants.yaml
│ │ ├── policies.yaml
│ │ └── teams.yaml # ApplicationSet for team Applications
│ ├── crds/
│ │ └── 0.x.y/ # deploy/crds/*-crd.yaml (or files/crds/) of this chart version
│ ├── secrets/
│ │ ├── base/ # ExternalSecrets, SecretStore reference
│ │ └── overlays/{staging,production}/
│ ├── server/
│ │ ├── values-common.yaml # production profile settings you share
│ │ ├── values-staging.yaml
│ │ └── values-production.yaml
│ ├── tenants/
│ │ ├── base/
│ │ │ ├── organizations/ # staff.yaml, acme.yaml (cluster-scoped)
│ │ │ ├── identity-providers/
│ │ │ │ ├── entra.yaml
│ │ │ │ └── entra-secret.yaml # ExternalSecret for the client secret
│ │ │ └── namespaces/ # team namespaces, admitted by spec.namespaces
│ │ └── overlays/{staging,production}/ # issuers differ per environment
│ └── policies/
│ └── oidcclient-organization.yaml
└── teams/
└── teams.yaml # team, repository, namespace, organization
billing-service/ # an application team's own repository
└── deploy/
├── base/
│ ├── deployment.yaml # reads the operator's Secret
│ ├── oidcclient.yaml
│ └── branding.yaml
└── overlays/{staging,production}/ # issuer and redirect URIs per environmentOne root Application per cluster keeps environments separate: staging and production have different issuers, keys and databases (Rollout), so nothing should let a staging change reach production except a pull request to the production overlay.
Sync order
Each layer needs the one before it. Order them with sync waves on the Application objects the root syncs:
| Wave | Application | Needs |
|---|---|---|
-2 | nauthera-crds | Nothing |
-1 | nauthera-secrets | The External Secrets operator and a SecretStore |
0 | nauthera-server | Secrets, cert-manager, PostgreSQL, Redis |
1 | nauthera-tenants, nauthera-policies | The CRDs, a Ready operator whose allowedClients entry grants organizations, broker_memberships and email_domains in the tenants' slugs |
2 | The team Applications | Their organization, the policy |
Argo CD waits for an Application to be healthy before the next wave only if it can assess
the health of Application resources, which it does not do by default since 1.8. Add
the health check from the
Argo CD docs
to argocd-cm, or the waves only order the creation of the Applications.
The platform AppProject
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: nauthera-platform
namespace: argocd
spec:
description: The Nauthera server, operator, tenants and policies
sourceRepos:
- https://github.com/example/platform-gitops.git
- ghcr.io/nauthera/charts # the OCI chart
destinations:
- server: https://kubernetes.default.svc
namespace: nauthera
- server: https://kubernetes.default.svc
namespace: "*" # namespaces with team labels
clusterResourceWhitelist:
- { group: apiextensions.k8s.io, kind: CustomResourceDefinition }
- { group: nauthera.io, kind: Organization }
- { group: admissionregistration.k8s.io, kind: ValidatingAdmissionPolicy }
- { group: admissionregistration.k8s.io, kind: ValidatingAdmissionPolicyBinding }
- { group: rbac.authorization.k8s.io, kind: ClusterRole }
- { group: rbac.authorization.k8s.io, kind: ClusterRoleBinding }
- { group: "", kind: Namespace }
roles:
- name: platform
groups: [alice@example.com, bob@example.com]
policies:
- p, proj:nauthera-platform:platform, applications, *, nauthera-platform/*, allowArgo CD matches a role's groups against the claims that scopes in argocd-rbac-cm
names, groups by default. Nauthera's tokens carry no groups claim, so when Argo CD signs
in through Nauthera, list the team's email addresses and set scopes: "[email]"
(Argo CD sign-in has the client and both
ConfigMaps). With another provider in front of Argo CD, its group names work as usual.
Narrow destinations to the namespaces you actually use once they are known.
The server Application
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: nauthera-server
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "0"
spec:
project: nauthera-platform
destination:
server: https://kubernetes.default.svc
namespace: nauthera
sources:
- repoURL: ghcr.io/nauthera/charts
chart: nauthera-server
targetRevision: 0.x.y # the only line an upgrade changes
helm:
releaseName: nauthera
valueFiles:
- $values/platform/nauthera/server/values-common.yaml
- $values/platform/nauthera/server/values-production.yaml
values: |
operator:
enabled: true
crds:
install: false # nauthera-crds owns them
certManager:
grpcTLS:
enabled: true # the operator calls the AdminService over mTLS
allowedClients:
- identity: dns:nauthera-operator
operations: [clients, brandings, identity_providers, ldap_directories,
organizations, broker_memberships, email_domains]
organizations: [default, staff, acme] # every tenant's slug
namespaces: ["*"]
config:
server:
grpc:
enabled: true
- repoURL: https://github.com/example/platform-gitops.git
targetRevision: main
ref: values
syncPolicy:
# No automated sync in production: migrations are forward only, so an upgrade is
# synced by a person, after a backup. Staging can use automated: { prune: true, selfHeal: true }.
syncOptions:
- CreateNamespace=falseThe chart and both images are public, so Argo CD needs no credentials and the cluster no image pull secret. Declare the registry once as an OCI Helm repository:
apiVersion: v1
kind: Secret
metadata:
name: nauthera-charts
namespace: argocd
labels:
argocd.argoproj.io/secret-type: repository
stringData:
type: helm
name: nauthera
url: ghcr.io/nauthera/charts
enableOCI: "true"Downloads lists every published version. The released chart pins both images by digest, so an upgrade is one change to
targetRevision. Let Renovate or Dependabot open that pull request.
Put values-common.yaml together from the production profile
(profiles/production.yaml) and your own settings. Argo CD reads value files from Git,
not from the chart, so copy the profile's content in rather than referencing it, and
review the profile's diff when you bump the chart.
Helm hooks under Argo CD
Argo CD renders the chart with helm template and applies the result; it does not run
helm install. Two things change:
- The migration Job still runs first. Argo CD maps the chart's
pre-installandpre-upgradehooks toPreSync, so withappendOnlyAuditthemaintainJob runs before the new pods, as with Helm. It runs on every sync, not only on upgrades;maintainis idempotent, so that is safe. - There is no
helm --timeout. A release whose migration builds an audit index on a large log is bounded by the Job's ownappendOnlyAudit.alone. Raise it in the values for that release (Upgrading).maintenance. activeDeadlineSeconds
The resource definitions
The chart renders the CRDs as templates by default (operator.crds.install), and Helm
keeps them on uninstall. Under Argo CD they are ordinary resources of the Application,
and whether a prune or a deleted Application spares them depends on your Argo CD version
and settings, not on Helm. Deleting a CRD deletes every OIDCClient with it, so don't
leave that to chance: sync them from their own Application with pruning off, and turn
the chart's copy off as above:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: nauthera-crds
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "-2"
spec:
project: nauthera-platform
destination: { server: https://kubernetes.default.svc }
source:
repoURL: https://github.com/example/platform-gitops.git
targetRevision: main
path: platform/nauthera/crds/0.x.y
syncPolicy:
automated: { prune: false, selfHeal: true }
syncOptions:
- ServerSideApply=true # large schemas exceed the client-side annotation limitBump the directory together with the chart's targetRevision, in the same pull request.
Secrets
The chart takes each of its secrets.*, ownerDatabaseDsn and auditCheckpointKey
included, by existingSecret. Create those Secrets with the External Secrets operator
from your secret store (or Sealed Secrets, or the Argo CD
Vault plugin), never as plain manifests in Git:
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: nauthera-keys
namespace: nauthera
annotations:
argocd.argoproj.io/sync-wave: "-1"
spec:
secretStoreRef: { kind: ClusterSecretStore, name: vault }
target:
name: nauthera-keys
deletionPolicy: Retain # losing the KEK makes tenant keys unreadable
data:
- secretKey: keys-encryption-key
remoteRef: { key: nauthera/production, property: kek }
- secretKey: mfa-encryption-key
remoteRef: { key: nauthera/production, property: mfa-key }Use the key names your chart version expects. cert-manager issues the token signing keys
(certManager.signingKey) and, unless secrets. gives one, the audit
checkpoint key (certManager.). To bring your own signing key, turn
certManager. off, mount the PEM with extraVolumes and
extraVolumeMounts, and set config. and config.
to it.
An IdentityProvider's client secret is an ExternalSecret beside it, in the same
directory: the operator sends the new secret whenever the Secret changes, so rotating it at
the upstream provider and in the store is the whole procedure. Remember the operator
refuses a value with a trailing newline (SecretPadded), which some stores add.
The KEK, the MFA seed key and the signing keys must also exist outside the cluster, in the recovery copy Operational Requirements describes. A secret store is that copy only if it is backed up separately from the cluster.
Tenants and identity providers
nauthera-tenants holds the Organization and IdentityProvider resources and the team
namespaces, which each Organization admits in spec.namespaces. A sync must never be what removes
a tenant, so protect them from pruning and from cascade deletion:
apiVersion: nauthera.io/v1alpha1
kind: Organization
metadata:
name: acme
annotations:
argocd.argoproj.io/sync-options: Prune=false,Delete=false
spec:
slug: acme
displayName: Acme Inc.
deletionPolicy: RetaindeletionPolicy: Retain keeps the tenant served even if the resource goes; the
annotations keep Argo CD from deleting the resource in the first place, on a pruned file
or a deleted Application. Do the same for IdentityProviders. Removing a tenant becomes a
deliberate step: delete the resource by hand after the pull request that removed its file.
Team applications
Each team gets an AppProject that allows its own repository, its own namespace, and only the kinds it owns:
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: team-billing
namespace: argocd
spec:
sourceRepos: [https://github.com/example/billing-service.git]
destinations:
- server: https://kubernetes.default.svc
namespace: billing
clusterResourceWhitelist: [] # nothing cluster-scoped
namespaceResourceWhitelist:
- { group: apps, kind: Deployment }
- { group: "", kind: Service }
- { group: "", kind: ConfigMap }
- { group: networking.k8s.io, kind: Ingress }
- { group: nauthera.io, kind: OIDCClient }
- { group: nauthera.io, kind: Branding }
roles:
- name: developers
groups: [team-billing]
policies:
- p, proj:team-billing:developers, applications, *, team-billing/*, allowIdentityProvider and Organization are not on the list, so a team's repository cannot
declare them however it is written. The team's namespace configures only the organization
whose spec.namespaces lists it, which the platform repository owns, and RBAC on the
namespace stays as in
Ownership. An ApplicationSet with a list
or Git generator over teams/teams.yaml creates one AppProject and one Application per
team and environment, so onboarding a team is a pull request adding one entry.
Things to know about the team side:
- The operator's Secret is not in Git, and that is right. It carries an owner
reference to the
OIDCClientbut not Argo CD's tracking label or annotation, so Argo CD shows it under the resource and never prunes it. Don't add aSecretwith the same name to the repository; the operator does not write over a Secret it does not own (SecretUndeliverable). - Rotating a client secret stays a delete of that Secret. Argo CD does not recreate it; the operator does.
- Deleting the Application deletes the
OIDCClientwith the rest. ItsdeletionPolicydecides what happens on the server,Retainby default. Leave it there for production clients. - Drift shows up twice: Argo CD reports a manifest changed by hand in the cluster, and
the operator reports a client changed in the server behind its back (
Drifted). Self-heal fixes the first;nauthera.io/restorefixes the second (OIDCClient).
Health checks for the operator's resources
Argo CD has no health check of its own for the nauthera.io kinds, so it gives an
OIDCClient no health status and leaves it out of the Application's health, even when the
server refused it (health code,
Argo CD 3.6.0). Teach it the operator's status:
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-cm
namespace: argocd
data:
resource.customizations.health.nauthera.io_OIDCClient: &nautheraHealth |
hs = { status = "Progressing", message = "Waiting for the operator" }
if obj.status == nil then return hs end
if obj.status.observedGeneration ~= nil and obj.metadata.generation ~= nil
and obj.status.observedGeneration < obj.metadata.generation then
return hs
end
local msg = ""
if obj.status.conditions ~= nil then
for _, c in ipairs(obj.status.conditions) do
if c.type == "Ready" then msg = (c.reason or "") .. ": " .. (c.message or "") end
end
end
if obj.status.phase == "Ready" then
hs.status = "Healthy"
elseif obj.status.phase == "Error" then
hs.status = "Degraded"
end
hs.message = msg
return hs
resource.customizations.health.nauthera.io_IdentityProvider: *nautheraHealth
resource.customizations.health.nauthera.io_Organization: *nautheraHealth
resource.customizations.health.nauthera.io_Branding: *nautheraHealth
resource.customizations.health.nauthera.io_LDAPDirectory: *nautheraHealth
resource.customizations.health.nauthera.io_BrokerMembership: *nautheraHealthNow a Refused client turns its Application Degraded, with the server's objection as
the message, and the team sees it where they deploy. Pending reasons such as
Unreachable stay Progressing. Point Argo CD notifications for Degraded at the owning
team's channel.
Promotion and upgrades
- Changes to tenants and clients go through staging first: merge to the staging
overlay, check the resource is
Readythere, then open the production pull request. Overlays differ only in issuers, hosts and redirect URIs. - Upgrading Nauthera is a pull request that bumps the chart's
targetRevisionand the CRD directory together, first for staging. In production, take the backup, then sync by hand, inside a sync window that matches your change calendar. A deny window during rollout waves keeps platform changes out of them (Rollout). - Rolling back the server is not a revert. Migrations are forward only, so reverting
targetRevisionafter a migration ran starts an older binary on a newer schema. Roll forward, or restore the backup (Upgrading). Reverting a tenant or client manifest is fine. - Check a sync is quiet. After the first sync, a second one with no Git change should
show nothing to apply. A resource that always differs, such as a field the API server
defaults, needs an
ignoreDifferencesentry, not self-heal fighting it.
Checklist
- One root Application per cluster; staging and production never share an overlay.
- CRDs in their own Application, with pruning off and server-side apply.
- No Secret in Git; the KEK and MFA key have a recovery copy outside the cluster.
-
OrganizationandIdentityProvidercarryPrune=false,Delete=false. - Team AppProjects allow
OIDCClientandBrandingonly, in their own namespace. - Health checks for the six
nauthera.iokinds, and forApplicationif you use waves. - Production server syncs by hand, after a backup, inside a sync window.
- Notifications route
Degradedresources to their owners.