# Rolling Out Nauthera Across a Company

> Rolling Nauthera out across a company: environments, a pilot, moving applications in waves, go/no-go criteria, watching each cutover and rolling an app back.

Identity is on the path of every sign-in, so roll Nauthera out the way you would a
database: in small steps, each reversible, each with someone watching. This page orders
the steps; [Migration](https://www.nauthera.io/docs/v1/adoption/migration) covers the mechanics of moving one
application.

An example timeline for a mid-sized company. The pilot and operational readiness run in
parallel, and the waves start only when both are done:

```mermaid
flowchart TD
  A["Decisions and owners · 2 weeks"] --> B["Staging install · 2 weeks"]
  B --> P1["Pilot in staging · 2 weeks"] --> P2["Pilot in production · 3 weeks"]
  B --> R["Operational readiness · 6 weeks"]
  P2 --> W1["Wave 1 · internal tools · 2 weeks"]
  R --> W1
  W1 --> W2["Wave 2 · workforce apps · 4 weeks"]
  W2 --> W3["Wave 3 · APIs and services · 3 weeks"]
  W3 --> W4["Wave 4 · customer-facing apps · 4 weeks"]
  W4 --> X["Retire the old provider · 3 weeks"]
```

## Environments

| Environment | Shape | Used for |
|---|---|---|
| Local | The [Quick Start](https://www.nauthera.io/docs/v1/quick-start) stack, or `kind-up` from the [installation guide](https://www.nauthera.io/docs/v1/installation#local-kind) | Application teams writing their `OIDCClient` and trying sign-in |
| Staging | The production Helm profile at a smaller size: same PostgreSQL and Redis topology, same certificates setup, same alerts | The pilot, every wave's rehearsal, every upgrade's rehearsal |
| Production | The production profile ([Operational Requirements](https://www.nauthera.io/docs/v1/adoption/operational-requirements)) | Real users |

Give each environment its own issuer URL, keys and database. Nothing should be shared
between staging and production, so a staging client can never sign in to production.
Keep the manifests the same across environments and change only the values, so what was
tested is what ships.

## 1. Pilot

Pick one application that is internal, has a forgiving user base and a team willing to
report problems, and move it in staging, then in production. The pilot is done when it
has shown:

- sign-in, sign-out and, if the app uses them, refresh and logout work end to end
  (front-channel logout for an `OIDCClient`, back-channel for a
  [dynamically registered](https://www.nauthera.io/docs/v1/sessions#telling-the-applications) client), through
  your upstream provider if you federate;
- the second-factor policy behaves as security expects, including `trustMfa`;
- the operator delivers the client's `Secret`, and a rotation (deleting the `Secret`)
  reaches the running application;
- alerts fire into the right channel, and someone has acknowledged a test page;
- a database backup has been restored into a scratch environment and served sign-ins;
- an upgrade to a newer chart version went through staging without downtime;
- support has handled at least one locked-out user and one lost second factor.

Write down what went wrong. Those notes become the runbook for the waves.

## 2. Waves

Group the remaining applications, smallest risk first:

1. **Internal tools** with few users and no external customers.
2. **Workforce applications** that everyone uses daily. These test the sign-in volume
   and the help desk.
3. **APIs and service-to-service clients** (`client_credentials`), which have no users but
   break silently. Move the API's token validation first, then its callers one at a time
   ([Migration](https://www.nauthera.io/docs/v1/adoption/migration#3-move-the-apis-first)).
4. **Customer-facing applications**, last, once the platform has run for weeks without
   an incident.

Keep a wave small enough that rolling all of it back would be a normal deploy. Don't
change Nauthera itself (an upgrade, a signing-key rotation, a policy change) during a
wave.

### Go / no-go

```mermaid
flowchart LR
  S["Rehearsed<br/>in staging"] --> G{"Go / no-go"}
  G -->|no-go| S
  G -->|go| C["Cut over"] --> W["Watch"]
  W -->|healthy| D["Next wave"]
  W -->|problems| RB["Roll back"] --> S
```

Before each wave, check:

- [ ] Every application in the wave has its `OIDCClient` `Ready` in production, and has
      signed in against it in staging.
- [ ] Every API it calls accepts tokens from both issuers.
- [ ] Its accounts exist in Nauthera, or the federated provider is offered for sign-up.
- [ ] Users were told what changes for them, and when.
- [ ] There are no firing alerts, and Nauthera's last upgrade was at least a few days ago.
- [ ] Each application's team is on hand for the cutover, and knows how to roll back.

## 3. Watching a cutover

For the first hours after an application moves, watch:

| Signal | Where |
|---|---|
| Sign-ins that do not complete, and why | `auth.sign_in.completed` [audit events](https://www.nauthera.io/docs/v1/audit-events) and their `cause`, filtered by the client |
| Federation failures | The same events with a `cause` of `upstream_denied`, `upstream_unavailable`, `upstream_token_invalid` or `upstream_email_missing`, where `upstream_error` holds the provider's own error code, and the `IdentityProvider`'s status |
| The application's daily users climbing to what the old provider showed | `GET /v1/admin/orgs/{slug}/clients/{id}/usage`, or the client in the [dashboard](https://www.nauthera.io/docs/v1/admin-dashboard) |
| Error rate and latency | The `errors` and `latency` [alert groups](https://www.nauthera.io/docs/v1/observability#alerting) |
| Lockouts and throttling, which spike when many users reset passwords at once | The `security` alert group and [Rate Limits & Lockout](https://www.nauthera.io/docs/v1/rate-limits) |
| Help desk tickets | Your ticketing system, tagged by wave |

A wave of invitations or password resets sends a burst of email. Check that the
organization's relay accepts the volume, and watch the [outbox](https://www.nauthera.io/docs/v1/email) drain.

## Talking to users

Users notice three things, so say them before the wave:

- the sign-in page looks different (brand it per client with a
  [`Branding`](https://www.nauthera.io/docs/v1/branding-resource) resource so it looks like the application);
- they sign in once more, and set a password or a second factor again unless they sign in
  through the company provider;
- where to go for help, and what an invitation or reset email will look like, so they
  don't take it for phishing.

## Rolling back

Rollback is per application, and it is the reverse of the cutover: deploy the
application with the old provider's issuer, client and secret again. That works for as
long as you keep:

- the old provider's client registered, which [Migration](https://www.nauthera.io/docs/v1/adoption/migration#5-cut-over-one-application)
  says to keep until its refresh tokens have expired;
- the APIs accepting both issuers;
- the application's user mapping able to read both `sub` values.

Users signed in through Nauthera sign in once more at the old provider. Accounts created
in Nauthera stay, so moving forward again later needs no new import. Leave the
`OIDCClient` in place during a rollback; its `deletionPolicy` decides what deleting it
would do.

## After the last wave

- Remove the old issuer from every API, and retire the old provider
  ([Migration](https://www.nauthera.io/docs/v1/adoption/migration#retiring-the-old-provider)).
- Narrow `operator.namespaces` and the gRPC allowlist to what is now in use.
- Hand on-call to the steady-state rota, and schedule the first restore drill and the
  first signing-key rotation.