Authentik deployment

This repository owns the production deployment of Authentik for sso.s1q.dev, including its Compose configuration, encrypted secrets, and Komodo Stack declaration.

Deployment architecture

The authentik-prod Stack follows the protected main branch. Its direct webhook is disabled. A Forgejo webhook calls the central authentik-deploy Procedure, which first runs the child Resource Sync and then runs DeployStackIfChanged. The central docker-compose/komodo-bootstrap repository owns that child Sync and Procedure.

Komodo passes docker-compose.yml first and docker-compose.override.yml second. The upstream-derived base and deployment-specific override remain separate by design. A regular deployment therefore follows this path:

1. merge a reviewed change into main;
2. Forgejo calls the authentik-deploy Procedure webhook;
3. the Procedure syncs komodo.toml; and
4. Komodo deploys authentik-prod if a tracked file changed.

For a first deployment or a manual retry, run authentik-deploy in Komodo instead of deploying the Stack directly. This keeps resource synchronization and deployment in the intended order.

Prerequisites

Before the first deployment, ensure that:

• NixOS has provisioned the shel1svc00 Komodo Server, Periphery's container permissions, the with-age-env wrapper, and its age-identity credential;
• PostgreSQL provisioning has created the authentik database on shel1svc00, and its database password is in secret.env.age;
• the root bootstrap Resource Sync has registered the authentik child Sync and the authentik-deploy Procedure; and
• the Forgejo repository webhook uses the Procedure's __ANY__ webhook URL.

Host provisioning, Tailscale, CrowdSec, and registry setup belong to NixOS or their respective infrastructure repositories and are intentionally not duplicated here.

Layout and ownership

.
├── .age-docker.toml             # Secret recipients and policy
├── docker-compose.yml           # Upstream-derived Compose base
├── docker-compose.override.yml  # Host database, secrets, and volumes
├── .env                         # Tracked non-secret configuration and version
├── komodo.toml                  # authentik-prod Stack declaration
├── secret.env.age               # Encrypted production secrets
└── data/authentik/custom-templates/

NixOS owns Komodo, Periphery, host credentials, Podman permissions, PostgreSQL, and Traefik. This repository owns the application definition. PostgreSQL lives in the host's provisioned cluster. Redis, media, certificate, and GeoIP data live in named container volumes and are not stored in Git.

Traefik terminates client TLS and forwards requests to Authentik's HTTP listener on port 9000 over the project's frontend network. Attach only trusted containers to that network; the server has no published host ports. Routing to HTTPS port 9443 without a trusted upstream certificate and matching server name makes Traefik return HTTP 500 during certificate verification.

To check a target through Traefik before or after deployment, preserving the public hostname and TLS verification, run:

bash scripts/check-traefik.sh shel1svc00.s1q.dev

The check requires the root request to redirect with HTTP 302 and the login page reached by following redirects to return HTTP 200.

.env is the tracked source for non-secret runtime settings and the Authentik image version. The age bundle supplies secret values during Komodo commands.

Secrets

The encrypted production bundle contains:

PG_PASS
AUTHENTIK_SECRET_KEY
AUTHENTIK_EMAIL__PASSWORD
GEOIPUPDATE_LICENSE_KEY

Local secret management uses only age-docker. Recipient keys, groups, and the rule for the encrypted bundle live in .age-docker.toml. The bundle is currently encrypted for the personal phg age key and the shel1svc00 SSH host key.

Edit the bundle with:

age-docker edit secret.env.age

When a configured recipient changes, re-encrypt the bundle and validate the complete policy:

age-docker rekey secret.env.age
age-docker check

age-docker creates mode-0600 temporary plaintext, removes it after the editor exits or a handled signal occurs, produces armored ciphertext, and verifies the new ciphertext with the personal identity before atomically replacing the tracked file. It does not add implicit recipients: every key with access is declared in .age-docker.toml.

Never add a plaintext secret file to Git. The Stack tracks the encrypted bundle so that secret rotation counts as a deployment change, but it does not track the temporary .komodo/prod.env file.

The decrypted values are Compose secret sources rather than service environment values. Containers receive only their declared files under /run/secrets; Authentik reads them through file:// configuration values, while PostgreSQL and GeoIP Update use their _FILE variables. Consequently, docker compose config renders secret source names and file paths without persisting the values in Komodo's merged configuration.

Komodo Compose wrapper

Komodo wraps config, pull, up, and run with the NixOS-provided with-age-env command. The wrapper:

• obtains an exclusive per-bundle lock;
• removes stale plaintext and temporary files;
• decrypts through the NixOS-provided systemd age-identity credential;
• creates plaintext with mode 0600; and
• removes plaintext on success, failure, or a handled signal.

The plaintext target and matching .lock and .tmp.* paths are exclusively owned by the wrapper. A SIGKILL or power loss can leave plaintext behind; the next invocation removes it before decrypting again.

Workload containers do not receive the age identity, but commands launched by Periphery can access it. Revisit that trust boundary if deployment authority is ever granted to another operator.

Upgrade

There is currently no long-lived test environment. Prepare and review every upgrade on a branch before merging it into production. If a test Stack is added later, perform the upgrade there first.

1. Confirm that the host backups are current and restorable.

2. Read the target version's Authentik release notes and upgrading section. Note all manual migrations, breaking configuration changes, and outpost compatibility requirements before changing files.

3. Create an upgrade branch.

4. Download the target version's upstream Compose file alongside the tracked file. Replace the example version with the intended target:

   authentik_version="2025.8"
   curl -fsSL \
     "https://goauthentik.io/version/${authentik_version}/lifecycle/container/compose.yml" \
     -o docker-compose.yml
   

5. Compare docker-compose.yml with the previously tracked version.

6. Review docker-compose.override.yml against the new base structure. Keep the host PostgreSQL connection, Traefik's HTTP upstream on the private frontend network, and persistent volumes working with the new release.

7. Update AUTHENTIK_TAG in .env. Review the supported PostgreSQL and Redis versions before changing either service. Plan to update all Authentik outposts to the same Authentik version.

8. Validate the resulting policy and Compose model. Dummy values satisfy Compose's secret-source checks without decrypting production secrets:

   age-docker check
   PG_PASS=dummy \
   AUTHENTIK_SECRET_KEY=dummy \
   AUTHENTIK_EMAIL__PASSWORD=dummy \
   GEOIPUPDATE_LICENSE_KEY=dummy \
     docker compose \
       --file docker-compose.yml \
       --file docker-compose.override.yml \
       config --quiet
   

9. Review the complete diff, commit it, push the branch, and merge it through a merge request. Do not run docker compose pull, down, or up manually on the host; the merge triggers the ordered Komodo deployment flow.

10. Follow the authentik-deploy Procedure and authentik-prod Stack logs in Komodo. Before public cutover, verify the target through Traefik with bash scripts/check-traefik.sh shel1svc00.s1q.dev and keep the public edge unchanged. At cutover, verify a login and update outposts to the matching version.

Record any repository-specific manual migration steps in this README before the rollout. None are currently documented.

Rollback

For a configuration or image regression, revert the merge on main; the webhook will run the same ordered deployment flow with the previous tracked configuration. Do not assume that an application or database migration can be downgraded. Follow the target release's rollback guidance and restore a tested backup when a migration is not reversible.

Backup and disaster recovery

The host's backup definition and restore instructions are in nixconfig/services/shel1svc00/postgresql/databases/authentik.md. Enable its timer only after the database and certs and media volumes are restored, and verify a backup and restore before relying on the target.

The 2026-10-02 export from snbg1svc00 is a point-in-time copy. The source was restarted immediately after the export and continues to accept writes. The target therefore requires a fresh final sync before any public cutover; never treat its restored database as current production data. The target worker is in the cutover Compose profile so it cannot run background tasks while the source remains active. Enable that profile only when the source is retired and a final data sync has completed.

Optional test environment

Do not create a test Stack or dummy bundle until there is a real long-lived test branch. When it is needed:

1. add secrets/test.env.age, encrypted to the test host and personal keys;
2. keep both Stack declarations in komodo.toml on main;
3. point authentik-test at branch test and its own encrypted/decrypted paths; and
4. add authentik-test to the Procedure's deployment stage in the central bootstrap repository.

The repository webhook should use the Procedure's __ANY__ endpoint so pushes to either protected environment branch enter the same ordered flow.