docker-compose/authentik
0
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat: deploy Authentik with age-encrypted secrets

Browse source
This commit is contained in:
Philip Henning 2026-08-21 14:04:26 +02:00
parent 9f8f619f3e
commit e966ec09ae
9 changed files with 240 additions and 260 deletions
Download patch file Download diff file Expand all files Collapse all files

341
README.md
View file

@ -1,269 +1,120 @@
# `sso.s1q.dev` - Base23 SSO for all services
# Authentik deployment
[Authentik](https://goauthentik.io/) based SSO for our sevices.
This repository owns the production Authentik Compose deployment and its
Komodo Stack declaration. The central `komodo-bootstrap` repository owns the
child Resource Sync and the ordered deployment Procedure.
## Table of Contents
## Layout
- [`sso.s1q.dev` - Base23 SSO for all services](#ssos1qdev---base23-sso-for-all-services)
- [Table of Contents](#table-of-contents)
- [Prerequisites](#prerequisites)
- [Tailscale](#tailscale)
- [CrowdSec](#crowdsec)
- [Setup CrowdSec Repo](#setup-crowdsec-repo)
- [Install CrowdSec](#install-crowdsec)
- [Configure CrowdSec](#configure-crowdsec)
- [Installation](#installation)
- [Clone \& configure initially](#clone--configure-initially)
- [Fist run](#fist-run)
- [Upgrade](#upgrade)
- [Test](#test)
- [Prod](#prod)
- [Disaster recovery / restore](#disaster-recovery--restore)
- [Rebuild containers locally](#rebuild-containers-locally)
## Prerequisites
- dokploy
### Tailscale
```shell
printf "Enter preauthkey for Tailscale: " \
&& read -rs TAILSCALE_PREAUTHKEY \
&& curl -fsSL https://pkgs.tailscale.com/stable/debian/bookworm.noarmor.gpg | sudo tee /usr/share/keyrings/tailscale-archive-keyring.gpg >/dev/null \
&& curl -fsSL https://pkgs.tailscale.com/stable/debian/bookworm.tailscale-keyring.list | sudo tee /etc/apt/sources.list.d/tailscale.list \
&& apt-get update \
&& apt-get install tailscale \
&& tailscale up --login-server https://vpn.s1q.dev --authkey ${TAILSCALE_PREAUTHKEY} \
&& sleep 2 \
&& tailscale status \
&& unset TAILSCALE_PREAUTHKEY
```text
.
├── docker-compose.yml
├── docker-compose.override.yml
├── komodo.toml
├── env/
│ └── common.env
├── secrets/
│ └── prod.env.age
├── scripts/
│ ├── compose.sh
│ └── init.sh
└── tests/
└── compose-secrets.sh
```
### CrowdSec
`env/common.env` contains tracked, non-secret configuration and explicit image
versions. `secrets/prod.env.age` contains only secrets and is encrypted to the
`sbx0docker01` SSH host key and the personal `agenix-phg` SSH key.
#### Setup CrowdSec Repo
The `authentik-prod` Stack follows protected `main`. Its direct webhook is
disabled. A Forgejo webhook calls the central `authentik-deploy` Procedure,
which runs the child Sync before `DeployStackIfChanged`.
```shell
apt update \
&& apt upgrade -y \
&& apt install -y debian-archive-keyring \
&& apt install -y curl gnupg apt-transport-https \
&& mkdir -p /etc/apt/keyrings/ \
&& curl -fsSL https://packagecloud.io/crowdsec/crowdsec/gpgkey | gpg --dearmor > /etc/apt/keyrings/crowdsec_crowdsec-archive-keyring.gpg \
&& cat << EOF > /etc/apt/sources.list.d/crowdsec_crowdsec.list \
&& apt update
deb [signed-by=/etc/apt/keyrings/crowdsec_crowdsec-archive-keyring.gpg] https://packagecloud.io/crowdsec/crowdsec/any any main
deb-src [signed-by=/etc/apt/keyrings/crowdsec_crowdsec-archive-keyring.gpg] https://packagecloud.io/crowdsec/crowdsec/any any main
EOF
Komodo passes `docker-compose.yml` first and `docker-compose.override.yml`
second. The base and deployment-specific override remain separate by design.
## Secrets
The encrypted production bundle contains:
```text
PG_PASS
AUTHENTIK_SECRET_KEY
AUTHENTIK_EMAIL__PASSWORD
GEOIPUPDATE_LICENSE_KEY
```
#### Install CrowdSec
To edit it locally, decrypt to a mode-`0600` temporary file, edit it, then
encrypt it to both recipients:
Install CrowdSec:
```bash
tmp=$(mktemp)
chmod 0600 "$tmp"
trap 'rm -f -- "$tmp"' EXIT HUP INT TERM
```shell
printf "Enter CrowdSec context: " \
&& read -rs CROWDSEC_CONTEXT \
&& apt install -y crowdsec crowdsec-firewall-bouncer-iptables \
&& cscli completion bash | tee /etc/bash_completion.d/cscli \
&& source ~/.bashrc \
&& cscli console enroll -e context ${CROWDSEC_CONTEXT} \
&& unset CROWDSEC_CONTEXT
age --decrypt \
--identity ~/.ssh/identities/agenix-phg \
--output "$tmp" \
secrets/prod.env.age
${EDITOR:-vi} "$tmp"
age --encrypt \
--recipient 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIH4umkUY5CG5aNJBhUjVcU8TWbh453N0pGHZhYDTOGQa' \
--recipient 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIr4Dji0wWzwSXyqbxdGa8LdWkhP+0T7kDKrLbyCoyos' \
--output secrets/prod.env.age.new \
"$tmp"
mv secrets/prod.env.age.new secrets/prod.env.age
```
Restart CordSec Service, after accepting the enrollment on the [CrowdSec Console](https://app.crowdsec.net/):
Do not add a plaintext secret file to Git. The Stack config tracks the
encrypted bundle so a rotation counts as a deployment change, but does not
track the temporary `.komodo/prod.env` file.
```shell
systemctl restart crowdsec; systemctl status crowdsec.service
```
The decrypted values are Compose secret sources rather than service
environment values. Containers receive only their declared files under
`/run/secrets`; Authentik reads them through `file://` configuration values,
while PostgreSQL and GeoIP Update use their `_FILE` variables. Consequently,
`docker compose config` renders secret source names and file paths without
persisting the values in Komodo's merged configuration.
#### Configure CrowdSec
## Compose wrapper
Whitelist Tailscale IPs:
Komodo wraps `config`, `pull`, `up`, and `run` with
the NixOS-provided `with-age-env` command. The wrapper:
```shell
cat << EOF > /etc/crowdsec/parsers/s02-enrich/01-s1q-dev-tailscale.yaml \
&& systemctl restart crowdsec; journalctl -xef -u crowdsec.service
name: s1q-dev/tailscale ## Must be unqiue
description: "Whitelist events from Tailscale Subnet"
whitelist:
reason: "Tailscale clients"
cidr:
- "100.64.0.0/10"
EOF
```
- obtains an exclusive per-bundle lock;
- removes stale plaintext and temporary files;
- decrypts through the NixOS-provided systemd `age-identity` credential;
- creates plaintext with mode `0600`; and
- removes plaintext on success, failure, or a handled signal.
Whitelist my current Public IPs:
The plaintext target and matching `.lock` and `.tmp.*` paths are exclusively
owned by the wrapper. A `SIGKILL` or power loss can leave plaintext behind;
the next invocation removes it before decrypting again.
```shell
mkdir -p /etc/crowdsec/postoverflows/s01-whitelist/ \
&& cat << EOF > /etc/crowdsec/postoverflows/s01-whitelist/01-s1q-dev-public-ips.yaml \
&& crowdsec -t && systemctl restart crowdsec; systemctl status crowdsec.service
name: s1q-dev/public-ips ## Must be unqiue
description: "Whitelist events from s1q-dev public IPs"
whitelist:
reason: "s1q-dev Public IPs"
expression:
- evt.Overflow.Alert.Source.IP in LookupHost("r3w.de")
EOF
```
NixOS owns the wrapper, its runtime dependencies, the host-key systemd
credential, Periphery's Podman permissions, and the external
`dokploy-network`. Workload containers do not receive the age identity, but
commands launched by Periphery can access it. Revisit that trust boundary if
deployment authority is ever granted to another operator.
Add Authentik integration:
Run `bash tests/compose-secrets.sh` to verify that sentinel secret values do
not appear in the rendered Compose model and that every service receives only
its intended secret files.
```shell
cscli collections install firix/authentik \
&& cat << EOF > /etc/crowdsec/acquis.d/authentik.yaml \
&& crowdsec -t && systemctl restart crowdsec
---
source: docker
container_name_regexp:
- sso-s1q-dev-de-server-*
- sso-s1q-dev-de-worker-*
labels:
type: authentik
EOF
```
## Optional test environment
Enable increasing ban time:
Do not create a test Stack or dummy bundle until there is a real long-lived
`test` branch. When it is needed:
```shell
sed -i -e 's/^#duration_expr/duration_expr/g' /etc/crowdsec/profiles.yaml \
&& crowdsec -t && systemctl restart crowdsec
```
1. add `secrets/test.env.age`, encrypted to the test host and personal keys;
2. keep both Stack declarations in `komodo.toml` on `main`;
3. point `authentik-test` at branch `test` and its own encrypted/decrypted
paths; and
4. add `authentik-test` to the Procedure's deployment stage in the central
bootstrap repository.
Setup notifications:
## Installation
### Clone & configure initially
1. [Create a Storage Box sub account](https://confluence.base23.de/pages/viewpage.action?pageId=27820074).
2. Enter the username to `env.template`.
3. Run the initial configuration script:
```shell
cd /root/apps \
&& git clone ssh://git@git.base23.de:222/base23/sso.s1q.dev.git \
&& cd sso.s1q.dev \
&& ./scripts/init.sh
```
4. Use the generated SSH key and copy it to the Hetzner Storage box for backups:
```shell
TARGET_DOMAIN=cloud.backup.base23.de \
TARGET_KEY_TYPES="ecdsa-sha2-nistp521,ed25519,ed25519-sk,rsa,dsa,ecdsa,ecdsa-sk" \
TARGET_IPV4=$(dig +short "${TARGET_DOMAIN}" A | grep -E '^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$') \
TARGET_IPV6=$(dig +short "${TARGET_DOMAIN}" AAAA | grep -E '^(([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:)|fe80:(:[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::(ffff(:0{1,4}){0,1}:){0,1}((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])|([0-9a-fA-F]{1,4}:){1,4}:((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9]))$') \
&& cat ./data/restic/ssh/id_ed25519.pub | ssh -p23 u291924-sub4@${TARGET_DOMAIN} install-ssh-key \
&& ssh-keyscan -p 23 -t ${TARGET_KEY_TYPES} ${TARGET_DOMAIN} > ./data/restic/ssh/known_hosts \
&& ssh-keyscan -p 23 -t ${TARGET_KEY_TYPES} ${TARGET_IPV4} >> ./data/restic/ssh/known_hosts \
&& ssh-keyscan -p 23 -t ${TARGET_KEY_TYPES} ${TARGET_IPV6} >> ./data/restic/ssh/known_hosts \
&& ssh-keyscan -p 22 -t ${TARGET_KEY_TYPES} ${TARGET_DOMAIN} >> ./data/restic/ssh/known_hosts \
&& ssh-keyscan -p 22 -t ${TARGET_KEY_TYPES} ${TARGET_IPV4} >> ./data/restic/ssh/known_hosts \
&& ssh-keyscan -p 22 -t ${TARGET_KEY_TYPES} ${TARGET_IPV6} >> ./data/restic/ssh/known_hosts
```
### Fist run
```shell
./scripts/compose.sh build --no-cache \
--build-arg BUILD_DATE=$(date -u +'%Y-%m-%dT%H:%M:%SZ') \
--build-arg SRC_REV=$(git rev-parse --short HEAD) \
&& ./scripts/compose.sh up -d; ./scripts/compose.sh logs -f
```
## Upgrade
### Test
This is intended for testing the upgrades before rollout on prod.
1. Check if the backups are up to date: `./scripts/compose.sh run --rm restore-cli /usr/local/bin/restic snapshots`
2. Create a new branch `git checkout -b <version>`.
3. Download the the docker compose for the version you want to update:
`curl -fsSL -o docker-compose.yml https://goauthentik.io/version/<version>/docker-compose.yml`
4. Update `AUTHENTIK_TAG` to the desired tag in `env.test.template`.
5. Check the upstream `docker-compose.yml` file against ours for changes made in the configuration. Check the Postgresql and Redis docker tags. Minor revisions of Postgresql *should be fine*, check the changelogs for any issues and if none are present raise to the latest Minor version (e.g. 16.6 -> 16.9). Redis should be less problematic for upgrades, check nonetheless.
6. Run `diff --color='auto' env.test.template .env` to display the diff between
`env.test.template` and `.env`.
7. Port the changes made to `.env`.
8. `./scripts/compose.sh pull`
9. `./scripts/compose.sh down`
10. `./scripts/compose.sh up -d; ./scripts/compose.sh logs -f`
9. Check the logs for any issues during during startup. Check if https://sso.test.base23.de is available and test the sso login (e.g https://whoami.test.base23.de)
10. Apply changes for test to prod files (`docker-compose.<stage>.yml`, `env.<stage>.template`), commit & push changes to the Repo in a new branch and create a merge request in preparation for the prod upgrade
### Prod
It is expected that the Upgrade has already been performed and tested on https://sso.test.base23.de, and the changes have been merged into main
1. Check if the backups are up to date: `./scripts/compose.sh run --rm restore-cli /usr/local/bin/restic snapshots`
2. Create a new branch `git checkout -b <version>`.
3. Download the the docker compose for the version you want to update:
`curl -fsSL -o docker-compose.yml https://goauthentik.io/version/<version>/docker-compose.yml`
4. Update `AUTHENTIK_TAG` to the desired tag in `env.prod.template`.
5. Commit & push changes to the Repo.
6. Run `diff --color='auto' env.prod.template .env` to display the diff between
`env.prod.template` and `.env`.
7. Port the made changes to `.env`.
8. `./scripts/compose.sh pull`
9. `./scripts/compose.sh down`
10. `./scripts/compose.sh up -d; ./scripts/compose.sh logs -f`
## Disaster recovery / restore
**IMPORTANT:**
You've to use different docker cli clients on prod/test.
- Prod
- Docker: `docker`
- Docker compose: `docker compose`
- Test
- Docker: `docker`
- Docker compose: `docker-compose-2.32.4`
For the ease of readability I'll use `docker` and `docker compose` in the documentation below,
please repalce it for restores on test!
1. Run the restore cli
```shell
docker compose run --rm restore-cli
```
2. Run the restore command and follow it's instructions
```shell
restore
```
3. If the restore was successfull, exit the restore container.
**DO NOT START THE APPLICATION YET!**
4. Run the PostgreSQL container without starting the main application
```shell
docker compose run --rm postgresql
```
5. Open another shell in the sso git directory.
6. Execute a shell in the running PostgreSQL container (replace `<containerid>` with the actual container id)
```shell
docker exec -it sso-base23-de-postgresql-run-<containerid> bash
```
7. If the database already contains data, delete an recreate it:
```shell
dropdb -U ${PG_USER:-authentik} ${PG_DB:-authentik}
createdb -U ${PG_USER:-authentik} ${PG_DB:-authentik}
```
8. Restore the database
```shell
psql ${PG_USER:-authentik} -d ${PG_DB:-authentik} -f /var/lib/postgresql/backups/authentik.sql
````
9. After the database is restored, exit the container
10. Now it's safe to start the complete application stack again
```shell
docker compose up -d; docker compose logs -f
```
## Rebuild containers locally
```shell
docker compose build --no-cache \
--build-arg BUILD_DATE=$(date -u +'%Y-%m-%dT%H:%M:%SZ') \
--build-arg SRC_REV=$(git rev-parse --short HEAD)
```
The repository webhook should use the Procedure's `__ANY__` endpoint so pushes
to either protected environment branch enter the same ordered flow.