Why#
There is a specific kind of project confidence that accumulates as things start working. Each service deployed, each integration completed, each monitoring alert configured — the stack feels increasingly solid. The danger of that feeling is that it becomes easy to confuse “things are working” with “things will keep working.”
Backup was not an afterthought in this project. BorgBackup was configured in Phase 0. PostgreSQL dumps were running daily. The borgmatic hooks were in place. But there is a difference between having a backup and having a tested, documented, externally executable recovery procedure. The first is a technical task. The second is a discipline.
Phase 7 was the discipline.
The question it asked was not “is there a backup?” but “could someone else restore this system from the backup, following only the documentation, without calling me?”
The honest answer, before this phase, was no.
Family#
Every service in the stack has a break-glass route — a way to log in when the authentication system itself is unavailable. Vaultwarden has an admin panel accessible without SSO. Seafile has a local account that works regardless of Authentik’s state. Grafana has a local admin. Immich has a local admin.
Outline did not. There was no working authentication path that bypassed the OIDC flow. If Authentik went down, the wiki was unreachable.
The fix was straightforward once identified. The more important thing was the identification itself — going through every service, one by one, and asking: if Authentik is completely unavailable, can this service still be accessed? The answer was no for one service. It could have been no for others if the exercise had not been done.
This is the break-glass matrix. A table, one row per service, documenting the fallback route and where the credentials live. It exists for the scenario where I am not there to explain it.
Tech#
The backup scope audit found eight gaps.
The most significant were not databases. Everyone audits databases. The gaps were in the files that sit alongside the databases — the Vaultwarden RSA key stored in the data directory, the PKI certificate authority private key, the Caddy CA chain, the Metabase application database that nobody had thought to add to the dump hooks. None of these would have been restored from a database dump alone. All of them would have been lost in a bare-metal recovery.
The fix was adding them explicitly to the backup scope. The lesson was that a backup strategy designed around “dump the databases” is incomplete for a stack where several services store critical state outside the database.
Monitoring was added to make the backup visible. A small bash script runs after each borgmatic backup and writes a Unix timestamp to a file. Node Exporter reads the file. Prometheus scrapes Node Exporter. Grafana shows a panel: time since last successful backup. Green below 24 hours, orange below 48, red above. The pipeline from bash script to Grafana panel took an afternoon. The value is that backup failure is now something Grafana can alert on, not something discovered retroactively when a restore is needed.
The Authentik incident was instructive. During MFA policy configuration, a group binding was accidentally placed on the authentication flow itself rather than on a specific stage within the flow. This blocked the flow for all users — including administrators. There was no way to log in to Authentik to fix it. The recovery required a one-minute token generated from the command line, used immediately before expiry, to access the admin interface and remove the binding. The lesson: in Authentik, policies on flows block everyone. Policies on stages are targeted. The distinction is not obvious from the interface.
Proton Drive remained blocked throughout the phase. Multiple authentication attempts triggered an anti-bot verification that did not resolve. The off-site backup is the one gap that was not closed — local BorgBackup on the attached drive remains the only copy outside the primary database. This is a known risk, documented, with Hetzner Storage Box as the fallback option.
Project#
The most useful output of this phase was a document written for someone else.
The DR runbook in the wiki is not written for me. It is written for a technically competent person who has never seen this system before, who needs to restore it after something has gone wrong, and who cannot ask questions. It explains the architecture in one paragraph. It lists the services and their dependencies in order. It documents each restore step with exact commands. It includes the break-glass matrix and the secret inventory locations.
Writing it required making explicit every assumption I had been carrying implicitly. Where are the backup keys? Off-site, secured with MFA, which requires a TOTP code from the same account. Is that circular? Slightly. The mitigation is that backup codes for the TOTP device are stored separately, also in the document.
Every system has this kind of dependency chain. Most systems never write it down. Writing it down makes the chain visible — and visible chains can be strengthened.
The restore simulation confirmed the backup was functional. Two databases restored into parallel test databases, row counts matched, test databases deleted. Recovery point objective measured at under 24 hours. These are numbers, not feelings.
The external validation — having someone else follow the runbook without guidance — remains open. That is the test that actually matters. It is planned for the next phase.