Skip to main content

Flipping the Switch

Author
Thornbush
HomeStack Part I - This article is part of a series.
Part 11: This Article

Why
#

At some point you have to stop building and start proving.

For over a month, HomeStack had been a construction site. Every phase added something new — a database, a service, a monitoring layer. Each piece worked in isolation. The stack felt solid. But “feels solid” is not the same as “is solid,” and I knew the difference mattered before I handed this infrastructure to my family.

Production Readiness was the phase where I stopped adding and started auditing. The question changed from “does it work?” to “can I prove it works, and prove it stays working?”

Family
#

My partner and daughter don’t think about infrastructure. They think about whether their photos are there, whether they can log in, whether things just work. That’s the right way to think about it — and it’s exactly why I needed to be certain before they started depending on it.

The goal of this phase wasn’t features. It was trust. Not my trust in the system, but the kind of trust that holds when I’m not around to fix things.

Tech
#

I started with a security audit. Internal port scan, external scan from a separate VPS, HTTP vulnerability scan against all seven public-facing services.

The internal scan found PostgreSQL exposed on a host port — reachable from anywhere on the local network. Not intentional. Not acceptable. Removed immediately, all dependent services restarted healthy. The external scan showed only Cloudflare infrastructure. The Pi was invisible, exactly as designed.

The HTTP scanner against the public services returned two findings per service. Both were false positives introduced by the Cloudflare proxy layer. No real vulnerabilities.

One scan, one real finding, one immediate fix. That’s what audits are for.

Before this phase, 28 containers were running with no central log visibility. Individual container logs existed but were unqueryable as a whole. If something suspicious happened across multiple services simultaneously, I would have had no way to see it.

Loki and Alloy closed that gap. Alloy collects logs from every Docker container and from the host journal. Loki stores and indexes them. Grafana queries them. Total RAM overhead: around 260MB for both services together.

Then came the alert rules — fourteen security alerts covering every service. Failed login attempts, brute force patterns, HTTP error spikes. Each alert required careful query design: Grafana’s Loki alerts need an explicit reduce step between the time-range query and the threshold evaluation. Without it, the alert structure looks correct but silently never fires. I found several existing alerts in exactly that broken state. Fixed all of them.

The lesson: you can’t defend what you can’t see. And alerts that look configured but never fire are worse than no alerts — they give false confidence.

Two small incidents stand out.

The first: a configuration variable with a misleading name. I was trying to disable an admin panel. The variable name suggested “disable the token requirement.” What it actually meant was “open the panel without requiring any token at all.” The panel was briefly wide open. Closed immediately, but the lesson stayed: variable names lie. Always verify behavior, not just configuration.

The second: a service was showing an old authentication URL in its UI even though the configuration file had the correct new value. The configuration file wasn’t the source of truth — the database was. The service had stored the original value at first setup and never re-read the file for that field. Three layers to check: environment file, database, UI. Always all three.

I had tested backups before. Partial restores, file-level verification. But I had never done a full bare-metal restore on completely clean hardware.

I provisioned a fresh ARM64 cloud server. No pre-existing configuration. Installed the backup tooling, connected to the off-site repository, extracted the latest archive, restored the databases, started the services.

From first action to both critical services responding: twenty-four minutes. That number is now documented. It’s not an estimate or a hope — it’s a measured recovery time on real hardware. The server stayed running as a permanent development and disaster recovery node.

The final piece was hardening everything that had been deferred. Password rotation across all services and databases. SSH access restricted to the management VPN only — local network SSH no longer works, which is exactly the point. Single sign-on validated end-to-end for every service. Session tokens short-lived. Admin interfaces closed or protected by an additional authentication layer.

One service update came mid-phase when I found critical security fixes in the release notes: authentication bypass and server-side request forgery. Updated immediately rather than waiting for the scheduled maintenance window. Some things don’t wait.

Project
#

Production Readiness is the phase that separates infrastructure from a system you can actually trust.

The pattern I keep returning to: the things that nearly went wrong were never the things I was worried about. The exposed database port wasn’t on my risk list. The misleading variable name wasn’t in the documentation. The broken alert structure looked correct in every dashboard view.

Auditing finds the gap between what you think you built and what you actually built. That gap is always smaller than you fear and always surprising in its specifics.

On the evening of May 10th, I renamed the server from its development hostname to its production name. A one-line command. Nine phases of work behind it.

The switch was flipped.

HomeStack Part I - This article is part of a series.
Part 11: This Article