Why#
There is a category of problem that is worse than something being broken: something appearing to work while silently failing. No error. No alert. No log entry that says “this is wrong.” Just a system that looks healthy while doing the wrong thing — or nothing at all.
This week was about finding those problems. Not because something had crashed. Because I decided to look carefully at things I had assumed were working.
What I found changed how I think about operating a self-hosted stack.
Family#
From the outside, nothing changed this week. All services stayed online. No one noticed anything because nothing was visibly broken.
That is exactly the point.
The work of running a private infrastructure is mostly invisible. Not building new things — maintaining what exists. Verifying that the systems watching over your data are actually watching. That the backups running every night are running the right version. That the security tools blocking attackers are blocking the right addresses.
This kind of work has no visible result. It is the difference between a house that looks fine and a house that is fine.
Tech#
Four separate systems turned out to be silently wrong.
The security scanner was watching the wrong addresses. The tool responsible for detecting and blocking malicious traffic had been analysing proxy addresses instead of real client IPs — for weeks. Every ban it issued hit the wrong target. A one-line configuration fix was all it took, but finding the problem required reading logs at three different layers simultaneously: what the web server logged, what the security tool parsed, and what it actually acted on.
The log pipeline was dropping everything without complaint. A filter in the log collector was written against a field that does not exist as a routing label in the log storage system. The result: every log entry was silently discarded. The pipeline showed no errors. The counters incremented. Nothing arrived. The fix was to remove the filter entirely and move filtering to the query layer where it belongs — but finding the root cause required stepping through each stage of the pipeline individually and measuring what entered and what left.
The backup tool was running an outdated version. The package manager had stopped tracking upstream releases. The installed version was a year and a half behind. The backups themselves were completing successfully, but improvements to database hook handling — directly relevant to how this stack dumps PostgreSQL and MariaDB — had accumulated in the meantime. Migration to a different installation method brought everything current. The first backup run after migration completed cleanly.
The container scanner had never sent a notification. A weekly security scan was running and writing results to a log file. The script was correctly placed, the scan was completing — but there was no alert on completion. The results existed only for someone who thought to look. Adding a push notification surfaced the first real scan result: over a hundred critical and high vulnerabilities across the running containers, most of them upstream dependencies waiting on fixes that do not yet exist.
Each of these had something in common. None produced an error. None triggered an alert. None were visible without deliberate investigation. The monitoring stack reported everything healthy because the monitoring stack was itself part of what needed monitoring.
Project#
The work this week covered ground across several areas simultaneously: security tooling, observability pipelines, backup operations, and container hygiene. None of it was planned. It emerged from reading the daily digest carefully and following what it implied.
One structural pattern became clear: silence is not the same as correctness. A system that produces no errors is not necessarily a system that is working. Verification requires more than checking that a process is running — it requires checking that the process is producing the right output at every stage.
The practical change this week was adding notification to anything that runs unattended. Scans now push results on completion. Alerts now cover two thresholds instead of one for storage that grows continuously. The log pipeline now routes correctly, which means the alert templates that depend on it can finally be built.
The work that cannot be finished yet — vulnerabilities in containers that depend on upstream fixes — is at least now visible. Visible problems can be tracked. Silent ones cannot.