Why#
There is a category of error that is more dangerous than a broken system: a system that appears to be working correctly while doing something fundamentally different from what was intended.
A security control that blocks the wrong traffic. An authentication layer that never fires. A middleware that every request bypasses without leaving a trace.
These errors are not visible in monitoring. They do not produce alerts. They show up only when you read the source carefully enough to understand what the software actually does — not what you assumed it did when you configured it.
This week produced two of them, discovered on the same day.
Family#
The services the family uses — password manager, photos, files, wiki — were not affected. They continued working throughout. The problem was in the administrative layer: the control panel for managing the gateway that routes all external traffic.
That control panel had been inaccessible for days. Every attempt to open it returned an error without explanation. The investigation into why took most of a day and rewrote two decisions in the process.
Tech#
The first discovery came from reading log files carefully. The gateway’s reverse proxy was logging the source address of every incoming request. That address was always the same: the internal IP address of the tunnel software itself, not the actual client making the request. The IP restriction that was supposed to limit access to the administrative dashboard to the management VPN had never worked. Every request from the public internet carried the same internal address. The filter had no effect. The gateway had been publicly reachable for weeks while appearing to be restricted.
The second discovery came from source validation. The original plan for fixing the authentication problem involved replacing the existing middleware with a different component — one that would connect the gateway’s authentication to the central identity system running on the home server. A decision record was written, reviewed, and approved. Then the documentation for the gateway software was read more carefully.
The middleware in question is not a replaceable component. It is integral to how the gateway software manages sessions. Replacing it would not fix the authentication flow — it would break session management entirely. The proposed architecture was based on a misunderstanding of how the software works.
The decision record was rewritten the same day it was written. The new version took a different approach: use the gateway software’s built-in identity provider integration, which connects to the central identity system through the documented interface rather than around it. That approach worked. The administrative dashboard became accessible. Authentication through the central identity system succeeded end-to-end.
The fix for the IP restriction problem was different. The root cause — the tunnel software passing its own internal address instead of the client’s — cannot be solved at the filter level. It requires a different configuration at the tunnel layer itself. That fix was documented and deferred; the non-functional filter was removed rather than left in place creating a false sense of security.
One additional problem emerged during the investigation. The containers on the gateway server needed to resolve the hostname of the central identity system to connect to it correctly. The workaround — editing a configuration file on the host — did not persist inside the containers and reset on every restart. The correct solution was to embed the hostname resolution directly in the container configuration, where it is version-controlled and survives any restart.
Project#
Three separate problems were resolved in a single day, each discovered while investigating the previous one. A non-functional security filter. An authentication architecture built on a misreading of the documentation. A DNS workaround that worked on the host but not inside the containers that actually needed it.
The day’s work produced four commits and a rewritten decision record.
The rewrite is worth noting. Writing a decision record, discovering it is wrong before implementation, and rewriting it on the same day is not a failure of the planning process. It is the planning process working correctly. The record existed to be scrutinised. The scrutiny found the error. The implementation that followed the corrected record worked on the first attempt.
The alternative — implementing first, discovering the error in production, and debugging a broken authentication system while the family tries to use their services — would have been significantly worse.
The security filter that never worked raises a different question. The system appeared secure. The monitoring showed nothing unusual. There was no alert, no log entry, no signal of any kind that the filter was not doing what it was configured to do. The discovery came from reading the proxy logs with enough attention to notice that the source address was always the same.
That kind of attention is not automatic. It requires time set aside specifically to look at things that appear to be working.