When my home server froze, the whole house lost DNS
My home server is the house's only DNS server, so each freeze took DNS down for everyone. I've had three guesses at why, and the third is still a guess.
My home server is a small, low-power mini PC that runs my self-hosted apps, and it’s also the only DNS server in the house. Every device that joins the home network is told to ask it for addresses, and it blocks ads while it’s at it. So when it started going down in May, the whole house lost DNS every time.
It went down four times between 14 May and 9 June. I’ve had three guesses at why. The first lasted about an hour, the second didn’t stop the freezes, and the third is still a guess. Through all of it I kept the house on one DNS server, on purpose.
One small box answers every lookup
The home server is a mini PC with a Ryzen 7 5825U, an AMD laptop chip, and 32 GB of memory. It runs my self-hosted apps in Docker containers, and one of those containers is an ad-blocking DNS server. The router tells every phone and laptop that joins the network to use it as their only DNS server. It also answers for the names of my apps at home, which public DNS servers don’t know.
So when the box stops, every device in the house still has a network, but none of them can look up a single name. For anyone at home, that looks the same as the internet being down.
A crash, a freeze and the first guess
The first outage, on 14 May, was a kernel crash, and I still don’t know what caused it. When I power-cycled the box it came back with no network, because its network settings let it stop trying without ever getting an address. The next day I made it recover on its own. The network keeps trying until it has an address, the apps wait for the network, and after a kernel crash the box restarts itself.
Three days later it froze. On 17 May the system log ended 6 seconds after a scheduled job started a web scraper I hosted on the box, which meant five containers starting at the same moment. Nothing else looked wrong: no shutdown messages, no programs killed for lack of memory, and 21 GB of memory free. The box stayed frozen until I restarted it by hand three and a half hours later.
Five containers starting at once means five sets of virtual network connections coming up together, and that was the last thing in the log before the freeze. So it was my first guess, and I turned the scheduled start off. I keep the box’s setup in a git repository, and that commit says the job “was crashing the box”. About an hour later the next commit said that reason was wrong.
The second guess: a kernel bug
The box was running Linux 6.17, and other owners of this processor had reported hard freezes on that kernel but not on the older 6.8. That fit better than the containers did, so I moved the box to 6.8.
It froze again anyway. On 26 May and again on 9 June it froze on 6.8, both times while it was doing nothing. The kernel reported no crash or error, and the log stopped in the middle of a line. The automatic restart needs the kernel to notice its own crash, and in these freezes it never did, so both times it took a hard power cycle.
The third guess: the processor’s deepest idle state
The kernel bug was in 6.17, so these freezes needed another explanation. When a processor has nothing to do, it steps down through idle states, and each step switches off more of the chip to save power. My third guess is that this processor freezes when it steps into its deepest idle state, which would also explain why both freezes came while the box was idle. If that’s right, no kernel version fixes it.
So I capped it. The box now boots with a kernel setting that keeps the processor in its lightest idle state. That costs some power, since the deeper states are where it saves the most.
It’s still a guess, and it isn’t the only power setting on this processor I’ve changed. In January, to save power, I lowered the most it may draw in the BIOS, from 25 W to 15 W, and I haven’t checked whether that plays any part.
Why the house still has one DNS server
Every one of these outages took DNS for the whole house with it. The obvious fix is a second DNS server in the router, a public one, for when the box is down. I didn’t add one.
Devices don’t keep a second DNS server for emergencies. They pick between the servers they’re given, so a public second server would skip the ad blocking at random and fail my home names at random. My notes put it in one line: a fallback turns “DNS is down” into “some things work”. With one server I know when DNS is down.
For repairs there’s a written way around it: point the router at a public DNS server, fix the box, and point the router back. The notes mark that last step as mandatory, because a public server left in place quietly turns off the ad blocking and hides the next failure.
What I learned
The first cause that fit the log was wrong. The log ended 6 seconds after five containers started, so I blamed the containers. The kernel bug came next, and the freezes after that pointed at the processor’s idle states. The first correction went into the very next commit, so the history shows the wrong guess too.
Recovering from a crash doesn’t cover a freeze. The automatic restart needs a kernel that knows it crashed. The freezes never got that far, so the box sat frozen until someone cut the power.
A second DNS server would have hidden all of this. Each outage would have looked like “some things work”, with home names failing at random. With one server the failure is loud, and the way around it is written down before I need it.
Where it stands
The house still has one DNS server. The box comes back on its own after a crash, and it now runs with the idle cap, my third guess at the freezes. A freeze, if one comes, still needs someone at the plug. The two freezes on the older kernel came two weeks apart, so the cap has to stay quiet for a lot longer than that before the guess counts as an answer.