For a few weeks straight, I'd wake up, check my phone, and see that half my homelab was unreachable. My first assumption, every single time, was that something had crashed overnight and I'd have to SSH in and figure out what happened.

This issue specifically took me three rounds in the ring before I actually landed a real punch on this thing. Every round I thought I'd won. I'm writing this mostly so future me remembers the process, because the symptom looked identical every time and the cause was different all three times.

Round 1: Assuming it crashed

First thing I always check when something's unreachable is whether the box actually rebooted.

journalctl --list-boots

Nope. Same boot ID the whole time. No OOM killer, no kernel panic, nothing in dmesg about the NIC dropping. The VM never went anywhere. So whatever was breaking, it wasn't the machine dying, it was something happening on top of a perfectly healthy host.

Went digging through the logs from around 5am and found dockerd spamming the same error over and over:

[resolver] failed to query external DNS server ... i/o timeout

Every container losing name resolution at the same moment, host included. Okay, that's not "an app crashed," that's DNS just falling over.

Throwing the first punch (the fix that felt right, and mostly was)

The 5am timing lined up suspiciously well with a scheduled restart on my gateway, a UniFi Cloud Gateway Max that handles routing between my VLANs. And when I checked my resolver config, yeah, I only had one DNS server pointed at that gateway. No backup, nothing.

Figured that was it. Added fallback resolvers:

# /etc/systemd/resolved.conf
DNS=10.1.101.1 1.1.1.1 8.8.8.8

Small thing that tripped me up here: I assumed systemd-resolved's FallbackDNS setting would just handle this. It doesn't, that only kicks in when you have zero DNS servers configured at all, not when your one server stops responding. You actually have to list multiple servers under DNS= yourself to get real failover.

Also threw together a quick watchdog, a systemd timer checking DNS resolution every minute and bouncing systemd-resolved if it's failing. It only logs when something's actually wrong so it doesn't clutter things up.

Ran clean for a couple weeks. I figured I'd nailed it.

Round 2: It gets back up, wearing a different face

Then one evening, out of nowhere, everything dropped again. Not 5am this time, random, and it stayed down a full hour before I noticed. Same DNS timeout errors everywhere, so my first thought was "did my fix not actually work?"

But looking closer, this time everything was failing to resolve, including 1.1.1.1 and 8.8.8.8. Not just the gateway's own DNS. And that's the detail that mattered, because if Cloudflare and Google's public resolvers are unreachable, that's not a DNS problem anymore. There's no working path out at all. My watchdog was faithfully restarting systemd-resolved every sixty seconds like I told it to, sixty times over the course of the outage, and it was completely powerless because there was no route for it to succeed over even once.

Also noticed SSH from another one of my subnets just never showed up in the logs during that window. Zero attempts logged. Made sense once I realized that traffic routes through the same gateway too, so one device having a bad time took out my ability to fix things remotely at the same moment it broke everything else.

I did the thing you're not supposed to do and power-cycled the VM, assuming that would clear it. It didn't need to. The gateway came back on its own around the same time, so it just looked like my reboot fixed it. Classic false correlation. Rebooting the VM for a problem that lives upstream on a totally separate device was pointless and just added risk for nothing.

Checked the gateway's uptime afterward through its API and it had never rebooted either, five and a half days of continuous uptime straight through the outage. So something inside its OS locked up, both DNS and inter-VLAN routing, and then quietly came back without ever restarting. SSH access to that box is disabled by default so I couldn't pull real logs to confirm what hung, just left with a guess that it's fighting itself for resources (it also runs the camera NVR stuff onboard).

Round 3: The one that finally stayed down

Same evening, right after that mess, a second outage hit almost immediately. Everything down again. This time I pulled up the switch logs and noticed something I'd missed before, both my Proxmox host itself and one of its VMs dropped off the same physical switch port at the exact same second.

That's the kind of detail that only points one direction. If the host and a VM sitting on top of it both vanish together on the same wire, that's not a gateway problem or a routing problem, that's the physical NIC underneath everything.

SSH'd straight into the Proxmox host and checked its kernel log for that window:

e1000e ... nic0: Detected Hardware Unit Hang

Over and over, every two seconds, until I finally rebooted the host manually and it cleared. The onboard NIC (an Intel I219, using the e1000e driver) had hit a hardware TX hang and just never came back on its own. Never seen this in any earlier boot log, brand new failure, and completely unrelated to the DNS story from two weeks earlier that I'd already mentally closed.

Turns out e1000e/I219 has a bit of a reputation for exactly this under Linux, tied to Energy Efficient Ethernet and TCP offload settings, and EEE was sitting on by default on this NIC.

ethtool -K nic0 tso off gso off gro off
ethtool --set-eee nic0 eee off

Wrapped that in a small systemd service tied to the device so it reapplies on every boot instead of me having to remember to run it by hand. Fair warning if you do this yourself, flipping EEE off bounces the link for about a second, totally harmless but startling if you're watching it happen live.

Decision

I'd like to tell you I learned some big lesson about not declaring victory early. I probably will do it again next time something breaks and the first fix seems to work. That's kind of the nature of homelabbing, you're never actually done, you're just undefeated until the next round starts.