TLDR; If your server has IPMI enabled make sure it’s set to dedicated mode (or better yet disabled) and not set to failover mode or else your pfsense WAN gateway will go offline randomly!
Over the weekend I visited my local electronics flea market and came across a Supermicro 1U server with 5 front facing ethernet ports. This guy is sporting an Atom C2758 8 core processor and 8GB of ram. My current pfsense box is also a 1U Supermicro server but it has an atom D510. This wimpy processor caps my gigabit internet to ~700 Mbps so it’s time for an upgrade.
So once I got home I grabbed the nearest spare hard drive and loaded pfsense 2.4.3 right up. Everything installed correctly and I got a new router going. Nice! Or so I thought.
About an hour later the internet stops working… But the LAN still does. I can get into the pfsense web interface and everything looks good except that my WAN Gateway went offline. Well that’s weird but maybe it’s just a hiccup? So I restarted the server and hoped for the best but no luck. This issue of “working” but dying after an hour or three kept happening. I tried different versions of pfsense, held off from doing any kind of custom configurations / portforwarding, and even some different CAT6 cables.
No luck. It’s at this point that my 5 front facing ethernet ports are actually 4 ethernet and 1 IPMI port. Ok, well I didn’t use the IPMI port anyways since pfsense doesn’t know it exists, but maybe logging into IPMI would shed some light on this weirdness. I connected a CAT6 wire up so that I could connect to it and found its IP via pfsense’s DHCP leases page.
After some poking around I found one innocuous looking dropdown under the network settings labled “LAN Interface” which was set to “Failover”. After some reading it appears that IPMI, when not connected will “faillover” and hi-jack the first ethernet port to try and run IPMI in tandem with whatever that first ethernet port is doing. Well, what my port was doing was running the WAN connection for the whole house. I set this setting over to “dedicated” and ta-da! My house now has had working internet for a whole 24 hours!
My guess is that pfsense is simply not compatible for running IPMI and WAN on the same port so it kind of soft fails when it hits some bandwidth condition. So kids, the moral of the story is to outright disable IPMI or atleast set it to dedicated mode when you’re running pfsense on that box.