[SOLVED] Bizarre Name Resolution/Routing Problem

[Kurt Wall]

As detailed in
http://mail.linux-sxs.org/pipermail/linux-users/2004-December/060479.html
(or http://tinyurl.com/5aqv6 for the URL-challenged), we found the culprit
of our name resolution/routing problems. I suspected a hardware problem on
the Firebox firewall, and I wasn't too far off. As it turned out, the
configuration off which the Firebox was running was quite broken. It
wouldn't load using the GUI config tool, which validates the config before
pushing it out. The runtime config had been hand-crafted by the previous
admin and tweaked via the serial console, which *doesn't* validate the
config. As a result of the broken configuration, incoming traffic to
the web site were getting to the right external address/interface, but
the NAT table would send packets to (at least) two different internal
addresses, one of which worked and one of which didn't. There was no
pattern that we could see to which internal interface got picked.
Meanwhile, the ARP cache would get trashed, which, understandably, 
destroyed address resolution. Clearing the cache helped, but only for
a little while.

We loaded a brand new config, had it blessed by the support guys at
Watchguard, and have been up without interruption for the last 23 hours.
Moral of the story? Don't bypass idiot checks and get a second, uninvolved
pair of eyes to look at this sort of thing.

Kurt
-- 
Atlee is a very modest man.  And with reason.
	-- Winston Churchill