Strange network thing

[Kurt Wall]

On Thu, Mar 06, 2008 at 06:10:08PM -0600, Stuart Biggerstaff wrote:
> We just moved a server from our internal network to our DMZ, and
> assigned it an appropriate new fixed IP.  Now, when eth0 tries to start,
> it shows the message "Error, some other host already uses address
> xxx.xxx.xxx.xxx," and the interface refuses to start.

First thing I'd do is look at the ARP tables. Somebody, somewhere,
thinks the problem child's IP address is xxx.xxx.xxx.xxx when it's really 
yyy.yyy.yyy.yyy (or oughta be). Judicious use of "arp" here should prove
fruitful to track this down. For that matter, tcpdump should be able to
tell you who's sending out packets with that address.

> We can start it on a separate switch and physically move the cable back,
> and it works, and other hosts in the subnet can ping it but show nothing
> live at that IP when it's down.  To me that says there really isn't
> another host with the IP, but the problem is still external, or I'd have
> it on the dummy network.

You changed the host's IP address. Does it still talk to the old gateway?
Are the DNS updates visible network-wide? That is, there's not a TTL delaying
an update? 

> I've had an issue with boxes with two NICs suddenly swapping them when
> upgrading the OS or even occasionally on kernel updates, so I disabled
> the second NIC in the configuration, and apparently caused it to do it
> preemptively.  So just after moving it, we ended up using a different
> MAC address, and apparently something in the network infrastructure has
> the association between that address and the IP cached, and isn't
> letting it go.  My first guesses would be the local switch and the
> firewall appliance, but I've confirmed both to show the current
> configuration.

The OS fiddling with the MAC address isn't out of the question, but I'm much
more suspicious that one of the switches involved has stale data. And rebooting
the switch in question might not solve the problem if the switch's config
hasn't been saved. That is, if you don't specifically  write a switch config
out before you reboot the switch, it boots and loads the last-saved config.

In summary, use "arp" and/or "tcpdump" to find out who is using the old IP
address, then make sure the switches involved have the right configs.

Kurt
-- 
Why do seagulls live near the sea?  'Cause if they lived near the bay,
they'd be called baygulls.