Strange network thing

[Stuart Biggerstaff]

Answering Lonni's questions, both the original IP and the new one are
fixed, and if I ping this IP with the system down or the network
interface down, no response.

As the NIC comes up fine when it's the only thing on the network, I have
to assume that it is finding something that tells it the IP is taken.
If that was the local ARP cache (for instance), that wouldn't make a
difference.  Querying ARP on the other hosts in the subnet, and on the
firewall appliance, I don't find anything--same for looking in the
configuration on the switch it plugs into.

Actually, when I changed the IP, I changed the gateway address but
probably not the actual gateway.  The network port was moved into a
different VLAN.  But the problem started not with that but when I
disabled the unused second NIC in the OS.  When I did that, the NIC that
came up with the active configuration was in fact the one originally
designated the second one.  That meant the IP was on a different MAC
address, and appparently something it is querying when it tries to load
"remembers" the old MAC the IP was associated with.

What I've been looking for is a diagnostic that would identify where
it's happening.


Stuart Biggerstaff
Systems Technician
Linda Hall Library of Science Engineering & Technology
5109 Cherry St.
Kansas City, Missouri 64110-2498

Phone:	(816) 926-8748
	(800) 662-1545 x748
FAX:	(816) 926-8790
URL:	www.lindahall.org 
-----Original Message-----
From: linux-users-bounces at linux-sxs.org
[mailto:linux-users-bounces at linux-sxs.org] On Behalf Of Kurt Wall
Sent: Thursday, March 06, 2008 8:18 PM
To: Linux tips and tricks
Subject: Re: Strange network thing

On Thu, Mar 06, 2008 at 06:10:08PM -0600, Stuart Biggerstaff wrote:
> We just moved a server from our internal network to our DMZ, and 
> assigned it an appropriate new fixed IP.  Now, when eth0 tries to 
> start, it shows the message "Error, some other host already uses 
> address xxx.xxx.xxx.xxx," and the interface refuses to start.

First thing I'd do is look at the ARP tables. Somebody, somewhere,
thinks the problem child's IP address is xxx.xxx.xxx.xxx when it's
really yyy.yyy.yyy.yyy (or oughta be). Judicious use of "arp" here
should prove fruitful to track this down. For that matter, tcpdump
should be able to tell you who's sending out packets with that address.

> We can start it on a separate switch and physically move the cable 
> back, and it works, and other hosts in the subnet can ping it but show

> nothing live at that IP when it's down.  To me that says there really 
> isn't another host with the IP, but the problem is still external, or 
> I'd have it on the dummy network.

You changed the host's IP address. Does it still talk to the old
gateway?
Are the DNS updates visible network-wide? That is, there's not a TTL
delaying an update? 

> I've had an issue with boxes with two NICs suddenly swapping them when

> upgrading the OS or even occasionally on kernel updates, so I disabled

> the second NIC in the configuration, and apparently caused it to do it

> preemptively.  So just after moving it, we ended up using a different 
> MAC address, and apparently something in the network infrastructure 
> has the association between that address and the IP cached, and isn't 
> letting it go.  My first guesses would be the local switch and the 
> firewall appliance, but I've confirmed both to show the current 
> configuration.

The OS fiddling with the MAC address isn't out of the question, but I'm
much more suspicious that one of the switches involved has stale data.
And rebooting the switch in question might not solve the problem if the
switch's config hasn't been saved. That is, if you don't specifically
write a switch config out before you reboot the switch, it boots and
loads the last-saved config.

In summary, use "arp" and/or "tcpdump" to find out who is using the old
IP address, then make sure the switches involved have the right configs.

Kurt
--
Why do seagulls live near the sea?  'Cause if they lived near the bay,
they'd be called baygulls.
_______________________________________________
Linux-users mailing list ( Linux-users at linux-sxs.org )
Unsub/Password/Etc: 
http://linux-sxs.org/mailman/listinfo/linux-users

Need to chat further on this subject? Check out #linux-users on
irc.linux-sxs.org !