Cannot ping other servers

Hi,

I have an IBM Bladecenter HS21 with 3 blade servers. All 3 blade
servers were initially installed with RedHat Linux 4. I then installed
ixMOS on one of the blade servers, and Mandriva Linux was also
installed (over RedHat) automatically along with ixMOS.

Now, I have some network problems. The other 2 servers with RedHat can
ping each other, but it appears that the server with ixMOS/Mandriva
Linux is isolated, that is, I can't ping the other 2 servers from it,
and I can't ping it from the other 2 servers.

Does anyone have any ideas on what is wrong and how I can solve this
problem? One more thing, my installation of Mandriva Linux has no GUI,
so I have to do everything using the command line interface.

Thank you.

On 2009-01-12, (E-Mail Removed) <(E-Mail Removed)> wrote:
>
>
> Hi,
>
> I have an IBM Bladecenter HS21 with 3 blade servers. All 3 blade
> servers were initially installed with RedHat Linux 4. I then installed
> ixMOS on one of the blade servers, and Mandriva Linux was also
> installed (over RedHat) automatically along with ixMOS.
>
> Now, I have some network problems. The other 2 servers with RedHat can
> ping each other, but it appears that the server with ixMOS/Mandriva
> Linux is isolated, that is, I can't ping the other 2 servers from it,
> and I can't ping it from the other 2 servers.
>
> Does anyone have any ideas on what is wrong and how I can solve this
> problem? One more thing, my installation of Mandriva Linux has no GUI,
> so I have to do everything using the command line interface.
>
> Thank you.

Could be a defective network interface. Try booting the server with a
live CD of another distro or installing a network card.


--
system-independent, adj.:
Works equally poorly on all systems.

<BIG SNIP>

*) ping localhost / 127.0.0.1 OK? TCP/IP stack OK?
*) ifconfig - link up? interface up? any collisions? any packets moved?
*) mii-tool - speed / duplex correct?
*) IPTables blocking anything? iptables -nvL look at the rules - if in
doubt flush them
*) Any activity on the NIC LED's?
*) Bad lead / port

--
Replica Watches - TRY LIDL - Cheap meds? Visit your GP

The current setup is
Mandriva server: 192.168.168.202
RedHat #1: 192.168.168.201
RedHat #2: 192.168.168.203

> *) ping localhost / 127.0.0.1 OK? TCP/IP stack OK?

Yes, I can ping both 127.0.0.1 and localhost. I can also ping its own
IP address 192.168.168.202.

> *) ifconfig - link up? interface up? any collisions? any packets moved?

The link and interface is up. I have the value 0 for all TX packets,
RX packets, collisions etc on the Mandriva server.

> *) mii-tool - speed / duplex correct?

I looked at /etc/sysconfig/network-scripts/ifcfg-eth0 of the Mandriva
server, which contains the following:

DEVICE = eth0
BOOTPROTO=static
IPADDR=192.168.168.202
NETMASK=255.255.255.0
GATEWAY=192.168.168.1
ONBOOT=yes
METRIC=5
MII_NOT_SUPPORTED=yes
USERCTL=no
DNS1=192.168.168.2
RESOLV_MODS=yes
LINK_DETECTION_DELAY=6
IPV6INIT=no
IPV6to4INIT=no

and noticed that if the line MII_NOT_SUPPORTED equals "yes", then the
server has the IP address 192.168.168.202 upon startup, but if
MII_NOT_SUPPORTED=no, then the server has no IP address upon startup,
and I have to manually assign one.

Also, when I typed "mii-tool" on the Mandriva server (with
MII_NOT_SUPPORTED=yes), I get the following:

SIOCGMIIPHY on 'eth0' failed: Operation not supported
SIOCGMIIPHY on 'eth1' failed: Operation not supported
SIOCGMIIPHY on 'eth2' failed: Resource temporarily unavailable
SIOCGMIIPHY on 'eth3' failed: Resource temporarily unavailable
no MII interfaces found

But on the 2 RedHat servers, which didn't contain the
MII_NOT_SUPPORTED lines, I get the following output:

eth0: negotiated 10baseT-HD, link ok
SIOCGMIIPHY on 'eth1' failed: Resource temporarily unavailable
SIOCGMIIPHY on 'eth2' failed: Operation not supported
SIOCGMIIPHY on 'eth3' failed: Operation not supported

> *) IPTables blocking anything? iptables -nvL look at the rules - if in doubt flush them

There are no entries on the iptables of all 3 servers. I think this is
because I am unable to get any of the 3 servers to connect to the
network, but that's another problem.

> *) Any activity on the NIC LED's?

The Link LED on the Cisco switch module on the bladecenter is lit up,
and the RX/TX LED is blinking, as are the RX/TX LEDs of the external
switch I have: Cisco switch module on bladecenter -> external switch -
> network point

--------------

> Did you try pinging by IP address? That would imply DNS problems.

All pings I tried were by IP address.

> /sbin/route -n

I have the following output:

Mandriva (192.168.168.202):
Destination Gateway Genmask Flags Metric Ref
Use Iface
192.168.168.0 0.0.0.0 255.255.255.0 U 5 0
0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 5
0 0 eth0
0.0.0.0 192.168.168.1 0.0.0.0 UG 5
0 0 eth0

RedHat #1 (192.168.168.201):
Destination Gateway Genmask Flags Metric Ref
Use Iface
192.168.168.0 0.0.0.0 255.255.255.0 U 0 0
0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0
0 0 eth0

RedHat #1 (192.168.168.203):
Destination Gateway Genmask Flags Metric Ref
Use Iface
192.168.168.0 0.0.0.0 255.255.255.0 U 0 0
0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0
0 0 eth0
0.0.0.0 192.168.168.1 0.0.0.0 UG 0
0 0 eth0

> grep host /etc/nsswitch.conf

(on mandriva server)
passwd: files compat
shadow: files
group: files

hosts: files nis dns
networks: files

(The following were all files
services, protocol, rpc, ethers, netmasks, netgroup, publickey,
bootparams, automount, aliases

> cat /etc/sysconfig/network

(On mandriva server)
NETWORKING=yes
GATEWAY=192.168.168.1
GATEWAYDEV=eth0
HOSTNAME=ixm1040

On Mon, 12 Jan 2009 00:31:48 -0800, (E-Mail Removed) wrote:


> Does anyone have any ideas on what is wrong and how I can solve this
> problem?

man ifconfig
man netstat

I ran the ethtool program and got the following output.

output of "ethtool -t etho" on Mandriva server

The test result is FAIL
The test extra info:
nvram test (online) 0
link test (online) 1
register test (offline) 0
memory test (offline) 0
loopback test (offline) 0
interrupt test (offline) 0

output of "ethtool -t etho" on both RedHat servers

The test result is PASS
The test extra info:
register test (offline) 0
memory test (offline) 0
loopback test (offline) 0
nvram test (online) 0
interrupt test (online) 0
link test (online) 0

output of "ethtool eth0" on Mandriva server:

Settings for eth0:
Supported ports: [ FIBRE ]
Supported link modes: 1000baseT/Half 1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 1000baseT/Half 1000baseT/Full
Advertised auto-negotiation: Yes
Speed: Unknown! (65535)
Duplex: Unknown! (255)
Port: FIBRE
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: d
Current message level: 0x000000ff (255)
Link detected: no

output of "ethtool eth0" on RedHat servers:

Settings for eth0:
Supported ports: [ FIBRE ]
Supported link modes: 1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000 Mb/s
Duplex: Full
Port: FIBRE
PHYAD: 2
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: g
Link detected: yes

Does this mean that the problem lies in the other servers not able to
detect the eth0 interface of the Mandriva server?

Thank you.

On Mon, 12 Jan 2009 21:07:36 -0800, (E-Mail Removed) wrote:


> Does this mean that the problem lies in the other servers not able to
> detect the eth0 interface of the Mandriva server?

Is that what "man ethtool" tells you?

Think about it. Where/how did you tell ethtool to look at the NIC in the
Mandriva server?

Never having used ethtool, it looks to me that all ethtool does is look a
"eth0" on the box you are running it on.

The first NIC in any box is usually etho, then if you have extras, eth1,
eth2, etc.

............................
Basically, you need to work out if your NIC hardware is seen by the OS
(lspci) and if the appropriate driver is loaded for the NIC (lsmod) and if
the NIC is configure properly (ifconfig eth0)

Caveat, this is what i do on a Debian system.
.................................................


This is why I like ifconfig
some host:somedirectory# ifconfig eth0 eth0 Link encap:Ethernet
HWaddr 00:60:67:72:93:A4
inet addr:192.168.X.Y Bcast:192.168.X.255 Mask:255.255.255.0 UP
BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX
packets:4414094 errors:0 dropped:0 overruns:0 frame:0 TX
packets:25614397 errors:0 dropped:0 overruns:7 carrier:0
collisions:0 txqueuelen:1000 RX bytes:645896469 (615.9 MiB) TX
bytes:2543905120 (2.3 GiB) Interrupt:9 Base address:

{yes, IP is fudged)

I would suggest an "lspci" (assumng pci NIC or mobo NIC) some host:some
directory# lspci
00:00.0 Host bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX Host
bridge (rev 03) 00:01.0 PCI bridge: Intel Corporation 440BX/ZX/DX -
82443BX/ZX/DX AGP bridge (rev 03) 00:07.0 ISA bridge: Intel Corporation
82371AB/EB/MB PIIX4 ISA (rev 02) 00:07.1 IDE interface: Intel Corporation
82371AB/EB/MB PIIX4 IDE (rev 01) 00:07.2 USB Controller: Intel Corporation
82371AB/EB/MB PIIX4 USB (rev 01) 00:07.3 Bridge: Intel Corporation
82371AB/EB/MB PIIX4 ACPI (rev 02) 00:09.0 Multimedia audio controller:
Rockwell International Riptide PCI Audio Controller 00:09.1 Communication
controller: Rockwell International Riptide HCF 56k PCI Modem 00:09.2 Input
device controller: Rockwell International Riptide PCI Game Controller
00:0a.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
RTL-8139/8139C/8139C+ (rev 10) 00:0c.0 VGA compatible controller: S3 Inc.
Savage 4 (rev 04) 01:00.0 VGA compatible controller: nVidia Corporation
NV4 [RIVA TNT] (rev 04) damselfly:/home/terryc#

Note 00:0a0
Then a "lsmod" and you will note that I have the 8139 module loaded for
the NIC

some host: some directoty# lsmod
Module Size Used by Not tainted af_packet
11048 0 (autoclean) nfsd 62288 8
nfs 62204 4 (autoclean) lockd 41616
1 (autoclean) [nfsd nfs] sunrpc 58528 1 (autoclean)
[nfsd nfs lockd] usb-uhci 19504 0 (unused) usbcore
52268 1 [usb-uhci] 8139too 12328 1
mii 1952 0 [8139too] crc32 2848
0 [8139too] agpgart 39108 0 (unused) dm-mirror
8548 0 (unused) dm-mod 36120 0 [dm-mirror]
ide-cd 27072 0
cdrom 26212 0 [ide-cd] rtc 5768
0 (autoclean) ext3 65388 2 (autoclean) jbd
34628 2 (autoclean) [ext3] ide-detect 288 0
(autoclean) (unused) piix 7784 1 (autoclean)
ide-disk 12448 4 (autoclean) ide-core 91832
4 (autoclean) [ide-cd ide-detect piix ide-disk] unix
12752 27 (autoclean)

On Jan 14, 4:02 am, ibupro...@painkiller.example.tld (Moe Trin) wrote:

> Look at the second lines of the '/sbin/ifconfig eth0' output on the
> Red Hat and the Mandriva systems. You stated in your original posting
> that these are IBM blades, and one would assume that they are at the
> least 'similar' to each other. Look at the 'HWaddr' value. Are they
> similar? (They will be different, but probably only in the last six
> digits of the HWaddr.)

HWaddr from ifconfig:
RedHat #1: 00:21:5E:22:03:78
RedHat #2: 00:21:5E:22:03:A2
Mandriva: 00:1A:64D:0E:C6

> Hmmm... and what do you see in /var/log/messages at boot time relating
> to the eth0 interface? Is it using the same driver module?

Looking through /var/log/messages of all 3 servers, I have (relevant
portions)

RedHat #1:
eth0: Broadcom NetXtreme II BCM5708 1000Base-SX(B2) PCI-X 64-bit
133MHz found at mem da000000, IRQ 209, node addr 00:21:5e:22:03:78
eth1: Broadcom NetXtreme II BCM5708 1000Base-SX(B2) PCI-X 64-bit
133MHz found at mem d8000000, IRQ 193, node addr 00:21:5e:22:03:7a
eth2: Tigon3[partno(BCM95704s) rev2100 PHY(serdes)] (PCIX:133MHz:64-
bit) 10/100/1000BaseT Ethernet 00:1a:64:dd:0e:a2
eth2: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] Spilt[0] WireSpeed[0]
TS0cap[1]
eth2: dma_rwctrl[769f4000]
eth3: Tigon3[partno(BCM95704s) rev2100 PHY(serdes)] (PCIX:133MHz:64-
bit) 10/100/1000BaseT Ethernet 00:1a:64:dd:0e:a3
eth3: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] Spilt[0] WireSpeed[0]
TS0cap[1]
eth3: dma_rwctrl[769f4000]
[...]
bnx2: eth0: using MSI
bnx2:eth0 NIC SerDesLink is Up, 1000Mbps full duplex

RedHat #2:
eth0: Broadcom NetXtreme II BCM5708 1000Base-SX(B2) PCI-X 64-bit
133MHz found at mem da000000, IRQ 255, node addr 00:21:5e:22:03:a2
eth1: Broadcom NetXtreme II BCM5708 1000Base-SX(B2) PCI-X 64-bit
133MHz found at mem d8000000, IRQ 193, node addr 00:21:5e:22:03:a4
eth2: Tigon3[partno(BCM95704s) rev2100 PHY(serdes)] (PCIX:133MHz:64-
bit) 10/100/1000BaseT Ethernet 00:1a:64:dd:09:a8
eth2: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] Spilt[0] WireSpeed[0]
TS0cap[1]
eth2: dma_rwctrl[769f4000]
eth3: Tigon3[partno(BCM95704s) rev2100 PHY(serdes)] (PCIX:133MHz:64-
bit) 10/100/1000BaseT Ethernet 00:1a:64:dd:09:a9
eth3: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] Spilt[0] WireSpeed[0]
TS0cap[1]
eth3: dma_rwctrl[769f4000]
[...]
bnx2: eth0: using MSI
bnx2:eth0 NIC SerDesLink is Up, 1000Mbps full duplex

Mandriva:
Broadcom NetXtreme II Gigabit Ethernet Driver bx2 v1.5.11 (June 4,
2007)
eth0: Broadcom NetXtreme II BCM5708 1000Base-SX(B2) PCI-X 64-bit
133MHz found at mem da000000, IRQ 17, node addr 00215e220366
eth1: Broadcom NetXtreme II BCM5708 1000Base-SX(B2) PCI-X 64-bit
133MHz found at mem d8000000, IRQ 19, node addr 00215e220368
eth0: Tigon3[partno(BCM95704s) rev2100 PHY(serdes)] (PCIX:133MHz:64-
bit) 1000Base-SX Ethernet 00:1a:64:dd:0e:c6
eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] WireSpeed[0] TS0cap[1]
eth0: dma_rwctrl[769f4000] dma_mask[64-bit]
eth1: Tigon3[partno(BCM95704s) rev2100 PHY(serdes)] (PCIX:133MHz:64-
bit) 1000Base-SX Ethernet 00:1a:64:dd:0e:c7
eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] WireSpeed[0] TS0cap[1]
eth1: dma_rwctrl[769f4000] dma_mask[64-bit]
[...]
ntpd[3758]: precision = 1.000 usec
ntpd[3758]: Listening on interface #0 wildcard, 0.0.0.0#123 Disabled
ntpd[3758]: Listening on interface #1 wildcard, ::#123 Disabled
ntpd[3758]: Listening on interface #2 lo, ::1#123 Enabled
ntpd[3758]: Listening on interface #3 lo, 127.0.0.1#123 Enabled
ntpd[3758]: Listening on interface #4 eth0, 192.168.168.202#123
Enabled
ntpd[3758]: Kernel time sync status 0040

So it appears that eth0 on both RedHat servers use Broadcom NetXtreme
II , and the Mandriva server used that initially, but then overwrote
that and used Tigon3 instead. The HWaddr from ifconfig confirms this.
Could this be the problem?

> On both Red Hat and Mandriva, '/sbin/lsmod' and see what modules are
> being used for the network driver.

lsmod on all 3 servers gave me a whole list of modules. The one that
looked familar was
On RedHat #1 and #2:
Module Size Used by
bnx2 181512 0

On Mandriva:
Module Size Used by
bnx2 160408 0

I also did a lspci but got a "command not found" message on the
Mandriva server.

The portions of the lspci output that I think are relevant:
RedHat #1:
03:00:0 PCI bridge: Broadcom: Unknown device 0103 (rev c3)
04:00:0 Ethernet controller: Broadcom Corporation NetXtreme II
BCM5708S Gigabit Ethernet (rev 12)
05:00:0 PCI bridge: Broadcom: Unknown device 0103 (rev c3)
06:00.0 Ethernet controller: Broadcom Corporation NetXtreme II
BCM5708S Gigabit Ethernet (rev 12)
0c:01.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704S
Gigabit Ethernet (rev 10)
0c:01.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704S
Gigabit Ethernet (rev 10)

RedHat #2:
03:00:0 PCI bridge: Broadcom: Unknown device 0103 (rev c3)
04:00:0 Ethernet controller: Broadcom Corporation NetXtreme II
BCM5708S Gigabit Ethernet (rev 12)
05:00:0 PCI bridge: Broadcom: Unknown device 0103 (rev c3)
06:00.0 Ethernet controller: Broadcom Corporation NetXtreme II
BCM5708S Gigabit Ethernet (rev 12)
08:01.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704S
Gigabit Ethernet (rev 10)
08:01.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704S
Gigabit Ethernet (rev 10)

Hello,

(E-Mail Removed) a écrit :
>
> RedHat #2:
> eth0: Broadcom NetXtreme II BCM5708 1000Base-SX(B2) PCI-X 64-bit
> 133MHz found at mem da000000, IRQ 255, node addr 00:21:5e:22:03:a2
> eth1: Broadcom NetXtreme II BCM5708 1000Base-SX(B2) PCI-X 64-bit
> 133MHz found at mem d8000000, IRQ 193, node addr 00:21:5e:22:03:a4
> eth2: Tigon3[partno(BCM95704s) rev2100 PHY(serdes)] (PCIX:133MHz:64-
> bit) 10/100/1000BaseT Ethernet 00:1a:64:dd:09:a8
> eth3: Tigon3[partno(BCM95704s) rev2100 PHY(serdes)] (PCIX:133MHz:64-
> bit) 10/100/1000BaseT Ethernet 00:1a:64:dd:09:a9
>
> Mandriva:
> eth0: Broadcom NetXtreme II BCM5708 1000Base-SX(B2) PCI-X 64-bit
> 133MHz found at mem da000000, IRQ 17, node addr 00215e220366
> eth1: Broadcom NetXtreme II BCM5708 1000Base-SX(B2) PCI-X 64-bit
> 133MHz found at mem d8000000, IRQ 19, node addr 00215e220368
> eth0: Tigon3[partno(BCM95704s) rev2100 PHY(serdes)] (PCIX:133MHz:64-
> bit) 1000Base-SX Ethernet 00:1a:64:dd:0e:c6
> eth1: Tigon3[partno(BCM95704s) rev2100 PHY(serdes)] (PCIX:133MHz:64-
> bit) 1000Base-SX Ethernet 00:1a:64:dd:0e:c7
> [...]
> So it appears that eth0 on both RedHat servers use Broadcom NetXtreme
> II , and the Mandriva server used that initially, but then overwrote
> that and used Tigon3 instead. The HWaddr from ifconfig confirms this.
> Could this be the problem?

Probably. So the first BCM5708 in Mandriva is now eth2 or any other
name. This is commonly caused by udev. When a new network interface is
discovered and udev is active, udev creates a persistent naming rule to
it. If for whatever reason during the Mandriva install process the
95704s interfaces were discovered first, they got the names eth0 and et1
and udev made these names persistent. You can look for a file whose name
contains "persistent-net.rules" in /etc/udev/rules.d/.

Your options are :
- use the actual interface names ("ifconfig -a" will show names and MAC
addresses) ;
- or edit the udev persistent naming rules so each interface name is
associated to each physical interface (by MAC addresses) the same way as
in RedHat.

Thank you so much, Pascal and Moe! That was the problem and after
editing the persistent rules, I finally got the servers to ping each
other!