eth0 works, but needs to ping the gateway to jumpstart it

Hello. I have a Gentoo server using the e1000 driver in a 2.6.10 kernel.
The network config information is all correct and the connection on eth0
works on boot. However, when I use the net.eth0 startup script to restart
eth0, the connection dies with no route to host.

If I start pinging the gateway, it will fail 16-40 times and then
suddenly the connection comes alive and this happens consistently.

My question is simply where does the problem likely lie? I've checked and
reconfigured the kernel several times, switched ports and even IP blocks
in the co-lo where the machine is, double-checked all the network
configuration files.

Thanks for your help! In this case, any direction will be helpful.

ak

Aaron E. Klemm wrote:
> Hello. I have a Gentoo server using the e1000 driver in a 2.6.10
kernel.
> The network config information is all correct ...

Correct or not, I cannot say. What I can say is that I have no idea
what it is. Troubleshooting network problems -- especially connection
problems -- requires config _data_, ie., the output of things like
ifconfig -a, and route -n, ascii art of connections, cat *.conf files,
etc.

> ... and the connection on eth0
> works on boot. ...

You mean the interface comes up. Big deal. How do your acquire
network config settings? DHCP? Static? Combination?

> However, when I use the net.eth0 startup script to restart
> eth0, the connection dies with no route to host.

Not familiar with gentoo -- keep meaning to correct this -- but it
sounds like you're using DHCP and you mean you are not receiving a
default route or are not connecting to DHCP server. "no route to host"
is as clear as mud.

> If I start pinging the gateway, it will fail 16-40 times and then
> suddenly the connection comes alive and this happens consistently.

What commandline are you using _exactly_ and what is the output of
ping? You have to act as our eyes by providing this. Copy-n-paste
from an X terminal.

So you are getting some routing info? You do this before using
net.eth0?

> My question is simply where does the problem likely lie?

Your ping results -- if I'm guessing at what they are correctly --
indicate that:
a) the router is flakey
b) you are not receiving arp replies unless you bang it on the head
really hard
c) you are using dns in ping instead of IP address and the dns servers
are p*ss*ng on you
d) there is a routing loop
e) something wrong/incomplete/incompatible with your config despite
what you believe
f) any of about 100 other things is a problem

> I've checked and
> reconfigured the kernel several times, switched ports and even IP
blocks
> in the co-lo where the machine is, double-checked all the network
> configuration files.

Very nice that you did all these things. So why don't it work?

We can't begin to diagnose/help without all config info and exact
command/output of your tests. If they were important enough for you to
try then share the results. You have more data that we do and if _you_
can't clear it up, how do expect us to do so with _no_ data?

We don't need to know about your "kernel config" -- just the networking
setup.

Post back with data...

prg
email above disabled

<snip>
O.k., thanks for the reply to my admittedly poor post. I will get all
the debugging information I can this afternoon...no connection to the
machine at the moment, obviously.

I mentioned the kernel because we *were* focused on it being a driver
problem with the nic...but the hardware worked on my home network
before going to the co-lo.

My knowledge of the networking on the co-lo side is shaky. It's a
static ip rather than dhcp, i know that. What other information should
I get from them?

In my network troubleshooting fantasy, I was hoping that with the
explanation of pinging the gateway to jumpstart the connection there
would be a hint what to focus on...;-)

Thanks,

Aaron

(E-Mail Removed) wrote:
> <snip>
> O.k., thanks for the reply to my admittedly poor post. I will get all
> the debugging information I can this afternoon...no connection to the
> machine at the moment, obviously.
>
> I mentioned the kernel because we *were* focused on it being a driver
> problem with the nic...but the hardware worked on my home network
> before going to the co-lo.
>
> My knowledge of the networking on the co-lo side is shaky. It's a
> static ip rather than dhcp, i know that. What other information
should
> I get from them?

Actually, it would be nice if you could ssh into the box and fetch

(These are Linux -- is your co-lo?)
ifconfig -a
route -n
or
netstat -rn

You can watch the arp cache to see if/how it changes before/after you
ping:
arp -vne
(most likely just your gw mac will appear)
In this order, ping these hosts with this switch -c10 (sends 10 pings):
a) localhost 127.0.01
b) your default GW -- normally doesn't change from lease to lease, so
use the one in an older lease if necessary
c) your DNS servers -- ditto
d) yahoo or google

You might also want to $ man arping for some arp troubleshooting.

The fact that you have a static IP does not mean that other networking
parameters are static -- they are usually dished up via DHCP. So check
your dhclient.leases file -- name/location varies with distro. My RH
box uses /var/lib/dhcp/dhclient-eth0.leases.

Ditto for resolv.conf (/etc/resolv.conf).

Who configured the box? You or them? Any customized startup scripts?
Do your network daemons load at startup? Xinet/TCPWrappers? Firewall
in place? Number of nics? Is forwarding turned on? Should it be? You
may have to start at the bottom (hardware) and check step by step,
config by config, setting by setting. Nothing planned for the weekend,
I hope

> In my network troubleshooting fantasy, I was hoping that with the
> explanation of pinging the gateway to jumpstart the connection there
> would be a hint what to focus on...;-)

Actually, when my cable ISP is beginning to have problems, I will have
to ping the router listed as my gw several times to wake it up. Know
what it's like to have such flakey connection/response. They are (or
pretend) cluelessness at the ISP's office -- I suspect
something/someone is flushing the router's arp cache (among other
possible things) and it takes a while to get re-populated with neighbor
info, etc. It's usually traced back (by me) to dns server troubles.

Any chance it's a bad cable or switch port? Is you nic properly
"synching" with the upstream connection -- ie., speed/duplex setting?
$man mii-tool and $ man ethtool
I'll keep an eye out,
prg
email above disabled

Hi,

I put pertinent information below as requested. One thing I verified is
this is actually a time problem...so just waiting a minute or two after
restarting the nic works just as well as pinging the gateway for starting
the connection. The net.eth0 startup script restarts sshd too, so that's
why I kept losing a connection to the box. With the ping -c 50 it now buys
enough time for sshd to have an ip to connect on.

Note below the my ip address is .34

Summary: the network settings are simply taking 1-2 minutes to take
effect. There's some kind of delay I can't pinpoint.


On Fri, 21 Jan 2005 22:19:29 -0800, prg wrote:

>
> (E-Mail Removed) wrote:
>> <snip>
>> O.k., thanks for the reply to my admittedly poor post. I will get all
>> the debugging information I can this afternoon...no connection to the
>> machine at the moment, obviously.
>>
>> I mentioned the kernel because we *were* focused on it being a driver
>> problem with the nic...but the hardware worked on my home network
>> before going to the co-lo.
>>
>> My knowledge of the networking on the co-lo side is shaky. It's a
>> static ip rather than dhcp, i know that. What other information
> should
>> I get from them?
>
> Actually, it would be nice if you could ssh into the box and fetch
>
> (These are Linux -- is your co-lo?)
> ifconfig -a

dummy0 Link encap:Ethernet HWaddr F2:10:6C:E3:C0:FF
BROADCAST NOARP MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

eth0 Link encap:Ethernet HWaddr 00:30:48:41:90:7E
inet addr:xx.xxx.xxx.34 Bcast:xx.xxx.xxx.39
Mask:255.255.255.248 UP BROADCAST RUNNING MULTICAST MTU:1500
Metric:1 RX packets:23542 errors:0 dropped:0 overruns:0 frame:0
TX packets:13438 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2166608 (2.0 Mb) TX bytes:2399025 (2.2 Mb) Base
address:0xa000 Memory:ec000000-ec020000

eth1 Link encap:Ethernet HWaddr 00:30:48:41:90:7F
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Base address:0xa400 Memory:ec020000-ec040000

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:20165 errors:0 dropped:0 overruns:0 frame:0
TX packets:20165 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:815124 (796.0 Kb) TX bytes:815124 (796.0 Kb)

> route -n

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
xx.xxx.xxx.32 0.0.0.0 255.255.255.248 U 0 0 0
eth0 127.0.0.0 127.0.0.1 255.0.0.0 UG 0 0
0 lo 0.0.0.0 xx.xxx.xxx.33 0.0.0.0 UG 0 0 0 eth0

> or
> netstat -rn
>
> You can watch the arp cache to see if/how it changes before/after you
> ping:
> arp -vne
> (most likely just your gw mac will appear)
> In this order, ping these hosts with this switch -c10 (sends 10 pings):
> a) localhost 127.0.01
> b) your default GW -- normally doesn't change from lease to lease, so
> use the one in an older lease if necessary
> c) your DNS servers -- ditto
> d) yahoo or google
>
> You might also want to $ man arping for some arp troubleshooting.
>
> The fact that you have a static IP does not mean that other networking
> parameters are static -- they are usually dished up via DHCP. So check
> your dhclient.leases file -- name/location varies with distro. My RH
> box uses /var/lib/dhcp/dhclient-eth0.leases.

There are no dhcp related files on this machine.

>
> Ditto for resolv.conf (/etc/resolv.conf).
>
> Who configured the box? You or them? Any customized startup scripts?

I configured it and it worked fine at home. The startup scripts are
straight Gentoo defaults.

> Do your network daemons load at startup? Xinet/TCPWrappers? Firewall
> in place? Number of nics? Is forwarding turned on? Should it be? You
> may have to start at the bottom (hardware) and check step by step,
> config by config, setting by setting. Nothing planned for the weekend,
> I hope
>

Weekend? Just this. There is no iptables stuff configured yet.

>> In my network troubleshooting fantasy, I was hoping that with the
>> explanation of pinging the gateway to jumpstart the connection there
>> would be a hint what to focus on...;-)
>
> Actually, when my cable ISP is beginning to have problems, I will have
> to ping the router listed as my gw several times to wake it up. Know
> what it's like to have such flakey connection/response. They are (or
> pretend) cluelessness at the ISP's office -- I suspect
> something/someone is flushing the router's arp cache (among other
> possible things) and it takes a while to get re-populated with neighbor
> info, etc. It's usually traced back (by me) to dns server troubles.
>
> Any chance it's a bad cable or switch port? Is you nic properly
> "synching" with the upstream connection -- ie., speed/duplex setting?

We've tried other ports and cables. The duplex stuff does seem to be
working fine.

> $man mii-tool and $ man ethtool
> I'll keep an eye out,
> prg
> email above disabled

Aaron E. Klemm wrote:
> Hi,
>
> I put pertinent information below as requested. One thing I verified
is
> this is actually a time problem...so just waiting a minute or two
after
> restarting the nic works just as well as pinging the gateway for
starting
> the connection. The net.eth0 startup script restarts sshd too, so
that's
> why I kept losing a connection to the box. With the ping -c 50 it now
buys
> enough time for sshd to have an ip to connect on.
>
> Note below the my ip address is .34
>
> Summary: the network settings are simply taking 1-2 minutes to take
> effect. There's some kind of delay I can't pinpoint.
>
[snip]
>
> dummy0 Link encap:Ethernet HWaddr F2:10:6C:E3:C0:FF
> BROADCAST NOARP MTU:1500 Metric:1
> RX packets:0 errors:0 dropped:0 overruns:0 frame:0
> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:0
> RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
>
> eth0 Link encap:Ethernet HWaddr 00:30:48:41:90:7E
> inet addr:xx.xxx.xxx.34 Bcast:xx.xxx.xxx.39
> Mask:255.255.255.248 UP BROADCAST RUNNING MULTICAST
MTU:1500
> Metric:1 RX packets:23542 errors:0 dropped:0 overruns:0
frame:0
> TX packets:13438 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:2166608 (2.0 Mb) TX bytes:2399025 (2.2 Mb) Base
> address:0xa000 Memory:ec000000-ec020000
>
> eth1 Link encap:Ethernet HWaddr 00:30:48:41:90:7F
> BROADCAST MULTICAST MTU:1500 Metric:1
> RX packets:0 errors:0 dropped:0 overruns:0 frame:0
> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
> Base address:0xa400 Memory:ec020000-ec040000
>
> lo Link encap:Local Loopback
> inet addr:127.0.0.1 Mask:255.0.0.0
> UP LOOPBACK RUNNING MTU:16436 Metric:1
> RX packets:20165 errors:0 dropped:0 overruns:0 frame:0
> TX packets:20165 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:0
> RX bytes:815124 (796.0 Kb) TX bytes:815124 (796.0 Kb)
>
> > route -n
>
> Kernel IP routing table
> Destination Gateway Genmask Flags Metric Ref
Use Iface
> xx.xxx.xxx.32 0.0.0.0 255.255.255.248 U 0 0
0
> eth0 127.0.0.0 127.0.0.1 255.0.0.0 UG 0 0

> 0 lo 0.0.0.0 xx.xxx.xxx.33 0.0.0.0 UG 0 0 0
eth0
>
[snip]

This all looks just fine if xx.xxx.xxx.32 is the _GW_ network address
and xx.xxx.xxx.33 is the GW host/router Localhost as a GW?

If it consistently takes several minutes (seconds?) to pull together
and the provider indicates that's "normal" you may be able to insert a
sleep in the startup script. But where? Aye, me lad, there be the rub
;-) /etc/conf.d/xxx ? SSH in rc.local?

I'm wondering if it's a combination of your startup and the providers
provisioning system -- authentication, dynamic table entries inserted,
messaging back to your box, info propogation, etc. They probably don't
expect you to take down your own system regularly -- you know, 24/7 and
stuff -- so that this may be a "lower" priority part of their system to
keep down a burst of maintenance traffic. How would they bring your
system up if the outage was from their end?

Would be nice if the delay were centered in one thing/area, but it is
likely a combination of things. Still, a couple of _minutes_ seems
rather excessive. If it _is_ consistent, can you live with it?

Good luck and report back any developments if you can,
prg
email above disabled