performance slowness (variable latency) in network

Hi

I am experiencing some slowness in the network performance on a computer
I have. Its a thinkpad T60, with intel core duo and Intel 82573L Gigabit
controller with linux 2.6.16.9 compiled with smp.

The slowness is f.ex. visible by running a ping request from the t60,
where every second packet has a round trip time of ca. 1000 ms while the
others have approx 20-40 ms. The problem is also evident if I run ping
from another computer on the same network with the t60 as the target.
So the problem seems to be with my t60's network subsystem somewhere.

The only problem I can think of is some sort network loop and timeout
occurence on the local machine, where f.ex. the local resolv.conf is not
properly set up so that the network system has to wait for timeout or
similar to continue with the next search step when resolving a name.

Does anybody have any suggestions on what the problem could or where to
look for "bottlenecks" on the local machine with regards to network
performance slowness?


regards

tom

> The only problem I can think of is some sort network loop and timeout
> occurence on the local machine, where f.ex. the local resolv.conf is not
> properly set up so that the network system has to wait for timeout or
> similar to continue with the next search step when resolving a name.
>
> Does anybody have any suggestions on what the problem could or where to
> look for "bottlenecks" on the local machine with regards to network
> performance slowness?

I can see a messed up resolv.conf getting in your way when you are
running ping on the T60. Use ping -n.

When the machine is the target of a ping, it shouldn't be doing any sort
of name resolution. An ARP going in the ditch could cause problems, but
regular alternation such as you seem to be observing is strange.

Other things to do:
- Look at the error count for the NIC.
- Look at logs for error messages.
- What distribution have you installed? Have you tried using one of
their stock kernels? Have you tried the latest? (I know someone who
had problems with FC5 on a T60, but I don't remember networking being an
issue.
- Try booting a distro-on-a-disk. See if that works.
- Try another distro-on-a-disk :-)
- Run tcpdump or ethereal and see what the timing looks like. You need
to have name resolution off. With tcpdump, that's "tcpdump -n". With
ethereal, it may be off by default.

Allen McIntosh wrote:
> Other things to do:
> - Look at the error count for the NIC.

There are no errors or dropped packets in the iconfig output.

> - Look at logs for error messages.

Nothing in /var/log/messages.
Is there a way to turn on increase log verbosity level for network
operations in the kernel?

> - What distribution have you installed? Have you tried using one of
> their stock kernels? Have you tried the latest? (I know someone who
> had problems with FC5 on a T60, but I don't remember networking being an
> issue.

Mandriva 2006.0. I have tried all their stock and devel kernels, but
they are all pre 2.6.16 version and that means the network is not
working properly, some driver/nic detection problem. I have made my own
2.6.16.9 kernel, which I hope is configured correctly, with the smp
feature (core duo processor) I also tried frequency and low latency
configs of different sorts and without it as well.

> - Try booting a distro-on-a-disk. See if that works.

Ubuntu Flight 6 is supposed to work nicely on t60. I haven't had a
chance to try it yet.

> other stuff

So on to what new info I find out, and there are lots of small pieces
that sort of confuses me so I have to try and make some sence of it all
in the end,

(FYI: my problem-machine and machine1 is running nearly identical
installations of mandriva, both have been rpm updated. problem-machine
is a core duo (dont know if that is relevant) and running 2.6.16.9, the
other is running stock kernel 2.6.4. A third machine is running older
mandrake)

- I have tested several network apps and nearly all of them are
experiencing variable network performance, except when I do f.ex file
transfers in firefox and wget (both ftp and http), then it runs smoothly.
- using "ping -n" seems to have some effect, by also using -i I can
verify that the latency is more or less constant.
- ping gives out some strange numbers. with "ping" its reveals the
variable latency, with "ping -n [-i x.x]" where -i is e.g. 0.1, 0.3, 3,
ping prints the entire time respectively 100ms, 300ms 3000ms instead of
the actual round trip time. On other machines it prints the actual round
trip time only.

ping results:
- ethereal on m1 shows match between ethereal times and ping times ->
latency is on problem machine not other hosts or network devices.
- when pinging the problem machine from the other machines on the local
network, latency is high and variable, all though using -i cuts the
round trip time the variable latency is proportionally smaller.
- when pinging machines on the local network from problem-machine,
latency is low.
- when pinging internet hosts with the problem machine latency is high
and variable.
- when pinging internet hosts with other machines on the local net, the
latency is low as expected.

So there is some inconsistency here which I cant seem to get my head
around. The problem might be network card driver having some sort of bug
under certain conditions, but this I am not sure of yet.

I can attach some ethereal and ping results if need be but then it has
to be decided exactly which measurement is the best one. I have tried
many different types and they show results in different directions. My
suggestion is that it is the output part of the network that is lagging
but I am not sure.

Any ideas would be greatly appreciated.

/tom

Tom Fredriksen <(E-Mail Removed)> wrote:
> I am experiencing some slowness in the network performance on a
> computer I have. Its a thinkpad T60, with intel core duo and Intel
> 82573L Gigabit controller with linux 2.6.16.9 compiled with smp.

> The slowness is f.ex. visible by running a ping request from the
> t60, where every second packet has a round trip time of ca. 1000 ms
> while the others have approx 20-40 ms. The problem is also evident
> if I run ping from another computer on the same network with the t60
> as the target. So the problem seems to be with my t60's network
> subsystem somewhere.

Could it be some interrupt coalescing run amok? You might try setting
InterruptThrottleRate=0 when doing a modprobe of the e1000 driver.

> The only problem I can think of is some sort network loop and
> timeout occurence on the local machine, where f.ex. the local
> resolv.conf is not properly set up so that the network system has to
> wait for timeout or similar to continue with the next search step
> when resolving a name.

I would have thought that after the initial startup, ping would not be
making any getaddrinfo() et al calls. You could check that by taking
a system call trace of ping() and see if it is doing any name
resolution once it actually starts sending requests and receiving
responses.

rick jones
--
The glass is neither half-empty nor half-full. The glass has a leak.
The real question is "Can it be patched?"
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...