poor performance over gigabit lan

hi everyone.

i've installed a gigabit lan a few weeks ago and even bought cat6 cables but i will get only 9-13MByte/s real data transfer over my network. my cpu utilization ranges from 44-99% depending on the machine, that is sending the data. the system configuration looks as follows:


pc1:

Linux eNTi 2.6.6-rc2-love1 #3 Fri Apr 30 16:25:02 CEST 2004 i686 AMD Athlon(tm) XP 2000+ AuthenticAMD GNU/Linux
Epox 8KHAL (VIA KT266A)
786MB Infineon DDR
Segate 160GB 7200rpm SATA UDMA6 (on PCI Promise Controller SATA150 TX2plus)


pc2:

Linux hydralisk 2.6.6-rc2-mm1 #1 Sat May 1 01:17:07 CEST 2004 i686 AMD Athlon(tm) Processor AuthenticAMD GNU/Linux
Asus A7V (VIA Apollo KT133)
512MB SDRAM
IBM 80GB 7200rpm UDMA5


network hardware (same on both machines):

Netgear GA302T NICs
Netgear GS105 Gigabit Switch
CAT6 cable


i now would like to know, where i should look for a possible bottleneck and what i can possibly do about it.
--
eNTi
--------------------------------------------
"In mathematics you don't understand things,
you just get used to them."

-- Johann von Neumann

Hi,

> i've installed a gigabit lan a few weeks ago and even bought cat6 cables but i will get only 9-13MByte/s real data transfer

How did you measure that, e.g. did you use FTP connection or did you sen files
via Samba, NFS or ????

> i now would like to know, where i should look for a possible bottleneck and what i can possibly do about it.

Anyways, modern Harddisks (like the 160 GB in Box 1) will be able to serve more
than 13 MB/s, but only if you transfer large files on the fly.
Have you enabled (U)DMA-Modes for your HDDs (ok, to SATA this might not apply)?

HDDs are almost always the bottleneck. Let's say your HDD could serve 20MB/s,
so there is (depending on your transfer protocol) some overhead by the server
daemon/protocol and network transport........13MB/s do not sound that bad
for non-RAID HDD systems, but it might not be the best you can get.

You say your CPU load is 55% to 99%?
Sounds a bit much for an AMD 2000+ serving only 13 MB/s.
Maybe your NIC driver module is poor or set up wrong.
But there might be another problem with duplex settings.....
auto-nego might not work correctly (although it should).
Try ifconfig and look for errors/collisions and so on....

Please provide more information....

transfer protocol?
UDMA setting of HDDs?
ifconfig output for your NICs?
mii-tool output for your NICs?


CU

Ralf

Also, MTU size and buffer size need to be looked at.

P. Danek

Ralf Herrmann wrote:

> Hi,
>
>> i've installed a gigabit lan a few weeks ago and even bought cat6
>> cables but i will get only 9-13MByte/s real data transfer
>
>
> How did you measure that, e.g. did you use FTP connection or did you sen
> files
> via Samba, NFS or ????
>
>> i now would like to know, where i should look for a possible
>> bottleneck and what i can possibly do about it.
>
>
> Anyways, modern Harddisks (like the 160 GB in Box 1) will be able to
> serve more
> than 13 MB/s, but only if you transfer large files on the fly.
> Have you enabled (U)DMA-Modes for your HDDs (ok, to SATA this might not
> apply)?
>
> HDDs are almost always the bottleneck. Let's say your HDD could serve
> 20MB/s,
> so there is (depending on your transfer protocol) some overhead by the
> server
> daemon/protocol and network transport........13MB/s do not sound that bad
> for non-RAID HDD systems, but it might not be the best you can get.
>
> You say your CPU load is 55% to 99%?
> Sounds a bit much for an AMD 2000+ serving only 13 MB/s.
> Maybe your NIC driver module is poor or set up wrong.
> But there might be another problem with duplex settings.....
> auto-nego might not work correctly (although it should).
> Try ifconfig and look for errors/collisions and so on....
>
> Please provide more information....
>
> transfer protocol?
> UDMA setting of HDDs?
> ifconfig output for your NICs?
> mii-tool output for your NICs?
>
>
> CU
>
> Ralf

eNTi <nt-@gmx.de> wrote in message news:<20040505005645.22493bd5.nt-@gmx.de>...
> hi everyone.
>
> i've installed a gigabit lan a few weeks ago and even bought cat6 cables but i will get only 9-13MByte/s real data transfer over my network. my cpu utilization ranges from 44-99% depending on the machine, that is sending the data. the system configuration looks as follows:
>
> pc1:
>
> Linux eNTi 2.6.6-rc2-love1 #3 Fri Apr 30 16:25:02 CEST 2004 i686 AMD Athlon(tm) XP >2000+ AuthenticAMD GNU/Linux
> Epox 8KHAL (VIA KT266A)
> 786MB Infineon DDR
> Segate 160GB 7200rpm SATA UDMA6 (on PCI Promise Controller SATA150 >TX2plus)
>
> pc2:
>
> Linux hydralisk 2.6.6-rc2-mm1 #1 Sat May 1 01:17:07 CEST 2004 i686 AMD Athlon(tm) >Processor AuthenticAMD GNU/Linux
> Asus A7V (VIA Apollo KT133)
> 512MB SDRAM
> IBM 80GB 7200rpm UDMA5
>
> network hardware (same on both machines):
>
> Netgear GA302T NICs
> Netgear GS105 Gigabit Switch
> CAT6 cable
>
> i now would like to know, where i should look for a possible bottleneck and what i can >possibly do about it.

Don't put too much trust in off-the-cuff network numbers till you're
familiar with the hosts' internal capabilities.

That said, I agree with Ralf that the cpu usage is high for a Linux
box -- it's more what you would expect on a Windows machine. The
increased cpu usage is almost always a sign of increased interrupt
processing -- something the nic processor and the I/O subsystem should
be handling better than this.

Check Ralf's suggested commands and also :
$ netstat -spc tcp
for more configuration/hardware related issues.

Most likely, you'll have to do some tuning to get decent performance.
Look here for a quick rundown of the most important changes:
http://www.psc.edu/networking/perf_tune.html

hth,
prg
email above disabled

On Wed, 05 May 2004 15:57:13 -0400
danek <(E-Mail Removed)> wrote:

> Also, MTU size and buffer size need to be looked at.

how do i find out about those two?

> >> i've installed a gigabit lan a few weeks ago and even bought cat6
> >> cables but i will get only 9-13MByte/s real data transfer
> >
> >
> > How did you measure that, e.g. did you use FTP connection or did you sen
> > files
> > via Samba, NFS or ????

scp tells you the throughput, also i watch my gkrellm network meter. i almost ever use nfs or scp.

> > Anyways, modern Harddisks (like the 160 GB in Box 1) will be able to
> > serve more
> > than 13 MB/s, but only if you transfer large files on the fly.
> > Have you enabled (U)DMA-Modes for your HDDs (ok, to SATA this might not
> > apply)?

i worte them to the hard disk, but maybe you misinterpreted that as the maximum. sata drive on box 1 runs at UDMA6 and pata on box2 at UDMA5.

> > HDDs are almost always the bottleneck. Let's say your HDD could serve
> > 20MB/s,
> > so there is (depending on your transfer protocol) some overhead by the
> > server
> > daemon/protocol and network transport........13MB/s do not sound that bad
> > for non-RAID HDD systems, but it might not be the best you can get.

i'll include hdparm -tT readings, as well as netperf benchmark tests:

netperf: BOX1 -> BOX2:
***************************
netperf -H hydra -f M -c -C
TCP STREAM TEST to hydra
Recv Send Send Utilization Service Demand
Socket Socket Message Elapsed Send Recv Send Recv
Size Size Size Time Throughput local remote local remote
bytes bytes bytes secs. MBytes /s % T % T us/KB us/KB

87380 16384 16384 10.00 29.54 92.80 29.11 30.680 9.623


netperf: BOX2 -> BOX1:
***************************
netperf -H enti -f M -c -C
TCP STREAM TEST to enti
Recv Send Send Utilization Service Demand
Socket Socket Message Elapsed Send Recv Send Recv
Size Size Size Time Throughput local remote local remote
bytes bytes bytes secs. MBytes /s % T % T us/KB us/KB

87380 16384 16384 10.00 32.56 68.48 19.60 20.538 5.877


hdparm: BOX1:
*****************
root@eNTi $ hdparm -tT /dev/sda

/dev/sda:
Timing buffer-cache reads: 1032 MB in 2.00 seconds = 514.79 MB/sec
Timing buffered disk reads: 100 MB in 3.02 seconds = 33.10 MB/sec


hdparm: BOX2:
*****************
root@hydralisk $ hdparm -tT /dev/hde

/dev/hde:
Timing buffer-cache reads: 540 MB in 2.01 seconds = 268.03 MB/sec
Timing buffered disk reads: 138 MB in 3.02 seconds = 45.63 MB/sec


strangly the "supposed-to-be-faster" sata disk brings a lot worse bufferd reads.


ifconfig BOX1:
****************
root@eNTi $ ifconfig
eth0 Link encap:Ethernet HWaddr 00:09:5B:62:1E:F8
inet addr:192.168.0.2 Bcast:192.168.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3871859 errors:0 dropped:0 overruns:0 frame:0
TX packets:2868305 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:76626653 (73.0 Mb) TX bytes:673987799 (642.7 Mb)
Interrupt:7


ifconfig BOX2:
****************
root@hydralisk $ ifconfig
eth0 Link encap:Ethernet HWaddr 00:09:5B:62:1E3
inet addr:192.168.0.3 Bcast:192.168.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:16662976 errors:0 dropped:0 overruns:0 frame:0
TX packets:18611361 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3691867954 (3520.8 Mb) TX bytes:2477305059 (2362.5 Mb)
Interrupt:3


mii-tool BOX1:
****************
root@eNTi $ mii-tool
eth0: negotiated 100baseTx-FD flow-control, link ok


mii-diag BOX2:
*****************
root@eNTi $ mii-diag
Using the default interface 'eth0'.
Basic registers of MII PHY #1: 1000 796d 0020 6110 05e1 cde1 000d 2001.
The autonegotiated capability is 01e0.
The autonegotiated media type is 100baseTx-FD.
Basic mode control register 0x1000: Auto-negotiation enabled.
You have link beat, and everything is working OK.
Your link partner advertised cde1: Flow-control 100baseTx-FD 100baseTx 10baseT-FD 10baseT, w/ 802.3X flow control.
End of basic transceiver information.


mii-tool BOX2:
****************
root@hydralisk $ mii-tool
eth0: negotiated 100baseTx-FD flow-control, link ok


mii-diag BOX2:
*****************
root@hydralisk $ mii-diag
Using the default interface 'eth0'.
Basic registers of MII PHY #1: 1000 796d 0020 6110 05e1 cde1 000d 2001.
The autonegotiated capability is 01e0.
The autonegotiated media type is 100baseTx-FD.
Basic mode control register 0x1000: Auto-negotiation enabled.
You have link beat, and everything is working OK.
Your link partner advertised cde1: Flow-control 100baseTx-FD 100baseTx 10baseT-FD 10baseT, w/ 802.3X flow control.
End of basic transceiver information.


thx for your help!
---
eNTi
--------------------------------------------
"In mathematics you don't understand things,
you just get used to them."

-- Johann von Neumann

eNTi <nt-@gmx.de> wrote in message news:<20040507194720.6cba7a12.nt-@gmx.de>...
> On Wed, 05 May 2004 15:57:13 -0400
> danek <(E-Mail Removed)> wrote:
>
> > Also, MTU size and buffer size need to be looked at.
>
> how do i find out about those two?
>
[snip]
> eNTi

These are all available when you cat/echo a file in:
/proc/sys/net/ipv4

Check out the lartc howto:
http://lartc.org/howto/lartc.kernel.html

You should have something, somewhere on disk similar to this:
/usr/src/linux-2.x.x/Documentation/networking/ip-sysctl.txt

My previous post has some other settings you may want to investigate.

hth,
prg
email above disabled

Hi again,

> scp tells you the throughput, also i watch my gkrellm network meter. i almost ever use nfs or scp.

Well scp has some reasonable overhead. Try to measure the speed with ftp.
It gives you a glue, if your hard discs will be able to deliver data
at a higher rate or if theay are limiting things.

Ok, the step from scp to ftp is not gaint, but ftp usually tends to
saturate the NICs capabilites, as long as the discs are fast enough.

> i worte them to the hard disk, but maybe you misinterpreted that as the maximum. sata drive on box 1 runs at UDMA6 and pata on box2 at UDMA5.

This should be ok.

>
> i'll include hdparm -tT readings, as well as netperf benchmark tests:
>
> netperf: BOX1 -> BOX2:
> ***************************
> netperf -H hydra -f M -c -C
> TCP STREAM TEST to hydra
> Recv Send Send Utilization Service Demand
> Socket Socket Message Elapsed Send Recv Send Recv
> Size Size Size Time Throughput local remote local remote
> bytes bytes bytes secs. MBytes /s % T % T us/KB us/KB
>
> 87380 16384 16384 10.00 29.54 92.80 29.11 30.680 9.623


> 87380 16384 16384 10.00 32.56 68.48 19.60 20.538 5.877


Hm, seems as if your ethernet link cannot provide more than about 30MB/s
(approx. <300 MBit) throughput.
I think netperf does not suffer from hard-discs, but it needs fast CPUs to
produce testing traffic......

Well, with FTP and fast HDDs, you could expect to get about 20Mb/s when
transfering large files.....


> /dev/sda:
> Timing buffer-cache reads: 1032 MB in 2.00 seconds = 514.79 MB/sec
> Timing buffered disk reads: 100 MB in 3.02 seconds = 33.10 MB/sec
>
> /dev/hde:
> Timing buffer-cache reads: 540 MB in 2.01 seconds = 268.03 MB/sec
> Timing buffered disk reads: 138 MB in 3.02 seconds = 45.63 MB/sec
>
>
> strangly the "supposed-to-be-faster" sata disk brings a lot worse bufferd reads.

Anyways, the discs seem fast enough, to serve more then 13MB/s :-)

> mii-tool BOX1:
> ****************
> root@eNTi $ mii-tool
> eth0: negotiated 100baseTx-FD flow-control, link ok

Hm, strange. If the NICs ran on 100MBit, no more than 10MB/s would be possible.
I guess your mii-tool is outdated, of the NIC's module doesn't report the real
data rate. Or did i miss something?

Or the auto-nego is really broken (which would cause slow transfers and CPU
loading, but in this case it should be much worse than 30 MB/s)



Ok, i think it's a combination of different things.
Please measure speed with FTP, that would surely tell us more.

But may i point you to the cables? CAT6 should be ok, but
signal quality depends on cable quality, shielding and last but not least
the quality of your RJ45 plugs.

If you can get a high end cable for testing, the results of netperf
might be interesting:-)

Ok, hope you find the bottleneck.

Ralf