Tweaking e1000 To The MAX

I'm trying to setup a test network between just two machines. The idea is
to make the connection as fast as possible. I got two Intel Pro 1000 MT
desktop adapters in a Quad 500MHz PowerEdge and an 800MHz PIII directly
connected with a regular cable. At first the numbers would pretty low
so I got the latest driver on Intel's site and download two tools iperf
and ttcp. The results are as follows:

TOOL SERVER Mbits/sec
==========================
iperf PowerEdge 303
iperf PIII 616
ttcp PowerEdge 142
ttcp PIII 243

How to use these tools to give compariable results?

Why is throughput so different in one direction opposed to the other?

Are there standard message sizes or parameters that should be used when
taking these measurements?

Can I go faster? Can I tweak the e1000 driver to increase throughput? The
point is to push some network software to it's limits (i.e. flood
buffers), not create a link optimal for transferring files between
100 PCs.

Thanks,
Mike

--8<-- detailed output --8<--

iperf
with PowerEdge as server:

[root@ts0 root]# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 6] local 192.168.3.100 port 5001 connected with 192.168.3.15 port 32797
[ ID] Interval Transfer Bandwidth
[ 6] 0.0-10.0 sec 361 MBytes 303 Mbits/sec

with PIII as server:

[root@ts0 root]# iperf -c 192.168.3.15
------------------------------------------------------------
Client connecting to 192.168.3.15, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 5] local 192.168.3.100 port 32769 connected with 192.168.3.15 port 5001
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-10.0 sec 735 MBytes 616 Mbits/sec

ttcp
with PowerEdge as server:

[root@ts0 root]# ttcp -p2001 -r -s > /dev/null
ttcp-r: nbuf=1024, buflen=1024, port=2001
ttcp-r: socket
ttcp-r: accept
ttcp-r: 1.1user 45.1sys 1:49real 42% 0i+0d 0maxrss 0+0pf 0+0csw
ttcp-r: 2000973424 bytes processed
ttcp-r: 46.31 CPU sec = 42195.5 KB/cpu sec, 337564 Kbits/cpu sec
ttcp-r: 109.583 real sec = 17832 KB/real sec, 142656 Kbits/sec

with PIII as server:

[miallen@nano miallen]$ ttcp -p2001 -r -s > /dev/null
ttcp-r: nbuf=1024, buflen=1024, port=2001
ttcp-r: socket
ttcp-r: accept
ttcp-r: 0.4user 10.8sys 0:46real 24% 0i+0d 0maxrss 0+0pf 0+0csw
ttcp-r: 1462139904 bytes processed
ttcp-r: 11.31 CPU sec = 126249 KB/cpu sec, 1.00999e+06 Kbits/cpu sec
ttcp-r: 46.9612 real sec = 30405.3 KB/real sec, 243243 Kbits/sec

On 2004-04-15, Michael B Allen <(E-Mail Removed)> wrote:
> I'm trying to setup a test network between just two machines. The idea is
> to make the connection as fast as possible. I got two Intel Pro 1000 MT
> desktop adapters in a Quad 500MHz PowerEdge and an 800MHz PIII directly
> connected with a regular cable. At first the numbers would pretty low
> so I got the latest driver on Intel's site and download two tools iperf
> and ttcp. The results are as follows:
>
are use using custom compiled kernels on both sides? Did you compile in NAPI
support on both machines? What is the hardware in the boxes, how are the
interupts distrubuted amongst the hardware in your box.

Do you have a 64bit bus in one box and a 32bit bus in the other, what is the
cache size in the CPU's......

The Intel architechure is not good with gigabit network cards and its very
sensitive to your hardware configuration.

Regards

Alex

On Thu, 15 Apr 2004 07:42:18 -0400, Alexander Clouter wrote:
>> I'm trying to setup a test network between just two machines. The idea
>> is to make the connection as fast as possible. I got two Intel Pro 1000
>> MT desktop adapters in a Quad 500MHz PowerEdge and an [900MHz Athlon]
>> directly connected with a regular cable. At first the numbers would
>> pretty low so I got the latest driver on Intel's site and download two
>> tools iperf and ttcp. The results are as follows:
>>
> are use using custom compiled kernels on both sides? Did you compile in
> NAPI support on both machines?

Not really. I built 2.4.20 kernels from the Red Hat 7.3 .src.rpm and
I built the e1000 driver against the redhat kernel-source-2.4.20-28
package (the latest). I just rebuilt/reinstalled the e1000 driver with
-DE1000_NAPI but it did not have any effect. I've searched through the
..config and grep'd for NAPI and google'd for a while without discovering
how one enables NAPI in the kernel let alone a redhat kernel. Would you
happen to know?

> What is the hardware in the boxes, how
> Do you have a 64bit bus in one box and a 32bit bus in the other, what is
> are the interupts distrubuted amongst the hardware in your box.
> the cache size in the CPU's......

Dell PowerEdge Quad Xeon 500MHz 1MB cache
There are 64bit card slots

CPU0 CPU1 CPU2 CPU3
0: 55225 55669 57963 56337 IO-APIC-edge timer
1: 265 395 399 382 IO-APIC-edge keyboard
2: 0 0 0 0 XT-PIC cascade
8: 0 1 0 0 IO-APIC-edge rtc
14: 133873 133823 133833 133919 IO-APIC-level eth0
20: 0 5 2 0 IO-APIC-level aic7xxx
21: 5422 5422 5413 5389 IO-APIC-level aic7xxx
22: 6 4 3 2 IO-APIC-level aic7xxx
NMI: 0 0 0 0
LOC: 225037 225035 225036 225033
ERR: 0
MIS: 0

Generic VIA KT 133 single Athlon Tbird 900MHz 256K cache

CPU0
0: 9237868 XT-PIC timer
1: 7 XT-PIC keyboard
2: 0 XT-PIC cascade
3: 608141 XT-PIC serial
5: 0 XT-PIC usb-uhci, usb-uhci
8: 1 XT-PIC rtc
10: 3456217 XT-PIC eth1
11: 160557 XT-PIC aic7xxx
12: 137159 XT-PIC eth0, PS/2 Mouse
NMI: 0
ERR: 0

Thanks,
Mike