Slow Connectivity via Gigabit

Recently I upgraded to gigabit, however the increase in speed I got was
far less than I thought. Copying a 20G tar file using scp transfers at
around 6.5 mb/sec, which while greater than the 4.5 mb/sec I used to
get using 100 mbps connection, is far less than the 15-20 mb/sec I
though I'd get. The gigabit lights are displayed correctly on both PCI
cards, as well as the switch. I put in new 3 ft cat 6 cables for both
machines. There are other computers connected to the switch which are
running at 100 mbps, including a separate 100mbps card on comp1. MTU is
set to 7200 on both machines. Does anyone have any idea how to speed it
up?

Netgear GS108 Gigabit Switch

Comp1
---------
P3 500 mhz
Fedora Core 5 2.6.17 kernel
Reiserfs
Netgear GA311 gigabit PCI card

[root@comp1]# ifconfig
eth1 Link encap:Ethernet HWaddr 00:14:6C:33:428
inet addr:192.168.1.1 Bcast:192.168.1.255
Mask:255.255.255.0
inet6 addr: fe80::216:6cff:fe35:40d8/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:7200 Metric:1
RX packets:4690518 errors:0 dropped:0 overruns:0 frame:0
TX packets:8788694 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:248804322 (237.2 MiB) TX bytes:2073137731 (1.9 GiB)
Interrupt:3 Base address:0xa800

[root@comp1]# ethtool eth1
Settings for eth1:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000033 (51)
Link detected: yes

Comp2
--------
P3 500 mhz
Fedora Core 5 2.6.17 kernel
ext3
Netgear GA311 gigabit PCI card

[root@comp2]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:14:6C:35:49:4D
inet addr:192.168.1.2 Bcast:192.168.1.255
Mask:255.255.255.0
inet6 addr: fe80::216:6cff:fe35:494d/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:7200 Metric:1
RX packets:4095533 errors:0 dropped:0 overruns:0 frame:0
TX packets:1540683 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:4246062689 (3.9 GiB) TX bytes:114723964 (109.4 MiB)
Interrupt:3 Base address:0x2400

[root@comp2]# ethtool eth0
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000033 (51)
Link detected: yes

In comp.os.linux.networking (E-Mail Removed):
> Recently I upgraded to gigabit, however the increase in speed I got was
> far less than I thought. Copying a 20G tar file using scp transfers at
> around 6.5 mb/sec, which while greater than the 4.5 mb/sec I used to
> get using 100 mbps connection, is far less than the 15-20 mb/sec I
> though I'd get. The gigabit lights are displayed correctly on both PCI
> cards, as well as the switch. I put in new 3 ft cat 6 cables for both
> machines. There are other computers connected to the switch which are
> running at 100 mbps, including a separate 100mbps card on comp1. MTU is
> set to 7200 on both machines. Does anyone have any idea how to speed it
> up?

Playing with MTU size might not be that helpful if the switch
doesn't support them?

> RX packets:4690518 errors:0 dropped:0 overruns:0 frame:0
> TX packets:8788694 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
[ stuff ]

> [root@comp2]# ethtool eth0
> Speed: 1000Mb/s
> Duplex: Full

Looks good. I'd try to rule out the switch at first, connect the
boxes directly, with GB nics you might not even need a
cross-over, many nics seem to adjust that. And see if this gives
more speed?

To rule out CPU/etc try ftp/nfs to transfer files.

Good luck

--
Michael Heiming (X-PGP-Sig > GPG-Key ID: EDD27B94)
mail: echo (E-Mail Removed) | perl -pe 'y/a-z/n-za-m/'
#bofh excuse 405: Sysadmins unavailable because they are in
a meeting talking about why they are unavailable so much.

Michael Heiming <michael+(E-Mail Removed)> writes:

>In comp.os.linux.networking (E-Mail Removed):
>> Recently I upgraded to gigabit, however the increase in speed I got was
>> far less than I thought. Copying a 20G tar file using scp transfers at
>> around 6.5 mb/sec, which while greater than the 4.5 mb/sec I used to
>> get using 100 mbps connection, is far less than the 15-20 mb/sec I
>> though I'd get. The gigabit lights are displayed correctly on both PCI
>> cards, as well as the switch. I put in new 3 ft cat 6 cables for both
>> machines. There are other computers connected to the switch which are
>> running at 100 mbps, including a separate 100mbps card on comp1. MTU is
>> set to 7200 on both machines. Does anyone have any idea how to speed it
>> up?

>Playing with MTU size might not be that helpful if the switch
>doesn't support them?

>> RX packets:4690518 errors:0 dropped:0 overruns:0 frame:0
>> TX packets:8788694 errors:0 dropped:0 overruns:0 carrier:0
>> collisions:0 txqueuelen:1000
>[ stuff ]

>> [root@comp2]# ethtool eth0
>> Speed: 1000Mb/s
>> Duplex: Full

>Looks good. I'd try to rule out the switch at first, connect the
>boxes directly, with GB nics you might not even need a
>cross-over, many nics seem to adjust that. And see if this gives
>more speed?

>To rule out CPU/etc try ftp/nfs to transfer files.


I would worry that the actual transfer by the computers themselves is the
limitation here. scp requires that all of th epackets be encrypted and
decrypted. That takes time and effort. Plus the load on the machine. Since
this cannot simply be piped out ( the data is transformed) it is going to
slow things down due to time slicing as well.

(E-Mail Removed) wrote:
> Recently I upgraded to gigabit, however the increase in speed I got was
> far less than I thought. Copying a 20G tar file using scp transfers at
> around 6.5 mb/sec, which while greater than the 4.5 mb/sec I used to
> get using 100 mbps connection, is far less than the 15-20 mb/sec I
> though I'd get. The gigabit lights are displayed correctly on both PCI
> cards, as well as the switch. I put in new 3 ft cat 6 cables for both
> machines. There are other computers connected to the switch which are
> running at 100 mbps, including a separate 100mbps card on comp1. MTU is
> set to 7200 on both machines. Does anyone have any idea how to speed it
> up?

7200 byte MTU? Where did that come from? "Proper" JumboFrame support
in GbE is a 9000 byte MTU. What happens if you keep the MTU like that
of 100BT?

FWIW, you now have experienced firsthand that changing just one
component of a multicomponent system may not have as large an effect
as the difference in the changed components themselves. File transfer
scp isn't "just" networking.

Things you need to investigate:

*) How much idle CPU did you have when transferring over 100BT? In
and of itself, gigabit ethernet does _nothing_ to make transfer any
easier on a host than 100BT. Just as 100BT did nothing in and of
itself to make transfer any easier on the host than 10BT. In
broadhandwaving terms, for "ethernet" links it takes just as many
CPU cycles to transfer a KB of data over 10BT as it does for 100BT,
or Gigabit, or even 10 Gigabit Ethernet.

Now, there can be differences in how NICs _implement_ those things
that will alter the level of effort required to transfer data, but
none of them are fundamental to the underlying "Ethernet" specs. Eg
ChecKsum Offload, Transport Segmentation Offload, non-standard MTUs.

So, if you were at 60ish percent CPU util for your 4.5 mb/s (did
you really mean to say 'B' for bytes instead of 'b' for bits?)
transfer, and the network were the bottleneck, it would stand to
reason that switching from 100BT to 1Gbit would not let you go more
than 50% faster than you did before.

*) How busy were your discs when you were transferring over 100BT?
The same sort of logic applies there.

*) What is the bandwidth-delay product of your connection? One limit
to the performance of a TCP connection (and IIRC scp uses TCP) is
the window size divided by the Round-Trip-Time (RTT) of the
connction. Going from 100BT to 1Gig does not mean the RTT will be
1/10th what it was before, so perhaps a larger TCP window is
required to achieve higher throughput over the Gigabit link. This
can be true even over a LAN.

*) What manner of I/O bus(ses) do you have in the system and what was
their level of utilization when transferring over 100BT?

If you want to see the network-level differences in isolation from the
rest of the components in the system you need to run a networking
benchmark - eg netperf - http://www.netperf.org/

--
firebug n, the idiot who tosses a lit cigarette out his car window
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Rick Jones wrote:
> (E-Mail Removed) wrote:
>> Recently I upgraded to gigabit, however the increase in speed I got was
>> far less than I thought. Copying a 20G tar file using scp transfers at
>> around 6.5 mb/sec, which while greater than the 4.5 mb/sec I used to
>> get using 100 mbps connection, is far less than the 15-20 mb/sec I
>> though I'd get. The gigabit lights are displayed correctly on both PCI
>> cards, as well as the switch. I put in new 3 ft cat 6 cables for both
>> machines. There are other computers connected to the switch which are
>> running at 100 mbps, including a separate 100mbps card on comp1. MTU is
>> set to 7200 on both machines. Does anyone have any idea how to speed it
>> up?
>
....
> Things you need to investigate:
>
> *) How much idle CPU did you have when transferring over 100BT?

Indeed. Don't forget also that scp requires CPU cycles at each end for
encryption/decryption. If you are somewhere relatively secure, try
using FTP and see if that makes a difference.

There is also the matter of window scaling. You really need it working
in order to get the full benefit of gigabit speeds. Some versions of
the kernel won't do it unless you tweak some of the TCP parameters. (I
haven't looked at recent kernels, so this may have been fixed.) I don't
think you are running into this, but you may once you remove other
things that are keeping the transfer rate down.

* Michael Heiming <michael+(E-Mail Removed)>:

> In comp.os.linux.networking (E-Mail Removed):
> > Recently I upgraded to gigabit, however the increase in speed I got was
> > far less than I thought. Copying a 20G tar file using scp transfers at
> > around 6.5 mb/sec, which while greater than the 4.5 mb/sec I used to
> > get using 100 mbps connection, is far less than the 15-20 mb/sec I
> > though I'd get. [...]
>
> Playing with MTU size might not be that helpful if the switch
> doesn't support them?

Should be fine, provided that its S/N is within the supported range:
http://kbserver.netgear.com/kb_web_files/n101539.asp

Thomas
--
=-------------------------------------------------------------------------=
- Thomas "ZlatkO" Zajic <(E-Mail Removed)> Linux-2.6.17 & slrn-0.9.8.1 -
- "It is not easy to cut through a human head with a hacksaw." (M. C.) -
=-------------------------------------------------------------------------=

* Rick Jones <(E-Mail Removed)>:

> (E-Mail Removed) wrote:
> > Recently I upgraded to gigabit, however the increase in speed I got was
> > far less than I thought. Copying a 20G tar file using scp transfers at
> > around 6.5 mb/sec, which while greater than the 4.5 mb/sec I used to
> > get using 100 mbps connection, is far less than the 15-20 mb/sec I
> > though I'd get. The gigabit lights are displayed correctly on both PCI
> > cards, as well as the switch. I put in new 3 ft cat 6 cables for both
> > machines. There are other computers connected to the switch which are
> > running at 100 mbps, including a separate 100mbps card on comp1. MTU is
> > set to 7200 on both machines. Does anyone have any idea how to speed it
> > up?
>
> 7200 byte MTU? Where did that come from? "Proper" JumboFrame support
> in GbE is a 9000 byte MTU. What happens if you keep the MTU like that
> of 100BT?

The GA311's maximum supported frame size is 7500 bytes, according to
http://kbserver.netgear.com/kb_web_files/n101539.asp, so it seems to
be a harware limitation in this case.

> [...]

Thomas
--
=-------------------------------------------------------------------------=
- Thomas "ZlatkO" Zajic <(E-Mail Removed)> Linux-2.6.17 & slrn-0.9.8.1 -
- "It is not easy to cut through a human head with a hacksaw." (M. C.) -
=-------------------------------------------------------------------------=

Thomas Zajic <(E-Mail Removed)> wrote:
> * Rick Jones <(E-Mail Removed)>:
>> 7200 byte MTU? Where did that come from? "Proper" JumboFrame
>> support in GbE is a 9000 byte MTU. What happens if you keep the
>> MTU like that of 100BT?

> The GA311's maximum supported frame size is 7500 bytes, according to
> http://kbserver.netgear.com/kb_web_files/n101539.asp, so it seems to
> be a harware limitation in this case.

This is the kind of thing that happens when something isn't made a de
jure standard. Being able to pass around 8KB of something with a bit
of header associated with it is quite useful, and I suspect the reason
the Alteon folks chose 9000 bytes.

My personal suggestion is that folks try, whenever possible, to obtain
kit that can do a full 9000 byte Jumbo Frame MTU (9014 byte "ethernet"
frame size).

rick jones
--
Wisdom Teeth are impacted, people are affected by the effects of events.
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...