Gigabit Ethernet, and Linux -- first observations

I installed a gigabit network switch and a gigabit enabled laptop wifi
adapter (with gigabit, obviously, available on ethernet ports only) in
my house.

Two computers in my home are connected to the switch and one (laptop)
to the wifi adaptor.

The highest possible speed of a gigabit connection is about 111
megabytes per second.

Naturally, I did some tests with a noncompressible 1 gigabyte long
file (fragment of some gzipped file exactly one GB long).

My first test was to scp files from one computer on the switch to
another. Here, I was disappointed as the highest speed was only 22
megabytes per second one way and 46 another way. About 20 and 40
percent of maximum.

Then I tried using HTTP to transfer the same file (both computers are
webservers). To my huge surprise, it made a world of difference and
the transfer speed was 111 or so megabytes per second.

So, now I have a dilemma, I have a fast pipe, but scp is not fast
enough (given my CPU) to encrypt/decrypt so much data.

I tried something else, which is doing wc -l on a NFS mounted drive
(same two computers). It was UNBELIEVABLY slow and the load average on
the NFS server shot WAY up. Transferring a 336 MB file took 157
seconds, or about e megabytes per second (vs 111 mbps that I achieved
with HTTP).

So, the conclusion is, HTTP is fast (no wonder), SSH is "medium",
and NFS is "slow, very bad".

The connection to laptop is a disappointment in its own right, since
even with all-ethernet connection, I get about 3 megabytes per
second. I think that I need to pull a new wire in the wall.

So, the short of it is that there is much work to be done.

i

Ignoramus26973 wrote:
> I installed a gigabit network switch and a gigabit enabled laptop wifi
> adapter (with gigabit, obviously, available on ethernet ports only) in
> my house.
>
> Two computers in my home are connected to the switch and one (laptop)
> to the wifi adaptor.
>
> The highest possible speed of a gigabit connection is about 111
> megabytes per second.
>
> Naturally, I did some tests with a noncompressible 1 gigabyte long
> file (fragment of some gzipped file exactly one GB long).
>
> My first test was to scp files from one computer on the switch to
> another. Here, I was disappointed as the highest speed was only 22
> megabytes per second one way and 46 another way. About 20 and 40
> percent of maximum.
>
> Then I tried using HTTP to transfer the same file (both computers are
> webservers). To my huge surprise, it made a world of difference and
> the transfer speed was 111 or so megabytes per second.
>
> So, now I have a dilemma, I have a fast pipe, but scp is not fast
> enough (given my CPU) to encrypt/decrypt so much data.
>
> I tried something else, which is doing wc -l on a NFS mounted drive
> (same two computers). It was UNBELIEVABLY slow and the load average on
> the NFS server shot WAY up. Transferring a 336 MB file took 157
> seconds, or about e megabytes per second (vs 111 mbps that I achieved
> with HTTP).

So, about 2.7 MB/s? :-)

>
> So, the conclusion is, HTTP is fast (no wonder), SSH is "medium",
> and NFS is "slow, very bad".

I ran into a similar problem with NFS ages back. Turns out you can
largely fix it by configuring NFS to increase the packet size. Been a
few years since I did anything with NFS, though, so you'll have to look
through the docs. I hear they use it in computer clusters, so it can't
be slow in *all* cases.

Around the same time, I also noticed that rsync can have similar speed
issues to SCP for a first copy, owing to it insisting on using SSH for RSH.

On Wed, 12 Sep 2007 01:14:07 -0400, Michael Mol wrote:

> Ignoramus26973 wrote:
>> I installed a gigabit network switch and a gigabit enabled laptop wifi
>> adapter (with gigabit, obviously, available on ethernet ports only) in
>> my house.
>>
>> Two computers in my home are connected to the switch and one (laptop)
>> to the wifi adaptor.
>>
>> The highest possible speed of a gigabit connection is about 111
>> megabytes per second.
>>
>> Naturally, I did some tests with a noncompressible 1 gigabyte long file
>> (fragment of some gzipped file exactly one GB long).
>>
>> My first test was to scp files from one computer on the switch to
>> another. Here, I was disappointed as the highest speed was only 22
>> megabytes per second one way and 46 another way. About 20 and 40
>> percent of maximum.
>>
>> Then I tried using HTTP to transfer the same file (both computers are
>> webservers). To my huge surprise, it made a world of difference and the
>> transfer speed was 111 or so megabytes per second.
>>
>> So, now I have a dilemma, I have a fast pipe, but scp is not fast
>> enough (given my CPU) to encrypt/decrypt so much data.
>>
>> I tried something else, which is doing wc -l on a NFS mounted drive
>> (same two computers). It was UNBELIEVABLY slow and the load average on
>> the NFS server shot WAY up. Transferring a 336 MB file took 157
>> seconds, or about e megabytes per second (vs 111 mbps that I achieved
>> with HTTP).
>
> So, about 2.7 MB/s? :-)
>
>
>> So, the conclusion is, HTTP is fast (no wonder), SSH is "medium", and
>> NFS is "slow, very bad".
>
> I ran into a similar problem with NFS ages back. Turns out you can
> largely fix it by configuring NFS to increase the packet size. Been a
> few years since I did anything with NFS, though, so you'll have to look
> through the docs. I hear they use it in computer clusters, so it can't
> be slow in *all* cases.

How do you configure NFS to increase the packet size? Also are you
talking about NFS V4 or V3?

On 12 Sep 2007 12:52:09 GMT, General Schvantzkoph <(E-Mail Removed)> wrote:


>On Wed, 12 Sep 2007 01:14:07 -0400, Michael Mol wrote:

>> Ignoramus26973 wrote:
>>> I installed a gigabit network switch and a gigabit enabled laptop wifi
>>> adapter (with gigabit, obviously, available on ethernet ports only) in
>>> my house.
>>>
>>> Two computers in my home are connected to the switch and one (laptop)
>>> to the wifi adaptor.
>>>
>>> The highest possible speed of a gigabit connection is about 111
>>> megabytes per second.
>>>
>>> Naturally, I did some tests with a noncompressible 1 gigabyte long file
>>> (fragment of some gzipped file exactly one GB long).
>>>
>>> My first test was to scp files from one computer on the switch to
>>> another. Here, I was disappointed as the highest speed was only 22
>>> megabytes per second one way and 46 another way. About 20 and 40
>>> percent of maximum.
>>>
>>> Then I tried using HTTP to transfer the same file (both computers are
>>> webservers). To my huge surprise, it made a world of difference and the
>>> transfer speed was 111 or so megabytes per second.
>>>
>>> So, now I have a dilemma, I have a fast pipe, but scp is not fast
>>> enough (given my CPU) to encrypt/decrypt so much data.
>>>
>>> I tried something else, which is doing wc -l on a NFS mounted drive
>>> (same two computers). It was UNBELIEVABLY slow and the load average on
>>> the NFS server shot WAY up. Transferring a 336 MB file took 157
>>> seconds, or about e megabytes per second (vs 111 mbps that I achieved
>>> with HTTP).
>>
>> So, about 2.7 MB/s? :-)
>>
>>
>>> So, the conclusion is, HTTP is fast (no wonder), SSH is "medium", and
>>> NFS is "slow, very bad".
>>
>> I ran into a similar problem with NFS ages back. Turns out you can
>> largely fix it by configuring NFS to increase the packet size. Been a
>> few years since I did anything with NFS, though, so you'll have to look
>> through the docs. I hear they use it in computer clusters, so it can't
>> be slow in *all* cases.

>How do you configure NFS to increase the packet size? Also are you
>talking about NFS V4 or V3?

when you mount it. Why didn't you just google for "nfs packet size"?

On Tue, 11 Sep 2007 23:02:37 -0500, Ignoramus26973 wrote:

> I installed a gigabit network switch and a gigabit enabled laptop wifi
> adapter (with gigabit, obviously, available on ethernet ports only) in
> my house.
>
> Two computers in my home are connected to the switch and one (laptop)
> to the wifi adaptor.
>
> The highest possible speed of a gigabit connection is about 111
> megabytes per second.
>
> Naturally, I did some tests with a noncompressible 1 gigabyte long
> file (fragment of some gzipped file exactly one GB long).
>
> My first test was to scp files from one computer on the switch to
> another. Here, I was disappointed as the highest speed was only 22
> megabytes per second one way and 46 another way. About 20 and 40
> percent of maximum.
>
> Then I tried using HTTP to transfer the same file (both computers are
> webservers). To my huge surprise, it made a world of difference and
> the transfer speed was 111 or so megabytes per second.
>
> So, now I have a dilemma, I have a fast pipe, but scp is not fast
> enough (given my CPU) to encrypt/decrypt so much data.
>
> I tried something else, which is doing wc -l on a NFS mounted drive
> (same two computers). It was UNBELIEVABLY slow and the load average on
> the NFS server shot WAY up. Transferring a 336 MB file took 157
> seconds, or about e megabytes per second (vs 111 mbps that I achieved
> with HTTP).
>
> So, the conclusion is, HTTP is fast (no wonder), SSH is "medium",
> and NFS is "slow, very bad".
>
> The connection to laptop is a disappointment in its own right, since
> even with all-ethernet connection, I get about 3 megabytes per
> second. I think that I need to pull a new wire in the wall.
>
> So, the short of it is that there is much work to be done.
>
> i

you didn't mention Jumbo Frames.

IF ( BIG IF ) your Gigabit hardware supports 9K MTUs, you can get
a big boost if you set your MTU to 9000 on your nics.

I'll post some numbers from my setup in the next couple of days.

jack


--
D.A.M. - Mothers Against Dyslexia

see http://www.jacksnodgrass.com for my contact info.

jack - Grapevine/Richardson

On Wed, 12 Sep 2007 13:32:12 GMT, Jack Snodgrass <(E-Mail Removed)> wrote:
> On Tue, 11 Sep 2007 23:02:37 -0500, Ignoramus26973 wrote:
>
>> I installed a gigabit network switch and a gigabit enabled laptop wifi
>> adapter (with gigabit, obviously, available on ethernet ports only) in
>> my house.
>>
>> Two computers in my home are connected to the switch and one (laptop)
>> to the wifi adaptor.
>>
>> The highest possible speed of a gigabit connection is about 111
>> megabytes per second.
>>
>> Naturally, I did some tests with a noncompressible 1 gigabyte long
>> file (fragment of some gzipped file exactly one GB long).
>>
>> My first test was to scp files from one computer on the switch to
>> another. Here, I was disappointed as the highest speed was only 22
>> megabytes per second one way and 46 another way. About 20 and 40
>> percent of maximum.
>>
>> Then I tried using HTTP to transfer the same file (both computers are
>> webservers). To my huge surprise, it made a world of difference and
>> the transfer speed was 111 or so megabytes per second.
>>
>> So, now I have a dilemma, I have a fast pipe, but scp is not fast
>> enough (given my CPU) to encrypt/decrypt so much data.
>>
>> I tried something else, which is doing wc -l on a NFS mounted drive
>> (same two computers). It was UNBELIEVABLY slow and the load average on
>> the NFS server shot WAY up. Transferring a 336 MB file took 157
>> seconds, or about e megabytes per second (vs 111 mbps that I achieved
>> with HTTP).
>>
>> So, the conclusion is, HTTP is fast (no wonder), SSH is "medium",
>> and NFS is "slow, very bad".
>>
>> The connection to laptop is a disappointment in its own right, since
>> even with all-ethernet connection, I get about 3 megabytes per
>> second. I think that I need to pull a new wire in the wall.
>>
>> So, the short of it is that there is much work to be done.
>>
>> i
>
> you didn't mention Jumbo Frames.
>
> IF ( BIG IF ) your Gigabit hardware supports 9K MTUs, you can get
> a big boost if you set your MTU to 9000 on your nics.

I think that it does support jumbo frames.

> I'll post some numbers from my setup in the next couple of days.

How do you set MTU? That would only work for local destinations,
right? It would not work for connections outside of my home LAN?

i

On Wed, 12 Sep 2007 13:08:06 GMT, AZ Nomad <(E-Mail Removed)> wrote:
> On 12 Sep 2007 12:52:09 GMT, General Schvantzkoph <(E-Mail Removed)> wrote:
>
>
>>On Wed, 12 Sep 2007 01:14:07 -0400, Michael Mol wrote:
>
>>> Ignoramus26973 wrote:
>>>> I installed a gigabit network switch and a gigabit enabled laptop wifi
>>>> adapter (with gigabit, obviously, available on ethernet ports only) in
>>>> my house.
>>>>
>>>> Two computers in my home are connected to the switch and one (laptop)
>>>> to the wifi adaptor.
>>>>
>>>> The highest possible speed of a gigabit connection is about 111
>>>> megabytes per second.
>>>>
>>>> Naturally, I did some tests with a noncompressible 1 gigabyte long file
>>>> (fragment of some gzipped file exactly one GB long).
>>>>
>>>> My first test was to scp files from one computer on the switch to
>>>> another. Here, I was disappointed as the highest speed was only 22
>>>> megabytes per second one way and 46 another way. About 20 and 40
>>>> percent of maximum.
>>>>
>>>> Then I tried using HTTP to transfer the same file (both computers are
>>>> webservers). To my huge surprise, it made a world of difference and the
>>>> transfer speed was 111 or so megabytes per second.
>>>>
>>>> So, now I have a dilemma, I have a fast pipe, but scp is not fast
>>>> enough (given my CPU) to encrypt/decrypt so much data.
>>>>
>>>> I tried something else, which is doing wc -l on a NFS mounted drive
>>>> (same two computers). It was UNBELIEVABLY slow and the load average on
>>>> the NFS server shot WAY up. Transferring a 336 MB file took 157
>>>> seconds, or about e megabytes per second (vs 111 mbps that I achieved
>>>> with HTTP).
>>>
>>> So, about 2.7 MB/s? :-)
>>>
>>>
>>>> So, the conclusion is, HTTP is fast (no wonder), SSH is "medium", and
>>>> NFS is "slow, very bad".
>>>
>>> I ran into a similar problem with NFS ages back. Turns out you can
>>> largely fix it by configuring NFS to increase the packet size. Been a
>>> few years since I did anything with NFS, though, so you'll have to look
>>> through the docs. I hear they use it in computer clusters, so it can't
>>> be slow in *all* cases.
>
>>How do you configure NFS to increase the packet size? Also are you
>>talking about NFS V4 or V3?
>
> when you mount it. Why didn't you just google for "nfs packet size"?

When I configure rsize and wsize, the mount fails for some reason, it
does not like these options, even if set at 4192. Client is Fedore 7,
server is Fedora Core 6.

i

Ignoramus26973 <(E-Mail Removed)> writes:
>So, now I have a dilemma, I have a fast pipe, but scp is not fast
>enough (given my CPU) to encrypt/decrypt so much data.

Using a different cipher like blowfish will speed this up a bit but http
or ftp will still be faster.

Later

Mark Hittinger
(E-Mail Removed)

On Wed, 12 Sep 2007 08:48:04 -0500, Ignoramus19897 wrote:


.... trimmed....

>>> So, the short of it is that there is much work to be done.
>>>
>>> i
>>
>> you didn't mention Jumbo Frames.
>>
>> IF ( BIG IF ) your Gigabit hardware supports 9K MTUs, you can get
>> a big boost if you set your MTU to 9000 on your nics.
>
> I think that it does support jumbo frames.
>

.... there is 'think' and 'does'.... say that you have 2 computers
connected via a switch... all 3 network devices have to support
jumbo frames. Until fairly recently, home network switches did NOT.
I don't think any of the NetGear nics support 9K jumbo frames...
some manuafactures claim jumbo frame support, but they are smaller
jumbo frames... 4.5K or 7K. I find it best when all the equipment
supports 9K Jumbo Frames. Some 'versions' of switches support
jumbo frames and other versions of the same switch do not. You really
need to look at the specs of the equipment you have to figure out if
this will work or not.


>> I'll post some numbers from my setup in the next couple of days.
>
> How do you set MTU? That would only work for local destinations,
> right? It would not work for connections outside of my home LAN?
>
ifconfig eth0 down
ifconfig eth0 mtu 9000
ifconfig eth0 up
ifconfig eth0 | grep MTU
should then show:
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1


if your 9K Jumbo Frame MTU is not working... you'll get something like
ssh remote_host
prompt# dmesg
....about 1500 bytes of output from the command
....
and then the session will hang..... you'll have to kill the session and
set the MTU back to the default ( 1500 )


jack

> i

--
D.A.M. - Mothers Against Dyslexia

see http://www.jacksnodgrass.com for my contact info.

jack - Grapevine/Richardson

On Wed, 12 Sep 2007 13:08:06 +0000, AZ Nomad wrote:

> On 12 Sep 2007 12:52:09 GMT, General Schvantzkoph
> <(E-Mail Removed)> wrote:
>
>
>>On Wed, 12 Sep 2007 01:14:07 -0400, Michael Mol wrote:
>
>>> Ignoramus26973 wrote:
>>>> I installed a gigabit network switch and a gigabit enabled laptop
>>>> wifi adapter (with gigabit, obviously, available on ethernet ports
>>>> only) in my house.
>>>>
>>>> Two computers in my home are connected to the switch and one (laptop)
>>>> to the wifi adaptor.
>>>>
>>>> The highest possible speed of a gigabit connection is about 111
>>>> megabytes per second.
>>>>
>>>> Naturally, I did some tests with a noncompressible 1 gigabyte long
>>>> file (fragment of some gzipped file exactly one GB long).
>>>>
>>>> My first test was to scp files from one computer on the switch to
>>>> another. Here, I was disappointed as the highest speed was only 22
>>>> megabytes per second one way and 46 another way. About 20 and 40
>>>> percent of maximum.
>>>>
>>>> Then I tried using HTTP to transfer the same file (both computers are
>>>> webservers). To my huge surprise, it made a world of difference and
>>>> the transfer speed was 111 or so megabytes per second.
>>>>
>>>> So, now I have a dilemma, I have a fast pipe, but scp is not fast
>>>> enough (given my CPU) to encrypt/decrypt so much data.
>>>>
>>>> I tried something else, which is doing wc -l on a NFS mounted drive
>>>> (same two computers). It was UNBELIEVABLY slow and the load average
>>>> on the NFS server shot WAY up. Transferring a 336 MB file took 157
>>>> seconds, or about e megabytes per second (vs 111 mbps that I achieved
>>>> with HTTP).
>>>
>>> So, about 2.7 MB/s? :-)
>>>
>>>
>>>> So, the conclusion is, HTTP is fast (no wonder), SSH is "medium", and
>>>> NFS is "slow, very bad".
>>>
>>> I ran into a similar problem with NFS ages back. Turns out you can
>>> largely fix it by configuring NFS to increase the packet size. Been a
>>> few years since I did anything with NFS, though, so you'll have to
>>> look through the docs. I hear they use it in computer clusters, so it
>>> can't be slow in *all* cases.
>
>>How do you configure NFS to increase the packet size? Also are you
>>talking about NFS V4 or V3?
>
> when you mount it. Why didn't you just google for "nfs packet size"?

Thanks. I was looking at the NFS server configuration in Webmin, it
didn't occur to me that it was a client parameter. I'm going to run some
experiments with different read and write buffer sizes to see how they
effect performance.