scp 3G file on 125MB/s network slow at 25MB/s why?

With a Gig nic on 2 boxes on same subnet I get only 25MB /s using scp
to transfer large movie files.

Why so low?

I should get 125MB /s from the Gig nics yes?

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

gavino wrote:
> With a Gig nic on 2 boxes on same subnet I get only 25MB /s using scp
> to transfer large movie files.

To benchmark network data transfer rate, I suggest use something like
iperf[1].

> Why so low?

Probably due to the I/O involved from storage.

> I should get 125MB /s from the Gig nics yes?

References:
[1] - http://dast.nlanr.net/Projects/Iperf/

HTH
- --
Ashish Shukla आशीष श�क�ल http://wahjava.wordpress.com/
·-- ·- ···· ·--- ·- ···- ·- ·--·-· --· -- ·- ·· ·-·· ·-·-·- -·-· --- --
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHWEseHy+EEHYuXnQRAiGwAKCewk+o0C1rJRLyjpnlND lDJWJShACgscAf
3rIykRJVhe6V+Y3a1IwazU4=
=/Gpn
-----END PGP SIGNATURE-----

On 2007-12-06, gavino <(E-Mail Removed)> wrote:
> With a Gig nic on 2 boxes on same subnet I get only 25MB /s using scp
> to transfer large movie files.
>
> Why so low?
It could be limited by:
- reading on one side
- writing on the other
- scp, which can slow things down a bit

> I should get 125MB /s from the Gig nics yes?
Round it to 100. But do you have disks that can handle even half of that ?
Basically, you're measuring the reading speed of your source disk.

And if someone starts talking about cache or buffers, on that machine, just
ignore them. No matter wxhat the cache, it has to be filled from the disk,
first. On the destination side, you can avoid HD bottleneck if your file
fits in the available ram.


Wanna find the max speed for a *file* transfer, with your hardware?
1. On the source: copy the file from 1 dir to another. That will put it in ram
(if you have enough).
2. scp it to the destination machine
3. now try the same with ftp


--
There is an art, it says, or rather, a knack to flying.
The knack lies in learning how to throw yourself at the ground and miss.
Douglas Adams

On Thu, 06 Dec 2007 10:41:18 -0800, gavino wrote:

> With a Gig nic on 2 boxes on same subnet I get only 25MB /s using scp to
> transfer large movie files.
>
> Why so low?
>
> I should get 125MB /s from the Gig nics yes?

If you are using OpenSSL, versions older than 4.7 have a rather
small window size that could be the cause of the problem. Try with the
latest 4.7 and see if that makes any difference.

> With a Gig nic on 2 boxes on same subnet I get only 25MB /s using scp
> to transfer large movie files.

> Why so low?

> I should get 125MB /s from the Gig nics yes?

Depends on many things. All what gigabit ethernet guarentees is that
you 100% will not get more than ~125MB/s. But there can be all kinds of
reasons to get something lower than that.

E.g. Most common disk drives are unable to reach 125MB/s so unless you
have a RAID array (or the file is in cache) you will not reach anywhere
near 125MB/s. E.g. "hdparm -t" tells me my 120GB 2.5" SATA drive can
read at about 39MB/s. For writes, this is often even slower.
And writing to a file (rather than straight to a device) is yet slower
depending on the file-system used (IIRC ext3 is not the hottest
performer w.r.t write speed).

Also at 1514 bytes per packets, 25MB/s implies 16Kpacket/s which is
a fairly high interrupt rate for tpyical desktop systems, so depending
on the network cards involved and their drivers and the OSes this may
also be a limitation: using jumbo frames could help (tho it comes
with its warts).
And of course, there's also the overhead imposed by the ssh protocol,
although current processors shouldn't have much trouble
encrypting/decrypting at 125MB/s.


Stefan

On Dec 6, 10:41 am, gavino <gavcom...@gmail.com> wrote:
> With a Gig nic on 2 boxes on same subnet I get only 25MB /s using scp
> to transfer large movie files.
>
> Why so low?
>
> I should get 125MB /s from the Gig nics yes?

The major variables are:

1) How fast can you read data from the disk?

2) How fast can you encrypt the data for transfer?

3) How efficiently can TCP utilize the available network bandwidth?

4) What is the available outbound bandwidth between the CPU and the
network interface (given that a lot of that I/O bandwidth is being
used to read the file from the disk)? How much data can be transferred
per interrupt, buffered on the network card, what is the maximum
sustainable interrupt rate? How much data is transferred per
interrupt? How much CPU is needed to keep up with the network transfer
and how does that affect how much CPU is available for encrypting?

5) What is the available network bandwidth?

6) What is the available inbound bandwidth between the CPU and the
network interface? How do all the factors described in 4 above affect
the receiver?

7) How fast can you decrypt the data for transfer? (Given that a lot
of CPU is being used to service interrupts, copy in data, and so on.)

8) Once you've buffered as much received data in memory as you are
able/willing to, how fast can you write the data to disk? How does all
this data being written to the disk affect the I/O bandwidth available
for the network interface?

To a first approximation, the effective speed will be the lowest of
these values. Most likely it's either the disk read rate, the disk
write rate, the available CPU to both encrypt and send or to both
decrypt and receive, or the available I/O bandwidth to both read the
disk and send to the network interface or both receive from the
network interface and write to the disk.

DS