Network speed discrepancy

Hello all,
I'm puzzled by the speed difference I get when doing transfers via netperf,
dd via NFS. Can somebody shed some light on it ?

The network is a direct 1Gbps connection between a PC and an embedded
device. All the tests are run from the device. The disk is on the PC and
mounted via NFS on the device.

# netperf -H MyPC
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to master
(192.168.1.1) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
87380 16384 16384 10.01 85.73

Quite far from the theoretical 1024Mbps... (only 8% of it indeed)

The blocks I want to transfer as fast as possible are 8192 bytes in size, so
here goes:
# time dd if=/dev/zero of=toto bs=8192 count=$((100*1024*1024/8192))
12800+0 records in
12800+0 records out
real 1m 3.22s

So that's only 14Mbps, or 1.4% or nominal performance !!!

I understand that an embedded device's network card is certainly not able to
reach its full potential, but does anybody can suggest some cause as to the
speed dicrepancy between netperf and NFS ?

(note: doing my own fwrite via NFS leads to the same speed as dd)
Thanks
--
Guillaume Dargaud
http://www.gdargaud.net/

On Tue, 2011-02-15, Guillaume Dargaud wrote:
> Hello all,
> I'm puzzled by the speed difference I get when doing transfers via netperf,
> dd via NFS. Can somebody shed some light on it ?
>
> The network is a direct 1Gbps connection between a PC and an embedded
> device. All the tests are run from the device. The disk is on the PC and
> mounted via NFS on the device.
>
> # netperf -H MyPC
> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to master
> (192.168.1.1) port 0 AF_INET
> Recv Send Send
> Socket Socket Message Elapsed
> Size Size Size Time Throughput
> bytes bytes bytes secs. 10^6bits/sec
> 87380 16384 16384 10.01 85.73
>
> Quite far from the theoretical 1024Mbps... (only 8% of it indeed)
>
> The blocks I want to transfer as fast as possible are 8192 bytes in size, so
> here goes:
> # time dd if=/dev/zero of=toto bs=8192 count=$((100*1024*1024/8192))
> 12800+0 records in
> 12800+0 records out
> real 1m 3.22s

You snipped the 'user' and 'sys' figures -- what did they say?

The output from 'mpstat -P ALL 1' on both machines would also be
useful.

> So that's only 14Mbps, or 1.4% or nominal performance !!!
>
> I understand that an embedded device's network card is certainly not able to
> reach its full potential, but does anybody can suggest some cause as to the
> speed dicrepancy between netperf and NFS ?
>
> (note: doing my own fwrite via NFS leads to the same speed as dd)
> Thanks

You probably cannot write to disk at anything *near* 1Gbit/s, but
~2Mbit/s seems too low for a disk I/O bottleneck, too.

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .

Guillaume Dargaud <(E-Mail Removed)> writes:
> Quite far from the theoretical 1024Mbps... (only 8% of it indeed)
>
> The blocks I want to transfer as fast as possible are 8192 bytes in size, so
> here goes:
> # time dd if=/dev/zero of=toto bs=8192 count=$((100*1024*1024/8192))
> 12800+0 records in
> 12800+0 records out
> real 1m 3.22s
>
> So that's only 14Mbps, or 1.4% or nominal performance !!!
>
> I understand that an embedded device's network card is certainly not
> able to reach its full potential, but does anybody can suggest some
> cause as to the speed dicrepancy between netperf and NFS ?

Is the nfs mount sync or async? AIUI sync will force each write()
syscall by dd to block until it gets a positive response from the NFS
server.

--
http://www.greenend.org.uk/rjk/

Richard Kettlewell wrote:
> Is the nfs mount sync or async? AIUI sync will force each write()
> syscall by dd to block until it gets a positive response from the NFS
> server.

MyPC $ cat /etc/exports
/rootfs 192.168.1.233(rw,no_root_squash,no_all_squash,asyn c,insecure,fsid=0)

MyPC $ cat /etc/dhcpd.conf
option root-path
"192.168.1.185:/rootfs,v3,rsize=1024,wsize=1024,proto=udp,nolock,i ntr";

# mount
rootfs on / type rootfs (rw)
/dev/root on / type nfs
(rw,relatime,vers=3,rsize=1024,wsize=1024,namlen=2 55,hard,nolock,proto=udp,port=65535,timeo=11,retra ns=3,sec=sys,mountport=65535,mountproto=,addr=192. 168.1.185)
proc on /proc type proc (rw,relatime)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620)
tmpfs on /tmp type tmpfs (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)


Jorgen Grahn wrote:
> You snipped the 'user' and 'sys' figures -- what did they say?
# time dd if=/dev/zero of=toto bs=8192 count=$((100*1024*1024/8192))
12800+0 records in
12800+0 records out
real 1m 3.68s
user 0m 0.19s
sys 0m 8.35s

> The output from 'mpstat -P ALL 1' on both machines would also be
> useful.

I couldn't find mpstat on either machine, so I just did:
# top -d20
PID PPID USER STAT VSZ %MEM %CPU COMMAND
79 2 root RW 0 0% 76% [rpciod/0]
352 351 root R 1144 1% 13% dd if /dev/zero of toto bs 8192
count 12800
101 2 root SW 0 0% 5% [nfsiod]
338 2 root SW 0 0% 5% [flush-0:10]
353 240 root R 1144 1% 0% top -d20

MyPC $ top -d20
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15536 root 15 0 0 0 0 S 8.3 0.0 24:07.99 nfsd
15535 root 15 0 0 0 0 S 4.7 0.0 22:15.49 nfsd
15534 root 15 0 0 0 0 S 1.9 0.0 22:39.29 nfsd
29148 myuser 15 0 54736 35m 14m S 0.5 0.9 25:48.45 konsole
28713 myuser 15 0 10808 2416 1188 S 0.1 0.1 10:31.36 sshd


> You probably cannot write to disk at anything *near* 1Gbit/s, but
> ~2Mbit/s seems too low for a disk I/O bottleneck, too.

When running directly on the server:
MyPC $ time dd if=/dev/zero of=toto bs=8192 count=$((10000*1024*1024/8192))
1280000+0 records in
1280000+0 records out
10485760000 bytes (10 GB) copied, 93.5844 seconds, 112 MB/s
real 1m40.572s
user 0m0.289s
sys 0m17.764s

--
Guillaume Dargaud
http://www.gdargaud.net/

On Wed, 2011-02-16, Guillaume Dargaud wrote:
>
> Richard Kettlewell wrote:
>> Is the nfs mount sync or async? AIUI sync will force each write()
>> syscall by dd to block until it gets a positive response from the NFS
>> server.
>
> MyPC $ cat /etc/exports
> /rootfs 192.168.1.233(rw,no_root_squash,no_all_squash,asyn c,insecure,fsid=0)
>
> MyPC $ cat /etc/dhcpd.conf
> option root-path
> "192.168.1.185:/rootfs,v3,rsize=1024,wsize=1024,proto=udp,nolock,i ntr";
>
> # mount
> rootfs on / type rootfs (rw)
> /dev/root on / type nfs
> (rw,relatime,vers=3,rsize=1024,wsize=1024,namlen=2 55,hard,nolock,proto=udp,port=65535,timeo=11,retra ns=3,sec=sys,mountport=65535,mountproto=,addr=192. 168.1.185)
> proc on /proc type proc (rw,relatime)
> devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620)
> tmpfs on /tmp type tmpfs (rw,relatime)
> sysfs on /sys type sysfs (rw,relatime)

Richard or someone will have to comment -- I don't use nfs myself.

> Jorgen Grahn wrote:
>> You snipped the 'user' and 'sys' figures -- what did they say?
> # time dd if=/dev/zero of=toto bs=8192 count=$((100*1024*1024/8192))
> 12800+0 records in
> 12800+0 records out
> real 1m 3.68s
> user 0m 0.19s
> sys 0m 8.35s

So dd *itself* (including the "upper" part of the kernel work
it causes) doesn't do a lot of work. It's mostly waiting for I/O.

>> The output from 'mpstat -P ALL 1' on both machines would also be
>> useful.
>
> I couldn't find mpstat on either machine,

'mpstat' is part of the 'sysstat' package. I find it very useful, but
you can find the raw information it uses in /proc/stat. It's good
because it also shows kernel time which *isn't* attributed to any one
specific process (and splits this time up a bit).

> so I just did:
> # top -d20
> PID PPID USER STAT VSZ %MEM %CPU COMMAND
> 79 2 root RW 0 0% 76% [rpciod/0]
> 352 351 root R 1144 1% 13% dd if /dev/zero of toto bs 8192
> count 12800
> 101 2 root SW 0 0% 5% [nfsiod]
> 338 2 root SW 0 0% 5% [flush-0:10]
> 353 240 root R 1144 1% 0% top -d20

Hm, that seems to indicate that your client *is* at CPU overload,
with some kind of NFS-related kernel work.

> MyPC $ top -d20
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 15536 root 15 0 0 0 0 S 8.3 0.0 24:07.99 nfsd
> 15535 root 15 0 0 0 0 S 4.7 0.0 22:15.49 nfsd
> 15534 root 15 0 0 0 0 S 1.9 0.0 22:39.29 nfsd
> 29148 myuser 15 0 54736 35m 14m S 0.5 0.9 25:48.45 konsole
> 28713 myuser 15 0 10808 2416 1188 S 0.1 0.1 10:31.36 sshd
>
>
>> You probably cannot write to disk at anything *near* 1Gbit/s, but
>> ~2Mbit/s seems too low for a disk I/O bottleneck, too.
>
> When running directly on the server:
> MyPC $ time dd if=/dev/zero of=toto bs=8192 count=$((10000*1024*1024/8192))
> 1280000+0 records in
> 1280000+0 records out
> 10485760000 bytes (10 GB) copied, 93.5844 seconds, 112 MB/s
> real 1m40.572s
> user 0m0.289s
> sys 0m17.764s

Damn, my disks are slow ;-)

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .

Guillaume Dargaud <(E-Mail Removed)> wrote:

> I'm puzzled by the speed difference I get when doing transfers via
> netperf, dd via NFS. Can somebody shed some light on it ?

> The network is a direct 1Gbps connection between a PC and an
> embedded device. All the tests are run from the device. The disk is
> on the PC and mounted via NFS on the device.

> # netperf -H MyPC
> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to master
> (192.168.1.1) port 0 AF_INET
> Recv Send Send
> Socket Socket Message Elapsed
> Size Size Size Time Throughput
> bytes bytes bytes secs. 10^6bits/sec
> 87380 16384 16384 10.01 85.73

> Quite far from the theoretical 1024Mbps... (only 8% of it indeed)

Just how much "oomph" does your embedded device have? I would suggest
adding a -c command and running the test again:

netperf -H MyPC -c

If the embedded device is single core, netperf reporting 100% CPU
utilization is pretty obvious. If it has two cores, netperf reporting
50% CPU util suggets (but does not guarantee) that one core was
saturated. You can add a -d option and netperf will emit the details
of its CPU calculation.

rick jones
--
The glass is neither half-empty nor half-full. The glass has a leak.
The real question is "Can it be patched?"
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

> Just how much "oomph" does your embedded device have? I would suggest
> adding a -c command and running the test again:

It's a 300MHz PowerPC processor with a lightweight busybox linux on it.

> netperf -H MyPC -c

# netperf -H controller -c
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to master
(192.168.1.185) port 0 AF_INET
Recv Send Send Utilization Service
Demand
Socket Socket Message Elapsed Send Recv Send Recv
Size Size Size Time Throughput local remote local
remote
bytes bytes bytes secs. 10^6bits/s % S % U us/KB us/KB
87380 16384 16384 10.01 83.37 100.00 -1.00 98.257
-1.000


> If the embedded device is single core, netperf reporting 100% CPU
> utilization is pretty obvious. If it has two cores, netperf reporting
> 50% CPU util suggets (but does not guarantee) that one core was
> saturated. You can add a -d option and netperf will emit the details
> of its CPU calculation.

# netperf -H controller -d
scan_sockets_args called with the following argument vector
netperf -H controller -d
Program name: netperf
Local send alignment: 8
Local recv alignment: 8
Remote send alignment: 8
Remote recv alignment: 8
Report local CPU 0
Report remote CPU 0
Verbosity: 1
Debug: 1
Port: 12865
Test name: TCP_STREAM
Test bytes: 0 Test time: 10 Test trans: 0
Host name: controller

installing catcher for all signals
Could not install signal catcher for sig 65, errno 22
remotehost is controller and port 12865
establish_control called with host 'controller' port '12865' remfam
AF_UNSPEC
local '0.0.0.0' port '0' locfam AF_UNSPEC
getaddrinfo returned the following for host 'controller' port '12865' family
AF_UNSPEC
cannonical name: 'master'
flags: 2 family: AF_INET: socktype: SOCK_STREAM protocol IPPROTO_TCP
addrlen 16
sa_family: AF_INET sadata: 50 65 192 168 1 185
getaddrinfo returned the following for host '0.0.0.0' port '0' family
AF_UNSPEC
cannonical name: '0.0.0.0'
flags: 3 family: AF_INET: socktype: SOCK_STREAM protocol IPPROTO_TCP
addrlen 16
sa_family: AF_INET sadata: 0 0 0 0 0 0
bound control socket to 0.0.0.0 and 0
successful connection to remote netserver at controller and 12865
complete_addrinfo using hostname controller port 0 family AF_UNSPEC type
SOCK_STREAM prot IPPROTO_TCP flags 0x0
getaddrinfo returned the following for host 'controller' port '0' family
AF_UNSPEC
cannonical name: 'master'
flags: 2 family: AF_INET: socktype: SOCK_STREAM protocol IPPROTO_TCP
addrlen 16
sa_family: AF_INET sadata: 0 0 192 168 1 185
local_data_address not set, using local_host_name of '0.0.0.0'
complete_addrinfo using hostname 0.0.0.0 port 0 family AF_UNSPEC type
SOCK_STREAM prot IPPROTO_TCP flags 0x1
getaddrinfo returned the following for host '0.0.0.0' port '0' family
AF_UNSPEC
cannonical name: '0.0.0.0'
flags: 3 family: AF_INET: socktype: SOCK_STREAM protocol IPPROTO_TCP
addrlen 16
sa_family: AF_INET sadata: 0 0 0 0 0 0
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to master
(192.168.1.185) port 0 AF_INET
create_data_socket: socket 4 obtained...
netperf: get_sock_buffer: send socket size determined to be 16384
netperf: get_sock_buffer: receive socket size determined to be 87380
send_tcp_stream: send_socket obtained...
recv_response: received a 256 byte response
remote listen done.
About to start a timer for 10 seconds.
recv_response: received a 256 byte response
remote reporting results for 10.01 seconds
calculate_confidence: itr 1; time 10.012977; res 83.279937
lcpu -1.000000; rcpu -1.000000
lsdm -1.000000; rsdm -1.000000
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

87380 16384 16384 10.01 83.28
shutdown_control: shutdown of control connection requested.


--
Guillaume Dargaud
http://www.gdargaud.net/

Guillaume Dargaud <(E-Mail Removed)> wrote:
> > Just how much "oomph" does your embedded device have? I would suggest
> > adding a -c command and running the test again:

> It's a 300MHz PowerPC processor with a lightweight busybox linux on it.

> > netperf -H MyPC -c

> # netperf -H controller -c
> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to master
> (192.168.1.185) port 0 AF_INET
> Recv Send Send Utilization Service
> Demand
> Socket Socket Message Elapsed Send Recv Send Recv
> Size Size Size Time Throughput local remote local
> remote
> bytes bytes bytes secs. 10^6bits/s % S % U us/KB us/KB
> 87380 16384 16384 10.01 83.37 100.00 -1.00 98.257
> -1.000

Well, there you have your answer then - your embedded device is CPU
saturated at 80-odd Mbit/s. Not much point to it having a GbE
interface really, but I suppose it can be thought of as being good for
"connectivity."

If you want it to go faster, likely as not you will have to find some
path length reductions in the stack. Or tweaks that in effect reduce
path length.

rick jones
--
web2.0 n, the dot.com reunion tour...
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

>> Recv Send Send Utilization Service
>> Demand
>> Socket Socket Message Elapsed Send Recv Send
>> Recv
>> Size Size Size Time Throughput local remote local
>> remote
>> bytes bytes bytes secs. 10^6bits/s % S % U us/KB
>> us/KB
>> 87380 16384 16384 10.01 83.37 100.00 -1.00 98.257
>> -1.000
>
> Well, there you have your answer then - your embedded device is CPU
> saturated at 80-odd Mbit/s. Not much point to it having a GbE
> interface really, but I suppose it can be thought of as being good for
> "connectivity."

I'm not sure how to read the above stats. Is the %S the CPU use ? Indeed
'top' jumps to above 85% during netperf.
Why would a pure network transfer need so much CPU ? It's set with DMA, and
I see a many DMA interrupts during the transfer.
The embedded system has a 300MHz PowerPC.

I really need to improve network speed... I've tried playing with everything
I could to no avail: MTU, tsqueuelen, netdev_max_backlog, tcp_sack,
mem_min/max, etc, but I guess if it's CPU bound then I need to figure
something else. I haven't been able to change the duplex.

> If you want it to go faster, likely as not you will have to find some
> path length reductions in the stack. Or tweaks that in effect reduce
> path length.

Not sure what you mean by that. The systems are connected directly by a
crossover cable...

--
Guillaume Dargaud
http://www.gdargaud.net/

On Fri, 2011-02-18, Guillaume Dargaud wrote:

(Rick Jones wrote the below, I think. Please don't remove attribution
lines.)

>> If you want it to go faster, likely as not you will have to find some
>> path length reductions in the stack. Or tweaks that in effect reduce
>> path length.
>
> Not sure what you mean by that. The systems are connected directly by a
> crossover cable...

I guess he means the code paths -- spending less CPU instructions (on
average) for sending each bit.

I didn't read all those netperf discussions above, but I assume they
proved that the bottleneck wasn't in NFS but in TCP and lower?

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .