Network speed discrepancy

Guillaume Dargaud <(E-Mail Removed)> wrote:
> >> Recv Send Send Utilization Service
> >> Demand
> >> Socket Socket Message Elapsed Send Recv Send
> >> Recv
> >> Size Size Size Time Throughput local remote local
> >> remote
> >> bytes bytes bytes secs. 10^6bits/s % S % U us/KB
> >> us/KB
> >> 87380 16384 16384 10.01 83.37 100.00 -1.00 98.257
> >> -1.000
> >
> > Well, there you have your answer then - your embedded device is CPU
> > saturated at 80-odd Mbit/s. Not much point to it having a GbE
> > interface really, but I suppose it can be thought of as being good for
> > "connectivity."

> I'm not sure how to read the above stats. Is the %S the CPU use ? Indeed
> 'top' jumps to above 85% during netperf.

Essentially, that "column" is the local (sending side) CPU utilization
while netperf is running. Specifically the "%S" is informing us as to
what method was used to measure CPU utilization.

http://www.netperf.org/svn/netperf2/...PU-Utilization

> Why would a pure network transfer need so much CPU ? It's set with
> DMA, and I see a many DMA interrupts during the transfer. The
> embedded system has a 300MHz PowerPC.

It takes a certain number of instructions to send a packet through the
stack. So, to get a given bandwidth, that means a certain number of
packets per second. That then means being able to service a certain
number of instructions per second in the processor. Your 300 MHz
PowerPC processor is able to execute enough instructions per second to
send a number of packets per second that results in a bandwidth of
80-something Mbit/s.

> I really need to improve network speed... I've tried playing with everything
> I could to no avail: MTU, tsqueuelen, netdev_max_backlog, tcp_sack,
> mem_min/max, etc, but I guess if it's CPU bound then I need to figure
> something else. I haven't been able to change the duplex.

> > If you want it to go faster, likely as not you will have to find some
> > path length reductions in the stack. Or tweaks that in effect reduce
> > path length.

> Not sure what you mean by that. The systems are connected directly by a
> crossover cable...

You have to remove instructions from the stack - that is, you have to
make it more efficient. Reduce the number of instructions required to
send a packet, so your processor can send more packets per second.

rick jones
--
denial, anger, bargaining, depression, acceptance, rebirth...
where do you want to be today?
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Jorgen Grahn <grahn+(E-Mail Removed)> wrote:
> I didn't read all those netperf discussions above, but I assume they
> proved that the bottleneck wasn't in NFS but in TCP and lower?

It does suggest that. Netperf won't measure entirely the same path as
taken by NFS code, but the results do suggest that the number of
instructions per second that the processor was able to execute were
such that with the path length of the stack the system could not go
faster than aboug 85 Mbits/s.

It may be worthwhile to see if the embedded NIC here can do things
such as:

*) Checksum offload (CKO)
*) TCP Segmentation Offload (TSO)
*) General Receive Offload (GRO)

as those, if supported, could reduce the "effective" path length to be
executed by the CPU by offloading some work to the NIC.

However, unless this embedded system is only ever going to be pushing
bulk data around, there will still be a need to reduce the basic
networking path length as those offloads above do little or nothing
for small sends/traffic.

rick jones
--
I don't interest myself in "why." I think more often in terms of
"when," sometimes "where;" always "how much." - Joubert
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

> > If the embedded device is single core, netperf reporting 100% CPU
> > utilization is pretty obvious. If it has two cores, netperf
> > reporting 50% CPU util suggets (but does not guarantee) that one
> > core was saturated. You can add a -d option and netperf will emit
> > the details of its CPU calculation.

> # netperf -H controller -d

But only if there is still the -c option present Although, given
that netperf is reporting 100% CPU util, the -d output probably isn't
necessary.

rick jones
--
The glass is neither half-empty nor half-full. The glass has a leak.
The real question is "Can it be patched?"
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

> It may be worthwhile to see if the embedded NIC here can do things
> such as:
>
> *) Checksum offload (CKO)
Yes, it's already doing it and it's not the main bottleneck (see other
thread started yesterday).

> *) TCP Segmentation Offload (TSO)
> *) General Receive Offload (GRO)
I'll have to check on that when I'm at work on monday...
--
Guillaume Dargaud
http://www.gdargaud.net/