Unbuffered network save

Hello all,

I have an application on an embedded device that needs to save large buffers
as fast as possible. We use a 1Gbps network chip but the speed is abysmal
(85Mbps max). I just profiled the system and the results are clear: the
system spends most of its time in __copy_tofrom_user(), in other words
copying my buffers to the network driver.

The buffers are saved with fwrite to an NFS mounted drive, without C
bufferisation (setvbuf).

Is there a way to tell the driver to NOT buffer the data ? I'd rather wait
for the call to return.

I would think this has nothing to do with the protocol NFS but with the
network driver itself, but I'm not so sure. Is there another network
protocol that doesn't use buffers ?

Thanks.
--
Guillaume Dargaud
http://www.gdargaud.net/

On Fri, 2011-02-25, Guillaume Dargaud wrote:
> Hello all,
>
> I have an application on an embedded device that needs to save large buffers
> as fast as possible. We use a 1Gbps network chip but the speed is abysmal
> (85Mbps max). I just profiled the system and the results are clear: the
> system spends most of its time in __copy_tofrom_user(), in other words
> copying my buffers to the network driver.
>
> The buffers are saved with fwrite to an NFS mounted drive, without C
> bufferisation (setvbuf).
>
> Is there a way to tell the driver to NOT buffer the data ? I'd rather wait
> for the call to return.
>
> I would think this has nothing to do with the protocol NFS but with the
> network driver itself, but I'm not so sure. Is there another network
> protocol that doesn't use buffers ?

I'll pass on that question, but answer anyway:

It would have helped if you had mentioned that this the same issue you
asked about before, in "Network speed discrepancy" (Message-ID:
<ijdem8$p13$(E-Mail Removed)>).

I thought you and Rick Jones did some work and confirmed that NFS
wasn't the problem? But I didn't read that part of the thread very
carefully.

One thought: would it be possible to decrease the amount of data to be
pushed over the network? For example by gzipping it. I don't know if
gzip beats your network stack in terms of bps, but perhaps if the data
is accumulated over time, you can gzip it on the fly so that you never
have a huge uncompressed buffer to deal with?

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .

If the profile(s) say that the copy from user space is taking most of
the cycles, then in the case of netperf at least, one might consider
running a TCP_SENDFILE test and seeing if that goes much faster. If
though, one's embedded device does not support CKU (ChecKsum Offload)
then even if you avoid the copy, you will not avoid the data access
when the checksum is calculated.

It isn't Linux, but this writeup:

ftp://ftp.cup.hp.com/dist/networking.../copyavoid.pdf

will show that CKU without copy avoidance will buy you some small but
non-trivial percentage, but for the big win it needs to be coupled
with copy-avoidance. (What isn't shown so much is that - at least
back then - copy avoidance without CKU was about the same benefit as
CKO without copy avoidance)

rick jones
--
The glass is neither half-empty nor half-full. The glass has a leak.
The real question is "Can it be patched?"
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Guillaume Dargaud <(E-Mail Removed)> wrote:

> I have an application on an embedded device that needs to save large
> buffers as fast as possible. We use a 1Gbps network chip but the
> speed is abysmal (85Mbps max). I just profiled the system and the
> results are clear: the system spends most of its time in
> __copy_tofrom_user(), in other words copying my buffers to the
> network driver.

Are you quite sure the copy is to the network driver? Can the profile
you are taking be coerced into telling you which routines happen to be
calling __copy_tofrom_user()?

> The buffers are saved with fwrite to an NFS mounted drive, without C
> bufferisation (setvbuf).

I would have expected the copies to be from user to kernel on the
write call I'm guessing is underneath the fwrite. That is, into
filecache. From there, I would expect the NFS client code to package
things up to hand-off to the stack - without a further copy I would
think. But I am far from familiar with the Linux NFS client code.

BTW, if there is no buffering of the stream, is there really any
advantage to fwrite() over a plain old write?

> Is there a way to tell the driver to NOT buffer the data ? I'd
> rather wait for the call to return.

Unless there is some sort of NIC chip addressing limitation, or the
chip being driven by the driver is really simple (cannot do
scatter/gather for example), I do not believe there is a "copy to the
network driver."

NFS (or TCP if NFS is not using UDP) must retain a reference to the
data in case it must be retransmitted. So, there must be buffering of
some sort. The question is not whether the data will be buffered, but
whether the data can be sent without being copied.

If your system is 100% CPU bound, waiting or not waiting for the call
to return isn't going to change much.

rick jones
--
It is not a question of half full or empty - the glass has a leak.
The real question is "Can it be patched?"
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

> It would have helped if you had mentioned that this the same issue you
> asked about before, in "Network speed discrepancy" (Message-ID:
> <ijdem8$p13$(E-Mail Removed)>).

Yes, I just checked with google, and apparently my newsgroups feed didn't
show me all the messages.
And another reason if that I'm now looking at the problem differently:
previous thread was about why netperf was going 5 times faster than NFS, and
now it's about saving buffers with copy-avoidance which I knew nothing about
until yesterday

> I thought you and Rick Jones did some work and confirmed that NFS
> wasn't the problem? But I didn't read that part of the thread very
> carefully.
>
> One thought: would it be possible to decrease the amount of data to be
> pushed over the network? For example by gzipping it. I don't know if
> gzip beats your network stack in terms of bps, but perhaps if the data
> is accumulated over time, you can gzip it on the fly so that you never
> have a huge uncompressed buffer to deal with?
It's possible (although it poses problems on the receiving side), but I
don't see how that would help since the CPU seems to be already the limiting
factor. Using MORE CPU to zip won't help, even if it decreases the CPU used
by the network driver. And it only zips by a factor two. If it had been 10,
then it would have been well worth it.

--
Guillaume Dargaud
http://www.gdargaud.net/

> If the profile(s) say that the copy from user space is taking most of
> the cycles, then in the case of netperf at least, one might consider
> running a TCP_SENDFILE test and seeing if that goes much faster. If
> though, one's embedded device does not support CKU (ChecKsum Offload)
> then even if you avoid the copy, you will not avoid the data access
> when the checksum is calculated.

I'm not at work right now, but off the top of my head, for heavy (but not
saturaling) load, the profiler gave me:
50% copy_from_user
8% checksum
3% DMA
1% my app
1% Misc Linux processes
The rest was idle

I think the checksum is already offloaded to the FPGA, but even if that is
not the case, there is much less to gain.

> It isn't Linux, but this writeup:
>
> ftp://ftp.cup.hp.com/dist/networking.../copyavoid.pdf
OK, I'll read it in front of the fireplace !

> will show that CKU without copy avoidance will buy you some small but
> non-trivial percentage, but for the big win it needs to be coupled
> with copy-avoidance. (What isn't shown so much is that - at least
> back then - copy avoidance without CKU was about the same benefit as
> CKO without copy avoidance)

First time I hear this term 'copy-avoidance', so that'll help with google
searches at least !

Have a nice WE.
--
Guillaume Dargaud
http://www.gdargaud.net/

On 26.02.2011, Guillaume Dargaud <(E-Mail Removed)> wrote:
> First time I hear this term 'copy-avoidance', so that'll help with google
> searches at least !

What about 'zero-copy'? It needs hardware supporting it. Google for
RDMA, iWARP(no changes in ethernet infrastructue, only NICs) and
InfiniBand(different network architecture with native support for RDMA).

--
Pozdrawiam
orcus

Thanks, That document was an interesting (if ancient) read.
So in summary I need to use copy-avoidance for writing my data. I'm no
network specialist, so excuse me if the following questions may be dumb:

- does this mean I can only do raw socket writes and need to write my own
app on the other side to receive the data ? (it's been 20 years since I last
looked at socket code)

- or is it possible to configure the socket of an existing protocole (NFS
for instance) to be in SO_SND_COPYAVOID mode ?

> What about 'zero-copy'? It needs hardware supporting it. Google for
> RDMA, iWARP(no changes in ethernet infrastructue, only NICs) and
> InfiniBand(different network architecture with native support for RDMA).

Don't those protocols use special hardware ?
I'm running on a custom embedded device and there is no choice of changing
hardware. I'm very far from the theoretical 1Gbps limit of the current
hardware.

Aren't there ways in Linux to do network writes with zero-copy, besides raw
socket writes ?
--
Guillaume Dargaud
http://www.gdargaud.net/

On Mon, 28 Feb 2011 10:04:08 +0100, Guillaume Dargaud wrote:
> Aren't there ways in Linux to do network writes with zero-copy, besides
> raw socket writes ?

Google this:
Linux network writes zero-copy

I found the first link to be enlightening:
http://www.linuxjournal.com/article/6345

On Sat, 2011-02-26, Guillaume Dargaud wrote:
....
>> One thought: would it be possible to decrease the amount of data to be
>> pushed over the network? For example by gzipping it. I don't know if
>> gzip beats your network stack in terms of bps, but perhaps if the data
>> is accumulated over time, you can gzip it on the fly so that you never
>> have a huge uncompressed buffer to deal with?

> It's possible (although it poses problems on the receiving side), but I
> don't see how that would help since the CPU seems to be already the limiting
> factor. Using MORE CPU to zip won't help, even if it decreases the CPU used
> by the network driver.

See "if the data is accumulated over time" above. If you spend a day
or a week creating the data and you can compress it as soon as it's
created, compression pretty much comes for free. But probably your
problem doesn't allow that approach.

> And it only zips by a factor two. If it had been 10,
> then it would have been well worth it.

True. I was hoping it was log files or something like that, where you
can get a factor 5 or 10. (Or like one case I've seen: 1GB core dumps
mostly filled with zeros. These compressed very well ...)

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .