Unbuffered network save

> Linux network writes zero-copy
>
> I found the first link to be enlightening:
> http://www.linuxjournal.com/article/6345

This copies from file to file, so it doesn't really apply in my case (saving
buffers)

So I tried to implement the socket code suggested by Rick Jones, only
SO_SND_COPYAVOID is nowhere to be found: not in socket.h or any other header
file...
I'm using 2.6.35 and the architecture is buildroot/uclibc/busybox.
--
Guillaume Dargaud
http://www.gdargaud.net/

Guillaume Dargaud <(E-Mail Removed)> wrote:
> > If the profile(s) say that the copy from user space is taking most of
> > the cycles, then in the case of netperf at least, one might consider
> > running a TCP_SENDFILE test and seeing if that goes much faster. If
> > though, one's embedded device does not support CKU (ChecKsum Offload)
> > then even if you avoid the copy, you will not avoid the data access
> > when the checksum is calculated.

> I'm not at work right now, but off the top of my head, for heavy (but not
> saturaling) load, the profiler gave me:
> 50% copy_from_user
> 8% checksum
> 3% DMA
> 1% my app
> 1% Misc Linux processes
> The rest was idle

> I think the checksum is already offloaded to the FPGA, but even if
> that is not the case, there is much less to gain.

"Half a percent, half a percent, half a percent onward..." (My
apologies to Mr. Kipling

The "value" in CKO isn't so much the instructions for computing the
checksum, but the "stalls" from the cache misses from accessing the
data (waiting for memory access). Similarly for the data copy. That
is why CKO gets one ~N% and copy avoidance gets one ~N%, but both
together get one > 2N%.

So, I'm guessing the checksum looks small here because it is the data
copy that is doing the "heavy lifting" of bringing the data into the
processor's data cache.

rick jones
--
a wide gulf separates "what if" from "if only"
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Guillaume Dargaud <(E-Mail Removed)> wrote:

> So I tried to implement the socket code suggested by Rick Jones,
> only SO_SND_COPYAVOID is nowhere to be found: not in socket.h or any
> other header file...

That should not be much of a surprise as all that was for HP-UX 9.X
from the 1990's. It was meant only as background data, not specific
coding - sorry if that was not clear at the outset. Linux tends to
want to do things its own way. Witness how it has a different set of
parameters for sendfile() vs say HP-UX or BSD.

rick jones
--
Wisdom Teeth are impacted, people are affected by the effects of events.
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Guillaume Dargaud <(E-Mail Removed)> wrote:

> And another reason if that I'm now looking at the problem
> differently: previous thread was about why netperf was going 5 times
> faster than NFS, and now it's about saving buffers with
> copy-avoidance which I knew nothing about until yesterday

BTW, you mentioned you are writing the file to an NFS server. Are you
sure that all the NFS operations being performed are writes, or might
there be some other operations involved? Perhaps some unfortunate
alignment of the writes relative to the block size that is triggering
read-modify-write?

I've no idea that it would help, nor even if the semantics of what you
are doing would be compatible with it, but perhaps mmap()ing the file
will avoid copies from user to kernel on your "writes" and then if the
NFS client code is able to reference filecache buffers directly
without copy... Of course that depends on whether your network
interface supports "scatter gather" - able to take multiple buffers
per packet to be sent.

BTW, did you ever try a TCP_SENDFILE test? (against a local file, not
one that is NFS mounted - even if, in theory, the netperf send-ring's
worth of data may end-up cached, why chance it

rick jones

--
firebug n, the idiot who tosses a lit cigarette out his car window
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

On Fri, 04 Mar 2011 00:08:35 +0000, Rick Jones wrote:

> Guillaume Dargaud <(E-Mail Removed)> wrote:
>
>> So I tried to implement the socket code suggested by Rick Jones, only
>> SO_SND_COPYAVOID is nowhere to be found: not in socket.h or any other
>> header file...
>
> That should not be much of a surprise as all that was for HP-UX 9.X from
> the 1990's. It was meant only as background data, not specific coding -
> sorry if that was not clear at the outset. Linux tends to want to do
> things its own way. Witness how it has a different set of parameters
> for sendfile() vs say HP-UX or BSD.

Interesting, after I posted something in this thread I did some more
mucking around and found it on Solaris... Is HP-UX also BSD based?

Kevin Snodgrass <(E-Mail Removed)> wrote:
> On Fri, 04 Mar 2011 00:08:35 +0000, Rick Jones wrote:

> > Guillaume Dargaud <(E-Mail Removed)> wrote:
> >
> >> So I tried to implement the socket code suggested by Rick Jones,
> >> only SO_SND_COPYAVOID is nowhere to be found: not in socket.h or
> >> any other header file...
> >
> > That should not be much of a surprise as all that was for HP-UX
> > 9.X from the 1990's. It was meant only as background data, not
> > specific coding - sorry if that was not clear at the outset.
> > Linux tends to want to do things its own way. Witness how it has
> > a different set of parameters for sendfile() vs say HP-UX or BSD.

> Interesting, after I posted something in this thread I did some more
> mucking around and found it on Solaris... Is HP-UX also BSD based?

HP-UX 9.X's (and 8.X and 10.X) networking stack had BSD lineage.
However, HP-UX did not get SO_SND_COPYAVOID from BSD, it got it on its
own. It has always been my understanding that Solaris' stack (vs
SunOS) was based on a Streams-based stack from Mentat. As such, it's
stack is a "cousin" of the stack in HP-UX 11.X.

When/how Solaris (or BSD) might have gotten an SO_SND_COPYAVOID I do
not know.

rick jones
--
The computing industry isn't as much a game of "Follow The Leader" as
it is one of "Ring Around the Rosy" or perhaps "Duck Duck Goose."
- Rick Jones
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...