How to achieve super low TCP latency?

I have an application for my Linux powered milling machine.

http://igor.chudov.com/projects/Brid...ct-2-CNC-Mill/

It requires, or rather benefits, from super low TCP/IP latency.

To avoid any network or router issues, I am only talking to 127.0.0.1,
so that network throughput is not even a part of my equation.

And yet, I see latencies that I would prefer to lower. To wit:

ping localhost
PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.010 ms
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.013 ms
64 bytes from localhost (127.0.0.1): icmp_seq=3 ttl=64 time=0.013 ms
64 bytes from localhost (127.0.0.1): icmp_seq=4 ttl=64 time=0.016 ms
64 bytes from localhost (127.0.0.1): icmp_seq=5 ttl=64 time=0.012 ms
64 bytes from localhost (127.0.0.1): icmp_seq=6 ttl=64 time=0.014 ms
64 bytes from localhost (127.0.0.1): icmp_seq=7 ttl=64 time=0.013 ms
^C
--- localhost ping statistics ---
7 packets transmitted, 7 received, 0% packet loss, time 5999ms
rtt min/avg/max/mdev = 0.010/0.013/0.016/0.001 ms

How can I reduce that 13 microsecond latency?

Also, in TCP based comms, I see about 60 microsecond delays between a
request sent and a reply received.

Of course, part of this is an app related stuff, but I estimate it to
be under one-two microseconds.

How can I improve this?

i

To further clarify, this milling application is such that the
bandwidth available FAR exceeds bandwidth that may ever be needed.

Thus, I would prefer to minimize delays pertaining to buffering.

The controller i talk to supports TCP only, so I cannot go to UDP or
anything of the sort.

i

On 2011-03-28, Ignoramus446 <(E-Mail Removed)> wrote:
> I have an application for my Linux powered milling machine.
>
> http://igor.chudov.com/projects/Brid...ct-2-CNC-Mill/
>
> It requires, or rather benefits, from super low TCP/IP latency.
>
> To avoid any network or router issues, I am only talking to 127.0.0.1,
> so that network throughput is not even a part of my equation.
>
> And yet, I see latencies that I would prefer to lower. To wit:
>
> ping localhost
> PING localhost (127.0.0.1) 56(84) bytes of data.
> 64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.010 ms
> 64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.013 ms
> 64 bytes from localhost (127.0.0.1): icmp_seq=3 ttl=64 time=0.013 ms
> 64 bytes from localhost (127.0.0.1): icmp_seq=4 ttl=64 time=0.016 ms
> 64 bytes from localhost (127.0.0.1): icmp_seq=5 ttl=64 time=0.012 ms
> 64 bytes from localhost (127.0.0.1): icmp_seq=6 ttl=64 time=0.014 ms
> 64 bytes from localhost (127.0.0.1): icmp_seq=7 ttl=64 time=0.013 ms
> ^C
> --- localhost ping statistics ---
> 7 packets transmitted, 7 received, 0% packet loss, time 5999ms
> rtt min/avg/max/mdev = 0.010/0.013/0.016/0.001 ms
>
> How can I reduce that 13 microsecond latency?
>
> Also, in TCP based comms, I see about 60 microsecond delays between a
> request sent and a reply received.
>
> Of course, part of this is an app related stuff, but I estimate it to
> be under one-two microseconds.
>
> How can I improve this?
>
> i

On Sun, 27 Mar 2011 20:36:47 -0500, Ignoramus446 wrote:

> I have an application for my Linux powered milling machine.
>
> http://igor.chudov.com/projects/Brid...nteract-2-CNC-
Mill/
>
> It requires, or rather benefits, from super low TCP/IP latency.
>
> To avoid any network or router issues, I am only talking to 127.0.0.1,
> so that network throughput is not even a part of my equation.
>
> And yet, I see latencies that I would prefer to lower. To wit:
>
> ping localhost
> PING localhost (127.0.0.1) 56(84) bytes of data. 64 bytes from localhost
> (127.0.0.1): icmp_seq=1 ttl=64 time=0.010 ms 64 bytes from localhost
> (127.0.0.1): icmp_seq=2 ttl=64 time=0.013 ms 64 bytes from localhost
> (127.0.0.1): icmp_seq=3 ttl=64 time=0.013 ms 64 bytes from localhost
> (127.0.0.1): icmp_seq=4 ttl=64 time=0.016 ms 64 bytes from localhost
> (127.0.0.1): icmp_seq=5 ttl=64 time=0.012 ms 64 bytes from localhost
> (127.0.0.1): icmp_seq=6 ttl=64 time=0.014 ms 64 bytes from localhost
> (127.0.0.1): icmp_seq=7 ttl=64 time=0.013 ms ^C
> --- localhost ping statistics ---
> 7 packets transmitted, 7 received, 0% packet loss, time 5999ms rtt
> min/avg/max/mdev = 0.010/0.013/0.016/0.001 ms
>
> How can I reduce that 13 microsecond latency?
>
> Also, in TCP based comms, I see about 60 microsecond delays between a
> request sent and a reply received.
>
> Of course, part of this is an app related stuff, but I estimate it to be
> under one-two microseconds.
>
> How can I improve this?

Probably by not using TCP at all.

This is an interesting topic and I'm curious about solid information.
I've done a lot of real-time applications, and if I needed sub-
microsecond latencies I'd be inclined to steer clear of big opaque
libraries like TCP/IP. The protocol is designed for reliability over long-
distance diverse networks, not low latency on a single computer. You must
have a powerful reason for wanting to use it in an application that
doesn't required long-distance communication.

You didn't say what OS, but your are asking on general linux groups. This
is hard real-time stuff, and you'd be better off asking on forums
dedicated to real-time programming. I don't know any, unfortunately. I
know that real-time is done using specialized versions of Linux, and I'm
curious about its possibilities for this kind of work.

["Followup-To:" header set to comp.os.linux.networking.]

On Mon, 2011-03-28, Ignoramus446 wrote:
> I have an application for my Linux powered milling machine.
>
> http://igor.chudov.com/projects/Brid...ct-2-CNC-Mill/
>
> It requires, or rather benefits, from super low TCP/IP latency.
>
> To avoid any network or router issues, I am only talking to 127.0.0.1,
> so that network throughput is not even a part of my equation.
>
> And yet, I see latencies that I would prefer to lower. To wit:
>
> ping localhost
> PING localhost (127.0.0.1) 56(84) bytes of data.
> 64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.010 ms
> 64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.013 ms
> 64 bytes from localhost (127.0.0.1): icmp_seq=3 ttl=64 time=0.013 ms
> 64 bytes from localhost (127.0.0.1): icmp_seq=4 ttl=64 time=0.016 ms
> 64 bytes from localhost (127.0.0.1): icmp_seq=5 ttl=64 time=0.012 ms
> 64 bytes from localhost (127.0.0.1): icmp_seq=6 ttl=64 time=0.014 ms
> 64 bytes from localhost (127.0.0.1): icmp_seq=7 ttl=64 time=0.013 ms
> ^C
> --- localhost ping statistics ---
> 7 packets transmitted, 7 received, 0% packet loss, time 5999ms
> rtt min/avg/max/mdev = 0.010/0.013/0.016/0.001 ms
>
> How can I reduce that 13 microsecond latency?
>
> Also, in TCP based comms, I see about 60 microsecond delays between a
> request sent and a reply received.
>
> Of course, part of this is an app related stuff, but I estimate it to
> be under one-two microseconds.

Don't estimate -- measure. You can get good numbers from low down in
the IP stack from tcpdump(1), and you can get numbers at the
process--kernel border using strace(1).

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .

Ping != TCP, but you may already know that. How do you write data to
the socket? Is it ever more than one send call per "message?" Is
your message size fixed? How does the controller software at the
other end pull data from the socket? (strace may be helpful if you
don't have source).

Ultimately, over loopback, your minimum latency will be a function of
the stack path length and whether you tickle unpleasant things (like
say doing multiple sends for a single message). There will also be
the question of whether or not your client or the controller software
gets timeslided.

For some idea of the path length, you can profile. You might also try
running a netperf TCP_RR test with CPU utilization enabled, and little
to nothing else running on the system at the time. (Of course, I have
a natural bias towards suggesting folks run netperf

happy benchmarking,

rick jones
--
It is not a question of half full or empty - the glass has a leak.
The real question is "Can it be patched?"
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Ignoramus7104 <(E-Mail Removed)> wrote:
> On 2011-03-28, Rick Jones <(E-Mail Removed)> wrote:

> > How do you write data to the socket?

> By sayind "send" and using TCP_NODELAY option.

> The amount of sending is minuscule, compared to the available
> bandwidth. So, the "pipe" will never be full, or even close to full.

> > Is it ever more than one send call per "message?"

> Never

Then, unless you have multiple messages outstanding at one time, the
TCP_NODELAY is extraneous.

> > Is your message size fixed?

> Not fixed, but usually small.

> > How does the controller software at the other end pull data from
> > the socket? (strace may be helpful if you don't have source).

> I have a source. I wrote both parts.

> Sadly, I am basically forced to use TCP, as opposed to say UDP, due
> to some stupid requirement, which I prefer not to debate.

Well, many of us here are fond of the "Paul Harvey" (as in, learning
"The rest of the story.")

> So, within the limitation of having to use TCP, I want to find the
> best way of sending.

And receiving. Frankly, it shoulds like you are already there for
sending - single send per message is about as good as you can get
unless you have multiple messages pending.

How many receive calls do you make per message?

> > Ultimately, over loopback, your minimum latency will be a function of
> > the stack path length and whether you tickle unpleasant things (like
> > say doing multiple sends for a single message). There will also be
> > the question of whether or not your client or the controller software
> > gets timeslided.

> This is where I am, indeed, concerned. I did time some things and just
> timing "send( ... )" alone gives something like 20 microseconds to
> just do "send". (!!!)

> So, just the send function seems to trigger pretty huge work,
> amounting to 20 microseconds.

Well, since this is loopback it may not always be just send. Much of
the receive side can end-up happening on "your" stack as well. I
believe if conditions are right, the processing can go all the way to
queuing the buffer to the receiver's socket.

rick jones
--
portable adj, code that compiles under more than one compiler
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...