How to write my own protocol driver?

On Thu, 2011-12-01, wzab wrote:
> On Dec 1, 11:47*am, wzab <wza...@gmail.com> wrote:
>> On Nov 30, 7:55*pm, Rick Jones <rick.jon...@hp.com> wrote:
>>
>> > wzab <wza...@gmail.com> wrote:
>> > > I have succeeded with implementation of my own layer 3 protocol in
>> > > user space, however the acknowledge was generated in user space
>> > > (using the libpcap), which significantly increased latency and
>> > > limited the throughput (as I have to buffer not acknowledged packets
>> > > in the FPGA's internal memory for possible retransmission)
>>
>> > Um, just how much latency are we talking about, and is your protocol
>> > streaming or request/response?
>>
>> I need to achieve at least 100Mb/s, while I have ca 40kB of memory in
>> my FPGA.
>> Assuming some protocol overhead and latency jitter it means, that the
>> buffered data
>> should occupy not more, than 30kB.
>> Finally it gives us the latency of 2.4ms
>> However I'd like to use smaller FPGA , so smaller latencies are
>> desired...
>>
>
> Of course what I was talking about is streaming.
> The control protocol is much more latency-tolerant, so streaming
> of measurement data is my main problem.
>
> I was thinking about different UDP or IP based solutions (RDS?),
> but as routing is anyway not necessary, I decided to get rid of the
> whole IP layer and use raw Ethernet frames...

That reminds me: what about the "fake IP stack" suggestion I made
elsewhere in the thread? It seems to me that if you *don't* have to
write a brand new L3 layer at the Linux end you'll save a lot of work,
and you'll benefit from all the work that has been done to make the
Linux IP stack as efficient as possible.

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .

On 2.12.11 2:20 , wzab wrote:
> On Dec 1, 11:06 pm, Rick Jones<rick.jon...@hp.com> wrote:
>> wzab<wza...@gmail.com> wrote:
>>> I need to achieve at least 100Mb/s, while I have ca 40kB of memory
>>> in my FPGA.
>>> Assuming some protocol overhead and latency jitter it means, that
>>> the buffered data> should occupy not more, than 30kB.
>>> Finally it gives us the latency of 2.4ms
>>
>> 2.4ms is an awfully long time. Is the receiver using a particularly
>> wimpy processor, or doing a great deal of processing before sending
>> the ACK? Is the user space code generating the ACKs presently
>> possibly being timesliced by other processes?
>>
> The process generating ACK in the current implementation is submitted
> to standard scheduling policy - so 2.4ms is quite short time!
> Maybe using the RT-enabled kernel or at least using the SCHED_FIFO
> scheduling could improve the situation.
> Anyway I'm quite sure, that sooner or later I'll face the efficiency
> problems.
> Solution with generation of ACK as soon as the packet is received,
> and it's integrity i checked seems to be much more natural and
> elegant - but this requires kernel space implementation...
> --
> Regards,
> WZab


Did you get the latencies using TCP as transport protocol?

There is a method called delayed ACK, which waits some for return
traffic before sending an ACK (which is a data segment with null data).

You cannot expect to get an ACK segment for every segment sent. To limit
return traffic, the TCP/IP stack sends an ACK for every second or third
incoming segment, acknowledging all octets received so far.

If you need an ACK for every packet, creating effectively a non-
windowed stop-and-go protocol, you should consider UDP on top
of a simple IP stack. It fits into very small space, and this is
why it is used in TFTP for network booting.

--

Tauno Voipio

On Dec 2, 7:49*pm, Rick Jones <rick.jon...@hp.com> wrote:
> wzab <wza...@gmail.com> wrote:
> > The process generating ACK in the current implementation is
> > submitted to standard scheduling policy - so 2.4ms is quite short
> > time! *Maybe using the RT-enabled kernel or at least using the
> > SCHED_FIFO scheduling could improve the situation.
> > Anyway I'm quite sure, that sooner or later I'll face the efficiency
> > problems.
> > Solution with generation of ACK as soon as the packet is received,
> > and it's integrity i checked seems to be much more natural and
> > elegant - but this requires kernel space implementation...
>
> Is your ACK currently both an "I've got the data now" and a "You can
> send more data now" indication? *If so, ACKing from the kernel will be
> strictly an "I've got the data now" and not a "You can send more data
> now" unless you delay it until after the application has done a
> receive of the data, which will still be subject to the same
> scheduling vagaries you currently seem to have.
>

In my current user-space based setup I have two levels of handshake
(the whole protocol will be opensourced and published, but it'll take
some
time ;-) ):
1. Each packet is acknowledged (and this should be done as soon as
possible,
so the kernel implementation is preferred - as this affects
the achievable throughput).
2. The software which processes data provides "ready for data",
"slow down" and "stop sending data" messages, which may be
generated
much slower (in user-space). Of course the last two must be
generated
in advance, and buffer with sufficient capacity must be provided.
--
WZab

On Dec 2, 10:24*pm, Tauno Voipio <tauno.voi...@notused.fi.invalid>
wrote:

>
> Did you get the latencies using TCP as transport protocol?
>
The TCP is simply to "heavy" for the FPGA end of the link.
I know, there are some implementations of TCP for FPGA's but they
require either big FPGA with sufficient internal memory for the TCP
stack or use of external memory.
I want to be able to use small FPGA without external memories.