How to write my own protocol driver?

Hi,

I need to write my own ethernet protocol driver for high speed
reliable transfer of data between Linux based embedded system and low-
resource measurement devices. The concept has been verified in user
space using the libpcap library, and now it is time to decrease
latency by implementing it as a kernel driver.

The problem however is the lack of documentation describing how to
write the network protocol (not device!) drivers for Linux.
I have checked, how the ipx driver is written.
I have found functions like proto_register (
http://lxr.linux.no/linux+v3.1.4/net/core/sock.c#L2362 )
proto_unregister ( http://lxr.linux.no/linux+v3.1.4/net/core/sock.c#L2433
) and the structure proto ( http://lxr.linux.no/linux+v3.1.4/inc...et/sock.h#L728
) itself.

However those functions and structure are very scarcely documented.
Is there any detailed description similar to Linux Device Driver of
Understanding the Linux Kernel but focused on kernel protocol drivers?

The only thing I could find is:
http://www.linuxquestions.org/questi...otocol-794965/

I'll appreciate any pointers/suggestions.
--
TIA & Regards,
WZab

On Tue, 2011-11-29, wzab wrote:
> Hi,
>
> I need to write my own ethernet protocol driver for high speed
> reliable transfer of data between Linux based embedded system and low-
> resource measurement devices. The concept has been verified in user
> space using the libpcap library, and now it is time to decrease
> latency by implementing it as a kernel driver.

What features of IP or IPv6 are not enough or too much, compared to
your new layer 3 protocol?

Asking just out of curiosity. You lose a lot of convenience (e.g all
userspace tools, libraries, test equipment and hardware offloading) by
rolling your own, so it better be worth it.

....
> Is there any detailed description similar to Linux Device Driver of
> Understanding the Linux Kernel but focused on kernel protocol drivers?

Sorry, I cannot answer that. It's probably not an area which many
people are interested in, so it's not surprising if the literature
doesn't focus on it.

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .

On Nov 30, 1:07*am, Jorgen Grahn <grahn+n...@snipabacken.se> wrote:
> On Tue, 2011-11-29, wzab wrote:
> > Hi,
>
> > I need to write my own ethernet protocol driver for high speed
> > reliable transfer of data between Linux based embedded system and low-
> > resource measurement devices. The concept has been verified in user
> > space using the libpcap library, and now it is time to decrease
> > latency by implementing it as a kernel driver.
>
> What features of IP or IPv6 are not enough or too much, compared to
> your new layer 3 protocol?
>
> Asking just out of curiosity. *You lose a lot of convenience (e.g all
> userspace tools, libraries, test equipment and hardware offloading) by
> rolling your own, so it better be worth it.
>

The measurement devices are based on price optimized FPGAs so it is
not possible to implement TCP/IP mainly due to memory limitations.
IP should be possible, but it still makes the design more resource
hungry,
and is simply not necessary - the whole system is supposed to use its
private ethernet network with simple 100Mb/1Gb hub (again - proce
optimzation)
to connect multiple measurement devices to a single embedded system,
which receives
concentrated data, acknowledging packets, performs initial data
processing
and sends the aggregated data upstream using the standard TCP/IP.
In the private segments I don't need to use IP. Raw Ethernet frames
with
MAC addresses are sufficient.

I have succeeded with implementation of my own layer 3 protocol in
user
space, however the acknowledge was generated in user space (using the
libpcap), which significantly increased latency and limited the
throughput
(as I have to buffer not acknowledged packets in the FPGA's internal
memory for possible retransmission)

> ..
> > Is there any detailed description similar to Linux Device Driver of
> > Understanding the Linux Kernel but focused on kernel protocol drivers?
>
> Sorry, I cannot answer that. It's probably not an area which many
> people are interested in, so it's not surprising if the literature
> doesn't focus on it.
>

OK. Then I simply need to analyze the sources :-(.
--
Thanks & Regards,
WZab

On Wed, 2011-11-30, wzab wrote:
> On Nov 30, 1:07*am, Jorgen Grahn <grahn+n...@snipabacken.se> wrote:
>> On Tue, 2011-11-29, wzab wrote:
>> > Hi,
>>
>> > I need to write my own ethernet protocol driver for high speed
>> > reliable transfer of data between Linux based embedded system and low-
>> > resource measurement devices. The concept has been verified in user
>> > space using the libpcap library, and now it is time to decrease
>> > latency by implementing it as a kernel driver.
>>
>> What features of IP or IPv6 are not enough or too much, compared to
>> your new layer 3 protocol?
>>
>> Asking just out of curiosity. *You lose a lot of convenience (e.g all
>> userspace tools, libraries, test equipment and hardware offloading) by
>> rolling your own, so it better be worth it.

>> ..
>> > Is there any detailed description similar to Linux Device Driver of
>> > Understanding the Linux Kernel but focused on kernel protocol drivers?
>>
>> Sorry, I cannot answer that. It's probably not an area which many
>> people are interested in, so it's not surprising if the literature
>> doesn't focus on it.
>>
>
> OK. Then I simply need to analyze the sources :-(.

Well, that's only *my* guess. I have some experience with the Linux
networking code, but not with replacing IPv4/IPv6. Perhaps there is
a lot of info out there!

Another thought:
What if you make your Layer 3 protocol a fake IPv4? E.g.
- put fake IPv4 and UDP headers in your Ethernet frames
- hardcode the IP addresses to the MAC addresses
- on Linux, hardcode the ARP table
- ignore fragmentation issues, disable UDP checksums etc
You'd avoid the complexity on the FPGA side[1], but it would still
look like IP from Linux' point of view.

/Jorgen

[1] Except perhaps IP header checksum calculation.

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .

wzab <(E-Mail Removed)> wrote:
> I have succeeded with implementation of my own layer 3 protocol in
> user space, however the acknowledge was generated in user space
> (using the libpcap), which significantly increased latency and
> limited the throughput (as I have to buffer not acknowledged packets
> in the FPGA's internal memory for possible retransmission)

Um, just how much latency are we talking about, and is your protocol
streaming or request/response?

rick jones
--
oxymoron n, commuter in a gas-guzzling luxury SUV with an American flag
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

On Nov 30, 7:55*pm, Rick Jones <rick.jon...@hp.com> wrote:
> wzab <wza...@gmail.com> wrote:
> > I have succeeded with implementation of my own layer 3 protocol in
> > user space, however the acknowledge was generated in user space
> > (using the libpcap), which significantly increased latency and
> > limited the throughput (as I have to buffer not acknowledged packets
> > in the FPGA's internal memory for possible retransmission)
>
> Um, just how much latency are we talking about, and is your protocol
> streaming or request/response?
>

I need to achieve at least 100Mb/s, while I have ca 40kB of memory in
my FPGA.
Assuming some protocol overhead and latency jitter it means, that the
buffered data
should occupy not more, than 30kB.
Finally it gives us the latency of 2.4ms
However I'd like to use smaller FPGA , so smaller latencies are
desired...

--
Regards,
Wojtek

On Dec 1, 11:47*am, wzab <wza...@gmail.com> wrote:
> On Nov 30, 7:55*pm, Rick Jones <rick.jon...@hp.com> wrote:
>
> > wzab <wza...@gmail.com> wrote:
> > > I have succeeded with implementation of my own layer 3 protocol in
> > > user space, however the acknowledge was generated in user space
> > > (using the libpcap), which significantly increased latency and
> > > limited the throughput (as I have to buffer not acknowledged packets
> > > in the FPGA's internal memory for possible retransmission)
>
> > Um, just how much latency are we talking about, and is your protocol
> > streaming or request/response?
>
> I need to achieve at least 100Mb/s, while I have ca 40kB of memory in
> my FPGA.
> Assuming some protocol overhead and latency jitter it means, that the
> buffered data
> should occupy not more, than 30kB.
> Finally it gives us the latency of 2.4ms
> However I'd like to use smaller FPGA , so smaller latencies are
> desired...
>

Of course what I was talking about is streaming.
The control protocol is much more latency-tolerant, so streaming
of measurement data is my main problem.

I was thinking about different UDP or IP based solutions (RDS?),
but as routing is anyway not necessary, I decided to get rid of the
whole IP layer and use raw Ethernet frames...

wzab <(E-Mail Removed)> wrote:
> I need to achieve at least 100Mb/s, while I have ca 40kB of memory
> in my FPGA.
> Assuming some protocol overhead and latency jitter it means, that
> the buffered data> should occupy not more, than 30kB.
> Finally it gives us the latency of 2.4ms

2.4ms is an awfully long time. Is the receiver using a particularly
wimpy processor, or doing a great deal of processing before sending
the ACK? Is the user space code generating the ACKs presently
possibly being timesliced by other processes?

rick jones
--
firebug n, the idiot who tosses a lit cigarette out his car window
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

On Dec 1, 11:06*pm, Rick Jones <rick.jon...@hp.com> wrote:
> wzab <wza...@gmail.com> wrote:
> > I need to achieve at least 100Mb/s, while I have ca 40kB of memory
> > in my FPGA.
> > Assuming some protocol overhead and latency jitter it means, that
> > the buffered data> should occupy not more, than 30kB.
> > Finally it gives us the latency of 2.4ms
>
> 2.4ms is an awfully long time. *Is the receiver using a particularly
> wimpy processor, or doing a great deal of processing before sending
> the ACK? *Is the user space code generating the ACKs presently
> possibly being timesliced by other processes?
>
The process generating ACK in the current implementation is submitted
to standard scheduling policy - so 2.4ms is quite short time!
Maybe using the RT-enabled kernel or at least using the SCHED_FIFO
scheduling could improve the situation.
Anyway I'm quite sure, that sooner or later I'll face the efficiency
problems.
Solution with generation of ACK as soon as the packet is received,
and it's integrity i checked seems to be much more natural and
elegant - but this requires kernel space implementation...
--
Regards,
WZab

wzab <(E-Mail Removed)> wrote:
> The process generating ACK in the current implementation is
> submitted to standard scheduling policy - so 2.4ms is quite short
> time! Maybe using the RT-enabled kernel or at least using the
> SCHED_FIFO scheduling could improve the situation.
> Anyway I'm quite sure, that sooner or later I'll face the efficiency
> problems.
> Solution with generation of ACK as soon as the packet is received,
> and it's integrity i checked seems to be much more natural and
> elegant - but this requires kernel space implementation...

Is your ACK currently both an "I've got the data now" and a "You can
send more data now" indication? If so, ACKing from the kernel will be
strictly an "I've got the data now" and not a "You can send more data
now" unless you delay it until after the application has done a
receive of the data, which will still be subject to the same
scheduling vagaries you currently seem to have.

rick jones
--
I don't interest myself in "why". I think more often in terms of
"when", sometimes "where"; always "how much." - Joubert
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...