CPU load due to IP networking

I've been looking for some information on the web, in white papers and
even on Usenet, but if it exists I can't seem to put together a search
that finds it. I'm asking here if anyone knows of something along
these lines and can point me to it.

I'm looking for some data on the approximate resource cost,
particularly CPU usage cost, of doing networking in a typical system,
in particular a Unix-like system (linux is a suitable example) and
vxworks (hence the crossposting).

I'm not looking for bandwidth / capacity type data (e.g., N kpkts/sec
or M MB/sec); these measures are fairly common and there are lots of
examples on the web. Rather I'm looking for an approximate cycle or
instruction count list, like this: I'd like to know that the CPU will
expend A instructions (or cycles) per IP packet in, B cycles per IP
packet out, C cycles per TCP packet in, D cycles per TCP packet out,
etc.; and T cycles per connection per second, and U cycles per second
for background processing, etc. Then using these numbers loosely as
rules of thumb, and given a traffic model and a CPU of a certain type
and speed, I could compute a rough estimate of what portion of the CPU
capacity would be consumed by networking activities. I'd also like to
include not only TCP/IP but UDP and some other router-like protocols
(e.g. BGP or OSPF routing) and layer 2 activity (such as implied by
ethernet connections.

I hope it's clear enough what kind of data I'm looking for, and that I
only need rough numbers. If anyone knows of something of this sort
that's been published, for any operating system, actually, I'd
appreciate a pointer.

Thanks / JBL

"jbl" <(E-Mail Removed)> writes:

>I've been looking for some information on the web, in white papers and
>even on Usenet, but if it exists I can't seem to put together a search
>that finds it. I'm asking here if anyone knows of something along
>these lines and can point me to it.

>I'm looking for some data on the approximate resource cost,
>particularly CPU usage cost, of doing networking in a typical system,
>in particular a Unix-like system (linux is a suitable example) and
>vxworks (hence the crossposting).

This is an extremely ill defined question. It will depend on exactly how
the driver is written, how the driver is accessed by the software, etc.



>I'm not looking for bandwidth / capacity type data (e.g., N kpkts/sec
>or M MB/sec); these measures are fairly common and there are lots of
>examples on the web. Rather I'm looking for an approximate cycle or
>instruction count list, like this: I'd like to know that the CPU will
>expend A instructions (or cycles) per IP packet in, B cycles per IP
>packet out, C cycles per TCP packet in, D cycles per TCP packet out,
>etc.; and T cycles per connection per second, and U cycles per second
>for background processing, etc. Then using these numbers loosely as
>rules of thumb, and given a traffic model and a CPU of a certain type
>and speed, I could compute a rough estimate of what portion of the CPU
>capacity would be consumed by networking activities. I'd also like to
>include not only TCP/IP but UDP and some other router-like protocols
>(e.g. BGP or OSPF routing) and layer 2 activity (such as implied by
>ethernet connections.

>I hope it's clear enough what kind of data I'm looking for, and that I
>only need rough numbers. If anyone knows of something of this sort
>that's been published, for any operating system, actually, I'd
>appreciate a pointer.

The numbers you are liable to get will be so rough that you could guess
them just as profitably.

Maybe if y ou told us the problem you are trying to solve, you would get
better answers.


>Thanks / JBL

jbl wrote:
> I've been looking for some information on the web, in white papers and
> even on Usenet, but if it exists I can't seem to put together a search
> that finds it. I'm asking here if anyone knows of something along
> these lines and can point me to it.
>
> I'm looking for some data on the approximate resource cost,
> particularly CPU usage cost, of doing networking in a typical system,
> in particular a Unix-like system (linux is a suitable example) and
> vxworks (hence the crossposting).
>
> I'm not looking for bandwidth / capacity type data (e.g., N kpkts/sec
> or M MB/sec); these measures are fairly common and there are lots of
> examples on the web. Rather I'm looking for an approximate cycle or
> instruction count list, like this: I'd like to know that the CPU will
> expend A instructions (or cycles) per IP packet in, B cycles per IP
> packet out, C cycles per TCP packet in, D cycles per TCP packet out,
> etc.; and T cycles per connection per second, and U cycles per second
> for background processing, etc. Then using these numbers loosely as
> rules of thumb, and given a traffic model and a CPU of a certain type
> and speed, I could compute a rough estimate of what portion of the CPU
> capacity would be consumed by networking activities. I'd also like to
> include not only TCP/IP but UDP and some other router-like protocols
> (e.g. BGP or OSPF routing) and layer 2 activity (such as implied by
> ethernet connections.

I think you will find that load vs. packets or load vs. bytes is
non-linear to the point that you are not going to get a reasonable
prediction or even a number you are likely to be able to repeat.
>
> I hope it's clear enough what kind of data I'm looking for, and that I
> only need rough numbers. If anyone knows of something of this sort
> that's been published, for any operating system, actually, I'd
> appreciate a pointer.

It's clear that you have an interesting time looking up details on doing
kernel profiling. Once you can do that you can apply any synthetic load
you think will be useful, and report your findings. Develop your
equations to include all of the tunables in a modern kernel, since they
all affect the performance.

--
bill davidsen <(E-Mail Removed)>
CTO TMR Associates, Inc
Doing interesting things with small computers since 1979

Unruh wrote:
> "jbl" <(E-Mail Removed)> writes:
> >I'm looking for some data on the approximate resource cost,
> >particularly CPU usage cost, of doing networking in a typical system,
> >in particular a Unix-like system (linux is a suitable example) and
> >vxworks (hence the crossposting).
>
> This is an extremely ill defined question. It will depend on exactly how
> the driver is written, how the driver is accessed by the software, etc.

This is why I'm interested only in rough estimates -- I should perhaps
have added examples.

> >I hope it's clear enough what kind of data I'm looking for, and that I
> >only need rough numbers. If anyone knows of something of this sort
> >that's been published, for any operating system, actually, I'd
> >appreciate a pointer.
>
> The numbers you are liable to get will be so rough that you could guess
> them just as profitably.
>
> Maybe if y ou told us the problem you are trying to solve, you would get
> better answers.

I'm trying to provide some guideliens for system engineers in an
environment where a CPU is already loaded a certain amount doing
control and telemetry with no internetworking within some allocation
(such as, say, 75%) of available CPU bandwidth. In this case it is
difficult to say "throw more bandwidth at it" because high speed
processors designed for this environment (radiation hardened) are not
yet available. An alternative, adding a new processor to handle
network and/or router functions, has substantial cost in increased
complexity, power and space required. Sorry to be so vague about this.

I realize that the most reasonable way to answer this question may in
fact be "build it and do the measurements".

/JBL

In article <(E-Mail Removed) .com>,
jbl <(E-Mail Removed)> wrote:

>> >I'm looking for some data on the approximate resource cost,
>> >particularly CPU usage cost, of doing networking in a typical system,
>> >in particular a Unix-like system (linux is a suitable example) and
>> >vxworks (hence the crossposting).
>>
>> This is an extremely ill defined question. It will depend on exactly how
>> the driver is written, how the driver is accessed by the software, etc.
>
>This is why I'm interested only in rough estimates -- I should perhaps
>have added examples.

Examples would not help. The costs to do any given network task vary
by orders of magnitude depending on many things too numerous to list
here. For example, the sum of the network work and the other application
might be more or less than the sum of their parts thanks to memory
bandwidth including cache effects. You might as well ask "many miles
will a gallon of fuel move a vehicle?" An example that any third
party might find instructive such as "bicycles," "trains," "earth to
orbit rockets," would not narrow the field enough to yield answer
useful to technical people. Non-technical discussions or what I call
policitics are likley to be otherwise, but apparently random guesses
shaded according to the agenda are more honest, cheaper, and usually
more effective there.


>> The numbers you are liable to get will be so rough that you could guess
>> them just as profitably.
>>
>> Maybe if y ou told us the problem you are trying to solve, you would get
>> better answers.
>
>I'm trying to provide some guideliens for system engineers in an
>environment where a CPU is already loaded a certain amount doing
>control and telemetry with no internetworking within some allocation
>(such as, say, 75%) of available CPU bandwidth. In this case it is

Somewhat better than guessing would be to
- take hardware and network software similar to what you might use,
- produce a relevant synthetic application that uses 100% of the CPU,
- run the network code and the synthetic appliation simultaneously
and see how much each slows down.


Vernon Schryver (E-Mail Removed)

In comp.protocols.tcp-ip jbl <(E-Mail Removed)> wrote:
> I realize that the most reasonable way to answer this question may in
> fact be "build it and do the measurements".

Indeed. You may find the service demand calculations of netperf
helpful. www.netperf.org.

rick jones
--
denial, anger, bargaining, depression, acceptance, rebirth...
where do you want to be today?
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

On 22 Jun 2005, (E-Mail Removed) wrote:
> In comp.protocols.tcp-ip jbl <(E-Mail Removed)> wrote:
>> I realize that the most reasonable way to answer this question may
>> in fact be "build it and do the measurements".
>
> Indeed. You may find the service demand calculations of netperf
> helpful. www.netperf.org.

I have implemented networking on several embedded processors. I think
the sad truth is that there is too much variation. Most expressed
this sentiment.

For instance, on vxWorks, the NE2000 driver versus an MPC860 and zbufs
will provide totally different results. The MPC860 allows DMA of
chained buffers. Most TCP/IP stack use mbuf or chained buffers to
separate memory for separate protocol layers. With an NE2000 driver,
you must iterate through the buffers and hit a register with each,
doing a memory fetch. On the MPC860, this is all done in hardware.

You could have a PPC with an NE2000 card and the MPC860 at a
significantly slower clock could still handle more traffic.

Another case, I used lwip on an 8051 (25Mhz). The pathetic 8bit data
transfers make a maximum Ethernet transmission (large ping packets) of
about 5 kBps. By coding in assembler, I was able to raise this to 50
kBbps. However, the lwip stack has optimized the icmp response to
just turn the buffer around. So this was simply to transmit and
receive data! The management at this company were a bunch of idiots
(perhaps stupider than most), so I had to do this exercise to make
them realize this was impossible [hint: Ethernet is 10Mbps]. Merging
checksums with copies can some time and on dumb CISC archs asm can pay
off huge. In the 8051, you have something like x86 real mode segment
registers.

Small packet versus large packet protocols will affect figures. The
stack itself will have trade offs. Code size, memory foot print,
flexibility, RFC conformance, etc.

I have heard one megahertz per mega bit as a rule of thumb. Ie, you
need a GHz PC to handle Gbps ethernet. This is more an IT rule that
something that should be applied to a control system.

fwiw,
Bill Pringlemeir.

--
Having sex is like playing bridge. If you don't have a good
partner,you'd better have a good hand. - Unknown

vxWorks FAQ, "http://www.xs4all.nl/~borkhuis/vxworks/vxworks.html"

On Wed, 22 Jun 2005, jbl wrote:

> say "throw more bandwidth at it" because high speed processors designed
> for this environment (radiation hardened) are not yet available. An
> alternative, adding a new processor to handle network and/or router
> functions, has substantial cost in increased complexity, power and space
> required. Sorry to be so vague about this.

Oh, that's fine, but when some guy comes to this group a year from now and
tells us he's got a nuclear bomb plugged into his network but he doesn't
know the IP address and wants to know how he can figure it out, we're
going to send him to you.

8)

tom

--
eviscerated by obfuscation

"jbl" <(E-Mail Removed)> writes:

>I've been looking for some information on the web, in white papers and
>even on Usenet, but if it exists I can't seem to put together a search
>that finds it. I'm asking here if anyone knows of something along
>these lines and can point me to it.

There was a lot of ink spilled on this topic in the 1990s -- a bunch of
good papers. Unfortunately, the results aren't as clear cut as you'd
like.

It also gets much tougher if you try to make a linux box act like
a router (which it sounds as if you're interested in doing).

But let's be simple minded and say you're running as a host. No OSPF
or BGP -- just TCP or UDP in and out to a default route.

Then assuming a reasonably good TCP implementation and a poor quality buffering
system, you'll expend about 3 or 4 instructions (or, 3 or 4 memory accesses,
whichever is slower) per word of data sent via TCP, NOT INCLUDING DEVICE
DRIVER/INTERRUPT COSTS. Note that a high quality TCP with a good
buffering system/zero copy memory will be under half that cost, while
a poor TCP can easily push it higher.

Device driver and interrupt costs vary so widely based on your interface
hardware that it is hard to estimate. (In the old days, a factor of two
difference was common using different cards in the same box with the same
software -- haven't done tests on interfaces in ages, so can't say today).

UDP costs are more variable and can be higher or lower than TCP depending
on buffering design. Most UDP implementations are not optimized very
much.

In many ways you're probably best off running a test of TCP tests
where the application writes at different sizes and plotting size vs.
CPU costs. Make sure to have other applications running (so TCP has
at least some competition for memory) and have several other TCP
connections running with at least low levels of traffic (so your
TCB cache gets some work).

Craig

Tom Anderson wrote:
> On Wed, 22 Jun 2005, jbl wrote:
>
>> say "throw more bandwidth at it" because high speed processors
>> designed for this environment (radiation hardened) are not yet
>> available. An alternative, adding a new processor to handle network
>> and/or router functions, has substantial cost in increased complexity,
>> power and space required. Sorry to be so vague about this.
>
>
> Oh, that's fine, but when some guy comes to this group a year from now
> and tells us he's got a nuclear bomb plugged into his network but he
> doesn't know the IP address and wants to know how he can figure it out,
> we're going to send him to you.

I guess I'm an optimist, most of the rad hard stuff I've worked with was
space related, not atomic decay. Someone better qualified than I could
talk about design for one high level vs. decades of low level.

--
bill davidsen
SBC/Prodigy Yorktown Heights NY data center
http://newsgroups.news.prodigy.com