pushing beyond 400mbit?

I have a performance issue. I have a box setup to proxy http traffic using
DNAT/SNAT rules. (So no userspace involved.)

p
client <-> r
o
server <-> x
y

All traffic comes and goes through the same interface. Hardware is DELL
1850
The network card is an Intel e1000, 66Mhz 32-bit pci (not the onboard)
Kernel is 2.6.14.3. e1000 driver as module default parameters.

Performace is stuk at ~46'700'000 byte/sec (rx and tx). Pinging the box at
this moment gives me huge amounts of packetloss so something is chocking.
But what?

Could I be hitting PCI bandwith limits? Assuming packets are transferd
from the card to main memory, and then back to the card for sending off
again I have <100 MB/sec to transfer. So that should be withing limits
right?

Also looked at some other stuff. Average over 10 seconds the card does:

rx: 46695389 byte/sec
51857 packet/sec
900 byte/packet avg.

tx is the same as what comes in also goes out again.

The number of interrupts for this card flat-lines at 8k/second. Which is
what the driver limits the card to. If there are more 'events' multiple
will be queued for a single interrupt.

The card uses a default number of 256 rx and tx descriptors which means
(if I understand correctly) that it could queue that many incoming packets
per interrupt it generates.

@ 51857 packet/sec and 8k int/second it only has to queue ~6.5 * 2 (tx and
rx) packets/events between interrupts. So I'm also nowhere near that limit.

The box is a 2.8Ghz XEON and CPU util shows: Cpu(s): 0.7% us, 0.2% sy,
0.0% ni, 89.2% id, 0.0% wa, 2.3% hi, 7.5% si So that's OK too.

What could it be? Any hints/clues as to what could be the bottleneck
and/or how I can locate it would be helpful.

Regards,
Mark.

Mark <(E-Mail Removed)> writes:


>I have a performance issue. I have a box setup to proxy http traffic using
>DNAT/SNAT rules. (So no userspace involved.)

> p
>client <-> r
> o
>server <-> x
> y

>All traffic comes and goes through the same interface. Hardware is DELL
>1850
>The network card is an Intel e1000, 66Mhz 32-bit pci (not the onboard)
>Kernel is 2.6.14.3. e1000 driver as module default parameters.

>Performace is stuk at ~46'700'000 byte/sec (rx and tx). Pinging the box at

And that is a problem why?

>this moment gives me huge amounts of packetloss so something is chocking.

That is a problem. Maybe cabling. What kinds of cables connect you to the
GB router?


>But what?

>Could I be hitting PCI bandwith limits? Assuming packets are transferd

Could be.

>from the card to main memory, and then back to the card for sending off
>again I have <100 MB/sec to transfer. So that should be withing limits
>right?

>Also looked at some other stuff. Average over 10 seconds the card does:

>rx: 46695389 byte/sec
> 51857 packet/sec
> 900 byte/packet avg.

>tx is the same as what comes in also goes out again.

>The number of interrupts for this card flat-lines at 8k/second. Which is
>what the driver limits the card to. If there are more 'events' multiple
>will be queued for a single interrupt.

>The card uses a default number of 256 rx and tx descriptors which means
>(if I understand correctly) that it could queue that many incoming packets
>per interrupt it generates.

>@ 51857 packet/sec and 8k int/second it only has to queue ~6.5 * 2 (tx and
>rx) packets/events between interrupts. So I'm also nowhere near that limit.

>The box is a 2.8Ghz XEON and CPU util shows: Cpu(s): 0.7% us, 0.2% sy,
>0.0% ni, 89.2% id, 0.0% wa, 2.3% hi, 7.5% si So that's OK too.

>What could it be? Any hints/clues as to what could be the bottleneck
>and/or how I can locate it would be helpful.

>Regards,
>Mark.

Might be good to look at link-level stats - eg ethtool.

Might be good to look at perf w/o the rules in place.

Might be good to look at unidrectional perf with something like netperf.

Sanity checking the top output might not be bad either - say a CPU
profile a la oprofile or q-syscollect.

If the interrupt throttle is precluding the card from being fed new
buffers quickly enough, you could disable the throttle. When you
modprobe specify "InterruptThrottleRate=N,N..." where you have the
comma-separated list for each of the interfaces "driven" by the e1000
driver. I suspect there is a config file (modules.conf?) you could
put that, and perhaps even later versions of ethtool can modify a
single interface on the fly.

Might be good to see if your harness' ability to generate load is tied
to the latency through the device.

rick jones
--
The glass is neither half-empty nor half-full. The glass has a leak.
The real question is "Can it be patched?"
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

On Mon, 24 Jul 2006 00:37:14 +0200, Mark <(E-Mail Removed)> wrote:

>
>I have a performance issue. I have a box setup to proxy http traffic using
>DNAT/SNAT rules. (So no userspace involved.)
>
>The number of interrupts for this card flat-lines at 8k/second. Which is
>what the driver limits the card to. If there are more 'events' multiple
>will be queued for a single interrupt.
>
>What could it be? Any hints/clues as to what could be the bottleneck
>and/or how I can locate it would be helpful.
>
>Regards,
>Mark.

I can't get my motherboard to see my gigabit NIC, so this is a wild
guess. Try using "jumbo frames".
--
buck

> I can't get my motherboard to see my gigabit NIC, so this is a wild
> guess. Try using "jumbo frames".

If the average bytes per packet is indeed only 900 bytes per the OP:

> Also looked at some other stuff. Average over 10 seconds the card
> does:
>
> rx: 46695389 byte/sec
> 51857 packet/sec
> 900 byte/packet avg.

and that hasn't been "deflated" by standalone ACKs, it would suggest
that JumboFrames won't buy all that much. Still worth considering,
since just knowing the average isn't the complete picture, and those
ACK's could indeed be dragging-down the average, but I'd not get hope
too high.

rick jones
--
Process shall set you free from the need for rational thought.
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Rick Jones <(E-Mail Removed)> wrote in news:97bxg.1019$ik1.782
@news.cpqcorp.net:

>> I can't get my motherboard to see my gigabit NIC, so this is a wild
>> guess. Try using "jumbo frames".
>
> If the average bytes per packet is indeed only 900 bytes per the OP:
>
>> Also looked at some other stuff. Average over 10 seconds the card
>> does:
>>
>> rx: 46695389 byte/sec
>> 51857 packet/sec
>> 900 byte/packet avg.
>
> and that hasn't been "deflated" by standalone ACKs, it would suggest
> that JumboFrames won't buy all that much. Still worth considering,
> since just knowing the average isn't the complete picture, and those
> ACK's could indeed be dragging-down the average, but I'd not get hope
> too high.
>
> rick jones

We did some tests of PCI NICs a while back and found a big difference from
motherboard to motherboard - maybe depends on the PCI chipset. OK that was
with 100BaseT cards but some motherboards PCI implementations seemed to
choke running five Nics flat out whereas others breezed it. Is the Gigabit
card you are using just PCI? If your motherboard supports it then a PCI-X
card might give better results.

Klazmon.