tc and bandwidth limiting: What am I doing wrong?

So I thought I'd limit the bandwidth on one of my network interfaces to
run a few tests on a few things. I tried:

tc qdisc add dev eth1 root tbf burst 20480 limit 20480 \
mtu 1514 rate 32000bps

....as was mentioned in the Traffic Control HOWTO. However, sustained
transfer rates weren't 64K/sec, but were 15K/sec. Doubling and
quadrupling the numbers for rate, burst, and limit made no difference to
the sustained transfer rates I observed.

Kernel 2.6.22.1 (vanilla), iproute2 2.6.22.20070710 . Is the HOWTO
outdated? Am I doing something stupid? Is there an easier way? The
machines are using forcedeth and e1000 cards, and sustained transfer
rates without tc are ~10M/sec. If you need more information, say what
you need and I'll find it. Thanks in advance.

(I ended up limiting bandwidth with an old 10bT DEC Tulip card I had
lying around. Yay for obsolete hardware, I guess.)

--
"Depress the button," it said. Gloria looked at the button. "You're
a very ugly button," she began. --Phil Janes, _The Galaxy Game_
My blog and resume: http://crow202.dyndns.org:8080/wordpress/
Matt G|There is no Darkness in Eternity/But only Light too dim for us to see

On Jun 14, 10:25*am, Dances With Crows <danceswithcr...@usa.net>
wrote:

> tc qdisc add dev eth1 root tbf burst 20480 limit 20480 \
> mtu 1514 rate 32000bps

> ...as was mentioned in the Traffic Control HOWTO. *However, sustained
> transfer rates weren't 64K/sec, but were 15K/sec. *Doubling and
> quadrupling the numbers for rate, burst, and limit made no difference to
> the sustained transfer rates I observed.

It's hard to figure out what exactly happened. You claim you set a
traffic limit of 32,000 bits per second, and expected a transfer rate
of 64,000 somethings per second (bits? bytes?) but got 15,000. Why
would you expect 64,000 if you asked for 32,000? No conceivable bits/
bytes conversion can make that work.

What was the command you used? What was the rate you got (and specify
if it's in bits or bytes) and what did you expect? And how did you
measure it?

The problem may be that you are confusing bits with bytes. It also may
be that you are expecting *application* cooked data throughput to
equal raw network throughput.

It's hard to tell though because your command seems not to match any
expectation, with or without such errors.

DS

David Schwartz staggered into the Black Sun and said:
> Dances With Crows wrote:
>> tc qdisc add dev eth1 root tbf burst 20480 limit 20480 \
>> mtu 1514 rate 32000bps

>> ...as was mentioned in the Traffic Control HOWTO. However, sustained
>> transfer rates weren't 64K/sec, but were 15K/sec.
> It's hard to figure out what exactly happened. You claim you set a
> traffic limit of 32,000 bits per second, and expected a transfer rate
> of 64,000 somethings per second (bits? bytes?) but got 15,000.

That's 64K/sec (kilobytes) expected, got 15K/sec. (Actually I should
have expected 32K/sec, but 32 still != 15.)

> What was the command you used?

See first paragraph. That was the exact syntax given in the HOWTO for
limiting an interface to 256 kbit/second. All right, they might have
gotten their math wrong, but I usually expect HOWTOs to be reasonably
accurate. Please also note that removing the rule, then replacing it
with the numbers doubled, quadrupled or *ed by 8 for burst, limit, and
rate produced the same measured 15K/sec. I said that in the first
message, but it's worth repeating.

> And how did you measure it?

Sustained transfer rate using scp.

> It also may be that you are expecting *application* cooked data
> throughput to equal raw network throughput.

The overhead isn't *that* high with scp, and see above.

> It's hard to tell though because your command seems not to match any
> expectation, with or without such errors.

That's sort of why I posted here, since what I saw is so different from
what I expected. So: Who's using tc, how are you using it, and how
do you make it work the way you want it to?

--
That which does not kill us makes us stranger.
--Trevor Goodchild, Aeon Flux
My blog and resume: http://crow202.dyndns.org:8080/wordpress/
Matt G|There is no Darkness in Eternity/But only Light too dim for us to see

David Schwartz wrote:
> On Jun 14, 10:25 am, Dances With Crows <danceswithcr...@usa.net>
> wrote:
>
>> tc qdisc add dev eth1 root tbf burst 20480 limit 20480 \
>> mtu 1514 rate 32000bps
>
> It's hard to figure out what exactly happened. You claim you set a
> traffic limit of 32,000 bits per second, and expected a transfer rate
> of 64,000 somethings per second (bits? bytes?)

Just to make things even more confusing TC reads bps as bytes/sec you
have to use bit/kbit/mbit for bits/sec.

When using tc on gig nics you really need to turn off tcp segmentation
offload with IIRC ethtool -k. With it on you can get strange results.

When you put qdiscs on the root of eths you need to consider arp - these
will be delayed/dropped as well - which can mess things up.

Andy.

One last clarification -- changing the configured 'tc' *rate* had no
effect on the transfer speed?!

DS

David Schwartz staggered into the Black Sun and said:
> One last clarification -- changing the configured 'tc' *rate* had no
> effect on the transfer speed?!

The bits in my notes here say that:

tc qdisc add dev eth1 root tbf burst 131122 limit 131122 mtu 1514 \
rate 2012144

gave the same results as--

tc qdisc add dev eth1 root tbf burst 262244 limit 262244 mtu 1514 \
rate 4024288

which gave the same results as--

tc qdisc add dev eth1 root tbf burst 524488 limit 524488 mtu 1514 \
rate 8048576

....which is why I said that I was using 2.6.22.1 (vanilla) and iproute2
2.6.22.20070710 , just in case there was some sort of stupid bug in
either/both that was fixed in later revisions.

--
I realised I was dyslexic after attending a toga party dressed
as a goat. --Author unknown
My blog and resume: http://crow202.dyndns.org:8080/wordpress/
Matt G|There is no Darkness in Eternity/But only Light too dim for us to see

Andy Furniss staggered into the Black Sun and said:
> David Schwartz wrote:
>> On Jun 14, 10:25 am, Dances With Crows <danceswithcr...@usa.net>
>> wrote:
>>> tc qdisc add dev eth1 root tbf burst 20480 limit 20480 \
>>> mtu 1514 rate 32000bps
>> It's hard to figure out what exactly happened. You claim you set a
>> traffic limit of 32,000 bits per second, and expected a transfer rate
>> of 64,000 somethings per second (bits? bytes?)
> Just to make things even more confusing TC reads bps as bytes/sec, you
> have to use bit/kbit/mbit for bits/sec.
>
> When using tc on gig nics you really need to turn off tcp segmentation
> offload with IIRC ethtool -k. With it on you can get strange results.

eth1 in this case was a forcedeth CK804 10/100 card. The other
machine's eth0 is an e1000, but since the tc'ing is being done on the
machine with the forcedeth, I didn't think that would make a difference.
Maybe it does. I can check later.

> When you put qdiscs on the root of eths you need to consider arp -
> these will be delayed/dropped as well - which can mess things up.

This could have screwed things up as well. However, I tried this as
well:

tc qdisc add dev eth1 handle 1:0 root dsmark indices 1 default_index 0
tc qdisc add dev eth1 handle 2:0 parent 1:0 tbf burst 20480 limit 20480
mtu 1514 rate 32000bps

....and got exactly the same results. Yeah, I'm confused, which is why I
posted here, hoping to find someone who knows more about tc than I do.

--
Jesus is the best radio producer in the beans. We need some saliva
and pickles to get mad. --MegaHAL, "The Best of MegaHAL"
My blog and resume: http://crow202.dyndns.org:8080/wordpress/
Matt G|There is no Darkness in Eternity/But only Light too dim for us to see

Dances With Crows wrote:

>> When using tc on gig nics you really need to turn off tcp segmentation
>> offload with IIRC ethtool -k. With it on you can get strange results.
>
> eth1 in this case was a forcedeth CK804 10/100 card. The other
> machine's eth0 is an e1000, but since the tc'ing is being done on the
> machine with the forcedeth, I didn't think that would make a difference.
> Maybe it does. I can check later.
>
>> When you put qdiscs on the root of eths you need to consider arp -
>> these will be delayed/dropped as well - which can mess things up.
>
> This could have screwed things up as well. However, I tried this as
> well:
>
> tc qdisc add dev eth1 handle 1:0 root dsmark indices 1 default_index 0
> tc qdisc add dev eth1 handle 2:0 parent 1:0 tbf burst 20480 limit 20480
> mtu 1514 rate 32000bps
>
> ...and got exactly the same results. Yeah, I'm confused, which is why I
> posted here, hoping to find someone who knows more about tc than I do.
>

Strange - I have a forcedeth 100mbit nic and just tried your tbf rule
from the first post and it works OK.

Could be unlucky kernel version I suppose, or maybe timers, if you have
multicore/smp you shouldn't use tsc. I haven't tested noHZ on the box I
shape on I still use Hz=1000.

Does htb work eg -

tc qdisc del dev eth1 root

Forgetting to do this between tests can sometimes cause weirdness

tc qdisc add dev eth1 root handle 1:0 htb
tc class add dev eth1 parent 1:0 classid 1:1 htb rate 256kbit
tc qdisc add dev eth1 parent 1:1 bfifo limit 20k
tc filter add dev eth1 protocol ip parent 1:0 u32 match u32 0 0 classid 1:1

It's not quite the same - ie it doesn't catch arp and I didn't use
burst/cburst because it doesn't seem to behave the same as burst on tbf.