traffic management performance problem

Hello
I have linux router/nat with couple of thousand iproute2 rules.
I use htb or hfsc for traffic management.

My setup is:
Linux router with 2 interfaces

eth0 is an internet interface with 8189 aliased addresses
eth1 is an lan interface with 4000 users and private ip's

On linux router/nat machine i do netmap for my clients and ip(inside) -->
ip(outside) NAT

I have 8000 iptables rules and this work for me ...
System in that setup working propertly ("8000 pps" and 20Mbit/s internet
link). System load is 0.1 avg

But if i want to do some traffic shaping with htb of hfsc then i occure some
problem
This problem is wery high system load and high latency to internet (more
than 4000ms)
This load is do by two procs ksoftirqd0 and ksoftirqd1 (99% of 2CPU)
Machine setup is
- 2x P3 1,4GHz
- 2GB RAM
- 2x Netgear 620GA

I have 4000 filters/classes/sfq qdiscs (in lan direction)

My rules:

tc qdisc add dev imq0 root handle 1: htb default 11

tc class add dev imq0 parent 1: classid 1:11 htb rate 20Mbit ceil 20Mbit mpu
64 prio 2

# # 10.151.17.1 # #
tc class add dev imq0 parent 1:11 classid 1:100 htb rate 2kbit ceil 320kbit
burst 16k mpu 64 quantum 1500 prio 1
tc filter add dev imq0 protocol ip parent 1:0 prio 5 u32 match ip dst
10.151.17.1 flowid 1:100
tc qdisc add dev imq0 parent 1:100 handle 100: sfq perturb 5

# # 10.7.61.1 # #
tc class add dev imq0 parent 1:11 classid 1:101 htb rate 2kbit ceil 320kbit
burst 16k mpu 64 quantum 1500 prio 1
tc filter add dev imq0 protocol ip parent 1:0 prio 5 u32 match ip dst
10.7.61.1 flowid 1:101
tc qdisc add dev imq0 parent 1:101 handle 101: sfq perturb 5

...
...
...

tc class add dev imq0 parent 1:11 classid 1:4000 htb rate 2kbit ceil 320kbit
burst 16k mpu 64 quantum 1500 prio 1
tc filter add dev imq0 protocol ip parent 1:0 prio 5 u32 match ip dst
10.x.x.x flowid 1:4000
tc qdisc add dev imq0 parent 1:4000 handle 4000: sfq perturb 5

Any help ??? or idea for this problem ??

Pawel Staszewski wrote:
> Hello
> I have linux router/nat with couple of thousand iproute2 rules.
> I use htb or hfsc for traffic management.
>
> My setup is:
> Linux router with 2 interfaces
>
> eth0 is an internet interface with 8189 aliased addresses
> eth1 is an lan interface with 4000 users and private ip's
>
> On linux router/nat machine i do netmap for my clients and ip(inside)
-->
> ip(outside) NAT
>
> I have 8000 iptables rules and this work for me ...
> System in that setup working propertly ("8000 pps" and 20Mbit/s
internet
> link). System load is 0.1 avg
>
> But if i want to do some traffic shaping with htb of hfsc then i
occure some
> problem
> This problem is wery high system load and high latency to internet
(more
> than 4000ms)
> This load is do by two procs ksoftirqd0 and ksoftirqd1 (99% of 2CPU)
> Machine setup is
> - 2x P3 1,4GHz
> - 2GB RAM
> - 2x Netgear 620GA
>
> I have 4000 filters/classes/sfq qdiscs (in lan direction)
>
> My rules:
>
> tc qdisc add dev imq0 root handle 1: htb default 11
>
> tc class add dev imq0 parent 1: classid 1:11 htb rate 20Mbit ceil
20Mbit mpu
> 64 prio 2
>
> # # 10.151.17.1 # #
> tc class add dev imq0 parent 1:11 classid 1:100 htb rate 2kbit ceil
320kbit
> burst 16k mpu 64 quantum 1500 prio 1
> tc filter add dev imq0 protocol ip parent 1:0 prio 5 u32 match ip dst
> 10.151.17.1 flowid 1:100
> tc qdisc add dev imq0 parent 1:100 handle 100: sfq perturb 5
>
> # # 10.7.61.1 # #
> tc class add dev imq0 parent 1:11 classid 1:101 htb rate 2kbit ceil
320kbit
> burst 16k mpu 64 quantum 1500 prio 1
> tc filter add dev imq0 protocol ip parent 1:0 prio 5 u32 match ip dst
> 10.7.61.1 flowid 1:101
> tc qdisc add dev imq0 parent 1:101 handle 101: sfq perturb 5
>
> ..
> ..
> ..
>
> tc class add dev imq0 parent 1:11 classid 1:4000 htb rate 2kbit ceil
320kbit
> burst 16k mpu 64 quantum 1500 prio 1
> tc filter add dev imq0 protocol ip parent 1:0 prio 5 u32 match ip dst
> 10.x.x.x flowid 1:4000
> tc qdisc add dev imq0 parent 1:4000 handle 4000: sfq perturb 5
>
> Any help ??? or idea for this problem ??


You have a lot of confidence in freeware Qos solutions! HTB doesn't
scale
well, and your machine is underpowered. The combination is fatal.

Certainly get a faster machine, and consider getting a solid commercial
linux-based solution like the one offered at www.etinc.com. When you
hit the wall with HTB (as you may have already) you'll have a lot of
work to do and you'll still have to buy something anyway.

Tommy

(E-Mail Removed) wrote:

>
> Pawel Staszewski wrote:
>> Hello
>> I have linux router/nat with couple of thousand iproute2 rules.
>> I use htb or hfsc for traffic management.
>>
>> My setup is:
>> Linux router with 2 interfaces
>>
>> eth0 is an internet interface with 8189 aliased addresses
>> eth1 is an lan interface with 4000 users and private ip's
>>
>> On linux router/nat machine i do netmap for my clients and ip(inside)
> -->
>> ip(outside) NAT
>>
>> I have 8000 iptables rules and this work for me ...
>> System in that setup working propertly ("8000 pps" and 20Mbit/s
> internet
>> link). System load is 0.1 avg
>>
>> But if i want to do some traffic shaping with htb of hfsc then i
> occure some
>> problem
>> This problem is wery high system load and high latency to internet
> (more
>> than 4000ms)
>> This load is do by two procs ksoftirqd0 and ksoftirqd1 (99% of 2CPU)
>> Machine setup is
>> - 2x P3 1,4GHz
>> - 2GB RAM
>> - 2x Netgear 620GA
>>
>> I have 4000 filters/classes/sfq qdiscs (in lan direction)
>>
>> My rules:
>>
>> tc qdisc add dev imq0 root handle 1: htb default 11
>>
>> tc class add dev imq0 parent 1: classid 1:11 htb rate 20Mbit ceil
> 20Mbit mpu
>> 64 prio 2
>>
>> # # 10.151.17.1 # #
>> tc class add dev imq0 parent 1:11 classid 1:100 htb rate 2kbit ceil
> 320kbit
>> burst 16k mpu 64 quantum 1500 prio 1
>> tc filter add dev imq0 protocol ip parent 1:0 prio 5 u32 match ip dst
>> 10.151.17.1 flowid 1:100
>> tc qdisc add dev imq0 parent 1:100 handle 100: sfq perturb 5
>>
>> # # 10.7.61.1 # #
>> tc class add dev imq0 parent 1:11 classid 1:101 htb rate 2kbit ceil
> 320kbit
>> burst 16k mpu 64 quantum 1500 prio 1
>> tc filter add dev imq0 protocol ip parent 1:0 prio 5 u32 match ip dst
>> 10.7.61.1 flowid 1:101
>> tc qdisc add dev imq0 parent 1:101 handle 101: sfq perturb 5
>>
>> ..
>> ..
>> ..
>>
>> tc class add dev imq0 parent 1:11 classid 1:4000 htb rate 2kbit ceil
> 320kbit
>> burst 16k mpu 64 quantum 1500 prio 1
>> tc filter add dev imq0 protocol ip parent 1:0 prio 5 u32 match ip dst
>> 10.x.x.x flowid 1:4000
>> tc qdisc add dev imq0 parent 1:4000 handle 4000: sfq perturb 5
>>
>> Any help ??? or idea for this problem ??
>
>
> You have a lot of confidence in freeware Qos solutions! HTB doesn't
> scale
> well, and your machine is underpowered. The combination is fatal.
>
> Certainly get a faster machine, and consider getting a solid commercial
> linux-based solution like the one offered at www.etinc.com. When you
> hit the wall with HTB (as you may have already) you'll have a lot of
> work to do and you'll still have to buy something anyway.
>
> Tommy

Thanks for reply

But what about hfsc ?? people talking that hfsc is much faster than htb or
cbq ...
But i have the same problem with it ...

And I do some test on 4x P4 Xeon 2GHz and effect was the same

My kernel is 2.6.11 with latest iproute2-bk package...



etbwr is a little to expensive for my company
And i know that people do traffic management for more than 4000 users with
htb or cbq or hfsc

Yes yes yes machine is old and weak but if I get 2x P4 3.2GHz and do the
same thing that on 2xP3 1,4GHz does problem want back ?? ( or this change
nothing !change! )

About etbwr ... hmm some time ago i have demo of it... and i was
thunderstruck that etbwr have many many functions for traffic management
and is that scalable.

But my company does not have founds for me and my toys
So i must play with htb or hfsc of cbq and Linux

Pawel Staszewski wrote:
> Hello
> I have linux router/nat with couple of thousand iproute2 rules.
> I use htb or hfsc for traffic management.
>
> My setup is:
> Linux router with 2 interfaces
>
> eth0 is an internet interface with 8189 aliased addresses
> eth1 is an lan interface with 4000 users and private ip's
>
> On linux router/nat machine i do netmap for my clients and ip(inside) -->
> ip(outside) NAT
>
> I have 8000 iptables rules and this work for me ...
> System in that setup working propertly ("8000 pps" and 20Mbit/s internet
> link). System load is 0.1 avg
>
> But if i want to do some traffic shaping with htb of hfsc then i occure some
> problem
> This problem is wery high system load and high latency to internet (more
> than 4000ms)
> This load is do by two procs ksoftirqd0 and ksoftirqd1 (99% of 2CPU)
> Machine setup is
> - 2x P3 1,4GHz
> - 2GB RAM
> - 2x Netgear 620GA
>
> I have 4000 filters/classes/sfq qdiscs (in lan direction)
>
> My rules:
>
> tc qdisc add dev imq0 root handle 1: htb default 11
>
> tc class add dev imq0 parent 1: classid 1:11 htb rate 20Mbit ceil 20Mbit mpu
> 64 prio 2
>
> # # 10.151.17.1 # #
> tc class add dev imq0 parent 1:11 classid 1:100 htb rate 2kbit ceil 320kbit
> burst 16k mpu 64 quantum 1500 prio 1
> tc filter add dev imq0 protocol ip parent 1:0 prio 5 u32 match ip dst
> 10.151.17.1 flowid 1:100
> tc qdisc add dev imq0 parent 1:100 handle 100: sfq perturb 5
>
> # # 10.7.61.1 # #
> tc class add dev imq0 parent 1:11 classid 1:101 htb rate 2kbit ceil 320kbit
> burst 16k mpu 64 quantum 1500 prio 1
> tc filter add dev imq0 protocol ip parent 1:0 prio 5 u32 match ip dst
> 10.7.61.1 flowid 1:101
> tc qdisc add dev imq0 parent 1:101 handle 101: sfq perturb 5
>
> ..
> ..
> ..
>
> tc class add dev imq0 parent 1:11 classid 1:4000 htb rate 2kbit ceil 320kbit
> burst 16k mpu 64 quantum 1500 prio 1
> tc filter add dev imq0 protocol ip parent 1:0 prio 5 u32 match ip dst
> 10.x.x.x flowid 1:4000
> tc qdisc add dev imq0 parent 1:4000 handle 4000: sfq perturb 5
>
> Any help ??? or idea for this problem ??
>
>

IMQ could be the problem - do you really need to use it?

I have read of crashes aswell with 2.6.10.

If you really need it you could try an older kernel just to see, alot of
people seem to use IMQ when they could really do without.

Saying that - I use it (low rates) and have tested it upto 30k pps on a
1.83 gig atholon box - probably not as many rules as you though.

FWIW sfq perturb causes packet reordering so 5 may be a bit low - you
can also now reduce queue size with limit - though changing MAX_DEPTH in
the source could save more resources - I am often wrong though :-)

Depending on exactly why you need IMQ/your setup you may be be able to
patch the dummy device to do the same thing.

Andy.

Andy Furniss wrote:

> Pawel Staszewski wrote:
>> Hello
>> I have linux router/nat with couple of thousand iproute2 rules.
>> I use htb or hfsc for traffic management.
>>
>> My setup is:
>> Linux router with 2 interfaces
>>
>> eth0 is an internet interface with 8189 aliased addresses
>> eth1 is an lan interface with 4000 users and private ip's
>>
>> On linux router/nat machine i do netmap for my clients and ip(inside) -->
>> ip(outside) NAT
>>
>> I have 8000 iptables rules and this work for me ...
>> System in that setup working propertly ("8000 pps" and 20Mbit/s internet
>> link). System load is 0.1 avg
>>
>> But if i want to do some traffic shaping with htb of hfsc then i occure
>> some problem
>> This problem is wery high system load and high latency to internet (more
>> than 4000ms)
>> This load is do by two procs ksoftirqd0 and ksoftirqd1 (99% of 2CPU)
>> Machine setup is
>> - 2x P3 1,4GHz
>> - 2GB RAM
>> - 2x Netgear 620GA
>>
>> I have 4000 filters/classes/sfq qdiscs (in lan direction)
>>
>> My rules:
>>
>> tc qdisc add dev imq0 root handle 1: htb default 11
>>
>> tc class add dev imq0 parent 1: classid 1:11 htb rate 20Mbit ceil 20Mbit
>> mpu 64 prio 2
>>
>> # # 10.151.17.1 # #
>> tc class add dev imq0 parent 1:11 classid 1:100 htb rate 2kbit ceil
>> 320kbit burst 16k mpu 64 quantum 1500 prio 1
>> tc filter add dev imq0 protocol ip parent 1:0 prio 5 u32 match ip dst
>> 10.151.17.1 flowid 1:100
>> tc qdisc add dev imq0 parent 1:100 handle 100: sfq perturb 5
>>
>> # # 10.7.61.1 # #
>> tc class add dev imq0 parent 1:11 classid 1:101 htb rate 2kbit ceil
>> 320kbit burst 16k mpu 64 quantum 1500 prio 1
>> tc filter add dev imq0 protocol ip parent 1:0 prio 5 u32 match ip dst
>> 10.7.61.1 flowid 1:101
>> tc qdisc add dev imq0 parent 1:101 handle 101: sfq perturb 5
>>
>> ..
>> ..
>> ..
>>
>> tc class add dev imq0 parent 1:11 classid 1:4000 htb rate 2kbit ceil
>> 320kbit burst 16k mpu 64 quantum 1500 prio 1
>> tc filter add dev imq0 protocol ip parent 1:0 prio 5 u32 match ip dst
>> 10.x.x.x flowid 1:4000
>> tc qdisc add dev imq0 parent 1:4000 handle 4000: sfq perturb 5
>>
>> Any help ??? or idea for this problem ??
>>
>>
>
> IMQ could be the problem - do you really need to use it?
>
> I have read of crashes aswell with 2.6.10.
>
> If you really need it you could try an older kernel just to see, alot of
> people seem to use IMQ when they could really do without.
>
> Saying that - I use it (low rates) and have tested it upto 30k pps on a
> 1.83 gig atholon box - probably not as many rules as you though.
>
> FWIW sfq perturb causes packet reordering so 5 may be a bit low - you
> can also now reduce queue size with limit - though changing MAX_DEPTH in
> the source could save more resources - I am often wrong though :-)
>
> Depending on exactly why you need IMQ/your setup you may be be able to
> patch the dummy device to do the same thing.
>
> Andy.


I think that IMQ is not a problem because i try do the same thing on
phisical interface (eth0) and the same problem was occured.

Now i do some tests on 2x P4 2.66Ghz Xeon with HT enabled
I load 4000 htb rules on eth0 with sfq perturb 20
And .....
16:48:07 CPU %user %nice %system %iowait %irq %soft %idle
intr/s
16:48:08 all 0.00 0.00 0.00 0.00 1.00 28.25 70.75
6853.00
16:48:08 0 0.00 0.00 0.00 0.00 0.00 0.00 100.00
1000.00
16:48:08 1 0.00 0.00 0.00 0.00 2.00 37.00 59.00
3099.00
16:48:08 2 0.00 0.00 0.00 0.00 2.00 75.00 24.00
2753.00
16:48:08 3 0.00 0.00 0.00 0.00 0.00 0.00 100.00
0.00

16:48:08 CPU %user %nice %system %iowait %irq %soft %idle
intr/s
16:48:09 all 0.25 0.00 0.00 0.00 0.75 26.18 72.82
6794.00
16:48:09 0 0.00 0.00 0.00 0.00 0.00 1.00 99.00
1002.00
16:48:09 1 0.00 0.00 0.00 0.00 1.00 36.00 63.00
3015.00
16:48:09 2 0.00 0.00 0.00 0.00 2.00 68.00 29.00
2776.00
16:48:09 3 0.00 0.00 0.00 0.00 0.00 0.00 100.00
0.00

16:48:09 CPU %user %nice %system %iowait %irq %soft %idle
intr/s
16:48:10 all 0.00 0.00 0.00 0.00 0.75 26.25 73.00
6760.00
16:48:10 0 0.00 0.00 0.00 0.00 0.00 0.00 100.00
1000.00
16:48:10 1 0.00 0.00 0.00 0.00 2.00 34.00 65.00
2991.00
16:48:10 2 0.00 0.00 0.00 0.00 1.00 71.00 28.00
2768.00
16:48:10 3 1.00 0.00 0.00 0.00 0.00 0.00 100.00
0.00

16:48:10 CPU %user %nice %system %iowait %irq %soft %idle
intr/s
16:48:11 all 0.00 0.00 0.00 0.00 1.25 24.50 74.25
6740.00
16:48:11 0 0.00 0.00 0.00 0.00 0.00 0.00 100.00
1001.00
16:48:11 1 0.00 0.00 0.00 0.00 3.00 28.00 69.00
2897.00
16:48:11 2 0.00 0.00 0.00 0.00 2.00 70.00 28.00
2845.00
16:48:11 3 0.00 0.00 0.00 0.00 0.00 0.00 100.00
0.00

16:48:11 CPU %user %nice %system %iowait %irq %soft %idle
intr/s
16:48:12 all 0.00 0.00 0.00 0.00 0.75 23.00 76.25
6648.00
16:48:12 0 0.00 0.00 0.00 0.00 0.00 1.00 99.00
1001.00
16:48:12 1 0.00 0.00 0.00 0.00 2.00 25.00 74.00
2826.00
16:48:12 2 0.00 0.00 0.00 0.00 2.00 67.00 32.00
2818.00
16:48:12 3 0.00 0.00 0.00 0.00 0.00 0.00 100.00
0.00

16:48:12 CPU %user %nice %system %iowait %irq %soft %idle
intr/s
16:48:13 all 0.00 0.00 0.00 0.00 1.00 24.44 74.56
6577.00
16:48:13 0 0.00 0.00 0.00 0.00 0.00 0.00 100.00
1001.00
16:48:13 1 0.00 0.00 0.00 0.00 1.00 28.00 70.00
2837.00
16:48:13 2 0.00 0.00 0.00 0.00 2.00 69.00 29.00
2741.00
16:48:13 3 0.00 0.00 0.00 0.00 0.00 0.00 100.00
0.00


I bind irq from eth0 to P1 and eth1 to P2
7 root 35 19 0 0 0 S 7.8 0.0 2:21.91 ksoftirqd/2
5100 paol 16 0 14688 2120 1736 S 1.6 0.2 0:04.26 sshd
5 root 34 19 0 0 0 S 0.7 0.0 0:39.19 ksoftirqd/1
1 root 16 0 1580 504 444 S 0.0 0.1 0:03.24 init
2 root RT 0 0 0 0 S 0.0 0.0 0:00.04 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
4 root RT 0 0 0 0 S 0.0 0.0 0:00.09 migration/1
6 root RT 0 0 0 0 S 0.0 0.0 0:00.64 migration/2
8 root RT 0 0 0 0 S 0.0 0.0 0:00.04 migration/3
9 root 34 19 0 0 0 S 0.0 0.0 0:09.08 ksoftirqd/3

test:/proc/irq/24# vmstat 5 5
procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----
r b swpd free buff cache si so bi bo in cs us sy id
wa
1 0 0 767920 19088 34836 0 0 2 3 834 23 0 12 87
0
0 0 0 767920 19148 34844 0 0 0 22 6876 33 0 32 68
0
1 0 0 767584 19172 34820 0 0 0 12 6994 36 0 31 69
0
1 0 0 767664 19260 34868 0 0 0 37 6917 29 0 29 70
0
0 0 0 767664 19260 34868 0 0 0 0 6891 27 0 31 69
0
test:/proc/irq/24#

test:/proc/irq/24# iptables -L -n -v -t nat | wc -l
7398

test:/proc/irq/24# tc -s -d class show dev eth0 | grep "class htb" | wc -l
3563
test:/proc/irq/24#

So i think that more powerfull cpu can handle more htb or hfsc rules
But now most powerfull cpu have 3.6GHz so when i get 1000 cluents more then
3.6GHz cpu will be unpowered


So ... what now ???

Thanks for help

Pawel Staszewski wrote:

>
> test:/proc/irq/24# iptables -L -n -v -t nat | wc -l
> 7398
>
> test:/proc/irq/24# tc -s -d class show dev eth0 | grep "class htb" | wc -l
> 3563
> test:/proc/irq/24#
>
> So i think that more powerfull cpu can handle more htb or hfsc rules
> But now most powerfull cpu have 3.6GHz so when i get 1000 cluents more then
> 3.6GHz cpu will be unpowered
>
>
> So ... what now ???
>
> Thanks for help

I don't know whether it's to do with SMP or not - it could be.

On Monday I'll try 4000 HTB + SFQ at 20meg and see what load I get.

Andy.

Andy Furniss wrote:

> Pawel Staszewski wrote:
>
>>
>> test:/proc/irq/24# iptables -L -n -v -t nat | wc -l
>> 7398
>>
>> test:/proc/irq/24# tc -s -d class show dev eth0 | grep "class htb" | wc
>> -l 3563
>> test:/proc/irq/24#
>>
>> So i think that more powerfull cpu can handle more htb or hfsc rules
>> But now most powerfull cpu have 3.6GHz so when i get 1000 cluents more
>> then 3.6GHz cpu will be unpowered
>>
>>
>> So ... what now ???
>>
>> Thanks for help
>
> I don't know whether it's to do with SMP or not - it could be.
>
> On Monday I'll try 4000 HTB + SFQ at 20meg and see what load I get.
>
> Andy.

Ok i wait for test performed by You


I try do do some hashing filtering ... but my users have many different
subnets and i could do this easy...
And i have no idea for this ...

Pawel Staszewski wrote:
> (E-Mail Removed) wrote:
>
> >
> > Pawel Staszewski wrote:
> >> Hello
> >> I have linux router/nat with couple of thousand iproute2 rules.
> >> I use htb or hfsc for traffic management.
> >>
> >> My setup is:
> >> Linux router with 2 interfaces
> >>
> >> eth0 is an internet interface with 8189 aliased addresses
> >> eth1 is an lan interface with 4000 users and private ip's
> >>
> >> On linux router/nat machine i do netmap for my clients and
ip(inside)
> > -->
> >> ip(outside) NAT
> >>
> >> I have 8000 iptables rules and this work for me ...
> >> System in that setup working propertly ("8000 pps" and 20Mbit/s
> > internet
> >> link). System load is 0.1 avg
> >>
> >> But if i want to do some traffic shaping with htb of hfsc then i
> > occure some
> >> problem
> >> This problem is wery high system load and high latency to internet
> > (more
> >> than 4000ms)
> >> This load is do by two procs ksoftirqd0 and ksoftirqd1 (99% of
2CPU)
> >> Machine setup is
> >> - 2x P3 1,4GHz
> >> - 2GB RAM
> >> - 2x Netgear 620GA
> >>
> >> I have 4000 filters/classes/sfq qdiscs (in lan direction)
> >>
> >> My rules:
> >>
> >> tc qdisc add dev imq0 root handle 1: htb default 11
> >>
> >> tc class add dev imq0 parent 1: classid 1:11 htb rate 20Mbit ceil
> > 20Mbit mpu
> >> 64 prio 2
> >>
> >> # # 10.151.17.1 # #
> >> tc class add dev imq0 parent 1:11 classid 1:100 htb rate 2kbit
ceil
> > 320kbit
> >> burst 16k mpu 64 quantum 1500 prio 1
> >> tc filter add dev imq0 protocol ip parent 1:0 prio 5 u32 match ip
dst
> >> 10.151.17.1 flowid 1:100
> >> tc qdisc add dev imq0 parent 1:100 handle 100: sfq perturb 5
> >>
> >> # # 10.7.61.1 # #
> >> tc class add dev imq0 parent 1:11 classid 1:101 htb rate 2kbit
ceil
> > 320kbit
> >> burst 16k mpu 64 quantum 1500 prio 1
> >> tc filter add dev imq0 protocol ip parent 1:0 prio 5 u32 match ip
dst
> >> 10.7.61.1 flowid 1:101
> >> tc qdisc add dev imq0 parent 1:101 handle 101: sfq perturb 5
> >>
> >> ..
> >> ..
> >> ..
> >>
> >> tc class add dev imq0 parent 1:11 classid 1:4000 htb rate 2kbit
ceil
> > 320kbit
> >> burst 16k mpu 64 quantum 1500 prio 1
> >> tc filter add dev imq0 protocol ip parent 1:0 prio 5 u32 match ip
dst
> >> 10.x.x.x flowid 1:4000
> >> tc qdisc add dev imq0 parent 1:4000 handle 4000: sfq perturb 5
> >>
> >> Any help ??? or idea for this problem ??
> >
> >
> > You have a lot of confidence in freeware Qos solutions! HTB doesn't
> > scale
> > well, and your machine is underpowered. The combination is fatal.
> >
> > Certainly get a faster machine, and consider getting a solid
commercial
> > linux-based solution like the one offered at www.etinc.com. When
you
> > hit the wall with HTB (as you may have already) you'll have a lot
of
> > work to do and you'll still have to buy something anyway.
> >
> > Tommy
>
> Thanks for reply
>
> But what about hfsc ?? people talking that hfsc is much faster than
htb or
> cbq ...
> But i have the same problem with it ...
>
> And I do some test on 4x P4 Xeon 2GHz and effect was the same
>
> My kernel is 2.6.11 with latest iproute2-bk package...
>
>
>
> etbwr is a little to expensive for my company
> And i know that people do traffic management for more than 4000 users
with
> htb or cbq or hfsc
>
> Yes yes yes machine is old and weak but if I get 2x P4 3.2GHz and
do the
> same thing that on 2xP3 1,4GHz does problem want back ?? ( or this
change
> nothing !change! )
>
> About etbwr ... hmm some time ago i have demo of it... and i was
> thunderstruck that etbwr have many many functions for traffic
management
> and is that scalable.
>
> But my company does not have founds for me and my toys
> So i must play with htb or hfsc of cbq and Linux

That is rather short-sighted. Are they not paying you to waste
your time with stuff that doesn't work? How much do you pay for
bandwidth? It seems that the cost of such a product can be
recovered rather quickly. Maybe they can fire you to recover
the cost?

also, how many pps do the people with 4000 customers have and how
many classes? All of these things are factors. There is a reason
that people pay for things. They are worth the cost.

Tommy

But won't buying such a machine cost more than getting bandwidth
manager software that works better? People are very funny. They
pay twice for a machine what they would need to so they can get the
free software to work better. Its very strange.

Paweł Staszewski wrote:
> Andy Furniss wrote:
>
>
>>Pawel Staszewski wrote:
>>
>>
>>>test:/proc/irq/24# iptables -L -n -v -t nat | wc -l
>>>7398
>>>
>>>test:/proc/irq/24# tc -s -d class show dev eth0 | grep "class htb" | wc
>>>-l 3563
>>>test:/proc/irq/24#
>>>
>>>So i think that more powerfull cpu can handle more htb or hfsc rules
>>>But now most powerfull cpu have 3.6GHz so when i get 1000 cluents more
>>>then 3.6GHz cpu will be unpowered
>>>
>>>
>>>So ... what now ???
>>>
>>>Thanks for help
>>
>>I don't know whether it's to do with SMP or not - it could be.
>>
>>On Monday I'll try 4000 HTB + SFQ at 20meg and see what load I get.
>>
>>Andy.
>
>
> Ok i wait for test performed by You
>
>
> I try do do some hashing filtering ... but my users have many different
> subnets and i could do this easy...
> And i have no idea for this ...
>

Well I didn't get to test your setup exactly as I don't have much of a
LAN or any PCs with two eth cards.

What I did do is generate 8000pps with netperf and then added tc filter
rules till I ran out of CPU - I only got to 1800.

I don't know how much more I could get if I were just forwarding rather
than generating - but I think optimising you rules by hashing filters or
using iptables is the way to go first.

The hashing chapter in lartc looks complicated, but from the ranges you
posted in the other thread I think you could do it with way way less
than a worst case of 4000 matches per packet.

It may be easier to use netfilter patches (iprange IIRC) and the
classify target.

As for 4000 htb classes - you would give the users a better experience
if you only use them for bulk and priorotise interactive dns - empty
acks etc.

If you are going to have 4000 sfq (& sfq is really best for bulk
traffic) I would also change SFQ_DEPTH from 128 to 16 in
net/sched/sch_sfq.c.

Andy.