multiple classes with target CLASSIFY

Hi,

I have a router where I use a CLASSIFY target on FORWARD chain to do
QoS.
Now I need a packet to be classified to 2 classes. The following:
iptables -t mangle -A FORWARD -s $SOME_IP -j CLASSIFY --set-class
1:$FIRST
iptables -t mangle -A FORWARD -s $SOME_IP -j CLASSIFY --set-class
1:$SECOND
doesn't work - only the last rule takes effect.

Intuitively I wrote two classes seperated with a comma (no space):
iptables -t mangle -A FORWARD -s $SOME_IP -j CLASSIFY --set-class
1:$FIRST,1:$SECOND
no errors... but only the 1:$FIRST class receives packets.

Couldn't find anything about it but it seems to be wrong, doesn't it?
Could someone test this rule? (should be easy - just replace an
existing rule with two classes and see if the second class works)

--
Tomek Gruca


Tomek Gruca

Tomek Gruca wrote:
> Hi,
>
> I have a router where I use a CLASSIFY target on FORWARD chain to do
> QoS.
> Now I need a packet to be classified to 2 classes. The following:

I can't see that this makes sense - you can't have a packet in two
queues at once.

What are you trying to do that needs different classes?

Andy.

On 9 Lut, 21:41, Andy Furniss <s...@andyfurniss.entadsl.com> wrote:
> What are you trying to do that needs different classes?

I have 2 ISPs and per port load balancing. I want my users to have
specified downstream (let's say 2Mb/s) no matter what ports they use.
So I set a class for each user where root class has unlimited bitrate.
But also I need queues on both links to avoid uncontrolled overload.

One solution is to use IMQ but I don't want to patch the kernel.

Another is to CLASSIFY in FORWARD and then in POSTROUTING (and it
works) but I need the "-i $DEV" match...

--
Tomek

On 10 Lut, 02:00, Tomek Gruca <tomekgr...@gmail.com> wrote:
> Another is to CLASSIFY in FORWARD and then in POSTROUTING (and it
> works) but I need the "-i $DEV" match...

Sorry, in POSTROUTING it doesn't work... it overrides FORWARD
CLASSIFY.

"you can't have a packet in two queues at once"
you mean it's technically impossible or you don't see a reason for
this?

--
Tomek

Tomek Gruca wrote:

> "you can't have a packet in two queues at once"
> you mean it's technically impossible or you don't see a reason for
> this?

I mean technically impossible.

You can using imq/ifb and set things up so the packet goes through more
than one queue, though.

IFB has been in 2.6s for quite a time now as a replacement for imq. I
don't know if all distros enable it - try modprobe ifb.

It's not quite the same as imq as on ingress it sees the traffic before
netfilter whereas imq is after prerouting/de-nat. You can use it on
egress aswell as ingress.

Depending on exactly how complicated you need your classification/qos
policied to be, one option I would consider is to just use policers.

Shaping from the wrong end of a wan is never perfect and you need to
sacrifice some bandwidth whether you police or shape. The nice thing
about policers is that they don't add latency.

You could police the wans at say 80% of their link speed with one
policer for each and have a 2meg per user policer on the lan interface.

It all really depends on exactly what you are trying to acheive, what
bandwidth you have, how many users, what they do etc etc

Andy.

On 10 Lut, 14:06, Andy Furniss <s...@andyfurniss.entadsl.com> wrote:
> I mean technically impossible.
>
> You can using imq/ifb and set things up so the packet goes through more
> than one queue, though.
>
> IFB has been in 2.6s for quite a time now as a replacement for imq. I
> don't know if all distros enable it - try modprobe ifb.
>
> It's not quite the same as imq as on ingress it sees the traffic before
> netfilter whereas imq is after prerouting/de-nat. You can use it on
> egress aswell as ingress.
>

Thanks,

I have IFB as kernel module (modprobe ifb works), bo I'm not sure how
to test it..
I have two classes:
tc class add dev eth0 parent 1:0 classid 1:1 htb rate 100kbit
tc class add dev ifb0 parent 2:0 classid 2:1 htb rate 100kbit

how could I make a packet to be classified to both of them?
I'm not sure if I understand actions correctly (probably not), can't
find any tutorial:
Let's say the packet is maked by iptables with mark=1001 in FORWARD
tc filter add dev eth0 protocol ip handle 1001 fw flowid 1:1 \
action ipt -j MARK --set-mark 2001 \
action mirred egress redirect dev ifb0
tc filter add dev ifb0 protocol ip handle 2001 fw flowid 2:1
Does it make sense? The first rule classifies it to 1:1 and passes on?

> Depending on exactly how complicated you need your classification/qos
> policied to be, one option I would consider is to just use policers.
>

Well... I've always used HTB for fair per user distribution of
bandwidth..
Don't know anything about policers but gonna read sth

> Shaping from the wrong end of a wan is never perfect and you need to
> sacrifice some bandwidth whether you police or shape. The nice thing
> about policers is that they don't add latency.
>

Could you explain a bit these 3 sentences ? Each seems to be very
interesting..

> You could police the wans at say 80% of their link speed with one
> policer for each and have a 2meg per user policer on the lan interface.
>
> It all really depends on exactly what you are trying to acheive, what
> bandwidth you have, how many users, what they do etc etc
>

I have one 15/0.8M (for asymmeric services: http,pop3 etc) and 6/6M
for the rest.
150 users, bandwidth per user varies.
I'd need to be able to specify CIR for a user on each link..

--
Tomek

Tomek Gruca wrote:

I will try and test/post some real rules soon, but this is a bit of an
untypical setup that I haven't done before, so I'll be in
trial/error/test/repeat mode and I won't have time for a couple of days.

> I have IFB as kernel module (modprobe ifb works), bo I'm not sure how
> to test it..
> I have two classes:
> tc class add dev eth0 parent 1:0 classid 1:1 htb rate 100kbit
> tc class add dev ifb0 parent 2:0 classid 2:1 htb rate 100kbit
>
> how could I make a packet to be classified to both of them?
> I'm not sure if I understand actions correctly (probably not), can't
> find any tutorial:
> Let's say the packet is maked by iptables with mark=1001 in FORWARD
> tc filter add dev eth0 protocol ip handle 1001 fw flowid 1:1 \
> action ipt -j MARK --set-mark 2001 \
> action mirred egress redirect dev ifb0
> tc filter add dev ifb0 protocol ip handle 2001 fw flowid 2:1
> Does it make sense? The first rule classifies it to 1:1 and passes on?

I am guessing that you are doing nat on the wan interfaces, if so this
complicates things somewhat. If you are not doing nat then the easy
solution would be to direct the packets to ifb as soon as they arrive at
the wan nic(s). If you are doing nat and have the need to seperate users
then this will not work because you won't be able to see the local IP
addresses or use any iptables rules to aid your classification.

So assuming this, you are going to have to try and do everything as the
packets leave you lan facing interface - this is a bit harder as in
order to prevent loops, packets redirected to ifb when about to leave an
interface can't directly be reclassified by another qdisc on that
interface.

What should work is to classify them before the redirect and then when
they return from ifb I beleive any previous classification will be
reinstated. So if you both classify and mark it may be possible to use
the marks on the two ifbs for the link specific qdiscs and the
classification for the third qdisc on the real lan facing interface.

>
>> Depending on exactly how complicated you need your classification/qos
>> policied to be, one option I would consider is to just use policers.
>>
>
> Well... I've always used HTB for fair per user distribution of
> bandwidth..
> Don't know anything about policers but gonna read sth

Policers don't queue packets they just drop or let pass depending on
whether the configured rate is exceeded (oversimplification). To make
this work they keep a virtual buffer whose length is set with the burst
parameter.

They do have a little used feature called shared meters which allows for
more complicated setups. Personally I think shared meter is a bit of a
misnomer shared policer would probably be better as they don't allow
different actions for different traffic that hits them. You may be able
to do a simple CIR with them, but on 2.6 kernels when used on ingress
they will see the traffic before it's been de-natted. On 2.4s they do
hook after de-nat. You can use them on egress aswell as ingress. So you
may be able to use them for your setup with ifb.


>
>> Shaping from the wrong end of a wan is never perfect and you need to
>> sacrifice some bandwidth whether you police or shape. The nice thing
>> about policers is that they don't add latency.

You have 15 & 6mbit that's not too bad. The less bandwidth you have the
harder things get when shaping from the wrong end of the bottleneck.

Take your 6/6 link as an example. For traffic headed from you to the wan
you have near enough total control - you put a qdisc on the interface
and set it to 6mbit and you have on your box any queue that builds up.
Your classification can do fairness and allow any interactive packets to
skip past the lower prio packets so they don't get delayed.

So what about traffic coming from the wan to your net - you can catch it
somewhere, but what rate do you set on the qdisc. it's never going to
come in faster than 6mbit so if you set 6mbit as the rate nothing will
happen, you will never build up a queue to be able to do fairness/prio
for some.

Exactly what happens will depend on what your ISP does - there may be a
big tail drop buffer that fills up so latency for all traffic suffers,
or maybe they police rate so latency is not so bad but you will get a
bit of loss and will be unable to do fairness. The way tcp works means
it will try and fill any buffer at the far end of your wan.

In practice what you do is have to compromise by setting a rate somewhat
lower than link speed and understanding that say at 80% the qdisc is
only going to fill up slowly as in the time 10 packets come in at 6meg
you will have dequeued 8. The way tcp slowstart works means that until
you get loss it will ramp up speed - so there is an incentive to use
shortish queues, but if you set them too short you hurt throughput, and
if you have lots of classes you can't be too mean to each class yet if
they are all active the length may add up to a queue that's a bit too
long. ESFQ can be handy here, but requires patching currently - sfq is
currently being enhanced aswell, but it's not mainstream yet.

Policers have another advantage for ingress - if you buffer then drop
it's a tail drop, so the tcp sender doesn't know to backoff until the
packets ahead in the queue have been dequeued - if in slowstart the
speed will still be rising and your ISP buffer still filling.

All of the above is a bit over simplistic but should give an idea of the
problems. As I said the lower the bandwidth the worse it is and you have
a reasonable amount, so all is not lost - even at lower rates it's
alot better than doing nothing, but it does mean sacrificing some
bandwidth and dropping packets that have already come in on your line.

Andy.

Andy Furniss wrote:

> I will try and test/post some real rules soon, but this is a bit of an
> untypical setup that I haven't done before, so I'll be in
> trial/error/test/repeat mode and I won't have time for a couple of days.

Well I tried and it seems to work - I only tested for classification and
I did it on my desktop so you would have to change things for a your
router.

I couldn't get -j CLASSIFY to work with ifb so you'll have to use marks
or tc filters direct or a combination. I used marks and masks just as an
example of using marks in a more complicated than normal way. I also
show one way to control queue length - you can use bfifos or sfq with
limit as well.

If I had to do this for real I would consider hashing or policers
instead of htb - but then I don't know exactly your requirements.

I just tested by adding the two addresses to my eth and used
my gateway 192.168.0.1 running netserver as the target

ping -I 192.168.0.101 192.168.0.1
ping -I 192.168.0.102 192.168.0.1

netperf -H 192.168.0.1 -l 2 -- -L 192.168.0.101
netperf -H 192.168.0.1 -l 2 -- -L 192.168.0.102

and looked at the counters with

tc -s class ls dev eth0/ifb0/ifb1

Looked OK to me but don't take my word that it works.

FWIW if you ever use 100s of netperfs in a script to generate traffic
you'll need to add all the addresses you use and 0.0.0.0 to /etc/hosts
or it will try and do loads of DNS lookups which will mess things up
(well it does for me).

The script - bound to be lame in many ways - I would be messing around
for days/weeks if it were for real, but it should show you how to use
ifb on egress at least.

I hope your distro enabled the u32 mark match!
Change the paths & mend any wrapping.

Andy.

#set -x
IP=/usr/sbin/ip
TC=/usr/sbin/tc
IPT=/usr/local/sbin/iptables

$TC qdisc del dev ifb0 root &>/dev/null
$TC qdisc del dev ifb1 root &>/dev/null
$TC qdisc del dev eth0 root &>/dev/null
$IP link set ifb0 down &>/dev/null
$IP link set ifb1 down &>/dev/null
modprobe -r ifb &>/dev/null
$IPT -F OUTPUT -t mangle


if [ "$1" = "stop" ]
then
exit
fi

modprobe ifb
$IP link set ifb0 up
$IP link set ifb1 up

# this is just a way to test on my host with a single eth
# icmp goes to what would be wan 2 all other wan 1
# my fictional 2 users then ORed so the wan marking remains
# note the length of the "u32 match mark" masks mean the
# filters on the ifbs don't see the wan marks

$IPT -A OUTPUT -t mangle -o eth0 -j MARK --set-mark 0x10000
$IPT -A OUTPUT -t mangle -o eth0 -p icmp -j MARK --set-mark 0x20000
$IPT -A OUTPUT -t mangle -o eth0 -s 192.168.0.101 -j MARK --or-mark 0x1
$IPT -A OUTPUT -t mangle -o eth0 -s 192.168.0.102 -j MARK --or-mark 0x2

# traffaic from 6mbit wan
# you would have lots of users here
# so rates will be 5mbit/num users
# the limit param on the pfifo controls queue length

$TC qdisc add dev ifb0 root handle 1: htb
$TC class add dev ifb0 parent 1: classid 1:ffff htb rate 5mbit

# user 1
$TC class add dev ifb0 parent 1:ffff classid 1:1 htb rate 2.5mbit ceil 5mbit
$TC qdisc add dev ifb0 parent 1:1 pfifo limit 10
$TC filter add dev ifb0 parent 1: protocol ip prio 1 \
u32 match mark 1 0xffff flowid 1:1

# user 2
$TC class add dev ifb0 parent 1:ffff classid 1:2 htb rate 2.5mbit ceil 5mbit
$TC qdisc add dev ifb0 parent 1:2 pfifo limit 10
$TC filter add dev ifb0 parent 1: protocol ip prio 1 \
u32 match mark 2 0xffff flowid 1:2

# traffic from 15mbit wan
# it's just a test so I didn't bother doing it properly
# as above - just wanted the counters to see
# where the traffic went

$TC qdisc add dev ifb1 root handle 1: htb

$TC class add dev ifb1 parent 1: classid 1:1 htb rate 100kbit
$TC filter add dev ifb1 parent 1: protocol ip prio 1 \
u32 match mark 1 0xffff flowid 1:1

$TC class add dev ifb1 parent 1: classid 1:2 htb rate 100kbit
$TC filter add dev ifb1 parent 1: protocol ip prio 1 \
u32 match mark 2 0xffff flowid 1:2


# 2mbit per user no sharing
# classids are set before redirect to ifbs they get saved
# and reinstated when the packets return - they do not get seen
# by the queue on the ifb

$TC qdisc add dev eth0 root handle 1: htb
$TC class add dev eth0 parent 1: classid 1:1 htb rate 2mbit
$TC class add dev eth0 parent 1: classid 1:2 htb rate 2mbit

# user 1
$TC filter add dev eth0 parent 1:0 \
protocol ip prio 1 u32 match mark 0x10001 0xfffff classid 1:1 \
action mirred egress redirect dev ifb0
$TC filter add dev eth0 parent 1:0 \
protocol ip prio 2 u32 match mark 0x20001 0xfffff classid 1:1 \
action mirred egress redirect dev ifb1

#user 2
$TC filter add dev eth0 parent 1:0 \
protocol ip prio 1 u32 match mark 0x10002 0xfffff classid 1:2 \
action mirred egress redirect dev ifb0
$TC filter add dev eth0 parent 1:0 \
protocol ip prio 2 u32 match mark 0x20002 0xfffff classid 1:2 \
action mirred egress redirect dev ifb1