bandwidth shaping traffic from/to specific subnet

Hi there,

I'd like to adjust the bandwidth avaliable for traffic from/to a specific
subnet. Well, actually, I'd like to limit the bandwith for all traffic
_except_ the traffic originating in the local subnet.

Background information: The hoster where I've got my server located provides
a free 1TB/month traffic volume into the internet. ATM I'm far from even
touching 10% of that, but still I'd like take precautions. Local traffic,
e.g. between servers in the data center, or to the provided backup storage
is unlimited though, so I'd like to not limit that.

There's however only one network interface, and so far I've read into
the "tc" manpages traffic control seems possible for the whole interface
only; but I may have missed something there.

The iptables limit module however works on whole packets, so this is packet
limiting, not bandwidth limiting. That's not what I want.

The best idea I had so far was somehow making eth0 into distinct interfaces
for local and internet traffic, adjusting the routing tables apropriately.
But then I had no idea, how to do that without coding a small program
creating a couple of tap interfaces, replicating the data to a master tap
which is bridged with eth0 and putting the traffic control on the slave
taps. I regard that idea as kinda like Frankenstein and would not like to
do it that way.

There must be a better one. Though I got no clue in which direction to look.

Wolfgang

"Wolfgang Draxinger" <(E-Mail Removed)> wrote in
message news:gmekfk$p5f$(E-Mail Removed)...
> Hi there,
>
> I'd like to adjust the bandwidth avaliable for traffic from/to a specific
> subnet. Well, actually, I'd like to limit the bandwith for all traffic
> _except_ the traffic originating in the local subnet.
>
> Background information: The hoster where I've got my server located
provides
> a free 1TB/month traffic volume into the internet. ATM I'm far from even
> touching 10% of that, but still I'd like take precautions. Local traffic,
> e.g. between servers in the data center, or to the provided backup
storage
> is unlimited though, so I'd like to not limit that.
>
> There's however only one network interface, and so far I've read into
> the "tc" manpages traffic control seems possible for the whole interface
> only; but I may have missed something there.

Not true. You haven't read it carefully.

Example: Limit GoogleBot to 1Mbit/sec.

tc qdisc add dev eth0 root handle 1:0 htb default 20
tc class add dev eth0 parent 1:0 classid 1:1 htb prio 1 rate 1Mbps
tc filter add dev eth0 parent 1:0 prio 1 protocol ip u32 match ip dst
66.249.64.0/19 match ip sport 80 0xFFFF flowid 1:1

This actually limits ALL fetches from Google's network for web pages (port
80) to a maximum of 1Mbit/s. If these three statements were the only
rules, then everything else would be unlimited.

> The iptables limit module however works on whole packets, so this is
packet
> limiting, not bandwidth limiting. That's not what I want.

There is still some anti-abuse [connection] limiting that should probably
be done.

> The best idea I had so far was somehow making eth0 into distinct
interfaces
> for local and internet traffic, adjusting the routing tables
apropriately.
> But then I had no idea, how to do that without coding a small program
> creating a couple of tap interfaces, replicating the data to a master tap
> which is bridged with eth0 and putting the traffic control on the slave
> taps. I regard that idea as kinda like Frankenstein and would not like to
> do it that way.
>
> There must be a better one. Though I got no clue in which direction to
look.

D. Stussy wrote:

> Not true. You haven't read it carefully.

Seems so :-)

> Example: Limit GoogleBot to 1Mbit/sec.
>
> tc qdisc add dev eth0 root handle 1:0 htb default 20
> tc class add dev eth0 parent 1:0 classid 1:1 htb prio 1 rate 1Mbps
> tc filter add dev eth0 parent 1:0 prio 1 protocol ip u32 match ip dst
> 66.249.64.0/19 match ip sport 80 0xFFFF flowid 1:1
>
> This actually limits ALL fetches from Google's network for web pages (port
> 80) to a maximum of 1Mbit/s. If these three statements were the only
> rules, then everything else would be unlimited.

Thank you a lot.

Wolfgang

"Andrew Gideon" <(E-Mail Removed)> wrote in message
news:498de23d$0$11543$(E-Mail Removed) om...
> On Thu, 05 Feb 2009 11:46:20 -0800, D. Stussy wrote:
> > This actually limits ALL fetches from Google's network for web pages
> > (port 80) to a maximum of 1Mbit/s.
>
> I'm pretty new to traffic shaping in Linux, and this comes close to a
> question about which I've wondered for a while: is it possible to apply a
> discipline (or something analogous to achieve the shaping) for traffic
> that passes out multiple interfaces?

One will need multiple rules - one for each interface. I don't know of a
way to combine across interfaces; maybe someone else does.

If one can create a virtual interface that routes via multiple real
interfaces, then set the bandwidth for the virtual. However, that's also
beyond my knowledge at this time.

> Most of the time simply shaping on a per interface is enough, because
> traffic destined to a single IP will tend to pass out a single
> interface. But if I want to shape traffic to a particular /23 and BGP
> selects a best path for the first /24 out eth0.10 and the second out
> eth0.11, then traffic to the /23 could reach twice the limit I've set.
>
> Is there some solution for this?

See above.

Andrew Gideon wrote:
> On Thu, 05 Feb 2009 11:46:20 -0800, D. Stussy wrote:
>
>> This actually limits ALL fetches from Google's network for web pages
>> (port 80) to a maximum of 1Mbit/s.
>
> I'm pretty new to traffic shaping in Linux, and this comes close to a
> question about which I've wondered for a while: is it possible to apply a
> discipline (or something analogous to achieve the shaping) for traffic
> that passes out multiple interfaces?
>
> Most of the time simply shaping on a per interface is enough, because
> traffic destined to a single IP will tend to pass out a single
> interface. But if I want to shape traffic to a particular /23 and BGP
> selects a best path for the first /24 out eth0.10 and the second out
> eth0.11, then traffic to the /23 could reach twice the limit I've set.

It's called ifb (intermediate functional block in kernel config)

Are eth0.10 vlans? (I can never remember which is vlan or the way people
used to add secondary addresses to nics with ifconfig)

Either way it should be possible - in the case of vlans you may need to
mess about with u32 and hex if tc won't attach to them directly - but it
should be possible somehow. In fact if the traffic is going out of one
physical nic you could do it without ifb anyway.

I am not on the my box with my old tests, but could dig out something as
a start if you can't find any docs.

Andrew Gideon wrote:

> You're suggesting - excluding ifb - I can attack a discipline to eth0 and
> it would apply to all VLANs on that physical interface? That had not
> occurred to me as a possibility (rather: I'd just assumed that it
> wouldn't work).

It is possible, but it may not be the easiest way for you because a
quick test I just did means that it looks like you can attach qdiscs to
vlans directly.

If this were not the case than what you would have needed to do is
filter on the real interface. Most tc examples you see use

tc filter add dev eth0 protocol ip ....

The protocol ip refers to the ethertype protocol number in the eth frame
- it is possible to filter types other than ip, protocol all gets
everything, others I can think of by name - arp,ipv6,8021q or you can
just use the number direct.

So you would use protocol 8021q for vlan. The problem then may be that
you have to work out u32 offsets and use hex matches/masks for the parts
of the ip packet you want to match - not as easy as letting tc do some
of the work for you, but tcpdump -e and a few tests and it should be
possible. FWIW using negative offsets in tc lets you match macs and
ethertype, I am not sure where the offsets for vlan headers would be.
You need to make sure the prio on tc filters is different for different
protocols, or it throws an error.


> If I'm grasping it, though, I'll need to direct packets from multiple
> interfaces into an IFB, do the shaping, and then get the packets out the
> original interface. I'm still not clear on how that last step works, but
> I've more to read (and eventually to try).

ifb was created specially to do this sort of thing - so you don't need
to worry getting the packets back, it will just happen.

Given that I think you can attach qdiscs to vlans I would use ifb and
redirect from the vlans so the traffic going through ifbs is protocol ip.

To get the traffic you need a classful qdisc on the interface you are
redirecting from - so just add prio to the vlans in this case as you
don't need to do any further shaping when the traffic returns. It is
possible to redirect incoming traffic as well by adding the ingress qdisc.

modprobe ifb

will give you two new interfaces ifb0 and ifb1 (use numifbs=x if you
need more). ip link ls will show you that the qlen is 32 vs 1000 for eth
which is worth remembering if you don't specify lengths on htb leafs.

ip link set ifb0 up

tc qdisc add dev eth0.1 handle 1:0 root prio

tc filter add dev eth0.1 parent 1:0 protocol ip prio 1\
u32 match u32 0 0 classid 1:1 action mirred egress redirect dev ifb0

and so on for the other vlans. The classid set is not seen by the ifb
but does get reinstated when the packet returns so it is possible to
double shape if you wanted to.

You then just add your tc rules and filters to ifb0 as if it were a
normal interface and it should see traffic from all the places you
redirected to it from.

Andrew Gideon wrote:

> What is it that causes the packet to return to eth0.1? Is if the nature
> of an ifb that returns the packet, so if I were to send it to a different
> device it would not return?

I think it's just the nature of ifb.

I doubt any other device would work in the same way. I've never tried
but I would think that redirecting to a real device would just fail most
of the time - wrong IP/MAC addresses. I know it's mentioned in the doc,
but it doesn't seem very useful apart from the case of cloning and
sending to dummy for tcpdump, which on a highly loaded link could be
more efficient than filtering with tcpdump on the real nic.