Traffic Shaping Question

Hello all,

I've run into a situation at work where I need to use traffic shaping.
I've done a lot of reading on the implementation details (and I'm
fairly seasoned with linux), but I'm new to this particular concept, so
I have a few questions.

The root of this problem is that when our outgoing bandwidth is maxed
out for whatever reason (sending out an email newsletter to 400+
addresses usually does it), traffic from our remote sites cannot get
back into us to update databases and so forth, so they get kicked out
of their applications.

Picture below (hope you don't mind the ASCII graphics):

____
/ \
[ Inet ]
\____/
|
_____
/ \
[ Frame ]
/ \_____/ \
1024K T1 / | \ 256K T1
/ | \
____ | _______
/ \ | / \
[ HQ ] | [ Remote2 ]
\____/ | \_______/
|
| 256K T1
_______
/ \
[ Remote1 ]
\_______/

(I have tried to get on to the LARTC mailing list to ask this question,
to no avail)

I'm going to try this: use two NICs and proxy ARP (or standard bridging
if necessary) and place the linux box between the internet router and
our network.
More ASCII:
____________ _______ ____
| | | | / \
| ISP ROUTER |------| Linux |------[ HQ ]
|____________| |_______| \____/

Main Question: Will I be able to shape traffic both ways?

And I don't mean shape and police. I mean shape on my router's
interface (egress from network) and also shape stuff on my network's
interface (egress from router). I would like to do this so that I can
guarantee bandwidth both incoming and outgoing to the remote sites (and
rate-limit the internet both ways so it can't drown out my remote
sites).

Another Question: I need to use the same prioritizing on the the
different rate-limited buckets, without specifying all the priority
rules on each interface?

E.g., this is what I think I'll have to do:
1: (root connection, rate limited to near-max bandwidth of 950k or so)
1:1 (1st remote connection, rate limited to 240k or so)
1:11 (priority traffic for 1st remote)
1:12 (bulk traffic for 1st remote)
1:2 (2nd remote connection, rate limited to 240k or so)
1:21 (priority traffic for 2nd remote)
1:22 (bulk traffic for 2nd remote)
1:3 (other traffic, rate-limited to 500k or so)
1:31 (priority traffic for internet/other)
1:32 (bulk traffic for internet/other)

I'd also like to figure out how to tack on SFQ to the leaf buckets.

I guess this is not that bad, but I was wondering, since all the
priorities will be kept the same among the different traffic shapes if
there was a way to set them globally, then define rate limits, so I
only have to set the priorities once and the rate limits once. Again,
I'm new to shaping, so that may seem obvious to some of you. Thanks
for taking the time to read this.

X

Well, I finally got a chance to actually play with this tonight, and
I'll answer some of my questions for the benefit of others out there
who might be looking at the same thing.

Proxy ARP was successful. I had to play with the routing to get it in
shape, but that didn't take too long. And of course it's obligatory to
forget "the magic words" somewhere in the process: echo 1 >
/proc/sys/net/ipv4/ip_forward

I was able to shape both ways using one root on eth0 and another on
eth1. (I tried it with just one first, then did a speed test, then
added a second). This is great!

I don't think it will be a problem to assign priorities correctly
without having too many rules, looking at the sample code I used ("The
Ultimate Traffic Conditioner" commands from the LARTC.org FAQ, sans any
ingress rules). However, I still have to figure out how to divvy up
the bandwidth on top of what those commands do. Any help on what
commands I will need will be appreciated!

X

On 10 Mar 2006 17:31:29 -0800, "X" <(E-Mail Removed)> wrote:

>Well, I finally got a chance to actually play with this tonight, and
>I'll answer some of my questions for the benefit of others out there
>who might be looking at the same thing.
>
>Proxy ARP was successful. I had to play with the routing to get it in
>shape, but that didn't take too long. And of course it's obligatory to
>forget "the magic words" somewhere in the process: echo 1 >
>/proc/sys/net/ipv4/ip_forward
>
>I was able to shape both ways using one root on eth0 and another on
>eth1. (I tried it with just one first, then did a speed test, then
>added a second). This is great!
>
>I don't think it will be a problem to assign priorities correctly
>without having too many rules, looking at the sample code I used ("The
>Ultimate Traffic Conditioner" commands from the LARTC.org FAQ, sans any
>ingress rules). However, I still have to figure out how to divvy up
>the bandwidth on top of what those commands do. Any help on what
>commands I will need will be appreciated!
>
>X

Have a look at http://yesican.chsoft.biz/lartc/ultimate_sh_InUse.txt
because the "Ultimate" script's classes don't add up to 100% of the
root rate, which will eventually cause you grief. Where that script
has IMQ think "My internal facing interface". There is also a bit of
stuff in the ftp://yesican.chsoft.biz/pub/lartc directory that you may
want to peek at because selective ProxyARP was a solution that normal
routing commands would not solve.
--
buck

Well, just about the time I left work, I got a call saying that our
remote site was down, so I had to go back. Long story short, I had our
application servers plugged into both networks (our old T1 is connected
to the old network that I want to shape, and our new T1, which I used
to test the "transparent" shaping server, is connected to a separate
network, the app servers have a NICs attached to both networks), and
the Proxy ARP linux server most likely made an entry in the app
servers' ARP cache. I have no idea why this would cause us to lose
connectivity, but it did. It was so strange, the rest of my network
could see the remote computer and vice versa, but the app servers could
not (and vice versa). The internet could also not see our application
servers from the outside. Very strange. Anyway, once I figured out
they were the problem, I guessed that it had to do with their ARP
cache, so I reset the default gateway on both of them to a different
(bogus) number and then back to what it's supposed to be and that
seemed to clear the cache and do the trick.

This is something to watch out for if you have multi homed computers
with one of them on a network that uses a proxy-arp server. I hope
this problem was only caused because I unplugged the server. I hope
this problem does not happen by default when you use a transparent
proxy arp server, as I need them to be on both networks and to
proxy-arp one. I guess worse comes to worse, I could fall back on
using bridging instead. Bridging is less manipulative, it seems, but I
am against the idea of using my network cards in promiscuous mode, and
also I don't know that the machine that we have for the shaping box
could handle as much traffic as promiscuous mode would force it to.

Food for thought,

X

Thank you very much for the script link and the info! Looking at the
script, it also using egress (shaping) both ways. I'm glad to know
this is effective. I'm curious as to why they didn't include filters
to prioritize ACKs. I did notice a note that said it would match
packets up to 64k with that rule. I just wonder if that causes the CPU
to be overburdened or the queues to get full or something. Anyway,
this is very helpful, thank you!

X

On 10 Mar 2006 19:15:19 -0800, "X" <(E-Mail Removed)> wrote:

>Thank you very much for the script link and the info! Looking at the
>script, it also using egress (shaping) both ways. I'm glad to know
>this is effective. I'm curious as to why they didn't include filters
>to prioritize ACKs. I did notice a note that said it would match
>packets up to 64k with that rule. I just wonder if that causes the CPU
>to be overburdened or the queues to get full or something. Anyway,
>this is very helpful, thank you!
>
>X

It is more effective to prioritize small (64 or 128 byte) packets than
to match ACK, and all real ACK are <= 64 bytes.

You can tweak the script where the ACK is commented out, but why?
--buck

On 10 Mar 2006 18:52:23 -0800, "X" <(E-Mail Removed)> wrote:

>Well, just about the time I left work, I got a call saying that our
>remote site was down, so I had to go back. Long story short, I had our
>application servers plugged into both networks (our old T1 is connected
>to the old network that I want to shape, and our new T1, which I used
>to test the "transparent" shaping server, is connected to a separate
>network, the app servers have a NICs attached to both networks), and
>the Proxy ARP linux server most likely made an entry in the app
>servers' ARP cache. I have no idea why this would cause us to lose
>connectivity, but it did. It was so strange, the rest of my network
>could see the remote computer and vice versa, but the app servers could
>not (and vice versa). The internet could also not see our application
>servers from the outside. Very strange. Anyway, once I figured out
>they were the problem, I guessed that it had to do with their ARP
>cache, so I reset the default gateway on both of them to a different
>(bogus) number and then back to what it's supposed to be and that
>seemed to clear the cache and do the trick.
>
>This is something to watch out for if you have multi homed computers
>with one of them on a network that uses a proxy-arp server. I hope
>this problem was only caused because I unplugged the server. I hope
>this problem does not happen by default when you use a transparent
>proxy arp server, as I need them to be on both networks and to
>proxy-arp one. I guess worse comes to worse, I could fall back on
>using bridging instead. Bridging is less manipulative, it seems, but I
>am against the idea of using my network cards in promiscuous mode, and
>also I don't know that the machine that we have for the shaping box
>could handle as much traffic as promiscuous mode would force it to.
>
>Food for thought,
>
>X

Bridging also sucks because to effectively firewall you must use
ebtables, further complicating things. ProxyARP is the way to go, but
as already stated we strongly prefer selective ProxyARP rather than
altering /proc to ARP the interface(s).

Don't forget 'ip route flush cache'
--
buck

What do you mean by selective proxy ARP? I just used a method that I
found on the internet where you have to do the echo 1 > /proc/...'s.
Is there another (better) way? Also, when I had the problem at work, I
left both the new T1 interface and the linux shaper unplugged when I
left work, and that's when I had the problems.

Thanks,

X

On 11 Mar 2006 21:18:59 -0800, "X" <(E-Mail Removed)> wrote:

>What do you mean by selective proxy ARP?

Let me start with this scenario:
eth1 is an external interface pointing to 1.2.3.4/29
eth2 is an external interface pointing to 5.6.7.8/29
eth0 is the internal interface 192.168.1.0/24.

I unplug the wire to eth1. What will then happen, and I DON'T want,
is that eth2 will now reply IS-AT to what belongs on eth1.

Selective ProxyARP specifies the IPs associated with eth1 (and also
eth2 if appropriate).

>I just used a method that I
>found on the internet where you have to do the echo 1 > /proc/...'s.
>Is there another (better) way?

Yes, there is a better way:
http://yesican.chsoft.biz/lartc/proxy-arp.sh
http://yesican.chsoft.biz/lartc/proxy-arp.conf

Don't forget to 'echo 0 >/proc/...'.

>Also, when I had the problem at work, I
>left both the new T1 interface and the linux shaper unplugged when I
>left work, and that's when I had the problems.
>
>Thanks,
>
>X
--
buck

Buck,

I don't know if I've said so, but I really appreciate your help in
this.

For my situation, I don't know that selective proxy arp will gain me
anything over the /proc method, because I suppose I do need the linux
box to answer ARP on the router side for all computers on the network
side, since all computers are on the same logical network. I do only
need one rule for ARP on the network side, since the only thing that is
on the other side is my router but that should be specified in the
routing table, right? e.g.:

ip route add 192.168.0.1 dev eth1
ip route add 192.168.0.0/24 dev eth0

My setup looks like this:

________
| |
| Router | 192.168.0.1
|________|
|
|
|
________
| | eth1
| Shaper | 192.168.0.18
|________| eth0
|
|
|
_______
/ \
[ Network ] 192.168.0.0/24
\_______/

I could be missing something here. Let me know if I am. Thanks again
for your help!

X