Traffic control: throttling downloads

Hi there,

I've successfully implemented a fairly complex traffic control setup
for a static SDSL line. All is working pretty nicely - multiple uploads
do not disturb parallel interactive ssh sessions any more.

Until somebody tried a large download. ssh got very sluggish just like
it used to be before I implemented traffic control.

My shaping policy roughly works like this:
- priorize DNS traffic
- priorize several TCP control packets (SYN, SYN+ACK etc.)
- priorize small ACK packets
- priorize ssh (which hasn't Maximize-Throughput TOS set)
- priorize small UDP packets
- medium size ACK packets get a slightly higher priority

This works very well in high-upload situations - small ssh packets come
through fast, big uploads get a fair share of bandwidth.

If somebody is just downloading, we've got almost no outgoing traffic,
the ACK packets come through very fast, so the sender will try to push
more through the line which get's saturated at my ISPs side of the line,
builds up a queue - dang! - ssh replies don't come through.

Is there anything I can do about this at my side - like detecting
connections which max out my bandwidth and throttling them somehow?

Maybe I could ask my ISP to employ some kind of traffic shaping on their
side - which is partly difficult since a lot of our traffic is via VPN,
so they cannot see what's inside and would have to rely on packet size
or some similar heuristics.

Any hints?

Thanks,

Tino.

--
"What we nourish flourishes." - "Was wir nähren erblüht."

www.tisc.de

Tino Schwarze <tino.schwarze-(E-Mail Removed)> wrote:
> Is there anything I can do about this at my side - like detecting
> connections which max out my bandwidth and throttling them somehow?

You might consider limiting or otherwise disabling Linux's autotuning
of socket buffer/window size to preclude the remote from dumping 4MB
of data into the buffers just upstream of you. That would be via
either your download apps making explicit setsockopt() calls for
SO_RCVBUF (prior to connection) or via sysctl against
net.ipv4.tcp_rmem and have it apply to all connections.

raj@tardy:~$ sysctl net.ipv4.tcp_rmem
net.ipv4.tcp_rmem = 4096 87380 4194304

The middle value is the starting point, the last value is the limit.
Units are bytes. Or you clould, IIRC go straight to
net.ipv4.tcp_moderate_rcvbuf to disable the autotuning entirely.

Some details in ip_sysctl.txt which may already be on your system, or
you can see a recent copy here:

http://git.kernel.org/?p=linux/kerne...tl.txt;hb=HEAD

> Maybe I could ask my ISP to employ some kind of traffic shaping on
> their side - which is partly difficult since a lot of our traffic is
> via VPN, so they cannot see what's inside and would have to rely on
> packet size or some similar heuristics.

You could do that. You might also ask them if they are following the
discussions on "buffer bloat." They may simply have too much
buffering for the bandwidthXdelay product of the link between them and
you.

rick jones
--
Process shall set you free from the need for rational thought.
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Tino Schwarze <tino.schwarze-(E-Mail Removed)> wrote:
> My shaping policy roughly works like this:
> - priorize DNS traffic
> - priorize several TCP control packets (SYN, SYN+ACK etc.)

Given that TCP tries to guess some things about the connection and the
characteristics of the path to the remote endpoint, based on the
behaviour of the SYN and SYN|ACK segments, it is probably best they
not be prioritized any higher than would be the data segments of that
connection. And you probably do not want TCP FIN segments to jump the
queue ahead of the data segments in the connection prior to the FIN.

> - priorize small ACK packets
> - priorize ssh (which hasn't Maximize-Throughput TOS set)
> - priorize small UDP packets
> - medium size ACK packets get a slightly higher priority

What do you mean by small and medium size ACK packets? Is size based
on how much is being ACKnowledged, or on how much data is also in the
segment? While TCP will "deal" with it, it does not like to have data
segments of a given connection reordered. Arrival of an out-of-order
data segment will trigger an immediate ACK, and if there are enough
out-of-order segments arriving at the destination, the returning ACKs
can trigger a spurrious fast retransmission.

rick jones
--
It is not a question of half full or empty - the glass has a leak.
The real question is "Can it be patched?"
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

On Fri, 18 Mar 2011 17:48:42 +0000 (UTC), Rick Jones <(E-Mail Removed)> wrote:

>Tino Schwarze <tino.schwarze-(E-Mail Removed)> wrote:
>> Is there anything I can do about this at my side - like detecting
>> connections which max out my bandwidth and throttling them somehow?
>
>You might consider limiting or otherwise disabling Linux's autotuning
>of socket buffer/window size to preclude the remote from dumping 4MB
>of data into the buffers just upstream of you. That would be via
>either your download apps making explicit setsockopt() calls for
>SO_RCVBUF (prior to connection) or via sysctl against
>net.ipv4.tcp_rmem and have it apply to all connections.
>
>raj@tardy:~$ sysctl net.ipv4.tcp_rmem
>net.ipv4.tcp_rmem = 4096 87380 4194304
>
>The middle value is the starting point, the last value is the limit.
>Units are bytes. Or you clould, IIRC go straight to
>net.ipv4.tcp_moderate_rcvbuf to disable the autotuning entirely.
>
>Some details in ip_sysctl.txt which may already be on your system, or
>you can see a recent copy here:
>
>http://git.kernel.org/?p=linux/kerne...tl.txt;hb=HEAD
>
>> Maybe I could ask my ISP to employ some kind of traffic shaping on
>> their side - which is partly difficult since a lot of our traffic is
>> via VPN, so they cannot see what's inside and would have to rely on
>> packet size or some similar heuristics.
>
>You could do that. You might also ask them if they are following the
>discussions on "buffer bloat." They may simply have too much
>buffering for the bandwidthXdelay product of the link between them and
>you.

One thing I did here long ago was to limit the size of the outgoing buffer
on the NIC connected to the ADSL modem. I run the modem in bridge mode, so
the connection is controlled by the Internet facing firewall box.

I've not yet got around to putting in priority streams or protecting my
streaming audio coming in. Instead, I try to throttle large downloads
by grabbing them with wget and using the --limit-rate option. This way
I can leave enough bandwidth for other uses while, for example, downloading
an .iso image.

It's not automatic, but good enough since mid-2004 on ADSL, for me to not
bother implementing more complex traffic control with netfilter.

Grant.

Tino Schwarze wrote:

> If somebody is just downloading, we've got almost no outgoing traffic,
> the ACK packets come through very fast, so the sender will try to push
> more through the line which get's saturated at my ISPs side of the line,
> builds up a queue - dang! - ssh replies don't come through.
>
> Is there anything I can do about this at my side - like detecting
> connections which max out my bandwidth and throttling them somehow?

You could limit large tcp packets to about 80% of your line speed and
use a short buffer or maybe a policer so that you drop packets.

Shaping from the wrong end of the bottleneck is not ideal, but you can,
at the expense of sacrificing bandwidth, do better than doing nothing.

You could shape or police on the inbound interface, but it would
probably be better with vpn to do it on the lan facing interface.
If you have more than one lan interface, it doesn't matter as you can
use something called ifb.

Andy Furniss <(E-Mail Removed)> wrote in news:im0ijf$tp3$1
@localhost.localdomain:

> You could limit large tcp packets to about 80% of your line speed and
> use a short buffer or maybe a policer so that you drop packets.

Could you please provide example code?

buck wrote:
> Andy Furniss<(E-Mail Removed)> wrote in news:im0ijf$tp3$1
> @localhost.localdomain:
>
>> You could limit large tcp packets to about 80% of your line speed and
>> use a short buffer or maybe a policer so that you drop packets.
>
> Could you please provide example code?

I am very rusty with Qos generally and don't know what sort of example
you want.

The easiest way is if you are just routing, then you can add a qdisc to
the lan facing interface and shape traffic as you would for upstream.

To set buffer size you just add a qdisc line fifo, bfifo or sfq to a htb
class and use the limit parameter.

Policers don't buffer, they just drop, but they do have a burst
parameter which is like a virtual buffer.

If say you had a 2mbit line on eth0 you could help stop it getting
flooded upstream with something like -

tc qdisc add dev eth0 ingress

tc filter add dev eth0 protocol ip prio 1 parent ffff: u32 match ip
protocol 6 0xff match u16 0x0000 0xff80 at 2 flowid 1:1

tc filter add dev eth0 protocol ip prio 2 parent ffff: u32 match ip
protocol 6 0xff flowid 1:2 action police rate 1600kbit burst 20k drop

The first filter matches tcp packets with length < 128 bytes by using a
match of 0x0000 and a mask of 0xff80 starting at byte 2 of the ip
packet, which is length - you can only match powers of 2 like this.
Packets that match this don't hit the second filter so only tcp >=128
get policed to 1600kbit.

You can see counters with -

tc -s filter ls dev eth0 parent ffff:

In practice you need to play around with rate and burst depending on
line speed - eg this would be too aggressive policing 100mbit eth
traffic down to 1600kbit.

If you want to use normal qdiscs on ingress traffic you have to use ifb.

ifb can also be used for egress traffic, eg if you want have more than
one lan facing nic you can still shape all traffic that came in from the
wan by putting a prio qdisc on the nics (just so you can add a filter to
redirect to ifb).

exanple ifb usage - here on ingress, catching all ip packets.

modprobe ifb numifbs=1
tc qdisc add dev eth0 ingress
ip link set ifb0 up

tc filter add dev eth0 parent ffff: \
protocol ip prio 1 u32 match u32 0 0 flowid 1:0 \
action mirred egress redirect dev ifb0

You can treat ifb0 like a real interface and add qdiscs and filters as
you would normally.
IIRC the flowid 1:0 set by the filter isn't seen by anything on ifb0 but
gets reinstated when traffic returns from ifb0.

The ingress qdisc isn't always as useful as shaping on egress lan
interfaces as in the latter case you can use iptables to mark etc, and
see de-nated packets if you are using nat.

If you need to shape ingress and some of the traffic is forwarded and
some destined to the shaping box, you may have a problem identifying
traffic using ingress if, say you are nating onto one IP address or
using something the uses iptables conntrack.

In this case there is a project called IMQ which can help, but you need
to patch kernel and netfilter to use it.

Andy Furniss <(E-Mail Removed)> wrote in news:im5k0q$8ge$1
@localhost.localdomain:

> buck wrote:
>> Andy Furniss<(E-Mail Removed)> wrote in news:im0ijf
$tp3$1
>> @localhost.localdomain:
>>
>>> You could limit large tcp packets to about 80% of your line speed
and
>>> use a short buffer or maybe a policer so that you drop packets.
>>
>> Could you please provide example code?
>
> I am very rusty with Qos generally and don't know what sort of
example
> you want.
>
> The easiest way is if you are just routing, then you can add a qdisc
to
> the lan facing interface and shape traffic as you would for
upstream.
>
> To set buffer size you just add a qdisc line fifo, bfifo or sfq to a
htb
> class and use the limit parameter.
>
> Policers don't buffer, they just drop, but they do have a burst
> parameter which is like a virtual buffer.
>
> If say you had a 2mbit line on eth0 you could help stop it getting
> flooded upstream with something like -
>
> tc qdisc add dev eth0 ingress
>
> tc filter add dev eth0 protocol ip prio 1 parent ffff: u32 match ip
> protocol 6 0xff match u16 0x0000 0xff80 at 2 flowid 1:1
>
> tc filter add dev eth0 protocol ip prio 2 parent ffff: u32 match ip
> protocol 6 0xff flowid 1:2 action police rate 1600kbit burst 20k
drop
>
> The first filter matches tcp packets with length < 128 bytes by
using a
> match of 0x0000 and a mask of 0xff80 starting at byte 2 of the ip
> packet, which is length - you can only match powers of 2 like this.
> Packets that match this don't hit the second filter so only tcp >=
128
> get policed to 1600kbit.
>
> You can see counters with -
>
> tc -s filter ls dev eth0 parent ffff:
>
> In practice you need to play around with rate and burst depending on
> line speed - eg this would be too aggressive policing 100mbit eth
> traffic down to 1600kbit.
>
> If you want to use normal qdiscs on ingress traffic you have to use
ifb.
>
> ifb can also be used for egress traffic, eg if you want have more
than
> one lan facing nic you can still shape all traffic that came in from
the
> wan by putting a prio qdisc on the nics (just so you can add a
filter to
> redirect to ifb).
>
> exanple ifb usage - here on ingress, catching all ip packets.
>
> modprobe ifb numifbs=1
> tc qdisc add dev eth0 ingress
> ip link set ifb0 up
>
> tc filter add dev eth0 parent ffff: \
> protocol ip prio 1 u32 match u32 0 0 flowid 1:0 \
> action mirred egress redirect dev ifb0
>
> You can treat ifb0 like a real interface and add qdiscs and filters
as
> you would normally.
> IIRC the flowid 1:0 set by the filter isn't seen by anything on ifb0
but
> gets reinstated when traffic returns from ifb0.
>
> The ingress qdisc isn't always as useful as shaping on egress lan
> interfaces as in the latter case you can use iptables to mark etc,
and
> see de-nated packets if you are using nat.
>
> If you need to shape ingress and some of the traffic is forwarded
and
> some destined to the shaping box, you may have a problem identifying
> traffic using ingress if, say you are nating onto one IP address or
> using something the uses iptables conntrack.
>
> In this case there is a project called IMQ which can help, but you
need
> to patch kernel and netfilter to use it.

Thanks, Andy. That's exactly what I was fishing for. You're matching
"small" packets and shaping on the remainder. Dunno why that never
occurred to me <shrug>.
--
buck

Hi Rick,

thanks for your response... I've been busy this week so I couldn't
answer earlier.

Rick Jones <(E-Mail Removed)> wrote:

>> Is there anything I can do about this at my side - like detecting
>> connections which max out my bandwidth and throttling them somehow?
>
> You might consider limiting or otherwise disabling Linux's autotuning
> of socket buffer/window size to preclude the remote from dumping 4MB
> of data into the buffers just upstream of you. That would be via
> either your download apps making explicit setsockopt() calls for
> SO_RCVBUF (prior to connection) or via sysctl against
> net.ipv4.tcp_rmem and have it apply to all connections.
>
> raj@tardy:~$ sysctl net.ipv4.tcp_rmem
> net.ipv4.tcp_rmem = 4096 87380 4194304
>
> The middle value is the starting point, the last value is the limit.
> Units are bytes. Or you clould, IIRC go straight to
> net.ipv4.tcp_moderate_rcvbuf to disable the autotuning entirely.

That would be difficult to implement since I've got a lot of machines,
some of which are not under my control. And it would of course affect
internal communication as well...

>> Maybe I could ask my ISP to employ some kind of traffic shaping on
>> their side - which is partly difficult since a lot of our traffic is
>> via VPN, so they cannot see what's inside and would have to rely on
>> packet size or some similar heuristics.
>
> You could do that. You might also ask them if they are following the
> discussions on "buffer bloat." They may simply have too much
> buffering for the bandwidthXdelay product of the link between them and
> you.

Good hint, thanks.

Tino.

--
"What we nourish flourishes." - "Was wir nähren erblüht."

www.tisc.de

Hi Rick,

Rick Jones <(E-Mail Removed)> wrote:

>> My shaping policy roughly works like this:
>> - priorize DNS traffic
>> - priorize several TCP control packets (SYN, SYN+ACK etc.)
>
> Given that TCP tries to guess some things about the connection and the
> characteristics of the path to the remote endpoint, based on the
> behaviour of the SYN and SYN|ACK segments, it is probably best they
> not be prioritized any higher than would be the data segments of that
> connection. And you probably do not want TCP FIN segments to jump the
> queue ahead of the data segments in the connection prior to the FIN.

I just took a closer look. I'm currently queuing SYN, RST, FIN+ACK,
SYN+ACK and RST+ACK packets to the "fast lane". The logic is: I want
connections to be established fast (e.g. ssh login).

>> - priorize small ACK packets
>> - priorize ssh (which hasn't Maximize-Throughput TOS set)
>> - priorize small UDP packets
>> - medium size ACK packets get a slightly higher priority
>
> What do you mean by small and medium size ACK packets? Is size based
> on how much is being ACKnowledged, or on how much data is also in the
> segment? While TCP will "deal" with it, it does not like to have data
> segments of a given connection reordered. Arrival of an out-of-order
> data segment will trigger an immediate ACK, and if there are enough
> out-of-order segments arriving at the destination, the returning ACKs
> can trigger a spurrious fast retransmission.

I'm classifying based on the payload of the ACK packet. This may lead to
packet reordering, of course. I'd rather classify connections by current
bandwidth usage but I could not find a way to do that - "tc filter" only
works by total bandwidth if I understood everything correctly.

I'm trying to catch the situation where someone does a bulk rsync or scp
transfer while working interactively at the same time at the same host.
I thought about using netfilter "hashlimit" but couldn't figure out a
sane way to detect high-bandwidth connections and reclassify them.

I assumed that TCP connections are usually either high-throughput or
low-throughput. Hmm... maybe that's a bit far-fetched.

Thanks,

Tino.

--
"What we nourish flourishes." - "Was wir nähren erblüht."

www.tisc.de