Maximum bandwidth usage

Apologies for the only marginally topical question.

On a properly tuned system (which all modern linux systems should be), is a
single TCP connection supposed to be able to use all the available
bandwidth?

The issue I'm seeing is as follows. Testing between two computers over a
transatlantic WAN link (RTT about 80ms). iperf and nuttcp report roughly 0.5
Mbit/sec throughput, no matter how I tune the TCP settings on the two hosts.
As far as I know, recent Linux kernels should support TCP autotuning, so
messing around shouldn't even be necessary. But still, I tried to modify the
suggested settings to no avail (net.core.rmem_max = 16777216,
net.core.wmem_max = 16777216, net.ipv4.tcp_rmem = 4096 87380 16777216,
net.ipv4.tcp_wmem = 4096 65536 16777216. I also tried other settings as
found in articles dealing with TCP tuning, but basically they all ended up
modifying those values, although the numbers differed slightly. TCP window
scaling and selective acknowledgements are all enabled).

But, if I run multiple connections concurrently (like eg nuttcp -N2 or
more), each connection gets 1Mbit/sec throughput: nuttcp -N4 (running 4
concurrent threads) totals 2 Mbit/sec.
I would expect to be able to get the same 2 Mbit/sec with a single TCP
connection, too.

I'm not sure whether this is expected behavior, or if not, if there's
something else I forgot to check or where to look to debug this.
I have very limited knowledge of the intermediate network topology, other
than what's shown by traceroute/mtr.

Any pointers appreciated. Thanks.

On 2010-06-15, pk <(E-Mail Removed)> wrote:
> Apologies for the only marginally topical question.
>
> On a properly tuned system (which all modern linux systems should be), is a
> single TCP connection supposed to be able to use all the available
> bandwidth?

Depends what you mean by that.
>
> The issue I'm seeing is as follows. Testing between two computers over a
> transatlantic WAN link (RTT about 80ms). iperf and nuttcp report roughly 0.5
> Mbit/sec throughput, no matter how I tune the TCP settings on the two hosts.

So. It is the link. Why you would try to test things on a translatlantic
cable I have no idea.


> As far as I know, recent Linux kernels should support TCP autotuning, so
> messing around shouldn't even be necessary. But still, I tried to modify the
> suggested settings to no avail (net.core.rmem_max = 16777216,
> net.core.wmem_max = 16777216, net.ipv4.tcp_rmem = 4096 87380 16777216,
> net.ipv4.tcp_wmem = 4096 65536 16777216. I also tried other settings as
> found in articles dealing with TCP tuning, but basically they all ended up
> modifying those values, although the numbers differed slightly. TCP window
> scaling and selective acknowledgements are all enabled).
>
> But, if I run multiple connections concurrently (like eg nuttcp -N2 or
> more), each connection gets 1Mbit/sec throughput: nuttcp -N4 (running 4
> concurrent threads) totals 2 Mbit/sec.
> I would expect to be able to get the same 2 Mbit/sec with a single TCP
> connection, too.

Depends on what you are trying to do.
How about transferring a large file(eg the kernel) over that link all at
once, and see what throughput you get.
Remember that with the 80ms any packet housekeeping will take far longer
than transfering the file will. (at 2Mb/s, 80ms is the equivalent of
20KB of transfered data.)


>
> I'm not sure whether this is expected behavior, or if not, if there's
> something else I forgot to check or where to look to debug this.
> I have very limited knowledge of the intermediate network topology, other
> than what's shown by traceroute/mtr.
>
> Any pointers appreciated. Thanks.

"pk" <(E-Mail Removed)> wrote in message news:2959923.OKlZ2vcFHj@xkzjympik...
> Apologies for the only marginally topical question.
>
> On a properly tuned system (which all modern linux systems should be), is
a
> single TCP connection supposed to be able to use all the available
> bandwidth?

If it's the only network bound process at the time, yes, there's no reason
not to.

If you wish to apportion the existing bandwidth among several network-bound
tasks, see the "tc" command from the IProute2 package (
http://www.linux-foundation.org/en/Net:Iproute2 ). The HTB classifier is
exceptionally good at splitting the bandwidth as allocated, and with the
"ceil[ing]" command, giving any unused allocations to those that are using
it. However, it may not help if the issue is concerning bandwidth of two
instances of the same application. Don't forget that this requires certain
items enabled in the kernel.

Rick Jones wrote:

> Just for grins, I would suggest some explicit socket buffer size
> settings. I don't know how that is done with iperf or nuttcp, but for
> netperf it would be something along the lines of:
>
> netperf -H <remote> -- -m 64K -s <size> -S <size>
>
> I'd probably try something like:
>
> for s in 32 64 128 256
> do
> netperf -H <remote> -- -m 64K -s $s -S $s
> done

This is what I get (pretty much same results in both directions, so I'm only
pasting one direction):

# for s in 32 64 128 256; do netperf -H 1x.x.x.x -p 5001 -- -P 5002 -m 64K -
s $s -S $s; done
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 5002 AF_INET to 1x.x.x.x
(1x.x.x.x) port 5002 AF_INET : demo
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

256 2048 65536 10.52 0.02
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 5002 AF_INET to 1x.x.x.x
(1x.x.x.x) port 5002 AF_INET : demo
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

256 2048 65536 10.88 0.02
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 5002 AF_INET to 1x.x.x.x
(1x.x.x.x) port 5002 AF_INET : demo
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

256 2048 65536 10.08 0.02
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 5002 AF_INET to 1x.x.x.x
(1x.x.x.x) port 5002 AF_INET : demo
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

512 2048 65536 10.09 0.05

>> But, if I run multiple connections concurrently (like eg nuttcp -N2
>> or more), each connection gets 1Mbit/sec throughput: nuttcp -N4
>> (running 4 concurrent threads) totals 2 Mbit/sec.
>> I would expect to be able to get the same 2 Mbit/sec with a single
>> TCP connection, too.
>
> Are there any TCP retransmissions recorded on the sender during the
> tests?

Yes, there is an average of ~85 retransimssions on the sending side when
running the 4 netperf tests above (in total, not per test).
A curious thing I noticed is that all the retransmissions occur with the
-[sS] 32, 64 and 128 tests only; NO retransmissions ever occur when running
the -[sS] 256 test. I've run the tests many times, and the 256 one
consistently showed no retransmissions (and the others did).

> What sort of interfaces are being used on the systems on either end?

One end is physical ethernet, the other end is a KVM virtual machine bridged
to an ethernet network (the guest is using the virtio driver). The KVM host
is not very loaded, as this is the only VM running on it.

> Any chance that interrupt coalescing is getting in the way?

# ethtool -c eth0
Coalesce parameters for eth0:
Adaptive RX: off TX: off
stats-block-usecs: 999936
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0

rx-usecs: 18
rx-frames: 6
rx-usecs-irq: 18
rx-frames-irq: 6

tx-usecs: 80
tx-frames: 20
tx-usecs-irq: 80
tx-frames-irq: 20

rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0

rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0

> Might also be interesting to know if the 2MBit/s WAN link is a single
> link "uinder the covers" or if it is some sort of bonding of lesser
> bandwidth links.

Yes, it might be interesting

>> I'm not sure whether this is expected behavior, or if not, if
>> there's something else I forgot to check or where to look to debug
>> this. I have very limited knowledge of the intermediate network
>> topology, other than what's shown by traceroute/mtr.
>
> A lot of this will not apply to your situation, but some will:
>
> Some of my checklist items when presented with assertions of poor
> network performance, in no particular order:
>
> *) Is *any one* CPU on either end of the transfer at or close to 100%
> utilization? A given TCP connection cannot really take advantage
> of more than the services of a single core in the system, so
> average CPU utilization being low does not a priori mean things are
> OK.

Neither box uses more than 0.3% with any test (one is a dual (virtual) CPU,
the other is a dual 4-core CPU).

> *) Are there TCP retransmissions being registered in netstat
> statistics on the sending system? Take a snapshot of netstat -s -t
> from just before the transfer, and one from just after and run it
> through beforeafter from
> ftp://ftp.cup.hp.com/dist/networking/tools:
>
> netstat -s -t > before
> transfer or wait 60 or so seconds if the transfer was already going
> netstat -s -t > after
> beforeafter before after > delta

Yes, see above.

> *) Are there packet drops registered in ethtool -S statistics on
> either side of the transfer? Take snapshots in a matter similar to
> that with netstat.

No packet drops at the ethernet level.

> *) Are there packet drops registered in the stats for the switch(es)
> being traversed by the transfer? These would be retrieved via
> switch-specific means.

No drops, at least in the switches I can access (ie those immediately
connected to the two hosts).

> *) What is the latency between the two end points. Install netperf on
> both sides, start netserver on one side and on the other side run:
>
> netperf -t TCP_RR -l 30 -H <remote>
>
> and invert the transaction/s rate to get the RTT latency. There
> are caveats involving NIC interrupt coalescing settings defaulting
> in favor of throughput/CPU util over latency:
>
> ftp://ftp.cup.hp.com/dist/networking...cy_vs_tput.txt
>
> but when the connections are over a WAN latency is important and
> may not be clouded as much by NIC settings.

This pretty much gives me the same latency I measured earlier, ie 1/12.87 =
0.078 msecs which is consistent with what ping shows.

> This all leads into:
>
> *) What is the *effective* TCP (or other) window size for the
> connection. One limit to the performance of a TCP bulk transfer
> is:
>
> Tput <= W(eff)/RTT
>
> The effective window size will be the lesser of:
>
> a) the classic TCP window advertised by the receiver (the value in
> the TCP header's window field shifted by the window scaling
> factor exchanged during connection establishment (why one wants
> to get traces including the connection establishment...)
>
> this will depend on whether/what the receiving application has
> requested via a setsockopt(SO_RCVBUF) call and the sysctl limits
> set in the OS. If the application does not call
> setsockopt(SO_RCVBUF) then the Linux stack will "autotune" the
> advertised window based on other sysctl limits in the OS.
>
> b) the computed congestion window on the sender - this will be
> affected by the packet loss rate over the connection, hence the
> interest in the netstat and ethtool stats.
>
> c) the quantity of data to which the sending TCP can maintain a
> reference while waiting for it to be ACKnowledged by the
> receiver - this will be akin to the classic TCP window case
> above, but on the sending side, and concerning
> setsockopt(SO_SNDBUF) and sysctl settings.
>
> d) the quantity of data the sending application is willing/able to
> send at any one time before waiting for some sort of
> application-level acknowledgement. FTP and rcp will just blast
> all the data of the file into the socket as fast as the socket
> will take it. scp has some application-layer "windowing" which
> may cause it to put less data out onto the connection than TCP
> might otherwise have permitted. NFS has the maximum number of
> outstanding requests it will allow at one time acting as a
> defacto "window" etc etc etc

Not sure this helps, but tcpdumping during the nuttcp test, the peers
initially advertise a window of 5840 and 5792 (sender/receiver), both with
window scaling == 2^9. As the test progresses, the receiver advertises
bigger and bigger window sizes (scaled values: 8704, 11264, 13824, 16896
etc. up to 82944 at the end of the test). The sender advertises 6144 from
the second segment onwards and sticks to that.

During the netperf test, on the other hand, both parties stick to a window
size of 512, which produces many "TCP Window Full" segments when seen in
Wireshark (actually, one every two).

Thanks.

pk <(E-Mail Removed)> wrote:
> Rick Jones wrote:

> > Just for grins, I would suggest some explicit socket buffer size
> > settings. I don't know how that is done with iperf or nuttcp, but for
> > netperf it would be something along the lines of:
> >
> > netperf -H <remote> -- -m 64K -s <size> -S <size>
> >
> > I'd probably try something like:
> >
> > for s in 32 64 128 256
> > do
> > netperf -H <remote> -- -m 64K -s $s -S $s
> > done

> This is what I get (pretty much same results in both directions, so I'm only
> pasting one direction):

Argh! I left-off the 'K' !-(

that should be

netperf -H <remote> -- -m 64K -s ${s}K -S ${s}K

sorry about that.

> > Are there any TCP retransmissions recorded on the sender during
> > the tests?

> Yes, there is an average of ~85 retransimssions on the sending side when
> running the 4 netperf tests above (in total, not per test).

While the netperf tests above were fubar thanks to my forgetting the
'K' a TCP retransmission is generally not goodness when it comes to
making throughput. That will tend to keep the congestion window (aka
cwnd) suppressed. How much depends on the congestion control
algorithm being used.


> > What sort of interfaces are being used on the systems on either end?

> One end is physical ethernet, the other end is a KVM virtual machine
> bridged to an ethernet network (the guest is using the virtio
> driver). The KVM host is not very loaded, as this is the only VM
> running on it.

Just for paranoia, you can achieve "good" throughput from the VM to a
local system?

> > A lot of this will not apply to your situation, but some will:
> >
> > Some of my checklist items when presented with assertions of poor
> > network performance, in no particular order:
> >

> > *) Are there TCP retransmissions being registered in netstat
> > statistics on the sending system? Take a snapshot of netstat
> > -s -t from just before the transfer, and one from just after
> > and run it through beforeafter from
> > ftp://ftp.cup.hp.com/dist/networking/tools:
> >
> > netstat -s -t > before
> > transfer or wait 60 or so seconds if the transfer was already going
> > netstat -s -t > after
> > beforeafter before after > delta

> Yes, see above.

Definitely check that with the corrected netperf (or other test).

> > *) Are there packet drops registered in ethtool -S statistics on
> > either side of the transfer? Take snapshots in a matter
> > similar to that with netstat.

> No packet drops at the ethernet level.

Probably at the point where the LAN meets the WAN.

> > *) What is the latency between the two end points. Install netperf on
> > both sides, start netserver on one side and on the other side run:
> >
> > netperf -t TCP_RR -l 30 -H <remote>
> >
> > and invert the transaction/s rate to get the RTT latency. There
> > are caveats involving NIC interrupt coalescing settings defaulting
> > in favor of throughput/CPU util over latency:
> >
> > ftp://ftp.cup.hp.com/dist/networking...cy_vs_tput.txt
> >
> > but when the connections are over a WAN latency is important and
> > may not be clouded as much by NIC settings.

> This pretty much gives me the same latency I measured earlier, ie
> 1/12.87 = 0.078 msecs which is consistent with what ping shows.

It will likely be somewhat higher under load.

> > This all leads into:
> >
> > *) What is the *effective* TCP (or other) window size for the
> > connection. One limit to the performance of a TCP bulk transfer
> > is:
> >
> > Tput <= W(eff)/RTT
> >
> > The effective window size will be the lesser of:
> >
> > a) the classic TCP window advertised by the receiver (the value in
> > the TCP header's window field shifted by the window scaling
> > factor exchanged during connection establishment (why one wants
> > to get traces including the connection establishment...)
> >
> > this will depend on whether/what the receiving application has
> > requested via a setsockopt(SO_RCVBUF) call and the sysctl limits
> > set in the OS. If the application does not call
> > setsockopt(SO_RCVBUF) then the Linux stack will "autotune" the
> > advertised window based on other sysctl limits in the OS.
> >
> > b) the computed congestion window on the sender - this will be
> > affected by the packet loss rate over the connection, hence the
> > interest in the netstat and ethtool stats.
> >
> > c) the quantity of data to which the sending TCP can maintain a
> > reference while waiting for it to be ACKnowledged by the
> > receiver - this will be akin to the classic TCP window case
> > above, but on the sending side, and concerning
> > setsockopt(SO_SNDBUF) and sysctl settings.
> >
> > d) the quantity of data the sending application is willing/able to
> > send at any one time before waiting for some sort of
> > application-level acknowledgement. FTP and rcp will just blast
> > all the data of the file into the socket as fast as the socket
> > will take it. scp has some application-layer "windowing" which
> > may cause it to put less data out onto the connection than TCP
> > might otherwise have permitted. NFS has the maximum number of
> > outstanding requests it will allow at one time acting as a
> > defacto "window" etc etc etc

> Not sure this helps, but tcpdumping during the nuttcp test, the
> peers initially advertise a window of 5840 and 5792
> (sender/receiver), both with window scaling == 2^9. As the test
> progresses, the receiver advertises bigger and bigger window sizes
> (scaled values: 8704, 11264, 13824, 16896 etc. up to 82944 at the
> end of the test).

Scale factor of 9 eh? That probably comes from setting the high end
of the [wr]mem to 16MB. Anyway, shows how autotuning is willing to
grow things quite large indeed.

> The sender advertises 6144 from the second segment
> onwards and sticks to that.

Shawn Osterman's "tcptrace" can take-in a tcpudmp trace and emit files
you can feed into xplot (or whatever replaces it) to show things like
cwnd (probably assuming an older congestion control algorithm) and
show where retransmissions likely took place and such. Sometimes it
is best to take the trace on the receiver, sometimes on the sender.

rick jones

--
the road to hell is paved with business decisions...
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Rick Jones wrote:

> pk <(E-Mail Removed)> wrote:
>> Rick Jones wrote:
>
>> > Just for grins, I would suggest some explicit socket buffer size
>> > settings. I don't know how that is done with iperf or nuttcp, but for
>> > netperf it would be something along the lines of:
>> >
>> > netperf -H <remote> -- -m 64K -s <size> -S <size>
>> >
>> > I'd probably try something like:
>> >
>> > for s in 32 64 128 256
>> > do
>> > netperf -H <remote> -- -m 64K -s $s -S $s
>> > done
>
>> This is what I get (pretty much same results in both directions, so I'm
>> only pasting one direction):
>
> Argh! I left-off the 'K' !-(
>
> that should be
>
> netperf -H <remote> -- -m 64K -s ${s}K -S ${s}K
>
> sorry about that.

Right, so now we're getting again the "expected" results:

# for s in 32 64 128 256; do netperf -H 1x.x.x.x -p 5001 -- -P 5002 -m 64K -
s $sK -S $sK; done
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 5002 AF_INET to 1x.x.x.x
(1x.x.x.x) port 5002 AF_INET : demo
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

1048576 2048 65536 10.30 0.52
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 5002 AF_INET to 1x.x.x.x
(1x.x.x.x) port 5002 AF_INET : demo
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

1048576 2048 65536 10.16 0.53
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 5002 AF_INET to 1x.x.x.x
(1x.x.x.x) port 5002 AF_INET : demo
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

1048576 2048 65536 10.88 0.49
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 5002 AF_INET to 1x.x.x.x
(1x.x.x.x) port 5002 AF_INET : demo
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

1048576 2048 65536 10.08 0.53

>> > Are there any TCP retransmissions recorded on the sender during
>> > the tests?
>
>> Yes, there is an average of ~85 retransimssions on the sending side when
>> running the 4 netperf tests above (in total, not per test).
>
> While the netperf tests above were fubar thanks to my forgetting the
> 'K' a TCP retransmission is generally not goodness when it comes to
> making throughput. That will tend to keep the congestion window (aka
> cwnd) suppressed. How much depends on the congestion control
> algorithm being used.

With the corrected tests, now I'm seeing about 30 retransmits per test
(including the 256K). The available congestion algorithms on the machine are
cubic and reno, both of which give roughly the same results.

>> > What sort of interfaces are being used on the systems on either end?
>
>> One end is physical ethernet, the other end is a KVM virtual machine
>> bridged to an ethernet network (the guest is using the virtio
>> driver). The KVM host is not very loaded, as this is the only VM
>> running on it.
>
> Just for paranoia, you can achieve "good" throughput from the VM to a
> local system?

Yes, both systems can (almost) max out the bandwidth in a single TCP
connection to another local machine.

>> The sender advertises 6144 from the second segment
>> onwards and sticks to that.
>
> Shawn Osterman's "tcptrace" can take-in a tcpudmp trace and emit files
> you can feed into xplot (or whatever replaces it) to show things like
> cwnd (probably assuming an older congestion control algorithm) and
> show where retransmissions likely took place and such. Sometimes it
> is best to take the trace on the receiver, sometimes on the sender.

That is a very good suggestion, I will try it. I'm starting to think that
the problem is in some intermediate node or link.

Thank you again!

pk <(E-Mail Removed)> wrote:
> Right, so now we're getting again the "expected" results:

> # for s in 32 64 128 256; do netperf -H 1x.x.x.x -p 5001 -- -P 5002 -m 64K -
> s $sK -S $sK; done

> ...

> 1048576 2048 65536 10.88 0.49
> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 5002 AF_INET to 1x.x.x.x
> (1x.x.x.x) port 5002 AF_INET : demo
> Recv Send Send
> Socket Socket Message Elapsed
> Size Size Size Time Throughput
> bytes bytes bytes secs. 10^6bits/sec

> 1048576 2048 65536 10.08 0.53

I'm a little concerned that the send socket size is being reported as
2048 there.

> >> > Are there any TCP retransmissions recorded on the sender during
> >> > the tests?


> With the corrected tests, now I'm seeing about 30 retransmits per
> test (including the 256K). The available congestion algorithms on
> the machine are cubic and reno, both of which give roughly the same
> results.

Pity there isn't a "Farragut" ("Damn the retransmissions full-speed
ahead!") module one can use for testing. I suppose though there is -
the UDP_STREAM test. I'd probably use 1460 as the value of -m - we
want to avoid IP fragmentation if we can. Likely as not the sending
side will report something close to local link-rate. It will be the
second line of data that will be of interest.

So, something like:

netperf -H 1.X.X.X.X -p 5001 -t UDP_STREAM -- -P 5002 -m 1460 -s 1M -S 1M

If that also happens to report about half a megabit, I might wonder if
you have bonded links in your path. If it reports more than half a
megabit, it is probably the packet losses holding things up.

Do be careful with that test. BTW, it may require you to add a -R 1
to the test-specific stuff. I had to put-in some code to cover the
backsides of people using netperf for functional testing and doing
link-down. They would do link-down on the test link(s) while netperf
UDP_STREAM was running, the stack would dutifully try to find another
way to get to the destination, and start pumping the UDP_STREAM
traffic out their default route - which happened to be their
production network... So for UDP_STREAM tests by default the socket
has SO_DONTROUTE set on it. It offended by sensibilities to cover the
backsides of folks doing something so stupid as functional testing on
systems without an airgap to their production networks but there you
are... the life of a benchmark maintainer

> > Shawn Osterman's "tcptrace" can take-in a tcpudmp trace and emit
> > files you can feed into xplot (or whatever replaces it) to show
> > things like cwnd (probably assuming an older congestion control
> > algorithm) and show where retransmissions likely took place and
> > such. Sometimes it is best to take the trace on the receiver,
> > sometimes on the sender.

> That is a very good suggestion, I will try it. I'm starting to think
> that the problem is in some intermediate node or link.

happy benchmarking,

rick jones
--
Process shall set you free from the need for rational thought.
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Rick Jones wrote:

> Pity there isn't a "Farragut" ("Damn the retransmissions full-speed
> ahead!") module one can use for testing. I suppose though there is -
> the UDP_STREAM test. I'd probably use 1460 as the value of -m - we
> want to avoid IP fragmentation if we can. Likely as not the sending
> side will report something close to local link-rate. It will be the
> second line of data that will be of interest.
>
> So, something like:
>
> netperf -H 1.X.X.X.X -p 5001 -t UDP_STREAM -- -P 5002 -m 1460 -s 1M -S 1M
>
> If that also happens to report about half a megabit, I might wonder if
> you have bonded links in your path. If it reports more than half a
> megabit, it is probably the packet losses holding things up.

$ netperf -H 1x.x.x.x -p 5002 -t UDP_STREAM -- -P 5003 -m 1460 -s 1M -S 1M
UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 5003 AF_INET to
1x.x.x.x (1x.x.x.x) port 5003 AF_INET : demo
Socket Message Elapsed Messages
Size Size Time Okay Errors Throughput
bytes bytes secs # # 10^6bits/sec

2097152 1460 10.00 1352768 0 1580.01
2097152 10.00 65165 76.11

Ok, now I'm puzzled. Surely 76 Mbit/sec look quite a lot to me. The
bandwidth purchased from the colo at the slowest of the two sites (the one
running the above test) should be around 10 Mbit...I'm not sure how to
interpret those results.

In a single isolated instance, I even got

UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 5003 AF_INET to
1x.x.x.x (1x.x.x.x) port 5003 AF_INET : demo
Socket Message Elapsed Messages
Size Size Time Okay Errors Throughput
bytes bytes secs # # 10^6bits/sec

2097152 1460 10.00 1399679 0 1634.82
2097152 10.00 384976 449.65

which doesn't make any sense to me. Except for this last one, tests in both
directions show figures around 75/79 Mbits/sec as the first one above.

pk <(E-Mail Removed)> wrote:
> Rick Jones wrote:

> > Pity there isn't a "Farragut" ("Damn the retransmissions full-speed
> > ahead!") module one can use for testing. I suppose though there is -
> > the UDP_STREAM test. I'd probably use 1460 as the value of -m - we
> > want to avoid IP fragmentation if we can. Likely as not the sending
> > side will report something close to local link-rate. It will be the
> > second line of data that will be of interest.
> >
> > So, something like:
> >
> > netperf -H 1.X.X.X.X -p 5001 -t UDP_STREAM -- -P 5002 -m 1460 -s 1M -S 1M
> >
> > If that also happens to report about half a megabit, I might wonder if
> > you have bonded links in your path. If it reports more than half a
> > megabit, it is probably the packet losses holding things up.

> $ netperf -H 1x.x.x.x -p 5002 -t UDP_STREAM -- -P 5003 -m 1460 -s 1M -S 1M
> UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 5003 AF_INET to
> 1x.x.x.x (1x.x.x.x) port 5003 AF_INET : demo
> Socket Message Elapsed Messages
> Size Size Time Okay Errors Throughput
> bytes bytes secs # # 10^6bits/sec

> 2097152 1460 10.00 1352768 0 1580.01
> 2097152 10.00 65165 76.11

> Ok, now I'm puzzled. Surely 76 Mbit/sec look quite a lot to me. The
> bandwidth purchased from the colo at the slowest of the two sites
> (the one running the above test) should be around 10 Mbit...I'm not
> sure how to interpret those results.

The top line of numbers is what the number of perceived successful
sendto() calls multiplied by the bytes for each sendto() divided by
the test time (along with unit conversion to 10^6 bits per second -
aka megabits) The second line is what the receiver reported receiving
over the same interval.

> In a single isolated instance, I even got

> UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 5003 AF_INET to
> 1x.x.x.x (1x.x.x.x) port 5003 AF_INET : demo
> Socket Message Elapsed Messages
> Size Size Time Okay Errors Throughput
> bytes bytes secs # # 10^6bits/sec

> 2097152 1460 10.00 1399679 0 1634.82
> 2097152 10.00 384976 449.65

> which doesn't make any sense to me. Except for this last one, tests
> in both directions show figures around 75/79 Mbits/sec as the first
> one above.

There may be some "anomalies" in the way the colo does bandwidth
throttling. And there being bandwidth throttling at/by the colo
tosses-in a whole new set of potential issues with the single-stream
TCP performance...

Do deal with the local (sending side) issue of reporting more than
local link-rate for sending there was recently a patch from Andrew
Gallatin to have netperf set IP_RECVERR on Linux which should at least
partially work-around things going awry in the Linux stack (the flow
control only works for certain relationships between the socket buffer
size and the driver transmit queue depth, or something along those
lines) - the sending side reporting more than local link rate. That
though will not address getting more than what the colo claims to be
giving you into the cloud.

One further thing you could do is add a global -F <filename> where
<filename> is a file with uncompressible data. I suppose there is a
small possibility there is something doing data compression, and that
would be a way to get around it.

If your colo agreement has total bytes transferred
limits/levels/charges, do be careful running netperf tests. It
wouldn't do to have a "free" benchmark cause a big colo bill...

rick jones
--
I don't interest myself in "why". I think more often in terms of
"when", sometimes "where"; always "how much." - Joubert
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Rick Jones wrote:

>> $ netperf -H 1x.x.x.x -p 5002 -t UDP_STREAM -- -P 5003 -m 1460 -s 1M -S
>> 1M UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 5003 AF_INET
>> to 1x.x.x.x (1x.x.x.x) port 5003 AF_INET : demo
>> Socket Message Elapsed Messages
>> Size Size Time Okay Errors Throughput
>> bytes bytes secs # # 10^6bits/sec
>
>> 2097152 1460 10.00 1352768 0 1580.01
>> 2097152 10.00 65165 76.11
>
>> Ok, now I'm puzzled. Surely 76 Mbit/sec look quite a lot to me. The
>> bandwidth purchased from the colo at the slowest of the two sites
>> (the one running the above test) should be around 10 Mbit...I'm not
>> sure how to interpret those results.
>
> The top line of numbers is what the number of perceived successful
> sendto() calls multiplied by the bytes for each sendto() divided by
> the test time (along with unit conversion to 10^6 bits per second -
> aka megabits) The second line is what the receiver reported receiving
> over the same interval.

Well, then...what can I say? Cool! There is more bandwidth than we'd ever
dream of. On the other hand, this makes the TCP performance I measured
earlier look even more ridiculous.

>> In a single isolated instance, I even got
>
>> UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 5003 AF_INET to
>> 1x.x.x.x (1x.x.x.x) port 5003 AF_INET : demo
>> Socket Message Elapsed Messages
>> Size Size Time Okay Errors Throughput
>> bytes bytes secs # # 10^6bits/sec
>
>> 2097152 1460 10.00 1399679 0 1634.82
>> 2097152 10.00 384976 449.65
>
>> which doesn't make any sense to me. Except for this last one, tests
>> in both directions show figures around 75/79 Mbits/sec as the first
>> one above.
>
> There may be some "anomalies" in the way the colo does bandwidth
> throttling. And there being bandwidth throttling at/by the colo
> tosses-in a whole new set of potential issues with the single-stream
> TCP performance...

I'm starting to think that they may be doing QoS on TCP only (eg RED and
so). I haven't got an answer from the colos to this specific question (yet).
I've also ran burst ping tests at 0.01 secs. interval (100 packets/sec) with
almost no packet loss (6 or 7 in 50000 packets, which is within the norm at
those speeds in my experience).

I know that scp isn't a good test, but it's useful because it shows the
speed while it's copying. The oscillating speed I see would be in accordance
with my hypothesis, and also the fact that multiple TCP connections get each
the same bandwidth, so it's possible to get a bigger throughput
cumulatively.

> One further thing you could do is add a global -F <filename> where
> <filename> is a file with uncompressible data. I suppose there is a
> small possibility there is something doing data compression, and that
> would be a way to get around it.
>
> If your colo agreement has total bytes transferred
> limits/levels/charges, do be careful running netperf tests. It
> wouldn't do to have a "free" benchmark cause a big colo bill...

Fortunately that is not the case!

Thanks for your help.