IPV4 Latency Issue: 2.4.18 Kernel

Hi there,

I'm currently working on a project that is, unfortunately, irrevocably
tied to the 2.4.18-3 kernel (RedHat 7.2). It's the runtime system for
what is supposed to be an "appliance", but I'm now arse-deep into the
OS.

Part of the project is to copy a certain amount of data from a data
center on the east coast to one on the west. We've got a sliver of an
OC-3 with a CIR of 10Mbps. RTT latency is 70-75ms.

So far, we've only been able to push (at max) 450KB/s down this line.
Immediately suspecting it to be a TCP window-size problem, I bumped up
the net.ipv4.tcp_wmem and net.ipv4.tcp_rmem parameters. I saw a
slight improvement moving from the defaults to:
min: 8K
def: 128K
max: 256K

I figured window size from:
(windowsize * 8bpB) / RTT = Throughput
or
Windowsize = (Throughput * RTT)/8

Windowsize = (10Mbps * .075)/8 = 96KB

I also changed net.core.rmem/wmem.default/max values to ensure
everything was sane. Currently net.core.rmem/wmem.max is 4MB.

The original throughput was 420KB/s, and upping the window sizes
increased it to 450. Further increasing the window size has had no
effect. SACK and FACK are enabled. The line is not oversubscribed.
Performing the same file copy with a Solaris box as the source
saturates the pipe.

I'm fresh outta ideas. Can anyone be of assistance?

--
Rob on the Web: http://rob.rnovak.net
Electronic Music: http://www.ugotawanit.com

On 2004-04-30, Rob Novak <(E-Mail Removed)> wrote:
>
> Part of the project is to copy a certain amount of data from a data
> center on the east coast to one on the west. We've got a sliver of an
> OC-3 with a CIR of 10Mbps. RTT latency is 70-75ms.
>
> So far, we've only been able to push (at max) 450KB/s down this line.
> Immediately suspecting it to be a TCP window-size problem, I bumped up
> the net.ipv4.tcp_wmem and net.ipv4.tcp_rmem parameters. I saw a
> slight improvement moving from the defaults to:
> min: 8K
> def: 128K
> max: 256K
>
> I figured window size from:
> (windowsize * 8bpB) / RTT = Throughput
> or
> Windowsize = (Throughput * RTT)/8
>
> Windowsize = (10Mbps * .075)/8 = 96KB
>
Aways round up (orders of magnitude you should work to), so use RWIN=128kB
(which you are anyway

> I also changed net.core.rmem/wmem.default/max values to ensure
> everything was sane. Currently net.core.rmem/wmem.max is 4MB.
>
> The original throughput was 420KB/s, and upping the window sizes
> increased it to 450. Further increasing the window size has had no
> effect. SACK and FACK are enabled. The line is not oversubscribed.
> Performing the same file copy with a Solaris box as the source
> saturates the pipe.
>
> I'm fresh outta ideas. Can anyone be of assistance?
>
whats the window size at both ends? Is it this speed in both directions?

I would attach tethereal to the device with

# tethereal -p -n -i ethx -s 40 -w dump

shunt some data and see what ethereal's tcp throughput and rtt graphs show.
This can really help no end.

Send me the blurb if you want.

You also have not mentioned what you are using to shunt the data across, if a
10Mb/s HUB is in the works these normally pack out at 40% if anything else is
on them; however I assume you have a *good* NIC plugged directly into the
OC-3 mcwhatsit.

Cheers

Alex