tcp send bigger data faster then small data

Hi,

I did:


typedef long long s64;
inline s64 getRealTime() {
s64 result;
__asm__ __volatile__ ("rdtsc" : "=A" (result));
return result;
}
s64 start64, end64;
tcpSender testSender2=tcpSender(argv[1],line,length);
testSender2.init(option); //SO_LINGER,bind,connect
start64=getRealTime();
testSender2.work();
closeid=testSender2.end(); //close(socket)
end64 = getRealTime();


I measured the time, I did 10 rounds to get the average value. I use Red
Hat 9.1, g++ 2.96, the data is produced by random, 100 MBit Ethernet,
switched. I used SO_LINGER to wait till the queue has been send correctly.

I send 8000, 10240 and 80000 bytes over the network.
I did this several times cause I couldn't believe it, but it was all
time the same
it send 8000 bytes in 32 ms, 10240 bytes in 40.8 ms and
80000 bytes in 10.5 ms.

why is it faster to send 80000 bytes then less bytes? who can explain? I
can use any idea.


Thanks
Stephan

"Stephan Absmeier" <(E-Mail Removed)> wrote:

> I send 8000, 10240 and 80000 bytes over the network.
> I did this several times cause I couldn't believe it, but it was all
> time the same
> it send 8000 bytes in 32 ms, 10240 bytes in 40.8 ms and
> 80000 bytes in 10.5 ms.
>
> why is it faster to send 80000 bytes then less bytes? who can explain? I
> can use any idea.

I think your timing method is flawed. Are you calculating and
printing the 64bit TSC-diff correctly, or maybe rdtsc wraps?
Some libs uses %lld and some uses %Ld to print a signed 64-bit
value. Why not use clock() instead? Plenty precision for what's you
are measuring.

--gv

Gisle Vanem schrieb:
> "Stephan Absmeier" <(E-Mail Removed)> wrote:
>
>
>>I send 8000, 10240 and 80000 bytes over the network.
>>I did this several times cause I couldn't believe it, but it was all
>>time the same
>>it send 8000 bytes in 32 ms, 10240 bytes in 40.8 ms and
>>80000 bytes in 10.5 ms.
>>
>>why is it faster to send 80000 bytes then less bytes? who can explain? I
>>can use any idea.
>
>
> I think your timing method is flawed. Are you calculating and
> printing the 64bit TSC-diff correctly, or maybe rdtsc wraps?
> Some libs uses %lld and some uses %Ld to print a signed 64-bit
> value. Why not use clock() instead? Plenty precision for what's you
> are measuring.
>
> --gv
>
>

I changed s64 to clo9ck_t and getRealTime() to clock() and it always
returned 0 and one time 1000.
Did I something wrong?

Gisle Vanem schrieb:
> "Stephan Absmeier" <(E-Mail Removed)> wrote:
>
>
>>I send 8000, 10240 and 80000 bytes over the network.
>>I did this several times cause I couldn't believe it, but it was all
>>time the same
>>it send 8000 bytes in 32 ms, 10240 bytes in 40.8 ms and
>>80000 bytes in 10.5 ms.
>>
>>why is it faster to send 80000 bytes then less bytes? who can explain? I
>>can use any idea.
>
>
> I think your timing method is flawed. Are you calculating and
> printing the 64bit TSC-diff correctly, or maybe rdtsc wraps?
> Some libs uses %lld and some uses %Ld to print a signed 64-bit
> value. Why not use clock() instead? Plenty precision for what's you
> are measuring.
>
> --gv
>
>
I tried gettimeofday and again: sending 8000 bytes needs 36 to 38
ms,10240 bytes needs 37 to 38 ms and 80000 bytes needs 8 to 47 ms, 14.9
ms in the average. But 800000 bytes need 68.6 to 85 ms.

double after;
timeval end;
gettimeofday(&end,NULL);
after=end.tv_sec+1.e-6*end.tv_usec;


the result is like the result of getRealTime
8 out of 10 runs transfering 80000 bytes are faster then 10 ms and for
that faster then all the runs sending 8000 or 10240 bytes.

have anybody an idea?
Stephan

Stephan Absmeier schrieb:
>
>
> Gisle Vanem schrieb:
>
>> "Stephan Absmeier" <(E-Mail Removed)> wrote:
>>
>>
>>> I send 8000, 10240 and 80000 bytes over the network.
>>> I did this several times cause I couldn't believe it, but it was all
>>> time the same
>>> it send 8000 bytes in 32 ms, 10240 bytes in 40.8 ms and
>>> 80000 bytes in 10.5 ms.
>>>
>>> why is it faster to send 80000 bytes then less bytes? who can explain? I
>>> can use any idea.
>>
>>
>>
>> I think your timing method is flawed. Are you calculating and
>> printing the 64bit TSC-diff correctly, or maybe rdtsc wraps?
>> Some libs uses %lld and some uses %Ld to print a signed 64-bit
>> value. Why not use clock() instead? Plenty precision for what's you
>> are measuring.
>>
>> --gv
>>
>>
> I tried gettimeofday and again: sending 8000 bytes needs 36 to 38
> ms,10240 bytes needs 37 to 38 ms and 80000 bytes needs 8 to 47 ms, 14.9
> ms in the average. But 800000 bytes need 68.6 to 85 ms.
>
> double after;
> timeval end;
> gettimeofday(&end,NULL);
> after=end.tv_sec+1.e-6*end.tv_usec;
>
>
> the result is like the result of getRealTime
> 8 out of 10 runs transfering 80000 bytes are faster then 10 ms and for
> that faster then all the runs sending 8000 or 10240 bytes.
>
> have anybody an idea?
> Stephan
>

I did some more tests, the runs are seconds:

Bytes fastest run average run slowest run
8000 0.031323 0.0315113 0.03211
10240 0.040979 0.0410652 0.041146
20000 0.040621 0.0407288 0.040818
30000 0.040002 0.0403257 0.040455
35000 0.003506 0.0035568 0.003608
40000 0.003849 0.0289824 0.039732
50000 0.004638 0.0046604 0.004724
80000 0.007648 0.0077132 0.007806
100000 0.008806 0.0097092 0.016487
200000 0.01764 0.0208155 0.048285
300000 0.02585 0.029469 0.058012
350000 0.030264 0.0304887 0.030698
400000 0.034348 0.041239 0.067274
500000 0.042902 0.0431682 0.043544
800000 0.068526 0.0735361 0.106607

it starts between 30000 and 35000 bytes to be faster then less bytes and
ends at 350000 bytes. 100 MBit means 35000 bytes need a minimum time of
0.0028 seconds (without overhead).

In comp.os.linux.development.system Stephan Absmeier <(E-Mail Removed)> wrote:
> I did some more tests, the runs are seconds:

That's good. Can you do some more graphing, please (and no tabs!!!
Fixing).

> Bytes fastest run average run slowest run
> 8000 0.031323 0.0315113 0.03211
> 10240 0.040979 0.0410652 0.041146
> 20000 0.040621 0.0407288 0.040818
> 30000 0.040002 0.0403257 0.040455
> 35000 0.003506 0.0035568 0.003608
> 40000 0.003849 0.0289824 0.039732
> 50000 0.004638 0.0046604 0.004724
> 80000 0.007648 0.0077132 0.007806
> 100000 0.008806 0.0097092 0.016487
> 200000 0.01764 0.0208155 0.048285
> 300000 0.02585 0.029469 0.058012
> 350000 0.030264 0.0304887 0.030698
> 400000 0.034348 0.041239 0.067274
> 500000 0.042902 0.0431682 0.043544
> 800000 0.068526 0.0735361 0.106607

Yes - this sort of accords with a vague idea I have that over a small
number of fragments the NIC will wait to see if there are any more
incoming before telling the system it has something.

> it starts between 30000 and 35000 bytes to be faster then less bytes and
> ends at 350000 bytes. 100 MBit means 35000 bytes need a minimum time of
> 0.0028 seconds (without overhead).

What I don't get is how you can send 800KB in 0.07 seconds. That's
11.2MB/s. Oh well, 100BT. Yes, some NICs probably do the wait-and-see
trick.

Peter

Peter T. Breuer schrieb:
> In comp.os.linux.development.system Stephan Absmeier <(E-Mail Removed)> wrote:
>
>>I did some more tests, the runs are seconds:
>
>
> That's good. Can you do some more graphing, please (and no tabs!!!
> Fixing).
>
>
>>Bytes fastest run average run slowest run
>> 8000 0.031323 0.0315113 0.03211
>> 10240 0.040979 0.0410652 0.041146
>> 20000 0.040621 0.0407288 0.040818
>> 30000 0.040002 0.0403257 0.040455
>> 35000 0.003506 0.0035568 0.003608
>> 40000 0.003849 0.0289824 0.039732
>> 50000 0.004638 0.0046604 0.004724
>> 80000 0.007648 0.0077132 0.007806
>>100000 0.008806 0.0097092 0.016487
>>200000 0.01764 0.0208155 0.048285
>>300000 0.02585 0.029469 0.058012
>>350000 0.030264 0.0304887 0.030698
>>400000 0.034348 0.041239 0.067274
>>500000 0.042902 0.0431682 0.043544
>>800000 0.068526 0.0735361 0.106607
>
>
> Yes - this sort of accords with a vague idea I have that over a small
> number of fragments the NIC will wait to see if there are any more
> incoming before telling the system it has something.
>
>
>>it starts between 30000 and 35000 bytes to be faster then less bytes and
>>ends at 350000 bytes. 100 MBit means 35000 bytes need a minimum time of
>>0.0028 seconds (without overhead).
>
>
> What I don't get is how you can send 800KB in 0.07 seconds. That's
> 11.2MB/s. Oh well, 100BT. Yes, some NICs probably do the wait-and-see
> trick.
That's one of my problems. That's not like tcp should act cause of slow
start.
This all was done with the default settings after insatllation. Turning off
the Nagle-Algorithm (TCP_NODELAY) speeds it up a little more. The fastest
was 0.06831 seconds for 800000 bytes.That means 93.6 MBit/s!! If I ping
the other host, it needs 0.00014 to 0.00015 seconds.

I go nuts on this.
Stephan

In comp.os.linux.development.system Stephan Absmeier <(E-Mail Removed)> wrote:
> Peter T. Breuer schrieb:
> > In comp.os.linux.development.system Stephan Absmeier <(E-Mail Removed)> wrote:
> >
> >>I did some more tests, the runs are seconds:
> >
> > That's good. Can you do some more graphing, please (and no tabs!!!
> > Fixing).

Can you please do that? The graphing, I mean. We need more data points.
And indicate how you are doing these tests, so that the results become
reproducible (and meaningful!).

> >>Bytes fastest run average run slowest run
> >> 8000 0.031323 0.0315113 0.03211
> >> 10240 0.040979 0.0410652 0.041146
> >> 20000 0.040621 0.0407288 0.040818
> >> 30000 0.040002 0.0403257 0.040455
> >> 35000 0.003506 0.0035568 0.003608
> >> 40000 0.003849 0.0289824 0.039732
> >> 50000 0.004638 0.0046604 0.004724
> >> 80000 0.007648 0.0077132 0.007806
> >>100000 0.008806 0.0097092 0.016487
> >>200000 0.01764 0.0208155 0.048285
> >>300000 0.02585 0.029469 0.058012
> >>350000 0.030264 0.0304887 0.030698
> >>400000 0.034348 0.041239 0.067274
> >>500000 0.042902 0.0431682 0.043544
> >>800000 0.068526 0.0735361 0.106607
> >
> >
> > Yes - this sort of accords with a vague idea I have that over a small
> > number of fragments the NIC will wait to see if there are any more
> > incoming before telling the system it has something.

> That's one of my problems. That's not like tcp should act cause of slow
> start.

Eh? What I suggest is just due to the way some 100BT NICs behave.

> This all was done with the default settings after insatllation. Turning off
> the Nagle-Algorithm (TCP_NODELAY) speeds it up a little more. The fastest

What I suggest is nothing to do with the o/s.

Please perform the further experiments to confirm or deny. Detail your
measurement mode. To make valid measurements you will need to let the
NIC quiesce after each "packet". At least 1s delay.

You should then remeasure on a continuous stream, taking the average
speed on the stream. If I am right, all should converge there.


Peter

"Peter T. Breuer" <(E-Mail Removed)> writes:

> Please perform the further experiments to confirm or deny. Detail your
> measurement mode. To make valid measurements you will need to let the
> NIC quiesce after each "packet". At least 1s delay.

Also, don't always start with the smaller files and work your way up
to bigger ones - there are probably some startup costs associated with
the first transfer. ARP comes to mind - are you sure that both hosts
have the other's MAC addrs in their ARP caches before you start
testing?

--KW 8-)
--
Keith Wansbrough <(E-Mail Removed)>
http://www.cl.cam.ac.uk/users/kw217/
University of Cambridge Computer Laboratory.

Keith Wansbrough schrieb:
> "Peter T. Breuer" <(E-Mail Removed)> writes:
>
>
>>Please perform the further experiments to confirm or deny. Detail your
>>measurement mode. To make valid measurements you will need to let the
>>NIC quiesce after each "packet". At least 1s delay.

I don't have root rights. How should I quiesce the card?
There is no packet delay module installed.

>
>
> Also, don't always start with the smaller files and work your way up
> to bigger ones - there are probably some startup costs associated with
> the first transfer. ARP comes to mind - are you sure that both hosts
> have the other's MAC addrs in their ARP caches before you start
> testing?
>
> --KW 8-)

Hi,

this time upside down. I started with 800000 Bytes going to
8000 Bytes:

Bytes fastest run average run slowest run
800000 0.068346 0.0688138 0.069734
500000 0.042749 0.046447 0.073162
400000 0.034251 0.0346192 0.035127
350000 0.030081 0.0334658 0.062391
300000 0.026015 0.0261624 0.026631
200000 0.017581 0.0177638 0.017923
100000 0.009186 0.0092719 0.009383
80000 0.007103 0.0109007 0.0411
50000 0.0048 0.0048566 0.005025
40000 0.003765 0.0258131 0.040525
35000 0.003641 0.0036926 0.003859
30000 0.039624 0.039726 0.039826
20000 0.038596 0.0392895 0.039463
10240 0.037367 0.0401245 0.049999
8000 0.03831 0.0386388 0.039061

nearly the same values as before but exactly the same picture.

It is done in a loop. There is a 1 second break between the
runs and a 5 second break between the different packets. The
socket has benn closed after each run, cause of time measure
methods.

arp can be a factor at the first run, but not later: I had a
look at /proc/net/arp, it wasn't there, I pinged the other
host, it was there, three minute later, the MAC address was
still there.

Stephan