Speed tests: are they actually worth anything?

I've just run (well, at around 5:30pm) a series of speed tests offered by
different sites... one straight after another. Results below, but first,
details of my setup:

ISP: Plus Net, "Broadband Your Way Option 2" -- a legacy package, originally
8Mbit, but now upgraded to ADSL2+ by agreement with Plus Net. I live about
half a mile from my local exchange; used to connect at around 7.5Mbits with
IP profile around 6.5 Mbits; after much variability following the
implementation of ADSL2+ at the exchange, it has settled down to around 22.5
Mbits with an IP profile of around 20 Mbits.

Router: Netgear DG834G v3 with up-to-date firmware. Short on displayable
stats, but:
Connection Speed (right now): down 22767 kbps; up 439 kbps
Line Attenuation: 11 db; up 3 db
Noise Margin: down 3 db; up 2147483647 db (an odd but apparently common
problem with this router; it appears to mean zero db... ???)

Pingtest net (using the best available server):
Packet loss: zero
Ping: 17ms
Jitter 1ms.


And the speed tests:

Speedtest.net <http://www.speedtest.net/index.php>--the one commonly
recommended by BT phone engineers:
Down: 19.67 Mbps; Up: 0.37 Mbps

My Broadband Speed <http://www.mybroadbandspeed.co.uk/>--Plus Net's own:
Down: 13383 Kbps; Up 373 Kbps.
The behaviour of this server has been peculiar ever since its recent outage
and repair by Plus Net: the download speed during the test jumps immediately
to around 6.5 Mbps and stays there for the majority of the duration; then,
when about two thirds of the data has been downloaded, the data rate starts
climbing rapidly and is still doing so when the test stops.

Broadband Max <http://www.speedtest.bbmax.co.uk/> (similar Ookla software to
that used by the above, but without the long pause at around 6.5 Mbps):
Down: 18611 Kbps; Up: 370 Kbps.

BBC iPlayer speed test <http://www.bbc.co.uk/iplayer/diagnostics> (it's
working again now after a period during which the "streaming speed (3)" test
refused to run if you weren't using the right ISP):
Download speed: 16957 Kbps
Streaming speed (1): 6207 Kbps (this, while never the fastest of the three
streaming speed tests, used to report a much higher figure but something was
changed such that for the past few weeks it's always around 6.2 Mbps).
Streaming speed (2): 11697 Kbps
Streaming speed (3): 18942 Kbps.

Thinkbroadband <http://www.thinkbroadband.com/speedtest.html>--unlike most
this isn't an Ookla http-based test; it uses Java:
Down: 17.2 Mbps; Up: 0.4 Mbps.

5 FWD <http://fwd.channel5.com/adv/broadbandspeedtest>:
Down: 17.19 Mbps; Up: 0.36 Mbps.

Broadband Speed Checker <http://www.broadbandspeedchecker.co.uk/>:
Down: 19792 Kbps; Up: 362 Kbps.

BT speed tester <http://www.speedtester.bt.com/>--BT's own Java-based test
complete with frequent outages and huge delays before and between tests:
IP profile: 20085 Kbps (in other words BT have reduced, for stability's
sake, the maximum throughput by around 2.5 Mbps)
Down: 15662 Kbps
Up: 363 Kbps.


These tests are supposed to tell you what your connection--or rather your
connection and ISP package--are capable of right now, IF the server from
which you're downloading is not itself the weak link and IF, since most ISPs
indulge in traffic shaping (i.e. selective throttling) they are treating
your current download as priority traffic; one assumes that ISPs always
treat a speed test server as top priority. Or at least they *used* to.

But are they worth anything when they vary as much as the above, with some
reporting speeds as much as 40 per cent greater than others? I *know* that
Plus Net's own speed test is either buggy or has been deliberately set to
lower expectations, since its low figure is pretty consistent and is some
four Mbps lower than the speed at which I lately downloaded a chunky video
from a server in Japan.

At the same time, very little of the data I access actually downloads at
even the lowest of the above speeds, regardless of the time of day. I have,
for example, an eight-tab browser session that I load several times a day;
each tab points at the home page of a popular news site, so data is being
requested of a minimum of eight servers simultaneously; yet the total
download rate while these eight pages are loading--as revealed by a
real-time graph provided by a traffic monitoring program called
Networx--*never* rises above six Mbps and the average throughput is closer
to two Mbps.

So what the hell are these tests really measuring?


I wonder whether the loathsome BT speed test software (which is so
unreliable and slow that I would never use it were it not the only means I
know of ascertaining my "IP profile") hasn't recently provided a clue: for a
few weeks it provided a little extra information, in the form of the
percentages of the traffic analysed by the test that were treated as "sub
best effort", "normal best effort" and "priority best effort". These
percentages varied wildly from test to test and seemed to bear no relation
to time of day.

This information is no longer displayed but the BT test remains, to put it
mildly, cantankerous. Yesterday I watched the Networx display is it ran the
download test: the speed climbed, over a period of around six seconds, to
around 18 Mbps--and then stayed there for the majority of the test, tailing
off to zero over the last three seconds or so. The test then reported a
download speed of less than 10 Mbps. But today's test, which resulted in a
virtually identical graph, reported a download speed of more than 15 Mbps.


So: can anyone explain to me--in reasonably clear terms--what's going on
here, and whether *any* of these speed tests are really doing anything for
me--other than using up my monthly download allowance?


--
Regards, Peter Boulding
(E-Mail Removed) (to e-mail, remove "UNSPAM")
Fractal Images and Music: http://www.pboulding.co.uk/
http://www.soundclick.com/bands/defa...&content=music

"Peter Boulding" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed)...
>
> I've just run (well, at around 5:30pm) a series of speed tests offered by
> different sites... one straight after another. Results below, but first,
> details of my setup:
>
> ISP: Plus Net, "Broadband Your Way Option 2" -- a legacy package,
> originally
> 8Mbit, but now upgraded to ADSL2+ by agreement with Plus Net. I live about
> half a mile from my local exchange; used to connect at around 7.5Mbits
> with
> IP profile around 6.5 Mbits; after much variability following the
> implementation of ADSL2+ at the exchange, it has settled down to around
> 22.5
> Mbits with an IP profile of around 20 Mbits.
>
> Router: Netgear DG834G v3 with up-to-date firmware. Short on displayable
> stats, but:
> Connection Speed (right now): down 22767 kbps; up 439 kbps
> Line Attenuation: 11 db; up 3 db
> Noise Margin: down 3 db; up 2147483647 db (an odd but apparently common
> problem with this router; it appears to mean zero db... ???)
>
> Pingtest net (using the best available server):
> Packet loss: zero
> Ping: 17ms
> Jitter 1ms.
>
>
> And the speed tests:
>
> Speedtest.net <http://www.speedtest.net/index.php>--the one commonly
> recommended by BT phone engineers:
> Down: 19.67 Mbps; Up: 0.37 Mbps
>
> My Broadband Speed <http://www.mybroadbandspeed.co.uk/>--Plus Net's own:
> Down: 13383 Kbps; Up 373 Kbps.
> The behaviour of this server has been peculiar ever since its recent
> outage
> and repair by Plus Net: the download speed during the test jumps
> immediately
> to around 6.5 Mbps and stays there for the majority of the duration; then,
> when about two thirds of the data has been downloaded, the data rate
> starts
> climbing rapidly and is still doing so when the test stops.
>
> Broadband Max <http://www.speedtest.bbmax.co.uk/> (similar Ookla software
> to
> that used by the above, but without the long pause at around 6.5 Mbps):
> Down: 18611 Kbps; Up: 370 Kbps.
>
> BBC iPlayer speed test <http://www.bbc.co.uk/iplayer/diagnostics> (it's
> working again now after a period during which the "streaming speed (3)"
> test
> refused to run if you weren't using the right ISP):
> Download speed: 16957 Kbps
> Streaming speed (1): 6207 Kbps (this, while never the fastest of the three
> streaming speed tests, used to report a much higher figure but something
> was
> changed such that for the past few weeks it's always around 6.2 Mbps).
> Streaming speed (2): 11697 Kbps
> Streaming speed (3): 18942 Kbps.
>
> Thinkbroadband <http://www.thinkbroadband.com/speedtest.html>--unlike most
> this isn't an Ookla http-based test; it uses Java:
> Down: 17.2 Mbps; Up: 0.4 Mbps.
>
> 5 FWD <http://fwd.channel5.com/adv/broadbandspeedtest>:
> Down: 17.19 Mbps; Up: 0.36 Mbps.
>
> Broadband Speed Checker <http://www.broadbandspeedchecker.co.uk/>:
> Down: 19792 Kbps; Up: 362 Kbps.
>
> BT speed tester <http://www.speedtester.bt.com/>--BT's own Java-based test
> complete with frequent outages and huge delays before and between tests:
> IP profile: 20085 Kbps (in other words BT have reduced, for stability's
> sake, the maximum throughput by around 2.5 Mbps)
> Down: 15662 Kbps
> Up: 363 Kbps.
>
>
> These tests are supposed to tell you what your connection--or rather your
> connection and ISP package--are capable of right now, IF the server from
> which you're downloading is not itself the weak link and IF, since most
> ISPs
> indulge in traffic shaping (i.e. selective throttling) they are treating
> your current download as priority traffic; one assumes that ISPs always
> treat a speed test server as top priority. Or at least they *used* to.
>
> But are they worth anything when they vary as much as the above, with some
> reporting speeds as much as 40 per cent greater than others? I *know* that
> Plus Net's own speed test is either buggy or has been deliberately set to
> lower expectations, since its low figure is pretty consistent and is some
> four Mbps lower than the speed at which I lately downloaded a chunky video
> from a server in Japan.
>
> At the same time, very little of the data I access actually downloads at
> even the lowest of the above speeds, regardless of the time of day. I
> have,
> for example, an eight-tab browser session that I load several times a day;
> each tab points at the home page of a popular news site, so data is being
> requested of a minimum of eight servers simultaneously; yet the total
> download rate while these eight pages are loading--as revealed by a
> real-time graph provided by a traffic monitoring program called
> Networx--*never* rises above six Mbps and the average throughput is closer
> to two Mbps.
>
> So what the hell are these tests really measuring?
>
>
> I wonder whether the loathsome BT speed test software (which is so
> unreliable and slow that I would never use it were it not the only means I
> know of ascertaining my "IP profile") hasn't recently provided a clue: for
> a
> few weeks it provided a little extra information, in the form of the
> percentages of the traffic analysed by the test that were treated as "sub
> best effort", "normal best effort" and "priority best effort". These
> percentages varied wildly from test to test and seemed to bear no relation
> to time of day.
>
> This information is no longer displayed but the BT test remains, to put it
> mildly, cantankerous. Yesterday I watched the Networx display is it ran
> the
> download test: the speed climbed, over a period of around six seconds, to
> around 18 Mbps--and then stayed there for the majority of the test,
> tailing
> off to zero over the last three seconds or so. The test then reported a
> download speed of less than 10 Mbps. But today's test, which resulted in a
> virtually identical graph, reported a download speed of more than 15 Mbps.
>
>
> So: can anyone explain to me--in reasonably clear terms--what's going on
> here, and whether *any* of these speed tests are really doing anything for
> me--other than using up my monthly download allowance?
>
>
> --
> Regards, Peter Boulding
> (E-Mail Removed) (to e-mail, remove "UNSPAM")
> Fractal Images and Music: http://www.pboulding.co.uk/
> http://www.soundclick.com/bands/defa...&content=music

The reported speeds depend not only on your connection to the ISP but also
the lottery of the Internet connections between you and the speed test
server. From my experience your download results today were actually rather
consistent. One of your sites was speedtest.net which offers a wide range of
test servers. Try a few and you will see what I mean - often the suggested
one is nowhere near the fastest.

The tests only show the results at the time the test is conducted. Therefore
any comparison has no real statistical meaning. The answer is not to get
obsessed with the figures because unless you can actually detect any
difference whilst you use the net then they have no relevance.


Peter Crosland

On Fri, 26 Aug 2011 21:35:05 +0100, "Peter Crosland" <(E-Mail Removed)>
wrote in <(E-Mail Removed) >:

>The tests only show the results at the time the test is conducted. Therefore
>any comparison has no real statistical meaning. The answer is not to get
>obsessed with the figures because unless you can actually detect any
>difference whilst you use the net then they have no relevance.

I am aware that the tests will vary with network loads at different times of
day. Or even, in theory, from second to second. But...

<retests the first three on the list>

Speedtest.net (yes, Mike, I have checked to see which is the fastest server)
Now: 18.25 Mbps. Earlier: 19.68 Mbps.

My Broadband Speed (Plus Net)
Now: 13,483 Kbps. Earlier: 13,383 Kbps.

Namesco Broadband Max
Now: 19,863 Kbps. Earlier: 18,611.


See what I mean?


The question remains: What, if anything, are they actually testing that's of
any use to us?


--
Regards, Peter Boulding
(E-Mail Removed) (to e-mail, remove "UNSPAM")
Fractal Images and Music: http://www.pboulding.co.uk/
http://www.soundclick.com/bands/defa...&content=music

"Peter Boulding" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed)...
> On Fri, 26 Aug 2011 21:35:05 +0100, "Peter Crosland" <(E-Mail Removed)>
> wrote in <(E-Mail Removed) >:
>
>>The tests only show the results at the time the test is conducted.
>>Therefore
>>any comparison has no real statistical meaning. The answer is not to get
>>obsessed with the figures because unless you can actually detect any
>>difference whilst you use the net then they have no relevance.
>
> I am aware that the tests will vary with network loads at different times
> of
> day. Or even, in theory, from second to second. But...
>
> <retests the first three on the list>
>
> Speedtest.net (yes, Mike, I have checked to see which is the fastest
> server)
> Now: 18.25 Mbps. Earlier: 19.68 Mbps.
>
> My Broadband Speed (Plus Net)
> Now: 13,483 Kbps. Earlier: 13,383 Kbps.
>
> Namesco Broadband Max
> Now: 19,863 Kbps. Earlier: 18,611.
>
>
> See what I mean?
>
>
> The question remains: What, if anything, are they actually testing that's
> of
> any use to us?


In the majority of cases nothing. Some people become quite obsessive and
paranoid about the figures. The bottom line is if you don't notice the
difference in normal use then forget about them!

Peter Crosland

Peter Boulding wrote:
> On Fri, 26 Aug 2011 21:35:05 +0100, "Peter Crosland"<(E-Mail Removed)>
> wrote in<(E-Mail Removed). uk>:
>
>> The tests only show the results at the time the test is conducted. Therefore
>> any comparison has no real statistical meaning. The answer is not to get
>> obsessed with the figures because unless you can actually detect any
>> difference whilst you use the net then they have no relevance.
>
> I am aware that the tests will vary with network loads at different times of
> day. Or even, in theory, from second to second. But...
>
> <retests the first three on the list>
>
> Speedtest.net (yes, Mike, I have checked to see which is the fastest server)
> Now: 18.25 Mbps. Earlier: 19.68 Mbps.
>
> My Broadband Speed (Plus Net)
> Now: 13,483 Kbps. Earlier: 13,383 Kbps.
>
> Namesco Broadband Max
> Now: 19,863 Kbps. Earlier: 18,611.
>
>
> See what I mean?
>
>
> The question remains: What, if anything, are they actually testing that's of
> any use to us?

They're giving you a crude instantaneous indication of backhaul congestion.

Some ISPs will show you the BRAS profile of your line via a management
web page, so you should always use that rather than trying to infer the
BRAS profile from the actual download speeds. The BRAS profile is the
limiting factor, and ideally is a fixed high proportion of the
synchronisation speed of the line (itself determined by the length and
quality of the line). But it may reduce for long periods following a
noise event that affects the synchronisation speed only momentarily.

If you test particular servers at regular times you might build up a
pattern of performance, and therefore be able to identify when
performance is different; but what are you going to do about it?

If you have paid for a SLA then you should have some agreed monitoring
procedure so you can establish when your ISP gives you less than you've
paid for.

--
Graham J

"Peter Boulding" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed)...
>
>
> So what the hell are these tests really measuring?
>
They are clearly showing that your download speeds are very much better than
mine, - by more that an order of magnitude.


--
Michael Chare

On Fri, 26 Aug 2011 23:29:58 +0100, "Michael Chare" <(E-Mail Removed)>
wrote in <j396p8$ds8$(E-Mail Removed)>:

>They are clearly showing that your download speeds are very much better than
>mine, - by more that an order of magnitude.

I'm sorry about that, even though I suspect you're referring to speed test
results--for what they're worth, which appears to be very little--rather
than real life results--e.g. those irritating delays you suffer when
surfing, usenetting, and so on.


--
Regards, Peter Boulding
(E-Mail Removed) (to e-mail, remove "UNSPAM")
Fractal Images and Music: http://www.pboulding.co.uk/
http://www.soundclick.com/bands/defa...&content=music

In article <(E-Mail Removed)>, Peter Boulding
wrote:
> So what the hell are these tests really measuring?

Very little. If you want to know the attainable speed of your internet
connection, the best way is to download something from your own ISP,
preferably several files simultaneously, adding the transfer rates
together. If you normally use a wireless connection, you should connect
by ethernet cable for the test.

A ping time is meaningless if you don't know the distance to the test
site - as if you were to express the speed of a car by giving a figure
called "journey time" without saying where the car was going. Generally
speaking if the ping time to your own gateway isn't in single digits,
there might be a problem, but other than that the figure is just an
indication of how far away the test website is from you, and nothing
much about your connection. Try pinging a site in America or Australia,
indiana.edu or sydney.edu.au for example, both via broadband and via a
dial-up connection, and you'll see what I mean.

Rod.
--
Virtual Access V6.3 free usenet/email software from
http://sourceforge.net/projects/virtual-access/

Peter Boulding wrote:
> On Fri, 26 Aug 2011 21:35:05 +0100, "Peter Crosland" <(E-Mail Removed)>
> wrote in <(E-Mail Removed) >:
>
>> The tests only show the results at the time the test is conducted. Therefore
>> any comparison has no real statistical meaning. The answer is not to get
>> obsessed with the figures because unless you can actually detect any
>> difference whilst you use the net then they have no relevance.
>
> I am aware that the tests will vary with network loads at different times of
> day. Or even, in theory, from second to second. But...
>
> <retests the first three on the list>
>
> Speedtest.net (yes, Mike, I have checked to see which is the fastest server)
> Now: 18.25 Mbps. Earlier: 19.68 Mbps.
>
> My Broadband Speed (Plus Net)
> Now: 13,483 Kbps. Earlier: 13,383 Kbps.
>
> Namesco Broadband Max
> Now: 19,863 Kbps. Earlier: 18,611.
>
>
> See what I mean?
>
>
> The question remains: What, if anything, are they actually testing that's of
> any use to us?
>
>
They are testing end to end connections.

Normally the local loop is the weakest link, but with FTTC et al that
can no longer be held to be the case.

I find them useful to guesstimate what my BRAS is, but beyond that they
do little.