Two-day outage on Be in Beds/Herts

Broadband has been off for almost two days, with still no estimate of
when the fault is going to be fixed, for Luton, Hitchin, and a number of
other telephone exchanges in Bedfordshire and Hertfordshire. The Be
helpline claims that other broadband suppliers are also affected.

The fault is apparently a break in some fibre owned by BT "between
Luton, Hemel Hempstead, and Berkhampsted" - which seems an awfully large
area of uncertainty.

This seems incredibly incompetent - surely the point of the Internet is
to route around damage. I don't understand why Be doesn't have a good
enough contract with BT or other infrastructure suppliers to guarantee
that they will use an alternative route for data in the event of a
single fibre break. A company the size of BT *must* have lots of
capacity on alternative routes. Maybe it's a ploy by BT to persuade us
migrate to them as a Broadband supplier? If so it may well work -
unless I hear that several lawyers and managers working for Be have been
fired for setting up an inadequate contract with BT, I shall leave Be as
soon as I can. Suggestions for Broadband companies which have more
robust services are welcome.

--
Clive Page

Sky and Orannge also affected
http://www.skyuser.co.uk/forum/sky-b...ge-down-2.html

More info here
http://www.beusergroup.co.uk/

"Plusnet Support Team" <(E-Mail Removed)> wrote in message
news:Tv2dnc9VK_-(E-Mail Removed)...
> On 11/03/2010 09:11, Clive Page wrote:
>> Broadband has been off for almost two days, with still no estimate of
>> when the fault is going to be fixed, for Luton, Hitchin, and a number of
>> other telephone exchanges in Bedfordshire and Hertfordshire. The Be
>> helpline claims that other broadband suppliers are also affected.
>>
>> The fault is apparently a break in some fibre owned by BT "between
>> Luton, Hemel Hempstead, and Berkhampsted" - which seems an awfully large
>> area of uncertainty.
>>
>> This seems incredibly incompetent - surely the point of the Internet is
>> to route around damage. I don't understand why Be doesn't have a good
>> enough contract with BT or other infrastructure suppliers to guarantee
>> that they will use an alternative route for data in the event of a
>> single fibre break. A company the size of BT *must* have lots of
>> capacity on alternative routes. Maybe it's a ploy by BT to persuade us
>> migrate to them as a Broadband supplier? If so it may well work - unless
>> I hear that several lawyers and managers working for Be have been fired
>> for setting up an inadequate contract with BT, I shall leave Be as soon
>> as I can. Suggestions for Broadband companies which have more robust
>> services are welcome.
>
> Only BT MSO I've received notification about recently on the Hitchin
> exchange is the one below from February.
>
> I guess the fault is probably something specific to LLU architecture?
>
> http://usertools.plus.net/exchanges/mso.php?id=4670


From beusergroup.co.uk

Major Outage
Lots of exchanges down..
2010-03-09 13:51:14, by drsox
Edit: This also seems to be affecting Sky, TalkTalk and Tiscali.. including
their phone service on the affected exchanges! Sky report that it is down to
a broken fibre run in the area.

Edit 2: Engineers have confirmed the location of the fibre break. Jointers
are working on the issue this moment. ETA still unknown, however.

Edit 3: Also affecting JANET services (Ashridge) in the area! No further
news though : (

Edit 4: We have the cause: Roadworks damaged an underground cable duct
carrying many cables [JANET, LLU, BT, cellphone services etc.] It also looks
like BE have decided to have website maintenance overnight too, so no status
updates from them :|

Edit 5: Repairs have been going overnight and are still not finished, due to
the location of the fault and the respective safety concerns. ETA *still*
unknown, sadly.

Edit 6: 13:24 10/03/2010 - Sky offer more information than BE do with the
following "Debris stuck in fibre duct, telco are trying to free it.". Still
no ETA : ( 24 hours of downtime now.

Edit 7: BE say: "1/7 lines cut [have been] fixed but safety concerns slowing
BT progress. Sadly still no ETR."

Edit 8: BT say up to 24 hour resolution time for complete repair, but
anytime between then and now there is a chance that some O2/Be lines will
come back up during the repair.

Edit 9: Sky are reporting similar but with longer timescale.. "Telco has
told us that they expect the outage to be a further 24 - 48 hours and have
declared it force majeure*." (*also know as MBORC)

Edit 10: Finally nice to see BE opening up! More information on the broken
cables: "these are major SDH trunk cable with extremely high bandwidth so no
reroute availability somewhere near Junction 6 of the M1"

Edit 11: 22:30 10/03/2010 - No other news to report other than BE and
suppliers are still repairing the damaged cables.

Edit 12: Update from BE "5/7 cables repair [eta] 0:30 - ~8am; last 2 cables
have no ETR [as] additional excavation required."

Edit 13: As far as we can tell (BE HAVE SINCE CONFIRMED!), at around 5am,
HARPENDEN (SMHR) and LEAGRAVE (SMLEA) service was restored. Others remain
down.

Edit 14: BE say "2 more cables restored overnight but BT unable to meet ETRs
previously provided. Still awaiting more updates on 4 cables."

Edit 15: Tiscali on Hitchin has now returned (first graph below) but BE
still out of service.

Edit 16: BE Say: "BT confirms 5/10 affected cables restored; 6th cable
should be restored by 16:30, *should* restore the 5 outstanding ISAMs."

Edit 17: Revised ETA for the 6th Cable = 18:30

Edit 18: JANET services restored shortly after the Tiscali service was,
looks like BE are going to be one of the last reconnected. Or as one twitter
use puts it "@bebroadband what was that whoosing sound?ah another ETR flying
past"

The following exchanges are down:
Luton (SMLT), Leagrave (SMLEA), Hitchin (SMHI), Letchworth (SMLH), Harpenden
(SMHR) and Houghton Regis (SMHUR).

Harpenden and Leagrave should be back up now

"Clive Page" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed)...
>
> This seems incredibly incompetent - surely the point of the Internet is to
> route around damage. I don't understand why Be doesn't have a good
> enough contract with BT or other infrastructure suppliers to guarantee
> that they will use an alternative route for data in the event of a single
> fibre break. A company the size of BT *must* have lots of capacity on
> alternative routes. Maybe it's a ploy by BT to persuade us migrate to
> them as a Broadband supplier? If so it may well work - unless I hear
> that several lawyers and managers working for Be have been fired for
> setting up an inadequate contract with BT, I shall leave Be as soon as I
> can. Suggestions for Broadband companies which have more robust services
> are welcome.

If you're not happy with the service you receive from BE good luck with the
rest....... Frying pan & fire are two words that come to mind!!

Paul

"Paulg0" <(E-Mail Removed)> wrote in message
news:hnbi0e$kfq$(E-Mail Removed)...
>
> "Clive Page" <(E-Mail Removed)> wrote in message
> news:(E-Mail Removed)...
>>

>
> If you're not happy with the service you receive from BE good luck with
> the rest....... Frying pan & fire are two words that come to mind!!
>

Be doing ok for me.
Regards
David

On Thu, 11 Mar 2010 13:39:08 +0000, Plusnet Support Team wrote:

> On 11/03/2010 12:22, Nicola Redwood wrote:
>> Sky and Orannge also affected
>> http://www.skyuser.co.uk/forum/sky-broadband-exchange-
information/36178-dunstable-exchange-down-2.html
>>
>> More info here
>> http://www.beusergroup.co.uk/
>
> Seems like it *is* mainly LLU operators affected then.

Add Tiscali LLU to the list.

--
<http://ale.cx/> (AIM:troffasky) ((E-Mail Removed))
22:18:23 up 24 days, 3:09, 4 users, load average: 0.00, 0.00, 0.00
It is better to have been wasted and then sober
than to never have been wasted at all

On Thu, 11 Mar 2010 09:11:32 +0000, Clive Page wrote:

> This seems incredibly incompetent - surely the point of the Internet is
> to route around damage.

'The internet' didn't get damaged, a fat bundle of fibres did. LLU
operators don't [AFAIK] mesh at all their exchanges because it's not
economically feasible.

> I don't understand why Be doesn't have a good
> enough contract with BT or other infrastructure suppliers to guarantee
> that they will use an alternative route for data in the event of a
> single fibre break.

Can we assume that you'll be willing to pay more to Be so that they can
pay more to BT for Excess Construction Charges to implement diverse
routing on all their backhaul? If they did just that, I'm sure you'd be
whining at how expensive they are.

> A company the size of BT *must* have lots of
> capacity on alternative routes. Maybe it's a ploy by BT to persuade us
> migrate to them as a Broadband supplier?

Yes, I'm sure BT have plenty of unlit capacity in places, but I very much
doubt they give it away for free.

> If so it may well work -
> unless I hear that several lawyers and managers working for Be have been
> fired for setting up an inadequate contract with BT,

I'd much rather Be invest money in their network, than spend it on suing
their suppliers for events that are outwith their control. If suing
people gives you a warm fuzzy feeling, don't bother with broadband at all
and get something that comes with an SLA instead.

> I shall leave Be as soon as I can. Suggestions for Broadband
> companies which have more robust services are welcome.

Yes, I'd be interested to hear who offer better value and a better
product than Be. As far as I'm concerned they're excellent, and the only
thing missing is IPv6.

--
<http://ale.cx/> (AIM:troffasky) ((E-Mail Removed))
22:19:50 up 24 days, 3:11, 4 users, load average: 0.07, 0.02, 0.00
It is better to have been wasted and then sober
than to never have been wasted at all

I'm only just replying to this because we've had the benefit of another
long outage in Luton, Hitchin, and surrounding exchanges, apparently
because the same cable has been damaged again.

In message <hnbra3$lgb$(E-Mail Removed)>, alexd
<(E-Mail Removed)> writes
>'The internet' didn't get damaged, a fat bundle of fibres did. LLU
>operators don't [AFAIK] mesh at all their exchanges because it's not
>economically feasible.

Given the experiences of what are apparently ~50,000 broadband users in
the area who have now been without service for approx 5 days this month,

>Can we assume that you'll be willing to pay more to Be so that they can
>pay more to BT for Excess Construction Charges to implement diverse
>routing on all their backhaul? If they did just that, I'm sure you'd be
>whining at how expensive they are.

Yes indeed.

I would welcome a suggestion for an ISP who does take resilience
seriously. I thought by subscribing to Be, who are by no means the
cheapest of ISPs, I was getting a superior service. Apparently not.

It appears that the non-LLU broadband providers have not been affected
by that. If that means a I don't get quite the same bandwidth but a
more reliable service, then I may switch suppliers.

>Yes, I'd be interested to hear who offer better value and a better
>product than Be. As far as I'm concerned they're excellent, and the only
>thing missing is IPv6.

I'm glad you are satisfied. After experience this month I'm not. And
if you look at ratings on e.g. ispreview.co.uk you'll find it's by no
means one of the top-rated suppliers.


--
Clive Page

Clive Page wrote:
>
>
> I would welcome a suggestion for an ISP who does take resilience
> seriously. I thought by subscribing to Be, who are by no means the
> cheapest of ISPs, I was getting a superior service. Apparently not.
>

years ago we laughed our socks off as the so called super resilient
Pipex, at that point just about the ONLY ISP, majoring in 'diverse
routes' and 'redundancy' was routing traffic from London to Manchester
via the USA, being as how a digger had smashed a bunch of fibres up the
M1, through which conduit ALL their 'multiply redundant' IP links
flowed. different backhaul suppliers were still using the same tier 1
comms providers in terms of light and/or dark fibre. :-)

So with respect Clive, its all pissing in the wind.

BT backhaul is a bit more resilient, but even then, it has its issues.

You want true multiply redundant routes, expect to spend 3-5 times as
much. Or more. I know. I've been involved in their construction and
commissioning.

The LLU operators are operating on the same constraints as anyone else:
which costs more, loss of customers from outages, and penalty payments
to customers who are 'outed' as it were, or loss of customers to
competing services at lower reliability, but cheaper costs.

The strak reality is that the major cost of operating a WAN is in the
provision and maintenance of the WAN links.

It costs a bit less to buy a virtual poipe over BT'S network, than a
virtual service from BT. that enables the LLUS to be slightly cost
competitive. The downside is that virtual pipe is slightly less
resilient, and BT may chose to repair its own circuits first, depending
on the contracts.

True resilience only happens at the tier 1 telco level, where they
actually know where their fibres run.

If you buy a pipe from e.g. London to Paris, and one from London to
Amsterdam and one from Paris to Amsterdam you may think you have bought
redundancy. Until someone slices a bundle in the channel tunnel or
somewhere, and you lose complete connectivity cross channel altogether.

It cots a fortune to lay long distance fibres, and no one in their right
minds lays them two completely different routes. It doubles the costs.

So there is always a significant chance that a major issue will result
in whole sections of the net being knocked out, or at least routing
around issues extremely sub-optimally.