Parts Of UK Booted Offline As Virgin Media Suffers Massive Broadband Outage

Updated Liberty-owned UK broadband pusher Virgin Media has fallen on its face this morning, with users across the country reporting complete broadband failure for a number of hours, among them some of the reporters on this news desk.

Earlier today, even its status page was unavailable, although prior to this readers who alerted The Reg were complaining that "their service status reports no issues like it always does." Its website staggered to its feet at around 0830 UTC.

Cloudflare tracked AS5089, Virgin Media's autonomous system number, as being subject to a "complete internet outage" with traffic dropping from 0030 UTC, and confirmed Virgin Media DNS was also unavailable.

Accordingly, several customers across the country reported being able to get online using a VPN as time went on, although for others the router simply wouldn't connect to the network, and repeatedly flashed "various lights as if it was going round in circles trying to connect to the mothership," as one put it.

Cloudflare noted that VM's authoritative nameservers are also hosted on AS5089, which explains the nameserver/ website outage.

Cisco's implementation of Border Gateway Protocol Stream software, which sifts through raw BGP data via live updates and offers "replays" of historical events, showed several bursts of outages at AS5089 this morning, with the initial outage at 00:50 – which you can see here, here, here, and here.

BGP is the routing protocol for the internet, so what you're looking at if you're clicking any of those last four replay links is one of the smaller networks within the larger internet, which are grouped into Autonomous systems (ASes), a clump of routers run by a single organization. As Cloudflare puts it, all the internal data from the relevant org is then "forwarded as an outbound transmission" to its AS (in this case AS5089, which still has the NTL moniker), which then uses BGP routing to get the transmissions to where they need to be. BGP then picks (or tries to) the most efficient route to get the data from point A to point B, hopping from system to system so it can allow that website to load or that search query response to be delivered.

According to DownDetector and Twitter, reports started flooding in just past 0100 local time, with a tally of 27,000-plus, itself only a proportion of Virgin Media users.

A notspot networker at Wavemobile grumbled on Twitter that they'd had a "sleepless night thanks to @virginmedia and ZERO information or apology." They added: "Things go wrong, the least you can do is tell us via a reliable service page. Would have saved me a trip to the office at 3am."

A Virgin Media spokesperson told The Register the company was "aware of an issue that is affecting broadband services for Virgin Media customers as well as our contact centres. Our teams are currently working to identify and fix the problem as quickly as possible and we apologise to those customers affected."

This will be the last thing British-based techies trying to get a jump on the weekend or organize upgrades over the Easter long weekend will have needed.

One angry Reg reader noted: "Mind, it took seven hours for them to acknowledge there was a problem!"

Worryingly for its network resilience status, contact centers were also out of reach, with another reader noting: "150 phone contact goes to number not available. Virgin website seems to have dropped back to a 'simple' format. Some websites work, others don't."

• UK watchdog still not ruled on Openreach wholesale fiber discounts
• Not satisfied with Virgin Media and O2 merger, Liberty Global takes 5% Vodafone stake
• British govt tech supplier Capita crippled by 'IT issue'
• Outage rates fall, but major ones will cost more. Oh and don't bank on SLAs

Last month Virgin Media/O2 reported its first full year results as a merged business after a deal by their respective parent companies, telco giant Liberty Global and Spain's Telefónica. It reported flat total revenue of £10.38 billion ($12.97 billion) but managed to wring more profits from that, with an adjusted post-transaction EBITDA that increased 6.3 percent year-on-year to reach £3.9 billion ($4.87 billion). It declared a shareholder dividend of £1.6 billion ($2 billion).

Nexfibre – a joint venture between Liberty Global, Telefónica and Infravia – completed in December. VM, which is an anchor tenant and build supplier of the network, says because of the JV, its Virgin Media O2 footprint will be expanded to 80 percent of the UK by 2026, at which point it hopes to present a "scaled wholesale fibre opportunity" that will challenge the nation's current broadband plumber, Openreach.

We have asked VM why there was no resiliency in terms of call centers and its own website, a situation that suggests failures of what should have been built-in network redundancies. ®

Updated at 1119 UTC to add:

A Virgin Media spokesperson got in touch mid-morning to say: "We've restored broadband services for customers but are closely monitoring the situation as our engineers continue to investigate. We apologise for any inconvenience caused."

Updated at 1720 UTC to add:

Several Virgin Media customers, among them Reg colleagues and readers below, have begun to report network wobbles are returning. We still have no technical detail from VM. Perhaps it's time to end this work day with a beverage of your choice.