OVH Blames Hour-long Global Outage On Human Error During 'routine' Network Reconfiguration

Hosting provider OVH reckons that its servers are "gradually returning" following a worldwide outage that lasted for around an hour.

The French co-location biz's services went AWOL from 7:20am UTC. The company had earlier talked of planned "maintenance on our routers" in its Vint Hill, Washington DC, data centre to "improve routing".

Network and racks:: VIN/DC: We will do a maintenance on our routers on VIN DC to improve our routing.

Maintenance is planned for 13/Oct/21 9:00 AM to 10:30 AM ( UTC+2). No impact expected, device will be isolated before the change. https://t.co/n9NjpGICrU #ovh

— OVH Status Feed (@ovh_status) October 13, 2021

CEO Octave Klaba then confirmed: "Following a human error during the reconfiguration of the network on our DC to VH (US-EST), we have a problem on the whole backbone. We are going to isolate the DC VH then fix the conf."

Worldwide outage at OVH

He added "In recent days, the intensity of DDoS attacks has increased significantly. We have decided to increase our DDoS processing capacity by adding new infrastructures in our DC VH (US-EST). A bad configuration of the router caused the failure of the network."

"The router with the wrong D2 configuration has been cut," Klaba concluded.

According to its outage report, the fix involved "the isolation of network equipment in the US."

The routine task is now marked "change cancelled".

The culprit? 'No impact expected'

OVH, which is among the larger non-hyperscale hosting and cloud services providers, with 400,000 servers across 30 data centres worldwide, is engaged in an IPO on Euronext Paris though the target price was reduced from €450m to €350m.

• OVH drops IPO target against figure mooted a month ago
• Like a phoenix rising from the smouldering ruins of its data centre, OVH sets sights on IPO
• OVH outlines three-point 'hyper resilience' plan after Strasbourg fire
• OVH services still not fully restored as boss rates ongoing recovery efforts a 'real nightmare'

The company is well placed to benefit from the concerns about European data sovereignty and the dominance of US-based global corporations in this space. It is the fourth biggest cloud provider in Europe.

Coming (or going) just before the IPO, this outage is therefore an embarrassment. "A few months ago you had a downtime of over 8 hours because of the poor measurements you take on the safety of the buildings... As soon as this downtime is over I am moving out of OVH," said one disgruntled customer.

The company hit the headlines earlier this year following a fire at its Strasbourg premises on 10 March that wiped out two data centres. At least today's incident was nowhere near the scale of that disaster. ®