'Once In A Lifetime' IT Outage At City Council Hit Datacenter, But No Files Lost

Nottingham City Council continues to deal with the fallout from its freak IT outage from last week as it confirms in-house IT specialists managed to prevent any data from going missing.

The council told The Register that despite the outage affecting much of the council's digital infrastructure, some of which remains down today, there is no risk to residents' data.

"Our IT teams have successfully worked around the clock, so we haven't lost any data," a spokesperson said. "This is why we've had to take time to power up systems – to ensure no loss of data as we bring back online [sic]."

The news comes as the council restores a number of its systems following a power outage caused by an electrical fault at Loxley House, its city center headquarters. 

A failure was detected within the electrical safety circuit of the building's high-voltage switchgear, a statement read.

"The power outage subsequently impacted the council's central datacenter, which houses computing and networking equipment for the whole council, " the statement went on to say.

"The serious nature of the unprecedented outage meant that all systems went offline, leading to disruptions to our phone lines and systems while we work to test systems before bringing them safely back online."

As of Monday, the council said it was able to restore its phones and other systems but encouraged all residents with emergencies to report them via email in the first instance since they were expecting a high volume of calls.

The outage started on March 13 and continues to affect most online services, according to the council's website. 

A banner no doubt penned by the tech team still advises residents that services may seem functional, and forms will load, but they may not work correctly. However, a council spokesperson contradicted this in a statement given to The Reg.

"The majority of our services are running and have been since last week, it's just that some have been working differently as a result of loss of payment systems and contact centers," they said.

Residents can expect normal service to resume later this week, according to the council's website, which also stated it would take some time to carry out the necessary repairs and tests before signing off on a "fully stable" power supply.

Council chief Sajeeda Rose praised staff for working through the disruptions and for adapting at "lightning speed" to the council's business continuity plans.

"Unexpected disruptions like these test our ability to adapt, and I want to take a moment to acknowledge the incredible resilience, patience, and professionalism colleagues have all shown in response to this unexpected challenge," she said.

• Hyperoptic customers left in dark as power outage takes down systems
• Mystery border control outage causes misery at Malaysia/Singapore frontier
• Internet Archive blames 'environmental factors' for overnight outages
• Techie saved the day and was then criticized for the fix

"While we are making progress, we appreciate that this situation has been disruptive. We appreciate the public's continued patience and cooperation. We will keep you updated as much as we can."

BBC Nottingham reporter Hugh Casswell shared an image of the council's executive board meeting on Tuesday from Loxley House with the effects of the power outage on display.

Members assembled in the barely lit room, sat around desks with no light and the wall-mounted TV plugged in but not operational.

"We're told a risk assessment has been carried out and it was deemed safe to hold the meeting as the room is on the ground floor and lit by natural light," said Casswell. ®