Alibaba Cloud Struggles After Blaze At Digital Realty Singapore Datacenter

At least a portion of Chinese web giant Alibaba's online services have been disrupted by a fire at a Singapore datacenter following what's said to be an Li-ion battery explosion.

Singapore media reports the blaze started Tuesday morning, local time, in the battery room of one of the datacenter's buildings.

A spokesperson for Digital Realty, which operates the facility, told us it “became aware of a fire alarm being triggered at our SIN11 datacenter at 0745 on September 10, 2024. All on-site personnel were safely evacuated by 0815 and no injuries are reported.”

The abnormality was caused by the explosion of lithium batteries

Singapore’s Civil Defence Force responded to the fire, and businesses affected by the blaze at the computing facility were told to activate their disaster continuity plans.

An Alibaba Cloud status report disclosed it detected anomalies in Availability Zone C of its Singapore region at 1020 SGT Tuesday, “causing some cloud services to function abnormally.”

“The abnormality was caused by the explosion of lithium batteries in the Singapore data center, which led to fire and elevated temperature,” reads a later update.

The Chinese cloud colossus claimed its disaster recovery and fail-over routines worked as intended, meaning high-availability cloud products met promised service levels, though said some users would still need to manually migrate workloads away from the flames.

At least some other services and offerings were taken offline as Alibaba waited for temperatures in the datacenter to fall.

• Singapore's banks to ditch texted one-time passwords
• Digital Realty CTO weighs in on AI's insatiable thirst for power
• OVH opens less flammable datacenter at site of 2021 fire
• Cigarette break burned out a huge chunk of Africa's internet

As of 2004 SGT Tuesday (1204 UTC, 0504 PT Tuesday) the cloudy concern reported “the fire alarm has not been completely eliminated,” staff could not enter the blaze-hit building, and “some network equipment in the datacenter has experienced abnormalities in high-temperature environments, affecting the network connectivity of some cloud products.”

Customers were warned “the possibility of a complete network outage in Availability Zone C of Singapore will increase,” and advised: “If your business is deployed in Availability Zone C of Singapore, we will assist you in migrating your business as soon as possible.”

At 0146 SGT Wednesday (1746 UTC, 1046 PT Tuesday), things took a turn for the worse.

“The server room has started to experience water accumulation and leaks, posing a risk of electrical short circuits,” said Alibaba, necessitating an emergency power shutdown for one of the buildings in Singapore Availability Zone C.

Network services for other buildings were gradually restored.

Digital Realty confirmed to The Register that as of 0145 SGT on Wednesday it had powered down some electrical systems.

The Register has seen complaints by netizens of outages or service degradation at Digital Ocean, IaaS service Coolify, and Cloudflare, seemingly related to blaze.

Battery fires are a known threat: French cloud OVH lost a datacenter after a fiery UPS mess in 2021.

This is a developing story: The Register will update it as more information comes to hand, and track investigations into the root cause of the incident. ®