Datacenter Fire Suppression System Wasn't Tested For Years, Then BOOM

On Call The "P6 rule" states that Proper Planning and Preparation Prevents Poor Performance. So as you prepare for the imminent weekend, The Register invites readers to run an eyeball over another edition of On Call, our weekly reader-contributed tale of tech support chores that involve another P – as in "going pear-shaped."

This week, meet a reader we'll Regomize as "Jack" who once worked in the computer room of a school board's HQ.

Next to the computer room was a walled-off shared office that had a window allowing a view of the computer room.

The two spaces shared a raised floor, so the very large tank crammed full of high-pressure halon gas in place to suppress fires would flood both rooms in case of emergency.

Both rooms were equipped with sensors to detect smoke, which would cause a bell to sound warning of imminent gas release.

But in case of a false alarm, or minor incident, the facility also included a Big Yellow Button that would abort the gas release. And in case the sensors didn't do their thing, the computer room also housed a Big Red Button that overrode everything and released the gas to put out any fire.

Those buttons and their respective roles became important when one of Jack's colleagues decided to do a spot of soldering in the office. A little smoke duly wafted past a sensor, and the warning bell began to sound.

Clearly the computer room was in no danger whatsoever. So one of Jack's colleagues pressed the Big Yellow Button to stop the release of Halon!

BOOOOOOM!

"The halon system immediately dumped its tank," Jack told On Call. "It was so fast that it sounded like a shotgun blast. Dust and paper flew everywhere in the air and one staff member sitting at their desk, not far from the halon nozzle in that office, dove under the desk, thinking that someone had fired a gun."

• Shocks from a hairy jumper crashed a PC, but the boss wouldn't believe it
• Thanks for fixing the computer lab. Now tell us why we shouldn’t expel you?
• Techie called out to customer ASAP, then: Do nothing
• Automation is great. Until it breaks and nobody gets paid

As explained above, the Big Yellow Button was there to stop gas being released. It had only one job – why had it failed so spectacularly?

Jack's investigation revealed that the Big Yellow Button and the Big Red Button had been wired in reverse many, many years before the soldering iron set off the alarm.

"Nobody had ever tested all the buttons with the system's dump mechanism disabled in order to ensure that everything was wired correctly," Jack explained.

"Seventeen years later, when the facility installed a new fire suppression system, the company that installed it worked with the IT staff to test each and every button to make sure that the system worked correctly and that all dump and countdown reset buttons were wired correctly," Jack wrote.

The new system sounds foolproof. As Jack tells it, at the first whiff of trouble a 30-second countdown commences – during which three high-pitched, warbling horns with strobe lights blare.

"The abort buttons reset the counter to zero with each momentary press, allowing a single staff member to run around the room, checking for smoke or fire and to allow that staff member to go to the fire suppression system's alarm panel and disarm the system if no issue is found."

What could possibly go wrong? On Call is sure readers will have stories of the frailties such a system possesses.

Click here to send your story to On Call and we may feature it here on a future Friday. ®