If You Have A Fan, And Want This Company To Stay In Business, Bring It To IT Now

On Call Weekends are the time for fun in the sun – but before you get to that, The Register offers another instalment of our weekly On-Call column, in which readers share stories of being put under the hot spotlight of being asked to fix flare-ups, fast.

This week, meet "Trent" who told us about his time as head of network operations for a payment processing company in Dublin.

Trent's tale took place a day he described to On-Call as "one of those rare occasions in Dublin when the sun shone brightly and the thermometer climbed well north of 20 degrees."

Which naturally meant everyone in the office was complaining about the heat.

Except Trent, who luxuriated in the air-conditioned comfort of the Network Operations Centre (NOC).

Said NOC wasn't just nicely air-conditioned – it was optimally air conditioned. Because on the very day on which this story happened, an air conditioning engineer had come out to talk about selling extra kit to cover more of the building.

As a demonstration of goodwill he topped up the existing aircon refrigerant tanks.

The small equipment room Trent tended, which he described as "small and crammed full of racks of file servers, payment processing servers, network switches and an ancient Netrix switch running the point of sale" was therefore filled with lovely frigid air as the giant aircon inside went about its work.

Until it didn't.

"A little after lunchtime … the usual quiet calm in the NOC was suddenly replaced by the strident wail of alarms," Trent recalled.

A glance at the NOC's screens revealed everything was down, so Trent dashed to the equipment room, opened the door and was hit by a hot wave of air and "the unmistakeable stink of refrigerant from the air conditioning unit."

"It appeared that the air conditioning engineer had not closed one of the valves fully when topping up the refrigerant and it had all leaked out of the system."

• What's up with IT, Doc? Rabbit hole reveals cause of outage
• No, you cannot safely run a network operations center from a corridor
• User was told three times 'Do Not Reboot This PC' – then unplugged it anyway
• This can’t be a real bomb threat: You've called a modem, not a phone

Trent's first call was to the aircon engineer, suggesting a rapid U-turn and faster journey back to fix the mess he left behind.

Which left Trent with the job of keeping a badly overheating datacenter up and running – an essential chore given that payment companies are all about 24x7 resilience.

"While we waited for the engineer's return, my colleague confiscated every fan we could find in the building and used them to blow air over the racks of overheated equipment," Trent told On-Call.

Which made Trent extremely unpopular, given the weather on that fine Dublin day.

Ten or fifteen minutes later, the fans had cooled the kit down enough for much of it to work.

The aircon engineer showed up about an hour later and normal service was restored.

"Afterwards, the bosses heaped praise on us for our quick thinking with the fans and getting the network back up and working so quickly. But everybody else was less than impressed by our actions and people bitched about it for a considerable time afterwards," Trent told On-Call.

Have you had to co-opt supplies from around the office to keep IT alive? Click here to send On-Call an email that tells your tale and you might find it on this page on some future Friday. ®