Standards-obsessed Boss Ignored One, And Suffered All Night For His Sin

On Call The Register loves standards – we have our very own Standards Bureau. Another standard we observe rigorously is that each Friday morning brings a new instalment of On Call, the weekly reader-contributed tale of tech support incidents that ripped up the rule book.

This week, meet a reader we’ll Regomize as "Laslo," who once managed a disaster recovery center – a specialist facility that housed its customers' emergency rigs with appropriately resilient and redundant everything, and even offered empty desks so that when trouble struck, clients' teams could assemble in their dedicated secondary workspace and carry on more or less as usual.

Laslo's facility also housed a few racks that contained servers used in production. And one of those racks included an IBM AS400 server and 42U of ye olde harde disques.

That enormous pile of storage made the rack too heavy for the raised floor. But the owner of the AS400 was one of the center's biggest and most prestigious clients, so Laslo and his crew had a local welder fabricate a steel plate to sit under the rack.

"We spread the load across ten raised floor supports rather than the usual four," Laslo told On-Call. This arrangement meant the rack stood out from all others in the facility – literally and metaphorically, and also because it was alone in having its cabling visible.

At about the time Laslo gave the monster rack the support it needed, the DR center's operators bought into ISO standards, big time. The management team was duly restructured, and the compliance operator given control of operations because it was felt his relentless pursuit of standardized perfection was just what the company needed.

This new manager duly visited the DR center and saw the AS400 rack, and decided it was both anomalous and dangerous. Someone could trip on the cables! And the bigness of the rack wasn't standard!

Laslo's perception of his boss's thoughts was that the rack "had to be rectified before the compliance police shut the whole place down for endangering life as we know it."

So the boss took it upon himself to sort out the situation by snipping the ties off the cables running to the rack to make enough slack that the AS400 and all those disks could be pushed back into line.

• Junior techie had leverage, but didn’t appreciate the gravity of the situation
• While we fire the boss, can you lock him out of the network?
• Tech support done bad sure makes it hard to do tech support good
• People power made payroll support in putrid places prodigiously perilous

That decision was taken without Laslo's knowledge, and at a rather late hour of the day.

When Laslo arrived the next morning, his manager did not appear in the standards-approved daily kickoff meeting which, in line with standard procedure, commenced with a review of overnight incidents.

Logs had recorded quite an interesting one: someone had gone into the datacenter, but nobody had come out!

An investigation of that anomaly found … the standards-obsessed manager, propping up the AS400 rack to prevent it from toppling over and almost sobbing with the effort of having done so for most of the night.

"It turned out he had wheeled the rack off the metal plate and as soon as the first two wheels rolled onto the raised floor, they had gone straight through," Laslo told On Call. The rack had therefore listed a little.

The standards-obsessed boss knew it was a production machine and that if it fell and stopped working, the result would be well and truly non-standard.

Which explained his epic all-nighter.

But as Laslo told us, it was not a necessary effort. As he set things right, he realized the rack would not have toppled beyond the slight angle at which it rested – because the non-standard steel plate would have held it in place.

"I still pass by the building from time to time. It's now an office," Laslo told On Call. "And whenever I do, I wonder if that manager is still there, valiantly and somewhat pointlessly, leaning on a rack."

"I hope he is."

Have standards made a mess of your IT? If so, click here to send On Call an email and we may feature your story on a future Friday. ®