Techie Wiped A Server, Nobody Noticed, So A Customer Kept Paying For Six Months

Who, me? Why hello, Monday! You beastly harbinger of another week of work and associated woes, which The Register each week welcomes with an instalment of “Who, Me?”, our reader-contributed tales of techies who make mistakes and mostly mollify their masters.

This week, meet a reader who asked to be Regomized as “Sam” and once worked at a company that, thanks to a series of acquisitions, had a diverse portfolio of infrastructure and apps it was migrating into a consolidated datacenter.

One of the apps handled what Sam described as “billing for inspections.”

“Plans were drawn to migrate the system into existing billing platforms,” Sam recounted. But as often happens when companies consolidate, staff with important knowledge were let go without care.

That meant the billing app’s sole developer was surplus to requirements but had already been hired as a contractor to assist with " some emergency temporary repairs" while the larger development team worked on the migration.

Two years later the app was still running on its original server, and Sam got word it was time for that box to move to another role.

“After a few emails confirming with the rest of IT that this server was no longer in production - because we surely did that migration - I removed it, wiped it, un-racked it, physically moved buildings, and repurposed it for an internal system,” Sam told Who, Me?

And then he thought nothing of it for six months, until he received a message about not being able to connect to the long-since-repurposed server.

“A scramble ensued,” Sam said, as he and colleagues checked the app had indeed been migrated.

• Don't worry, that system's not actually active – oh, wait …
• Backup tech felt the need – the need for speed. And pastries and Tomb Raider
• A tip for content filter evaluators: erase the list of sites you tested, don't share them on 100 PCs
• Learn the art of malicious compliance: doing exactly what you were asked, even when it's wrong

It had not, and nobody knew why.

Nasty questions followed.

Why wasn’t it migrated?

Do we have backups?

What do you mean when you say “There are no backups”?

Why did it take six months for someone to report this app was down?

The best question of all was: “How long have we been billing clients for this?”

The sole developer who understood the app was again asked to help, and according to Sam “told the company to buzz off and that he would not help us for any amount of money.”

At which point the migration project shot to the top of the development team’s to-do list and was given all the resources needed to get the job done fast.

“It was integrated into the existing system in just a few months, agile timelines be damned,” Sam said.

Sam told us he was not blamed for this mess. “I had done all my checks first,” he said.

But he’s still a little down on himself for not quite doing all he could to ensure the server was not indispensable.

“I guess it goes to show, no matter how many times you check, always rely on that powered-down ‘what does it break’ analysis before pulling something out, no matter how many people say you're in the clear,” he told Who, Me?

If you have a similar tale of a resource being removed without anyone noticing, click here to send Who, Me? And email and it might be your story kicking of another week of Reg fun. ®