Bank's Datacenter Died After Travelling Back In Time To 1970

On Call The steady process of time means that The Register has once again arrived at Friday and the timeslot we reserve for On Call – our weekly reader-contributed tale of tech support trials and tribulations.

This week, meet a reader we will Regomize as "Colin" who shared a story of his time working as a support engineer for a major UK bank's foreign exchange division.

This was the sort of facility where smartly dressed and highly strung traders made money by manipulating numbers on the other side of the world and worked hours to match.

The support team therefore started very early. In this story Colin rolled into work at 06:00 and found "nobody in the building could work."

Thankfully, only a few traders were there to grumble about it, and the fact they'd missed out on some moneymaking opportunities in Asia.

But Colin knew in an hour or two he would have hundreds of traders, dealers, and other staff all demanding that systems come online – ASAP.

"I was immediately summoned to explain what had happened and had to tell them that I wasn't going to be able to fix it, unless they stopped questioning me, and let me get on with the job," Colin recalled.

• Bank boss hated IT, loved the beach, was clueless about ports and politeness
• Lawyer guilty of arrogance after ignoring tech support
• CompSci academic thought tech support was useless – until he needed it
• Suits ignored IT's warnings, so the tech team went for the neck

Thankfully other staff soon arrived, including help desk staff who became a barrier between traders – who were calling for Colin's head – and other techies who helped reboot servers and conduct diagnostics.

The problem was eventually found on a windowsill that held a pair of dusty radio time clocks – devices that sync with radio signals to ensure they tell the time accurately.

One of those clocks provided the time for the bank's Novell servers. The other did likewise for Windows machines.

Or it would have done, had its quartet of AA batteries not expired overnight. The hundreds of servers that relied on the clock therefore made their best guess about the time, reverting to January 1, 1970 – a date regarded as the beginning of time by many computers.

Four new cells were swiftly produced and installed, and before you could say "Short the Yen, buy the Dollar," the bank was back in business.

Next, the incident assessment meeting.

At which it was discovered that the Help Desk was supposed to replace the batteries once a year but had forgotten to do so.

Oops.

"Solidarity between colleagues meant that we didn't rat on the Help Desk staff and made up some other cause – a loose network cable or some such," Colin told On Call.

Have you covered for a colleague that caused a calamity? Click here to send On Call an email with your story and we may feature it in coming weeks.

We're also keen to receive stories of festive foul-ups, as On Call's author will work across the holidays and fancies slipping in a few seasonal specials. ®