I Made This Network So Resilient Nothing Could Possibly Go Wro...

Who, Me? Greetings and salutations, dear reader, welcome to yet another fun-filled Monday at The Register. As you well know, each Monday (which is today) The Register (which you're reading) brings you an instalment of Who, Me? – our reader-contributed tales of tech gone wrong. This is that very column. To tell you again would surely be redundant.

As it happens, redundancy is at the core of today's horrifying tale – which our hero, whom we shall Regomize as "Clint", referred to as his "brown trouser day." We very much hope he had a spare pair with him – redundancy, you know.

Back in his halcyon youth, Clint worked as a network admin for a credit card company. The biz worked out of two offices, right across the road from each other. They were joined by a dedicated fiber, connecting a pair of Cisco 6509 routers – very modern for the time.

Of course that meant that there was only one "path" between the two buildings. Given both offices housed huge call centers and hundreds of employees, this was seen as far too risky. A second fiber link was installed, and a second pair of 6509 routers. So if one of the paths between the two buildings failed, the other could easily handle the traffic.

Building this second path had been Clint's special project, and he was rightly proud.

One day, after it had all been set up and tested, he found himself bored – "having finished reading The Register," Clint wrote, because he knows we are suckers for flattery – and was nosing around the ports via Telnet.

He noticed that one of the ports was returning an error condition. After checking that no-one seemed to be having any issues with the network, he assumed this meant that the redundancy was working as intended. All he had to do was toggle that port, and all would be well. No-one else would even notice!

So he typed the appropriate command, hit enter, and …

• Linux admin asked savvy scientist for IT help and the boffin blew it
• Compression? What's that? And why is the network congested and the PCs frozen?
• After we fix that, how about we also accidentally break something important?
• Personalized pop-up was funny for about a second, until it felt like stalking

His session immediately ended. And he could not reconnect.

People around him started wondering what in the world was going on with the network. The VoIP-based call center fell deathly quiet.

Clint instantly realized what he had done. He was logged into the wrong router – the link to the other building was obviously down, so he'd been shuffled over to the router in the building he was in, without realizing it. He could see the error on one port, but in attempting to enable it he had in fact flipped the wrong switch and disabled the path that had been working perfectly well. Now there was no way to get back in without physically connecting.

He grabbed his laptop and a serial cable, then sprinted across the road like a man with a mission – for indeed he was a man with a mission.

He reached the router, connected to it, typed the appropriate magics, and everything came to life. For good measure he cleared the logs – because no-one needed to know, right?

Just then the head of the IT team came in looking for someone to blame. He found Clint moments after he made the save. Naturally our hero won praise for having acted so quickly to fix this mysterious fault, for which no cause could be found. Clint suggested additional monitoring on the ports – in case of future strange dropouts.

As Clint told us: "Fail fast, learn fast couldn't come fast enough!"

Have you ever had a "brown trouser day" like Clint? How did you manage to talk your way out of it – or even end up looking like the hero? Tell us all about it in an email to Who, Me? and we may share your exploits on a future Monday. ®