Health System Network Turned Out To Be A House Of Cards – Cisco Cards, That Is

who, me? Welcome once again, gentle reader, for that cushion into the working week that we like to call Who, Me? in which readers like your good selves entertain us with tales of times technology did not go quite right.

This week our hero is someone we shall Regomize as "Tim" who moved continents some years ago, bringing his networking skills from Europe to the frozen wastelands of Canada. Or at least to a healthcare authority residing in said wastelands.

The authority serviced an area that Tim told us was slightly larger than France. The primary points on the network were two datacenters and a server room in a hospital, which was the hub for the rest of the network.

When Tim arrived, bursting with cosmopolitan knowledge and experience, he found that while most of the networking equipment was in place (Cisco Catalyst 6500s throughout) a bunch of dual supervisors had been purchased but not installed.

This was odd, he thought, as supervisor cards improved the giant switches' resilience. Tim attributed their absence to a lack of knowledge or experience on the part of his predecessors. Or perhaps simply an oversight by staff, who didn't realize the benefits they could glean from the dual supervisor cards.

Whatever the reasons the switch had been left unsupervised, Tim promptly went about setting IOS options to download code from each of the current supervisors in preparation for popping the new cards in place.

At each turn, all went smoothly. Tim even troubled to demonstrate to the staff at each datacenter how easy this all was, and told them not to be worried about installing cards – nothing can go wrong, naïve provincial types.

• If we plug this in without telling anyone, nobody will know we caused the outage
• Self-taught-techie slept on the datacenter floor, survived communism, ended a marriage
• 'Crash test dummy' smashed VIP demo by offering a helping hand
• Developer's default setting created turbulence in the flight simulator

Tim finally added the supervisors to the last very last chassis. This was the one at the hospital – which, as we've mentioned, served as the hub for the whole network.

Tim plugged the card in and headed off to Canada's favorite purveyor of sweet baked goods for a well-earned snack.

And while he was eating his donut, his phone started vibrating furiously. Not happily at all, but indeed furiously.

And at the other end of those calls were the 9000+ employees of the health network, who found themselves suddenly unable to do anything but call tech support. Furiously.

It transpired that the last Cisco 6500 chassis had a fault on its main board that meant, when the dual supervisor card was installed, it froze up harder than spit in a blizzard. And because that was the hub for the whole network, it meant that no-one – doctors, nurses, admin staff or even Tim's own team – had any network access at all.

Tim removed the dual supervisor card from the faulty chassis, allowing everyone to get back to work. Then, as he tells it, "after depositing my underwear in a suitable biohazard disposal bin I went ahead and started on a project to redesign the network and replace every piece of network equipment."

If you've ever thought nothing could possibly go wrong until it did, tell us about your experience by clicking here to send Who, Me? an email. Our mailbag is a little threadbare so please send your story so we can share it with your fellow readers on some future Monday.