Techie Climbed A Mountain Only Be Told Not To Touch The Kit On Top

On Call Every Friday, The Register presses the OFF button the week with a fresh instalment of On Call, our column that recounts readers' experiences of taking on tricky tech support jobs in exotic places.

This week, meet a reader we'll Regomize as "Edmund" who once worked for a very large internet service provider and was responsible for the operation and maintenance of several routers that were essential for the provision of broadband services in towns near his home.

These were the sort of routers that could easily support hundreds or even thousands of customers. Edmund's job was complicated by the fact his routers were mostly located in cold and mountainous locations.

So when the call to fix one came, it was of course winter, as the sun was going down, on a Friday, and the router was two hours away by car.

"The network operations center (NOC) requested I head to a site and power cycle one of the two redundant routers that carried all the traffic for the town," Edmund told On Call. "I was about two hours away, but it wasn't customer impacting so it wasn't a big deal other than my Friday night activity was canceled."

So off Edmund went. When he arrived, he joined a conference call in which he was asked to connect his laptop to the console port and share his screen before he cycled the router off and on again.

Edmund did as he was asked. As soon as he connected he saw a terminal filled with gibberish, and was swiftly told that NOC staff knew what was going on and that the router must be replaced.

Our hero reminded his NOC masters that he hadn't yet power cycled the router and was told that didn't matter.

His new instruction: wait for the vendor to deliver a replacement – which should take four hours to arrive.

After buying himself dinner, then checking the weather forecast and learning of imminent snowfall, Edmund rated his chances of receiving the replacement as very low. Around midnight the NOC agreed and sent him home because the router was not going to make it up the hill that night.

• Standards-obsessed boss ignored one, and suffered all night for his sin
• Junior techie had leverage, but didn’t appreciate the gravity of the situation
• While we fire the boss, can you lock him out of the network?
• Tech support done bad sure makes it hard to do tech support good

Come Saturday, the news was better. A courier would arrive at 10:00 AM, but as the router weighed over 100 pounds (40kg) Edmund would need a colleague to help move it into place atop a six-foot rack.

Edmund duly found a colleague, hit the road, schlepped the router into the facility, and dialed in to a conference call with his NOC colleagues.

Edmund didn't fancy a full router replacement, so reminded his remote collaborators that nobody had power cycled the machine. "Oh, you didn't do that?" was the reply

Edmund reminded them "You guys specifically said not to!"

"You can guess what happens next," Edmund told On Call. "I pulled the power, and it came back up like nothing happened. Twelve hours of overtime and a wasted trip because of an 'expert'."

This story had a strange sequel: three months later, Edmund was asked to locate the defective kit, because he had sent the unused replacement back and the serial numbers didn't match.

Have you ever been prevented from power cycling? Or sent somewhere cold and nasty for a futile job? We're sure some of you have and we want you to click here to send On Call an email that shares your story.

Many of you very kindly say you enjoy this column. Step up to support it with an email! We never breach a confidence or write in a way that will land you in trouble. ®