Service Desk Tech Saved Consultancy Capita From VPN Meltdown, Got A Smack For It

On Call Welcome, dear reader, to yet another instalment of On-Call, The Register's weekly column featuring readers' tales of being asked to show up and save the day.

This week, something a little different.

Readers usually ask us not to identify the scene of their adventures – but the On-Call inbox recently received a tale set at controversial consultancy Capita. As is usual the contributor asked not to be named, but was happy for us to mention his employer.

With that bit of background, meet a chap we'll Regomize as "Terry" who accepted a role on the Capita help desk a decade ago.

The gig featured standard help desk fare: account unlocks, password resets and deciphering software errors.

But a couple of years into his tenure, Terry got to work and "all of a sudden our phone switchboard started lighting up like a Christmas tree on steroids!" The Reg is not sure you can give steroids to a Christmas tree, but would not recommend trying it at home.

Anyway, the cause of the steroidal kerfuffle was an upgrade to a VPN that refused connections from certain sales teams and management bods, who weren't shy about expressing their anger.

Terry did some triage, and found those who could not connect were running Windows XP – Capita's Windows 7 users were fine.

"As one can imagine, things escalated quickly to the 'major incident team' while service delivery managers were bravely BSing angry customers to save their necks."

Terry, meanwhile, started a rollback of the VPN as colleagues tried to figure out what went wrong on Windows XP.

Then an order came to down tools: more than half of users were able to connect, so a rollback was ruled out.

"At this point, our service desk was unceremoniously ousted from the troubleshooting effort, which was left in the hands of senior server engineers and the like," Terry told On-Call.

The service desk team therefore updated its interactive voice menus with a message that said, essentially: "If you are calling about VPN issues: Go away, we're not allowed to help."

With incoming calls back to regular levels, Terry started to ponder why only XP was afflicted by the VPN change.

He took the job home and eventually learned that the first time Cisco's VPN successfully connects to a machine, it saves a config file into the Windows C:\Users\<username>\AppData directory. Armed with VMs of Capita's standard XP and Win 7 images, Terry found the file on the Windows 7 image and copied it to the XP image.

He booted the XP image, hooked up to the VPN … and solved the problem.

• PC tech turns doctor to diagnose PC's constant crashes as a case of arthritis
• If you have a fan, and want this company to stay in business, bring it to IT now
• What's up with IT, Doc? Rabbit hole reveals cause of outage
• No, you cannot safely run a network operations center from a corridor

Terry decided his next job was to save the day – and make sure everyone knew how clever he had been.

So he purchased a domain, with a name that suggested it was the savior of Capita's VPN mess, and hosted a site. On said site he posted a self-extracting .rar file that injected the config file into the correct folder on Windows XP.

The next morning, Terry found Capita in a state of nervous tension as attempts to find a fix continued.

"I calmly sidled up to one of the juniors and whispered into his ear: 'I fixed it mate'."

Said junior looked up Terry's web site, downloaded the .rar file, "and nearly fell off his overly priced swivel chair, when his XP VPN connected within seconds!"

"Hours later, my fix was officially approved by corporate and we service desk bods got to work doling out the 50KB patch file to the XP users."

For his trouble Terry was officially thanked by management and given a couple of hundred pounds worth of gift vouchers. Peers gave him the nickname "Capita Savior" – a nod to the website he created.

But a couple of weeks later, Terry was "dragged into a disciplinary meeting" and given two black marks.

One was issued because his fix had circumvented company security policy. The other was because the domain he bought had "supposedly brought the company name into disrepute." Terry was told to take it down ASAP.

Which might be why, lo these many years later, he wrote to On-Call!

Have you been smacked down for saving the day? If so, click here to send On-Call an email and we may share your story here on a future Friday. ®