It's hard to get a good job in IT these days, but it's all too easy to lose
one.
There are lots of reasons for instant termination. Failure to fulfill your
obligation to protect your employer's digital assets or abusing your vast powers
for your own nefarious ends are two sure ways to end up on the unemployment
line. You could be fired for opening your mouth at the wrong time or not opening
your mouth at the right one. Spying on the boss, lying to your superiors, or
being directly responsible for the loss of millions of dollars in downtime
through your own negligence are all excellent ways to end up on the chopping
block.
Everyone messes up at some point. But some screwups are almost always fatal --
to jobs, if not entire careers.
Here are seven true tales of IT pros who screwed up big and got fired quick --
even if some were terminated for the right reasons. The names have been withheld
to protect the guilty. Don't let their fatal mistakes become yours.
Fatal IT mistake No. 1: Slacking on backup
It was 10:30 on a Thursday night when Eric Schlissel's phone rang. On the line
was the chief operating officer of a midsize clothing manufacturer with whom
Schlissel had never spoken before. The COO, who found his company's phone number
via Google, was frantic. His plant's ERP system had been wiped out by a virus,
and they had a major deadline in the morning.
Schlissel, CEO of managed service provider GeekTek IT Services, hopped in his
car and headed down to the L.A. garment district to handle the situation
personally.
"Within three minutes of logging in, I realized there was nothing on the
server," says Schlissel. "All the data files were gone, the database was gone,
and the ERP software was nowhere to be found. I told him this was no virus.
Someone had purged the system."
It turned out a disgruntled IT contractor had enacted revenge by wiping the
garment maker's servers. But worse news was yet to come. The backups, which were
supposed to run every night, hadn't been working for a very long time. The most
recent data Schlissel could find was a year old, making it virtually worthless.
The company only survived because someone in accounting, who did not trust
technology, had kept paper copies of everything. It took Schlissel and his team
six months to restore all the data by hand.
"It was a $10 or $12 million company, and they probably lost $2 million as a
result of this," he says. "It was the most catastrophic IT disaster I've ever
seen."
The factory's general-purpose IT guy, who was responsible for ensuring backups
were made, had simply forgotten about them. He was on the unemployment lines the
next day.
Failure to maintain backups is an all too common screwup, and the mistake is
often fatal to one's job security, Schlissel says.
"The first thing we do when we visit a new client is to check the backups,"
Schlissel says. "This is a classic IT horror story, one we often tell clients.
We're not trying to scare them, we just want to make sure their assets are
protected."
Moral of the story: A backup in hand is worth two bushels of paper.
Fatal IT mistake No. 2: Snooping on the boss
A few months ago, Oli Thordarson got a call from the CFO of a midsize health
care provider in Southern California. As CEO of Alvaka, an advanced network
management services company, Thordarson and his staff are often asked to act as
a virtual CIO for small businesses and to perform forensic investigations.
The CFO told Thordarson he thought someone was secretly reading his email, and
he had a pretty good idea who it was: the director of IT.
The CFO said that, over the past two years, this guy had made comments about
things he had no business knowing, says Thordarson. "The running joke was that
the director of IT knew more about what was going on inside the company than
anybody else," he says.
Thordarson had one of his techs modify a real-time network probe so that it
would send a silent alert if anyone was reading emails they shouldn't be
accessing. Within a few days, Alvaka discovered that the director of IT was
indeed reading the CFO's email -- as well as messages from the CEO, the
chairman, and other top brass. The next day he was reading the want ads at
Monster.com.
This problem is more common than you might think, Thordarson adds. In roughly
two-thirds of the companies Alvaka advises, techs have the ability to read any
employee's email, including that of top executives.
"Did they do it to enable support and then forget to undo it, or did they do it
because they wanted to snoop?" asks Thordarson. "We don't really know."
Moral of the story: A fool and his job are soon parted.
Fatal IT mistake No. 3: Covering up the crime
It was a mistake that could have happened to anyone. The IT staff at a major
financial institution needed to replace a disk tray for an older storage array.
A staffer called the vendor and had one shipped out. But the junior sales guy at
the vendor made a mistake and shipped the wrong tray -- one for a newer array
that was incompatible with the old one.
The array then failed catastrophically, taking the entire bank's system offline
for nearly a week and costing millions of dollars in lost transactions. That's
when they called in Anthony R. Howard to troubleshoot.
There were three big screwups, says Howard, a best-selling author ("The
Invisible Enemy: Black Fox") and independent technology consultant for Fortune
50 companies and the U.S. military. One, of course, was that the vendor shipped
the wrong unit. The second was that the bank's IT staff tried to install the
array itself without waiting for the vendor to send out a qualified technician
to do it for them.
The third problem was the big one, though. Almost everyone involved in this
screwup lied about it, says Howard. Only one staffer had the courage to admit
what really happened.
"When the IT staff saw their jobs were in danger, they began to try to protect
themselves and blaming the tech support staff of the vendor," says Howard.
"After the bank's internal team was done with its investigation and found out
that only one person told the truth, he was the only one who managed to keep his
job."
Moral of the story: If the crime doesn't get you, the cover-up will.
Fatal IT mistake No. 4: The porn identity
Late one evening a couple years back, a network admin for a Fortune 100 firm was
looking for an empty backup tape. He grabbed one from the desk drawer of a
senior system administrator and popped it into the drive, but was surprised to
find it was already full of data. What, he wondered, could be on it? So he
looked at it.
You can guess what he found.
"It was filled with porn," says Dave Amsler, president and CIO of Foreground
Security, which was called in to handle the incident. "And so were dozens of
other 'blank' tapes in the admin's desk. There was nothing illegal on any of the
tapes, thank goodness. Still, he was terminated on the spot."
Yet that's hardly the worst Amsler has seen in his 14 years with Foreground,
which provides managed security services for major U.S. corporations and
government agencies. Amsler says he's been called in to deal with porn problems
for at least 10 clients. Twice he found IT employees running adult sites on
company servers. In those cases, the personnel suddenly found themselves with
lots more spare time to pursue their hobbies.
Porn filters are useless against this kind of behavior because the IT guys know
how to turn them off. Even when organizations have strict policies and filters
in place, high-level admins often exempt themselves from these restrictions,
says Amsler.
"Sometimes rightfully so," he adds. "Often high-level admins need to get to
sites that would normally be blocked in order to do their jobs. But that doesn't
mean they shouldn't at least be monitored. Even good people end up doing things
they normally wouldn't when they think no one's watching. If the admin knows
he's being watched, that would eliminate a significant portion of this
behavior."
Moral of the story: Some things are better done at home.
Fatal IT mistake No. 5: Keeping the wrong secrets
Until recently, Dana B. was a network engineer at a major U.S. Internet
provider. One day, a former colleague was told to change the IP addresses on
some production routers. Because these changes could impact Internet
subscribers, taking them briefly offline, the ISP typically made such changes
overnight.
But this engineer didn't like to stay late, so he changed the addresses at the
end of the day before he went home, then turned off his phone so that nobody
would disturb him during his off-hours.
That was his first mistake. His bigger mistake was that he consistently refused
to document anything he'd done, says Dana. That meant he had no idea which IP
addresses he had already used in the past -- and neither did anyone else.
After he left, the interfaces failed to come up because their IP addresses had
already been used, leaving nearly 5,000 subscribers without Internet access.
When other engineers tried to call him to figure out what went wrong, they
couldn't reach him.
"It took a team of five network engineers several hours to find the issue and
correct the problems," says Dana. "The next day he came in and was promptly
walked out."
Moral of the story: Some secrets are better left unkept.
Fatal IT mistake No. 6: Unmitigated disaster
They thought they were ready for anything. An organization in a heavily
regulated industry had spent millions building out a comprehensive
disaster-recovery plan, including a dedicated fail-over data center humming with
hundreds of virtual hosts and a Gigabit Ethernet connection.
But when an unplanned network outage cut the connection to its primary data
center, the money the organization spent on its DR solution was for naught.
"The CTO did not have the confidence to activate the disaster-recovery plan,
because they had never tested it," says Michael de la Torre, vice president of
recovery services product management for SunGard Availability Services, which
was called in by the organization later to shore up its DR strategy. "Instead,
he stood by for more than a day hoping the circuit would be repaired. Everyone
was offline that entire time. Employees had no access to email or data files,
and the organization took a pretty big hit to its reputation."
Shortly thereafter, the CTO's career also suffered an unplanned outage.
More than half of all organizations with disaster-recovery plans in place fail
to adequately test them, notes de la Torre. Even those that do test uncover an
average of five critical errors in the people, process, and tools needed to make
DR work.
Disaster recovery is neither glamorous nor easy, but it's vital to the survival
of your company, he adds.
"Protecting the business may never get you promoted. But failing to do so will
almost always get you fired."
Moral of the story: Test your umbrella before the **** hits the fan.
Fatal IT mistake No. 7: Speaking truth to power
Ten years ago, "Bob" was working for a payday loans franchise with more than
1,000 locations nationwide. (Bob asked that his real name be withheld from this
story.) He had been hired to rearchitect the chain's ASP-based system, which was
running ancient code on servers in every store. But first he had to prove
himself by converting the stores' dozens of Web-based legal forms into a
database.
One Friday afternoon, six months into the job and two weeks before his initial
trial was over, the vice president of IT came into the weekly staff meeting to
present his five-year vision for the company. The veep's two-hour speech could
be summed up in four bullet points, says Bob:
Stay the course
Fix bugs
Don't rock the boat
No new technology
"I was floored," says Bob. "I thought, 'What about all the stuff they hired me
for?' They were spending millions of dollars a year maintaining creaky sites
written by 50 different people."
Later that afternoon, Bob went into the VP's office and closed the door.
"He asked me, 'So what do you think of my vision?'" Bob says. "I said, 'Frankly,
sir, you don't have one. What you just described was a maintenance plan.'"
The VP thanked him for his candor and complimented him on his courage. The
following Monday when Bob returned to his office his key no longer worked. He
was gone.
"I drove home whistling," he says. "I've never been so happy to be unemployed. I
decided I would never have my career depend on an empty suit ever again. The
next day I started my own business, which has kept me busy ever since."
Moral of the story: Sometimes getting yourself fired is the right thing to do.
