Real costs of data loss and imperfect backups

Impact of the costs of data loss

The costs of data loss impact businesses, the people that are the backbone of these businesses. Families can be shattered as a result of the loss of data as shown in the following story.

Imperfect backup and costs of data loss

Imperfect backups can bring down a business to its knees. Many executives across the board are realizing how important is data for their business survival and growth. The costs of data loss are a significant threat to businesses’ financial and human capital.

A system administrator’s experience with data loss and imperfect backup

On a recent flight from Paris to Minneapolis, I sat next to a system administrator called John. After our acquaintance and talks about our mutual love of IPA and fishing, we pivoted and started talking about risk and how much data is worth for business entities. I judged that John was becoming a bit agitated during our discussion. I later learned that John left his job and took off four weeks and toured many places in Europe, after his company was devastated by a ransomware attack. He had several weeks to think through what happened to his former employer X Co and its disaster recovery and backup plans or …

I asked John “How much was X Co’s data worth?”.

I can see that John was upset. He let out a few big sighs and he resumed:
“A lot El. A lot. For starters the costs of the attack were very big. My team and I and our families were shouldering a great portion of the costs as well. I guess everyone who is in direct contact with X Co was affected directly or indirectly.”

“Didn’t you have a disaster recovery plan? And a backup plan John?”

“We did. But our servers’ backups where old and we spent weeks restoring from tapes and from NAS boxes. While waiting for the restores to finish, our sales team, marketing team, accounting, devs and everyone in the freaking … was waiting for our IT group. Many staff viewed us like villains and not heroes. I was in the data center for twelve days without taking a shower or eating a decent meal. Just Pizza and pop. It was very stressful. Management did not make it easy. We heard an earful 24/7. I just could not take it any more. Actually, I was the last one to quit in my group. I quit as soon as I restored operations to the level they were at a month earlier. It was the last straw for one of my co-workers who got divorced two weeks after the incident. He went home and his wife had left the house. None of us went home for two weeks. I missed my children’s soccer games and dance recital. I loved my job, X Co and the pay check, but that was too much stress to take. We looked like a bunch of chickens with their heads cutoff. I wish I spoke up early … I guess I felt intimidated and did not raise enough red flags after my proposals, to revamp our data protection, were rejected several times in a row.”

I saw John swallow Rolaids several times during our flight so I asked him “John, why are you stressed out?”
“Yes, he replied. I got a job offer from the new CEO, installed by the board immediately after I left. He wants me back and he offered me a 30K raise”.

I froze for a bit and cautiously asked:
“John? Didn’t you guys do a lousy job by not protecting one of your company’s most valuable assets? No backup?”

“We tried El. Believe me. Management turned down our proposals several times. Management threw sales and marketing bashes and spent hundreds of thousands of dollars on exhibits. Yet, they said we were a cost center. I even sent a report where I wrote about risks etc to our board after I ran into many road blocks. The new CEO got hold of it and emailed me and my team trying to get us back. I guess everything has a cost. X. Co made $100 million in profits last year. We were growing at 15% year over year and our leaders would not spend peanuts to protect our data. Our CFO told me that our data protection modernization proposal was sound but since it is over a certain amount, the CEO had to approve it. And he did not. We submitted it 2 times in 2016 and 3 times in 2015 and 2014. It was met with the same argument[We don’t want to be on the bleeding edge]. So we continued to cobble a bunch of goo together.”

“Everything has a cost El. However, I believe that the new CEO has a great vision and  that will succeed in turning things around and in restoring our reputation. I would like to help him but I am nervous about going back, nevertheless I was touched by how the new CEO reached out to me and my team. I have to make my decision as soon as soon as I get back home?”

Where is home John?

“On the east coast.”

What will make you accept the job then ?
“I guess, I will go back if they promise to treat us as professionals, fund our team projects, backup proposal and a few other things. We really can automate a lot of the manual stuff that they have. Just as you said earlier we could create multiple levels of redundancy while cutting costs and risks. I will inform you about how it goes.”

Time to say goodbye to my new system administrator friend but  …

We landed in Minneapolis, agreed to stay in contact and parted our ways.

While going through customs, I kept thinking how many  systems people like John are out there? How many system administrators and IT professionals are visible only while things go wrong. How many pull  all nighters, weekends to get ready for the staff when they show up in the morning and yet they are invisible except when an event like when a system or network fails or a breach or a data leak occurs? These brave men and women stand between many businesses and chaos.  I grab my bag and go home.

The following are some insights from John’s experience and how to avoid being in John’s shoes (except for his decompression time in Italy and Spain).

If you are a follower, be proactive and do not be passive. You need to be independent and  to use critical thinking to voice your concerns in a respectful way. Be a soldier and be vigilant. Be a great follower.

The following are a few examples of costs of data loss as John and I have discussed in our nine-hour flight from France.

• Labor costs
• Opportunity cost: sales, business development
• Operations or manufacturing, design, R&D
• Reputation cost
• Fixed costs
• Variable costs
• Accounts receivable: not able to bill and collect
• Accounts payable: not able to pay bills
• Material costs due to waste
• Setup time of manufacturing, R&D
• Inventory of finished goods not shipped

Variable costs due to data loss

• Labor
• Materials
• Opportunity
• Sales
• Marketing (marketing and advertising campaign costs while not able to deliver or sell)
• Reputation
• Future value of money of lost sales
• Customers
• Distribution channel
• Media
• Data recovery costs
• Suspected media and hardware costs

…

And above all twitter, Facebook, the “real and fake “ news will light up with X Co’s name.

Fixed costs of data loss

• Labor
• Rent
• Employee health care costs
• Insurance
• Utilities
• Depreciation
• Interests on capital
• Property taxes

…

Data loss costs are highly variable. They depend on the industry and the mission. They can even involve a real person’s life as in the case of health care, first responders, emergency management or national security.

I decided to write this post after I exchanged a few emails recently with John who gave me permission to write about this case without revealing X Co or his full name. John returned to his old job and the company deployed a new solution. Unfortunately for John, none of his other team members came back to work with him. I found out yesterday that John is also in the middle of a divorce that started after X Co’s data loss ordeal.

Are you a system administrator? A storage manager or a security person? What do you think? Share your insights.

Please send/post your comments and stories about how data loss affected you in the trenches and the heroic efforts of the IT team to recover (email to blogstories@enduradata.com). I guarantee your anonymity and that of your business.

PS: I changed the real names to protect the innocent.

–el haddi

Related posts