NSI specializes in Virtualization, Disaster Recovery, Managed Print and is a New England IT Consulting company, this has allowed us to be exposed to an array of IT problems all over Connecticut, New York, and Massachusetts areas and has led us to see not only the problems within different companies IT departments, but also a good understanding of where most companies lack in IT policy and general flaws in their DR Plans. We hope this will help people in preparing for disaster from the human aspect of things.
When most people think of a network going down they general attribute the problem to a hardware failure, whether it be a server or hard drive failure most people blame the devices themselves. But 29% of all data loss is contributed to human error, either from an IT professional who forgot to perform the correct backup, or an office employee who accidently deletes an important file; data loss is real and happens all too often. Unplanned downtime occurs whenever something serious happens to your network that wasn’t planned and can have different effects based on when and how bad the incident is, if a server crashes at 2am and your business operates 9-5 then you are probably alright, but if things were to shut down at 10am during the holiday season then there can be some serious revenue loss for the company. Juniper Networks reports that Human Error is the cause of 50-80% of all downtime, this, along with our 29% of data loss also being human error, shows that large amounts of money loss and headaches can be avoided by implementing policies that don’t just focus on hardware and software fixes, but instead to add policies to help avoid mistakes from happening.
Most IT staff know about disaster recovery, not all have a plan, but most know having redundant servers and reducing the amount of single points of failure can go a long way in securing the IT center if the hardware should break. But where many IT centers lack is the human aspect of ensuring that data loss and downtime are kept as low as possible. When setting up a DR plan the question of who is often downplayed but is extremely important; Who is in charge of damage assessment? Who is charge of the disaster recovery? Who is charge of calling the correct people? Having these answers is just the beginning to having a properly put in place plan for when disaster strikes, every company needs a DR Team, a group of people who are in charge during a period of disaster.
First having the right team is most important, just because one person is a great manager doesn’t mean they would fit any of the rolls if they live an hour away, so make sure these are people who can get to the company ASAP if a disaster should happen during the night. Next these people must be put on a list with their contact information so that they can all be called quickly; nobody wants to have to scramble around looking for Bill’s cell number if there is a water leak in the server room. Each person the list should have a specific duty and have it stated, crisis time is not when you want to have an argument over leadership, so having each person dedicated to a specific role will keep it so there aren’t 10 people doing the same task.
Having a vendor sheet is very important. Make sure you have a direct line to each of your vendor reps, this allows you to contact them directly versus having to go through their corporate phone tree in order to speak with your rep. This list should also come with a copy of their SLA’s, this way you know how long they will take to act upon your disaster. Remember nothing can get done if your agreement says they will fix it within the next 24 to 72 hours.
Having these lists is important but so is keeping them all up to date. If you made this list 2 years ago then many of the employees might have changed jobs, or simply moved/changed cell phone number. Having an outdated list is almost as bad as having no list at all, if you spend the time to call your DR team and find out 2 of them now live on the other side of the city and 1 of them changed their number and the other 3 are no longer employed by you, then having a list would seem useless. So having one person who is in charge of checking the list ever 3 months by calling everyone on it and verifying phone number, address, and if they are currently an employee is critical. This also applies to the Vendor list, you want to make sure the corporate number is still in working order, that your rep is still employed, and that your warranties or SLA agreements haven’t changed.
Check out Part 2 of our blog to learn more on how to set up the right team for your DR plan.
