It was the 3rd night before Christmas. The elves were busy working, building, and assembling everything to do with the “Big Day”. The toys would travel down the line with each adopting their own expertise to the required task at hand, along with similes, songs and the hand off “high fives”.
The inevitable happened at 12:13pm as everything suddenly stopped! The line was idle, the toys were stationary, the elves confused, and then all of them simply looked at Tommy. Tommy was the Grand Elf. He was responsible for the entire workshop; everything from children’s requests, inventory management, manufacturing, packaging and labeling, right down to the toy placement into the sleigh. Any other month, everyone would be on lunch now, but there are no lunch breaks in December. Just shorter hours with more shifts.
Today he was challenged as Bobby “IT” left early for the day – something to do with family & doctors, he remembered.
Pausing and reflecting for a moment, Tommy went to the computer room to see if there is anything he could do. As he was walking there, he got a call from Bobby. With a sigh of relief, and grateful it was him on the phone’s display, he took the call.
“Bob glad you called, something happened today” said Tommy. “We have a huge problem! Nobody is working as the line as it has stopped, and Santa will ….”
Bobby jumped in, “I know, I just got an alert that one of our servers is down. Can you go into the computer room and check something out for me?”
Tommy walked into the room. Looking at the rack, “Bob, one of the servers does not have any lights on, and besides something smells funny too!”
Bob, looking to direct Tommy to the center sticker said, “Is it the WList Server?”
“Yes” Tommy replied.
Bobby thanked him for his confirmation and continued with “It’s the Wish List Server. Everything we do revolves around it. The children’s wishes are maintained on it, along with all their information, addresses, language, right down to proper positioning on the sleigh for organized for drop offs. That’s why the line stopped.”
“Ok; now what?” said Tommy
“Get on the PA and tell everyone that everything will be fine and operational in about 12 minutes”.
Are you sure? What do we do, if it doesn’t?” said Tommy.
“I will call you back shortly, talk to you later” and with that the line dropped.
In the workshop, Tommy got on the PA and addressed everyone, “Attention, this is a minor stoppage and everything will resume in approximately 20 minutes. Please take your break now, if you can.” Minor stoppage, he thought, this is a CATASTROPHE, with not much time left! Surely, Santa will be here in no time.
Pacing the floor, he could feel his blood pressure rising. Of all times, this has to be the worst time for this to happen, he thought. He went outside to calm down but did not stay long as it was a crisp -40.
Back in, his cell rang and he was hopeful that it might be Bobby. With a smile, he answered the call.
“Login, everything should be fine and call me back” said Bobby.
With that, he walked to the station and logged in. Upon hitting Enter, he knew that things would be fine. In short time, the line was moving, everyone was working and the Christmas mood was back in full swing!
He looked at his watch – it was close to 12:25pm and decided to confirm all with Bobby.
In the truck, Bobby got the ….
Suddenly, the alarm clock sounded 6:30 and Bob reached over to shut it off.
“A dream?” he thought, as he got out of bed to start his morning routine.
As he was getting dressed, he smiled and reflected upon it, as some time ago, he experienced a similar challenge, but with much greater downtime…..
Last year he encountered a server malfunction in his office and even though he maintained daily backups, the recovery process was a major headache! The full recovery took over well over 10 hours. He will never forget it as he had internal challenges with personnel that day regarding the data loss, work stoppage, customers that were confused and THE CHAT with his boss…as he thought he was going to get FIRED.
The next day knowing that everything was back to normal, his boss approached him. Again, he thought he was about to be fired. Instead, his boss started a discussion on his business – “the loss was not just about downtime and costs – it was about customer perceptions, integrity and responsibility, as it may take months to find a customer, but mere seconds to lose them”. He instructed Bob to “come with a complete fail-over plan, so that it NEVER happens again. He also asked him to send an email to all of their customers communicating the business interruption and that due diligence is currently underway to insure that it does not happen again. I should have tested my backups, he thought.
The emails were sent and then began his due diligence. Searching the web, he came across “Backup options’, but he wanted more than just backup – the recovery is where the challenge is, he thought. He came across Disaster Recovery options with RTP and RPO, but he was inundated with information and he needed to educate himself further, but he didn’t want this to linger for weeks. As he left for the day, he decided that his game plan would be to contact a few providers and pick the best.
Call number three was to Lanworks, as they have a DRWorks offering. Lanworks was somewhat different, in that he actually had a DR discussion with someone and it wasn’t a sales pitch. They offered to bring an engineer onsite to review the current environment, and together develop the DR requirements. Once they completed the “discovery process” they drafted the documentation, visio’d the layout and communicated several options. Throughout the process, not only did he get educated, but he knew that DR was going to be his best “insurance”.
Lanworks provided him with;
- Flexible RTO & RPO options of which he selected 15 minutes and 8 hours respectively
- Offsite data provisioning of his 1.2TB, which was safe, secure and encrypted
- Automated replication of his 4 virtualized servers
- Automated daily monitoring of DR operations
- Ability to recover from a single server or all his servers
Plus other benefits-but at least these will protect his long term tenure, he thought. Armed with the information at hand, he scheduled a meeting and presented his findings to his boss.
“I suggest that we implement a DR solution that has a reasonable cost and designed specifically for businesses just like us”, he said. He referenced the diagram; the process, benefits and what would happen if their office “blows” up.
The boss interjected, “How do you know our data can be easily recovered, in time of need?
“They perform regular testing and provide us a quarterly compliance reports – if we want monthly, they can do that to”, said Bob.
Boss, “If the building blows up, we don’t have a business to operate”.
“Not true”, said Bob. “If the building blew up, they also have DR seating, or a place where we can relocate our employees and have access to all our data or information. Besides, we can always access the data remotely via VPN”.
Suddenly, his boss knew this was the right thing to do. Bob stated the cost, his boss agreed, and as always he wanted it… “yesterday”!
Within a week Bob had his DR operational. An email went out to all customers stating that the company now has a DR plan and that business interruptions will NEVER happen again. There is huge benefit in being proactive, rather than reactive, he thought.
As he finished his thoughts, he walked downstairs and just smiled when he looked at the “Tree”.
