Campfire 365 Podcast – Episode 3: When The Dust Clears: Getting Your Organization Back On Track After A Big Disaster.

Carolyn Norton  00:13

Hello, happy campers. This is your host, Carolyn North. In our last episode, we dissect this the big disaster and how poor decision making can lead to a catastrophe. Many organizations get swept away by these disasters and never recover. Others get by with Justice great, what separates the organizations that succeed and those that fail? In today’s episode, we discuss what effective disaster response looks like, and how organizations can emerge from the rubble after a disruptive business event. Joining us today,

Daryl Moll  00:52

Darryl Hall, I’m a principal Cloud Architect with over 25 years of cloud and it experience.

Rich Fowler  00:57

Now I’m rich Fowler, I’ve got 20 plus years of experience of working with our partners worldwide. Hey, Amy

Amy McKie  01:03

Mackay, Director of Client sales have been working with clients, it clients for over 25 years.

Eric Robertson  01:10

And this is Eric Robertson, Director of it. 20 plus years of IT experience.

Carolyn Norton  01:16

Thanks, everyone. So the worst has happened? How do we react? What does a business leader do to determine what’s critical? What needs to come back online? And what’s not necessarily critical to start backup? How does an organization even determine how bad the damage was? Is there anyone they should be going to help? What do you do once the bleeding has stopped? To bring the business back on track?

Rich Fowler  01:43

Well, I think Darrell talked about it in episode two. Darrell talked about it earlier. And if you had your prioritization done correctly, then your marching orders are in front of you know what bleeding, you know, where it needs to be stopped, you know, what can go forward? And what can’t think the question might be, what if we hadn’t? What if we didn’t do that? What now? What

Daryl Moll  02:04

I think, yeah, and I think, you know, an Eric hit it to with the, you know, the the documented policies in place. But if all of that has failed, I mean, I think the first thing you need to do is really assess what happened. So, you know, assess what this what this disaster was, was it a, was it a, was it Mother Nature? Was it, you know, a, you know, a criminal act was a day, you know, was it it was technology related? Or was it physically, you know, affecting our plant and stuff like that, you need to, you know, first of all have a quick assessment of saying, Here’s what the issue is, and here’s, you know, the spectrum of what’s what’s affected. And then you look at that, and you say, how does that impact the priority of things? You know, at what point in my priority list does this fall in? have, you know, where that impact is happening? And if it’s across the board, then you need to start at your highest priority, you know, items, and the things that are most importantly, deemed most important to your business. You need to get those back on track. First.

Carolyn Norton  03:03

Let’s pretend I had a plan. I listened to episode two. The disaster happens. I marched in with my, my plan. And then I realized, whoa, I, I need help here.

Daryl Moll  03:17

I didn’t account for this.

Carolyn Norton  03:19

Actually, I just got hit with something completely unplanned. Think of COVID as a good example. What does an organization do at that point, I’ve done everything I needed to do. I thought I had everything on track. But Google got hit in the face and I’m not ready.

Rich Fowler  03:37

The first the first answer is cry. That’s always the best option. Once you get that out of your system, then you can go forward. But I don’t know if especially if you’re a business owner and small business folks, specifically, they spent a lot of time up at nights and stuff like this. That’s livelihood. So we we make jokes of it sometimes. And we laugh, but this is somebody’s livelihood. And if we think the rich come here, Eric, I’m glad we’re on flaperon virtual meeting. There are hundreds of people that whose entire families are affected by this, this isn’t a little thing. This is a big deal. So people have to pay attention to it. And sometimes to get them to notice that it. It’s the conversations you have to have early, but now now they’ve got to come to the table and say, Okay, did we have the conversations? If we didn’t, we’re going to have to have them in a hurry and under fire. So let’s go.

Daryl Moll  04:30

Well, but I think I think the important thing to really do is is again, that assessment and figure out where where something’s wrong and what needs you know, corrected and where does it impact your stack of priorities? I think the big thing to do is reach out to somebody in an organization, you know, and bring in an expert because you don’t have you know, at that point you don’t have you didn’t have the know how or the or the knowledge to, you know, realize that this was going to be an issue to begin with, which means you probably don’t have the ability to make a quick to decision and make great decisions on the fly as to how to remediate that issue or anything like that. At this point, you need to step back, you need to bring in somebody that’s an expert in whatever it happened, whether it was a fire, and you need to get a building back in to code so that you can do this, whether it was ransomware, and you need to bring in a security company to protect yourself from that and remediate from that, whether it was, you know, all the different facets, you need to have that assessment, figure out what the problem was, bring in an expert, and part of bringing in that expert, a big piece of that is, is hey, I need you to get me up and running as quickly as possible in a in a proper manner to best practices. But I also need to know what I, you know, I need you to glean some information back to me, as to how I prevent this from happening again in the future and how I recover, you know, if it ever happens again in the future, so that you do this as a learning experience,

Group  05:50

Daryl. Yeah, I

Amy McKie  05:52

think Darrell, you hit it right there, it’s because we’ve had clients call, you know, it’s like the chicken little sky is falling. Because they you know, they went to another company to try to help them get back online. That company had no idea what they were doing. They didn’t implement best practices. And then they’re in, you know, a double bad situation, you know, by the time they get to us.

Eric Robertson  06:15

And hopefully that this is part of the plan is identifying who to contact when issues occur.

Daryl Moll  06:22

Trusted vendor lists and valued vendor lists are huge.

Carolyn Norton  06:26

Absolutely.

Eric Robertson  06:27

And so, no, no, go ahead. No, now you can even get, you know, retainers for, for people who are ready to jump in, in case of a cyber incident or something. So you can have them ready and, you know, willing to help and drop of a dime, because you’ve already, you know, have that relationship set up with them.

Carolyn Norton  06:46

The worst scenarios, you’re in the mid year, you just got out of it, and you’re just getting out of it, and you’re looking for help and you’re having I’m gonna use an old reference, you’re gonna go to the phonebook and scrounge for resources, while you’re in the midst of it’s not the best way, how do you know they’re even going to help you? To Amy’s point do they know what they’re doing, you’re, you’re stuck in such a scenario where you’re trying to get yourself out of it, and you don’t know who to turn to for help. If you need that help. That’s, that definitely needs to be part of the planning process that way, whether you have a strong plan or not, you have a contingency, a contingency plan for your plan. So that if for some reason you need to go beyond the scope of effort, let’s a scenario that happens that you weren’t planning for, you can actually go ahead and have that list of contacts and resources, vendors, you know, anything you need that’s possible to help bring your organization back online.

Daryl Moll  07:43

Having that relationship is really important because think about it, you know, what happens if it’s a regional you know, disaster or something larger than just your your individual business? There might be 1000 other companies that are reaching out to that same vendor or different, you know, those types of vendors to get them back up and running before you. Do you know, to do that, you know, you don’t want to be the person walking around on a Friday night trying to find something to eat on find out you don’t you needed reservations for everywhere in a 10 mile radius.

Carolyn Norton  08:10

I’m think I’m assuming you mean, like natural disasters is affected at the same time?

Daryl Moll  08:16

Yep. Hey,

Amy McKie  08:17

let’s just go with supply chain problems.

Carolyn Norton  08:20

That’s also a big one. Yeah. Yeah. Supply and demand is also a big problem. Yeah, absolutely.

Rich Fowler  08:27

And anybody that’s tried to buy a new car in the last six months, understands that chips need to be made, and they need to come from somewhere.

Daryl Moll  08:36

I’m driving, I’m driving a great car because of that exact reason. That was the only one that is the only one that was on the, on the ladder at the time.

Rich Fowler  08:44

So everything works. Yes.

Carolyn Norton  08:50

So now we get to the point where we’re trying to get through this, we had a plan. We know who to turn to for help. But how do we know how exposed we we’ve been? What what actually happened? I get we get that a lot to where organizations say, Hey, we’re compromised, but we don’t know the level of severity that it’s been something happened and I know something happened, but how bad is it that doesn’t necessarily get planned out? So how does an organization triage and properly determine what all has gotten affected in their org so that they can best address it?

Rich Fowler  09:28

I think that kind of comes back to some of the prioritization that we’ve talked about, you’ve got to go in order of importance, because if you spend all your time on the little rocks, the big rocks fall through. So you know what, I can tell what the big rocks are, have, we can taking care of the big rocks and then working our way down through the priority list until we get to the small stuff. And we can we can figure out what was there and from the reaction especially if you’ve got people in the field. They’re going to help you know when they can’t get into systems or can’t get their job done or can’t talk to suppliers and things like that. So You’re going to get information coming in from lots of areas. Yeah. And

Daryl Moll  10:03

I mean, I think the bottom line is you bring in an expert, according to whatever your issue is, or whatever you’re impacted, you know, you know, system or process is, at that point, you want to make sure you bring in somebody who’s been there done that seen that and knows exactly what to do. Because the reality is, if you had the experts on staff to be able to answer those questions and do that, you probably wouldn’t have ran into this situation in the first place. So the fact that you’re in the situation means that you need some outside assistance from somebody who knows what they’re talking about.

Eric Robertson  10:35

And that’s somebody who knows should be providing some sort of root cause analysis so that we can identify how we prevent it in the future. figuring out why it happened is that the key in preventing it, I’m going

Rich Fowler  10:47

to relate it back to your basic business, almost every business has an attorney, and an accountant and a janitor jobs, you don’t really want to do yourself, but have to be done when it’s time. A disaster recovery person or somebody like that, I think that that’s a retainer that has to happen. And if a business isn’t thinking about that, now they’re going to shortly. And I know many attorneys will be annoyed that I put them in the same category.

Eric Robertson  11:17

Well, to take your as your your analogy a step further. It’s almost like everybody has car insurance, and cyber insurance is now becoming the norm. So you know, having a cyber insurance, okay, something does happen. And now you have to pay for it. Well, what are you going? How are you going to pay for it? Do you do you have that just those funds laying around? Or have you kind of already paid for part of it, knowing that something might happen, and now you’re, you’re able to cover those costs. And those costs can be pretty big.

Carolyn Norton  11:45

I circled back to the triaging aspect that reminds me of are doing EMS work, where you got to look at what all has happened, who’s critically affected and get those guys addressed first, and then you work your way down the chain. So you’re right, the planning aspect is really going to dictate what all you’re gonna focus on to bring the business back online. And ideally, the plan with the external resources, understanding the priority level is going to help the organization come back online. And Eric, to your point, when you have the security, the cybersecurity insurance to help with whatever actually happened. If it was a cyber issue, you might have other insurance that could cover other scenarios. And ideally, your exposure is covered, because you’ve looked at it from all angles. How do you how do we deal with the impact to staff and employees, though? So let’s say the organization plan is underway. But how do we make sure that, you know, our external relationships are addressed? We talked in our last episode of making sure that we’re communicating effectively what happened and how an organization’s planning to remediate and prevent it from happening again. But how do we also bring up to speed? What’s going on with the staff members, with our relationship with our clients and customers so that they understand what’s a go forward as well?

Daryl Moll  13:13

Yeah, I mean, I don’t mean to take it back to episode two. But I mean, it all goes back to your plan, and what you had that plan for, it’s funny, you know, communication, your communication piece of that plan is is probably at the you know, in this episode that we’re talking about right now is probably the most important and it’s how you communicate, whether it’s, you know, when it’s something just happened, and you’re just finding out about it communicating to your employees, hey, this is what we need to do. This is what you don’t do, this is what you do. You know, and then you communicate out to your vendors, you communicate out to your clients in your customers. But that communication plan is key. But it all goes back to what we mentioned in episode two, which is coming up with that plan in having that plan documented. I mean, I go through a lot of conversations with customers, and they’re like, you know, you ask them the question, you know, what’s your appeal, or to the Eric mentioned, your recovery point objective and your recovery time objective, and they’re like, I can’t be down for you know, more than 15 minutes, you know, and they throw out these wildly crazy numbers, and you’re like, Okay, that’s fine, that gives us a mark, we can, we can go to that mark, but then we start bringing in the cost factor. And we say, okay, for an RPO of 15 minutes and an RTO of, you know, 15 minutes, that’s gonna cost you, you know, a million dollars. And everybody’s like, Whoa, I can do, I’m like, Well, you know, I can get you to a RPO of, you know, one day and an RTO of one hour for $500. And they’re like, how, what’s the big difference? Well, the shorter recovery points and the quicker recovery times that you get to the more expensive it is, and you can you can find these balances. But part of that is again, having that plan and having those honest conversations, how long can I be down for? If you tell me 15 minutes, it’s gonna cost you a million dollars. If you tell me you can be down for a day, it’s gonna cost you 500 bucks. I mean, I’m obviously making those numbers up, but the numbers go change drastically. And having that conversation and having that plan of knowing what to do on how to get back after those disasters, whatever it may be, whether it’s technical, you know, or physical or anything along those lines, you’re going to have $1 amount associated with that, and you need to, you know, as Amy mentioned, weigh those risks, or what that that financial impact is, of the financial impact against what you’re gonna have to pay to, to, you know, to have a plan for that, or to have something like that ready to go beforehand, well,

Amy McKie  15:31

and it’s also the plan of, Hey, okay, if we’re gonna go with we can be down a day, here’s the other 15 steps that we need to implement. And these 10 employees need to switch what they’re doing, you know, in this job and start doing this, you know, to be proactive in that. So it all goes back into the big plan.

Eric Robertson  15:51

And I think part of this, and we haven’t really addressed yet is testing the plan, and acting on the plan and making sure that the plan is feasible. And you don’t know putting something down on paper is going to work until you actually try it. And having again, a regular cadence of testing your disaster recovery of each of the pieces is a big piece of that. And making sure you know what, you know, how, how long is the estimated time for these recovery actions to actually take because it’s not just flipping a switch, there’s going to be pieces that you either have to restore or turn back on or, you know, so on and so forth, that unless you practice it, and get better at it and know what those steps are going to be so that it’s not so nerve wracking when the issue does occur, you’re now prepared for it because you’ve not only written out a plan, but now you’ve tested that plan, and are somewhat prepared. Of course, there’s gonna be curveballs thrown in the mix. But you’re better prepared than never testing it at all.

Rich Fowler  16:45

And I think we’re forgetting one, one big piece of it, that doesn’t necessarily have to cost the company money. And we can look at FEMA for New Orleans and some other things. One of the big failures was communication to the world and to their own employees, if we’ve got marketing involved, and we’ve got the press releases pre written or at least storyboarded out and we know what’s going to happen, we just don’t know when or how bad. But if we have that that communication plan, both internally and externally as part of our disaster recovery plan, then that’s going to make a huge difference in the reputation of the business, the reputation of employees, because you’ve got employees sitting around and let’s be honest, no employee sits around wondering what’s going on. They’re all talking to each other lighting up instant messaging, lighting up Facebook, lighting up everything they can light up. So, you know, rumors and bad information. Misinformation gets out there quickly. So to have marketing involved and let marketing take the lead on what do we tell? And who do we tell? I think that’s a critical piece.

Carolyn Norton  17:49

To your point, both internally and externally. For sure, corrupted. So

Amy McKie  17:54

clients, it’s vendors, it’s all of it.

Eric Robertson  17:57

And making sure you have a communication platform in case your communication platform is the one that goes down to

Amy McKie  18:04

be you know, outside of normal.

Eric Robertson  18:08

There’s vendors for it, there’s backups to backups, how do I send out these messages? How do I how do I let people know? You know, if there’s a fire in a building, how do I how do I how do I let all the people in that building know that there’s a fire and they need to get out and then check on the status of you know, that they got out?

Daryl Moll  18:24

Yeah, we’re the issue is, hey, that the internet is down, which is a huge would be a huge disaster to people, hey, my VoIP phones don’t work. My Facebook doesn’t work I can’t do and, you know, the top nine communications, you know, platforms that I’m doing don’t work anymore. How do I you know, still get information out to people?

Amy McKie  18:41

Yeah, and it’s electricity or natural disaster or tornado, you know, comes through when you’ve got half, you know, half an area that’s down. So

Eric Robertson  18:49

smoke signals, right? Yeah.

Rich Fowler  18:52

Tell your nearest Junior High kid, they can get messages out from anywhere.

Carolyn Norton  19:00

So when an organization has a plan, and they’re in the midst of recovering what are the employee supposed to be doing in the meantime, twiddling their

Eric Robertson  19:07

fingers, part of

Rich Fowler  19:08

that goes back to that communication part of the communication shouldn’t be telling them what to do.

Daryl Moll  19:13

They have a they have a plan for like Amy mentioned, you know, hey, you know, you know, the shipping docks are down because there was a fire loading docks are down because there’s a fire them, hey, we’re gonna go you know, stage everything over here in this location and move different things over there, there’s there should be a plan for what every person is doing.

Eric Robertson  19:30

And that should be part of what’s communicated and and the employees should be part and should know what the disaster recovery process is for some certain of these scenarios so that they know like, hey, if this happened, here’s where I’m supposed to look for that type of information or here’s what my expectation is to move

Daryl Moll  19:46

forward. And one key piece that I would say is important to is, you know, the higher up you are, the lower the further away you are from those daily things or those those those, you know, boots on the ground type of operations that are happening But while this is going on, people should be paying attention to what’s working and what’s not working so that they can have that conversation after the fact to tweak the plan going forward. Because, again, that frequency of conversations and the fact that happens, there’s no conversation that is more important from a frequency standpoint than having the conversation just after something just happened. What did we learn? What can it how do we prevent it from happening again? And how do we do better the next time something happens?

Eric Robertson  20:27

And there’s always room for improvement? That’s right. But always,

Daryl Moll  20:31

so somebody should be paying attention to, you know, not, you know, obviously, the prioritization, again, going back to that keyword, is, you know, to get things back up and running, and make sure everybody makes sure everybody’s safe and get things back up and running. But somebody should also be paying attention to how things are working and what needs tweaked moving forward. Yeah,

Amy McKie  20:47

it’s really listening to feedback from, from employees,

Daryl Moll  20:50

from employees, from vendors, and from customers,

Carolyn Norton  20:53

almost surveying How well did the plan actually work? And where can it be improved, and just reiterating, you know, revisiting the plan and making it better, ideally, only annually, but of course, after a disaster, if possible.

Daryl Moll  21:09

And, you know, I mean, you know, we’re talking about after, you know, when that when the dust clears, the biggest thing is you want to be the the your company wants to be the one in charge of getting the message out, you don’t want your, you know, your competitors, you don’t want random media to be the one reporting your issue, because that’s gonna all go back to whether it’s your, your brand, and that’s gonna go back to, you know, whether or not your customers still respect you, that goes back to whether or not, you know, investors or, you know, everybody in anybody, if they if you want to retain some credibility and some of your reputation in that brand, you need to be the person out there, not only trying to put your spin or but just being that person that’s reporting this goes a long way to, you know, doing that in that communication plan of, hey, here’s what happened. Here’s, you know, again, here’s what happened, here’s, you know, how we’re reacting to it. And here’s what we’re doing to prevent it from happening again,

Carolyn Norton  22:06

by doing that, you illustrate that you as an organization did have some thought leadership, spent some time planning and communicating to whoever it is that you need to communicate to what has happened and what’s occurred, it just makes you look that much more organized. I can’t remember who said it before, but these things do happen. It’s it’s almost expected from everyone in organizations. So it’s not surprising when it does. But when you come out strong, leading the charge and communicating effectively, here’s what happens. Here’s what’s going on, and we were aware of it, and we’re fixing it, you just you just look so much better than having someone else, some having someone else fill in the gap for you having someone else speak for your organization, which we never want to do to your point, Darrell. Absolutely. So

Amy McKie  22:58

it goes to show

Carolyn Norton  22:59

you that planning is a big key. But we also understand that there might be scenarios where plans for as good as they can be, there might be holes, there might be gaps or areas of improvement. And just making sure that after everything happens, you know exactly what you can do on a turn of a dime. So that you can reach out for help. If you need an additional assistance, or you need guidance beyond what you’ve planned out for you have that factored in as well. And making sure that after everything has been done, you take an opportunity to sit down and revisit what worked, what didn’t work, what was missed. So that way that’s incorporated into your new plans day, and you’re iterating off of that every single time. So that way, the organization just continues to evolve and make sure that they’re they’re providing that safety net that the plan is supposed to provide.

Daryl Moll  23:56

Yeah, I mean, my closing thoughts would be simply, you know, get in there, assess the situation, assess what the risks are, what the what the disaster was and what what it relates to, and then reach out and bring in an expert because you know, if something happened, that means that you didn’t have it accounted for or, or what you thought you had accounted for wasn’t exactly a great the greatest plan. Reach out to somebody who’s an expert in that area and have them come in, remediate the issue, give you the root cause analysis of what why it happened so that you can prevent it going in the future.

Rich Fowler  24:29

And I always heard it was the ACA for you for using Eric’s acronyms, assess the situation, communicate our plan, and then adjust to whatever the situation throws back at us.

Carolyn Norton  24:41

Nice. I like it.

Amy McKie  24:43

Well, I mean, even if there’s not a plan, I mean, companies don’t always have a plan at least have a list of maybe trusted vendors so that you’re not running around like Chicken Little with your head cut off, you know, in the middle of this disaster and you don’t know what’s going on. You’re not going to leave the babysitter without you know, giving them phone number refers to a couple of people or the doctor, the pediatrician or whatever. So it’s at least maybe have a shortlist of some people that you can call in the middle of everything.

Eric Robertson  25:09

And right now we’re giving you that time to create these plans. So hopefully you’re not just with a short list and you’re taking the time to create these plans, create the response, playbooks identify those key people internally, externally, what processes are going to get affected how all these interconnected systems that we have may be affected, just try to get that big picture. So you know, if something was to happen, what’s going to be affected? Because if you don’t know, you’re only going to find out what to does.

Carolyn Norton  25:39

I think we can all agree that just start with a plan, it could be as light, obviously, we’d prefer it to be robust, but start with a plan. It’s not no plan is ever perfect, but at least you’ve put thought into it and you have a starting point. And that ideally gets you into the mindset of planning for the big disaster. The big disaster can be a devastating event for your organization, but it doesn’t have to be the end of it. With proper planning and quick thinking. Even the most battered of organizations can still put up a fight to those organizations that can survive the worst of the worst. They have one thing on their mind. How do I prevent this from happening again? Join us in our season finale of the big disaster, where we put together everything, creating the disaster proof organization, developing a solid strategy for disaster recovery and business continuity.