You’ve certainly heard the infamous quote from Mike Tyson, “Everyone has a plan, until they get punched in the mouth.” This is certainly true when it comes to surviving a zero-day attack on your company. We’ve all be through the planning and tabletop exercises, but are you really ready? Miguel will share some practical advice beyond the “incident response plan” that speaks to mindset, decision-making, communication and leadership during a crisis moment.
Transcription:
Miguel Edwards: (
Well, good afternoon, everyone. Hope everyone's doing well. Before we begin, let me just offer a shout out and recognition for the folks that dig in and all of the sponsors for braving this new world. It's so good to be back in person. It's been a long time since I've seen a ton of you guys, and so thank you, Nathan, and the rest of the dig in team round of applause for you guys for doing this and, thank you for supporting it. Fantastic. So, we're gonna talk a little bit about, some of the practical approaches, to surviving a zero day attack, the actual event itself. I'm not gonna be technical at all, but, I am a cyber event survivor. And after that I think that's a term. I think if not, I've just coined it.
Miguel Edwards: (
But after that, I actually went out and started chatting with a lot of my peers, other CIOs and other, VPs of infrastructure and security to really try to understand what their experience was and see if that could be something that I can kind of collect into a series of thoughts around some of the things that we often don't go through in the tabletops. So speaking of tabletop exercises, show of hands who here has been through a tabletop exercise, couple of you guys. Okay. Give that some thought. Think about that tabletop. Okay. Who can share with me your thoughts around that experience? How long was that tabletop exercise as an example, anybody volunteer few hours. Okay. Anybody else value that you got out of it? Were you multitasking while it was going on? Yeah. On the phone texting Facebook, right?
Miguel Edwards: (
The problem with those tabletop exercises that we don't, we don't take it seriously. And there's a lot of technical stuff that goes on in those exercises. And what occurs to me is that there's an entire body of knowledge that's perhaps missing. And what I want to do is just try to provoke some of that thought around some of those tabletop exercises and maybe some of the additional softer skills that we need to think about. So let me set the stage for you real quick, and you're hanging out and you're thinking about a pina colada, cuz you're on the beach somewhere. Can you close your eyes and imagine this setting you're on a beach. You're on vacation. You're on PTO. All right, let me help you out a little bit
Miguel Edwards: (
Better. Okay. Got it. All right. So you're on the beach and everything is cool. Kids are playing and then you get a little bit of a distraction and what is that? Oh, probably a car warranty ad or something. I don't need this. I'll just cancel that. And then a couple minutes you get a text message followed by another text message. And I recognize that phone number is that Jennifer? Oh man. Maybe I should pick, I should pick this up and see what's going on back in the home office and sure enough, it's a problem. And it may sound like this. Hey
Matt: (
Paul, this is Matt. We're getting multiple reports of applications being down. And we just got a single report in Wisconsin from some sort of green screen message, demanding a rant payment. What, what should we do?
Miguel Edwards: (
That was my 14 year old son, by the way, first gig as a voice actor. But that typically is what happens. You might be faced down on a massage table, right? You're on a beach, you're on your commute, but you're definitely not prepared for this. And now it's time for you to jump into action. They don't talk about that in the tabletops that you're somewhere in The Bahamas. When you get this call to come in and now you have to take charge. So what are some of the first things that you have to think about? Well, I'm not a stranger to crisis and it's a disaster. And one of the first things that you have to think about is that when is there crisis or there's a disaster, you go to that plan and what is that plan worth? Right. And we all know Mike, Tyson's famous quote, everybody's got a plan until you get punched in the mouth, right?
Miguel Edwards: (
And that's oftentimes what happens when you're dealing with a crisis event. So you have that incident response plan and you go to it. But for some reason, there isn't any instructions on how you lead into that plan. So now there's all this instruction about recovery and what to do and all of these other things, but it doesn't always match the situation. So the first thing that occurs to me retrospectively thinking back is that there needed to be a recognition that what has just happened is a trauma. That's the first thing. Now I said before, I'm not a stranger to crisis, my company was like 10 blocks away from the world trade center during nine 11. And so I remembered that event and I remembered the crisis management that was required to start figuring out how do you rally a team around the fact that we have to deal with a trauma?
Miguel Edwards: (
So if you think about, if your home was invaded, if you think about your car was broken into, we've been violated essentially as what has happened. And we have to recognize the trauma that's associated with that. It requires a mindset shift. You're no longer necessarily just a CIO. You're no longer just a CSO. You're no longer just ahead of infrastructure. You're not only just ahead of applications, but you now are leading through a trauma. You have to recognize that you have to recognize that there's gonna be people who are gonna be working nights and days for an undefined period of time. Wondering if the company's even gonna survive. What has just happened? Everyone's worried about the reputational damage, man. Should I start looking for another job right now? Was this my fault that this happened? That's a thought process, right? As a CIO, I'm thinking I have the accountability for what has just gone on right now.
Miguel Edwards: (
I hope I have a job, right? So there is that mindset shift that's required. The next thing that you have to think about is potentially depending on how your company should, this might be your company now who else is qualified to make decisions about the next step in this recovery is the head of distribution gonna make decisions about what to do next is the head of product is your chief actuary gonna make some decisions around what's going on? So now all of a sudden you're in this position now where you are going to make decisions that will have an existential impact on the future of your company. So if you've gotta prepare yourself for that and you have to be prepared again, as I mentioned before, to deal with your team and you have to be prepared to help your team, see the future, to see the recovery and to see all of the opportunity that's gonna come out of this.
Miguel Edwards: (
Okay. So now what are some things that we're gonna do now? Next, the first thing I would suggest in talking to all of my colleagues and again, thinking about my own experience. So super critical is to establish command and control. That is the most important thing at this time. Now there are a couple about four key elements that I'll talk you through that are important to establishing command to control. The first thing is the recognition that you have to control information, information that's coming in about this event. And you have to recognize that that information is gonna be largely imperfect. So I once had, and I'm not, she's actually in this room. She once told me to be careful, Miguel, cuz a lot of times you act on the best, last information you've had and this is not the moment to do that, my mentor.
Miguel Edwards: (
Okay. And so now is the time to recognize the fact that all the information that you're getting is potentially imperfect, it's potentially wrong, but it's also potentially right. And so how do you control that information as that information is evolving and making sure that we know what happened? Do we really know what happened? Are we sure? Was it only one thing that happened or was it multiple things that happened and how do we make sure that that flow of information continues to come in, but it's coming in through a very narrow funnel so that you don't have a bunch of people running off making decisions, not fully understanding the globality of that information. Also too, as you're thinking about that information, you may have heard the saying, what did when did you know it and what did you do when you found out about it?
Miguel Edwards: (
That's also too very important in your note taking, because what people don't talk about in these tabletops is that in the cyber event, there's a lot of potential future litigation that you have to be worried about. And by the way, nothing I say here should be construed as legal advice or technical advice, please consult with your family physician. But at the end of the day, you have to make sure that you're documenting all of this information, how you got it. What did you do when you got that information? Who did you share it with and how did you operate with that information? So super important. And how do you make sure that you have a process to vet the information that you're getting, ensuring that it's accurate? How do you test that information to make sure that it's something that you should act on?
Miguel Edwards: (
So now you're gonna act on information that you have, and you want to make key decisions because we've gotta move quickly. But how do we make decisions that ensure that we're not foreclosing future optionality? This is a question around, should we just burn the ships? Do we just shut everything down and start all over? That might foreclose future options as new information emerges? Are we gonna start the recovery process right away? I'm gonna just give you a couple of reasons and a couple slides as to why that might not be the right idea. So you have to always be thinking about multifaceted decisions and what's the impact of the various decisions that you're going to make, but we have to make sure that we limit the damage as quickly as possible. So how do we make sure we close everything down again without foreclosing future options?
Miguel Edwards: (
And can we quickly establish a perimeter so that we can gain control, command and control of what is going on? The next element that we want to think about is making sure that we have decision frameworks in place. So one of the most important things to figure out is who actually has the authority to make decisions, have you, as the incident commander, if you will been granted the authority to make decisions, or do you have to take everything up by committee? How many committees are there in your job right now in your company? You wanna get a project to prove how many people do you have to go through it? Is that gonna be the same thing during your incident response? So what does that look like? And how do we determine what the timeframe are for the timeframe is for various decisions that have to be made so we can make those decisions quickly.
Miguel Edwards: (
Also importantly, going back to that information record that you're building is how are you recording and memorializing those informations. So for I'm sorry, those decisions. So for example, one of the things you might want to think about is an architectural decision log, so that as you're making these decisions for your recovery, that you're recording why it is that you burned a ship, why it is that you shut this off? What are you going to do instead? And what's gonna be step one, step two, step three, for each of those decisions. So developing a decision log is key for memorializing the event. And then finally not the last and not the only thing, but an important thing to control is the flow of communication. Communication is key during these events. And it's a very, very tricky, tricky game to play because you don't want to create a vacuum of information.
Miguel Edwards: (
Cuz I have a saying that says in the absence of information, people create their own and it's almost always wrong. So you don't want to create an information vacuum because then you're gonna have people run in the hallways talking about, oh, the company's dead. We're all gonna go home. We're gonna lose our jobs. You want to make sure that you're creating some amount of information, but how are you communicating internally is one of the important things that you have to think about. And depending on how large your company is communicating internally could be communicating publicly. The press is very, very interesting when they find out something's wrong, they know exactly who to call inside of the company usually starts with the call centers, by the way. Right? So that's something that just think about what is the communication strategy for your call centers. As an example, in the event of an attack, you have to be thinking about your leadership team and how you're communicating with your leadership team.
Miguel Edwards: (
What's the involvement of your board of directors? How often do you meet? What type of information do you share with them? How about legal? All of those types of things have to be thought about in terms of communicating. And then finally, what are you gonna do with respect to external communication? When one of the worst things you see is when companies come out half cocked with communications, oh, it only affected just this number of PO, oh, sorry. No, no it's oh no, no, no, it's all this whole, you have to be very, very careful with how much you communicate and when, but then also too, you don't want to be skirting the facts that you've got a problem cuz you will be found out. Okay. So then now that we've got that command and control established, now we're ready to start the recovery, right? So let's just recover.
Miguel Edwards: (
It's super easy, right? Nope. We are working in an active crime scene. Folks. It is literally an active, a crime was committed and there's an active crime scene. So imagine if your house was broken into its ransack and the police are doing their investigation and all of a sudden you come in with a vacuum and you start cleaning up and you start putting things back where they belong. You just got the new TV from best buy you're putting while the cops are still investigating. Would that work or are you gonna be trampling evidence impeding their investigation? So what's unique about these types of scenarios. Again, that I don't think they teach us in the tabletop is that there are two things going on simultaneously that we have to be aware of. We have to be aware of the forensic analysis that is ongoing. And we also have to be aware of the cyber recovery that will be ongoing.
Miguel Edwards: (
And we need both of those things to happen at the same time. In fact, there's a little bit of a symbiosis between the two and these things don't come out. So let's talk about the forensic portion. First, the first most important thing about the forensic analysis is identifying what they call patient zero. Why is that important? If we don't know who patient or what patient zero is, we don't know what the attack vector was. How did they get in? How do I know if they're still there or not? I have to know when they got in and what they did. So as they figure out who did what I figure out what they did and how it worked, and then I can figure out where they moved, what's called lateral movement. So did they go from a workstation to a server? How did they operate?
Miguel Edwards: (
Were they dropping little latent pieces of malware around so that if I fix this thing, this one, would call home and activate. That is all of the interesting dynamics. And by the way, this is a very highly specialized team. You're kidding yourself. If you think you have these people in your shop, it's a very highly specialized people who know how to get these packages. And I learned an interesting term. They detonate these packages in a safe way. It's really fascinating the way they talk about this stuff. Yeah. We found the malware and we detonated it. And here's what we discovered about. It gives us Intel on the forensics. We would need to know how the malware functions. So was this an encryption based malware was a control based malware. What is it that this malware is supposed to actually be doing? And again, these are all important things that help us determine how it is that we're going to recover.
Miguel Edwards: (
Was there data that was exfiltrated as part of this, we need to understand that. Cause that's gonna form the basis of all the communication and how much life lock you're gonna have to buy. Right. Gotta know how much and what was it, sensitive data. And by the way, knowing how much data was exfiltrated knowing how the malware functions may also affect or inform how it is that you negotiate with the threat actors. Otherwise you're negotiating blindly. Oh, we've got your folks and your people and your data. And we make, a billion dollars. I did the analysis. No you don't. That might be your position. Okay. And then finally are the threat actors internal to your organization and perhaps in cahoots with the external threat actor, cuz that happens sometimes have to be careful. We have to know if we have threat actors inside, we have to know who the threat actors are outside. Knowing who the threat actor is externally, knowing what crime ring they're part of could also too, be important to understanding the M.O How they typically operate. So that's why it's also too important that that forensic analysis be conducted. But while that forensic analysis is being conducted, while you're trying to not trample in a crime scene, you are also still trying to recover your organization. Your company is down hard. So now you also have another set of folks that are trying to help support business continuity plans.
Miguel Edwards: (
Now your question is, are your business continuity plans effective for a cyber attack? Cause that's interesting. A lot of the bus BCP plans that I see a lot of the D.R Plans, I see really speak to weather risk in smoking whole scenarios, right? You lost your data center, but do they does the BCP planet? You have speak to a potential cyber threat, something to just go back and ask your folks about you're negotiating perhaps also too, with these threat actors along the way that negotiation is important. You have to think of a couple things. I mean, so is your data encrypted and do you need the encryption key? Will it work when they give it to you? These things don't come with warranties guys, right? You can't go out and Sue somebody if they gave you the key and it didn't work. So you think about the negotiations with them.
Miguel Edwards: (
Are they threatening to put all of your sensitive data up on their big boards for the world to see? Do you care about that? Are they gonna do it anyway? Even if you pay them questions, you gotta ask. You gotta think about how you restore your network. Now, as I think about disaster recovery plans that I've seen does, I guess I'll ask you rhetorically. Do you guys have an entire data center of spare networking gear that you would just start standing up and lighting up while you're not trying to trample on the crime scene servers, spare servers laying around. That's a lot of capital. So there's this magic that you have to be trying to be prepared for on how it is that you're gonna recover while you're doing this forensics. And by the way, guys take take a guess. I'll take volunteers. And it says, how long do you think typically the forensic analysis to identify patient zero takes take a guess six weeks, eight hours, two and a half weeks is on average what it takes. So really long or really short. Yeah, it was either really long, really short. I'll give you in the middle though, two and a half weeks. So now let me ask you this. Is it tenable for you guys to remain down for two and a half weeks while you're waiting for the experts to come back and tell you what happened?
Miguel Edwards: (
So we gotta be thinking about those options and that's where we get into this data recovery data restoration. Again, if I don't have the forensic analysis done, what is my recovery point? So of recovery, time objectives, recovery point objectives at what point do I recover? If I don't know what the attack vector was and when the actual attack occurred, I might be potentially restoring data and systems that are still infected. So I thought I shut everything down, restore everything. Guess what? E.T Phone home happens all over again. I gotta start from scratch. So having that forensic analysis is important so that exactly what your recovery point should be. And then finally, you've got end users out there. Who's like, how do I get back online? How do I do my job for, your sales organizations? They take their job very, very seriously. They have relationships with their brokers and with their agents and with their customers. And they're down hard. How do you support them? What do you do?
Miguel Edwards: (
So I don't have the answers for you guys, but what I'm said I'm trying to do is provoke some thought about some of the things that may not be coming out in the tabletops and in the preparations that you're doing. So let me give you a couple of three areas of support that you may find helpful going to the outside for. So the first area of support is as it relates to public relations, having a public relations firm on retainer, on tap, ready to go, who knows your organization, who knows your customer base, who knows your style is important to help you with things like responding to press release. I'm sorry to media inquiries to creating the press release when it comes time to doing so. These are very, very delicate matters that have to be thought out where you may want a public relations firms.
Miguel Edwards: (
Who's dealt with cyber events who have a better idea on how to communicate with the public and with your customer base in the event of a cyber event. So having a public relations team is important for that, as well as all managing the internal communications. Again, as I said, if your company is large enough, internal communications could be just as dangerous as external communications, the next area. In social media as well. Social media is an interesting thing because what happens, especially if your company is large, all of a sudden you'll start seeing social media posts, where it says, X, Y, Z insurance company, they're not responding to me. Something must be wrong. And then you start seeing that on social media. How do you respond to that?
Miguel Edwards: (
So that damage control piece is really, really important from a public relations perspective. The second pillar that you may want to think about from external support is dedicated and specialized legal counsel. And having those folks on retainer in our business, it's highly regulated, right? And so in our business, there might be regulations on the books that speak to how, and when we have to inform our lead regulator on a cyber event and how you do that is very important as well. Okay. You have to remember that there's this potential legal activity. That's all gonna happen after this thing is closed. You've got criminal, you've got civil proceedings that could definitely follow a cyber event. So they're there to help you with all of the legal and the regulatory stuff, whether or not you should notify law enforcement. That's an interesting question. Should we call the FBI and let 'em know that this has happened.
Miguel Edwards: (
There's pros and cons to that. And again, that's a legal decision. That's a decision that you're gonna have to make with counsel, but reaching out to the FBI and informing them that you've been hit, they might be able to tell you things about this threat actor's M.O Where they're seeing other types of activity, the types of ways that they've seen other folks recover from it. And talking to my colleagues, I've heard by and large, that notifying law enforcement is fairly useless. You don't get a whole lot of help. There was this interesting, story that I heard, if you to go outside, find a local bank and go Rob a bank and they probably have what, like a hundred thousand dollars in there, there will be a stream of cops following you, wherever you go. And they will probably gun you down.
Miguel Edwards: (
Once they find you a cyber attack, which is responsible for hundreds of millions of dollars of losses. There's like one guy in all of Louisiana helping that out from the FBI. So it doesn't make a ton of sense. I don't know if there's, notifying law and enforcement is terribly helpful, but they will also legal counsel will help you prepare for potential civil action. And that's gonna be a big deal. You've gotta make sure that you've got this stuff documented. You've gotta make sure that you're being proactive and they're helping you all throughout every one of your decisions. They're helping you make the right decisions, that position you best in for, a civil defense. I talked a lot about that forensic analysis. First thing that happens during a cyber event, you'll probably get a legal hold. Evidence. Preservation is super, super important to protect your executives, to protect your directors, to protect the company.
Miguel Edwards: (
We need to make sure that we're preserving evidence and doing that while the it folks are moving things around and plugging things in and plugging things out and deleting things, evidence preservation can be pretty fascinating thing to deal with. And then you wanna think about threat actor negotiations. And at the end of the day, it's, probably a little unnerving calling somebody with a Russian accent or whatever, and trying to have a discussion with them about the ransom. That's probably something you wanna leave to a professional. And there are there's an entire industry out there that deals with negotiating with threat actors. And, I don't know if you're gonna use your own personal Bitcoin wallet to make the payment or whatever it is, but that's not an easy thing to do. And you've gotta worry about anti-money laundering, provisions and all kinds of like, how do you do this?
Miguel Edwards: (
How do you legally do this, foreign corrupt practices that comes into play? There's lots of different legal issues with actually negotiating with and paying a threat actor. And then finally, I'm assuming that a lot of people have cyber insurance. And so what are you doing in terms of documenting all of the things that would support a cyber claim so that you can be compensated renumerated for, the event. And that's another long drawn out process is actually filing all of the evidence that's necessary to get the benefit of a cyber insurance claim. And then, as I mentioned before, that highly specialized group of people that help you with that forensic analysis, you need to have think about having someone on retainer that you can call up that has experience dealing with cyber threats. That understands how to do that forensic analysis.
Miguel Edwards: (
Not very many. It shops have that skill in house. It's very, very difficult to procure that skill. And these guys are professional scientists. They're computer like literally computer scientists that know how to deal with this work. They're gonna also to prepare independent reports that you're gonna need for your insurance company, for your third parties, by the way, that want to know how did you recover? Is my data now safe? And so these third party, party report written by a forensic analysis is very helpful. It's very credible to illustrate how you have recovered from this event. They're also gonna provide you technical support around your recovery, especially that support that's necessary. So it's not conflicting with or frustrating that forensic analysis that I was talking about. So you may want to check in with them and say, Hey, we're gonna do this.
Miguel Edwards: (
What do you think? Nope, you gotta hold off for 24 hours because we're still analyzing that piece of hardware. Don't touch it. Okay. Got it. Helpful. I'll move on to the next thing. So they can provide a lot of technical recovery support. And then the most important element from a technology perspective is they're gonna tell you where the vulnerabilities are. So, Hey, while we were scanning things and we found out how the bad guys got it, we found these 15 other things you might want to take a look at. So that's also super helpful to have those technical guys on tap to do that. Let me wrap up with a couple of additional considerations that you want to think about, and then I would love to take any questions. So the first thing is make sure that you're thinking about this context in terms of disaster recovery versus cyber recovery, they are not the same thing.
Miguel Edwards: (
They are very, very different dynamics and circumstances. And I will almost guarantee you that your disaster recovery plan is useless in the face of a cyber attack. They're not translatable. So make sure that you actually do have, and not just an incident response plan, but that you have a technical cyber recovery plan. Okay. I'd also encourage you to think about as part of your contingency plan. Think about all of the major in flight initiatives that you have so that if you actually did have to pull a rip cord on a cyber event, how many consultants and contractors do you have running that 15 million project that are now gonna be idled? What are you gonna do with those guys? Do they just still hang out and bill you, do you have good vendor relationships? Do you have provisions in your sows that would, you'd be able to, pull a parachute clause on say, all right, we're gonna pause this.
Miguel Edwards: (
Everybody go home for now. What are you doing with your big major in flight initiatives, along those lines, you're thinking about your third party vendors. So if you have any TPAs, you have claims administrators, all these folks that are on the hook waiting for, to operate in your company. And all of a sudden the lights go out. How do you interact with those folks? So third party conversations are important. How are you gonna pay your people? Payrolls. Next Friday systems are down. Folks have to get paid. Some of the, very simple things that our company does every day, paying vendors, AP, right? We could have a little bit more grace with those folks. We're on net 30 net, 40, whatever our vendors, but we're not on net 30 with our employees. They're expecting a paycheck next Friday. So how are you gonna handle that?
Miguel Edwards: (
Your claims operations, especially if you're in a medical space, that's difficult, right? You gotta think about how we're getting our providers paid and life insurance claim, no one wants to hear, I'm sorry. Our systems are down. We'll get back to you. I'm sorry for your loss. By the way, we've gotta really be thinking about how we deal with our customers specifically in a claims environment. And when I was talking about business continuity, it's important to think about what does business continuity look like when you have no access to your systems? So there's this concept out there called digital bunkers that says, do you put data that you need and data in terms of maybe spreadsheets and customer lists or whatever it is in a bunker that's accessible in the event of a disaster. And then think about business continuity in the broader context.
Miguel Edwards: (
And then finally go back to the tabletops, the testing, but thinking about it in terms of this emotional context that I'm talking about, the softer skills, do you have, are you ready? Or do you have what it takes? Is your team ready to operate in this type of environment? Do you know the right things to say, do you know the right types of imagery to convey that type of thing is gonna be super important as you practice through your tabletops and your testing. So with that, I'd love to open it up for any questions that you have.
Nate Golia: (
Good. We got about hello. We got about one minute. I probably take one question
Miguel Edwards: (
One minute, one question, sir,
Audience Member 1: (
What's more vulnerable public cloud or what would you prefer?
Nate Golia: (
I'll repeat that question. What's more vulnerable public cloud or on-prem. And which one would you recommend?
Miguel Edwards: (
So I gotta give you, it's gonna be a 1000% opinionated response here. I've gotta believe that a well designed public cloud environment is a gajillion times more secure than anything I can do in my own data center now, but that said, it's gotta be well designed. There's very, very easy ways to build very vulnerable public cloud environments. But when I think about just whether or not I wanna secure a data center, I'm not interested.
Nate Golia: (
We could take maybe one more, one want one more. Well, in that case,
Miguel Edwards: (
Thank you so much. Thanks,
Nate Golia: (
Miguel. Appreciate you.