1 00:00:03,799 --> 00:00:05,879 Speaker 1: If you didn't listen to a single thing I said, 2 00:00:06,519 --> 00:00:13,599 you can listen to these three things, Collaborate, plan, and practice. 3 00:00:14,439 --> 00:00:18,359 Speaker 2: Welcome listeners to the Industrial Security Podcast. My name is 4 00:00:18,440 --> 00:00:22,120 Nate Nelson. I'm here with Andrew Ginter, the vice president 5 00:00:22,199 --> 00:00:26,399 of Industrial Security at Waterfall Security Solutions, who's going to 6 00:00:26,480 --> 00:00:30,120 introduce the subject and guest of our show today. Andrew, 7 00:00:30,320 --> 00:00:30,679 how are you? 8 00:00:31,960 --> 00:00:34,320 Speaker 3: I'm very well, Thank you, Nate. Our guest today is 9 00:00:34,439 --> 00:00:37,920 Chris Cistrunk. He is the technical lead of the Mandient 10 00:00:38,399 --> 00:00:41,840 ICs or ot security consulting team, whatever you wish to 11 00:00:41,880 --> 00:00:46,359 call it. Google purchased Mandian in twenty twenty two, but 12 00:00:46,560 --> 00:00:50,119 they're still keeping the Mandian name, so he still identifies 13 00:00:50,240 --> 00:00:55,719 as technical lead of Industrial Security Consulting at Mandian. And 14 00:00:55,759 --> 00:00:58,679 our topic. You know, they as part of their consulting practice, 15 00:00:58,679 --> 00:01:00,479 they do a lot of incident response. He's going to 16 00:01:00,520 --> 00:01:04,799 talk about lessons learned from incident response in the industrial 17 00:01:04,840 --> 00:01:05,599 security space. 18 00:01:06,480 --> 00:01:09,159 Speaker 2: Then, without further ado, here's your interview. 19 00:01:12,319 --> 00:01:16,599 Speaker 3: Hello Chris, and welcome to the podcast. Before we get started, 20 00:01:16,680 --> 00:01:18,760 can I ask you to say a few words for 21 00:01:18,799 --> 00:01:21,599 our listeners about your background and about the good work 22 00:01:21,680 --> 00:01:22,719 that you're doing at Mandian. 23 00:01:23,519 --> 00:01:29,079 Speaker 1: Okay, thanks Andrew. I'm again at Mandient on the ICSOT 24 00:01:29,319 --> 00:01:32,400 consulting team, been doing that for over eleven years now, 25 00:01:33,200 --> 00:01:38,519 focus on ICSOT security consulting around the world with every 26 00:01:38,599 --> 00:01:45,560 type of critical infrastructure, doing incident response, strategic and technical assessments, 27 00:01:45,920 --> 00:01:50,079 and doing training as well. Before that, I was electrical engineer. 28 00:01:50,560 --> 00:01:54,319 I still am, but for a large electric utility energy 29 00:01:55,079 --> 00:01:58,840 was there over eleven years as a senior electrical engineer 30 00:01:59,079 --> 00:02:04,000 Transmission dish should skate a substation automation and distribution design. 31 00:02:04,640 --> 00:02:07,959 So that's a little bit about me and again just 32 00:02:08,080 --> 00:02:10,680 working for Mandian, part of Google Cloud. 33 00:02:11,080 --> 00:02:14,879 Speaker 3: And our topic is incidents. It's lessons from incidents. But 34 00:02:14,919 --> 00:02:18,000 let's talk the big picture of incidents. I mean, Waterfall 35 00:02:18,039 --> 00:02:22,479 puts out a threat report annually. I'm one of the contributors. 36 00:02:22,879 --> 00:02:27,000 You know, we go through thousands of public incident reports 37 00:02:27,039 --> 00:02:29,479 looking for the needles in the haystack, the incidents where 38 00:02:29,479 --> 00:02:32,759 there were physical consequences, where there were shutdowns, where you know, 39 00:02:32,800 --> 00:02:38,120 sometimes equipment was damaged, and we rely on the public 40 00:02:38,280 --> 00:02:44,840 record on public disclosure. And so I've always believed that 41 00:02:44,919 --> 00:02:49,319 we were under reporting because I'm guessing again, I don't 42 00:02:49,319 --> 00:02:52,879 have that many confidential disclosures that people tell me about, 43 00:02:53,080 --> 00:02:55,319 but I'm guessing that there's a lot more out there 44 00:02:55,360 --> 00:02:58,800 that never makes it into the public eye. You folks 45 00:02:58,879 --> 00:03:01,960 work behind the scene, you know, without reaching any non 46 00:03:02,000 --> 00:03:05,159 disclosure agreements or anything. Can you talk about the big picture. 47 00:03:05,319 --> 00:03:09,520 Do you see incidents, especially incidents you know, in the 48 00:03:09,520 --> 00:03:15,400 industrial space with physical consequences, incidents that triggered shutdowns, incidents 49 00:03:15,520 --> 00:03:18,840 that are not public reported. How many are there, what 50 00:03:18,879 --> 00:03:21,159 do they look like? Can you talk anything about sort 51 00:03:21,199 --> 00:03:23,960 of what I would not see by looking at the 52 00:03:23,960 --> 00:03:24,599 public record. 53 00:03:25,360 --> 00:03:28,280 Speaker 1: Sure, thanks for the question. You know, I think we're 54 00:03:28,319 --> 00:03:31,879 talking about cybersecurity incidents here, and there's many incidents that 55 00:03:31,960 --> 00:03:36,800 happen every day, right but life goes on. Squirrels happen 56 00:03:36,919 --> 00:03:42,240 right in the grid. But for cybersecurity incidents, I do 57 00:03:42,439 --> 00:03:47,560 believe we're seeing an increase. I can't go into how many. 58 00:03:48,439 --> 00:03:53,360 We actually have a report that m Trends Many has 59 00:03:53,360 --> 00:03:55,240 put out every year. It's going to come out later 60 00:03:55,280 --> 00:04:00,240 this month and for RSA, and this is a yearly report. 61 00:04:00,759 --> 00:04:09,039 We report on the different themes, the different targeted victims, 62 00:04:09,439 --> 00:04:14,639 the different threat groups, the TTPs. But for cyber attacks 63 00:04:14,759 --> 00:04:20,480 that impact say production or cause of a company to 64 00:04:20,680 --> 00:04:25,120 shut their operations down. I don't have any hard fast 65 00:04:25,240 --> 00:04:31,079 numbers to talk about, but we have seen an increase 66 00:04:31,720 --> 00:04:35,839 and you can look in not just our report, but 67 00:04:36,079 --> 00:04:41,560 also the reports of others IBM X Force, Verizon, d 68 00:04:41,639 --> 00:04:49,120 b I, r Drago's others. There are increasing reports of these, 69 00:04:49,199 --> 00:04:50,879 and a lot of it has to do with things 70 00:04:50,920 --> 00:04:57,399 like ransomware UH and ransomware either directly impacting the control 71 00:04:57,480 --> 00:05:03,120 system environment, which we have responded to in a manufacturer 72 00:05:03,959 --> 00:05:10,319 and a few others, but we have seen in the public, 73 00:05:11,199 --> 00:05:13,920 you know, news where a company might have to shut 74 00:05:13,959 --> 00:05:18,800 down operations due to indirect impact. Maybe their enterprise resource 75 00:05:18,879 --> 00:05:24,560 planning software or manufacturing execution software was impacted, which is 76 00:05:24,600 --> 00:05:29,000 an indirect impact to the OT critical data flowing that 77 00:05:29,160 --> 00:05:33,199 was halted, which means I can't produce my orders anymore, 78 00:05:33,399 --> 00:05:36,920 or track shipping or logistics things like that. So we're 79 00:05:36,959 --> 00:05:40,439 seeing a lot of those. There's others in the electric 80 00:05:40,480 --> 00:05:42,920 sector that they kind of have to be reported to 81 00:05:43,279 --> 00:05:48,480 OE four seventeen reports. If there's a material impact, obviously 82 00:05:48,480 --> 00:05:51,560 they'll be filed in the U or they're supposed to 83 00:05:51,600 --> 00:05:55,199 be filed in the eight K or ten K with SEC, 84 00:05:57,160 --> 00:05:59,680 and so I think if you take all of those 85 00:05:59,680 --> 00:06:04,079 source and look together and see, we see there's an 86 00:06:04,120 --> 00:06:11,240 increase of operational impact. Mhm. But it's I think the 87 00:06:11,279 --> 00:06:14,199 engineers are doing a good job of UH and the 88 00:06:14,240 --> 00:06:21,199 folks that run these systems are minimizing the impact in 89 00:06:21,240 --> 00:06:26,439 these situations, and especially for electric and water and other 90 00:06:26,600 --> 00:06:31,519 critical infrastructure. Manufacturing is critical. But I'd say it is 91 00:06:31,519 --> 00:06:37,079 probably the highest targeted outside of you know, healthcare and 92 00:06:37,120 --> 00:06:38,879 other other areas. 93 00:06:39,439 --> 00:06:41,680 Speaker 3: So work with me on on the numbers, just you know, 94 00:06:42,120 --> 00:06:46,439 for one more minute. I'm on the record in the 95 00:06:46,560 --> 00:06:49,879 in the the Waterfall Threat Report speculating as to what's 96 00:06:49,920 --> 00:06:54,920 going on with public disclosures. It's my opinion, but you know, 97 00:06:55,000 --> 00:06:57,439 I have limited information to back it up. It's my 98 00:06:57,519 --> 00:07:01,199 opinion that the new disc closure rules in the SEC 99 00:07:01,319 --> 00:07:04,759 and other jurisdictions around the world are in fact reducing 100 00:07:04,759 --> 00:07:07,399 the amount of information in the public domain rather than 101 00:07:07,480 --> 00:07:10,839 increasing it. And the reason I suggest this is because 102 00:07:10,879 --> 00:07:14,000 it seems to me that with the new rules, every 103 00:07:14,000 --> 00:07:18,040 incident response team on the planet roughly you know, I overgeneralize, 104 00:07:18,040 --> 00:07:20,480 has a new step two in their playbook. Step two 105 00:07:20,519 --> 00:07:23,240 is called the lawyers, and what the lawyers say, they 106 00:07:23,279 --> 00:07:28,680 say say nothing, because if you disclose improperly, if you 107 00:07:28,759 --> 00:07:31,519 fail to disclose widely enough, you can be accused of 108 00:07:31,560 --> 00:07:36,680 facilitating insider trading. If you disclose too much information, you 109 00:07:36,759 --> 00:07:39,240 might get sued. I mean people have been sued for 110 00:07:39,759 --> 00:07:45,079 disclosing incorrect information about security into the public. People buy 111 00:07:45,079 --> 00:07:48,000 and trade shares, and then they know, they find out 112 00:07:48,040 --> 00:07:51,000 the information was incorrect and they get sued. And so 113 00:07:51,519 --> 00:07:54,480 to me, the mandate for the lawyers is say the 114 00:07:54,519 --> 00:07:58,439 minimum the law requires, because if you say too much, 115 00:07:59,279 --> 00:08:02,040 you risk make a mistake and getting sued, and you 116 00:08:02,040 --> 00:08:05,639 don't want to get sued. And if you you know, 117 00:08:05,639 --> 00:08:07,600 if you say too little, you're going to get sued. 118 00:08:08,120 --> 00:08:10,879 You know, the lawyers minimize. And if you have, you know, 119 00:08:10,920 --> 00:08:13,360 a material incident, you must report it. If it turns 120 00:08:13,360 --> 00:08:16,560 out the incident is not material to the finances of 121 00:08:16,600 --> 00:08:19,560 the company, you don't have to report it. And again, 122 00:08:19,600 --> 00:08:22,920 to minimize the risk of getting sued by reporting incorrect information, 123 00:08:23,439 --> 00:08:27,040 you report nothing. So my sense is that we're seeing 124 00:08:27,160 --> 00:08:30,680 fewer reports because of these mandatory rules, not more of 125 00:08:30,680 --> 00:08:33,879 than what do you see? You know, you see this 126 00:08:33,919 --> 00:08:35,799 from the other side does this make any sense? Do 127 00:08:35,840 --> 00:08:36,960 you have a different perspective. 128 00:08:38,039 --> 00:08:40,960 Speaker 1: I can say that, you know, as an incident responder 129 00:08:41,080 --> 00:08:45,600 working with you know, victims in critical infrastructure, but also outside. 130 00:08:45,600 --> 00:08:49,000 I think this is a broader question you bring is. 131 00:08:50,240 --> 00:08:53,679 I can definitely confirm that we work with external counsel 132 00:08:54,080 --> 00:08:57,720 that a victim may have hired to bring in to 133 00:08:57,840 --> 00:09:03,120 handle a lot of these reporting or not reporting requirements. 134 00:09:03,240 --> 00:09:09,000 I can't say or confirm that the lawyers themselves external 135 00:09:09,039 --> 00:09:13,399 counsel under reports. I can't say that I don't know. 136 00:09:14,200 --> 00:09:16,679 I'm not a lawyer, nor do I play one on Facebook. 137 00:09:16,960 --> 00:09:20,200 So I will just stick to say yes, we have 138 00:09:20,320 --> 00:09:26,080 worked with external counsel, and usually we do not say 139 00:09:26,080 --> 00:09:30,639 anything in public for us as the incident responder, unless 140 00:09:32,080 --> 00:09:37,759 the victim company or our client asked us to, because 141 00:09:37,799 --> 00:09:42,039 sometimes sharing information is a helpful thing, especially if it's 142 00:09:42,080 --> 00:09:48,120 a big breach. Sharing that lessons learned about what has 143 00:09:48,120 --> 00:09:52,360 happened to them with others, just like we did back 144 00:09:52,360 --> 00:09:55,360 in the day when we had the solar winds breach. 145 00:09:55,559 --> 00:09:58,519 So you know, there's two ways of thinking of that, 146 00:09:58,720 --> 00:10:02,320 and maybe you can pull on that throw with some 147 00:10:02,360 --> 00:10:05,240 other experts, but not me. I don't know about the 148 00:10:05,240 --> 00:10:06,320 External Council part. 149 00:10:09,399 --> 00:10:13,679 Speaker 2: Andrew, you'd referenced Waterfalls Annual Threat Report, a report which 150 00:10:13,799 --> 00:10:17,080 I've covered in the past for dark reading. I'm not 151 00:10:17,120 --> 00:10:20,159 sure i've seen this year's iteration, So maybe you could 152 00:10:20,559 --> 00:10:23,679 tell listeners just a bit about what the report covers 153 00:10:23,840 --> 00:10:25,799 and what the numbers are showing lately. 154 00:10:26,559 --> 00:10:31,440 Speaker 3: Sure, the report uses a public data set. The entire 155 00:10:31,480 --> 00:10:33,320 data sets in the appendix. You can click through to 156 00:10:33,320 --> 00:10:38,960 it if you wish we cover we count in our statistics. 157 00:10:39,000 --> 00:10:43,799 We count deliberate attacks, cyber attacks with physical consequences, not 158 00:10:43,840 --> 00:10:48,360 stole some money, physical consequences in heavy industry and critical infrastructure, 159 00:10:48,519 --> 00:10:52,120 the industries we serve in the public record. Okay, no 160 00:10:52,519 --> 00:10:59,200 confidential disclosures. The numbers last year were seventy two attacks, 161 00:10:59,240 --> 00:11:03,120 I believe with I don't know some you know, a 162 00:11:04,879 --> 00:11:06,879 one hundred, one hundred and fifty something like that. I 163 00:11:06,879 --> 00:11:11,639 forget the numbers sites affected. Many of the attacks affected 164 00:11:11,720 --> 00:11:15,720 multiple sites. This year, you know, we're up from seventy two. 165 00:11:15,879 --> 00:11:19,519 We have seventy six attacks affecting a little over a 166 00:11:19,600 --> 00:11:22,200 thousand sites, So there was more sites affected, but the 167 00:11:22,320 --> 00:11:28,799 number of attacks did not increase sharply. And this is why, Again, 168 00:11:28,879 --> 00:11:32,200 I speculate, why have we sort of seen a plateau. 169 00:11:32,399 --> 00:11:35,600 We went up from you know zero essentially give or 170 00:11:35,639 --> 00:11:42,159 take in you know, let's say twenty nineteen to you know, 171 00:11:42,240 --> 00:11:45,960 seventy two, and then you know in twenty twenty four 172 00:11:46,000 --> 00:11:49,200 to seventy six. Why do we seem to have a 173 00:11:49,200 --> 00:11:51,320 bit of a plateau? And I'm speculating it has to 174 00:11:51,320 --> 00:11:55,600 do with the SEC rules. People are now legally obliged 175 00:11:55,679 --> 00:11:58,000 not just in you know, the United States, the Security 176 00:11:58,039 --> 00:12:01,480 and Exchange Commission, They're legally not just in that jurisdiction, 177 00:12:01,559 --> 00:12:04,000 but in other jurisdictions around the world. There's similar rules 178 00:12:04,039 --> 00:12:06,759 around the world. If you have an incident that is 179 00:12:06,840 --> 00:12:10,600 material that any reasonable investor would use as grounds to 180 00:12:10,799 --> 00:12:14,600 buy or share or sell or value shares, you must 181 00:12:14,679 --> 00:12:18,159 disclose it. But I have the sense that we are 182 00:12:18,279 --> 00:12:23,200 seeing fewer disclosures because by law you're required to disclose 183 00:12:23,320 --> 00:12:28,399 material incidents. And again because I speculate that because the 184 00:12:28,480 --> 00:12:33,720 lawyers are involved, we are seeing fewer disclosures. You know, 185 00:12:33,799 --> 00:12:36,679 they disclose the material incidents and they squash everything else. 186 00:12:36,919 --> 00:12:39,480 Is the sense I have. But you know, you asked 187 00:12:39,480 --> 00:12:44,879 about the numbers seventy six last year. Nation state attacks 188 00:12:44,879 --> 00:12:48,159 are up from you know, there were two the year before, 189 00:12:48,240 --> 00:12:50,960 there were six last year? You know, is this a trend? 190 00:12:50,960 --> 00:12:56,919 It's still small numbers, who knows? And industrial control system 191 00:12:57,080 --> 00:13:01,799 capable malware malware that understands industrial protocols and that is 192 00:13:01,879 --> 00:13:07,039 apparently designed to manipulate industrial systems is up sharply with 193 00:13:07,159 --> 00:13:11,639 there were three new different kinds of malware disclosed last 194 00:13:11,720 --> 00:13:13,679 year or you know, found in the wild last year 195 00:13:14,000 --> 00:13:17,159 that had that capability versus you know, seven in the 196 00:13:17,200 --> 00:13:21,840 preceding fifteen years. Again, small numbers. Is it a blip? 197 00:13:22,039 --> 00:13:25,480 Is it a trend? Is AI helping these people write stuff? 198 00:13:25,480 --> 00:13:28,399 We don't know? So these are all sort of you 199 00:13:28,440 --> 00:13:29,960 look at the numbers and you scratch your head and 200 00:13:30,039 --> 00:13:32,879 you go, I wonder this. Doesn't you know what's going 201 00:13:32,919 --> 00:13:37,120 on here? So that's that's the threat report in a nutshell. 202 00:13:37,320 --> 00:13:39,159 There's other statistics in there, but those are sort of 203 00:13:39,159 --> 00:13:46,159 the headlines that leads us into the topic of the show, 204 00:13:46,200 --> 00:13:49,120 which is lessons learned from incidents. You folks do incident 205 00:13:49,120 --> 00:13:52,000 response all the time. You know, can you talk to me? 206 00:13:52,360 --> 00:13:54,240 What are you seeing out there? You know, is there 207 00:13:54,240 --> 00:13:56,639 an incident or three that sticks in your mind. As 208 00:13:56,720 --> 00:13:58,879 you know, Andrew, the most important thing I have to 209 00:13:58,879 --> 00:14:01,039 tell you is and you know or the most recent 210 00:14:01,440 --> 00:14:02,559 Where would you like to start? 211 00:14:03,480 --> 00:14:08,679 Speaker 1: Okay? Sure? We have been doing OT incint response since 212 00:14:08,720 --> 00:14:12,840 I've been here, and I can give you a few examples. 213 00:14:13,919 --> 00:14:17,559 Last year in twenty twenty four, we responded to a 214 00:14:17,639 --> 00:14:24,240 North American manufacturing company that had their OT network for 215 00:14:24,320 --> 00:14:30,279 looking at a Purdue model. It's the third layer or 216 00:14:30,519 --> 00:14:34,519 level three of the network was directly impacted by the 217 00:14:34,600 --> 00:14:41,480 Akira ransomware game. And what had happened was an unknown 218 00:14:41,559 --> 00:14:45,320 Internet connection was made by this third party who was 219 00:14:45,399 --> 00:14:49,120 running the site. They had put in their own Cisco 220 00:14:49,240 --> 00:14:55,159 ASA firewall, and it just so happened to be that 221 00:14:55,759 --> 00:15:00,399 there was two critical vulnerabilities in that firewall at the time, 222 00:15:01,120 --> 00:15:07,320 and the Cure ransomware game was targeting targeting those exposed firewalls. 223 00:15:08,200 --> 00:15:13,519 So don't necessarily think this was a targeted manufacturing OT attack. 224 00:15:13,759 --> 00:15:18,080 It's just ransomware gangs doing what they do, trying to 225 00:15:18,120 --> 00:15:21,679 make money, and so they were able to log in 226 00:15:21,840 --> 00:15:27,120 and get in through these vulnerabilities and deployed the ransomware 227 00:15:27,240 --> 00:15:32,320 on directly on the OT network, which was flat and 228 00:15:32,799 --> 00:15:36,159 every system but about five or six or seven were 229 00:15:36,240 --> 00:15:43,240 completely encrypted, including their o T DCS vendors, And there 230 00:15:43,279 --> 00:15:46,919 was multiple not pick picking on any one in particular, 231 00:15:47,000 --> 00:15:53,240 but ge A, B. B Rockwell several others that were there, 232 00:15:54,320 --> 00:16:00,279 and the backups server was impacted in the back up 233 00:16:00,600 --> 00:16:03,399 of the backup server was impacted. They were all on 234 00:16:03,440 --> 00:16:08,279 the same flat network. So this was a really tough 235 00:16:08,320 --> 00:16:16,240 situation since the company manufacturing did not have any backups 236 00:16:17,720 --> 00:16:22,480 that were offline. The OT vendors, like I mentioned for 237 00:16:22,600 --> 00:16:26,000 I had to come on site to completely rebuild the 238 00:16:26,039 --> 00:16:33,200 Windows systems, the Windows servers, the engineering workstations, the hmis, 239 00:16:33,360 --> 00:16:36,879 all the things that were Windows and or Linux that 240 00:16:37,000 --> 00:16:40,919 had to completely rebuild. Client didn't pay the ransom in 241 00:16:41,000 --> 00:16:45,559 other words, and so the lessons learned here. Work with 242 00:16:45,639 --> 00:16:52,000 your OT vendors and OEMs and even your contractors to 243 00:16:52,000 --> 00:16:56,000 make sure that your Windows systems and Linux systems have antivirus, 244 00:16:57,879 --> 00:17:02,639 make sure that you have OT backups that are segmented 245 00:17:02,679 --> 00:17:08,599 from the main OT network, and keep offline backups and 246 00:17:08,680 --> 00:17:13,319 test them on a at least a year basis. Backups 247 00:17:13,359 --> 00:17:16,400 will get you out of a bad day, even if 248 00:17:16,400 --> 00:17:19,480 it's an honest mistake at five o'clock on Friday. So 249 00:17:19,640 --> 00:17:23,480 this is a basic win here having good backup strategy. 250 00:17:24,640 --> 00:17:30,319 And then in the last case here we recommended they 251 00:17:30,640 --> 00:17:36,440 eliminate this external firewall and leverage the existing itot DMZ 252 00:17:36,599 --> 00:17:42,839 firewall that came in from the main owner of that site. 253 00:17:43,279 --> 00:17:48,319 And so they had a back door essentially that this 254 00:17:48,480 --> 00:17:52,279 third party contractor had installed in a new Internet connection. 255 00:17:52,440 --> 00:17:56,200 So give away from the shadow it go back to 256 00:17:56,279 --> 00:18:00,680 your normal itot dmz with jump box two, factory authentication 257 00:18:00,799 --> 00:18:04,519 and all those things. But if you do the basics 258 00:18:04,559 --> 00:18:09,079 and do them well, keep good segmentation, have backups, and 259 00:18:09,559 --> 00:18:12,960 patch your firewalls on a regular basis, I think that 260 00:18:13,039 --> 00:18:16,440 will go a long way, especially in this case. 261 00:18:19,920 --> 00:18:23,680 Speaker 2: You know, I feel like I've heard of variant of 262 00:18:24,000 --> 00:18:28,039 that advice that Chris just gave a million times, and 263 00:18:28,359 --> 00:18:31,839 I don't work in industrial security, so you folks must 264 00:18:31,839 --> 00:18:34,359 hear it all the time, or it must just be 265 00:18:34,880 --> 00:18:37,640 such basic knowledge that you don't even think about it. 266 00:18:38,039 --> 00:18:42,119 So are there really industrial sites out there that still 267 00:18:42,160 --> 00:18:44,480 need to hear that you shouldn't be making an Internet 268 00:18:44,480 --> 00:18:46,359 connection from your critical systems? 269 00:18:47,160 --> 00:18:53,519 Speaker 3: Short answers, Yes, you know, people who do security assessments, 270 00:18:53,559 --> 00:18:55,640 you know, not just incident response that we're talking here, 271 00:18:55,680 --> 00:19:00,319 but security assessments come back and say they regularly find 272 00:19:00,920 --> 00:19:05,759 connections out to the IT network and occasionally straight out 273 00:19:05,799 --> 00:19:08,680 to the Internet. The connections to the IT network tend 274 00:19:08,680 --> 00:19:13,319 to have been deployed by the the you know, the 275 00:19:13,359 --> 00:19:16,519 engineering team or the IT team to make their lives easier. 276 00:19:17,720 --> 00:19:20,920 You know, people with gray hair enough gray hair like me, 277 00:19:21,079 --> 00:19:24,200 they talk about, you know, how the systems used to 278 00:19:24,240 --> 00:19:27,079 be air gapped. This was a very long time ago. 279 00:19:27,119 --> 00:19:29,559 We're talking thirty forty years. The systems used to be 280 00:19:29,559 --> 00:19:33,839 air gaped, and you know, people with gray hair like 281 00:19:33,920 --> 00:19:37,039 me might assume that's still the case. It's not. You know, 282 00:19:37,119 --> 00:19:42,359 everybody who does audits reports these connections. The really disturbing stuff, 283 00:19:42,519 --> 00:19:45,880 you know, yes, it's it's disturbing that there are connections 284 00:19:45,880 --> 00:19:48,440 to the IT network that are poorly secured, that, you know, 285 00:19:48,799 --> 00:19:51,720 But the really disturbing stuff is the vendors going in 286 00:19:51,880 --> 00:19:54,279 and if you do an audit on a site time 287 00:19:54,359 --> 00:19:57,720 and time again, I hear people saying, yeah, they discovered 288 00:19:57,799 --> 00:20:01,559 three different Internet connections. The vendors stuck in there, and 289 00:20:01,599 --> 00:20:05,079 you're going, well, wouldn't you notice if there was a 290 00:20:05,119 --> 00:20:08,079 new Internet connection. I mean, no internet service provider gives 291 00:20:08,079 --> 00:20:10,720 you a connection for free. You got to run wires, 292 00:20:10,759 --> 00:20:12,799 you got to You got to pay for this thing. 293 00:20:12,839 --> 00:20:14,400 Every month. It's showing up on your bill. 294 00:20:14,440 --> 00:20:14,920 Speaker 1: No, it's not. 295 00:20:16,119 --> 00:20:18,359 Speaker 3: You know, there's a lot of wires being run while 296 00:20:18,440 --> 00:20:20,880 stuff is being deployed. You don't notice a new wire. 297 00:20:21,440 --> 00:20:25,480 And the vendors pay month after month for the Internet connection. 298 00:20:25,680 --> 00:20:27,920 Doesn't even show up on the bill of the owner 299 00:20:27,960 --> 00:20:32,279 and operator. Because the vendors are providing a remote management 300 00:20:32,400 --> 00:20:35,880 or vote maintenance service and they want to minimize their costs, 301 00:20:35,880 --> 00:20:38,799 they want to maximize their convenience in terms of getting 302 00:20:38,799 --> 00:20:42,200 into the site, so they deploy rogue DSL routers, they 303 00:20:42,200 --> 00:20:47,119 deploy rogue firewalls to the site's Internet connection. They deploy 304 00:20:48,640 --> 00:20:52,920 they might deploy rogue cellular access points where you know, 305 00:20:52,960 --> 00:20:55,359 there's not even wires to run. It's just a box 306 00:20:55,519 --> 00:20:57,960 sitting there that has a label on it saying, you 307 00:20:58,000 --> 00:21:01,119 know important, do not remove. And of course God makes 308 00:21:01,160 --> 00:21:03,799 it invisible to everybody who's looking at it says, oh what, 309 00:21:04,039 --> 00:21:09,200 don't touch that one. Yes, it's very common. The advice 310 00:21:09,359 --> 00:21:12,680 I try to give people is when you do like 311 00:21:13,119 --> 00:21:16,559 a risk assessment or a walk through or an audit 312 00:21:16,640 --> 00:21:22,200 of your site look for these rogue connections. Unfortunately, you're 313 00:21:22,279 --> 00:21:26,279 probably going to find one or two of these contractual 314 00:21:26,319 --> 00:21:32,920 penalties with the vendor help, but they're no guarantee. You 315 00:21:33,000 --> 00:21:39,039 said that the victim decided not to pay the ransom. 316 00:21:40,319 --> 00:21:44,960 You know, do you see victims ever paying the ransom 317 00:21:45,440 --> 00:21:49,000 to recover an OT network, to recover the HMI, to 318 00:21:49,079 --> 00:21:52,039 recover worse than that, the PLCs and the safety systems. 319 00:21:52,480 --> 00:21:57,480 You know, why would does anyone trust a criminal to 320 00:21:57,640 --> 00:22:00,359 take the tool the criminal provides and light it on 321 00:22:00,400 --> 00:22:04,880 their safety system and you know, restore it because they 322 00:22:04,880 --> 00:22:07,279 trust the criminal. Does anyone trust the criminal that far 323 00:22:07,279 --> 00:22:08,200 as that does that happen? 324 00:22:09,240 --> 00:22:16,400 Speaker 1: We have seen traditional I T systems where they pay 325 00:22:16,519 --> 00:22:23,039 the ransom and get access back to these systems and 326 00:22:23,119 --> 00:22:30,400 some that are OT adjacent such as colonial pipeline and hospitals. Right, 327 00:22:31,519 --> 00:22:35,240 we do know that those systems and both of those 328 00:22:35,359 --> 00:22:39,359 incidents or well those examples the are colonnal pipeline are 329 00:22:40,240 --> 00:22:44,039 name a hospital breach. In some cases there were OTS 330 00:22:44,319 --> 00:22:50,680 type data OT type critical information that was impacted and 331 00:22:50,759 --> 00:22:55,440 so they of course paid and due to the fact 332 00:22:55,440 --> 00:23:03,240 that there's if someone didn't trust these ransomware gangs to 333 00:23:03,400 --> 00:23:05,000 do what they say they were going to do, they 334 00:23:05,000 --> 00:23:08,440 would those ransomware games would be out of business. So 335 00:23:09,319 --> 00:23:13,960 if you pay them, they don't, they decryptor doesn't work, 336 00:23:15,000 --> 00:23:19,119 then it's no good in their ransomware gain job is 337 00:23:19,160 --> 00:23:25,359 over with at least for this thing. But for OT 338 00:23:26,440 --> 00:23:30,240 in this instance I talked about, they did not pay. 339 00:23:31,880 --> 00:23:36,839 I don't have enough data to know if OT direct 340 00:23:37,160 --> 00:23:42,440 on the control systems themselves, the Windows, HMI s, engineering workstations, 341 00:23:42,519 --> 00:23:48,400 DCS servers, SKATAS servers, if they've paid in those instances. 342 00:23:50,279 --> 00:23:55,160 But I'd say it's it's plausible, And it really comes 343 00:23:55,200 --> 00:23:59,160 down to the business decision of the plant owner, the 344 00:23:59,200 --> 00:24:06,200 CEO of the cut based on what the engineers at 345 00:24:06,240 --> 00:24:09,279 the lower level, Hey, can we get do we have backups? 346 00:24:09,319 --> 00:24:12,880 Can we get the vendors to come in? And so 347 00:24:13,160 --> 00:24:16,519 I really don't have enough information to say about do 348 00:24:16,680 --> 00:24:23,000 OT asset owners like a plant, like a site, like 349 00:24:23,079 --> 00:24:29,559 a you know that directly operates h O T, if 350 00:24:29,599 --> 00:24:34,839 they trust these ransomware games or not. It may just 351 00:24:34,960 --> 00:24:41,440 roll up higher than them about that. Uh. Also, there's 352 00:24:41,680 --> 00:24:48,880 usually an advice from a ransomware negotiator that's a third 353 00:24:48,880 --> 00:24:54,559 party that specializes in negotiating with ransom so they may 354 00:24:54,720 --> 00:24:58,359 advise to pay or not to pay, or to get 355 00:24:58,359 --> 00:25:02,759 a reduced pain as well. So it's very, very complicated. 356 00:25:03,559 --> 00:25:09,200 I know I didn't answer your question directly, but in 357 00:25:09,359 --> 00:25:13,240 the instances we've seen, we have seen them not pay 358 00:25:13,319 --> 00:25:17,680 and we have seen them pay and what's OT or not. 359 00:25:18,279 --> 00:25:21,880 Speaker 3: So so coming back to our theme here, you know, 360 00:25:22,000 --> 00:25:25,680 lessons from incidents. You know, the lesson from this incident is, 361 00:25:26,039 --> 00:25:29,680 you know, get rid of that firewall, use the existing infrastructure, 362 00:25:30,119 --> 00:25:33,519 and you know, look at your backups. I mean, if 363 00:25:33,559 --> 00:25:35,839 the backups are encrypted, it's it's all over. That makes 364 00:25:35,839 --> 00:25:39,240 perfect sense. What else have you got? What else you 365 00:25:39,240 --> 00:25:41,640 know have you been running into lately? That that's that's 366 00:25:41,680 --> 00:25:42,720 interesting and noteworthy. 367 00:25:43,119 --> 00:25:46,440 Speaker 1: Yeah, I mean it's basically boths down to either ransomware 368 00:25:46,519 --> 00:25:51,160 or commodity malware. So I've got I've got another example 369 00:25:51,200 --> 00:25:56,000 about ransomware. Electric utility was impacted by ransomware on the 370 00:25:56,039 --> 00:26:00,319 IT side, but they had a good incident response plan 371 00:26:00,960 --> 00:26:07,640 and they severed the IT and OT connections there and 372 00:26:08,839 --> 00:26:13,200 even down to the power plant type networks, and so 373 00:26:13,319 --> 00:26:17,160 that was really amazing, and so that's a good story, 374 00:26:17,720 --> 00:26:22,680 and we were able to actually verify the IT team 375 00:26:22,720 --> 00:26:25,599 were able to verify that the threat actor, the ransomware 376 00:26:25,640 --> 00:26:31,200 game Quantum, were scanning the OT DMZ, but they didn't 377 00:26:31,200 --> 00:26:34,839 get a chance to get be let through. We did 378 00:26:34,920 --> 00:26:40,880 do a full assessment of their DMZ and looked at 379 00:26:40,960 --> 00:26:46,519 the domain control of the firewalls and even the domain 380 00:26:46,559 --> 00:26:51,480 controller and the firewalls and others down inside the OT networks, 381 00:26:52,359 --> 00:26:54,920 and we found that that they were actually pretty lucky 382 00:26:54,960 --> 00:26:58,400 because they had some weakness in some of the firewalls. 383 00:26:58,400 --> 00:27:01,960 So eventually do if they had enough time, if the 384 00:27:02,119 --> 00:27:05,440 ransomware actor had persisted long enough, they could have gotten 385 00:27:05,440 --> 00:27:09,119 through that firewall, made it to the DMZ And the 386 00:27:09,160 --> 00:27:12,960 active directory had some weaknesses as well, and they could 387 00:27:13,000 --> 00:27:18,359 have gotten domain access domain admin and pivoted to the 388 00:27:18,400 --> 00:27:22,920 OT network. But the great thing is highlight again that 389 00:27:23,160 --> 00:27:25,720 they had a good instant response plan. They were able 390 00:27:25,759 --> 00:27:29,279 to segment quickly and then they were able to have 391 00:27:29,680 --> 00:27:34,640 their OT vendor in this case it was Emerson Ovation 392 00:27:34,880 --> 00:27:39,920 were able to go on site and they were able 393 00:27:40,000 --> 00:27:43,519 to not only take the IOCs that we had from 394 00:27:43,519 --> 00:27:48,759 the ransomware, but they were able to sweep because it 395 00:27:48,799 --> 00:27:52,000 was their contract to do so, to look in the 396 00:27:52,000 --> 00:27:56,759 POC logs, the OT workstations, endpoint protection and all that stuff. 397 00:27:57,119 --> 00:27:59,920 So we all worked in concert together in this incident. 398 00:28:01,119 --> 00:28:05,119 And then actually they hardened the firewalls, hardened the domain controllers, 399 00:28:05,200 --> 00:28:11,799 hardened the workstation configurations before doing anything else. They did 400 00:28:11,839 --> 00:28:18,000 all of that. When the ransomware was eradicated and hardened, 401 00:28:19,559 --> 00:28:22,440 then they said, okay, now we'll reconnect everything back the 402 00:28:22,440 --> 00:28:25,640 way it was. So that was a really great lesson 403 00:28:25,720 --> 00:28:30,640 learned with another ransomware, and it wasn't a direct impact 404 00:28:30,640 --> 00:28:37,160 to OT, but this is a great opportunity to leverage 405 00:28:37,200 --> 00:28:40,400 that incident response plan that they had. 406 00:28:43,440 --> 00:28:49,119 Speaker 3: So nate the concept of separating it from OT networks 407 00:28:49,160 --> 00:28:52,920 in an emergency. This is the concept that I see increasingly. 408 00:28:53,839 --> 00:28:56,559 I mean, I think we've reported on the show here 409 00:28:56,680 --> 00:29:00,559 a few times that this is what the TSAD demands 410 00:29:00,599 --> 00:29:05,160 of pipeline operators, petrochemical pipeline operators ever since colonial the 411 00:29:05,240 --> 00:29:08,000 ability to separate the networks in an emergency so that 412 00:29:08,079 --> 00:29:10,799 you can keep the pipeline running wild it's being cleaned up. 413 00:29:12,359 --> 00:29:18,200 I haven't I'm told I haven't actually read the translated 414 00:29:18,200 --> 00:29:20,680 the Danish law, but apparently in Denmark there's a recent 415 00:29:20,920 --> 00:29:24,720 law in the last twelve months saying exactly the same thing. 416 00:29:25,000 --> 00:29:27,559 You know, the TSA applies to pipelines and rails, in 417 00:29:27,640 --> 00:29:31,440 Denmarket applies to critical infrastructure, and it says, in an emergency, 418 00:29:31,480 --> 00:29:33,119 you have to be able to separate. They call it 419 00:29:33,240 --> 00:29:39,039 islanding the industrial control network, and as Chris points out, 420 00:29:39,119 --> 00:29:45,920 it can be effective, but it relies on really rapid 421 00:29:46,279 --> 00:29:50,359 intrusion detection and rapid response because as Chris said, you know, 422 00:29:50,480 --> 00:29:53,839 the bad guys had been testing the OT firewall. If 423 00:29:53,839 --> 00:29:55,839 they had had just a little bit longer, they could 424 00:29:55,839 --> 00:30:00,400 have gone through. So, you know it, even though it 425 00:30:00,400 --> 00:30:05,119 it's you know, imperfect, it is a measure that I'm 426 00:30:05,200 --> 00:30:10,119 seeing increasingly required of critical infrastructure operators and you know, 427 00:30:10,240 --> 00:30:14,079 recommended to non critical operators as a as a measure 428 00:30:13,920 --> 00:30:20,119 that helps, especially on the incident response side. Have you 429 00:30:20,119 --> 00:30:22,039 got another example for us? I mean three is the 430 00:30:22,079 --> 00:30:24,240 magic number you've given us, you know, sort of two 431 00:30:24,319 --> 00:30:26,480 sets of insights. What what else have you got for us? 432 00:30:26,519 --> 00:30:28,079 And in terms of lessons. 433 00:30:27,799 --> 00:30:32,319 Speaker 1: Learned, yeah, there's lessons learned I can just name a 434 00:30:32,319 --> 00:30:36,000 few other lessons learned from it, just about any attack, right, 435 00:30:37,119 --> 00:30:41,759 making sure that you have these at least Windows systems 436 00:30:41,799 --> 00:30:44,839 with antivise. In a lot of cases the OT network 437 00:30:44,839 --> 00:30:49,960 didn't have anti just basic antivirse, not necessarily an agent 438 00:30:50,160 --> 00:30:54,839 or EDR solutions. If you have those, great. If you 439 00:30:54,920 --> 00:30:58,720 don't have any antivirse, that's you need to get at 440 00:30:58,799 --> 00:31:04,079 least corner version of Windows or operating system and with antivirus, 441 00:31:05,599 --> 00:31:10,799 having good backups, having good that good vendor support. Now, 442 00:31:11,880 --> 00:31:17,279 this last incident we responded to was using a living 443 00:31:17,319 --> 00:31:21,599 off the land attack, So we responded to electric utility 444 00:31:21,799 --> 00:31:28,720 in Ukraine in twenty twenty two and it was a 445 00:31:28,759 --> 00:31:33,559 distribution utility that the attacker came in through the IT 446 00:31:33,880 --> 00:31:41,359 network deployed their typical wiper malware. This was the group 447 00:31:41,480 --> 00:31:45,400 APT forty four or Sandworm team, which has been targeting 448 00:31:45,400 --> 00:31:50,240 critical infrastructure around the world for quite a while, and 449 00:31:50,440 --> 00:31:55,440 they were able to pivot to the Skater system and 450 00:31:56,279 --> 00:32:02,960 used the feature of the Skater system to trip breakers 451 00:32:04,319 --> 00:32:07,000 using a tool that was built in the skate of 452 00:32:07,079 --> 00:32:10,519 system itself, So just giving it a list of breakers 453 00:32:10,519 --> 00:32:16,920 to trip and calling that executable in the system to 454 00:32:17,359 --> 00:32:23,440 trip those breakers on. Behalf of the attackers is the 455 00:32:23,799 --> 00:32:32,160 too long did read of that incident? And so the 456 00:32:32,279 --> 00:32:38,000 lesson learned here is targeted attacks. They're going to not 457 00:32:38,160 --> 00:32:42,039 use malware. They're going to use the features or the 458 00:32:42,240 --> 00:32:46,880 inherent vulnerabilities in an OT network stealing valid credentials like 459 00:32:46,920 --> 00:32:54,359 an operator workstation or an engineering administrator account. And if 460 00:32:54,400 --> 00:33:01,279 you can even spearfish an engineer or administrator network AM 461 00:33:01,519 --> 00:33:04,960 on the IT network and you don't have good segmentation 462 00:33:05,119 --> 00:33:09,759 of roles from IT to O T, then that's that 463 00:33:10,680 --> 00:33:13,880 attacker is going to use every one of those tools 464 00:33:14,640 --> 00:33:22,359 to evade detection, to bypass your normal detections because they're 465 00:33:22,559 --> 00:33:27,240 they're coming in as a valid user. So the lessons 466 00:33:27,319 --> 00:33:34,680 learned there is to limit the amount of administrative access. 467 00:33:34,960 --> 00:33:38,440 And this is you know, role based authentication, right and 468 00:33:39,000 --> 00:33:41,680 does the person that got promoted and now is in 469 00:33:41,680 --> 00:33:45,640 a different department, does he still need ADMIN rights? You know, 470 00:33:46,480 --> 00:33:51,039 does this person having enough control for just their area 471 00:33:51,119 --> 00:33:55,119 only or their responsibilities too wide? And now we say, okay, 472 00:33:55,119 --> 00:34:00,720 we need to reduce the amount of admin. Do we 473 00:34:00,759 --> 00:34:05,839 require two factor authentication or even hardware two factor authentication 474 00:34:06,079 --> 00:34:11,760 to really reduce the attacker down to an insider threat, 475 00:34:12,440 --> 00:34:16,360 because remotely that's very hard to do to to to 476 00:34:16,360 --> 00:34:24,199 to bypass hardware token based two factor authentication. And so 477 00:34:24,280 --> 00:34:28,360 there's some there's some living off the land guides out there. 478 00:34:28,679 --> 00:34:33,519 The US government DEWE has put out a threat hunting 479 00:34:33,559 --> 00:34:38,599 guy for living off the land attacks after the volt 480 00:34:38,639 --> 00:34:45,480 typhoon announcements last year. But I would also go as 481 00:34:45,559 --> 00:34:53,199 step above and beyond that, learning good ways to detect 482 00:34:53,440 --> 00:34:58,480 anomalous logins, even from your own folks. If it's out 483 00:34:58,519 --> 00:35:04,000 of a normal time, out of a normal location, you're 484 00:35:04,039 --> 00:35:06,519 really gonna have to have some tuning on some of 485 00:35:06,559 --> 00:35:11,719 these detections. And the only way to really test those 486 00:35:11,800 --> 00:35:15,639 is with a red team that's trying to be quiet 487 00:35:15,679 --> 00:35:23,400 and not trigger your detections. And that is some of 488 00:35:23,440 --> 00:35:29,199 the more advanced asset owners and end users they're using 489 00:35:29,559 --> 00:35:33,679 leveraging red teams, hiring red teams like what we do 490 00:35:33,719 --> 00:35:36,320 at Mandiant to come in and see if we can 491 00:35:36,639 --> 00:35:39,480 do living off the land attacks to bypass their detections. 492 00:35:42,639 --> 00:35:46,400 Speaker 2: Since Chris mentioned it but moved on before we could 493 00:35:46,400 --> 00:35:49,440 actually define it, let me just for listeners, living off 494 00:35:49,480 --> 00:35:53,920 the land is the process by which an attacker, rather 495 00:35:53,960 --> 00:35:58,960 than using their own malicious tooling, would make use of 496 00:35:59,280 --> 00:36:04,280 legitimate software or functionality of the system they're attacking to 497 00:36:04,400 --> 00:36:09,840 perform malicious actions on it. It's been a growing trend 498 00:36:10,639 --> 00:36:15,320 in recent years, I believe, because it's so effective in 499 00:36:15,360 --> 00:36:20,039 that it is so difficult to detect. You know, you 500 00:36:20,039 --> 00:36:23,280 could spot malware with certain kinds of tools, but can 501 00:36:23,280 --> 00:36:27,519 you spot somebody doing things with legitimate aspects of Windows 502 00:36:27,599 --> 00:36:31,199 or whatever you might be using. It sounds though like 503 00:36:31,760 --> 00:36:36,840 Chris is talking about detecting living off the land tactics, 504 00:36:36,840 --> 00:36:38,280 which seems difficult to Andrew. 505 00:36:38,880 --> 00:36:41,079 Speaker 3: That's right, I mean, I you know, have been have 506 00:36:41,159 --> 00:36:45,320 been following Living off the land to a degree. You 507 00:36:45,360 --> 00:36:49,679 know the what's the right where the short The short 508 00:36:49,679 --> 00:36:52,599 answer is you run an anti virus scan on a 509 00:36:52,599 --> 00:36:54,800 machine that's been compromised by a living off the land 510 00:36:55,199 --> 00:36:59,400 attack and it comes up squeaky clean, there's nothing nasty 511 00:36:59,440 --> 00:37:02,360 on the machine. And what I heard Chris say is 512 00:37:02,400 --> 00:37:08,000 that this is because the bad guys are using normal mechanisms, 513 00:37:08,079 --> 00:37:12,599 especially remote access, to log into these systems as if 514 00:37:12,639 --> 00:37:16,440 they were normal users and use the tools on the 515 00:37:16,480 --> 00:37:20,320 machine to attack the network or you know, to wait 516 00:37:20,400 --> 00:37:22,920 for a period of time until it's opportune, and then 517 00:37:22,960 --> 00:37:27,440 attack the network. And you know what I heard him 518 00:37:27,440 --> 00:37:31,079 say is that because it's a lot of a lot 519 00:37:31,119 --> 00:37:34,679 of this is remote access, he says, you can detect 520 00:37:34,679 --> 00:37:39,280 this by focusing hard on your remote access system. A. 521 00:37:39,440 --> 00:37:41,800 You can prevent it by throwing in some hardware based 522 00:37:42,000 --> 00:37:44,159 two factor. You know, that will solve a lot of 523 00:37:44,199 --> 00:37:46,800 the problem, not necessarily all of it. There's always vulnerabilities 524 00:37:46,800 --> 00:37:50,360 and zero days, but two factor helps enormously. It's way 525 00:37:50,360 --> 00:37:53,880 better than not having two factor. But that's preventive. On 526 00:37:53,920 --> 00:37:57,320 the detective side, he said, pay attention to your remote 527 00:37:57,360 --> 00:38:03,719 access If normal users are logging in at strange times, 528 00:38:04,639 --> 00:38:07,960 that should raise a red flag. If normal users are 529 00:38:08,199 --> 00:38:11,119 logging in from strange places, the IP address coming in 530 00:38:11,199 --> 00:38:14,559 is from China, Well is Fred in China this week? 531 00:38:14,719 --> 00:38:18,400 No he's not. So you know what I heard was 532 00:38:18,440 --> 00:38:22,639 one way to to you know, help detect living off 533 00:38:22,639 --> 00:38:25,960 the land techniques is to pay close attention in your 534 00:38:25,960 --> 00:38:30,880 intrusion detection system to the intelligence that you're getting about 535 00:38:31,119 --> 00:38:37,079 remote users logging in. So one more question, you know, 536 00:38:37,119 --> 00:38:38,519 we talked to a lot of folks on the on 537 00:38:38,559 --> 00:38:40,880 the podcast, A lot of them are our vendors with 538 00:38:41,239 --> 00:38:44,519 technology that we talk about, and you know, sort of 539 00:38:44,559 --> 00:38:47,719 a consistent theme for most of these vendors most of 540 00:38:47,760 --> 00:38:54,840 these technologies is operational benefits. Yes, the technology whatever it is, 541 00:38:54,840 --> 00:38:59,760 is helping with cybersecurity, but often this stuff helps with 542 00:39:00,440 --> 00:39:05,440 just general operations in sometimes surprising ways. We've been talking 543 00:39:05,440 --> 00:39:08,679 about incidents and lessons, and you know, a lot of 544 00:39:08,719 --> 00:39:13,239 what you do is incident response. Are there operational benefits 545 00:39:13,000 --> 00:39:15,719 that you run into that people say, you know, I 546 00:39:15,800 --> 00:39:18,800 did what you told me and everything is working smooth 547 00:39:19,039 --> 00:39:21,440 more smoothly than not just on the security side. Can 548 00:39:21,480 --> 00:39:23,880 you talk you do you have anything like that for us? 549 00:39:24,360 --> 00:39:27,480 Speaker 1: Oh? Absolutely. And one of the things that I always 550 00:39:27,719 --> 00:39:31,440 teut is looking at your network, looking at the packet 551 00:39:31,480 --> 00:39:37,679 captures in the network can aid in not just cybersecurity benefits, 552 00:39:37,719 --> 00:39:41,760 but these operational benefits. You can see things like switch 553 00:39:41,800 --> 00:39:48,199 failures happening, TCP retransmissions happening. All this traffic may be 554 00:39:48,320 --> 00:39:51,880 went like your Windows hmis maybe trying to reach out 555 00:39:51,880 --> 00:39:55,280 to Windows Update, but it's blocked by the it OT 556 00:39:55,440 --> 00:40:00,440 firewall or anything else. It may not have a connection 557 00:40:00,519 --> 00:40:03,480 at all as trying to reach out all this unnecessary 558 00:40:03,599 --> 00:40:12,159 traffic or indications of proper configurations, misconfigurations and things like that. 559 00:40:12,440 --> 00:40:15,000 So just looking at your network with some of these 560 00:40:15,039 --> 00:40:19,079 tools that are out there, free tools, pay tools, ICs 561 00:40:19,079 --> 00:40:22,559 specific tools, or IT specific tools. It doesn't matter if 562 00:40:22,559 --> 00:40:24,360 you look at If you take any one of those, 563 00:40:24,400 --> 00:40:28,079 say just even wireshark and look in your OT network, 564 00:40:30,239 --> 00:40:34,920 you can get an idea on what traffic doesn't need 565 00:40:34,960 --> 00:40:39,159 to be there that you can eliminate it, make your 566 00:40:39,400 --> 00:40:44,440 improvements to the system, and now I have better visibility 567 00:40:44,519 --> 00:40:49,079 if there is an incident. I can easier to detect 568 00:40:49,119 --> 00:40:52,719 if there's a cyber incident or if something's operationally wrong 569 00:40:54,199 --> 00:40:57,960 like a switch failure or something, and so there's a 570 00:40:58,039 --> 00:41:02,760 really great benefit there. Also helps improve reliability. We've done 571 00:41:02,840 --> 00:41:11,000 an assessment at a company that had a conveyor belt 572 00:41:12,039 --> 00:41:14,920 that they were having problems with. If the conveyor belt 573 00:41:15,000 --> 00:41:17,320 wasn't timed exactly right, if they had too much latency 574 00:41:17,320 --> 00:41:20,639 on the network, the conveyor belt would stop and all 575 00:41:20,639 --> 00:41:23,400 the things on the conveyor belt would just go everywhere, 576 00:41:23,440 --> 00:41:25,880 and it was a disaster. So we just looked in 577 00:41:25,960 --> 00:41:29,159 the network, Oh, you got all these TCP re transmissions 578 00:41:29,239 --> 00:41:32,480 is and you look at the map in the in 579 00:41:32,559 --> 00:41:34,559 the software and say, oh, it's coming from these two 580 00:41:34,599 --> 00:41:38,840 IP addresses. Oh, we know what those equipment is. And 581 00:41:38,880 --> 00:41:40,920 we had the network person coming, Oh, I've been trying 582 00:41:40,960 --> 00:41:44,639 to figure this out for weeks and just looking looking, 583 00:41:44,840 --> 00:41:47,760 just using a duel like that. Uh, they were able 584 00:41:47,800 --> 00:41:50,199 to find and fix the problem and they fixed their 585 00:41:50,280 --> 00:41:56,639 latency issue because of that. So going back to you know, 586 00:41:57,000 --> 00:42:04,079 incident response, having these having an incant response plan. A 587 00:42:04,119 --> 00:42:13,400 lot of ot already has this because of disasters, fire, floods, storms, 588 00:42:16,320 --> 00:42:23,800 it spills, air releases, safety issues, and that's all part 589 00:42:23,800 --> 00:42:29,880 of their normal disaster recovery or incident response plans. If 590 00:42:29,920 --> 00:42:32,199 you already have one of those, you've already done ninety 591 00:42:32,239 --> 00:42:35,559 percent of the work to have a cyber incident response plan. 592 00:42:36,400 --> 00:42:40,960 You just now have a cyber incident response added to that. 593 00:42:41,199 --> 00:42:44,039 So that's the whole premise behind things like ICs or 594 00:42:44,199 --> 00:42:50,679 ICs incident Command system for industrial control systems and having 595 00:42:50,719 --> 00:42:56,840 that say, chief person in charge of cybersecurity for a 596 00:42:57,000 --> 00:42:59,679 site for a paper mill, for a power plant, for 597 00:43:00,559 --> 00:43:04,519 manufacturing facility, even though that's not your day to day 598 00:43:04,599 --> 00:43:08,599 job all the time. If you say the lead that 599 00:43:09,679 --> 00:43:14,440 and you say you have multiple plants, have multiple leads 600 00:43:14,800 --> 00:43:21,159 for those plants. That still the every decision will go 601 00:43:21,239 --> 00:43:24,079 through the plant manager, the general manager of the plant 602 00:43:24,559 --> 00:43:28,440 or site. But at least you have someone that is 603 00:43:28,480 --> 00:43:32,000 in charge of cybersecurity, just like you have a designated 604 00:43:32,000 --> 00:43:39,480 firewatch person or anything else. So if you take safety 605 00:43:39,519 --> 00:43:42,639 culture that we've known about for over one hundred years 606 00:43:43,840 --> 00:43:49,599 and mold your cybersecurity culture to fit with that, things 607 00:43:49,639 --> 00:43:53,119 will make a lot more sense. We've already we've already 608 00:43:53,119 --> 00:43:57,000 invented this. We're not reinventing the wheel here now. We're 609 00:43:57,039 --> 00:44:02,519 just including another paradigm of cyber security, network security, and 610 00:44:02,679 --> 00:44:07,000 endpoint security into these things that we have been doing. 611 00:44:08,199 --> 00:44:11,960 There's a fire, okay, let's put out incident response. So 612 00:44:16,199 --> 00:44:19,119 and if you have a plan, that's great. If you 613 00:44:19,119 --> 00:44:24,039 don't have a plan and you run around, that's not good. 614 00:44:24,280 --> 00:44:26,320 So if you have a plan, you can at least 615 00:44:26,360 --> 00:44:30,079 prepare for it, and sometimes that's the win. Being prepared 616 00:44:30,880 --> 00:44:32,239 is better than not being prepared. 617 00:44:33,280 --> 00:44:35,719 Speaker 3: Well, this has been tremendous. Thank you Chris for joining us. 618 00:44:35,960 --> 00:44:37,760 Before we let you go, can you sum up for 619 00:44:37,760 --> 00:44:39,840 our listeners what you know? What are the key points 620 00:44:39,840 --> 00:44:40,840 we should take away here. 621 00:44:41,639 --> 00:44:44,239 Speaker 1: Sure if you didn't listen to a single thing I said, 622 00:44:45,280 --> 00:44:51,880 and you can listen to these three things. Collaborate, plan 623 00:44:52,039 --> 00:44:56,760 and practice. So collaborate. You get your IT teams, talking 624 00:44:56,800 --> 00:45:02,400 to your OT teams, talking to your manufacturers, and identify 625 00:45:02,480 --> 00:45:05,920 the right roles within each of those and make sure 626 00:45:06,039 --> 00:45:08,960 you get together and talk about these things. Have some 627 00:45:09,079 --> 00:45:14,559 donuts and coffee. So collaborating. Knowing who is in charge 628 00:45:14,559 --> 00:45:18,159 of what is half the battle. Knowing who to call 629 00:45:18,199 --> 00:45:24,719 when plan, having an incent response plan, or including OT 630 00:45:24,920 --> 00:45:29,079 security in your incident response plan and or engineering procedures. 631 00:45:30,000 --> 00:45:36,960 That's gonna help when an incident impacts OT directly or indirectly. 632 00:45:38,280 --> 00:45:43,320 And then practice even can start with a simple question, Hey, 633 00:45:44,079 --> 00:45:47,320 what would we do in an incident or even going 634 00:45:47,440 --> 00:45:53,440 to having a tabletop exercise collecting logs from a PLC 635 00:45:53,800 --> 00:45:56,880 of security logs? How long does that take? How many 636 00:45:56,880 --> 00:46:00,119 devices do we have? If the general manager says how 637 00:46:00,119 --> 00:46:01,480 long is this going to take to pull all the 638 00:46:01,519 --> 00:46:04,719 logs from all of our systems, you won't be able 639 00:46:04,719 --> 00:46:07,360 to say I don't know. You'll have to say no, 640 00:46:07,440 --> 00:46:11,599 this will take two hours and forty five minutes. Because 641 00:46:11,599 --> 00:46:18,719 we've tested it so collaborate plan and practice. If you 642 00:46:18,920 --> 00:46:22,880 need help with OT security or IT security, we do that. 643 00:46:22,960 --> 00:46:29,079 At Mandian. We offer incidant response retainer that covers IT 644 00:46:29,360 --> 00:46:31,760 and OT. There's no separate retainer. If you have an 645 00:46:31,800 --> 00:46:35,559 IT incident and don't need OT, not a problem. If 646 00:46:35,599 --> 00:46:38,639 you have OT only incident, not a problem. If it's 647 00:46:39,000 --> 00:46:42,800 IT cloud and OT all at the same time, we 648 00:46:42,880 --> 00:46:45,719 can help you around the world twenty four to seven. 649 00:46:47,400 --> 00:46:50,239 And lastly, if you want to learn more about this, 650 00:46:50,400 --> 00:46:54,000 you can reach out to me Chris's drunk at Google 651 00:46:54,239 --> 00:47:00,679 dot com, my email LinkedIn social media blue Sky, and 652 00:47:01,079 --> 00:47:03,679 we check out some of our blogs on the Google 653 00:47:03,719 --> 00:47:09,360 Cloud or Mandian security blog. We have great content out 654 00:47:09,400 --> 00:47:16,239 there that is actual actionable not marketing fluff. It is 655 00:47:16,360 --> 00:47:23,079 actual actionable reports. The next M Trends Report is coming 656 00:47:23,119 --> 00:47:27,480 out next week r s a time prime end of April, 657 00:47:28,000 --> 00:47:31,760 so that's a free report. It's a great report to 658 00:47:32,119 --> 00:47:35,119 look at and gain some insights on what we've been 659 00:47:35,159 --> 00:47:38,920 responding to over the last year. And with that, I 660 00:47:39,159 --> 00:47:45,280 appreciate it. Collaborate plan practice, Andrew. 661 00:47:45,320 --> 00:47:49,440 Speaker 2: That just about concludes your interview with Chris Sistroke. Do 662 00:47:49,480 --> 00:47:51,800 you have any final words to add on to his 663 00:47:52,039 --> 00:47:53,960 to take us out with today. 664 00:47:53,559 --> 00:47:57,360 Speaker 3: I mean Chris, Chris summed up collaborate plan practice. You know, 665 00:47:57,760 --> 00:48:00,880 what I heard, especially earlier in the interview, was, you know, 666 00:48:01,559 --> 00:48:05,440 do the basics, guys. They some people call it basic 667 00:48:05,519 --> 00:48:08,559 hygien is. Basically do on an OT network as much 668 00:48:08,599 --> 00:48:10,320 as you can of what you would do on an 669 00:48:10,360 --> 00:48:13,360 IT network. Put a little anti virus in on the 670 00:48:13,360 --> 00:48:17,880 systems that tolerated, you know, get some backups, get some 671 00:48:18,039 --> 00:48:20,679 offsite backups so that if the bad guys get in 672 00:48:20,719 --> 00:48:23,639 they can't encrypt the off site backups. There's somewhere else, 673 00:48:25,880 --> 00:48:31,880 you know, look for the vendors leaving behind internet connections, 674 00:48:32,639 --> 00:48:36,119 get rid of them, you know. And you know, on 675 00:48:36,159 --> 00:48:38,400 the in terms of living off the land, you know, 676 00:48:38,440 --> 00:48:42,800 he gave some very concrete advice that I've never heard before, saying, look, 677 00:48:43,239 --> 00:48:46,280 these people are coming in as users. Get two factor. 678 00:48:46,360 --> 00:48:49,559 Two factor will do a lot to breaking up living 679 00:48:49,559 --> 00:48:52,519 off the land attacks. And in your intrusion detection systems, 680 00:48:52,559 --> 00:48:55,639 look hard at what your remote users are doing, and 681 00:48:55,639 --> 00:48:58,719 if it seems at all unusual, that's the clue that 682 00:48:58,760 --> 00:49:02,239 you're being a tact And you know, in terms of 683 00:49:02,280 --> 00:49:05,519 his collaborate, plan and practice. I really liked his his 684 00:49:05,519 --> 00:49:12,599 his uh the fire warden analogy, saying, look, you know, 685 00:49:12,639 --> 00:49:16,320 if you have an industrial site that is flammable, okay, 686 00:49:16,840 --> 00:49:20,039 your firewarden does not just sit on her hands until 687 00:49:20,039 --> 00:49:22,639 the place bursts into flames. Okay. The fire warden is 688 00:49:22,679 --> 00:49:26,360 someone who's active in terms of actively h you know, 689 00:49:26,960 --> 00:49:31,079 looking at managing, raising the alarm when they see dangerous 690 00:49:31,119 --> 00:49:34,920 practices in this flammable plant. It's not you know, it's 691 00:49:34,920 --> 00:49:38,639 not just a reactive position. It's also a proactive position. 692 00:49:39,159 --> 00:49:45,960 And we need that for cybersecurity because basically every site is, 693 00:49:46,119 --> 00:49:52,480 you know, in a sense, a flammable cybersecurity situation. So 694 00:49:52,559 --> 00:49:55,079 it's not just that they sit on their hands until 695 00:49:55,079 --> 00:49:57,920 there's an incident and then they're in charge. They are 696 00:49:58,000 --> 00:50:01,079 actively looking around, just like a fire warden wouldn't say, 697 00:50:01,119 --> 00:50:03,760 you know, we shouldn't be doing this. You know, my 698 00:50:03,920 --> 00:50:06,280 job is not just to put the fire out when 699 00:50:06,320 --> 00:50:09,480 it occurs, or coordinate putting the fire out. My job 700 00:50:09,559 --> 00:50:12,159 is to help prevent these things. And so I love 701 00:50:12,239 --> 00:50:15,119 that analogy. That makes so much sense. Anyhow, That's what 702 00:50:15,159 --> 00:50:16,119 I took from the episode. 703 00:50:16,880 --> 00:50:20,000 Speaker 2: Sure well, thank you Chris for speaking with us and 704 00:50:20,000 --> 00:50:22,679 Andrews always thank you for speaking of it. 705 00:50:22,679 --> 00:50:23,960 Speaker 3: It's always a great pleasure. 706 00:50:24,000 --> 00:50:24,280 Speaker 1: Thank you. 707 00:50:25,199 --> 00:50:29,559 Speaker 2: This has been the Industrial Security Podcast from Waterfall. Thanks 708 00:50:29,559 --> 00:50:31,239 to everyone out there listen.