1 00:00:07,240 --> 00:00:12,560 So we started working on defensive measures. How can we automate that, just 2 00:00:12,640 --> 00:00:27,719 as we automate engineering actunities. Welcome, everyone's the Industrial Security Podcast. My 3 00:00:27,800 --> 00:00:31,640 name is Nate Nelson. I'm here as usual with Andrew Ginter, the vice 4 00:00:31,679 --> 00:00:37,119 president of Industrial Security at Waterfall Security Solutions. Who's going to introduce the subject 5 00:00:37,200 --> 00:00:41,200 and guest of our show today. Andrew, how are you? I'm very 6 00:00:41,200 --> 00:00:44,119 well, Thank you, Nate. Our guest today is Yusuf's Jad. He 7 00:00:44,359 --> 00:00:51,320 is the chief technology officer and co founder at Sivault and his topic is active 8 00:00:51,439 --> 00:00:55,399 defense in ot how to make it work? And you know when I heard 9 00:00:55,439 --> 00:01:00,119 about this concept, I was very interested. Common wisdom has it that you 10 00:01:00,280 --> 00:01:06,000 cannot do this on industrial network, yet here's Cybald doing it all right, 11 00:01:06,040 --> 00:01:11,159 then, without further ado, here's your conversation with Yusef. Hello, Yusuf, 12 00:01:11,439 --> 00:01:15,680 welcome to the podcast. Before we get started, can I ask you 13 00:01:15,760 --> 00:01:21,040 to introduce yourself a bit and tell us about the good work that you do 14 00:01:21,079 --> 00:01:25,200 at Cybald. Well, thank you for having me, Andrew. My name 15 00:01:25,239 --> 00:01:30,200 is Yusef jd CTU O T I C S cyber Defense and co founder at 16 00:01:30,200 --> 00:01:37,280 Cybald, which is a division of pm SKADA cyber Defense, Originally an engineering 17 00:01:37,319 --> 00:01:47,560 company. We deployed manufacturing plants and such substations and so forth. So as 18 00:01:47,599 --> 00:01:51,519 part of my role, I'm leading cyber defense operations and novel R and D 19 00:01:51,680 --> 00:02:00,920 products. Have over twenty years experience in osh so IoTs also blockchain and quantum 20 00:02:01,079 --> 00:02:07,240 cyber applications when it's possible, So we're trying to be ahead of the curve 21 00:02:07,520 --> 00:02:15,159 whenever it's it's possible. I'm also course consultants for fourteen ten companies delivered turn 22 00:02:15,199 --> 00:02:21,319 key solutions to multiple organizations, including the HSBI. I've trained military units on 23 00:02:21,400 --> 00:02:28,280 defense and offense exercises UM and to them. Also part of an same or 24 00:02:28,280 --> 00:02:31,639 for ICs. For ICs which you might know, it's is the Incident's Command 25 00:02:31,759 --> 00:02:38,719 system for industrial command control systems. It's a US initiative UM and I was 26 00:02:38,800 --> 00:02:43,479 also a lead on the Want to Cry v too or hansomwhere leading the data 27 00:02:43,560 --> 00:02:50,439 task force globally and as I wold, we serve mostly critical infrastructure including energy, 28 00:02:50,479 --> 00:02:55,199 healthcare, smart series utilities, sometimes defense, and others. I didn't 29 00:02:55,199 --> 00:03:00,879 know you were involved in in uh quantum. We'll have to get you back 30 00:03:00,879 --> 00:03:05,080 on on the show again for for for a different topic. But you know 31 00:03:05,120 --> 00:03:10,039 today's topic is active defense and I mean, you folks have a product that 32 00:03:10,080 --> 00:03:14,520 does this in the OT space. And you know this was a surprise to 33 00:03:14,560 --> 00:03:17,080 me. Um. You know, there's a there's a widespread perception, there's 34 00:03:17,120 --> 00:03:23,280 common knowledge in the industry that you don't do active defense in OT. It's 35 00:03:23,280 --> 00:03:27,680 it's you know, too risky. But you know, before we get into 36 00:03:27,719 --> 00:03:30,680 the the the why and the how, um, and you talk to us 37 00:03:31,039 --> 00:03:35,719 about what you've got in the active decent species you space. You've got a 38 00:03:35,800 --> 00:03:38,479 cyball dome product. What what is it? What does what does that product 39 00:03:38,520 --> 00:03:43,639 do? And then we'll dig into the details. Yeah. Sure. So 40 00:03:44,919 --> 00:03:51,240 a few years ago we looked at the attack landscape and the otics industries, 41 00:03:51,840 --> 00:03:54,479 right, and we saw that there is actually a need for for defending them 42 00:03:54,599 --> 00:04:00,400 against this attack. So we started working on defensive measures. How can we 43 00:04:00,439 --> 00:04:06,280 automate that, just as we automate engineering activities, and we created dome. 44 00:04:06,879 --> 00:04:13,560 Dome came from you know, a dome principle and it's also a very familiar 45 00:04:13,800 --> 00:04:20,199 Israeli irondome. Um. So we focused especially for otics, which was the 46 00:04:20,240 --> 00:04:29,639 sector lacking these kind of defenses, so that it is not too late when 47 00:04:29,680 --> 00:04:34,079 an attack happens or when it is initiated. M So, we created a 48 00:04:34,160 --> 00:04:43,560 hybrid model. It's not a fully automated as anyone might might think. It's 49 00:04:43,600 --> 00:04:47,399 a hybrid model where there is automated incident response actions and there is also manual 50 00:04:48,519 --> 00:04:55,439 incidental response actions, and those we will provide through our DOME platform to our 51 00:04:55,480 --> 00:05:01,199 customers or partners would be operators, owners and such, so that they can 52 00:05:01,240 --> 00:05:08,040 execute those and then we support them along the way. So as a service, 53 00:05:08,560 --> 00:05:15,600 it's industries first managed or co managed Cyber Defense Operations Center or o g 54 00:05:15,759 --> 00:05:20,759 i CS, and it comes with technology that is software. We have appliances 55 00:05:20,800 --> 00:05:28,560 that are for different environment We have ruggodized appliances that take action, provide visibility 56 00:05:29,600 --> 00:05:34,959 into those environments and so forth. So we take care from the visibility standpoints 57 00:05:35,040 --> 00:05:43,399 or stage all the way to defending actively in case of a confirmed attack. 58 00:05:45,600 --> 00:05:51,959 Okay um, and you know where the beginning. Some of your some of 59 00:05:51,959 --> 00:05:57,560 your some some attacks have manual responses, others are automatic. I'm especially interested 60 00:05:57,639 --> 00:06:00,959 in the automatic responses. I mean common with is that you don't do this 61 00:06:00,000 --> 00:06:04,759 because you know years ago, what I remember is people would take intrusion prevention 62 00:06:04,800 --> 00:06:11,040 systems, which were basically firewalls or you know, stuff hanging off the side 63 00:06:11,040 --> 00:06:15,040 of the network looking at packet traces. Basically, if you've got an intrusion 64 00:06:15,399 --> 00:06:17,839 detection system that's looking at the network saying, oh hey, look that's an 65 00:06:17,879 --> 00:06:23,199 attacking progress, and triggers a prevention action, which is usually I don't know, 66 00:06:23,199 --> 00:06:28,120 a TCP reset pack it's saying, turn off that TCP connection that is 67 00:06:28,199 --> 00:06:31,639 launching the attack. It's not there anymore. You wound up with, you 68 00:06:31,639 --> 00:06:38,120 know, a signature update would come through and suddenly half of your connections between 69 00:06:38,160 --> 00:06:43,399 the SKATA system and the PLC would be interpreted as attacks and interrupted and everything 70 00:06:43,439 --> 00:06:47,279 would just break. And so the common wisdom became. You know, you 71 00:06:47,279 --> 00:06:51,319 can do intrusion prevention very high in the architecture, you know, in the 72 00:06:51,360 --> 00:06:58,160 IT network close to the IT network, you can do intrusion detection very low 73 00:06:58,199 --> 00:07:01,560 in the architecture. But you never do prevention low in the architecture. Yet 74 00:07:01,560 --> 00:07:06,079 here you are not sorry, you know, here you are responding to attacks. 75 00:07:06,160 --> 00:07:12,160 As you know, an intrusion prevention system of all would have deep in 76 00:07:12,199 --> 00:07:19,959 the architecture. How do you do that without risk? Two? Normal operations. 77 00:07:19,680 --> 00:07:24,560 That's a very good question, Andrew, and yes, that's why we 78 00:07:24,680 --> 00:07:30,959 created a sivault and we've done all these efforts. So our principle is being 79 00:07:31,319 --> 00:07:38,720 automated as much as possible, and the first level is providing security controls that 80 00:07:39,399 --> 00:07:46,040 are built for otics not that's come from it, and they are trying to 81 00:07:47,079 --> 00:07:53,639 implement them in ot environments and adapt them to these environments and processes so that 82 00:07:53,680 --> 00:07:58,240 they do not impact them. It's the right way from the beginning. So 83 00:07:58,319 --> 00:08:03,439 this is for the security controls. The r aspect is we have Cybold Labs, 84 00:08:03,480 --> 00:08:13,279 which includes partnerships with different engineering and vendors that provide engineering equipment where we 85 00:08:13,360 --> 00:08:18,399 test and officialize our high fidelity response actions. So if an action for a 86 00:08:18,399 --> 00:08:26,920 specific asset is confirmed hundred percent that it's not going to impact operations delivered by 87 00:08:28,000 --> 00:08:35,759 that equipment, that goes into our automation actions that are part of that pool, 88 00:08:37,720 --> 00:08:43,399 and of course the engineering engineers can confirm the the you know, if 89 00:08:43,600 --> 00:08:50,600 that action could be implemented in their environment or not. And that's of course 90 00:08:50,879 --> 00:09:00,399 collaboration between us and the plant resources and it's strictly blocking attack specifics as an 91 00:09:00,440 --> 00:09:05,240 intern so it's not if you think of an IPS having all those rules and 92 00:09:05,360 --> 00:09:13,799 signatures or behavior analytics turned on and then pushes updates like you said, Well, 93 00:09:13,840 --> 00:09:18,000 that's not the way we're doing it. It's the reverse. So let's 94 00:09:18,039 --> 00:09:26,080 take this IPS and then it's not enabled completely. It's enabled on specific attacks 95 00:09:26,159 --> 00:09:31,120 that are happening in this environment, and that we confirm that those signatures have 96 00:09:31,240 --> 00:09:37,480 been tested and could not impact the operations. So they're not going to block 97 00:09:37,519 --> 00:09:41,960 a mod bus or an obviously away command. Our communication that is going through 98 00:09:43,519 --> 00:09:48,960 that is all tested on our end. So any what I'm hearing here, 99 00:09:48,559 --> 00:09:54,720 you know, it's let me back up a bit sort of general principles engineering 100 00:09:54,799 --> 00:09:56,799 change control. You know, if you've got a safety critical network or a 101 00:09:56,799 --> 00:10:03,000 reliability critical network, every change is a potential threat to safe and reliable operations, 102 00:10:03,000 --> 00:10:07,600 so you have to study the change before you make it. All of 103 00:10:07,600 --> 00:10:11,320 that is true, but sort of the next level of detail is that, 104 00:10:11,720 --> 00:10:16,960 look, some changes on these networks are carried out on many networks are carried 105 00:10:16,960 --> 00:10:20,720 out routinely. How can you do that? How's that possible? What's an 106 00:10:20,720 --> 00:10:24,720 example of that is, you know, a new screen on the human machine 107 00:10:24,759 --> 00:10:30,120 interface on the graphic user interface the twenty four by seven operator uses. They've 108 00:10:30,120 --> 00:10:33,039 got one hundred thousand data points. They're managing the only hope they have of 109 00:10:33,080 --> 00:10:35,879 You know, you can't see one hundred thousand data points on the screen at 110 00:10:35,879 --> 00:10:41,039 once. And so you've got summary screens. You've got summaries of summaries. 111 00:10:41,080 --> 00:10:45,320 You're constantly summarizing the information differently. There's new screens. You know, you've 112 00:10:45,320 --> 00:10:48,960 got a thousand screens. You might invent a new screen every month or so 113 00:10:50,799 --> 00:10:54,960 because you've discovered a new need in terms of seeing summary information. And it's 114 00:10:56,000 --> 00:10:58,759 the plant operators who are the gurus on using these screens, because they do 115 00:10:58,799 --> 00:11:03,399 this time. And so most sites, not all, but most sites, 116 00:11:03,080 --> 00:11:11,000 the engineering team will study the process of how to safely define new screens on 117 00:11:11,039 --> 00:11:16,000 the HMI and they will put together a procedure for the operator saying, when 118 00:11:16,039 --> 00:11:18,679 you need to define a new screen, this is what you do, and 119 00:11:18,759 --> 00:11:22,879 you go step by step through the procedure, and the operators follow the procedure 120 00:11:22,120 --> 00:11:28,360 to change the screens on the HMI. And so this is an example of 121 00:11:28,639 --> 00:11:33,879 a change that has to be done routinely, and so the engineering team has 122 00:11:33,919 --> 00:11:39,480 studied it, put a procedure together, it becomes part of the system. 123 00:11:39,559 --> 00:11:43,480 What I'm hearing that Yusuf is telling us is that his technology is being treated 124 00:11:43,559 --> 00:11:50,120 the same way saying an interruption. You know, if there's an attack detected, 125 00:11:50,200 --> 00:11:54,200 we need to interrupt that attack. That in principle is a change to 126 00:11:54,240 --> 00:12:01,320 the operating of the system. And you know they're not saying an intrusion prevention 127 00:12:01,399 --> 00:12:03,679 system from the IT network. Throw it on the OT network. Throw all 128 00:12:03,720 --> 00:12:07,279 the signatures in there, cross your fingers that nothing bad happens, because you 129 00:12:07,279 --> 00:12:11,320 don't know what these signatures are. They were designed for a different environment. 130 00:12:11,799 --> 00:12:16,240 He's saying that his system was designed for the the engineering environment and has been 131 00:12:16,279 --> 00:12:22,879 tested on specific vendors, specific assets, specific attack scenarios, and the vendors 132 00:12:22,840 --> 00:12:26,320 and engineering teams have said, yeah, this is safe to do to this 133 00:12:26,440 --> 00:12:35,879 device in this context, and it's only sort of approved changes, approved attack 134 00:12:37,039 --> 00:12:41,480 interruptions on approved assets, you know, in scenarios that the you know, 135 00:12:41,519 --> 00:12:46,159 the engineering teams understand thoroughly. It's only those approved changes that he'll carry out. 136 00:12:46,200 --> 00:12:50,919 And this is why I think he's got a manual process. If you've 137 00:12:50,919 --> 00:12:54,480 got anything else going on, you've got to have manual process. So this 138 00:12:54,559 --> 00:12:56,480 was the deep insight that I got out of you know, the sort of 139 00:12:56,480 --> 00:13:01,600 the first insight I got out of this this inter you, Um, it 140 00:13:01,679 --> 00:13:07,240 is possible to do routine change provided it's been engineered, and this is what 141 00:13:07,279 --> 00:13:13,679 they've done. You've got equipment into your labs, you've been testing. Uh 142 00:13:13,720 --> 00:13:20,639 you know whether your your system would would interfere with normal operations? Um. 143 00:13:20,679 --> 00:13:24,279 You know when you take your equipment and out into the field, You've got 144 00:13:24,360 --> 00:13:28,840 equipment in the field that you've tested against. You've got equipment that you haven't 145 00:13:28,879 --> 00:13:33,279 tested against. How fine grained you know? Ore is your your system in 146 00:13:33,320 --> 00:13:39,879 the field? You know you can presumably tell it this asset is you know, 147 00:13:39,039 --> 00:13:43,759 safe to to send messages to to interrupt the tax in progress because we've 148 00:13:43,759 --> 00:13:50,080 tested it. You know that asset possibly not? Is it? You know? 149 00:13:50,240 --> 00:13:56,360 Do you control the active response per asset or you know, per attack 150 00:13:56,480 --> 00:13:58,480 per asset or purport on the asset? How how fine grain? How do 151 00:13:58,480 --> 00:14:05,240 you control or know what is safe and what is not working With our customers, 152 00:14:07,039 --> 00:14:11,759 we act only on assets that do not impact operations or safety. So, 153 00:14:11,879 --> 00:14:16,840 for example, an engineering workstation that has no involvement in real time activities. 154 00:14:18,159 --> 00:14:22,960 We could, for example, take an action that says isolate this machine, 155 00:14:22,600 --> 00:14:30,240 so if we act automatically, that action is per asset and also on 156 00:14:30,279 --> 00:14:37,360 those attacks specific signals, so would have criticality levels assigned to those assets working 157 00:14:37,399 --> 00:14:43,440 with the operatures or the engineers in the plant or other environments, and then 158 00:14:43,840 --> 00:14:50,960 we assign them criticality in terms of are they part of the real time process? 159 00:14:52,840 --> 00:14:58,080 Could we automate these actions? Which actions should we automate or can automate? 160 00:14:58,759 --> 00:15:05,799 And we we leverage things like the five DS, so we could defend, 161 00:15:05,399 --> 00:15:13,360 which is automatic action of blockage. We can also delay or just go 162 00:15:13,559 --> 00:15:18,679 revert to detect in some cases. So it really depends. But the concern 163 00:15:18,720 --> 00:15:24,039 here is the automeric actions. So I understand that is going to be part 164 00:15:24,120 --> 00:15:33,080 of our collaboration with the field. In the beginning of the industrial security revolution, 165 00:15:33,320 --> 00:15:37,519 engineers were told to use it security principles, protect the information. We 166 00:15:37,519 --> 00:15:41,720 were told. We knew this was a poorfeit, but it was all we 167 00:15:41,759 --> 00:15:46,879 had. Today. The top security priority at industrial sites is safety. Don't 168 00:15:46,960 --> 00:15:54,480 kill anyone, don't cause an environmental disaster, and the second priority is reliability. 169 00:15:54,879 --> 00:16:00,759 Do not shut down our factory or infrastructure Today, safe and reliable operations 170 00:16:00,039 --> 00:16:07,440 use unhackable protections from cyber risks, not just cybersecurity. For a deeper look 171 00:16:07,480 --> 00:16:11,320 at the evolution of the revolution, we invite you to download Waterfall's report on 172 00:16:11,360 --> 00:16:17,919 the Emerging Consensus for Industrial Security Engineering. You can access the report at the 173 00:16:17,960 --> 00:16:25,440 Waterfall website Waterfall dash Security dot com, slash Engineering dash Consensus, or just 174 00:16:25,519 --> 00:16:30,440 go to the resources menu and click on white papers and ebooks. So the 175 00:16:30,559 --> 00:16:34,799 second insight that I've got here, and you know, he didn't quite say 176 00:16:34,840 --> 00:16:37,600 it this way, but you know, he gave the example of sort of 177 00:16:37,600 --> 00:16:41,399 an asset that's almost always safe to target, which is the engineering workstation, 178 00:16:41,919 --> 00:16:47,000 which is a full Windows machine or a full Linux machine with a lot of 179 00:16:47,039 --> 00:16:51,360 software on it and the ability to interact with just about everything to reprogram it. 180 00:16:51,360 --> 00:16:55,440 It's a very powerful tool, and so of course it's a natural draw 181 00:16:55,799 --> 00:17:03,080 for cyber attacks reaching into the OT system. And you know, there's other 182 00:17:03,880 --> 00:17:08,119 what what's in an industrial network. There's a lot of you know, PLCs 183 00:17:08,119 --> 00:17:12,880 and r to us and very low level devices that might have some horribly stripped 184 00:17:12,880 --> 00:17:18,119 down Linux under the hood, or a real time operating system that has very 185 00:17:18,160 --> 00:17:23,079 limited capability. You give these devices limited capability deliberately so that you can test, 186 00:17:25,079 --> 00:17:29,880 you know, deeply, the limited software they have to make sure it's 187 00:17:29,920 --> 00:17:33,720 safe, it's reliable. So there tend to be two kinds of assets in 188 00:17:33,720 --> 00:17:38,440 industrial networks. Assets that are sort of full fledged Windows Linux powerful tools, 189 00:17:38,960 --> 00:17:42,319 and assets that are much less powerful that have one job to do that are 190 00:17:42,319 --> 00:17:49,079 extremely reliable at that one job. And you know, in my read of 191 00:17:49,119 --> 00:17:55,519 the space, if an attackers coming in, they are going to want to 192 00:17:55,519 --> 00:18:00,000 get a foothold on the most powerful machines in the space so that they can 193 00:18:00,079 --> 00:18:04,359 use those machines to manipulate the sort of less intelligent machines in the space. 194 00:18:04,400 --> 00:18:08,720 That machines are actually connected to the physical process doing stuff. And so what 195 00:18:08,799 --> 00:18:14,559 I heard was that, you know, it probably makes more sense. You 196 00:18:14,599 --> 00:18:18,079 know, I didn't hear this use of didn't say this, but my read 197 00:18:18,119 --> 00:18:23,960 on it is that, you know, printers and engineering workstations and remote access 198 00:18:25,000 --> 00:18:29,279 jump posts, all of these full fledged Windows Linit machines, these are the 199 00:18:29,279 --> 00:18:36,279 ones that are likely to be the most tolerant of intrusion prevention actions sending them 200 00:18:36,279 --> 00:18:41,759 a message saying, you know, interrupt this or turn that off or because 201 00:18:41,359 --> 00:18:45,079 they're the most powerful, and they tend to be you know, the stuff 202 00:18:45,079 --> 00:18:49,720 that's essential to second by second safe operation is well bluntly dumb. It has 203 00:18:49,759 --> 00:18:55,160 one job to do, stay safe. And the stuff that is more powerful 204 00:18:55,400 --> 00:18:59,839 tends to be the stuff that manipulates and programs these other dumb systems. So 205 00:19:00,640 --> 00:19:07,920 every attack has two pieces. If you apply the intrusion interruption process to the 206 00:19:10,559 --> 00:19:17,039 non essential machines that are attacking everybody else the dumb machines, that strikes me 207 00:19:17,079 --> 00:19:19,039 as as brilliant. I thought, Hey, I never thought of that. 208 00:19:19,440 --> 00:19:22,440 You know what a clever approach this is. This is the second thing I 209 00:19:22,440 --> 00:19:26,240 got out of it. Even though I'm not sure he quite said these words. 210 00:19:26,240 --> 00:19:27,960 I might be reading something into what he's saying, but it makes sense 211 00:19:27,960 --> 00:19:33,920 to me. Can you give me a couple of examples? What what's an 212 00:19:33,920 --> 00:19:38,400 example of an attack that you would most likely interrupt because it's it's all been 213 00:19:38,400 --> 00:19:41,599 tested, and what's you know, maybe an example of an attack that you 214 00:19:41,640 --> 00:19:45,559 would I don't know, maybe just report rather than get in the middle of. 215 00:19:45,960 --> 00:19:53,200 Can you give us some some concrete examples here? Yes, So let's 216 00:19:53,200 --> 00:20:02,200 take an example of a ransomware attack Let's say that engineering workstation as an example, 217 00:20:02,400 --> 00:20:06,440 or a PLC that is not sending real time event so it's all depending 218 00:20:06,480 --> 00:20:12,359 on real time activities. If that engineering workstation get infected by our ransomware, 219 00:20:12,440 --> 00:20:18,720 even an unknown variant that we have never detected before, and now we confirm 220 00:20:18,799 --> 00:20:23,519 that it is, so our actions will include isolating, for example, that 221 00:20:23,640 --> 00:20:30,240 engineering workstation and then validating after that. What what are we going to do 222 00:20:32,079 --> 00:20:37,480 more than that just that isolation. The other example where we're going to take 223 00:20:37,559 --> 00:20:42,519 manual action is a PLC or an engineering workstation that is involved in real time 224 00:20:45,039 --> 00:20:49,880 process. It's sending commands, it's sending changes, and those are almost real 225 00:20:49,880 --> 00:20:53,920 time or real time, so those we cannot going to impact the operations. 226 00:20:53,960 --> 00:21:00,319 So what we do is we send man response actions that can be executed by 227 00:21:00,400 --> 00:21:06,960 the field resources, and then we're with them during that execution, supporting them 228 00:21:07,720 --> 00:21:15,559 so we can minimize and you know, contain that attack and minimize the impact. 229 00:21:15,359 --> 00:21:21,079 So that's interesting. You've used the word isolate a couple of times. 230 00:21:22,440 --> 00:21:25,880 How do you isolate an asset? Do you like log into the windows and 231 00:21:26,160 --> 00:21:30,279 tell it to drop the network interface or how do you isolate? I'm used 232 00:21:30,319 --> 00:21:34,839 to interrupting communications with like I said, TCP reset packets of old what do 233 00:21:34,880 --> 00:21:42,039 you do nowadays to isolate? So our device is a consider it an orchestrator 234 00:21:42,400 --> 00:21:49,640 of the security controls that are already in place into viruses, machines that we 235 00:21:49,680 --> 00:21:56,519 can interact with safely, like Windows prcs that we can communicate with, and 236 00:21:56,640 --> 00:22:03,119 others. So it orchestrates all this incident response action at the network level. 237 00:22:03,359 --> 00:22:07,559 Or if an agency is installing a machine, would it be a third party 238 00:22:07,559 --> 00:22:14,319 agent or our own proprietary agent that we put in place for for insident response 239 00:22:14,359 --> 00:22:21,359 purposes. In the network level, there could be a industry switch stick, 240 00:22:21,400 --> 00:22:26,519 for example, a rugged gom swhich would take action on all an inty virus 241 00:22:26,559 --> 00:22:33,119 that is already sitting on that machine that we would leverage. So this makes 242 00:22:33,160 --> 00:22:38,640 sense. Um, you know you are are interrupting only those communications that are 243 00:22:40,039 --> 00:22:44,519 a less critical and B you've tested for the impact on the physical process. 244 00:22:45,680 --> 00:22:49,960 Um. But as I said, you know, in my experience, there's 245 00:22:49,960 --> 00:22:57,880 a fair amount of resistance out there to any kind of active response because of 246 00:22:57,960 --> 00:23:03,200 the experience you know ten years ago of the old intrusion prevention systems. You 247 00:23:03,240 --> 00:23:06,759 know there's a lot of a lot of engineers are that old, they remember 248 00:23:07,079 --> 00:23:11,960 those problems. How do you get you know, how do you overcome that 249 00:23:11,039 --> 00:23:15,960 sort of resistance in the marketplace? How do you get buying from the engineering 250 00:23:15,000 --> 00:23:19,720 team who are are very focused on controlling change, controlling everything? How do 251 00:23:19,759 --> 00:23:23,640 you get them to buy into this. What we do is we start first 252 00:23:23,680 --> 00:23:27,960 of all with visibility and detection, right, Um, that's the first step. 253 00:23:29,039 --> 00:23:33,319 We don't come in and say automates everything. After that, we start 254 00:23:33,400 --> 00:23:40,200 automating little by little, working with the engineering teams and the plant resources and 255 00:23:40,240 --> 00:23:45,839 the field resources which are involved with us in defining what is critical, what 256 00:23:45,079 --> 00:23:51,759 is not critical as assets, what actions could we take, what actions could 257 00:23:51,759 --> 00:23:56,359 we not take at all? And there is sometimes where we have semi automated 258 00:23:57,319 --> 00:24:03,079 responses. So the work is done with the engineering team. We do not 259 00:24:03,200 --> 00:24:10,039 just rely on tech. Tech is just there to help an assistant facilitate, 260 00:24:10,559 --> 00:24:15,119 but it's actually this collaborative work. So if we take, for example, 261 00:24:17,519 --> 00:24:23,279 let's say the engineering change control, it's a very crucial process. So they 262 00:24:23,359 --> 00:24:30,480 leverage thinks like automated changes or automated routine challenges. So we could take these 263 00:24:30,519 --> 00:24:36,519 incidental response actions that are automated as part of engineering challenges and activities, and 264 00:24:36,640 --> 00:24:41,039 that way, they are part of it. So we always look at this 265 00:24:41,400 --> 00:24:47,920 from the engineering aspect, actually not from the IT cyber defense aspect, so 266 00:24:48,160 --> 00:24:52,000 because we are very aware of the impact that it can cause if something goes 267 00:24:52,000 --> 00:24:59,839 wrong. And it seems like in your question there you're referring to maybe some 268 00:25:00,079 --> 00:25:03,720 specific personal experiences that you had some years ago. At this it wasn't so 269 00:25:03,799 --> 00:25:07,839 much personal experience. It was sort of, as I said, Colin Wisdom, 270 00:25:07,839 --> 00:25:10,799 in the industry, it was a long time ago. I don't even 271 00:25:10,839 --> 00:25:15,079 remember specific examples. I know there were specific cases where, you know, 272 00:25:15,319 --> 00:25:19,599 entire plants were shut down because of a malfunctional because of a miss you know, 273 00:25:21,000 --> 00:25:26,480 a misfire of the intrusion prevention system. And you know, it doesn't 274 00:25:26,480 --> 00:25:32,720 even take more than a couple of these incidents before you know, multiple industries 275 00:25:33,160 --> 00:25:36,480 draw the conclusion that this is not going to work. So you know, 276 00:25:36,559 --> 00:25:41,519 for example, if in uh, you know, in any industry um A, 277 00:25:42,079 --> 00:25:47,279 you know, you put a let's say a an intrusion prevention system built 278 00:25:47,319 --> 00:25:51,960 into a firewall. You put a firewall between the control system, the HMI 279 00:25:52,599 --> 00:26:00,799 and the devices, the PLCs and that system conclude. You know, it 280 00:26:00,839 --> 00:26:03,599 gets a new a new signature, download that and you know, and it 281 00:26:03,599 --> 00:26:08,720 doesn't understand industrial protocols, and so it sees a new signature comes through saying 282 00:26:08,799 --> 00:26:12,880 when you're when somebody, someone's targeting an attack on this port that you you 283 00:26:12,920 --> 00:26:18,200 know they haven't targeted before, but that port is being used for a different 284 00:26:18,200 --> 00:26:22,839 purpose in the PLCs, in the you know, the real time communications. 285 00:26:22,119 --> 00:26:26,319 Now it says, oh, there's communications on this port and it matches this 286 00:26:26,400 --> 00:26:30,480 crudely defined signature, you know, poorly tested signature. Bang, shut down 287 00:26:30,519 --> 00:26:34,599 the communications to the PLC. Now the operator can't control half a dozen its 288 00:26:34,599 --> 00:26:40,480 plc's or her plc's and has no choice but to shut the plant down because 289 00:26:40,519 --> 00:26:45,119 the operators lost control of the physical process. This only has to happen a 290 00:26:45,119 --> 00:26:51,680 couple of times before entire industries. I mean again, that's sort of a 291 00:26:51,720 --> 00:26:55,000 generic example. I don't remember where exactly had happened, but I know it 292 00:26:55,039 --> 00:26:59,119 happened. You know, these things happened. But you know, someone like 293 00:26:59,200 --> 00:27:03,200 refining to turn around and looks at this and says, oh, if I 294 00:27:03,279 --> 00:27:10,759 have to trip my plant, I process one hundred million dollars worth of product 295 00:27:10,880 --> 00:27:14,039 a day. If I trip my plant and I've got to start from a 296 00:27:14,119 --> 00:27:17,079 cold start to go back up to full production, it's going to take me 297 00:27:17,160 --> 00:27:21,519 twelve days to get back up to full production. So I've lost, you 298 00:27:21,559 --> 00:27:25,000 know, fifty percent of that. The average between zero one one hundred percent 299 00:27:25,160 --> 00:27:27,400 is fifty percent fifty percent times six days. Do the math. At one 300 00:27:27,440 --> 00:27:33,079 hundred million dollars, I've lost a half billion dollars of product. Here a 301 00:27:33,119 --> 00:27:37,480 half billion, now five hundred and six hundred million dollars worth of product. 302 00:27:37,680 --> 00:27:41,920 Never going to use one of those, you know, it just bang gone. 303 00:27:41,359 --> 00:27:45,640 Entire industries groups of industries look at one of these incidents and say that's 304 00:27:45,720 --> 00:27:52,119 unacceptable. So I don't remember the specifics, but you know, a couple 305 00:27:52,160 --> 00:27:56,599 of them happened and everybody turned around and said not doing that ever, the 306 00:27:56,960 --> 00:28:02,000 you know, it just costs too much. Then, Andrew, how, 307 00:28:02,359 --> 00:28:04,160 you know, give me a sense how strong is the resistance, because his 308 00:28:04,359 --> 00:28:08,079 use have mentioned, you know, he's working with engineering teams on this seems 309 00:28:08,119 --> 00:28:15,000 like it's pretty amenable. I imagine there's a fair bit of resistance initially, 310 00:28:15,000 --> 00:28:18,559 and you know, I thought his answer was interesting. He said Andrew. 311 00:28:18,599 --> 00:28:23,960 We start with detection. I mean they are detecting attacks as well as interrupting 312 00:28:23,960 --> 00:28:29,039 them. Start with detection, you know, show the engineers what's happening in 313 00:28:29,079 --> 00:28:33,599 their network, and you know, show them presently develop a relationship. Now 314 00:28:33,680 --> 00:28:37,119 now they're learning stuff from you. Whenever, whenever you turn on detection on 315 00:28:37,160 --> 00:28:40,440 a network that you've been blind to before, you learn stuff about it. 316 00:28:40,480 --> 00:28:44,079 So you're developing a relationship with these engineers. You're teaching them things about their 317 00:28:44,079 --> 00:28:49,240 network, You're develop developing credibility with them. And you know, he's got 318 00:28:49,240 --> 00:28:53,319 in his pocket, you know, endorsements by vendors, testing by vendors, 319 00:28:53,640 --> 00:28:57,799 the same vendors that produced the control system that the engineers are operating in the 320 00:28:57,799 --> 00:29:02,359 plant. You know, he's got the endorsements of the vendors saying it's safe 321 00:29:02,400 --> 00:29:04,440 to do this, it's safe to do that. And the engineers are learning 322 00:29:04,480 --> 00:29:07,640 about their networks. You know, they learn together. They they the engineers 323 00:29:08,319 --> 00:29:15,400 are not resisting change for the sake of resisting change. The resistant change because 324 00:29:15,720 --> 00:29:18,359 there's a risk. And once you persuade them, you show them, you 325 00:29:18,400 --> 00:29:22,680 know what's going on, you show them the evidence. Yeah, they're absolutely 326 00:29:22,720 --> 00:29:32,160 willing to you know, design, you know, include in their change control 327 00:29:32,680 --> 00:29:37,960 system, you know, ways to carry out routine changes if there's a benefit. 328 00:29:37,000 --> 00:29:41,079 And the benefit here is interrupting a taxing progress and that's a huge benefit. 329 00:29:41,160 --> 00:29:45,440 So you know, you can't come in and drop it on them, 330 00:29:45,519 --> 00:29:48,640 turn it on and walk away. That's they're going to turn it off and 331 00:29:48,720 --> 00:29:52,519 walk away, thank you. But work with them. And yeah, these 332 00:29:52,559 --> 00:29:59,319 people are reasonable, they want the benefits. You know, just just have 333 00:29:59,359 --> 00:30:03,200 to make it happen. And maybe there's a generational component to it too, 334 00:30:03,319 --> 00:30:07,200 you know, like the ways that people thought and the experiences that they had 335 00:30:07,240 --> 00:30:15,319 ten years ago maybe aren't quite as relevant today to a degree. But engineers 336 00:30:15,400 --> 00:30:22,039 new or old, nobody wants to trip the plant. And so yeah, 337 00:30:22,079 --> 00:30:26,880 you know, the younger folks might not have suffered quite as many hard knocks 338 00:30:26,920 --> 00:30:30,400 as the older folks. It's this is why you have, you know, 339 00:30:30,440 --> 00:30:33,960 these engineering teams. We haven't talked about it. I should bring someone on 340 00:30:34,119 --> 00:30:41,319 talking about this. But these engineering teams design succession plans into their planning. 341 00:30:41,359 --> 00:30:45,680 They have like thirty year timelines they are executing too, and so yeah, 342 00:30:45,759 --> 00:30:49,680 they deliberately mix up you know, young and old so that you know, 343 00:30:49,680 --> 00:30:52,240 one can learn from the others, so that you know, each can learn 344 00:30:52,319 --> 00:30:59,519 from the other. So yeah, there there is some of that. We 345 00:30:59,599 --> 00:31:03,480 know. You also talked about manual incident response. Now you know, I'm 346 00:31:03,519 --> 00:31:07,720 familiar with the term flyaway teams. Uh. You know, something goes wrong 347 00:31:07,839 --> 00:31:14,000 at a an IT network and OT network and you know, incident response experts 348 00:31:14,559 --> 00:31:18,400 get on the aircraft, fly out there with suitcases full of equipment and deal 349 00:31:18,440 --> 00:31:25,680 with the problem. You're doing a bunch of this remotely. How does that 350 00:31:25,720 --> 00:31:29,039 work? I mean, is it as simple as you pick up the phone 351 00:31:29,079 --> 00:31:33,920 and call the engineer who's your contact and tell them what to do? Or 352 00:31:33,000 --> 00:31:38,440 you know, is there is there more to it than that. Yeah, 353 00:31:38,480 --> 00:31:48,640 so it's a combination. First, we are present in North America covering it 354 00:31:48,759 --> 00:31:55,559 from Montreal and officially and also in Europe from France, and Africa is covered 355 00:31:55,599 --> 00:32:00,640 from Morocco. So what we do, like you said, Andrews, most 356 00:32:00,640 --> 00:32:07,920 of it is automatically is remotely done with the teams from from our from the 357 00:32:07,960 --> 00:32:13,599 field resources. But part of it is we have experts that are ready to 358 00:32:13,640 --> 00:32:19,480 go on site with their GEM bags, ready to go to take care of 359 00:32:19,480 --> 00:32:28,319 these incidents, but still handled from a centralized crisis crisis unit part of the 360 00:32:28,960 --> 00:32:35,000 So the manual actions that we take our part of our platform. So technicians 361 00:32:35,079 --> 00:32:40,960 or engineers or field experts can log into the platform or have those playbooks, 362 00:32:43,119 --> 00:32:46,559 men will be clarify to them where is the process that needs to be done, 363 00:32:46,720 --> 00:32:51,759 what are the steps to follow. It's a step by step kind of 364 00:32:52,799 --> 00:33:00,200 guides that are also tested in our laps unprovided part of this offering. I 365 00:33:00,200 --> 00:33:02,759 mean, this is this is unfascinating. I learned something today. I thought 366 00:33:02,799 --> 00:33:08,519 intrusion prevention on on you know, industrial networks was an experiment failed long ago, 367 00:33:09,480 --> 00:33:14,799 but here you folks are doing it, and you know, you're looking 368 00:33:14,799 --> 00:33:19,519 into the future as well. I mean this, you know is clearly an 369 00:33:19,559 --> 00:33:23,880 evolution over what was available ten years ago. But I imagine you guys are 370 00:33:24,559 --> 00:33:28,319 working on this into the future as well. What are you working on? 371 00:33:28,359 --> 00:33:31,440 What can we look forward to? What's coming? Yeah, So, so 372 00:33:32,000 --> 00:33:37,400 DOME is very crucial aspect of our business, and we've been working on this 373 00:33:37,519 --> 00:33:43,400 for years and making it as perfect as possible for whatever our missions we provide. 374 00:33:44,079 --> 00:33:49,119 So we're continuously simplifying the implementation. That's that's one of the aspects that 375 00:33:50,640 --> 00:33:54,640 we have to concentrate on and also helping industry adoption of what we spoke about 376 00:33:54,640 --> 00:34:00,559 today. For our dome offering, we're also involved with different entities, would 377 00:34:00,680 --> 00:34:08,000 be vendors, partners, government entities to push this principle so that industry or 378 00:34:08,119 --> 00:34:13,400 community cannot cannot be afraid. We don't. We don't have to act after 379 00:34:13,440 --> 00:34:19,679 the attack happened, which is the case today for organizations that are not our 380 00:34:19,719 --> 00:34:23,199 customers, where an attack has already happened, the substations are already blocked, 381 00:34:23,639 --> 00:34:30,559 the grades shutdown. Well, now we need to revert back right. So 382 00:34:30,599 --> 00:34:34,719 that's the first piece. We want to be as proactive and as active as 383 00:34:34,719 --> 00:34:38,719 possible. We can do it. We're very confident we have done it with 384 00:34:38,800 --> 00:34:45,480 different entities. The other aspect is we want to push more and more also 385 00:34:45,559 --> 00:34:51,280 so well, what we help with is consequence driven cyber informed engineering so CC 386 00:34:51,559 --> 00:34:58,599 or just cyberinformed engineering to help these engineers in the field engineer out most possible 387 00:34:58,679 --> 00:35:02,719 cyber risks. And it's what we call in the community. I see as 388 00:35:02,880 --> 00:35:12,840 unhackable mitigation so or mitigation aspects that are part of the engineering design. And 389 00:35:12,960 --> 00:35:17,239 I like the example that that you give Andrew, which is very simple to 390 00:35:17,320 --> 00:35:23,519 understand. It's we have a boiler that can explode, and you know if 391 00:35:23,599 --> 00:35:30,239 we control the PLC and we hack it and now we control the parameters of 392 00:35:30,280 --> 00:35:35,559 that boiler to make it explode and kill people. Well, as a simpler 393 00:35:35,679 --> 00:35:45,840 way of countering that cyber risk is putting an overpressure valve which will automatically, 394 00:35:46,079 --> 00:35:50,960 you know, mitigate that risk. So we're working. I'm working with all 395 00:35:50,960 --> 00:35:57,920 these entities in our company also, so we're involved with the our labs and 396 00:35:58,239 --> 00:36:02,599 Caesar and others to make sure, you know, we bring this active aspect 397 00:36:04,039 --> 00:36:08,840 to the community and not be afraid and I'm sure everybody will appreciate it. 398 00:36:10,119 --> 00:36:14,079 Before we let you go, can you sum up for our listeners what you 399 00:36:14,119 --> 00:36:16,960 know? What should what should listeners take away from the episode here? Well, 400 00:36:17,079 --> 00:36:25,159 the OT I see as cyber defense aspect is something where I'm calling everyone 401 00:36:25,199 --> 00:36:30,119 to contribute and collaborate, you know, practitioners, vendors, service providers to 402 00:36:30,280 --> 00:36:36,880 safeguard or critical infrastructure. This this is this is a community work and we 403 00:36:36,920 --> 00:36:42,840 need to provide the best ways of doing it. How can we prevent these 404 00:36:42,840 --> 00:36:49,639 attacks not just detect? Prevention is key. We cannot let these criminal gangs 405 00:36:49,840 --> 00:36:52,679 win over us. We can do it, as as Cybald. We have 406 00:36:52,800 --> 00:37:00,840 tested it, we have customers that already have our hybrid automated model and dome, 407 00:37:00,039 --> 00:37:07,400 which is our offering. Is the first step in indefendedness. And I 408 00:37:07,480 --> 00:37:10,440 thank you for inviting me today, Andrew. It was a pleasure being here. 409 00:37:14,519 --> 00:37:17,079 Andrew. That was your interview with use of JAD. Do you have 410 00:37:17,119 --> 00:37:22,239 any final thoughts to close us out today? Nothing new, I mean I'm 411 00:37:22,280 --> 00:37:28,880 repeating myself here, but I was fascinated by by this technology because common wisdom 412 00:37:29,000 --> 00:37:30,920 is you can't do this, and yet here they are doing it. I 413 00:37:30,960 --> 00:37:37,199 mean, you know, innovation the world, the world continues to move forward. 414 00:37:37,280 --> 00:37:39,199 Good job. Okay, Well, thanks to you so for speaking with 415 00:37:39,199 --> 00:37:43,159 you, Andrew. And Andrew, as always, thank you for speaking with 416 00:37:43,159 --> 00:37:45,360 me. It's always a pleasure, Nate, thank you. This has been 417 00:37:45,360 --> 00:37:51,639 the Industrial Security Podcast from Waterfall. Thanks to everybody out there listening.