1 00:00:04,879 --> 00:00:09,320 Speaker 1: Welcome listeners to the Industrial Security Podcast. My name is 2 00:00:09,400 --> 00:00:13,039 Nate Nelson. I'm here with Andrew Ginter, the vice president 3 00:00:13,160 --> 00:00:17,440 of Industrial Security at Waterfall Security Solutions, who's going to 4 00:00:17,480 --> 00:00:25,519 introduce the subject and guest of our show today. Andrew, 5 00:00:25,839 --> 00:00:26,399 how's it going. 6 00:00:27,000 --> 00:00:29,839 Speaker 2: I'm doing very well, Thank you, Nate. Our guest today 7 00:00:29,920 --> 00:00:33,159 is Marcel Rickson. He is the founder and lead instructor 8 00:00:33,200 --> 00:00:37,719 at fox Grid International, and our topic is hardware hacking. 9 00:00:37,920 --> 00:00:42,759 Picking apart the hardware, finding the vulnerabilities, you know, arguably 10 00:00:43,000 --> 00:00:45,640 essential attack knowledge. We need to understand how we're going 11 00:00:45,679 --> 00:00:48,560 to be attacked if we're going to design effective defenses. 12 00:00:48,640 --> 00:00:50,439 So that's that's the topic for today. 13 00:00:51,240 --> 00:00:55,000 Speaker 1: Then, without further Ado, here's your conversation with Marcel. 14 00:00:57,799 --> 00:01:01,280 Speaker 2: Hello Marcel, and welcome to the podcast. Before we get started, 15 00:01:01,280 --> 00:01:03,359 can I ask you to introduce yourself. Please tell our 16 00:01:03,359 --> 00:01:05,680 listeners a little bit about your background and about the 17 00:01:05,680 --> 00:01:07,400 good work that you're doing at Fox Grid. 18 00:01:07,959 --> 00:01:10,519 Speaker 3: Yeah, thank you, Andrew. Hi everyone, my name is MARSA 19 00:01:10,599 --> 00:01:13,560 Riksen and if it would introduce me in one sentence, 20 00:01:13,719 --> 00:01:17,959 I am an automation engineer turned ot security nerd. To 21 00:01:18,000 --> 00:01:21,439 my background, I have a master's in automation engineering. I 22 00:01:21,480 --> 00:01:25,359 have global experience in commissioning automation systems as well as 23 00:01:25,760 --> 00:01:31,159 programming planning industrial operations. Now, during my day job, I 24 00:01:31,200 --> 00:01:35,280 am an OT and IoT security consultant and a product 25 00:01:35,359 --> 00:01:39,959 owner of our in houset remote access solution. During my nighttime, 26 00:01:40,040 --> 00:01:42,040 I am an hacker, or if you want to put 27 00:01:42,079 --> 00:01:45,920 it more formal, I am an independent OT security researcher 28 00:01:45,959 --> 00:01:50,480 that looks at what makes and breaks OT devices. Coming 29 00:01:50,519 --> 00:01:54,000 from that, I also founded Foxgrid, where I want to 30 00:01:54,079 --> 00:01:58,560 teach industrial cybersecurity and safety to newcomers. 31 00:01:59,439 --> 00:02:03,400 Speaker 2: Thank you for that. And our topic is hardware hacking. 32 00:02:03,920 --> 00:02:05,920 Can we start with an example. You've got a couple 33 00:02:05,959 --> 00:02:07,799 of reports out, Can you pick one? Can you tell 34 00:02:07,879 --> 00:02:10,520 us about, you know, a concrete example of what what 35 00:02:10,759 --> 00:02:11,120 is that? 36 00:02:11,919 --> 00:02:15,240 Speaker 3: Yeah, let's talk about the hardware hacking that led to 37 00:02:15,280 --> 00:02:18,280 a CVE that I found last year where I found 38 00:02:18,400 --> 00:02:22,759 hard coded root credentials hidden deep in the device's filmware memory. 39 00:02:23,520 --> 00:02:25,360 Speaker 2: Can you go a little deeper? What was the device, 40 00:02:25,439 --> 00:02:28,879 how's it supposed to work? And you know how important 41 00:02:28,960 --> 00:02:29,680 is what you found? 42 00:02:30,439 --> 00:02:33,840 Speaker 3: So the device is a remote access skateway that machine 43 00:02:33,879 --> 00:02:38,159 builders usually built into the electric cabinet so which connects 44 00:02:38,199 --> 00:02:41,879 the machine to the service provider. In case there's an 45 00:02:42,000 --> 00:02:47,759 unplanned interruption or any other operational bug coming up, the 46 00:02:47,800 --> 00:02:51,719 service provider can directly connect over the cloud portal to 47 00:02:51,919 --> 00:02:54,400 this edge device and start troubleshooting. 48 00:02:55,639 --> 00:02:58,439 Speaker 2: If I made. This is something that's used in manufacturing. 49 00:02:58,439 --> 00:03:01,360 When you say the manufacturer, you mean someone who's building 50 00:03:01,400 --> 00:03:04,319 a robot, someone who's building a stamping machine, someone who's 51 00:03:04,319 --> 00:03:07,520 building I don't know, conveyor. Is that the use case here? 52 00:03:08,159 --> 00:03:12,759 Speaker 3: This basically can be used in any operational so from 53 00:03:12,800 --> 00:03:17,439 your maybe water treatment planned to your manufacturing to your 54 00:03:17,439 --> 00:03:20,759 building automation. There are really no limits. This is really 55 00:03:20,800 --> 00:03:25,680 a network connection from the service engineer's laptop directly into 56 00:03:25,759 --> 00:03:27,919 the heart of the device or into the heart of 57 00:03:27,960 --> 00:03:28,560 the operation. 58 00:03:29,680 --> 00:03:32,639 Speaker 2: Okay, So it's not just used for like a robot 59 00:03:32,759 --> 00:03:36,319 for a manufacturer of equipment. It might also be used 60 00:03:36,319 --> 00:03:39,759 by a service provider, by the engineer who's responsible for 61 00:03:40,759 --> 00:03:44,159 you know, occasionally coming in and servicing parts of a 62 00:03:44,240 --> 00:03:47,599 water treatment system. It's used to access systems as well 63 00:03:47,639 --> 00:03:49,479 as devices, is what I'm hearing. 64 00:03:49,840 --> 00:03:55,000 Speaker 3: Yes, correct, So this acts as the gateway to the 65 00:03:55,039 --> 00:03:57,039 machine or operational network. 66 00:03:57,800 --> 00:04:01,199 Speaker 2: So you've found the default credentials. Does that mean that 67 00:04:01,240 --> 00:04:04,479 any fool who wants to can connect to the cloud 68 00:04:04,520 --> 00:04:07,360 connect into this thing, or how would you use those 69 00:04:07,360 --> 00:04:09,360 default credentials. 70 00:04:09,599 --> 00:04:14,120 Speaker 3: Luckily, the attack vector is really narrow. These default credentials. 71 00:04:14,199 --> 00:04:17,720 They grant root access to the device, and you only 72 00:04:17,759 --> 00:04:21,000 can get root access when you are physically connected to 73 00:04:21,120 --> 00:04:25,519 the device. So luckily the cloud attack surface or the 74 00:04:25,560 --> 00:04:28,120 cloud is not exposed to disvulnerability. 75 00:04:31,600 --> 00:04:33,399 Speaker 1: Andrew, I don't know if I just missed it, but 76 00:04:33,720 --> 00:04:36,879 what is the actual device that we're talking about here? 77 00:04:38,079 --> 00:04:42,000 Speaker 2: It is an Ixon device I XO N I forget 78 00:04:42,000 --> 00:04:44,399 the exact name of it, but it's it's physically it's 79 00:04:44,439 --> 00:04:47,079 a little device about you know, six inches square in 80 00:04:47,120 --> 00:04:48,040 an inch thick. 81 00:04:49,279 --> 00:04:49,680 Speaker 3: And. 82 00:04:51,040 --> 00:04:54,480 Speaker 2: In my understanding, it's you know, it's a remote access device. 83 00:04:54,959 --> 00:04:59,639 You can connect into it from the cloud. Who uses this? 84 00:05:00,839 --> 00:05:03,720 The sense I have is that it's used in manufacturing. 85 00:05:03,759 --> 00:05:08,199 If you're building a laser cutter or a stamping machine, 86 00:05:08,560 --> 00:05:11,439 you might build one of these into the thing so 87 00:05:11,480 --> 00:05:13,639 that when the customer calls you up and says your 88 00:05:13,680 --> 00:05:17,360 machine isn't working, something is worn out, you can remote 89 00:05:17,399 --> 00:05:20,600 into it, do the diagnostics and say I think it's 90 00:05:20,600 --> 00:05:24,000 this part, replace this part, see if the problem is solved, 91 00:05:24,399 --> 00:05:26,839 because you know, moving parts where out friction is the 92 00:05:27,120 --> 00:05:30,279 is the enemy of moving parts. But you know, when 93 00:05:30,279 --> 00:05:36,000 I asked the gentleman Rick, he said, yeah, manufacturers of 94 00:05:36,240 --> 00:05:40,079 physical equipment use it so they can maintain the equipment 95 00:05:40,160 --> 00:05:44,279 or diagnose the equipment remotely. But it's remote access. You know, 96 00:05:44,360 --> 00:05:47,720 a service provider, an engineer who's responsible for, you know, 97 00:05:48,319 --> 00:05:51,879 keeping the automation running at a dozen small water utilities 98 00:05:51,959 --> 00:05:55,360 in the geography might well buy, you know, a half 99 00:05:55,399 --> 00:05:57,959 dozen of these and drop one of them into each 100 00:05:58,079 --> 00:06:02,079 water system to access the HMI and the automation and whatnot. 101 00:06:02,279 --> 00:06:04,560 So it's remote access. The sense I have that it's 102 00:06:04,639 --> 00:06:10,759 used frequently by manufacturers of equipment that's used in manufacturing, 103 00:06:11,000 --> 00:06:14,079 but it could be used by service providers as well. 104 00:06:14,920 --> 00:06:17,399 Speaker 1: And I think it's the remote access thing that has 105 00:06:17,480 --> 00:06:21,759 me a little bit confused here. We're talking about hard 106 00:06:21,800 --> 00:06:26,199 coded credentials as vulnerability, something I'm rather used to in 107 00:06:26,319 --> 00:06:30,480 the IT space, right, Like a public repository or a 108 00:06:30,560 --> 00:06:37,040 server that's been incorrectly configured will leak credentials to the 109 00:06:37,040 --> 00:06:39,680 web that then actors could use to get in and 110 00:06:39,720 --> 00:06:44,360 we're talking about a remote access device, and yet I 111 00:06:44,360 --> 00:06:47,920 think he mentioned there that you can only actually exploit 112 00:06:48,000 --> 00:06:53,079 this vulnerability if you have local physical access to the machine. 113 00:06:53,399 --> 00:06:56,240 So can you help me explain that gap. 114 00:06:57,240 --> 00:06:59,079 Speaker 2: We go into this in sort of more detail later 115 00:06:59,120 --> 00:07:01,600 in the interview, but let me let me let people 116 00:07:01,600 --> 00:07:05,560 know kind of what's happening. This is you know, there 117 00:07:05,560 --> 00:07:10,319 are basically two user interfaces to the device. One is 118 00:07:10,720 --> 00:07:13,680 the remote access user interface with users configured and blah 119 00:07:13,720 --> 00:07:19,160 blah blah. That's not where the vulnerability is. The other 120 00:07:19,279 --> 00:07:22,319 interface is if you touch the device and you connect 121 00:07:22,319 --> 00:07:24,040 to it. I don't know, I was a little weak 122 00:07:24,079 --> 00:07:26,920 on the details. If you connect through the USB port 123 00:07:27,040 --> 00:07:30,839 or if you connect you know pins, you know, you're 124 00:07:31,040 --> 00:07:34,279 too electrically to pins sitting there on the on the 125 00:07:34,439 --> 00:07:36,480 on the circuit board. If you open the device up, 126 00:07:37,360 --> 00:07:41,319 you can get access to the operating system of the device. 127 00:07:42,079 --> 00:07:45,240 And it's the operating system credentials that were lead so 128 00:07:45,279 --> 00:07:48,000 in order to use those credentials, they don't work on 129 00:07:48,079 --> 00:07:53,680 the remote user interface. They work locally when you're when 130 00:07:53,680 --> 00:07:56,319 you're able to physically touch the device and plug stuff 131 00:07:56,360 --> 00:08:00,399 into it. The CVE was twenty twenty four, dash five seven, 132 00:08:00,600 --> 00:08:04,079 seven nine zero. It was given a five or a 133 00:08:04,120 --> 00:08:06,240 five point nine or something like this, not a ten. 134 00:08:06,360 --> 00:08:09,800 This is not a remote code execution vulnerability. You can't 135 00:08:09,800 --> 00:08:11,759 do this remotely. You have to be local. It's a 136 00:08:11,839 --> 00:08:15,199 local escalation of privilege vulnerability. 137 00:08:15,680 --> 00:08:17,879 Speaker 1: That explanation makes a lot of sense to me, But 138 00:08:18,079 --> 00:08:20,959 why is it that, Like, how can you even leak 139 00:08:21,040 --> 00:08:25,560 credentials to somebody who's physically using a computer? Right, Like, 140 00:08:25,800 --> 00:08:28,439 any credentials on my computer that get leaked to me 141 00:08:28,839 --> 00:08:31,639 doesn't matter because I'm the user. So what I suppose 142 00:08:31,680 --> 00:08:36,159 I'm asking attack scenarios are we worried about with this vulnerability. 143 00:08:37,120 --> 00:08:39,279 Speaker 2: Well, actually we didn't go into that, but you know, 144 00:08:39,360 --> 00:08:43,679 as far as I know, the scenario is that you're 145 00:08:43,679 --> 00:08:47,399 there locally touching the device. Now normally, you know, you 146 00:08:47,440 --> 00:08:50,440 look at the device, it's got network ports, it's got 147 00:08:50,879 --> 00:08:53,679 you know, one of the ports is connected out to 148 00:08:54,120 --> 00:08:57,320 the world. You come in remotely, you know it does 149 00:08:57,320 --> 00:09:03,080 its thing. There is no other supported user interface. But 150 00:09:03,559 --> 00:09:05,360 if you touch the device, you can get in there. 151 00:09:05,399 --> 00:09:08,240 You can tamper with the firmwaire, you can tamper with stuff. 152 00:09:08,279 --> 00:09:13,320 You can you know, you could presumably create credentials that 153 00:09:13,360 --> 00:09:15,559 you could use remotely. You'd need a little bit of 154 00:09:15,600 --> 00:09:18,480 skill to do that, but you know you could break 155 00:09:18,519 --> 00:09:22,159 the device, so it's again if you're standing there with 156 00:09:22,200 --> 00:09:24,519 a hammer, you could also break the device. This is 157 00:09:24,559 --> 00:09:27,519 why it was given a lower priority. Yes, technically it's 158 00:09:27,519 --> 00:09:34,240 a vulnerability, it's not a really alarming one. What's interesting 159 00:09:34,399 --> 00:09:37,480 is how did you find it? Because the way that 160 00:09:37,559 --> 00:09:40,480 he found it, the technique is what he teaches at 161 00:09:40,519 --> 00:09:44,519 Foxgrid you can also use to find more interesting stuff. 162 00:09:47,120 --> 00:09:48,679 What I wanted to ask you about is, you know, 163 00:09:48,720 --> 00:09:54,960 we've never had someone on the show who picks things 164 00:09:54,960 --> 00:09:57,960 apart like this, or maybe we did once about three 165 00:09:58,000 --> 00:10:00,759 years ago, but it's been a long time, you know. 166 00:10:00,919 --> 00:10:05,120 Can you talk about the process? How did you find this? 167 00:10:05,320 --> 00:10:08,879 How does one pick these things apart? What is what 168 00:10:08,919 --> 00:10:09,559 does that mean? 169 00:10:10,399 --> 00:10:13,000 Speaker 3: If I would just describe it to you maybe in 170 00:10:13,000 --> 00:10:16,240 a pub or over our coffee. This really is like 171 00:10:16,320 --> 00:10:19,720 a hardware and digital scavenger hunt because you have to 172 00:10:19,759 --> 00:10:22,360 look at so many things. You also go down a 173 00:10:22,399 --> 00:10:24,960 rabbit hole, see it's the wrong way, turn around, and 174 00:10:25,000 --> 00:10:29,399 then keep digging. And to find this I just needed 175 00:10:29,879 --> 00:10:37,039 tools for about thirty euros, so a multimeter, screwdriver, prying tools, 176 00:10:37,679 --> 00:10:41,320 a U s BI logic analyzer, and a usbut interface 177 00:10:42,039 --> 00:10:44,480 was all I needed to find this ulnerability. 178 00:10:45,480 --> 00:10:47,600 Speaker 2: Can you give me a little more detail? Those are 179 00:10:47,639 --> 00:10:51,200 the parts. I've never used a logic analyzer. I mean, 180 00:10:51,480 --> 00:10:53,240 how technical is that? Do I need to be an 181 00:10:53,279 --> 00:10:56,080 engineer to use a logic analyzer? How did? How did 182 00:10:56,120 --> 00:10:58,840 you go about this? What you know? Can you tell 183 00:10:58,919 --> 00:11:01,720 us a story? You know, what did you start with? 184 00:11:01,840 --> 00:11:04,279 What do you do next? What does that mean? And 185 00:11:04,720 --> 00:11:08,159 you know a blind alley went down? How did it work? 186 00:11:08,960 --> 00:11:11,240 Speaker 3: Yeah? I can walk you through the all the six 187 00:11:11,360 --> 00:11:15,399 or seven steps that led to root access, from opening 188 00:11:15,480 --> 00:11:19,039 up to the device until I was greeted with the 189 00:11:19,120 --> 00:11:26,600 root banner. Okay, And before anyone gets started with hardware 190 00:11:26,639 --> 00:11:30,440 hacking and picking a dior, before anyone gets started with 191 00:11:30,600 --> 00:11:35,159 taking a device apart, here are four electrical safety rules 192 00:11:35,200 --> 00:11:38,759 that you should follow because your own life is more 193 00:11:38,799 --> 00:11:45,320 important than your curiosity. And never ever open wall plucked 194 00:11:45,399 --> 00:11:49,440 devices because if they're directly plucked into the socket, this 195 00:11:49,639 --> 00:11:55,120 means that hazardous voltage is inside the PCB, inside the device, 196 00:11:55,200 --> 00:11:58,480 and there's the risk that you can touch a live 197 00:11:58,559 --> 00:12:04,759 wire and it's urisk and electric shock. Therefore, use devices 198 00:12:04,799 --> 00:12:08,200 that have external power adapters only so that the voltage 199 00:12:08,240 --> 00:12:13,200 conversion happens outside the area where you're working with. Also 200 00:12:13,279 --> 00:12:18,720 avoid mixing power sources. This is important for example if 201 00:12:18,759 --> 00:12:24,360 you really go into firmware extraction. And of course prevent 202 00:12:24,440 --> 00:12:27,960 in short circuits because this will fry your PCB and 203 00:12:28,000 --> 00:12:31,639 then you have a very expensive brick. But if you 204 00:12:31,639 --> 00:12:34,639 stick to these rules, you can open up the device 205 00:12:34,759 --> 00:12:39,200 and first start with a hardware reconnaissance. We'll just take 206 00:12:39,240 --> 00:12:43,960 a look on what chips are on there. And many 207 00:12:44,000 --> 00:12:47,720 industrial embedded devices they'll run on a so called system 208 00:12:47,840 --> 00:12:51,000 on chip and they are somewhere close to the system 209 00:12:51,039 --> 00:12:56,440 on chip the flash memory filmware chip. So this is 210 00:12:56,480 --> 00:13:00,519 basically where the brains and oil information are still and 211 00:13:00,799 --> 00:13:03,759 once it is and once the system is powered on, 212 00:13:04,399 --> 00:13:08,279 the system on chip pulls the filmware information from the 213 00:13:08,320 --> 00:13:13,480 firmware memory chip. If you identify these, then you take 214 00:13:13,519 --> 00:13:17,120 a look around the board. Are there any debug interfaces 215 00:13:17,320 --> 00:13:20,799 and and on this device I found the so called 216 00:13:21,159 --> 00:13:27,840 Ewert debugging interface. So with these we can move to 217 00:13:27,879 --> 00:13:31,960 the next steps. And first we do some electrical measurements 218 00:13:32,320 --> 00:13:35,879 just to prevent that our us be US BWARED and 219 00:13:36,039 --> 00:13:40,519 USB Signal analyzers get fried because they are very sensitive 220 00:13:40,559 --> 00:13:46,039 to voltage. And first things first, we first confirm the 221 00:13:46,360 --> 00:13:51,440 common electrical ground on the debug interface we identified. Once 222 00:13:51,440 --> 00:13:54,600 we identify the common ground, we know where we well 223 00:13:54,600 --> 00:13:59,960 connect the ground wire of our USB logic analyzer. Then 224 00:14:00,120 --> 00:14:04,840 a viewered interface usually has two more pins RX and TX, 225 00:14:05,039 --> 00:14:10,720 which stands for receive and transmit. And then we turn 226 00:14:10,799 --> 00:14:14,960 on the power and then measure these pins against the 227 00:14:15,279 --> 00:14:20,480 electrical common ground and in most cases we will find 228 00:14:20,519 --> 00:14:25,480 a voltage range between three walls and five walls, which 229 00:14:25,519 --> 00:14:29,360 means the devices are communicating on the so called transistor 230 00:14:29,399 --> 00:14:34,440 transistor logic level. When this is identified, we can move 231 00:14:34,519 --> 00:14:38,960 on with the logic analyzer, power off the device, connect 232 00:14:38,960 --> 00:14:42,320 the logic analyzer. We connect the logic analyzer's ground to 233 00:14:42,399 --> 00:14:46,679 the boards ground, and then the RX and TX WIA. 234 00:14:47,120 --> 00:14:52,480 Although the RX and TX pin are already labeled and 235 00:14:52,519 --> 00:14:56,120 we only could connect to the TX, it's always good 236 00:14:56,120 --> 00:14:59,639 to connect to all pins that we have a full 237 00:14:59,679 --> 00:15:03,000 picture of what's going on. Because Andrew at this board 238 00:15:03,120 --> 00:15:08,039 is what's this what's easy mode? The pins are already labeled, 239 00:15:08,080 --> 00:15:11,000 but that's not always the case sometimes you just have 240 00:15:11,120 --> 00:15:14,679 a three four five pin sticking out and you don't 241 00:15:14,720 --> 00:15:18,000 know what I mean. Then you also do the same procedure. 242 00:15:18,080 --> 00:15:21,600 You measure for the electric common ground and then start 243 00:15:21,639 --> 00:15:25,039 measuring the voyage levels, and this gives you an idea 244 00:15:25,840 --> 00:15:28,879 if you can find logical signals going on. 245 00:15:30,000 --> 00:15:32,279 Speaker 2: So thanks for that. I mean, that's you know, giving 246 00:15:32,399 --> 00:15:35,600 giving us some insight into the mechanics of dealing with 247 00:15:35,600 --> 00:15:38,759 the device. I mean, I'm I'm a software guy. I 248 00:15:38,840 --> 00:15:42,120 never have to worry about electrocuting myself if if I 249 00:15:42,200 --> 00:15:47,639 bring up a compiler on my laptop. But let me 250 00:15:47,679 --> 00:15:51,320 ask you know, you've talked about two sort of devices 251 00:15:51,360 --> 00:15:55,679 here that seem to ring a bell with me. There's 252 00:15:55,759 --> 00:16:01,600 the the you aret, which a quick question is that 253 00:16:01,679 --> 00:16:04,799 you are USB or is it RS two thirty two. 254 00:16:05,960 --> 00:16:11,600 Speaker 3: This is an RS arras thirty two connection to USB. 255 00:16:11,759 --> 00:16:17,120 So this basically converts the signal, converts the logic serial 256 00:16:17,240 --> 00:16:21,399 signals to USB so that your machine, your computer can 257 00:16:21,519 --> 00:16:22,039 work with that. 258 00:16:25,120 --> 00:16:28,120 Speaker 2: Nate real quick here, I had a very short sort 259 00:16:28,120 --> 00:16:32,519 of interaction with with Rick there asking about the difference 260 00:16:32,559 --> 00:16:34,960 between USB and RS two thirty two. For anyone who 261 00:16:35,039 --> 00:16:38,320 didn't quite track that RS two thirty two is a 262 00:16:38,559 --> 00:16:42,080 very old hardware signaling protocol. I mean I remember using 263 00:16:42,200 --> 00:16:45,159 RS two thirty two back in the day to connect 264 00:16:45,200 --> 00:16:47,120 you know, this was thirty years ago to connect to 265 00:16:47,320 --> 00:16:52,480 three hundred bits per second modems. Okay, ancient ancient technology. 266 00:16:52,679 --> 00:16:55,480 Why would there be such an ancient interface on this 267 00:16:55,600 --> 00:16:58,360 modern device, is roughly what I asked him, and he said, 268 00:16:58,399 --> 00:17:01,600 there isn't. What there is is is a USB port. 269 00:17:01,840 --> 00:17:03,919 It turns out that what he connected to, the t 270 00:17:04,240 --> 00:17:08,839 X and RX he connected to were signaling USB. And 271 00:17:09,160 --> 00:17:12,119 when he looked at the signals, he discovered that the 272 00:17:12,200 --> 00:17:16,759 messages coming across the USB were r S two thirty 273 00:17:16,759 --> 00:17:19,880 two over USB. So he looked around. He said, well, 274 00:17:20,000 --> 00:17:24,160 you know, I have a dongle that can that you know, 275 00:17:24,200 --> 00:17:26,480 can take USB and gives me RS two thirty two, 276 00:17:26,519 --> 00:17:29,079 and he connected to it and there he can see 277 00:17:29,119 --> 00:17:33,079 the messages coming across. So that's that's what's going on 278 00:17:33,160 --> 00:17:35,920 there is it's a US thirty two thirty it's a 279 00:17:36,039 --> 00:17:41,680 USB connector on the device, but the signaling is r 280 00:17:41,759 --> 00:17:47,359 S two thirty two over USB. The other one that 281 00:17:47,519 --> 00:17:50,960 that struck me, and again I'm a software guy. You 282 00:17:51,079 --> 00:17:55,119 mentioned the flash chip. You know. To me, if I 283 00:17:55,200 --> 00:17:58,359 get what's on the flash, I can start looking at instructions, 284 00:17:58,400 --> 00:18:02,519 I can start running my business. Embler, Is it possible 285 00:18:02,640 --> 00:18:05,240 to to sort of go under the nose of the 286 00:18:05,240 --> 00:18:08,480 device and just read the flash chip? Or do you 287 00:18:08,519 --> 00:18:09,759 have to go through the front door? Do you have 288 00:18:09,799 --> 00:18:11,839 to go through the CPU in order to get access 289 00:18:11,839 --> 00:18:13,920 to the flash? No, you don't. 290 00:18:13,960 --> 00:18:17,240 Speaker 3: You also can basically perform a chip off of the 291 00:18:17,279 --> 00:18:21,119 flash chip and then read out the contents with a programmer. 292 00:18:22,119 --> 00:18:25,960 This is also possible, but at the time of my research, 293 00:18:26,039 --> 00:18:28,960 I didn't have such equipment here, so I went through 294 00:18:29,000 --> 00:18:29,640 the front door. 295 00:18:30,640 --> 00:18:33,319 Speaker 2: Okay, that's fair, So please carry on. You're you're talking 296 00:18:33,319 --> 00:18:36,279 about the u R. I interrupted you. You know, finish 297 00:18:36,440 --> 00:18:38,000 finish the story here. How how are we? 298 00:18:38,160 --> 00:18:38,279 Speaker 3: Uh? 299 00:18:38,960 --> 00:18:39,759 Speaker 2: How did you get in? 300 00:18:40,440 --> 00:18:44,319 Speaker 3: Okay? The logic analyser revealed that indeed, logical data is 301 00:18:44,519 --> 00:18:48,359 transmitted over the t xpin. So this means we can 302 00:18:48,559 --> 00:18:53,279 connect our us BURET interface and open a serial console 303 00:18:53,400 --> 00:18:54,720 to that. 304 00:18:54,720 --> 00:18:57,160 Speaker 2: That's fair. I didn't realize it USB will I mean, 305 00:18:57,200 --> 00:18:59,759 I knew USB was cereal, that's what it means, universal 306 00:19:00,119 --> 00:19:05,400 real bus, but I guess I never put two and 307 00:19:05,440 --> 00:19:08,440 two together that you could just, you know, I don't know, 308 00:19:08,480 --> 00:19:10,920 connect an RS two thirty two to it. 309 00:19:12,000 --> 00:19:15,319 Speaker 3: Well, you always need this interface device that you pluck 310 00:19:15,440 --> 00:19:18,759 between that you plug into your USB socket and then 311 00:19:18,920 --> 00:19:21,400 on the other end of the device you can connect 312 00:19:21,440 --> 00:19:25,559 it to the target device. So once the usbu WED 313 00:19:25,599 --> 00:19:29,680 interface is connected, and I started the terminal on and 314 00:19:29,720 --> 00:19:33,119 I started a serial console on my Linux machine. Then 315 00:19:33,160 --> 00:19:36,960 I powered up the device again. I could see the 316 00:19:37,119 --> 00:19:41,079 bootlock flashing in front of the screen. You know, it 317 00:19:41,160 --> 00:19:45,160 was a basic Linux lute bop. It was a very 318 00:19:45,759 --> 00:19:48,799 It was a basic Linux bootlock, and at the very 319 00:19:48,920 --> 00:19:51,640 end there was a lock in prompt to lock into 320 00:19:51,680 --> 00:19:55,319 this device. This is where it got interesting. And here 321 00:19:55,400 --> 00:19:59,119 my curiosity was really on fire because I really wanted 322 00:19:59,119 --> 00:20:01,559 to get into this d and I started to look 323 00:20:01,599 --> 00:20:06,000 at the bootlock itself. First. Here I learned that the 324 00:20:06,039 --> 00:20:11,200 filmware memory is partitioned into several partitions. And if you 325 00:20:11,279 --> 00:20:16,200 look at the common IoT hardware hacker courses, then they 326 00:20:16,240 --> 00:20:18,839 always tell you to go for the root FS file 327 00:20:18,920 --> 00:20:22,839 system because that's where all the binaries are stored off 328 00:20:22,839 --> 00:20:26,400 this Linux device. But there was another petition that was 329 00:20:26,559 --> 00:20:29,920 interesting for me. This was the so called factory partition. 330 00:20:31,119 --> 00:20:34,119 Scrawling further up in the bootlock, there was also a 331 00:20:34,160 --> 00:20:37,799 brief prompt to pass a There was also a brief 332 00:20:37,839 --> 00:20:42,160 prompt to press space bar to enter the bootloader, but 333 00:20:42,319 --> 00:20:45,000 andrew the timing for this was so narrow that it 334 00:20:45,079 --> 00:20:48,200 was almost impossible to hit the timing right to enter 335 00:20:48,240 --> 00:20:52,039 the bootloader. You know, you can imagine I was jamming 336 00:20:52,119 --> 00:20:55,160 the I was hammering the space bar like a lunatic. 337 00:20:56,079 --> 00:20:58,960 And then maybe at the fourth or fifth time, I 338 00:20:59,000 --> 00:21:01,720 succeeded to get the timing right, and then I was 339 00:21:02,000 --> 00:21:06,200 presented with the next option to choose the operation. And 340 00:21:06,279 --> 00:21:10,440 here a very interesting option was presented to me. By 341 00:21:10,440 --> 00:21:13,039 pressing the number four, I would be able to enter 342 00:21:13,759 --> 00:21:17,759 the boot command line interface. And this was something what 343 00:21:17,799 --> 00:21:20,519 I was interested in and wanted to go. But with 344 00:21:20,880 --> 00:21:26,960 this narrow timing, I turned to chat GPT, asking it 345 00:21:27,000 --> 00:21:29,400 is this way. Is there a way that I can 346 00:21:29,480 --> 00:21:33,440 automate the keypresses and can send a space bar press 347 00:21:33,480 --> 00:21:37,519 and a number four press at rapid speed? The AI 348 00:21:38,240 --> 00:21:43,720 gave me a five line shell script code which uses 349 00:21:44,359 --> 00:21:47,759 an on board tool of Kelly Linux to send space 350 00:21:47,839 --> 00:21:51,759 bar and number fours, and this immediately lended me into 351 00:21:51,799 --> 00:21:57,000 the bootshell. And the bootshell of this device is based 352 00:21:57,039 --> 00:22:01,000 on the U boot bootloader, and all the hardware hackers 353 00:22:01,000 --> 00:22:03,480 out there that are familiar with you boot would immediately 354 00:22:03,519 --> 00:22:05,960 see that this is already a very stripped down and 355 00:22:06,599 --> 00:22:12,200 secured restricted version of you Boot. There was almost no 356 00:22:12,359 --> 00:22:16,519 way of manipulating the device, but they're left in the 357 00:22:16,559 --> 00:22:20,480 so called SPI command which enabled me to read the 358 00:22:20,519 --> 00:22:24,279 content of the factory partition. So that's what I did. 359 00:22:24,279 --> 00:22:28,759 I issued the command to read the factory petition and 360 00:22:28,799 --> 00:22:32,440 then and then it printed out the content of the 361 00:22:32,480 --> 00:22:37,599 factory petition in the hexadecimal format. And here's something really 362 00:22:37,640 --> 00:22:41,319 strange occurred to me that the data was not always 363 00:22:41,400 --> 00:22:47,279 represented in two hexadecimal digits. You know that, You know 364 00:22:47,359 --> 00:22:51,720 hexadecimal data always needs to have two digits. If not, 365 00:22:51,920 --> 00:22:57,400 the data gets misaligned and then gets corrupted. So the 366 00:22:57,440 --> 00:23:00,319 problem I was facing here is that some digits where 367 00:23:00,559 --> 00:23:05,119 or some data was represented with single digits missing the 368 00:23:05,160 --> 00:23:10,880 second digits, so the data was not usable for me. Okay, 369 00:23:10,960 --> 00:23:13,880 Then I used another script to align the data and 370 00:23:13,920 --> 00:23:19,720 then convert the text has the text hexidescimal data back 371 00:23:19,759 --> 00:23:23,599 into binary hexadescimal data, and then I was able to 372 00:23:23,839 --> 00:23:28,039 view the binary data and the ask the interpretation of that. 373 00:23:28,920 --> 00:23:34,400 And here's something really interesting stood out. There were basically 374 00:23:34,680 --> 00:23:40,640 three strings of data that at first made not really 375 00:23:40,720 --> 00:23:45,200 sense to me, but somehow felt familiar. And suddenly I 376 00:23:45,240 --> 00:23:49,240 realized this is the information which is also printed on 377 00:23:49,279 --> 00:23:53,359 the device's label on the site. I could see that 378 00:23:53,400 --> 00:23:58,640 in this factory partition of this device, the version number, 379 00:23:58,920 --> 00:24:03,319 the serial number, the device version, and the lock in 380 00:24:03,400 --> 00:24:07,359 password for the web management surface for the web management 381 00:24:07,440 --> 00:24:13,279 interface was stored. But there was another string that also 382 00:24:13,359 --> 00:24:17,759 kept me guessing and puzzling for quite a while. But 383 00:24:18,079 --> 00:24:22,640 this unknown string had the same characteristics as the weblock 384 00:24:22,680 --> 00:24:26,839 in password. It had ten characters, capital and lower case 385 00:24:26,920 --> 00:24:30,119 letters and numbers, and I tell you this had to 386 00:24:30,160 --> 00:24:35,400 be another password. So I restarted the device again and 387 00:24:35,480 --> 00:24:38,079 at the very end of the boot process, I was 388 00:24:38,160 --> 00:24:41,359 prompted for the lock in once more. I entered the 389 00:24:41,480 --> 00:24:46,000 user name route and entered this data I found inside 390 00:24:46,039 --> 00:24:49,359 the memory, and this gave me root access to the device. 391 00:24:52,680 --> 00:24:55,400 Speaker 1: Andrew, it's not that anything that Marcell said at any 392 00:24:55,400 --> 00:24:59,880 point there wasn't clear. But we've now gone a while 393 00:25:00,279 --> 00:25:03,759 and he's expressed a lot of technical steps. Can you 394 00:25:03,880 --> 00:25:08,200 just give me the big picture summary what we're talking 395 00:25:08,240 --> 00:25:10,400 about here, what he achieved and why it's important. 396 00:25:10,920 --> 00:25:15,920 Speaker 2: Absolutely, he real quick managed to connect to the boot 397 00:25:16,079 --> 00:25:22,680 shell with the space for you know, script constantly blasting, 398 00:25:22,960 --> 00:25:24,960 and he got in and discovered there was almost nothing 399 00:25:24,960 --> 00:25:27,440 he could do there, but he could look at this 400 00:25:27,559 --> 00:25:31,000 one tiny partition and he managed to you know, get 401 00:25:31,079 --> 00:25:34,240 the data, decode the data, and he looked at he said, 402 00:25:34,240 --> 00:25:37,000 you know, this looks like a serial number. It looks 403 00:25:37,319 --> 00:25:40,480 it looks like a password. And so he said, well, 404 00:25:40,680 --> 00:25:43,160 let's try it, and he reboots the device again. He 405 00:25:43,160 --> 00:25:44,960 doesn't do the space for this time. He lets it 406 00:25:45,000 --> 00:25:47,400 completely boot and it comes up and says, okay, I'm 407 00:25:47,400 --> 00:25:49,720 ready log in. Do you want to log in? And 408 00:25:49,720 --> 00:25:54,160 he said, yeah, let's log in as root and it says, well, 409 00:25:54,160 --> 00:25:56,240 what's the root password? And he says, well, here's the 410 00:25:56,279 --> 00:25:58,400 string that I saw in the partition. He enters it 411 00:25:58,440 --> 00:26:03,680 and he's in and now he's in route. It's not 412 00:26:04,279 --> 00:26:07,519 like you looked at the file system and said, oh, 413 00:26:07,599 --> 00:26:11,519 here's files. You know, look, there's a file with the 414 00:26:11,599 --> 00:26:15,240 name password. It wasn't nearly that obvious. 415 00:26:15,720 --> 00:26:18,799 Speaker 3: Now it was very well hidden, but I think also 416 00:26:18,880 --> 00:26:23,640 on purpose, because there's nothing written in this memory area 417 00:26:24,200 --> 00:26:27,519 before or after this partition. You really just find the 418 00:26:28,279 --> 00:26:31,759 version number, the serial number, the web management password, and 419 00:26:32,000 --> 00:26:36,039 well the root password. So somewhere in production when the 420 00:26:36,119 --> 00:26:41,000 device gets so to speak, gets the breath of life 421 00:26:41,000 --> 00:26:45,599 and the data for the label, at this moment, the 422 00:26:45,720 --> 00:26:48,640 data must be flashed into this filmweb memory chip. 423 00:26:49,880 --> 00:26:52,519 Speaker 2: Okay, so this is a very small partition. Then we're 424 00:26:52,559 --> 00:26:57,000 not talking, you know, tons of megabytes, We're talking tons 425 00:26:57,039 --> 00:26:57,720 of kilobytes. 426 00:26:58,480 --> 00:26:59,960 Speaker 3: Yeah, it was very small indeed. 427 00:27:00,680 --> 00:27:06,720 Speaker 2: Okay, Okay, cool. So you found the vulnerability and then 428 00:27:06,920 --> 00:27:10,920 I assume you know there's something called a responsible disclosure process. 429 00:27:10,960 --> 00:27:15,400 I assume you contacted the vendor, you contacted the government. 430 00:27:15,640 --> 00:27:17,519 What was the next step there? 431 00:27:18,200 --> 00:27:21,960 Speaker 3: So the next step was to contact the security contact 432 00:27:22,359 --> 00:27:25,480 of this company, and luckily I was already in contact 433 00:27:25,839 --> 00:27:29,680 with him on LinkedIn. So on a Sunday morning, I 434 00:27:29,680 --> 00:27:32,880 sent him a screenshot of hey, mister x y Z, 435 00:27:33,079 --> 00:27:38,119 I got root access to your ot gateway and within 436 00:27:38,160 --> 00:27:41,400 two hours he replied and said, okay, this is very concerning. 437 00:27:41,880 --> 00:27:45,119 Please send your findings and everything you have to our 438 00:27:45,240 --> 00:27:48,799 security email address, and we will look into this first 439 00:27:48,799 --> 00:27:52,279 thing Monday morning. Then I wrote a quick report attached 440 00:27:52,319 --> 00:27:56,200 the screenshots and the proof of concept video, and around 441 00:27:56,559 --> 00:28:02,400 Monday lunchtime they replied said, yes, this root password is 442 00:28:02,880 --> 00:28:08,480 uniquely generated per device and inserted here during production. But 443 00:28:08,680 --> 00:28:13,599 since everything is uniquely they kind of hinted at that 444 00:28:13,640 --> 00:28:18,240 they're accepting the risks so that the probability of this 445 00:28:18,400 --> 00:28:23,680 being exploited is rather low. They also said, if machine builders, 446 00:28:23,920 --> 00:28:29,519 integrateors operators stick to their security requirements, they do not 447 00:28:29,680 --> 00:28:32,440 see really a risk of this being exploited. 448 00:28:33,759 --> 00:28:38,400 Speaker 2: Okay, so the vendor said, you know it's a low 449 00:28:38,440 --> 00:28:43,119 priority because you know people are expected at physical security. 450 00:28:43,160 --> 00:28:45,240 No full off the street can come in, take one 451 00:28:45,240 --> 00:28:48,480 of these devices, walk away with it, pick it apart, 452 00:28:48,680 --> 00:28:52,440 bring it back. That's that's not a realistic threat. Do 453 00:28:52,519 --> 00:28:53,200 you agree with that? 454 00:28:54,039 --> 00:28:57,079 Speaker 3: Yes, totally. I agree with that from on my experience 455 00:28:57,160 --> 00:29:00,000 on the shop floor and in the field. You cannot 456 00:29:00,279 --> 00:29:03,640 just walk up to an electric cabinet, take out their device, 457 00:29:03,799 --> 00:29:07,480 screw it open, extract the root credentials and then put 458 00:29:07,519 --> 00:29:10,039 it back in with the back door you implanted right, 459 00:29:10,440 --> 00:29:12,640 This would hopefully catch some attention. 460 00:29:14,039 --> 00:29:16,799 Speaker 2: Okay, and you know, can you can you finish the 461 00:29:16,799 --> 00:29:18,799 the the thought? I mean, you wound up with a 462 00:29:18,839 --> 00:29:23,000 CVE for this, you know you've interacted with a vendor, 463 00:29:23,160 --> 00:29:25,039 then then what how do you how do you finish 464 00:29:25,079 --> 00:29:25,640 the process. 465 00:29:26,960 --> 00:29:31,680 Speaker 3: Then I contacted MITRO to file a CVE, also reported 466 00:29:32,519 --> 00:29:35,519 the things I found and the implications for this, and 467 00:29:36,720 --> 00:29:39,200 after two months the CVE was. 468 00:29:39,200 --> 00:29:42,720 Speaker 2: Assigned and at that point you're able to disclose publicly. 469 00:29:42,759 --> 00:29:43,240 Is that right? 470 00:29:43,920 --> 00:29:47,880 Speaker 3: Yes, all that being said, there is a tiny, tiny 471 00:29:47,960 --> 00:29:52,680 risk that you may receive the backdoor device. But then 472 00:29:52,720 --> 00:29:56,319 someone really must be targeting your operations. They need to 473 00:29:56,359 --> 00:29:59,960 know that you're operating such device, and if you expe 474 00:30:00,039 --> 00:30:04,000 acting a new shipment, they could intercept the shipment, open 475 00:30:04,079 --> 00:30:08,160 up the device, extract the root credentials, implant a backdoor, 476 00:30:08,440 --> 00:30:12,039 pack it back up, and ship it forward to your operations. 477 00:30:12,680 --> 00:30:16,759 So for that, if you are operating something critical, or 478 00:30:16,799 --> 00:30:21,119 if you're operating or if you're having critical infrastructure and operations, 479 00:30:21,640 --> 00:30:26,640 you should definitely opt for temper detection and protection. You know, 480 00:30:26,720 --> 00:30:31,799 some devices they have these little sticker on their warranty 481 00:30:31,920 --> 00:30:33,880 void if removed. 482 00:30:34,799 --> 00:30:37,160 Speaker 2: So fascinating stuff, at least at least to me. I've 483 00:30:37,160 --> 00:30:39,960 always wondered, you know, how some of this hardware hacking 484 00:30:40,039 --> 00:30:43,880 was done. But you know, as far as I know, 485 00:30:44,359 --> 00:30:47,680 you don't get paid to do the hardware hacking, unless 486 00:30:47,720 --> 00:30:50,160 I don't know there's a bounty or something. You know, 487 00:30:50,200 --> 00:30:53,440 how does this relate to making a living for you? 488 00:30:54,039 --> 00:30:56,359 Speaker 3: This is not my day job. And I also don't 489 00:30:56,359 --> 00:30:59,599 get paid to find these fulndabilities. Let's just say this 490 00:30:59,640 --> 00:31:04,559 is a very expensive hobby. I've been in the I've 491 00:31:04,599 --> 00:31:07,599 been in the domain of automation systems for half of 492 00:31:07,640 --> 00:31:11,599 my life, and after my work, I'm still interested, especially 493 00:31:11,599 --> 00:31:15,839 in what makes and breaks these devices. And that's also 494 00:31:16,480 --> 00:31:20,119 how my trainings got born. I took all the experience 495 00:31:20,240 --> 00:31:24,200 I made from while breaking these devices and turned them 496 00:31:24,200 --> 00:31:24,880 into training. 497 00:31:26,039 --> 00:31:28,200 Speaker 2: Okay, and you know this is what you do at 498 00:31:28,200 --> 00:31:29,960 Fox Squid. Can you go a little deeper? I mean, 499 00:31:31,599 --> 00:31:34,400 if I sign up for one of these courses, what 500 00:31:34,440 --> 00:31:35,480 are you going to lead me through? 501 00:31:36,480 --> 00:31:39,400 Speaker 3: If we stay with hardware hacking, you could sign up 502 00:31:39,440 --> 00:31:43,640 to Industrial Embedded Systems hardware penetration testing, where you will 503 00:31:43,680 --> 00:31:47,880 also go through these six or seven steps from investigating 504 00:31:47,920 --> 00:31:52,839 the PCB to hopefully getting route access. But this course 505 00:31:52,880 --> 00:31:55,759 has a unique approach because if you look at the 506 00:31:55,839 --> 00:32:00,680 IoT hardware hacking courses, you usually hack some key camera 507 00:32:00,960 --> 00:32:05,000 or a home router, but it's almost impossible to hack 508 00:32:05,039 --> 00:32:08,640 an industrial device because there is an entry barrier problem. 509 00:32:09,240 --> 00:32:12,640 First of all, this hardware is really expensive. You usually 510 00:32:12,759 --> 00:32:16,119 pay five hundred dollars or more, and it's risky because 511 00:32:16,119 --> 00:32:19,680 you can brick it and then you wasted five hundred dollars. 512 00:32:21,000 --> 00:32:24,559 To get around this, I built a custom firmware for 513 00:32:24,680 --> 00:32:28,400 a cheap ESP eighty two sixty six micro controller that 514 00:32:28,640 --> 00:32:32,960 mimics the behavior of an industrial device and introduces the 515 00:32:33,000 --> 00:32:36,319 student to the same challenges I faced. 516 00:32:36,519 --> 00:32:39,519 Speaker 2: Okay, so that's the hardware hacking. Have you got other courses? 517 00:32:39,880 --> 00:32:44,440 Speaker 3: Yes? I have my flagship course, Practical Offensive Industrial Security Essentials, 518 00:32:45,240 --> 00:32:49,960 which gives students from diverse backgrounds, whether they're automation engineers, 519 00:32:50,160 --> 00:32:57,039 IT professionals or total newcomers, a holistic introduction to industrial cybersecurity. 520 00:32:57,720 --> 00:33:00,680 Of course, there are assrom gaps that needs to be filled, 521 00:33:00,720 --> 00:33:06,599 but anyone with anyone with enough curiosity will get through 522 00:33:06,640 --> 00:33:10,359 this course or will have success with the course and 523 00:33:10,960 --> 00:33:14,480 then get a holistic understanding of industrial cybersecurity. 524 00:33:15,400 --> 00:33:17,279 Speaker 2: So if I can take you sort of on a 525 00:33:17,319 --> 00:33:22,359 side trip real quick here. You know, throughout this interview, 526 00:33:22,720 --> 00:33:28,039 I have been surprised by you personally. I mean I 527 00:33:28,039 --> 00:33:32,680 had always had a stereotype in mind for people who 528 00:33:32,920 --> 00:33:36,200 found vulnerabilities, who hacked stuff, hardware, software or whatever. The 529 00:33:36,880 --> 00:33:39,559 stereotype that I had in mind was sort of somebody 530 00:33:39,559 --> 00:33:42,240 with a big ego, somebody with an ego, you know, 531 00:33:42,519 --> 00:33:45,799 saying to themselves, I'm smarter than you are. I can 532 00:33:45,920 --> 00:33:50,160 find these problems. You know, you the vendor have messed 533 00:33:50,160 --> 00:33:54,119 it up. I always thought you needed that kind of 534 00:33:54,359 --> 00:33:58,640 attitude to be able to go in and tackle you know, 535 00:33:58,720 --> 00:34:02,079 the vendor's defenses in you know, and incorporated in the product. 536 00:34:02,200 --> 00:34:04,319 I always thought that you needed attitude. But you know, 537 00:34:04,720 --> 00:34:07,200 what's coming across from you is something different. Can you 538 00:34:07,240 --> 00:34:11,519 talk about who, you know, what do you need sort 539 00:34:11,559 --> 00:34:14,960 of in your brain, in your personality to be successful here? 540 00:34:15,639 --> 00:34:20,199 Speaker 3: Well, to make sure, you just need curiosity and persistence. 541 00:34:20,480 --> 00:34:23,639 I think people with a big ego they're more successful 542 00:34:23,719 --> 00:34:27,320 in finding more vulnerabilities. But like I said earlier, this 543 00:34:27,400 --> 00:34:30,679 is more an expensive hobby for me, so I do 544 00:34:30,800 --> 00:34:35,239 not really have the pressure to find vulnerability after vulnerability 545 00:34:35,360 --> 00:34:38,719 For me, it's more well being on this scavenger hunt 546 00:34:38,719 --> 00:34:42,400 to go away that or find a way to operate 547 00:34:42,440 --> 00:34:45,840 the device it was not intended to and then really 548 00:34:45,880 --> 00:34:48,559 find a way in. And to be honest, I also 549 00:34:48,599 --> 00:34:52,760 have a whole box of scrap OT devices where I 550 00:34:52,800 --> 00:34:56,159 did not find a vulnerability. So this is where we 551 00:34:56,199 --> 00:35:00,800 come back to the expensive hobby. I think if someone 552 00:35:00,920 --> 00:35:05,320 is understanding a bit of the domain these devices are 553 00:35:05,440 --> 00:35:10,039 operated in, and have enough curiosity and then persistence to 554 00:35:10,159 --> 00:35:14,320 stick to it, they can definitely find some vulnerabilities, or 555 00:35:14,559 --> 00:35:17,119 if not well, I can at least learn a lot 556 00:35:17,519 --> 00:35:21,440 about the devices, how they operate, and how they interact 557 00:35:21,480 --> 00:35:23,840 with other devices in the OT domain. 558 00:35:27,119 --> 00:35:33,719 Speaker 1: So, Andrew, we've been talking hardware vulnerabilities. It seems relatively serious, 559 00:35:33,840 --> 00:35:37,679 but bring it to a practical level for me, if 560 00:35:37,880 --> 00:35:43,719 I'm running an industrial site and I discover, you know, 561 00:35:43,760 --> 00:35:47,159 a hard coded issue in one of my gateways, am 562 00:35:47,199 --> 00:35:54,039 I running around red alarms, ringing to patch immediately? Or 563 00:35:54,760 --> 00:35:57,880 am I more focused on these systems and data flows 564 00:35:57,920 --> 00:36:02,039 around it that enable sort of legacy technologies to occasionally 565 00:36:02,719 --> 00:36:06,000 have vulnerabilities like this. How would you interpret it in 566 00:36:06,039 --> 00:36:07,639 the grand scheme of things in. 567 00:36:07,599 --> 00:36:09,199 Speaker 2: The grand scheme of things, there's there's sort of a 568 00:36:09,199 --> 00:36:11,519 couple of different questions. Let's let's let's pick it apart. 569 00:36:11,559 --> 00:36:15,280 What we've been talking about primarily is how to find 570 00:36:15,400 --> 00:36:19,559 these vulnerabilities. Once you've found a vulnerability, now you've got 571 00:36:19,559 --> 00:36:22,960 to ask the question, you know, a can I patch 572 00:36:23,039 --> 00:36:25,199 the system? Because if it's you know, a vulnerability in 573 00:36:25,239 --> 00:36:28,760 your safety system, well I'm sorry. The testing cost of 574 00:36:28,800 --> 00:36:31,320 the new version is going to be prohibitive. It's just 575 00:36:31,440 --> 00:36:33,719 really hard to patch some things. Other things are easier 576 00:36:33,719 --> 00:36:36,599 to patch. So can I patch it? Second question is 577 00:36:36,760 --> 00:36:40,320 do I need urgently to patch it? And that's sort 578 00:36:40,320 --> 00:36:42,480 of a different skill set. It's one skill set to 579 00:36:42,519 --> 00:36:44,840 find the vulnerability, it's a different skill set to say, well, 580 00:36:45,199 --> 00:36:48,119 how would an attacker? So it's an imagination thing, imagine 581 00:36:48,159 --> 00:36:51,800 how would an attacker use this against me? And you know, 582 00:36:51,880 --> 00:36:54,519 we talked about two scenarios for this vulnerability. One is 583 00:36:54,599 --> 00:36:57,199 physically walking up and stealing the device and taking it 584 00:36:57,239 --> 00:37:00,840 apart and putting it back, which seems not a very 585 00:37:00,920 --> 00:37:04,639 credible threat because you're going to be discovered. The second 586 00:37:04,719 --> 00:37:09,159 scenario was, you know, someone with much more resources discovers 587 00:37:09,199 --> 00:37:12,159 that you've just ordered fifty of these intercepts. The shipment 588 00:37:12,400 --> 00:37:15,079 bribes the driver to go take a long coffee break, 589 00:37:15,559 --> 00:37:17,599 breaks into you know, five or ten or fifteen of 590 00:37:17,639 --> 00:37:23,280 these devices, inserts malware, packages them all up again again, 591 00:37:24,119 --> 00:37:26,400 is that a credible threat. It's a credible threat for 592 00:37:26,639 --> 00:37:31,320 some very high value targets. Is it a credible threat 593 00:37:31,320 --> 00:37:35,400 for a small bakery? You know, probably not. So you know, 594 00:37:35,480 --> 00:37:39,039 first step is fined it. Second step is figure out 595 00:37:39,519 --> 00:37:42,840 can I even patch it? Third step is how would 596 00:37:42,960 --> 00:37:47,079 a bad guy exploit this? Are there credible threats? Is 597 00:37:47,119 --> 00:37:51,679 there a third scenario that we haven't imagined? So it's 598 00:37:52,199 --> 00:37:55,440 a question of imagination and studying what people have done 599 00:37:55,480 --> 00:37:59,960 in the past, and then you know, the decision part 600 00:38:00,119 --> 00:38:02,719 part of it is you know, how easy is this 601 00:38:02,760 --> 00:38:08,039 to exploit? So we're talking about devices generally, we're also 602 00:38:08,079 --> 00:38:11,440 talking about cloud connected devices because a lot of the 603 00:38:11,480 --> 00:38:13,880 devices that Marcelle has focused on that he teaches you 604 00:38:13,880 --> 00:38:17,159 about his industrial Internet devices, they're connected out to the cloud. 605 00:38:17,639 --> 00:38:23,079 So that's more Internet connected, more Internet exposed. But really 606 00:38:23,280 --> 00:38:27,440 what he looked at here was an O T cloud 607 00:38:27,679 --> 00:38:34,559 remote access device. It's arguably the most exposed piece of 608 00:38:34,599 --> 00:38:37,679 technology in the OT network. It's the technology that gives 609 00:38:38,440 --> 00:38:42,920 Internet based users access to the OT system. So normally 610 00:38:43,639 --> 00:38:47,199 you would set these things on automatic update. Why what 611 00:38:47,280 --> 00:38:49,920 if they blue screen, Well, nobody cares if their blue screen. 612 00:38:50,000 --> 00:38:53,199 It's inconvenient if they blue screen. If the bad guys 613 00:38:53,239 --> 00:38:56,039 get in, you know, they can work whatever they want 614 00:38:56,159 --> 00:39:00,639 sabotage on your OT network. So normally people pay a 615 00:39:00,840 --> 00:39:04,519 lot of attention to defects in there. You know, there 616 00:39:04,519 --> 00:39:07,239 are too too vulnerabilities in their O TI remote access. 617 00:39:07,519 --> 00:39:10,119 This one we just you know, we couldn't imagine a 618 00:39:10,199 --> 00:39:14,599 credible attack scenario pre mere mortals. You know, it might 619 00:39:14,639 --> 00:39:18,239 not be that that big to worry about, but generally speaking, 620 00:39:19,239 --> 00:39:21,679 you know, this is the kind of device you want 621 00:39:21,920 --> 00:39:26,199 people like Marcel picking apart the most thoroughly because this 622 00:39:26,280 --> 00:39:28,840 is the device that has to be the most thoroughly protected. 623 00:39:31,440 --> 00:39:33,159 Before we let you go, can I ask you to 624 00:39:33,239 --> 00:39:35,239 sum up for our listeners, what should we take away 625 00:39:35,280 --> 00:39:38,360 from this? What you know, what's important to know about 626 00:39:38,400 --> 00:39:40,880 this stuff, and how do we use it going forward? 627 00:39:41,400 --> 00:39:44,159 Speaker 3: Okay, looking at the vulnerability, I found this was a 628 00:39:44,400 --> 00:39:49,000 prime example that just one part of security was completely overlooked. 629 00:39:49,440 --> 00:39:51,760 When you look at the device from a network perspective, 630 00:39:51,840 --> 00:39:57,000 you see a very fortified device, but security doesn't stop 631 00:39:57,000 --> 00:39:59,880 at the network in our face and also the PC 632 00:40:00,079 --> 00:40:04,599 be the hardware level should be taken into consideration. And 633 00:40:04,639 --> 00:40:08,599 in general, I think OTI security needs more curious minds 634 00:40:08,599 --> 00:40:12,920 that are looking under the hood. For example, if you're 635 00:40:12,920 --> 00:40:17,960 an engineer, you already understand the industrial processes and here 636 00:40:18,039 --> 00:40:21,800 I just can recommend you to level up your cybersecurity skills. 637 00:40:22,840 --> 00:40:26,719 And this is exactly what I'm doing with Foxgrit. This 638 00:40:26,840 --> 00:40:31,639 platform exists to teach industrial security in an affordable and 639 00:40:31,840 --> 00:40:38,119 practical way. The flagship cause Practical Offensive and Practical Offensive 640 00:40:38,440 --> 00:40:43,000 Industrial Security Essentials comes with an open source lab where 641 00:40:43,000 --> 00:40:46,840 you not only learn about penetration testing tools, but also 642 00:40:47,400 --> 00:40:51,639 how you can use them on simulated industrial controllers and 643 00:40:51,719 --> 00:40:55,920 that way you also can understand how your real devices 644 00:40:55,960 --> 00:41:01,559 would behave under such conditions. For the next steps. If 645 00:41:01,559 --> 00:41:05,960 you're curious, check out Fox Sprit for resources and connect 646 00:41:05,960 --> 00:41:09,599 with me on LinkedIn, and of course keep pushing OT 647 00:41:09,800 --> 00:41:10,760 security forward. 648 00:41:14,119 --> 00:41:17,039 Speaker 1: So that seems suggest about do it. For your interview, 649 00:41:17,079 --> 00:41:20,559 Andrew with Marcel rick Zen, do you have any final 650 00:41:20,639 --> 00:41:23,280 thoughts that you'd like to share before we leave today. 651 00:41:23,400 --> 00:41:26,960 Speaker 2: I guess so. I mean, I had always been curious, 652 00:41:27,079 --> 00:41:31,760 you know, how people do this stuff. What surprised me 653 00:41:31,840 --> 00:41:35,239 about the interview here was that I actually followed what 654 00:41:35,280 --> 00:41:38,280 he did. I kind of understood it. You know, I 655 00:41:38,280 --> 00:41:40,360 thought it'd be harder than that, and I suppose it 656 00:41:40,400 --> 00:41:42,280 could be if you have to, you know, if you 657 00:41:42,280 --> 00:41:44,239 don't have a small amount of information to look at, 658 00:41:44,280 --> 00:41:46,039 if you've got to look at the entire firmware and start, 659 00:41:46,159 --> 00:41:49,840 I don't know, disassembling megabytes of firmware looking for vulnerabilities, 660 00:41:50,880 --> 00:41:55,320 that would strike me as harder. You know, this seemed 661 00:41:55,920 --> 00:42:00,719 really straightforward. You know, I don't know if I don't know, 662 00:42:00,719 --> 00:42:04,000 if I'm curious enough about, you know, how this stuff 663 00:42:04,039 --> 00:42:06,039 works that I would do the work myself. But I 664 00:42:06,079 --> 00:42:08,199 sure wouldn't mind another two or three guests like this 665 00:42:08,320 --> 00:42:10,960 to walk us through how they did the hard work, 666 00:42:11,679 --> 00:42:14,800 so that you know, we can satisfy our curiosity and 667 00:42:14,920 --> 00:42:18,639 beyond my curiosity. You know, I agree with Marcel. We 668 00:42:18,719 --> 00:42:22,840 need people tracking down vulnerabilities. That's you know, it's because 669 00:42:22,840 --> 00:42:27,760 that's the good way to persuade vendors to invest more 670 00:42:27,760 --> 00:42:30,480 in security, to you know, make these devices more secure. 671 00:42:30,519 --> 00:42:32,800 To begin with is to point out afterwards they've got 672 00:42:32,840 --> 00:42:35,280 problems and you know, the next time around, hopefully they 673 00:42:35,320 --> 00:42:38,559 will be more careful. The bad way is to wait 674 00:42:38,599 --> 00:42:40,760 for the bad guys to find the vulnerabilities and exploit 675 00:42:40,760 --> 00:42:42,920 them and take advantage of us. So, you know, we 676 00:42:42,960 --> 00:42:45,199 need more of the good guys. You know, we need 677 00:42:45,239 --> 00:42:49,599 more more technical, curious people out there fight in the fight. 678 00:42:49,760 --> 00:42:51,280 So you know, thank you to Marcel. 679 00:42:52,039 --> 00:42:56,719 Speaker 1: Well, thanks to Marcel for satisfying our curiosity. And Andrews 680 00:42:56,760 --> 00:42:58,320 always thank you for speaking to me. 681 00:42:58,920 --> 00:43:00,000 Speaker 2: It's always a pleasure. Thank you. 682 00:43:01,039 --> 00:43:05,280 Speaker 1: This has been the Industrial Security Podcast from Waterfall. Thanks 683 00:43:05,320 --> 00:43:07,199 to everyone out there listening.