1 00:00:04,360 --> 00:00:10,320 The genesis of Aurora started with Mike and others motivating us to ask the question, 2 00:00:10,759 --> 00:00:16,000 what are some interesting accidents that have taken place relative to control systems and 3 00:00:16,039 --> 00:00:29,559 infrastructure. Welcome everyone to the Industrial Security Podcast. My name is Nate Nelson. 4 00:00:29,760 --> 00:00:33,920 I'm here as usual with Andrew Ginter, the vice president of Industrial Security 5 00:00:34,159 --> 00:00:38,240 at Waterfall Security Solutions. He's going to introduce the subject and guests of our 6 00:00:38,280 --> 00:00:42,320 show today. Andrew, how are you. I'm very well, Thank you, 7 00:00:42,399 --> 00:00:45,880 Nate. Our guest today is Aaron Turner. He is part of the 8 00:00:45,880 --> 00:00:50,640 faculty at Ion's Research. I a n S Research. You know these people, 9 00:00:51,159 --> 00:00:56,520 they do managerial they do SISO training, and you know our topic today's 10 00:00:56,679 --> 00:01:02,759 failures of imagination from the nine to eleven attacks through the Aurora demo. Aaron 11 00:01:02,920 --> 00:01:07,319 was instrumental in the history the genesis of the industrial security field, and he's 12 00:01:07,359 --> 00:01:11,319 going to tell us a bit about how this all came to be. Then, 13 00:01:11,359 --> 00:01:18,040 without further Ado, here's your conversation with Aaron. Hello, Aeron, 14 00:01:18,120 --> 00:01:22,840 and thank you for joining us before we Before we get started, can you 15 00:01:22,840 --> 00:01:26,159 say a few words for our listeners about yourself and about the good work that 16 00:01:26,200 --> 00:01:34,040 you're doing at Ions. Yeah, thank you for this opportunity to talk about 17 00:01:34,120 --> 00:01:38,159 the history of cybersecurity. It's something I'm really passionate about. I've been doing 18 00:01:38,239 --> 00:01:45,719 some form of breaking into systems or hardening systems since early nineteen nineties, and 19 00:01:46,560 --> 00:01:49,879 I got my starting a penetration tester, but caught a lucky break in the 20 00:01:49,959 --> 00:01:57,480 late nineties to join Microsoft security teams. And today I work at Ion's Research 21 00:01:57,560 --> 00:02:00,719 as a faculty, and what that means is I try to help people take 22 00:02:00,760 --> 00:02:07,159 a non vendor driven approach to solving problems, and the Iron's Research has been 23 00:02:07,200 --> 00:02:08,960 a great platform to help me do that. I work with over six hundred 24 00:02:08,960 --> 00:02:15,039 customers around all sorts of different industries and it's a great form for me to 25 00:02:15,080 --> 00:02:19,759 just get access to great information and collaborate people without the filter that we have. 26 00:02:20,120 --> 00:02:27,159 Sometimes our topic is failures of imagination, I mean, in my dim 27 00:02:27,240 --> 00:02:32,319 understanding, you know, third and fourth hand, the industrial control system, 28 00:02:32,360 --> 00:02:37,360 the Skata Security Initiative, if you like, it started after nine eleven. 29 00:02:37,439 --> 00:02:42,240 Nine eleven was a physical assault on the World Trade Center. But in the 30 00:02:42,240 --> 00:02:50,360 months after I'm told that authorities around the world looked around and said that that 31 00:02:50,560 --> 00:02:53,800 was unexpected, that was a failure of imagination. Where else have we failed? 32 00:02:54,159 --> 00:03:00,560 And one of the ways that I'm told came back was industrial cyber security. 33 00:03:00,599 --> 00:03:04,360 And you know, whereas before the turn of the century there might have 34 00:03:04,400 --> 00:03:07,560 been a dozen people on the planet looking at the topic, mostly in universities 35 00:03:07,639 --> 00:03:14,159 academics, it became a mainstream concern. This is you know, that's it. 36 00:03:14,240 --> 00:03:17,400 That's my depth of understanding. I understand that you were part of that 37 00:03:17,520 --> 00:03:21,800 process. Can you talk about that sort of the next level of detail? 38 00:03:21,879 --> 00:03:28,439 You know, what what did it look like from the inside. Yeah. 39 00:03:28,520 --> 00:03:32,199 When I was asked to join Microsoft in nineteen ninety eight, I joined an 40 00:03:32,280 --> 00:03:37,560 organization that didn't really have a clear focus on security, but that focus had 41 00:03:37,560 --> 00:03:40,400 to get sharpened over time. And because I also have a little bit of 42 00:03:40,400 --> 00:03:45,719 training in the law and the law school dropout, I would often be paired 43 00:03:45,960 --> 00:03:50,719 with law enforcement to go try to solve tough problems, tough questions. And 44 00:03:50,800 --> 00:03:53,919 so by the time nine to eleven happened in two thousand and one, I 45 00:03:53,919 --> 00:03:59,039 had already developed strong relationships with the Secret Service and Department Justice, DEA, 46 00:03:59,280 --> 00:04:03,159 FBI, And so when they came to me and said, Aaron, what's 47 00:04:03,240 --> 00:04:08,439 the craziest thing you could think about happening as the result of computer problems. 48 00:04:09,120 --> 00:04:13,080 Well, this was in light of the fact that I had just helped the 49 00:04:13,199 --> 00:04:18,160 FBI Kart Lab to do some investigative research on the laptops associated with the DC 50 00:04:18,279 --> 00:04:23,279 Sniper. That same lab was the one that did some of the analysis on 51 00:04:23,360 --> 00:04:28,879 the laptops that Daniel Pearl purchased in Pakistan that were used by Muhammadada and others 52 00:04:28,879 --> 00:04:33,240 to do flight simulator training into the World Trade Center. And so as I 53 00:04:33,319 --> 00:04:36,879 sat back and said, Okay, what would be the thing that I would 54 00:04:36,879 --> 00:04:42,279 do, I said, you know, whenever I've worked with folks who embed 55 00:04:42,319 --> 00:04:48,199 computers into systems to do good, very rarely do those engineers have or whether 56 00:04:48,240 --> 00:04:54,439 you'd call it the malicious imagination or the threat modeling mindset to go what's the 57 00:04:54,480 --> 00:05:00,120 worst thing that could happen? And my background in that area came from a 58 00:05:00,120 --> 00:05:02,519 side project that I was working on at Microsoftware for a period of time. 59 00:05:02,560 --> 00:05:10,879 I would help the licensees of Windows XP embedded evaluate how that embedded system was 60 00:05:10,879 --> 00:05:15,079 being used. So, for example, in a medical imaging system, they 61 00:05:15,120 --> 00:05:18,519 had decided to embed a Windows XP subsystem into that large medical imager. It 62 00:05:18,600 --> 00:05:24,720 was an MRI system, and in MRIs, you have these massive magnets that 63 00:05:24,879 --> 00:05:30,160 rely on polarizing the human body and water in ways to get those images. 64 00:05:30,759 --> 00:05:33,519 Well, when someone showed me that, my first thought was, I guess 65 00:05:33,519 --> 00:05:38,040 being somewhat broken inside, being a bad kid, or I guess just having 66 00:05:38,040 --> 00:05:41,759 an evil imagination. I said, well, wouldn't it be funny if you 67 00:05:41,800 --> 00:05:44,399 know, you reverse the plarity on one side of the magnets and you turn 68 00:05:44,439 --> 00:05:47,199 that MRI into a human meed grinder. And they didn't think that was very 69 00:05:47,199 --> 00:05:51,279 funny. In fact, the response from the engineers on that project were like, 70 00:05:51,879 --> 00:05:57,360 you're sick, you're broken. And my response to them was that, 71 00:05:57,439 --> 00:06:00,759 okay, well I might be broken, but you have to think you've got 72 00:06:00,800 --> 00:06:04,000 to apply threat models to the way you embed these systems. And so that 73 00:06:04,079 --> 00:06:10,759 began a journey that I went down, and it was really sharpened with some 74 00:06:10,800 --> 00:06:15,360 interactions that I had through CSO Magazine. Bob Bragden, the publisher of CSO 75 00:06:15,399 --> 00:06:19,240 Magazine, put together a working group probably around two thousand and three two thousand 76 00:06:19,240 --> 00:06:23,920 and four timeframe, where I was introduced to a man named Mica Sante. 77 00:06:23,959 --> 00:06:27,680 Mica Sante at the time was working for American Electric Power. He was the 78 00:06:27,720 --> 00:06:32,240 CISO there. He had just cleaned up a major disruption that had happened in 79 00:06:32,519 --> 00:06:38,120 his grid that coincided with a major incident that Microsoft had had in August of 80 00:06:38,160 --> 00:06:42,079 two thousand and three, and so we started collaborating in ways, and I 81 00:06:42,120 --> 00:06:45,959 really found an affinity of working with Mike that we sort of both were, 82 00:06:46,680 --> 00:06:50,160 I guess broken in our own way, and it was a really interesting opportunity 83 00:06:50,199 --> 00:06:56,120 to start to ask those difficult questions of what's the worst thing that can happen 84 00:06:56,199 --> 00:07:00,079 if we started embedding distributed computers and something else that had all these different in 85 00:07:00,079 --> 00:07:03,120 two thousand and three was the Northeast blackout. Millions of people without power for 86 00:07:04,240 --> 00:07:09,759 hours, some of them I think possibly for days, but most of them 87 00:07:09,800 --> 00:07:15,000 I think it was restored within twenty four hours. The post mortem analysis on 88 00:07:15,120 --> 00:07:19,240 that said that, you know, in my understanding, if I my recollection 89 00:07:19,279 --> 00:07:24,120 I read the thing years ago, said that it was like a memory leak 90 00:07:24,279 --> 00:07:29,920 in an alarm server. Alarms were delayed that could have told the operators there 91 00:07:29,959 --> 00:07:32,120 was a problem, and they could have, you know, taken preventive corrective 92 00:07:32,160 --> 00:07:36,319 action to prevent the blackout, but they didn't. See the alarms. Because 93 00:07:36,319 --> 00:07:41,560 of this failure, there was widespread speculation that it was a cyber attack. 94 00:07:42,000 --> 00:07:49,000 You were involved in that as well. How what happened there? Yes, 95 00:07:49,079 --> 00:07:54,519 in August of two thousand and three, so twenty years ago now, there 96 00:07:54,560 --> 00:07:58,000 was an event on the Microsoft side of things. It was called the blaster 97 00:07:58,120 --> 00:08:03,040 Worm. The blaster Worm, over the course of several days, infected over 98 00:08:03,120 --> 00:08:07,360 two billion computers around the world with an attack package that was designed to try 99 00:08:07,360 --> 00:08:13,800 to take down Windows Update. So basically the attackers wanted to disable the ability 100 00:08:13,800 --> 00:08:18,240 to people let people fix the problem. So we were focused on the blaster 101 00:08:18,319 --> 00:08:22,600 instant and it was so bad that, you know, the inbound support cues 102 00:08:22,600 --> 00:08:26,120 at Microsoft were overloaded and we're having trouble going through, you know, and 103 00:08:26,399 --> 00:08:33,519 actually helping people get get help. Well, that was the same time when 104 00:08:33,960 --> 00:08:41,279 there was this accident in American Electric Power switchyard that caused this series of events 105 00:08:41,360 --> 00:08:45,600 that pushed you know, those substations into a safe state, and a safe 106 00:08:45,639 --> 00:08:52,039 state is disconnected. Well as a result of that, plus the network being 107 00:08:52,120 --> 00:08:56,519 congested from the blaster traffic between sites and within the enterprise network in American Electric 108 00:08:56,600 --> 00:09:03,559 Power, it probably serve is a contributing factor. Now, in the haze 109 00:09:03,000 --> 00:09:09,240 of digital uncertainty, that is, were these massive events and incidents, there 110 00:09:09,240 --> 00:09:15,639 were some people within government that suggested that maybe the Microsoft impacted worm, the 111 00:09:15,679 --> 00:09:18,480 blaster worm, had something to do with the power grid. Now, eventually, 112 00:09:18,559 --> 00:09:22,639 as you mentioned, it was traced back to a system failure that was 113 00:09:22,679 --> 00:09:28,679 not related to the Microsoft operating system problem. But it probably was a contributing 114 00:09:28,720 --> 00:09:35,720 factor in the delay in response, and it probably forced that outage to grow 115 00:09:35,799 --> 00:09:39,840 longer than it should have for some people. But that was another period of 116 00:09:39,840 --> 00:09:43,399 time when you know, myself, Mike and other people basically sat down and 117 00:09:43,440 --> 00:09:46,159 said, wow, this was an accident. What if somebody did that on 118 00:09:46,200 --> 00:09:52,039 purpose? Like, what would happen if someone decided to go and manipulate a 119 00:09:54,039 --> 00:09:58,960 digital network in a way that reduced the fidelity or the reliability or the integrity 120 00:10:00,480 --> 00:10:03,600 of the network that was controlling things like the power grid or cell phone networks 121 00:10:03,720 --> 00:10:11,279 or water delivery systems or whatever it may be. And so in that world 122 00:10:11,320 --> 00:10:16,720 where we had proof that blastered impaired the restart on the IT side, then 123 00:10:18,320 --> 00:10:22,480 maybe control systems needed to be thought about it in a new threat model, 124 00:10:22,559 --> 00:10:26,399 what's the trust relationship between it and OT and what kinds of boundaries should be 125 00:10:26,440 --> 00:10:31,759 there, And it sort of served as a genesis for myself and Mike and 126 00:10:31,799 --> 00:10:35,720 others to start asking those questions. I only would have been seven years old 127 00:10:35,720 --> 00:10:39,320 at the time, but I distinctly remember that Northeast blackout. My family was 128 00:10:39,360 --> 00:10:45,039 taking a trip to Canada and on the way back, we stopped in an 129 00:10:45,080 --> 00:10:52,080 ice cream place, not realizing that half of the Northeast was totally in darkness 130 00:10:52,480 --> 00:10:56,240 and they were giving away free ice cream because it was all melting. Yeah. 131 00:10:56,320 --> 00:11:00,600 I mean, that was a big event, and in the heat of 132 00:11:00,600 --> 00:11:05,879 the moment, in the weeks that followed the event, there was widespread speculation, 133 00:11:05,720 --> 00:11:09,279 you know, that this was a cyber attack. I remember, you 134 00:11:09,279 --> 00:11:16,840 know, reading these reports, and you know, the bizarre thing is I 135 00:11:16,879 --> 00:11:22,080 started, I got into sort of the public. I started interacting with the 136 00:11:22,080 --> 00:11:28,799 public on cybersecurity almost a decade later, sort of in the eight nine timeframe, 137 00:11:28,320 --> 00:11:33,440 and I remember, you know, into the middle of the teens. 138 00:11:33,440 --> 00:11:37,960 We're talking twenty fourteen, twenty fifteen. I remember, this is almost you 139 00:11:37,000 --> 00:11:41,799 know, it's more than a decade after the event. I remember experts standing 140 00:11:41,879 --> 00:11:48,080 up in public saying that the two thousand three blackout was a cyber attack, 141 00:11:48,000 --> 00:11:52,399 you know, and one after another I tapped these people on their shoulder and 142 00:11:52,440 --> 00:11:56,440 say, have you read the report. This is a decade later, and 143 00:11:56,480 --> 00:12:01,399 you're spreading misinformation. I mean, this was again such widespread speculation that that 144 00:12:03,039 --> 00:12:05,399 you know, a decade later, people were still talking about the cyber attack 145 00:12:05,440 --> 00:12:09,080 when in fact, it was a failure. It was a you know, 146 00:12:09,080 --> 00:12:13,840 equipment failure, was a software failure. The the alarm server eventually rebooted spit 147 00:12:13,840 --> 00:12:18,759 out all the alarms, but it was too late by then. So yeah, 148 00:12:18,799 --> 00:12:24,879 this this, uh, And what I didn't realize until just now speaking 149 00:12:24,879 --> 00:12:28,399 to Aaron, is that the blaster worm did have a role. It did 150 00:12:28,440 --> 00:12:33,919 not cause the outage, but in his estimation, it impaired the response and 151 00:12:33,960 --> 00:12:39,240 may have delayed the you know, may have may have prolonged the blackout for 152 00:12:39,320 --> 00:12:43,600 some customers by you know, up to a handful of hours because it delayed 153 00:12:43,639 --> 00:12:52,000 response because communications facilities were all messed up, failures of imagination, concerns about 154 00:12:52,080 --> 00:13:00,000 you know, laptops, and nine to eleven concerns about blaster possibly having connections 155 00:13:00,320 --> 00:13:03,919 to the two thousand and three blackout. What was next what you know, 156 00:13:03,600 --> 00:13:11,000 It sounds like you and Michaelasante were identifying the problem. You know, we 157 00:13:11,039 --> 00:13:18,080 need a solution. You know what, what did you do with the problem? 158 00:13:18,200 --> 00:13:22,480 Well, I think we really need to make sure that we attribute the 159 00:13:22,559 --> 00:13:28,600 first action to Mike. He he had the guts. He had a pretty 160 00:13:28,600 --> 00:13:31,840 good job at American Electric Power, Like he was one of the first CISOs. 161 00:13:31,840 --> 00:13:37,360 He was featured as I think CISO of the Year by several publications, 162 00:13:37,399 --> 00:13:41,360 and so, you know, he he had a pretty cush life, like 163 00:13:41,399 --> 00:13:43,480 he could have just gone on that path, but what he decided to do 164 00:13:45,159 --> 00:13:48,600 was to take a risk, and he approached some folks at the Department Energy 165 00:13:48,159 --> 00:13:56,279 and basically asked them the question, could we build a research test bed to 166 00:13:56,320 --> 00:14:01,960 prove out some of these theories? Can we move from speculation to actual data 167 00:14:03,039 --> 00:14:05,399 that would show us, you know, what's the actual impact? And how 168 00:14:05,440 --> 00:14:09,960 do we protect these things? And so Mike's first miracle, I'll say, 169 00:14:11,000 --> 00:14:16,720 to get this project started was convincing the folks at DOE to combine forces with 170 00:14:16,759 --> 00:14:20,399 the Departmental Land Security, which just oftentimes hard in the federal government. Sometimes 171 00:14:20,559 --> 00:14:26,200 people don't like to play nicely with each other and basically set up this test 172 00:14:26,279 --> 00:14:31,080 lab out at the Idaho National Lab. Now, he brought a few other 173 00:14:31,080 --> 00:14:37,159 people along for the ride, Very Coonley and other you know, really interesting 174 00:14:37,559 --> 00:14:43,519 a wide variety of folks, power engineers and cyber people and military folks, 175 00:14:43,519 --> 00:14:46,399 and it was just a really good conglomerate conglomeration of people that he brought together. 176 00:14:48,120 --> 00:14:50,679 And in two thousand and six he invited me to come along for the 177 00:14:50,759 --> 00:14:56,399 ride, and I felt supremely honored. It's like, oh, there's sort 178 00:14:56,399 --> 00:15:01,720 of like this cast of characters from different parts of the universe that are coming 179 00:15:01,759 --> 00:15:05,759 together to trying to solve a tough problem. And it was going to be 180 00:15:05,799 --> 00:15:11,240 a sacrifice. I mean, moving from a company like Microsoft at going and 181 00:15:11,279 --> 00:15:16,320 getting in a federal government job wasn't exactly the easiest thing to convince my wife 182 00:15:16,360 --> 00:15:20,960 to do. It wasn't the easiest thing on my personal finances trajectory, but 183 00:15:22,159 --> 00:15:24,639 it was the right thing to do. And so I moved my family from 184 00:15:26,519 --> 00:15:30,240 Seattle, the suburbs of Seattle where we were living, to Idaho, and 185 00:15:30,919 --> 00:15:35,440 we start on this project to basically say, how do we put our brains 186 00:15:35,480 --> 00:15:39,639 together to prove to the world that this is really a problem, and so 187 00:15:39,799 --> 00:15:43,600 we started to go out and do a sort of marketing show to go pitch 188 00:15:43,679 --> 00:15:48,879 for funding because we had the facility, but we didn't necessarily have the funding 189 00:15:48,919 --> 00:15:52,200 to actually run a full test, and so we would fly from Idaho out 190 00:15:52,240 --> 00:15:54,960 to Washington, d C. You know, usually Sunday night, we'd get 191 00:15:54,960 --> 00:15:58,519 into DC, we'd set up meetings Monday through Friday, and then fly back 192 00:15:58,519 --> 00:16:03,200 Friday night. So that was our rhythm, as you know, essentially spending 193 00:16:03,200 --> 00:16:06,440 the whole week, got in DC pitching the people saying, Hey, we've 194 00:16:06,440 --> 00:16:10,840 got this idea, can we get some help to fund it? And we 195 00:16:10,919 --> 00:16:17,519 wandered from civilian agencies like Dewey and DHS into the Pentagon and into some crazy 196 00:16:17,559 --> 00:16:22,120 places in the intelligence community, and you know, we're essentially just kind of 197 00:16:22,159 --> 00:16:25,080 kind of got hat in hand looking for the resources we need to put this 198 00:16:25,080 --> 00:16:32,240 thing together. There was some tough experiences along that path. I can remember 199 00:16:32,279 --> 00:16:34,639 one time in the Pentagon when we got to invite it in to give a 200 00:16:34,679 --> 00:16:41,240 briefing, and during that briefing, an individual fairly rudely stood up in the 201 00:16:41,240 --> 00:16:45,360 middle of the briefing and just turned his back and was walking out. And 202 00:16:45,440 --> 00:16:48,360 before he walked out. He said, you know, if I if I 203 00:16:48,399 --> 00:16:52,799 want to go kinetic, I'll call in artillery. So this was a senior 204 00:16:53,120 --> 00:16:57,000 army official. And because what we were pitching in our talk was, hey, 205 00:16:57,440 --> 00:17:02,919 maybe digital attack can have these physical consequences. Maybe you could actually, 206 00:17:03,240 --> 00:17:08,200 you know, severely disable fighting for us by eliminating the support of the infrastructure 207 00:17:08,200 --> 00:17:14,799 that's around them. And there were some other people who basically said, you 208 00:17:14,839 --> 00:17:17,519 and your you and your R two D two language. You know, you 209 00:17:17,559 --> 00:17:21,039 guys can go off and play video games or whatever. And so we didn't 210 00:17:21,079 --> 00:17:25,480 have the most receptive audience. This was two thousand and six time frame. 211 00:17:26,599 --> 00:17:33,920 Now luckily there were some folks who listened. We finally found some listening ears 212 00:17:33,960 --> 00:17:36,920 inside of the Pentagon, inside of DHS, inside of DEE, where essentially 213 00:17:36,960 --> 00:17:40,240 combined forces said, look, we were going to put together the budget where 214 00:17:40,240 --> 00:17:45,480 we can do one test to really show what this thing can do and and 215 00:17:45,559 --> 00:17:48,599 all of that hard work that Mike could work for for years, and that 216 00:17:48,839 --> 00:17:51,400 I got to go along from the ride on, several others got to pitch. 217 00:17:52,000 --> 00:17:56,559 You know, we finally got the resources to then start dreaming up the 218 00:17:56,599 --> 00:18:00,839 tests that we were going to do and That's when we went back to Idaho 219 00:18:00,960 --> 00:18:03,720 to kind of put our heads together to say what's the best thing we can 220 00:18:03,759 --> 00:18:07,640 do and how do we actually live the Aurora test? Was it not? 221 00:18:07,839 --> 00:18:12,519 I mean, the the the test was controversial. I remember a video leaked 222 00:18:12,519 --> 00:18:18,240 and just about everything else was confidential. You know, you were on you 223 00:18:18,279 --> 00:18:21,960 were on the inside of that. You know, where did where did Aurora 224 00:18:22,079 --> 00:18:25,759 come from? What was it really and sort of what what can you tell 225 00:18:25,839 --> 00:18:27,559 us? What can you I mean, what can you tell us today about 226 00:18:27,599 --> 00:18:37,160 what happened behind the scenes there. The genesis of Aurora started with Mike and 227 00:18:37,240 --> 00:18:42,079 others motivating us to ask the question, what are some interesting accidents that have 228 00:18:42,160 --> 00:18:51,000 taken place relative to control systems and infrastructure? And we canvassed all over North 229 00:18:51,000 --> 00:18:56,559 America and we ended up having a conversation with uh, a Canadian power engineer 230 00:18:56,599 --> 00:19:00,440 who told us a story and I don't know how apocryphal was, but he 231 00:19:00,839 --> 00:19:03,400 told the story of, yeah, one time someone tried to bring a coal 232 00:19:03,440 --> 00:19:07,920 fire power plant online and the power was out of phase and ended up blowing 233 00:19:07,960 --> 00:19:12,519 this coal fire facility up and everything had to get fixed and oh interesting, 234 00:19:12,680 --> 00:19:19,359 Okay, So this aspect of large scale generating facility trying to link into the 235 00:19:19,400 --> 00:19:22,559 grid and the power being out of phase. That was bad. So we 236 00:19:23,079 --> 00:19:27,880 started to look at that, and then in conjunction with that research, we 237 00:19:27,920 --> 00:19:33,440 started to look at, well, what are the digital components that marry these 238 00:19:33,839 --> 00:19:38,599 generation and transmission and delivery capabilities together. And we started to zero in on 239 00:19:40,200 --> 00:19:45,680 these these safety relays, these these relays that sit inside of the substations that 240 00:19:45,759 --> 00:19:51,720 really serve as those those breakpoints where you can shut stuff down and stuff' out 241 00:19:51,720 --> 00:19:56,720 of whack and you can try to marry stuff together. And in looking at 242 00:19:56,720 --> 00:20:04,559 that particular technology, it it was very ripe for cyber attacks because the original 243 00:20:04,680 --> 00:20:11,960 inventors of those pieces of those relays, they did not really do a good 244 00:20:11,200 --> 00:20:17,039 cyber threat model. So they had things like hard coded use names and passwords 245 00:20:17,079 --> 00:20:22,160 and always open network connections and just stuff that you didn't want connected to the 246 00:20:22,200 --> 00:20:26,400 Internet and you didn't want bad people thinking about. So as we started to 247 00:20:26,519 --> 00:20:30,519 fuse this information together, we said, well, if we can manipulate or 248 00:20:30,559 --> 00:20:37,200 relay in a way that makes one side of the relay essentially a weapon to 249 00:20:37,240 --> 00:20:41,599 the other side, that could be really interesting. And that was essentially the 250 00:20:41,599 --> 00:20:45,960 genesis of Aurora. We really wanted to show a test that actually shook the 251 00:20:47,000 --> 00:20:52,920 ground like we wanted something dramatic. And as we worked with the power engineers 252 00:20:52,920 --> 00:20:56,920 and we started modeling this, couple of the senior power engineers who were involved, 253 00:20:56,920 --> 00:20:59,640 they said, well, I mean, if the generator is big enough, 254 00:21:00,319 --> 00:21:04,359 you can you could do some serious shaken. And so, as is 255 00:21:04,400 --> 00:21:12,000 shown in the YouTube video that's up now, that generator shook when the array 256 00:21:12,640 --> 00:21:17,880 the phases of the power on the two sides of that safety relay were essentially 257 00:21:18,240 --> 00:21:22,480 put out a whack and in a certain way and it would shake one side. 258 00:21:22,519 --> 00:21:26,519 And so we took that idea and showed that it was reality and it 259 00:21:26,559 --> 00:21:33,279 was I remember the day that the test happened how ecstatic we were because it 260 00:21:33,319 --> 00:21:34,720 was all just theory at the time, right, we had written this stuff 261 00:21:34,759 --> 00:21:37,440 down and it was supposed to work, and you know how it is when 262 00:21:37,440 --> 00:21:41,640 you go down the paths and like this, how how often does it actually 263 00:21:41,640 --> 00:21:45,359 work? And we really had the budget for one try at this, so 264 00:21:45,400 --> 00:21:49,559 we didn't have the ability to to do you know, multiple tries and so 265 00:21:51,240 --> 00:21:55,119 okay, so that was to see it pulled off. You know, when 266 00:21:55,160 --> 00:22:00,480 I talked to people about Aurora. I talked to them years later, you 267 00:22:00,519 --> 00:22:04,319 know they there are there are voices in the community who were who were critical 268 00:22:04,319 --> 00:22:10,839 about how the aftermath was handled. I've been I mean, I wasn't there. 269 00:22:10,880 --> 00:22:15,079 I wasn't proud of this, but I've been told that the details of 270 00:22:15,119 --> 00:22:18,119 the test were immediately I don't know, either classified or made for official use 271 00:22:18,160 --> 00:22:26,119 only and basically hidden away. You know, very superficial details were you know, 272 00:22:26,160 --> 00:22:30,440 became public knowledge and it experts were shown some of the details, and 273 00:22:32,279 --> 00:22:37,200 bluntly, they weren't physicists, they weren't engineers. They didn't understand the physical 274 00:22:37,960 --> 00:22:41,319 characteristics of of what happened, and there were accusations of the whole thing being, 275 00:22:42,519 --> 00:22:47,319 you know, a fake, like I said it was. The public 276 00:22:47,359 --> 00:22:51,559 reception was very confused. Can you tell us anything about what happened behind the 277 00:22:51,599 --> 00:22:56,720 scenes. Whenever you do something for the first time, no one knows how 278 00:22:56,720 --> 00:23:00,680 to handle it. And and that's the situation where on ourselves in that the 279 00:23:00,759 --> 00:23:07,440 test had been conducted without necessarily you know, like a top secret classification around 280 00:23:07,440 --> 00:23:11,240 it. The test was put together in a way where you know, so 281 00:23:11,279 --> 00:23:15,400 many people were involved, it didn't necessarily have the same level of classification like 282 00:23:15,400 --> 00:23:21,559 a pure DoD project would. And so you know, by the way it 283 00:23:21,599 --> 00:23:25,720 was designed, and I think Mike did this on purpose. He wanted to 284 00:23:25,759 --> 00:23:29,440 share the information to help people protect themselves. And I think that's why Mike 285 00:23:29,519 --> 00:23:32,920 designed it that way. He could have designed the test to be ultra high 286 00:23:32,920 --> 00:23:34,839 classify that sort of thing. So it was it was designed from the beginning 287 00:23:34,880 --> 00:23:40,680 of something where Mike wanted to share that information. And because of my background 288 00:23:40,759 --> 00:23:45,880 doing vulnerability reporting at Microsoft, he asked me to lead the report, to 289 00:23:45,920 --> 00:23:49,680 write the report of sort of what was going to get sent upstream to the 290 00:23:49,720 --> 00:23:53,559 sponsors, the people who had, you know, helped to support the test 291 00:23:53,599 --> 00:24:00,480 financially and eventually to DHS because they were the they were positioning themselves as the 292 00:24:00,480 --> 00:24:06,400 industrial control systems certed. Right, So we get the report written, and 293 00:24:06,640 --> 00:24:11,440 the report was written on you know, non classified systems on my laptop sitting 294 00:24:11,480 --> 00:24:15,480 on just the enterprise network and I and L and we took that report and 295 00:24:15,559 --> 00:24:21,559 sent it up the chain. And exactly as you said, the people who 296 00:24:21,599 --> 00:24:26,640 are on the receiving arm of that, the folks at DHS were much more 297 00:24:26,680 --> 00:24:34,000 accustomed to traditional cybersecurity problems not industrial security problems, and that's where there was 298 00:24:34,039 --> 00:24:38,119 some confusion about well is this real, what's the impact, like how should 299 00:24:38,160 --> 00:24:42,160 this be treated? And because you know, we at I and L, 300 00:24:42,240 --> 00:24:45,440 we didn't really have good guidance about what we should do. We wanted to 301 00:24:45,920 --> 00:24:51,559 balance protecting the information so it didn't enable malicious use of what we just just 302 00:24:52,079 --> 00:24:56,960 discovered, but still providing guidance to infrastructure owners should protect themselves from these types 303 00:24:56,000 --> 00:25:04,200 of attacks. And that's the agan almost ninety days of really really crazy conflicts 304 00:25:04,200 --> 00:25:12,039 between people, and whenever there's uncertainty, people tend to become their worst selves, 305 00:25:12,359 --> 00:25:18,720 self protecting, territorial, egotistical in some of the things that happened, 306 00:25:19,319 --> 00:25:25,000 and I think that really set back what was the potential to be able to 307 00:25:25,039 --> 00:25:30,039 talk about this now. Once the video leaked to CNN, there was immediately 308 00:25:30,079 --> 00:25:32,640 a witch hunt to say, okay, who leaked this thing? It was 309 00:25:32,640 --> 00:25:36,440 the one that leaked this thing at CNN, and lots of fingers were pointed 310 00:25:36,480 --> 00:25:40,240 all sorts of directions. But I think that was probably the best thing that 311 00:25:40,279 --> 00:25:45,000 could have happened because it basically allowed for other people to look at it to 312 00:25:45,039 --> 00:25:48,880 go, wait a second, this could make sense you had people from other 313 00:25:48,960 --> 00:25:56,119 disciplines outside of the typical cybersecurity domain that we're looking at it. And I 314 00:25:56,160 --> 00:26:02,079 think once that video was leaked took a lot of the pressure off of us 315 00:26:02,079 --> 00:26:03,599 at I and L because at that point, the horse left the barn, 316 00:26:03,960 --> 00:26:07,240 train left the station, and that's when more we got drug along for the 317 00:26:07,319 --> 00:26:14,119 ride. The ride at times was not fun because again there was there's politics 318 00:26:14,119 --> 00:26:18,839 involved, there's egos involved, and whenever something new happens within the government, 319 00:26:18,440 --> 00:26:21,920 there are vested interests to say, well, I want to own that, 320 00:26:21,960 --> 00:26:25,319 I want to own that program. And so there was some competition that went 321 00:26:25,400 --> 00:26:27,599 down between the labs about who got who was going to get new funding and 322 00:26:27,759 --> 00:26:33,480 what was going to happen. And and that's where there was a huge tax 323 00:26:33,519 --> 00:26:37,680 on us as a team, and there were and it showed in people's personal 324 00:26:37,759 --> 00:26:40,640 lives, like you take a look at what was happening, you know, 325 00:26:40,920 --> 00:26:44,400 outside of work, and it just wasn't a fun situation. And all of 326 00:26:44,440 --> 00:26:48,960 that that great team that we would put together, that cross domain interdisciplinary team, 327 00:26:49,039 --> 00:26:52,720 people from all over the world and all over the country who are working 328 00:26:52,720 --> 00:27:00,319 together, you know, it wasn't fun anymore. And so myself and included 329 00:27:00,400 --> 00:27:04,480 I sort of separated myself to say, you know, maybe maybe this isn't 330 00:27:04,519 --> 00:27:08,160 what I'm cutting out from what I'm cut out for. Maybe there's better ways 331 00:27:08,200 --> 00:27:12,200 I can, you know, go after my desire to protect the world in 332 00:27:12,240 --> 00:27:18,480 the universe by you know, following by promoting cybersecurity in other ways. And 333 00:27:18,519 --> 00:27:22,599 so, you know, by by the two thousand and eight timeframe, we 334 00:27:22,640 --> 00:27:26,240 had lost probably about half the team, and that's when I left, I 335 00:27:26,240 --> 00:27:29,480 know, I was in late two thousand and eight, and I went on 336 00:27:29,559 --> 00:27:34,160 to go do a series of cybersecurity startups focusing on everything from mobile to cloud 337 00:27:34,240 --> 00:27:37,960 and everything in between. And you look at that team that was there, 338 00:27:38,160 --> 00:27:41,359 excellent, great people that went on to do great things, sometimes within the 339 00:27:41,359 --> 00:27:47,160 industrial community, sometimes outside. But it was sort of sad to see it 340 00:27:47,200 --> 00:27:52,279 get torn apart because of that the uncertainty about how to handle this, and 341 00:27:52,319 --> 00:27:55,799 I think that's the danger of whenever you do something new, you know, 342 00:27:56,000 --> 00:28:00,279 people don't know how to handle it. The latest numbers in the twenty twenty 343 00:28:00,319 --> 00:28:06,880 three Threat Report on OT cyber incidents show that the threat environment has changed fundamentally. 344 00:28:07,000 --> 00:28:11,000 At the beginning of this decade, OT cyber attacks with physical consequences have 345 00:28:11,240 --> 00:28:17,480 changed from a theoretical problem to a very real problem more than doubling every year. 346 00:28:18,240 --> 00:28:22,240 The new report is focused on deliberate cyber attacks in the public record. 347 00:28:22,799 --> 00:28:27,599 These are attacks that cause physical consequences in process industries and discrete manufacturing. Most 348 00:28:27,599 --> 00:28:32,640 of these attacks are ransomware, though the fraction of activist attacks is growing, 349 00:28:33,200 --> 00:28:37,359 and the report's appendix includes a complete list of all cyber attacks since Stuxnet that 350 00:28:37,400 --> 00:28:41,920 meet these criteria. To see how today's OT cyber threat environment has changed, 351 00:28:42,160 --> 00:28:47,720 I invite you to download the report, a joint effort between Waterfall Security and 352 00:28:47,759 --> 00:28:52,720 the ICs drive OT Incident Repository. You can download the report at Waterfall dash 353 00:28:52,799 --> 00:28:59,839 security dot com, slash twenty twenty three dash Threat dash Report, or just 354 00:28:59,839 --> 00:29:03,000 go to the resources menu at the Waterfall Security site and click on white papers 355 00:29:03,000 --> 00:29:12,319 and Ebooks. Andrew, I must have seen the grainy footage of the Aurora 356 00:29:12,400 --> 00:29:18,160 generator test by now dozens of times, just because it comes up so often 357 00:29:18,720 --> 00:29:27,160 when you're talking about OT cybersecurity, with stucks Net being the big overall attack 358 00:29:27,200 --> 00:29:32,920 that everybody knows about, but Aurora being that progenitor of this whole conversation, 359 00:29:33,079 --> 00:29:37,039 and so it's sort of interesting to me just to hear Aaron's background on it 360 00:29:37,079 --> 00:29:42,640 as somebody who is directly involved. I'm even just watching the video now. 361 00:29:44,079 --> 00:29:47,920 It's it's sort of it's a very interesting case because you see this giant, 362 00:29:48,480 --> 00:29:55,119 hulking, green metal machine of a thing that is clearly in distress and then 363 00:29:56,279 --> 00:30:00,880 creating black smoke and it almost seems like it's about to blow up. The 364 00:30:00,920 --> 00:30:03,359 notion that that could happen just from a cyber incident, as much as I 365 00:30:03,359 --> 00:30:10,319 can understand that academically, is still to this day interesting, very much so. 366 00:30:10,400 --> 00:30:14,680 And you know, in the moment what I remember when it was released 367 00:30:14,759 --> 00:30:18,319 the information, at least the video in O seven, I mean, the 368 00:30:18,359 --> 00:30:22,839 rest of the detail didn't become public knowledge until years later. In O seven, 369 00:30:22,880 --> 00:30:26,160 there was there was you know, it wasn't released on the news. 370 00:30:26,200 --> 00:30:32,759 It was on CNN. You had cybersecurity experts weighing in on CNN on you 371 00:30:32,839 --> 00:30:36,519 know, social media, what social media existed in the day. A lot 372 00:30:36,559 --> 00:30:40,119 of the feedback that, you know, a lot of the experts weighing in 373 00:30:40,240 --> 00:30:47,480 were cybersecurity experts, not physicists, not engineers, with really little or no 374 00:30:47,559 --> 00:30:49,720 understanding of the physical process, and some of them were coming in saying it's 375 00:30:49,759 --> 00:30:55,880 all fake. It couldn't really have happened that way without again without understanding the 376 00:30:55,880 --> 00:31:00,119 physical process, and in my understanding in terms of the physical process, what 377 00:31:00,200 --> 00:31:08,759 happened was inl has a full power grid. It's a massive test installation that 378 00:31:10,480 --> 00:31:15,680 the generator was connected to as one of many generators on this simulated power grid, 379 00:31:15,160 --> 00:31:21,799 and what they did was trip the breaker. So disconnect the generator from 380 00:31:21,880 --> 00:31:26,400 the grid for a short period of time I assume a fraction of a second, 381 00:31:26,240 --> 00:31:30,599 and what happens. I mean the generator is underload, it's supplying energy 382 00:31:30,680 --> 00:31:34,119 to the grid, The grid is consuming the energy. The generator is working. 383 00:31:34,680 --> 00:31:37,559 The moment you're disconnected from the grid, it has no load anymore, 384 00:31:37,920 --> 00:31:42,240 but there's still energy in terms of the diesel engine spinning the generators, still 385 00:31:42,319 --> 00:31:47,599 energy going into the generator. The generator speeds up, and now the power 386 00:31:47,640 --> 00:31:49,759 it's producing and going nowhere, you know, just heating up the wires. 387 00:31:51,000 --> 00:31:56,119 The power it's producing is out of phase with the power in the simulated grid. 388 00:31:56,400 --> 00:32:00,640 A fraction of a second later you reconnect it, and now there's enormous 389 00:32:00,839 --> 00:32:06,920 stress torque. They call it on the generator because when you've got you know, 390 00:32:07,319 --> 00:32:09,599 a generator and the grid fighting it out for who's going to win. 391 00:32:09,759 --> 00:32:15,559 I'm sorry, the grid always wins. The generator is forced back into phase 392 00:32:15,279 --> 00:32:20,960 in nothing flat, you know, with enormous stress, enough stress to destroy 393 00:32:21,000 --> 00:32:28,079 the generator. You saw the video there and the you know the so we 394 00:32:28,359 --> 00:32:30,759 saw that in the public sphere. What I didn't realize was sort of a 395 00:32:30,799 --> 00:32:36,680 different debate happening in the in the in confidence in government, where people are 396 00:32:36,680 --> 00:32:38,640 saying, oh, it is real, you know, I want to own 397 00:32:39,000 --> 00:32:45,640 this problem going forward. I didn't realize that that that was happening. I 398 00:32:45,720 --> 00:32:50,079 don't want to preempt anything you ended up discussing with Aaron, But from your 399 00:32:50,160 --> 00:32:55,960 perspective, was there any major shift in the way that government worked with ot 400 00:32:57,160 --> 00:33:00,640 sites or the way that OT sites worked on their own that may have directly 401 00:33:00,680 --> 00:33:06,759 resulted from this? The incident was was widely reported. It was you know, 402 00:33:07,319 --> 00:33:13,200 people talked about it for half a decade or longer after the incident. 403 00:33:14,240 --> 00:33:17,359 You know, the the big news that the biggest news that happened after that 404 00:33:17,440 --> 00:33:21,160 was sort of stuck snet that sort of preempted it. But you know, 405 00:33:21,319 --> 00:33:25,960 there weren't a lot of examples in the public domain of cyber attacks that could 406 00:33:27,200 --> 00:33:31,240 or did cause physical consequences, and so you know, the the the incident 407 00:33:31,359 --> 00:33:37,799 was was influential. And you know, in in Aaron's estimation, you know, 408 00:33:37,880 --> 00:33:45,400 the the turf war that took place within the government, you know, 409 00:33:45,799 --> 00:33:49,519 was a turf war for funding and responsibility. It was you know, when 410 00:33:49,559 --> 00:33:54,680 when that turf warce settled out, there was funding, there was an initiative, 411 00:33:55,359 --> 00:34:02,440 and you know it was it was sort of instrumental in cementing that initiative 412 00:34:02,440 --> 00:34:07,359 going forward, is my understanding. But now coming back to the test itself, 413 00:34:07,480 --> 00:34:12,800 you maybe I'm misremembering mentioned that the generator was destroyed. Now, from 414 00:34:12,800 --> 00:34:16,840 the publicly available video that I've seen over and over, you do see a 415 00:34:16,880 --> 00:34:21,159 ton of black smoke coming out of it, and it's sort of shaking, 416 00:34:21,360 --> 00:34:24,599 and it seems like it's in a state of real panic, this machine. 417 00:34:25,440 --> 00:34:30,320 But the notion of this thing being destroyed, and if anybody's interested, just 418 00:34:30,360 --> 00:34:37,039 look up a picture of this Aurora generator or blowing up in any meaningful way, 419 00:34:37,119 --> 00:34:39,440 is still sort of unbelievable. You're telling me that there was more damage 420 00:34:39,480 --> 00:34:43,599 than what we see in this video, or you're just using a different word 421 00:34:43,639 --> 00:34:46,559 for it. No, So, I mean the generator did not blow up. 422 00:34:46,599 --> 00:34:51,119 It did not explode. You know. The video says that the smoke 423 00:34:51,199 --> 00:34:54,280 rose out of the generator, that there was obvious vibration, and the analysis 424 00:34:54,320 --> 00:34:59,079 of the generator afterwards, you know, the internal report to the government was 425 00:34:59,119 --> 00:35:02,519 the generator was when you open that generator up, there's nothing useful inside anymore. 426 00:35:02,559 --> 00:35:06,320 You can't generate power with it. You have to throw it away. 427 00:35:06,440 --> 00:35:10,960 It was it was a write off, don't I don't know that the diesel 428 00:35:12,320 --> 00:35:16,840 engine was affected as badly, but the generator was shot. And you know, 429 00:35:17,039 --> 00:35:22,599 the diesel engine provides energy to the generator. The generator turns rotational energy 430 00:35:22,719 --> 00:35:29,760 into electricity. And you know, I've had the privilege of visiting large power 431 00:35:29,760 --> 00:35:32,559 plants in the past. When I see a large generator, I mean that 432 00:35:32,599 --> 00:35:36,800 was a ten megawak generator, it's nothing by the scale of the grid. 433 00:35:36,960 --> 00:35:40,440 A large generator is three hundred, five hundred, eight hundred megawatts, so 434 00:35:40,480 --> 00:35:46,599 it's you know, between between thirty and eighty times as big. I saw 435 00:35:46,639 --> 00:35:51,719 a five hundred megawat generator once, and it's you know, it's as big 436 00:35:51,760 --> 00:35:58,199 as a bungalow, and it looks like a very large lump of molten metal. 437 00:35:58,719 --> 00:36:00,760 You know, it just looks like you took big drop of metal and 438 00:36:00,880 --> 00:36:05,840 dropped it, and you know, it landed, it hardened, and that's 439 00:36:05,840 --> 00:36:07,800 what it looks like. And I'm going, that's not what I expected. 440 00:36:07,840 --> 00:36:10,800 You know, I expected a generator to be rounder, you know, I 441 00:36:10,840 --> 00:36:15,920 expected sort of And they said, no, no, you don't understand, 442 00:36:15,960 --> 00:36:19,920 Andrew. They said, all of that metal on the outside of the generator 443 00:36:20,000 --> 00:36:25,239 is to protect you and me standing here, because if that generator fails in 444 00:36:25,320 --> 00:36:30,039 the worst case, and you know, an outer phase reconnect is pretty close 445 00:36:30,079 --> 00:36:31,800 to a worst case. But you know, I was told, if that 446 00:36:31,840 --> 00:36:38,119 generator fails in the worst place, it basically blows up. It's turning at 447 00:36:38,280 --> 00:36:45,760 at least sixty cycles a second, sixty rpm, and if it flies apart, 448 00:36:46,000 --> 00:36:51,159 this is three hundred tons of metal that's flown apart. And all of 449 00:36:51,199 --> 00:36:55,039 that metal you see on the outside is to prevent that metal inside flying apart 450 00:36:55,360 --> 00:36:59,800 from striking you and me and the building and all of these other generators that 451 00:36:59,840 --> 00:37:04,559 you see down the massive building. So you know, it's a it's a 452 00:37:04,599 --> 00:37:08,440 real concern. And in the modern world, like I said, people protect 453 00:37:08,599 --> 00:37:13,840 these generators. There have been cases in the past where generators have blown up 454 00:37:15,400 --> 00:37:19,159 or turbines have blown up. I think it was a hydro turbine in two 455 00:37:19,159 --> 00:37:22,599 thousand and nine killed seventy five people. So these are very large pieces of 456 00:37:22,639 --> 00:37:29,519 equipment. They're dangerous pieces of equipment. This little demonstration managed to destroy a 457 00:37:29,559 --> 00:37:34,079 ten megawatt generator. But you know, the concern everyone has is that much 458 00:37:34,079 --> 00:37:39,360 worse is clearly possible, so that, you know, that begs the question 459 00:37:39,480 --> 00:37:45,360 here we are going on fifteen years later than two thousand and eight. You 460 00:37:45,360 --> 00:37:49,280 know, there's a lot of water under the bridge. Since then, industrial 461 00:37:49,320 --> 00:37:52,079 cybersecurity is a is a mainstream activity. You know, we still have we 462 00:37:52,119 --> 00:37:57,320 still have lots of engineering teams who are just beginning to come up to speed. 463 00:37:57,360 --> 00:38:00,199 But there's widespread recognition that, you know, this is a thing, 464 00:38:00,440 --> 00:38:05,480 it's real. We have to you know, we have to act on it. 465 00:38:07,639 --> 00:38:09,320 Did you you know, did you stay in touch with the community, 466 00:38:09,840 --> 00:38:14,400 you know, in in your sort of contacts, your your view of the 467 00:38:14,599 --> 00:38:19,079 of of the history. You know, how how was all of this confusion 468 00:38:19,159 --> 00:38:22,000 resolved? How did we wind up sort of on a track to get to 469 00:38:22,039 --> 00:38:25,880 where we are today. Well, again, I think we need to pay 470 00:38:25,960 --> 00:38:30,079 tribute to Mike for being courageous enough to stay the course. Like he he 471 00:38:30,119 --> 00:38:32,719 could have bowed out and said, hey, I'm going to go do something 472 00:38:32,719 --> 00:38:37,679 else, but he leaned in with with Firk and Nirk and said, look, 473 00:38:37,719 --> 00:38:43,000 we've got to do something about this. And and as the result he 474 00:38:43,119 --> 00:38:49,840 spent some time researching where would be the best place to land to keep driving 475 00:38:49,840 --> 00:38:54,079 this this forward. And the other person I think we should really pay tribute 476 00:38:54,079 --> 00:38:58,840 to, who also unfortunately is not with us, is Alan Power, the 477 00:38:58,880 --> 00:39:04,039 founder of SANDS. So Mike and Allan had known each other through other you 478 00:39:04,079 --> 00:39:07,519 know, training relationships, and Alan really put himself out there to say, 479 00:39:07,559 --> 00:39:13,960 you know what, because SANS has this platform to provide meaningful technical training, 480 00:39:14,239 --> 00:39:19,239 because SANS has this great certification mechanism where you go for this training, and 481 00:39:19,719 --> 00:39:22,199 SANDS certificates, you know, still to this day, really stand above others 482 00:39:22,199 --> 00:39:28,760 because of the depth of technical training that you get through those those courses. 483 00:39:29,480 --> 00:39:34,159 And so Alan and Mike basically agreed to say, you know, let's create 484 00:39:34,239 --> 00:39:38,880 an industrial control curriculum. And that was the best thing that could have happened, 485 00:39:38,880 --> 00:39:43,800 because at that point Alan had the resources to push it forward, to 486 00:39:43,840 --> 00:39:50,960 basically fund the creation of a vendor neutral forum for people to go and learn 487 00:39:51,039 --> 00:39:57,239 meaningful things. But Allan also had the political connections because Alan and I had 488 00:39:57,280 --> 00:40:00,639 known Alan from the time when he first started SANDS and I was wearing at 489 00:40:00,679 --> 00:40:07,360 Microsoft. We collaborated on sharing course materials around Windows security because Microsoft needed some 490 00:40:07,360 --> 00:40:12,320 folks to go teach the US military about how to secure Windows systems, and 491 00:40:12,679 --> 00:40:16,400 Microsoft did and wanted to maintain an arms like relationship there. So SANDS became 492 00:40:16,440 --> 00:40:22,840 a great channel that I collaborated with there. And so with that connection with 493 00:40:22,960 --> 00:40:28,719 SANDS, that's really where the what I'll call the flowering of public knowledge in 494 00:40:28,760 --> 00:40:36,320 a proactive, well defined way. And as a result of that SANDS curriculum, 495 00:40:36,840 --> 00:40:42,920 DOE sort of I guess there was a there was a peace movement between 496 00:40:43,280 --> 00:40:47,360 what had happened between the Aurora test and some of the DHS stuff that had 497 00:40:47,360 --> 00:40:52,239 gone on, and so DHS and DOE went along with that and created their 498 00:40:52,280 --> 00:40:55,800 own course materials. And to this day you can still go out to the 499 00:40:55,800 --> 00:41:00,920 IDON National Laboratory and participate in hands on technical training around industrial control and so 500 00:41:00,920 --> 00:41:07,960 I think that was really the combination of sands plus the ability of dealing and 501 00:41:07,039 --> 00:41:13,440 DHS to put together a curriculum there that was really what put this in the 502 00:41:13,480 --> 00:41:16,480 position where we're at today. And now you take a look and there's been 503 00:41:16,480 --> 00:41:22,559 a flowering of startups, you know, folks like Dregs and others that are 504 00:41:22,559 --> 00:41:28,000 out there that have really tried their best to help this community. And I 505 00:41:28,039 --> 00:41:30,880 think that's what really it's us in a situation where in today, which is 506 00:41:30,880 --> 00:41:37,800 a much healthier one where people can have open and honest discussions about the convergence 507 00:41:37,840 --> 00:41:43,639 of control systems cyber physical attacks and you know, the price we have to 508 00:41:43,639 --> 00:41:46,519 pay now is that we've seen several I mean just in the last year or 509 00:41:46,559 --> 00:41:50,880 two years. Probably the ones that are most interesting to me or what happened 510 00:41:50,880 --> 00:41:54,800 with the bul Russian railroad system as a result of some probably Ukrainian attacks against 511 00:41:55,079 --> 00:42:00,159 that railroad system to stop the delivery of tanks to their northern border. But 512 00:42:00,360 --> 00:42:02,400 you know, there's there's been some terrifying things what you've seen as a result 513 00:42:02,480 --> 00:42:07,000 of cyber physical convergence. But it's the world we live in now, and 514 00:42:07,039 --> 00:42:12,519 I think now we have the ability to have open and on his conversations about 515 00:42:12,639 --> 00:42:15,760 what we can actually do about it. So that's really interesting. I mean, 516 00:42:15,880 --> 00:42:22,199 I I knew Mike. I knew Mike uh Asante to see him, 517 00:42:22,440 --> 00:42:25,079 you know, he was he was a fixture at DHS and other events. 518 00:42:25,239 --> 00:42:30,480 I kind of I kind of knew him as the the He was one of 519 00:42:30,519 --> 00:42:36,599 the senior managers at NIRK, and you know, he he was infamous. 520 00:42:36,960 --> 00:42:37,840 He was I think he was only there a couple of years, but he 521 00:42:37,920 --> 00:42:42,719 was infamous for sending out a letter saying, guys, uh you know. 522 00:42:42,920 --> 00:42:49,880 This version of NIRK says NRK sip rather says that you have to self assess 523 00:42:50,280 --> 00:42:54,440 as to which of your assets are critical to the reliability of the ball electric 524 00:42:54,480 --> 00:43:00,239 system. Some large power utilities out there have identified you knows or even hundreds 525 00:43:00,280 --> 00:43:07,039 of uh you know, physical assets and cybersystems that control them as critical to 526 00:43:07,119 --> 00:43:12,280 the grid and have taken measures to protect them. Other utilities, just as 527 00:43:12,360 --> 00:43:15,519 large have come back and said, absolutely, none of our equipment is critical. 528 00:43:16,159 --> 00:43:20,760 We all know that these both can't be true, you know, fix 529 00:43:20,880 --> 00:43:25,000 this. I remember I'm paraphrasing that that was what I the the the sort 530 00:43:25,039 --> 00:43:29,639 of the takeaway that I recall from the letter that was sort of where I 531 00:43:29,760 --> 00:43:31,800 was introduced to Mike, and then you know, I saw him later on 532 00:43:31,840 --> 00:43:37,199 at Sands. You know, I had, I had, I had none 533 00:43:37,239 --> 00:43:42,039 of this this background before. Yeah, so if you think about, you 534 00:43:42,079 --> 00:43:47,800 know, what the what Mike did is he put himself out there to basically 535 00:43:47,840 --> 00:43:52,960 say, we've got to make a change. And I think that letter was 536 00:43:52,000 --> 00:43:57,119 part of it. You know, he he continued to work closely with Congress 537 00:43:57,639 --> 00:44:02,079 to you know, motivate folks to make sure that the right at least partial 538 00:44:02,159 --> 00:44:05,880 legislation was in place, to try to say, hey, we've got to 539 00:44:05,880 --> 00:44:09,039 do better about protecting critical systems. He did a ton of lobbying with dhs 540 00:44:09,079 --> 00:44:14,280 to make sure that they were empowered with knowledge so that they could build the 541 00:44:14,360 --> 00:44:17,039 right working groups and keep moving it forward. And so he was critical to 542 00:44:17,079 --> 00:44:21,559 it. And I think what a lot of folks don't understand is that, 543 00:44:21,880 --> 00:44:25,400 you know, he he was a cancer survivor, and that was one of 544 00:44:25,400 --> 00:44:29,119 the things that attracted me to work with him. I'm also a cancer survivor, 545 00:44:29,159 --> 00:44:31,239 and so you know, whenever you face death, you know, both 546 00:44:31,239 --> 00:44:36,719 he and I got terminal diagnoses where we were supposed to die sometime in two 547 00:44:36,760 --> 00:44:39,039 thousand and six, and that also motivated us to go out the I n 548 00:44:39,119 --> 00:44:42,800 L. Because if the diagnosed was right, we kind of both wanted to 549 00:44:42,800 --> 00:44:46,679 go out with the bank. Well, you know, fortunately I have continued 550 00:44:46,719 --> 00:44:52,039 to fight mine I was. I suffered from melanoma, but he suffered from 551 00:44:52,199 --> 00:44:57,559 a non honchosuphoma, and unfortunately he had a reoccurrence and that's the reason why 552 00:44:57,559 --> 00:45:00,559 he passed away a couple of years ago. But I think the the thing 553 00:45:00,599 --> 00:45:06,760 that we look at now is, you know, Mike's ability to focus people, 554 00:45:06,840 --> 00:45:09,159 to get people on the right path. And that's why we're we are 555 00:45:09,199 --> 00:45:13,559 where we are today, is because he had the courage to write letters like 556 00:45:13,599 --> 00:45:15,960 he did at to basically stand up in people's faces and say we've got to 557 00:45:16,000 --> 00:45:21,320 do something about this. And and that's the reason why there's scholarships named after 558 00:45:21,400 --> 00:45:27,000 him and awards and the cybersecurity community and it's all it's it's all Merid like 559 00:45:27,039 --> 00:45:30,400 there's there's a whole bunch of stuff that Mike did that no one will probably 560 00:45:30,400 --> 00:45:34,960 ever know because he wasn't a bragger, he wasn't a guy who wore all 561 00:45:35,000 --> 00:45:37,719 of his achievements on his sleeve. Will probably never know the full extent to 562 00:45:37,760 --> 00:45:43,000 which he dedicated his life to make the world a better place. And I 563 00:45:43,039 --> 00:45:45,320 just got myself as lucky that I got to go, I got to work 564 00:45:45,320 --> 00:45:49,559 with him and got to know him. So yeah, Nate, as I, 565 00:45:49,679 --> 00:45:52,679 as I said on the on the interview, you know, I knew 566 00:45:52,320 --> 00:45:57,079 Mica Sante from his days at RK. I think he was the chief security 567 00:45:57,119 --> 00:46:01,880 officer officer there for like two or three years, and you know, then 568 00:46:01,960 --> 00:46:06,800 he moved on and I remember him eventually, you know, in before he 569 00:46:06,840 --> 00:46:10,480 passed away, he was in charge of the industrial control system training program at 570 00:46:10,559 --> 00:46:15,920 SANDS. But you know what little I knew about him personally is that, 571 00:46:17,559 --> 00:46:22,880 you know, he wasn't afraid to make waves. I remember that letter that 572 00:46:22,960 --> 00:46:27,920 came out and I think it was two thousand and nine talked about Look, 573 00:46:28,039 --> 00:46:35,599 you know Version three says you're required to you know, these power utilities are 574 00:46:35,599 --> 00:46:39,320 required to define a risk assessment methodology. You're required to apply the methodology to 575 00:46:39,440 --> 00:46:45,559 your physical assets, the generators and the transformers and the substation. You're required 576 00:46:45,559 --> 00:46:50,119 to identify which of these physical assets are essential to the reliability of the grid. 577 00:46:50,840 --> 00:46:53,840 You are required then to figure out which computers, if any, are 578 00:46:54,039 --> 00:46:59,039 essential to the correct operation of those physical assets. Those are your critical cyber 579 00:46:59,159 --> 00:47:02,360 assets. You have to apply the rules in NIRKSIP to the critical cyber assets. 580 00:47:02,400 --> 00:47:08,159 He said, a lot of you large power utilities that you know probably 581 00:47:08,239 --> 00:47:13,840 have cyber critical assets and critical cyber assets have come back and said we have 582 00:47:13,920 --> 00:47:17,880 none. You know, this is going to have to change. And you 583 00:47:17,920 --> 00:47:22,719 know it was controversial, I think because people interpreted it as you know, 584 00:47:22,880 --> 00:47:30,840 accusing the power companies of not caring about the reliability the grid. And you 585 00:47:30,880 --> 00:47:35,599 know, I reread the letter and uh, you know, I don't I 586 00:47:35,639 --> 00:47:42,079 don't see that. I mean, there he's identified a problem. He says, 587 00:47:42,639 --> 00:47:47,400 this methodology has been applied inconsistently, and you know he gives he gives 588 00:47:47,519 --> 00:47:51,159 you the power companies. Now, he says, look, you know, 589 00:47:51,599 --> 00:47:54,639 in his estimation from talking to the utilities, it has to do with redundancy. 590 00:47:54,719 --> 00:47:59,039 The grid is massively redundant. If a generator goes down, there's other 591 00:47:59,199 --> 00:48:02,239 generators that can up the load. If a substation goes down, there's other 592 00:48:02,400 --> 00:48:07,079 paths through the mesh that is the transmission grid to get power from sources to 593 00:48:07,159 --> 00:48:12,400 destinations, and he says that, you know, the fact that you have 594 00:48:12,519 --> 00:48:16,840 redundancy does not make these devices not critical. Yes, any one of them 595 00:48:16,840 --> 00:48:21,880 can fail and the grid keeps going. But he says these devices are still 596 00:48:21,920 --> 00:48:25,360 critical to the grid because in the world of sort of random equipment failures, 597 00:48:25,400 --> 00:48:30,039 you can count on redundancy. In the world of cyber attacks, deliberate attacks, 598 00:48:30,440 --> 00:48:34,840 you might have an attack that takes down multiple similar assets that are similarly 599 00:48:34,880 --> 00:48:39,360 defended, and now the redundancy has been bypassed. And so you know, 600 00:48:39,880 --> 00:48:44,599 to me, it was it was, it was reasonable. But again it 601 00:48:45,000 --> 00:48:50,079 was controversial in the day because he pointed out this inconsistency in a very public 602 00:48:50,119 --> 00:48:55,360 way. Wow, well, thank you for that, and thank you for 603 00:48:55,480 --> 00:49:00,239 joining us. I mean this has been a you know, insightside. I 604 00:49:00,280 --> 00:49:05,239 didn't have into you know, the history, the beginnings of the industry that 605 00:49:05,320 --> 00:49:09,599 now has thousands and thousands of practitioners in it. You know, before we 606 00:49:09,679 --> 00:49:13,880 let you go, can you sum up for us what you know, what 607 00:49:13,920 --> 00:49:15,719 should we what what should we all take away from the history, What lessons 608 00:49:15,760 --> 00:49:21,400 should we should we you know, carry around with us. So I think 609 00:49:21,400 --> 00:49:27,000 the first thing is is that the older we get, the more rigid our 610 00:49:27,039 --> 00:49:32,079 thinking becomes. And luckily Mike and I were both young kids who were willing 611 00:49:32,119 --> 00:49:37,760 to challenge the status quo. We were willing to challenge the the incumbents and 612 00:49:37,920 --> 00:49:43,679 basically think evilly right. We were the ones who really started to say, 613 00:49:43,679 --> 00:49:47,639 look, what's the worst thing we can do? And I think that's something 614 00:49:49,000 --> 00:49:52,599 that we always have to be willing to consume. And whether that's you know, 615 00:49:52,719 --> 00:49:58,199 inviting you know, outside folks to come and do penetration tests and and 616 00:49:58,320 --> 00:50:01,280 be able to evolve models, I think that is so so important. And 617 00:50:01,320 --> 00:50:06,360 so I would say, you know, if you're a security leader, someone 618 00:50:06,400 --> 00:50:09,480 who's been around in the industry for a while, someone who owns large infrastructure 619 00:50:09,480 --> 00:50:14,639 systems or whatever, be willing to bring young folks in who have new thinking 620 00:50:14,679 --> 00:50:19,159 about new ways to approach how do you compromise these systems? How do you 621 00:50:19,239 --> 00:50:22,599 turn a protection what was maybe a control of designs of protection into a weapon? 622 00:50:23,239 --> 00:50:28,960 And we always need that fresh thinking. So I think step one always 623 00:50:29,000 --> 00:50:34,840 make sure that you're open to critical thinking and to evolving threat models so that 624 00:50:34,880 --> 00:50:37,719 you can understand, you know, how to go about doing things. The 625 00:50:37,760 --> 00:50:44,440 next thing I would recommend to folks is as you make investments in cybersecurity, 626 00:50:45,039 --> 00:50:51,199 sometimes simpler is better. So over the last thirty years there's been several phases 627 00:50:51,239 --> 00:50:52,840 of my career where I've seen people say, you know what, I'm going 628 00:50:52,880 --> 00:50:57,519 to go out and buy every security tool on the planet and just start layering 629 00:50:57,519 --> 00:51:01,079 the stuff all over the place because more is better. Well, the situation 630 00:51:01,119 --> 00:51:06,320 we find ourselves in now is more may not be better because it's too noisy, 631 00:51:06,559 --> 00:51:12,920 because it's too it's giving you telemetry that's maybe false positives. And you 632 00:51:12,920 --> 00:51:17,079 know, as much as sometimes we want to avoid single points of failure, 633 00:51:17,199 --> 00:51:25,039 want to avoid situations where we don't have great resiliency through through distributed or diversification, 634 00:51:27,320 --> 00:51:30,440 you know, we're nearing a time now where we're seeing the proliferation of 635 00:51:30,480 --> 00:51:37,079 attacks, especially through identity control systems, where you know, even very supposedly 636 00:51:37,079 --> 00:51:42,440 strong identity systems that have features like multi factor authentication, that identity system itself 637 00:51:42,519 --> 00:51:45,159 is compromised, thereby eliminating the need for MFA to get into the system. 638 00:51:45,159 --> 00:51:51,480 And so sometimes those complex identity systems come back to bidas because we've cobbled these 639 00:51:51,480 --> 00:51:57,719 things together. So simplification in things like identity ecosystems, simplification and things like 640 00:51:57,920 --> 00:52:02,400 network segmentation. I think those are things that we need to engineer towards as 641 00:52:02,440 --> 00:52:08,320 system owners of how do we simplify to get better security results. And the 642 00:52:08,400 --> 00:52:15,119 last thing that I that I'll put out there for the community is we need 643 00:52:15,159 --> 00:52:19,280 to find the next version of Mike. I don't know where that person sits, 644 00:52:19,760 --> 00:52:25,039 very likely not within the cybersecurity domain. I think the diversity of thought 645 00:52:25,119 --> 00:52:30,960 that comes from other disciplines is what we need to keep ourselves fresh in cybersecurity. 646 00:52:31,360 --> 00:52:35,079 And we've got to be looking for those people and giving them chances to 647 00:52:35,079 --> 00:52:39,159 come in and participate in meaningful ways. And I think with those three things 648 00:52:39,239 --> 00:52:45,559 we can we can keep moving forward to what got started fifteen twenty years ago. 649 00:52:47,960 --> 00:52:52,000 Andrew, that was your interview with Aaron Turner. Do you have any 650 00:52:52,039 --> 00:52:54,480 final thoughts that you might want to end with today? Yeah, I mean, 651 00:52:57,239 --> 00:53:00,400 let me repeat his three points. He went on for a little bit. 652 00:53:00,440 --> 00:53:02,800 You know, he said, in my recollection, be paranoid, challenge 653 00:53:04,039 --> 00:53:07,599 the status quo in terms of, you know, bad stuff that could happen. 654 00:53:08,440 --> 00:53:13,960 He said, simplify, you know, simpler is better, he said, 655 00:53:14,320 --> 00:53:17,880 you know, diversity across disciplines. Uh, you know, bring bring 656 00:53:19,079 --> 00:53:22,079 fresh knowledge in, especially when we're talking you know, he didn't say it, 657 00:53:22,119 --> 00:53:23,880 but in my mind, especially when we're talking about physical consequences, you 658 00:53:23,920 --> 00:53:30,320 can't you cannot really get an understanding of the physical consequences without bringing in people 659 00:53:30,320 --> 00:53:35,280 who are experts on the physics, experts on the engineering. So you know, 660 00:53:36,039 --> 00:53:38,599 be paranoid, challenge the status quotes, simplify and you know, bring 661 00:53:38,639 --> 00:53:44,960 people in who know about, you know, how things work. Makes great 662 00:53:45,039 --> 00:53:51,159 sense. You know, lately, I've been very involved in the cyber informed 663 00:53:51,280 --> 00:53:53,760 Engineering initiative and it's saying some of the same things he's saying. It's saying 664 00:53:53,760 --> 00:54:00,400 that, you know, we have to teach engineers to be more pa aoid. 665 00:54:00,400 --> 00:54:07,679 We have to you know, use powerful simple tools that engineers have, 666 00:54:07,800 --> 00:54:13,280 you know, over pressure relief valves, mechanical overspeed governors use these simple tools 667 00:54:13,440 --> 00:54:19,199 as last ditch stop gaps, so that even if all of our cyber defenses 668 00:54:19,239 --> 00:54:24,000 fail, we still have physical protection from catastrophe, and you know, diversify, 669 00:54:24,280 --> 00:54:29,199 you know, bring in the physical experts. There's a lot of knowledge 670 00:54:29,480 --> 00:54:32,000 that's needed in the space. A lot of it's in the head of engineers. 671 00:54:32,000 --> 00:54:36,360 Some of it's in the head of you know, chemists and physicists. 672 00:54:36,880 --> 00:54:39,559 This all makes this all makes perfect sense. So you know, I think 673 00:54:39,639 --> 00:54:46,599 you know, Aaron has sort of not been active in the field in most 674 00:54:46,639 --> 00:54:52,719 of a decade, but his advice is right on the money. All right. 675 00:54:52,760 --> 00:54:55,559 Well then, thank you to Erin for sharing all this with us, 676 00:54:55,639 --> 00:55:00,760 and Andrew, thank you as always for speaking with me. Thank you very 677 00:55:00,800 --> 00:55:05,599 much, Niate. This has been the Industrial Security Podcast from Waterfall. Thanks 678 00:55:05,639 --> 00:55:07,320 to everybody out there listening.