The genesis of Aurora started with Mike and others motivating us to ask the question,
what are some interesting accidents that have taken place relative to control systems and
infrastructure. Welcome everyone to the Industrial Security Podcast. My name is Nate Nelson.
I'm here as usual with Andrew Ginter, the vice president of Industrial Security
at Waterfall Security Solutions. He's going to introduce the subject and guests of our
show today. Andrew, how are you. I'm very well, Thank you,
Nate. Our guest today is Aaron Turner. He is part of the
faculty at Ion's Research. I a n S Research. You know these people,
they do managerial they do SISO training, and you know our topic today's
failures of imagination from the nine to eleven attacks through the Aurora demo. Aaron
was instrumental in the history the genesis of the industrial security field, and he's
going to tell us a bit about how this all came to be. Then,
without further Ado, here's your conversation with Aaron. Hello, Aeron,
and thank you for joining us before we Before we get started, can you
say a few words for our listeners about yourself and about the good work that
you're doing at Ions. Yeah, thank you for this opportunity to talk about
the history of cybersecurity. It's something I'm really passionate about. I've been doing
some form of breaking into systems or hardening systems since early nineteen nineties, and
I got my starting a penetration tester, but caught a lucky break in the
late nineties to join Microsoft security teams. And today I work at Ion's Research
as a faculty, and what that means is I try to help people take
a non vendor driven approach to solving problems, and the Iron's Research has been
a great platform to help me do that. I work with over six hundred
customers around all sorts of different industries and it's a great form for me to
just get access to great information and collaborate people without the filter that we have.
Sometimes our topic is failures of imagination, I mean, in my dim
understanding, you know, third and fourth hand, the industrial control system,
the Skata Security Initiative, if you like, it started after nine eleven.
Nine eleven was a physical assault on the World Trade Center. But in the
months after I'm told that authorities around the world looked around and said that that
was unexpected, that was a failure of imagination. Where else have we failed?
And one of the ways that I'm told came back was industrial cyber security.
And you know, whereas before the turn of the century there might have
been a dozen people on the planet looking at the topic, mostly in universities
academics, it became a mainstream concern. This is you know, that's it.
That's my depth of understanding. I understand that you were part of that
process. Can you talk about that sort of the next level of detail?
You know, what what did it look like from the inside. Yeah.
When I was asked to join Microsoft in nineteen ninety eight, I joined an
organization that didn't really have a clear focus on security, but that focus had
to get sharpened over time. And because I also have a little bit of
training in the law and the law school dropout, I would often be paired
with law enforcement to go try to solve tough problems, tough questions. And
so by the time nine to eleven happened in two thousand and one, I
had already developed strong relationships with the Secret Service and Department Justice, DEA,
FBI, And so when they came to me and said, Aaron, what's
the craziest thing you could think about happening as the result of computer problems.
Well, this was in light of the fact that I had just helped the
FBI Kart Lab to do some investigative research on the laptops associated with the DC
Sniper. That same lab was the one that did some of the analysis on
the laptops that Daniel Pearl purchased in Pakistan that were used by Muhammadada and others
to do flight simulator training into the World Trade Center. And so as I
sat back and said, Okay, what would be the thing that I would
do, I said, you know, whenever I've worked with folks who embed
computers into systems to do good, very rarely do those engineers have or whether
you'd call it the malicious imagination or the threat modeling mindset to go what's the
worst thing that could happen? And my background in that area came from a
side project that I was working on at Microsoftware for a period of time.
I would help the licensees of Windows XP embedded evaluate how that embedded system was
being used. So, for example, in a medical imaging system, they
had decided to embed a Windows XP subsystem into that large medical imager. It
was an MRI system, and in MRIs, you have these massive magnets that
rely on polarizing the human body and water in ways to get those images.
Well, when someone showed me that, my first thought was, I guess
being somewhat broken inside, being a bad kid, or I guess just having
an evil imagination. I said, well, wouldn't it be funny if you
know, you reverse the plarity on one side of the magnets and you turn
that MRI into a human meed grinder. And they didn't think that was very
funny. In fact, the response from the engineers on that project were like,
you're sick, you're broken. And my response to them was that,
okay, well I might be broken, but you have to think you've got
to apply threat models to the way you embed these systems. And so that
began a journey that I went down, and it was really sharpened with some
interactions that I had through CSO Magazine. Bob Bragden, the publisher of CSO
Magazine, put together a working group probably around two thousand and three two thousand
and four timeframe, where I was introduced to a man named Mica Sante.
Mica Sante at the time was working for American Electric Power. He was the
CISO there. He had just cleaned up a major disruption that had happened in
his grid that coincided with a major incident that Microsoft had had in August of
two thousand and three, and so we started collaborating in ways, and I
really found an affinity of working with Mike that we sort of both were,
I guess broken in our own way, and it was a really interesting opportunity
to start to ask those difficult questions of what's the worst thing that can happen
if we started embedding distributed computers and something else that had all these different in
two thousand and three was the Northeast blackout. Millions of people without power for
hours, some of them I think possibly for days, but most of them
I think it was restored within twenty four hours. The post mortem analysis on
that said that, you know, in my understanding, if I my recollection
I read the thing years ago, said that it was like a memory leak
in an alarm server. Alarms were delayed that could have told the operators there
was a problem, and they could have, you know, taken preventive corrective
action to prevent the blackout, but they didn't. See the alarms. Because
of this failure, there was widespread speculation that it was a cyber attack.
You were involved in that as well. How what happened there? Yes,
in August of two thousand and three, so twenty years ago now, there
was an event on the Microsoft side of things. It was called the blaster
Worm. The blaster Worm, over the course of several days, infected over
two billion computers around the world with an attack package that was designed to try
to take down Windows Update. So basically the attackers wanted to disable the ability
to people let people fix the problem. So we were focused on the blaster
instant and it was so bad that, you know, the inbound support cues
at Microsoft were overloaded and we're having trouble going through, you know, and
actually helping people get get help. Well, that was the same time when
there was this accident in American Electric Power switchyard that caused this series of events
that pushed you know, those substations into a safe state, and a safe
state is disconnected. Well as a result of that, plus the network being
congested from the blaster traffic between sites and within the enterprise network in American Electric
Power, it probably serve is a contributing factor. Now, in the haze
of digital uncertainty, that is, were these massive events and incidents, there
were some people within government that suggested that maybe the Microsoft impacted worm, the
blaster worm, had something to do with the power grid. Now, eventually,
as you mentioned, it was traced back to a system failure that was
not related to the Microsoft operating system problem. But it probably was a contributing
factor in the delay in response, and it probably forced that outage to grow
longer than it should have for some people. But that was another period of
time when you know, myself, Mike and other people basically sat down and
said, wow, this was an accident. What if somebody did that on
purpose? Like, what would happen if someone decided to go and manipulate a
digital network in a way that reduced the fidelity or the reliability or the integrity
of the network that was controlling things like the power grid or cell phone networks
or water delivery systems or whatever it may be. And so in that world
where we had proof that blastered impaired the restart on the IT side, then
maybe control systems needed to be thought about it in a new threat model,
what's the trust relationship between it and OT and what kinds of boundaries should be
there, And it sort of served as a genesis for myself and Mike and
others to start asking those questions. I only would have been seven years old
at the time, but I distinctly remember that Northeast blackout. My family was
taking a trip to Canada and on the way back, we stopped in an
ice cream place, not realizing that half of the Northeast was totally in darkness
and they were giving away free ice cream because it was all melting. Yeah.
I mean, that was a big event, and in the heat of
the moment, in the weeks that followed the event, there was widespread speculation,
you know, that this was a cyber attack. I remember, you
know, reading these reports, and you know, the bizarre thing is I
started, I got into sort of the public. I started interacting with the
public on cybersecurity almost a decade later, sort of in the eight nine timeframe,
and I remember, you know, into the middle of the teens.
We're talking twenty fourteen, twenty fifteen. I remember, this is almost you
know, it's more than a decade after the event. I remember experts standing
up in public saying that the two thousand three blackout was a cyber attack,
you know, and one after another I tapped these people on their shoulder and
say, have you read the report. This is a decade later, and
you're spreading misinformation. I mean, this was again such widespread speculation that that
you know, a decade later, people were still talking about the cyber attack
when in fact, it was a failure. It was a you know,
equipment failure, was a software failure. The the alarm server eventually rebooted spit
out all the alarms, but it was too late by then. So yeah,
this this, uh, And what I didn't realize until just now speaking
to Aaron, is that the blaster worm did have a role. It did
not cause the outage, but in his estimation, it impaired the response and
may have delayed the you know, may have may have prolonged the blackout for
some customers by you know, up to a handful of hours because it delayed
response because communications facilities were all messed up, failures of imagination, concerns about
you know, laptops, and nine to eleven concerns about blaster possibly having connections
to the two thousand and three blackout. What was next what you know,
It sounds like you and Michaelasante were identifying the problem. You know, we
need a solution. You know what, what did you do with the problem?
Well, I think we really need to make sure that we attribute the
first action to Mike. He he had the guts. He had a pretty
good job at American Electric Power, Like he was one of the first CISOs.
He was featured as I think CISO of the Year by several publications,
and so, you know, he he had a pretty cush life, like
he could have just gone on that path, but what he decided to do
was to take a risk, and he approached some folks at the Department Energy
and basically asked them the question, could we build a research test bed to
prove out some of these theories? Can we move from speculation to actual data
that would show us, you know, what's the actual impact? And how
do we protect these things? And so Mike's first miracle, I'll say,
to get this project started was convincing the folks at DOE to combine forces with
the Departmental Land Security, which just oftentimes hard in the federal government. Sometimes
people don't like to play nicely with each other and basically set up this test
lab out at the Idaho National Lab. Now, he brought a few other
people along for the ride, Very Coonley and other you know, really interesting
a wide variety of folks, power engineers and cyber people and military folks,
and it was just a really good conglomerate conglomeration of people that he brought together.
And in two thousand and six he invited me to come along for the
ride, and I felt supremely honored. It's like, oh, there's sort
of like this cast of characters from different parts of the universe that are coming
together to trying to solve a tough problem. And it was going to be
a sacrifice. I mean, moving from a company like Microsoft at going and
getting in a federal government job wasn't exactly the easiest thing to convince my wife
to do. It wasn't the easiest thing on my personal finances trajectory, but
it was the right thing to do. And so I moved my family from
Seattle, the suburbs of Seattle where we were living, to Idaho, and
we start on this project to basically say, how do we put our brains
together to prove to the world that this is really a problem, and so
we started to go out and do a sort of marketing show to go pitch
for funding because we had the facility, but we didn't necessarily have the funding
to actually run a full test, and so we would fly from Idaho out
to Washington, d C. You know, usually Sunday night, we'd get
into DC, we'd set up meetings Monday through Friday, and then fly back
Friday night. So that was our rhythm, as you know, essentially spending
the whole week, got in DC pitching the people saying, Hey, we've
got this idea, can we get some help to fund it? And we
wandered from civilian agencies like Dewey and DHS into the Pentagon and into some crazy
places in the intelligence community, and you know, we're essentially just kind of
kind of got hat in hand looking for the resources we need to put this
thing together. There was some tough experiences along that path. I can remember
one time in the Pentagon when we got to invite it in to give a
briefing, and during that briefing, an individual fairly rudely stood up in the
middle of the briefing and just turned his back and was walking out. And
before he walked out. He said, you know, if I if I
want to go kinetic, I'll call in artillery. So this was a senior
army official. And because what we were pitching in our talk was, hey,
maybe digital attack can have these physical consequences. Maybe you could actually,
you know, severely disable fighting for us by eliminating the support of the infrastructure
that's around them. And there were some other people who basically said, you
and your you and your R two D two language. You know, you
guys can go off and play video games or whatever. And so we didn't
have the most receptive audience. This was two thousand and six time frame.
Now luckily there were some folks who listened. We finally found some listening ears
inside of the Pentagon, inside of DHS, inside of DEE, where essentially
combined forces said, look, we were going to put together the budget where
we can do one test to really show what this thing can do and and
all of that hard work that Mike could work for for years, and that
I got to go along from the ride on, several others got to pitch.
You know, we finally got the resources to then start dreaming up the
tests that we were going to do and That's when we went back to Idaho
to kind of put our heads together to say what's the best thing we can
do and how do we actually live the Aurora test? Was it not?
I mean, the the the test was controversial. I remember a video leaked
and just about everything else was confidential. You know, you were on you
were on the inside of that. You know, where did where did Aurora
come from? What was it really and sort of what what can you tell
us? What can you I mean, what can you tell us today about
what happened behind the scenes there. The genesis of Aurora started with Mike and
others motivating us to ask the question, what are some interesting accidents that have
taken place relative to control systems and infrastructure? And we canvassed all over North
America and we ended up having a conversation with uh, a Canadian power engineer
who told us a story and I don't know how apocryphal was, but he
told the story of, yeah, one time someone tried to bring a coal
fire power plant online and the power was out of phase and ended up blowing
this coal fire facility up and everything had to get fixed and oh interesting,
Okay, So this aspect of large scale generating facility trying to link into the
grid and the power being out of phase. That was bad. So we
started to look at that, and then in conjunction with that research, we
started to look at, well, what are the digital components that marry these
generation and transmission and delivery capabilities together. And we started to zero in on
these these safety relays, these these relays that sit inside of the substations that
really serve as those those breakpoints where you can shut stuff down and stuff' out
of whack and you can try to marry stuff together. And in looking at
that particular technology, it it was very ripe for cyber attacks because the original
inventors of those pieces of those relays, they did not really do a good
cyber threat model. So they had things like hard coded use names and passwords
and always open network connections and just stuff that you didn't want connected to the
Internet and you didn't want bad people thinking about. So as we started to
fuse this information together, we said, well, if we can manipulate or
relay in a way that makes one side of the relay essentially a weapon to
the other side, that could be really interesting. And that was essentially the
genesis of Aurora. We really wanted to show a test that actually shook the
ground like we wanted something dramatic. And as we worked with the power engineers
and we started modeling this, couple of the senior power engineers who were involved,
they said, well, I mean, if the generator is big enough,
you can you could do some serious shaken. And so, as is
shown in the YouTube video that's up now, that generator shook when the array
the phases of the power on the two sides of that safety relay were essentially
put out a whack and in a certain way and it would shake one side.
And so we took that idea and showed that it was reality and it
was I remember the day that the test happened how ecstatic we were because it
was all just theory at the time, right, we had written this stuff
down and it was supposed to work, and you know how it is when
you go down the paths and like this, how how often does it actually
work? And we really had the budget for one try at this, so
we didn't have the ability to to do you know, multiple tries and so
okay, so that was to see it pulled off. You know, when
I talked to people about Aurora. I talked to them years later, you
know they there are there are voices in the community who were who were critical
about how the aftermath was handled. I've been I mean, I wasn't there.
I wasn't proud of this, but I've been told that the details of
the test were immediately I don't know, either classified or made for official use
only and basically hidden away. You know, very superficial details were you know,
became public knowledge and it experts were shown some of the details, and
bluntly, they weren't physicists, they weren't engineers. They didn't understand the physical
characteristics of of what happened, and there were accusations of the whole thing being,
you know, a fake, like I said it was. The public
reception was very confused. Can you tell us anything about what happened behind the
scenes. Whenever you do something for the first time, no one knows how
to handle it. And and that's the situation where on ourselves in that the
test had been conducted without necessarily you know, like a top secret classification around
it. The test was put together in a way where you know, so
many people were involved, it didn't necessarily have the same level of classification like
a pure DoD project would. And so you know, by the way it
was designed, and I think Mike did this on purpose. He wanted to
share the information to help people protect themselves. And I think that's why Mike
designed it that way. He could have designed the test to be ultra high
classify that sort of thing. So it was it was designed from the beginning
of something where Mike wanted to share that information. And because of my background
doing vulnerability reporting at Microsoft, he asked me to lead the report, to
write the report of sort of what was going to get sent upstream to the
sponsors, the people who had, you know, helped to support the test
financially and eventually to DHS because they were the they were positioning themselves as the
industrial control systems certed. Right, So we get the report written, and
the report was written on you know, non classified systems on my laptop sitting
on just the enterprise network and I and L and we took that report and
sent it up the chain. And exactly as you said, the people who
are on the receiving arm of that, the folks at DHS were much more
accustomed to traditional cybersecurity problems not industrial security problems, and that's where there was
some confusion about well is this real, what's the impact, like how should
this be treated? And because you know, we at I and L,
we didn't really have good guidance about what we should do. We wanted to
balance protecting the information so it didn't enable malicious use of what we just just
discovered, but still providing guidance to infrastructure owners should protect themselves from these types
of attacks. And that's the agan almost ninety days of really really crazy conflicts
between people, and whenever there's uncertainty, people tend to become their worst selves,
self protecting, territorial, egotistical in some of the things that happened,
and I think that really set back what was the potential to be able to
talk about this now. Once the video leaked to CNN, there was immediately
a witch hunt to say, okay, who leaked this thing? It was
the one that leaked this thing at CNN, and lots of fingers were pointed
all sorts of directions. But I think that was probably the best thing that
could have happened because it basically allowed for other people to look at it to
go, wait a second, this could make sense you had people from other
disciplines outside of the typical cybersecurity domain that we're looking at it. And I
think once that video was leaked took a lot of the pressure off of us
at I and L because at that point, the horse left the barn,
train left the station, and that's when more we got drug along for the
ride. The ride at times was not fun because again there was there's politics
involved, there's egos involved, and whenever something new happens within the government,
there are vested interests to say, well, I want to own that,
I want to own that program. And so there was some competition that went
down between the labs about who got who was going to get new funding and
what was going to happen. And and that's where there was a huge tax
on us as a team, and there were and it showed in people's personal
lives, like you take a look at what was happening, you know,
outside of work, and it just wasn't a fun situation. And all of
that that great team that we would put together, that cross domain interdisciplinary team,
people from all over the world and all over the country who are working
together, you know, it wasn't fun anymore. And so myself and included
I sort of separated myself to say, you know, maybe maybe this isn't
what I'm cutting out from what I'm cut out for. Maybe there's better ways
I can, you know, go after my desire to protect the world in
the universe by you know, following by promoting cybersecurity in other ways. And
so, you know, by by the two thousand and eight timeframe, we
had lost probably about half the team, and that's when I left, I
know, I was in late two thousand and eight, and I went on
to go do a series of cybersecurity startups focusing on everything from mobile to cloud
and everything in between. And you look at that team that was there,
excellent, great people that went on to do great things, sometimes within the
industrial community, sometimes outside. But it was sort of sad to see it
get torn apart because of that the uncertainty about how to handle this, and
I think that's the danger of whenever you do something new, you know,
people don't know how to handle it. The latest numbers in the twenty twenty
three Threat Report on OT cyber incidents show that the threat environment has changed fundamentally.
At the beginning of this decade, OT cyber attacks with physical consequences have
changed from a theoretical problem to a very real problem more than doubling every year.
The new report is focused on deliberate cyber attacks in the public record.
These are attacks that cause physical consequences in process industries and discrete manufacturing. Most
of these attacks are ransomware, though the fraction of activist attacks is growing,
and the report's appendix includes a complete list of all cyber attacks since Stuxnet that
meet these criteria. To see how today's OT cyber threat environment has changed,
I invite you to download the report, a joint effort between Waterfall Security and
the ICs drive OT Incident Repository. You can download the report at Waterfall dash
security dot com, slash twenty twenty three dash Threat dash Report, or just
go to the resources menu at the Waterfall Security site and click on white papers
and Ebooks. Andrew, I must have seen the grainy footage of the Aurora
generator test by now dozens of times, just because it comes up so often
when you're talking about OT cybersecurity, with stucks Net being the big overall attack
that everybody knows about, but Aurora being that progenitor of this whole conversation,
and so it's sort of interesting to me just to hear Aaron's background on it
as somebody who is directly involved. I'm even just watching the video now.
It's it's sort of it's a very interesting case because you see this giant,
hulking, green metal machine of a thing that is clearly in distress and then
creating black smoke and it almost seems like it's about to blow up. The
notion that that could happen just from a cyber incident, as much as I
can understand that academically, is still to this day interesting, very much so.
And you know, in the moment what I remember when it was released
the information, at least the video in O seven, I mean, the
rest of the detail didn't become public knowledge until years later. In O seven,
there was there was you know, it wasn't released on the news.
It was on CNN. You had cybersecurity experts weighing in on CNN on you
know, social media, what social media existed in the day. A lot
of the feedback that, you know, a lot of the experts weighing in
were cybersecurity experts, not physicists, not engineers, with really little or no
understanding of the physical process, and some of them were coming in saying it's
all fake. It couldn't really have happened that way without again without understanding the
physical process, and in my understanding in terms of the physical process, what
happened was inl has a full power grid. It's a massive test installation that
the generator was connected to as one of many generators on this simulated power grid,
and what they did was trip the breaker. So disconnect the generator from
the grid for a short period of time I assume a fraction of a second,
and what happens. I mean the generator is underload, it's supplying energy
to the grid, The grid is consuming the energy. The generator is working.
The moment you're disconnected from the grid, it has no load anymore,
but there's still energy in terms of the diesel engine spinning the generators, still
energy going into the generator. The generator speeds up, and now the power
it's producing and going nowhere, you know, just heating up the wires.
The power it's producing is out of phase with the power in the simulated grid.
A fraction of a second later you reconnect it, and now there's enormous
stress torque. They call it on the generator because when you've got you know,
a generator and the grid fighting it out for who's going to win.
I'm sorry, the grid always wins. The generator is forced back into phase
in nothing flat, you know, with enormous stress, enough stress to destroy
the generator. You saw the video there and the you know the so we
saw that in the public sphere. What I didn't realize was sort of a
different debate happening in the in the in confidence in government, where people are
saying, oh, it is real, you know, I want to own
this problem going forward. I didn't realize that that that was happening. I
don't want to preempt anything you ended up discussing with Aaron, But from your
perspective, was there any major shift in the way that government worked with ot
sites or the way that OT sites worked on their own that may have directly
resulted from this? The incident was was widely reported. It was you know,
people talked about it for half a decade or longer after the incident.
You know, the the big news that the biggest news that happened after that
was sort of stuck snet that sort of preempted it. But you know,
there weren't a lot of examples in the public domain of cyber attacks that could
or did cause physical consequences, and so you know, the the the incident
was was influential. And you know, in in Aaron's estimation, you know,
the the turf war that took place within the government, you know,
was a turf war for funding and responsibility. It was you know, when
when that turf warce settled out, there was funding, there was an initiative,
and you know it was it was sort of instrumental in cementing that initiative
going forward, is my understanding. But now coming back to the test itself,
you maybe I'm misremembering mentioned that the generator was destroyed. Now, from
the publicly available video that I've seen over and over, you do see a
ton of black smoke coming out of it, and it's sort of shaking,
and it seems like it's in a state of real panic, this machine.
But the notion of this thing being destroyed, and if anybody's interested, just
look up a picture of this Aurora generator or blowing up in any meaningful way,
is still sort of unbelievable. You're telling me that there was more damage
than what we see in this video, or you're just using a different word
for it. No, So, I mean the generator did not blow up.
It did not explode. You know. The video says that the smoke
rose out of the generator, that there was obvious vibration, and the analysis
of the generator afterwards, you know, the internal report to the government was
the generator was when you open that generator up, there's nothing useful inside anymore.
You can't generate power with it. You have to throw it away.
It was it was a write off, don't I don't know that the diesel
engine was affected as badly, but the generator was shot. And you know,
the diesel engine provides energy to the generator. The generator turns rotational energy
into electricity. And you know, I've had the privilege of visiting large power
plants in the past. When I see a large generator, I mean that
was a ten megawak generator, it's nothing by the scale of the grid.
A large generator is three hundred, five hundred, eight hundred megawatts, so
it's you know, between between thirty and eighty times as big. I saw
a five hundred megawat generator once, and it's you know, it's as big
as a bungalow, and it looks like a very large lump of molten metal.
You know, it just looks like you took big drop of metal and
dropped it, and you know, it landed, it hardened, and that's
what it looks like. And I'm going, that's not what I expected.
You know, I expected a generator to be rounder, you know, I
expected sort of And they said, no, no, you don't understand,
Andrew. They said, all of that metal on the outside of the generator
is to protect you and me standing here, because if that generator fails in
the worst case, and you know, an outer phase reconnect is pretty close
to a worst case. But you know, I was told, if that
generator fails in the worst place, it basically blows up. It's turning at
at least sixty cycles a second, sixty rpm, and if it flies apart,
this is three hundred tons of metal that's flown apart. And all of
that metal you see on the outside is to prevent that metal inside flying apart
from striking you and me and the building and all of these other generators that
you see down the massive building. So you know, it's a it's a
real concern. And in the modern world, like I said, people protect
these generators. There have been cases in the past where generators have blown up
or turbines have blown up. I think it was a hydro turbine in two
thousand and nine killed seventy five people. So these are very large pieces of
equipment. They're dangerous pieces of equipment. This little demonstration managed to destroy a
ten megawatt generator. But you know, the concern everyone has is that much
worse is clearly possible, so that, you know, that begs the question
here we are going on fifteen years later than two thousand and eight. You
know, there's a lot of water under the bridge. Since then, industrial
cybersecurity is a is a mainstream activity. You know, we still have we
still have lots of engineering teams who are just beginning to come up to speed.
But there's widespread recognition that, you know, this is a thing,
it's real. We have to you know, we have to act on it.
Did you you know, did you stay in touch with the community,
you know, in in your sort of contacts, your your view of the
of of the history. You know, how how was all of this confusion
resolved? How did we wind up sort of on a track to get to
where we are today. Well, again, I think we need to pay
tribute to Mike for being courageous enough to stay the course. Like he he
could have bowed out and said, hey, I'm going to go do something
else, but he leaned in with with Firk and Nirk and said, look,
we've got to do something about this. And and as the result he
spent some time researching where would be the best place to land to keep driving
this this forward. And the other person I think we should really pay tribute
to, who also unfortunately is not with us, is Alan Power, the
founder of SANDS. So Mike and Allan had known each other through other you
know, training relationships, and Alan really put himself out there to say,
you know what, because SANS has this platform to provide meaningful technical training,
because SANS has this great certification mechanism where you go for this training, and
SANDS certificates, you know, still to this day, really stand above others
because of the depth of technical training that you get through those those courses.
And so Alan and Mike basically agreed to say, you know, let's create
an industrial control curriculum. And that was the best thing that could have happened,
because at that point Alan had the resources to push it forward, to
basically fund the creation of a vendor neutral forum for people to go and learn
meaningful things. But Allan also had the political connections because Alan and I had
known Alan from the time when he first started SANDS and I was wearing at
Microsoft. We collaborated on sharing course materials around Windows security because Microsoft needed some
folks to go teach the US military about how to secure Windows systems, and
Microsoft did and wanted to maintain an arms like relationship there. So SANDS became
a great channel that I collaborated with there. And so with that connection with
SANDS, that's really where the what I'll call the flowering of public knowledge in
a proactive, well defined way. And as a result of that SANDS curriculum,
DOE sort of I guess there was a there was a peace movement between
what had happened between the Aurora test and some of the DHS stuff that had
gone on, and so DHS and DOE went along with that and created their
own course materials. And to this day you can still go out to the
IDON National Laboratory and participate in hands on technical training around industrial control and so
I think that was really the combination of sands plus the ability of dealing and
DHS to put together a curriculum there that was really what put this in the
position where we're at today. And now you take a look and there's been
a flowering of startups, you know, folks like Dregs and others that are
out there that have really tried their best to help this community. And I
think that's what really it's us in a situation where in today, which is
a much healthier one where people can have open and honest discussions about the convergence
of control systems cyber physical attacks and you know, the price we have to
pay now is that we've seen several I mean just in the last year or
two years. Probably the ones that are most interesting to me or what happened
with the bul Russian railroad system as a result of some probably Ukrainian attacks against
that railroad system to stop the delivery of tanks to their northern border. But
you know, there's there's been some terrifying things what you've seen as a result
of cyber physical convergence. But it's the world we live in now, and
I think now we have the ability to have open and on his conversations about
what we can actually do about it. So that's really interesting. I mean,
I I knew Mike. I knew Mike uh Asante to see him,
you know, he was he was a fixture at DHS and other events.
I kind of I kind of knew him as the the He was one of
the senior managers at NIRK, and you know, he he was infamous.
He was I think he was only there a couple of years, but he
was infamous for sending out a letter saying, guys, uh you know.
This version of NIRK says NRK sip rather says that you have to self assess
as to which of your assets are critical to the reliability of the ball electric
system. Some large power utilities out there have identified you knows or even hundreds
of uh you know, physical assets and cybersystems that control them as critical to
the grid and have taken measures to protect them. Other utilities, just as
large have come back and said, absolutely, none of our equipment is critical.
We all know that these both can't be true, you know, fix
this. I remember I'm paraphrasing that that was what I the the the sort
of the takeaway that I recall from the letter that was sort of where I
was introduced to Mike, and then you know, I saw him later on
at Sands. You know, I had, I had, I had none
of this this background before. Yeah, so if you think about, you
know, what the what Mike did is he put himself out there to basically
say, we've got to make a change. And I think that letter was
part of it. You know, he he continued to work closely with Congress
to you know, motivate folks to make sure that the right at least partial
legislation was in place, to try to say, hey, we've got to
do better about protecting critical systems. He did a ton of lobbying with dhs
to make sure that they were empowered with knowledge so that they could build the
right working groups and keep moving it forward. And so he was critical to
it. And I think what a lot of folks don't understand is that,
you know, he he was a cancer survivor, and that was one of
the things that attracted me to work with him. I'm also a cancer survivor,
and so you know, whenever you face death, you know, both
he and I got terminal diagnoses where we were supposed to die sometime in two
thousand and six, and that also motivated us to go out the I n
L. Because if the diagnosed was right, we kind of both wanted to
go out with the bank. Well, you know, fortunately I have continued
to fight mine I was. I suffered from melanoma, but he suffered from
a non honchosuphoma, and unfortunately he had a reoccurrence and that's the reason why
he passed away a couple of years ago. But I think the the thing
that we look at now is, you know, Mike's ability to focus people,
to get people on the right path. And that's why we're we are
where we are today, is because he had the courage to write letters like
he did at to basically stand up in people's faces and say we've got to
do something about this. And and that's the reason why there's scholarships named after
him and awards and the cybersecurity community and it's all it's it's all Merid like
there's there's a whole bunch of stuff that Mike did that no one will probably
ever know because he wasn't a bragger, he wasn't a guy who wore all
of his achievements on his sleeve. Will probably never know the full extent to
which he dedicated his life to make the world a better place. And I
just got myself as lucky that I got to go, I got to work
with him and got to know him. So yeah, Nate, as I,
as I said on the on the interview, you know, I knew
Mica Sante from his days at RK. I think he was the chief security
officer officer there for like two or three years, and you know, then
he moved on and I remember him eventually, you know, in before he
passed away, he was in charge of the industrial control system training program at
SANDS. But you know what little I knew about him personally is that,
you know, he wasn't afraid to make waves. I remember that letter that
came out and I think it was two thousand and nine talked about Look,
you know Version three says you're required to you know, these power utilities are
required to define a risk assessment methodology. You're required to apply the methodology to
your physical assets, the generators and the transformers and the substation. You're required
to identify which of these physical assets are essential to the reliability of the grid.
You are required then to figure out which computers, if any, are
essential to the correct operation of those physical assets. Those are your critical cyber
assets. You have to apply the rules in NIRKSIP to the critical cyber assets.
He said, a lot of you large power utilities that you know probably
have cyber critical assets and critical cyber assets have come back and said we have
none. You know, this is going to have to change. And you
know it was controversial, I think because people interpreted it as you know,
accusing the power companies of not caring about the reliability the grid. And you
know, I reread the letter and uh, you know, I don't I
don't see that. I mean, there he's identified a problem. He says,
this methodology has been applied inconsistently, and you know he gives he gives
you the power companies. Now, he says, look, you know,
in his estimation from talking to the utilities, it has to do with redundancy.
The grid is massively redundant. If a generator goes down, there's other
generators that can up the load. If a substation goes down, there's other
paths through the mesh that is the transmission grid to get power from sources to
destinations, and he says that, you know, the fact that you have
redundancy does not make these devices not critical. Yes, any one of them
can fail and the grid keeps going. But he says these devices are still
critical to the grid because in the world of sort of random equipment failures,
you can count on redundancy. In the world of cyber attacks, deliberate attacks,
you might have an attack that takes down multiple similar assets that are similarly
defended, and now the redundancy has been bypassed. And so you know,
to me, it was it was, it was reasonable. But again it
was controversial in the day because he pointed out this inconsistency in a very public
way. Wow, well, thank you for that, and thank you for
joining us. I mean this has been a you know, insightside. I
didn't have into you know, the history, the beginnings of the industry that
now has thousands and thousands of practitioners in it. You know, before we
let you go, can you sum up for us what you know, what
should we what what should we all take away from the history, What lessons
should we should we you know, carry around with us. So I think
the first thing is is that the older we get, the more rigid our
thinking becomes. And luckily Mike and I were both young kids who were willing
to challenge the status quo. We were willing to challenge the the incumbents and
basically think evilly right. We were the ones who really started to say,
look, what's the worst thing we can do? And I think that's something
that we always have to be willing to consume. And whether that's you know,
inviting you know, outside folks to come and do penetration tests and and
be able to evolve models, I think that is so so important. And
so I would say, you know, if you're a security leader, someone
who's been around in the industry for a while, someone who owns large infrastructure
systems or whatever, be willing to bring young folks in who have new thinking
about new ways to approach how do you compromise these systems? How do you
turn a protection what was maybe a control of designs of protection into a weapon?
And we always need that fresh thinking. So I think step one always
make sure that you're open to critical thinking and to evolving threat models so that
you can understand, you know, how to go about doing things. The
next thing I would recommend to folks is as you make investments in cybersecurity,
sometimes simpler is better. So over the last thirty years there's been several phases
of my career where I've seen people say, you know what, I'm going
to go out and buy every security tool on the planet and just start layering
the stuff all over the place because more is better. Well, the situation
we find ourselves in now is more may not be better because it's too noisy,
because it's too it's giving you telemetry that's maybe false positives. And you
know, as much as sometimes we want to avoid single points of failure,
want to avoid situations where we don't have great resiliency through through distributed or diversification,
you know, we're nearing a time now where we're seeing the proliferation of
attacks, especially through identity control systems, where you know, even very supposedly
strong identity systems that have features like multi factor authentication, that identity system itself
is compromised, thereby eliminating the need for MFA to get into the system.
And so sometimes those complex identity systems come back to bidas because we've cobbled these
things together. So simplification in things like identity ecosystems, simplification and things like
network segmentation. I think those are things that we need to engineer towards as
system owners of how do we simplify to get better security results. And the
last thing that I that I'll put out there for the community is we need
to find the next version of Mike. I don't know where that person sits,
very likely not within the cybersecurity domain. I think the diversity of thought
that comes from other disciplines is what we need to keep ourselves fresh in cybersecurity.
And we've got to be looking for those people and giving them chances to
come in and participate in meaningful ways. And I think with those three things
we can we can keep moving forward to what got started fifteen twenty years ago.
Andrew, that was your interview with Aaron Turner. Do you have any
final thoughts that you might want to end with today? Yeah, I mean,
let me repeat his three points. He went on for a little bit.
You know, he said, in my recollection, be paranoid, challenge
the status quo in terms of, you know, bad stuff that could happen.
He said, simplify, you know, simpler is better, he said,
you know, diversity across disciplines. Uh, you know, bring bring
fresh knowledge in, especially when we're talking you know, he didn't say it,
but in my mind, especially when we're talking about physical consequences, you
can't you cannot really get an understanding of the physical consequences without bringing in people
who are experts on the physics, experts on the engineering. So you know,
be paranoid, challenge the status quotes, simplify and you know, bring
people in who know about, you know, how things work. Makes great
sense. You know, lately, I've been very involved in the cyber informed
Engineering initiative and it's saying some of the same things he's saying. It's saying
that, you know, we have to teach engineers to be more pa aoid.
We have to you know, use powerful simple tools that engineers have,
you know, over pressure relief valves, mechanical overspeed governors use these simple tools
as last ditch stop gaps, so that even if all of our cyber defenses
fail, we still have physical protection from catastrophe, and you know, diversify,
you know, bring in the physical experts. There's a lot of knowledge
that's needed in the space. A lot of it's in the head of engineers.
Some of it's in the head of you know, chemists and physicists.
This all makes this all makes perfect sense. So you know, I think
you know, Aaron has sort of not been active in the field in most
of a decade, but his advice is right on the money. All right.
Well then, thank you to Erin for sharing all this with us,
and Andrew, thank you as always for speaking with me. Thank you very
much, Niate. This has been the Industrial Security Podcast from Waterfall. Thanks
to everybody out there listening.