Speaker 1: We use a reactive AI in our methodology to actually
pick the right countermeasure which blocks the technique that the
attacker is using that in that moment in time.
Speaker 2: Welcome, Everyone's the Industrial Security Podcast. My name is Nate Nelson.
I'm here with Andrew Ginter, the vice president of Industrial
Security at Waterfall Security Solutions, who's going to introduce the
subject and the guest of our show today. Andrew, how's going.
Speaker 3: I'm very well, Thank you, Nate. Our guest today is
Gary Southwell. He is the general manager and vice president
at ARIA Cybersecurity Solutions, and we're going to be talking
about AI, a new use for AI in protecting critical infrastructure.
Speaker 2: Then, without further ado, here's your conversation with Gary Southwell.
Speaker 3: Hello Gary, and thank you for joining us. Before we
get started, can I ask you to, you know, say
a few words about yourself for our listeners, and you know,
say a few words about the good work that you're
doing at ARIA Cybersecurity.
Speaker 1: A little bit about me, So my background, I've been
in cybersecurity really since the early days. I was a
system engineer in the nineties, you know, working on some
of the initial checkpoint deployments as firewalls. Spent a lot
of time at Juniper Networks trying to improve the way
we did network security with more of an intrusion detection
prevention systems. Moved on into a company called Secon where
we worked on adding artificial intelligence to a SIM product
so that we could do more advanced managed detection response
solutions for MSSPs. And really over the last seven years
I've been here at ARIA Cybersecurity and we've really looked
at how do we actually stop the tax that we're
seeing in the news. And if you date back, you
know the attacks we're talking about are the ones that
are really going after the critical assets that are out there,
everything from the colonial pipeline on. We want to make
sure that we've got a better way to actually go
in utilize the artificial intelligence properly to find and stop
these types of attacks.
Speaker 3: Thanks for that. And our topic today is using AI
to protect critical infrastructure. And my understanding of the way
you know that you folks use AI is in the hosts.
I mean, you know, outfits like Juniper and others have
used AI in their firewalls for a long time to
look at the messages coming to and try and figure
out you know, do these things? Are these things attacks?
You're doing something different? Is my understanding. You're not focused
on the firewalls, You're focused on the hosts. So you know,
we already have in a sense anti virus on the hosts.
We have whitelisting on the hosts. What is the problem
you guys are trying to solve and why isn't it
already solved with antivirus and whitelisting.
Speaker 1: We believe you want to actually stop the attacks at
the point where they're actually attacking their critical applications, which
is what drives your critical infrastructure. So the challenge with
solutions we call these active solutions that they're going on
the host is that if you look at traditional anti
virus that's been around with us for twenty years, is
that they are looking for files typically that land on
the device and they have a known bad signature that's
calculated off of looking at the file. You can calculate
a hash and it comes up with a value. If
it matches something bad that I've been told about, I
can block it from running. It's a great concept. The
problem is malware of all kinds, including your ransomware. The
signatures are now polymorphic really over the last five years,
so you get different value every time it lands on
a different device. That approach really isn't effective anymore as
we've entered this decade. The other approach, as you mentioned,
is whitelisting. It kind of got a bad name because
it's been difficult to use, But whitelisting just means, or
we call it access control these days. Is it says
I can delineate which application should be running on this device.
I can check their SERTs and then make sure that
it is that application and then allow it to run,
and by default I'll block everything else. So these are
good approaches if we could make them work in today's environments.
So we start there. We were saying, we need to
make sure that the applications, the critical applications can run,
and anything else that would land like a file based
malware ransomware, we're not run. That's just your very basics.
The challenges with these zero data attacks, as I said before,
that they can't detect detect them with signatures. So the
industry kind of evolved over the last five years to
looking at patterns of behavior or what we really cause.
The industry indicators of compromise. When something bad lands on
a device, it does this, and then this and then
this through the kill chain, we can identify the pattern.
Once we know the pattern, if we have a way
to then block one of those steps, we can disable
the attack. We call that next generation anti virus. So
vendors like CrowdStrike, Centa one one do a really good job,
you know with those types of approaches. However, when we
get into what are the rest of the attacks, these
are more sophisticated level attacks like the ones we've seen
in the industry, starting with Solar Winds, where you've actually
got humans behind the attacks. They may use some form
of credential use or deposit software through a legit channel
like they did with Solar Winds, and then they progress
an attack once they get in, get a foothold, and
they begin to take additional steps. So we've got a
myriad of attacks. So bringing this back to how do
we use AI. We want an approach that can actually
detect the various attack techniques that are happening, and if
they are somehow adult trading an application, trying to deposit
a new one on the device, or you're coming in
and using existing processes against themselves. We used to call
those advanced persistent threats. They're more generic, very specific ones
are like living off the land, where they're using the
actual OS processes against the actual device. We need to
be able to detect that and apply the right measure
to stop that type of attack. And this is what
we believe is the role of AI on the device.
It's not your generative AI. Just make sure you understand
that the typicoally needs massive horse power. This is the opposite.
We use or reactive AI in our methodology to actually
pick the right countermeasure which blocks the technique that the
attacker is using in that moment in time.
Speaker 2: Andrew, I would just like to second a lot of
the points that Gary has made here. It feels like
we might have crossed some sort of vague threshold in
recent years where traditional detection antivirus has started to work
a little bit less. I mean, as he mentioned, the
number of variants and samples out there of your typical
malware can get really crazy. I mean, only in recent months.
I recall a report about mobile malware. I think this
was published by I Forget the vendor, where you wouldn't
think of mobile malware as necessarily quite as subject to
security analysis as traditional PC malware, and yet so many
malware families that target mobile devices these days have hundreds
or thousands of samples out there. There's a malware called
Godfather with well over a thousand nexus sedarit picks pirate.
So those kinds of solutions that would have picked up
based on traditional fingerprints may not suffice anymore, even in
the mobile realm let alone PCs where we would expect it.
And there's also the fact of he highlighted briefly their
behavioral indicators of compromise. This to me seems like where
cybersecurity across IT and OT has been going lately. Traditional
indicators of compromise can help. But I recently came across
report from Mandiant about Chinese ORB. They call them networks
ORB being short for Operational Relay Box Networks. It's not
an entirely new concept, but basically in China there is
an entire economy of folks who provide infrastructure to thread
actors across the Chinese spectrum. And so where once we
might have been able to say, you know, look out
for these servers they've been used by this group, well
now those same servers are being used by all kinds
of groups, and so it's less helpful now. So the
way that we are adapting is by tracking behaviors of
threat actors rather than static indicators like that. So this
is a very long way to say that I agree
with all of the points that CARRY has made.
Speaker 3: Yeah, and you know you're talking about polymorphic. These our
viruses that change frequently. Either the bad guys change them frequently,
so there's thousands of variants out there, or they change
themselves they sort of self evolve. So again there's thousands
of variants out there, so that signature based solutions have
a hard time keeping track, you know, publishing enough signatures
fast enough to track things as they change. The traditional
sort of alternative to signature based anti virus in the
industrial space has been whitelisting or application control some people
call it or allow listing. What it is is a
list of programs that's allowed to run on your industrial
computer and blocks everything else. So it doesn't matter what
the latest signature of the virus is if it isn't allowed,
if you know it's not on the list of allowed,
then it's blocked. But you know that class of solution
allow listing is itself limited. So for example, when you
install software updates, you have to update the list of
allowed applications because you've just changed a whole bunch of
your applications. You've changed their signatures, you've changed their sizes
and their you know, so having changed all that, that's
a vulnerability. If the bad guys can get in there
and hack the update process, they can get their nasty
listed as an allowed executable and they're off to the races.
Other ways that allow listing is is you know, limited,
Everything is limited. I'm not knocking allow listing. It's a
it's a useful solution, but bear in mind that you know,
it tends to focus entirely on executable codes. So DLLs,
dot xs, dot COM's coming off the hard drive into memory,
and that's when they apply their checks. If your malware
is scripted, well, i'm sorry. The Pearl dot x is
allowed because the operating system needs it, or the the
control system needs it. Python dot x might be allowed,
and now you're loading a nasty as text off the drive,
and allow listing is kind of blind to that. They
don't really check text files. And you know another class
of application that that are of bad stuff that you know,
allow listing is blind to is in memory attacks. So
if you can comprom something and start executing with a
buffer overflow or something and start executing your own code
and inject insert the malware into memory. Again, allow listing
looks at things coming off the disc. It doesn't look
at what's happening in memory. So yeah, you know we
need as well, malware becomes more sophisticated, We need more
sophisticated tools to diagnose it and deal with it. And
you know, here's a new kind of tool. And these
are some examples of why we are always looking for
sort of the next step in these tools. You talked
about sort of attacks that can defeat the access control
as well. Can you go into a little more detail.
You know, we're going to talk about AI in a second,
but you know, can we nail down what is sort
of a modern advanced piece of malware?
Speaker 1: What's it look like?
Speaker 3: And you know, can you sort of not comprehension, not
every kind of a maount where that come after us?
Can you give us sort of one example of the
kind of thing that existing anti virus and access control
might struggle with.
Speaker 1: Let me pick one example to let the audience get
their heads around the use of advanced malware and what
do I mean by a sophisticated level of attack. So
one comes to mind I mentioned a minute ago, and
this was the type of attack that really set the
industry kind of its ear. So I'll date myself back
to January twenty twenty one. This is when the discovery
of the Solar Winds attack was happening. Now, this was
a nation state backed attack and if I step into
it a minute, it was very very clever. The way
they did it. It was called a sunburst attack used malware,
but the way they decided to get in was to
actually go into Solar Winds infestrutructure and as they were
packaging there oryan software for an update, they actually put
in we call a shim form of malware inside that update,
very very small so that would go through and not
be easily detected. You didn't want to be doubling the
size of the update that might get picked off. So
it was very very thin and its sole objective was
to get in and as they orian software update was initiated,
it would be ready to do its one task, and
that's one task was to call back out of the
location back to a commanding control location where the rest
of the attack had happened. Okay, so this is very
very small piece of malware. Actually being embedded as one
of the tool sets inside of the Orion update was clever,
but they also made it what would call polymorphic, so
as it deposited inside there, it slightly modified itself so
that if you were to calculate a signature, it would
be slightly different for each form of deposit. This makes
it a lot harder for us to go and say
we can pick it off, and because it's now embedded
to a tool set, that makes it very difficult because
most of today's more sophisticated solutions weren't looking at that
level trying to figure out what was happening, and it
only really exhibited one behavior, it did a callback. What
happened next, though, was after the callback, the attack actually
had twelve more steps, and the callback brought in another
form of malware, and that was more what I would
call it the business end of the attack, so that
had more capabilities inside of it to launch and step
up the attack to allow it to figure out what
was on. And then its other role was then to
attempt to spread inside the environment if it could, and
then pull back information and that's when humans would get
involved and then would begin to take additional steps. So
that's an example of polymorphic malware enabling the beginning of
a sophisticated attack that would get by most of the
protections we have today as an industry. So the next
part of the attack as it began to evolve, had
multiple steps, and one of the steps that was in
the attack, and this was brought up in the Senate
hearings where they brought in some of the industry leaders
Palo Alto and CrowdStrike. Of course, Microsoft and Solar Winds
were in the panel, and they said to them and
they all agreed, it's like in some of the situations
you were present on these devices when this attack was occurring.
I'm trying to quote Marco Ruby Oakes. He asked the
question it was reported that you were bypassed, and everyone
in the panel said yes, that was in fact the case.
So bypass just means that in effect, they either couldn't
see the attack or more likely, the attackers are actually
able to have control of the system and basically disable
them at the task master location at least temporarily. They
probably come back right up and boot up, but it
then lets them get by if they're doing something else
that they might might see an attack. So these were
fundamentally new sets of challenges that the industry now had
to face. That you could use basic tools like malware,
but in the hands of sophisticated actors, they could do
an awful lot of harm. And as we saw with
solar winds, it went on for almost a year without
actually being detected. It was fortunate enough that fireri actually
saw some of their tools actually leaving their environment, and
that was the first time that was ever picked up.
Since then, we've had a series of similar I think
the industry, the OT industry itself from last count had
about seven hundred of these types of attacks actually happening,
not all from the same actor, by the way, but
other actors, because once the formula was figured out, sophisticated
attackers could then utilize some of these same techniques. So
this has been so The ones right after it were
the colonial pipeline that was a simplified variant of what
we just heard about, But there's other types of attacks
that have gone over the years.
Speaker 3: Can we talk about AIE. You know, this is the
topic here. If we had a magic AI sitting in
our industrial control system hosts, what would that AI do?
How would it detect attacks like this?
Speaker 1: Well, that's the part that's the most challenging, and that's
why you need some form of AI. There are very
different techniques that are being applied by the attack type.
Some cases, it's just recognizing that there's form code that's
appeared here. In other cases, it's like I've got to
understand that there's an abnormal operation happening in conjunction with
this legitimate application. There's some form I will call adulteration
going on here. In other cases, it's the application is
running fine, but for some reason, it's going from a
user level trying to escalate itself to a system level
that allows them to get control of the application. Or
it's trying to use processes inside the OS that are
not affiliated with an application, or it's a spoofed application
variant that's actually trying to initiate the processes on the OS.
Some of these. I'll bring that up. Is it came
out in the Pool Party attacks that the last Blackhead
in the UK. They showed eight different forms of thread
processes that are available from the OS for the applications
to use that attackers could easily take advantage of. So
I've really described three different types of techniques and the
inside there there's a variety of combinations. So this makes
it very difficult to figure out how to stop all
forms of attack. If I want to make sure that
we're doing the best job we can at the host
to stop whatever may be happening, whether it's a zero
day form of malware or ransomware that we haven't seen before,
we haven't seen the IOC patterns and it's trying to
just do its thing, or it's one of these variations
of attacks that these sophisticated typically nation state back but
now it's a crime backed attacks that are out there.
That these kits are out there and they can vary
their attacks. So the AI really needs to make sure
it's saying I can pick off what's happening, and then
what do I do about it, just like a human would.
I've identified it's a sophisticated attack, someone's using a privileged escalation.
I'm going to apply this countermeasure to block it. I've
discovered this is an interesting piece of code that's arrived
here and I need to block it. I recognize that
this application is no longer working the way it should.
It's actually coppying things off into buffer spaces that it
normally doesn't do. I need to stop that from happening
and block that operation. Or I've got unattached processes from
the OS that should not be running. I need to
make sure I can block them at this moment in time.
So this is where the AI comes into play from
our experience here in the industry.
Speaker 3: Okay, so let's get specific here. You know, we've been
talking about the problem sort of in the abstract and we're
drifting into you know, you folks have this stuff. Can
you give us just a quick rundown? What do you have?
How does it work?
Speaker 1: You know?
Speaker 3: Why do people deploy it? What are we talking about?
Speaker 1: You? So, in our particular application of this technology is
we built a very lightweight agent and it's different from
your typical agents. We are going in at If you
understand kernels, especially in the Windows environment or Linux environment
where we play is wetach it ring zero right at
the kernel level. The reason we do that is we
want to see everything that's going on as far as
processes from applications that are leveraging what's happening into the
kernel and vice versa. The other thing that we do
that makes us fairly unique. We actually have some patents
on this, so we're hoping it'll stay unique is that
we actually watch device memory continuously. This is the way
that we can actually pick off some of these techniques
when you've got adorable use of bupper memory, or you're
actually seeing some process kicking off where something's being written
over here into notepad and that's now being imported into
the application because it's probably giving them access to some
form of change the application that they want to leverage.
We do that inside our agent. The markets that we've
chosen to go after, though, you know, have a variety
of requirements. We're typically talking about operational technology environments, and
there's many of them. You know, if you think about manufacturing,
we're talking about manufacturing floors. We think about utilities. We
are in the process devices out there that have an
OS on them that are helping them run and control
electrical generation and distribution. You're talking about oil and gas,
same way you're dealing with the processes. So these are
environments we chose to go after because they are high
value targets. As we saw what the colonial pipeline is
a good example. When we get into these environments, we
also find that you have got other constraints. Typically there's
not continuous Internet connectivity. In fact, they purposely try to
limit that as one of the protections. They often have
limited processing power available left to anything else that's going
to run besides the production applications. In some cases, some
of these devices that are old or they're running what
I would call old versions of the OS because they've
been trying to sweat that asset for many, many years.
You know. For example, we're deployed in a a large
pharmaceutical around the world, and different locations will have devices
that are in these various lines and you'll see, oh,
I see Windows Server two thousand and eight over here.
Some locations we actually see Windows XP typically service pack too,
which is nice because it's got nice controls. And they're
still using that asset because they built their applications on
top of that, and everything just runs and in typical
OT fashion, if it works, don't change it. And then
their hope was let's wall off the environment using passive
protections from the networks if we can. And yet what
they found is supply chain attacks get around the network protections.
So what we did is made sure that our application
could run on these older operating systems. It could run
with very very limited amounts of CPU and limited amounts
of memory, so that we could perform and not impact
the performance of the production applications. The benefit of this
approach is that we can go on a myriad of
these devices with many, many different forms of applications. And
I'll go one step further here is when we're deploying
in these environments, we expect to see tens of applications.
In fact, in some cases we actually see upwards of
a thousand of these applications. So our approach as we
deploy is to prevent the adulteration of applications. One of
the side benefits is in these environments is that when
you have this many applications, you're going to have known
vulnerabilities inside those applications by published by cvees, and the
chance of for ever having in them all continuously patched
even an IT environment is almost know and when you
only have a chance to patch maybe once a quarter
at the fastest and more likely in this pharmaceutical company,
it's once a year. You're never going to be patched.
So one of the side benefits was because we go
in and protect these applications from adulteration, the ability to
exploit these vulnerabilities becomes significantly less. I won't say zero,
but we can go down to you a ninety nine
percent chance that we are going to block the exploit
of these applications. And this then becomes a real benefit
because now you've dramatically improved the likelihood that we can
keep these operations operational even during an attack because we
will continuously block those attackers. That's the benefit. When people
went through the risk analysis, they're saying, Okay, I can
look at it. The cost of my line if it
goes down for X number of hours is thousands of dollars,
and if it goes down for a month, it's millions
of dollars. And if you've taken down my risk by
a factor of ninety nine percent, I can actually calculate
a value to that. So these are the benefits that
we're offering. No, the challenge really that is you've got
all these applications, how do we actually make it easy
for these operators to use this technology, so.
Speaker 3: There was a lot of stuff there. Nate, let me
come back to the Solar Winds example. Gary said, the
malware came in as part of the Solar wind security updates.
So that would have defeated the white listing the application
control I was talking about, because it would have come
in saying, hey, here's a new authorized executable and the
malware had been flagged as authorized. And then what the
malware was, you know, was something that phone home that
called out to the Internet and said, hey, boss, I
got a live one here, and did not much else.
It was very thin, it was small, It was benign looking.
A lot of malware phones home to the vendor. Not
just malware, a lot of legit software phones home to
the vendor and says, you know, here's what's going on,
because the vendor is helping manage the software. So it's
not that suspicious, you know that the malware is phoning
out to the Internet. The alarming thing is that what
it got on the Internet was here's another whole bunch
of code, and it copied the code that it got
from the Internet into memory and started executing it. And
at that point it became dangerous. It really started doing
nasty stuff, and so again, you know, whitelisting would have
missed it as part of the software update, would have
missed it as in memory, you know, pulling stuff off
the internet or off the socket the connection out to
the internet and inserting it into memory and starting to
execute it. This is where we need, you know, sort
of a deeper insight into execution so that I think
that one example sort of hit all the marks there.
You're doing things that that antivirus and access control don't.
Can I ask, you know, just a clarifying question, when
when your stuff is deployed, do you tend to see
it deployed in addition to antivirus or you know, whitelisting
access control, or are you deployed sort of instead of that.
Speaker 1: We've seen it both ways. We designed it so it
would run in parallel, you know, with some of these
ab solutions that are out there, because again we're trying
to go in there and say we're not disrupting your
existing infrastructure. If you have a reason to keep running that, great,
we'll just come in and run alongside. We've never gone
in and they've kept an application. Whitelisting solution there they've
typically just moved to us. So in many cases they
will be running things like Windows Defender and we're running
in addition to that.
Speaker 3: You know, your stuff is installed in the kernel. In
my experience, there's longstanding reluctance to deploy any kind of
security technology on existing hosts, in existing OT network, you know,
existing systems.
Speaker 1: You know, the.
Speaker 3: Vendors sometimes push back and say no, no, no, you've
installed somebody else's software on my system. I don't support
this anymore. You're on your own. There's vendor support agreements,
there's legal agreements that it gets complicated. Can you talk
about that? How you know, how has this technology been
received in an environment that just doesn't want to change anything.
Speaker 1: Well, that's an excellent point that you're making, and it
is one of the inhibitors you know, we see, but
it is changing the The industrial automation vendors typically which
you speak are recognizing that there's problems here, and you know,
they can either be part of the solution or they
can be held liable as part of the problem. And
you're seeing a movement there. You know, for instance, we
have just gone public with a relationship with Rockwell and
they are bringing US now in as part of their
solution to solve these types of problems where appropriate. And
we're having these discussions with these other other vendors. In
some cases they are very very regimented, and others are
much more open to provide more modern approaches, if you will,
to stopping these threats, because they are happening, and they're
starting to happen in increasing fashion. So they can't just
tell the customer my leug agreement with you says you
can't run protection to that I haven't approved on your system,
and then find out that their system was compromised because
their applications had vulnerabilities in them that were exploited. So
you can see the challenge is there, right, So these
vendors are now starting to move and it is something
that really has just really happened over the last couple
of years.
Speaker 3: You know, you were talking about bad stuff, shims that
download other bad stuff. You know, the shim looks benign,
but it winds up downloading code and xing executing Code's
that's not so benign, you know, just thinking about it,
this is what happens in browsers. Browsers download JavaScript routinely,
they execute the code routinely. You know, if if your
AI forbids downloaded code, does it break the browser? How
do you work with browsers?
Speaker 1: Yeah, so that's that's an interesting point. So you know,
when we get into OT environments, you don't typically see
these types of behaviors happening. So they're not downloading apps
through browsers and dynamically executing code, so that's not your
typical behavior that you're going to see. We do have
certain countermeasures though, to all all of us to deal
with things like malicious javascripts running and picking off those
those types of techniques because that may be running independent
of your typical browser download. So we do stop that.
Speaker 3: There's a lot of applications in an IT environment, and
to a lot of people surprised, there's a lot of
applications in an industrial control system environment as well. You
might imagine that. You know, there's fewer control systems in
the world than IT networks, but there's still a huge
diversity of applications of software of even hardware out there.
Is it I mean, is it possible for you guys
to learn all of those applications and keep track of
them as vendors release new security updates? Do you do
this sort of centrally, how do you manage that diversity?
Speaker 1: Yeah, there are multiple ways to do this, and we
do try to work with the larger industrial automation vendors
in advance to get as many of these as we can.
But we realized early on that we couldn't depend on that,
so we built our application so that once our agent
became active on the device, it would quickly inventory everything
had found running on the host, and they would slowly
also look at everything that was over on the disc. Again,
the word slowly is there because we're trying to make
sure we stay within operational parameters, so we don't slow
down the device at all. So when we do that,
we can inventory everything and in some cases we'll say
we find one hundred applications, you know, we'll build that
list on the device, and each of our individual agents
and all devices will build a list. Now, to make
it easy on the operator, we can come up in
a mode we call prevent mode where we say, okay,
we're going to assume that all the application is we
just built are good, right, Chances are that probably the situation,
and then we have these additional countermeasures that are going
to watch and see if anything else happens which is
not good, you know, one of these tech techniques, and
then those will then trigger us to stop those techniques
and zero it in on the application so that that
can be explored. So this allows the vendors to deploy
our agents out on the devices and just say, okay,
you come up and prevent you allow everything that's running
to run. You turn on your countermeasures to look for
bad techniques. And then what they do is they communicate
to a centralized application that's running inside this customer site
typically and it could be right down on these manufacturing lands.
In some cases we've got running an air gapped environment
factory flowers as we speak, so they don't have normal
connectivity outside world. But the one hundred devices in a
manufacturing floor communicate everything that they've learned on their devices,
and it builds the central manifest and then now you've
got depopulated stuff you like, but now you've actually got
an inventory of every single application on which devices or
which lines, because it'll actually give you that capability to
name the lines. And we found that of extreme use
to a lot of these manufacturers because a lot of
things they don't know. All the applications or the application
variants that are running there on each of their OS platforms,
So it gives them visibility of that, and then they
can say, okay, this is a great I can now
have an approved level of manifest and if something goes wrong,
I can look through that and take out those those
bad applications. Or I can say, you know what, I'm
looking through this list and I really don't like that
notepad is running on these applications because that's something that
could be used by an attacker. So I'm going to
say I want to block that. So the centralized control
system we call our trust center can then an update
out and say block the use of Noepad. So it's
a way for you to control the policies in which
you allow certain applications to run. So this was well
received because I said, we have this large global manufacturer
and they were like, this is good because the people
on the site trying to run these things don't have
time to go figure everything out. They need something that's simple.
But then when we're looking and doing some periodic reviews,
we can sit there and say, okay, we can examine
what we have here. We can decide what we don't
like running, and we also can get an indication of
all the different variants of these various applications so that
we can do better planning going forward. In the meantime,
they're fully protected from these types of attacks that they're
most concerned with, everything from the zero day ransomware and
malware all the way up to these very sophisticated nations
date backed attacks. They're typically coming in through their supply chain, by.
Speaker 3: The way, So that struck me as interesting, Nate. You
know a lot of people have been on the show
talking about asset inventory. You can only protect what you
know you have, but asset inventory in most implementations, in
my understanding, tends to focus on what kind of devices
are there. There's PLCs, there's RTUs, there's protective relays. You know,
there's Windows machines, there's Linux machines. What version of what
OS are they running? What's what version of the OS
are they running, what patch level are they running? What
software has been installed? Has this software been patched? What
these folks are doing is sort of coming at it.
I assume all of that, and and they're going through
and making a long list of all of the executables
that are installed on the machine, which is sort of
the next level of detail. I mean, you know, he
mentioned the example of Notepad. Notepad. You know, there's no
there's no there's nothing in the list of installed software
that says notepads installed. It installs when you install all
the OS. It's not a separate install So having that
sort of more detailed asset inventory is you know, to me,
is interesting. It strikes me as potentially useful in terms of,
you know, additional hardening that you can apply to these machines.
Speaker 2: You know, I'm not sure that this was the point
of what you just said there, but you mentioned notepad,
and he mentioned Notepad. I'm wondering, why is Notepad coming
into any of this. That's just the application that I
never use on my computer.
Speaker 3: Yeah, it's an application I never use either. It's it's
an application that lets you edit text files. And you know,
if you ever read me file, nobody wants to edit it.
They want to read it. You can read it in
lots of things, the browser will let you read it.
Notepad lets you create text files as well, and apparently attackers,
you know, it's it's one of the tools that attackers
tend to use more than regular users, because the attackers
always need to put some script file you know, down
so you can execute it, or you know, put a
store and license key into a text file so it
can be imported, and so on. So I guess this
is again one of the tools that owners and operators
might look at and say, I never use that, We
don't need that. The only people that are going to
use that are the bad guys. Take it off the machine.
You know, we had an episode I think recently talking
about living off the land when touched on it briefly,
the using tools that are part of the operating system
to launch attacks. This sounds like one of those tools.
Never really occurred to me. But yeah, you know, when
you say it, I never use notepad either. So if
it's only the bad guys doing it, you know, that's
a candidate to take off the machine. You were talking
about the ability to run in sort of an advisory
mode versus an enforcing mode. Is this relevant to let's
call it upset conditions? I mean, how often do you
start the plant from scratch, maybe once a year, and
everything behaves a little bit differently during start up? How
often do you do an emergency shutdown? Hopefully no more
than every few years. And in an emergency shutdown, everything
is different, everything is changing. Do do you make sort
of provision make exceptions in those cases?
Speaker 1: Prevent mode is our normal mode. But when we go
into these let's call them updates, usually it's maintenance windows
emergency shutdowns, we just asked that the operators turn us
into detect mode. That way the product keeps running. But
as they make these changes, we're not going to try
to get in the way of things happening, especially like
you said, when there's an emergency going on, or you're
just getting a batch of updates coming in, you know,
from patches to the new revisions of applications coming in
across the board. Then once things settle down, you can
look at what we detected as showing alerts you or not.
It depends on what they want to do. It depends
on the situation, of course. And then if you see
everything is fine from what we reported, you just accept
all those changes. You can say these are all okay
to run as is, basically started fresh and move back
into prevent mode. So all the changes have been accepted
and we run from there. So that's the way that
we deal with that and we find it works out
pretty well because it's just simple toggling of a switch
and then toggling it back on once everything is stabilized,
and you still have an ability to track everything that
happened during that mode where you're doing all those changes.
So we have all these wonderful operational logs to tell
you about exactly what happened. So in a lot of
these environments that are definitely under a lot of scrutiny
compliance reasons, now you now have a complete history of
what's happened.
Speaker 3: Well, this has been great, Gary, thank you for joining us.
Before I let you go, so can I ask you
to sum up for us what should know? What should
we be thinking about when we're thinking about this space.
Speaker 1: Well, as I start off, you've got to make sure
you've got a solution that has an ability to stop
all these different variations of attack. It doesn't help if
you're only covering twenty percent of the attacks out there.
You've got to cover the full level of attacks in
order to have a solution with efficacy. The other point
I think we want to briefly touch on is that
you can have the best solutions out there, but if
they're not easy to actually implement and deploy and update,
then the solution will not be successful. That's going to
be that simple that operators with minimal training can figure
out how to deploy it. They can come up and
then deal with this as they go through their normal operations,
as they run it or are going through a period.
Whenever I've got a maintenance window running and they're making
updates to all their applications in their environment, I would
say there's a call to action going on right here,
because for so long the industry has tried to stick
with the old ways and the old ways in the
OT world where or try to use passive defenses as
much as possible air gap which means there's no Internet
connectivity as much as possible, and yet the attacks keep coming.
The problem is there's the human element. The industrial automation vendors.
I don't want to pick on them, but they have
to update their applications at some point, so either they're
bringing in people, or they've got third parties that are
coming in, or the customer has third parties coming in.
And that's when we have people walking past the network
and then plugging into these devices, often with USB sticks
or I mean it's laptops and the updates happen, and
so do the problems. So you can't be myopic and
think we can get away with approaches that worked in
the last decade when there's actually ways that defeat them
every day in our environments. So I would say the
takeaway is you're going to look at a solution. You've
got to find one that will work well, drastically reduce
your risk, that's easy to deploy, and then can deal
with these situations where traditional defenses just don't cover the problem.
Speaker 2: So Andrew to close out here, Gary's talking a lot
about choosing the right solutions, which solution is a tricky word, right,
Are we really solving something here or are we iterating
on a long history of what we've been doing prior.
Speaker 3: That's a good question. I would use the word innovating
rather than iterating. The bad guys keep getting better at
what they're doing. They keep inventing new and different and
subtler ways of a acting us, and so our defenses
need to become more capable. You know, as time goes
by as well as the threat environment changes. And here's
an innovation. Here's a way to address a kind of
attack that is becoming more widely used by the sophisticated,
the high end of the attack spectrum. You know, putting
something benign looking into a software update, putting something benign
looking on a machine, and then loading the nasty in
memory into that benign looking thing. This is you know,
this is the world we live in. This is starting
to happen reasonably regularly. We need technology that's going to
address this threat, you know, the bad guys innovates we
need to as well.
Speaker 2: Well. Thank you to Gary Southwell for speaking with you, Andrew.
And Andrew is always thank you for speaking with me.
Speaker 3: It's always a pleasure. Thank you, Nan.
Speaker 2: This has been the Industrial Security podcast from Waterfall. Thank
you to everybody of their listening.
Speaker 1: Mm hmm