Speaker 1: If you didn't listen to a single thing I said,
you can listen to these three things, Collaborate, plan, and practice.
Speaker 2: Welcome listeners to the Industrial Security Podcast. My name is
Nate Nelson. I'm here with Andrew Ginter, the vice president
of Industrial Security at Waterfall Security Solutions, who's going to
introduce the subject and guest of our show today. Andrew,
how are you?
Speaker 3: I'm very well, Thank you, Nate. Our guest today is
Chris Cistrunk. He is the technical lead of the Mandient
ICs or ot security consulting team, whatever you wish to
call it. Google purchased Mandian in twenty twenty two, but
they're still keeping the Mandian name, so he still identifies
as technical lead of Industrial Security Consulting at Mandian. And
our topic. You know, they as part of their consulting practice,
they do a lot of incident response. He's going to
talk about lessons learned from incident response in the industrial
security space.
Speaker 2: Then, without further ado, here's your interview.
Speaker 3: Hello Chris, and welcome to the podcast. Before we get started,
can I ask you to say a few words for
our listeners about your background and about the good work
that you're doing at Mandian.
Speaker 1: Okay, thanks Andrew. I'm again at Mandient on the ICSOT
consulting team, been doing that for over eleven years now,
focus on ICSOT security consulting around the world with every
type of critical infrastructure, doing incident response, strategic and technical assessments,
and doing training as well. Before that, I was electrical engineer.
I still am, but for a large electric utility energy
was there over eleven years as a senior electrical engineer
Transmission dish should skate a substation automation and distribution design.
So that's a little bit about me and again just
working for Mandian, part of Google Cloud.
Speaker 3: And our topic is incidents. It's lessons from incidents. But
let's talk the big picture of incidents. I mean, Waterfall
puts out a threat report annually. I'm one of the contributors.
You know, we go through thousands of public incident reports
looking for the needles in the haystack, the incidents where
there were physical consequences, where there were shutdowns, where you know,
sometimes equipment was damaged, and we rely on the public
record on public disclosure. And so I've always believed that
we were under reporting because I'm guessing again, I don't
have that many confidential disclosures that people tell me about,
but I'm guessing that there's a lot more out there
that never makes it into the public eye. You folks
work behind the scene, you know, without reaching any non
disclosure agreements or anything. Can you talk about the big picture.
Do you see incidents, especially incidents you know, in the
industrial space with physical consequences, incidents that triggered shutdowns, incidents
that are not public reported. How many are there, what
do they look like? Can you talk anything about sort
of what I would not see by looking at the
public record.
Speaker 1: Sure, thanks for the question. You know, I think we're
talking about cybersecurity incidents here, and there's many incidents that
happen every day, right but life goes on. Squirrels happen
right in the grid. But for cybersecurity incidents, I do
believe we're seeing an increase. I can't go into how many.
We actually have a report that m Trends Many has
put out every year. It's going to come out later
this month and for RSA, and this is a yearly report.
We report on the different themes, the different targeted victims,
the different threat groups, the TTPs. But for cyber attacks
that impact say production or cause of a company to
shut their operations down. I don't have any hard fast
numbers to talk about, but we have seen an increase
and you can look in not just our report, but
also the reports of others IBM X Force, Verizon, d
b I, r Drago's others. There are increasing reports of these,
and a lot of it has to do with things
like ransomware UH and ransomware either directly impacting the control
system environment, which we have responded to in a manufacturer
and a few others, but we have seen in the public,
you know, news where a company might have to shut
down operations due to indirect impact. Maybe their enterprise resource
planning software or manufacturing execution software was impacted, which is
an indirect impact to the OT critical data flowing that
was halted, which means I can't produce my orders anymore,
or track shipping or logistics things like that. So we're
seeing a lot of those. There's others in the electric
sector that they kind of have to be reported to
OE four seventeen reports. If there's a material impact, obviously
they'll be filed in the U or they're supposed to
be filed in the eight K or ten K with SEC,
and so I think if you take all of those
source and look together and see, we see there's an
increase of operational impact. Mhm. But it's I think the
engineers are doing a good job of UH and the
folks that run these systems are minimizing the impact in
these situations, and especially for electric and water and other
critical infrastructure. Manufacturing is critical. But I'd say it is
probably the highest targeted outside of you know, healthcare and
other other areas.
Speaker 3: So work with me on on the numbers, just you know,
for one more minute. I'm on the record in the
in the the Waterfall Threat Report speculating as to what's
going on with public disclosures. It's my opinion, but you know,
I have limited information to back it up. It's my
opinion that the new disc closure rules in the SEC
and other jurisdictions around the world are in fact reducing
the amount of information in the public domain rather than
increasing it. And the reason I suggest this is because
it seems to me that with the new rules, every
incident response team on the planet roughly you know, I overgeneralize,
has a new step two in their playbook. Step two
is called the lawyers, and what the lawyers say, they
say say nothing, because if you disclose improperly, if you
fail to disclose widely enough, you can be accused of
facilitating insider trading. If you disclose too much information, you
might get sued. I mean people have been sued for
disclosing incorrect information about security into the public. People buy
and trade shares, and then they know, they find out
the information was incorrect and they get sued. And so
to me, the mandate for the lawyers is say the
minimum the law requires, because if you say too much,
you risk make a mistake and getting sued, and you
don't want to get sued. And if you you know,
if you say too little, you're going to get sued.
You know, the lawyers minimize. And if you have, you know,
a material incident, you must report it. If it turns
out the incident is not material to the finances of
the company, you don't have to report it. And again,
to minimize the risk of getting sued by reporting incorrect information,
you report nothing. So my sense is that we're seeing
fewer reports because of these mandatory rules, not more of
than what do you see? You know, you see this
from the other side does this make any sense? Do
you have a different perspective.
Speaker 1: I can say that, you know, as an incident responder
working with you know, victims in critical infrastructure, but also outside.
I think this is a broader question you bring is.
I can definitely confirm that we work with external counsel
that a victim may have hired to bring in to
handle a lot of these reporting or not reporting requirements.
I can't say or confirm that the lawyers themselves external
counsel under reports. I can't say that I don't know.
I'm not a lawyer, nor do I play one on Facebook.
So I will just stick to say yes, we have
worked with external counsel, and usually we do not say
anything in public for us as the incident responder, unless
the victim company or our client asked us to, because
sometimes sharing information is a helpful thing, especially if it's
a big breach. Sharing that lessons learned about what has
happened to them with others, just like we did back
in the day when we had the solar winds breach.
So you know, there's two ways of thinking of that,
and maybe you can pull on that throw with some
other experts, but not me. I don't know about the
External Council part.
Speaker 2: Andrew, you'd referenced Waterfalls Annual Threat Report, a report which
I've covered in the past for dark reading. I'm not
sure i've seen this year's iteration, So maybe you could
tell listeners just a bit about what the report covers
and what the numbers are showing lately.
Speaker 3: Sure, the report uses a public data set. The entire
data sets in the appendix. You can click through to
it if you wish we cover we count in our statistics.
We count deliberate attacks, cyber attacks with physical consequences, not
stole some money, physical consequences in heavy industry and critical infrastructure,
the industries we serve in the public record. Okay, no
confidential disclosures. The numbers last year were seventy two attacks,
I believe with I don't know some you know, a
one hundred, one hundred and fifty something like that. I
forget the numbers sites affected. Many of the attacks affected
multiple sites. This year, you know, we're up from seventy two.
We have seventy six attacks affecting a little over a
thousand sites, So there was more sites affected, but the
number of attacks did not increase sharply. And this is why, Again,
I speculate, why have we sort of seen a plateau.
We went up from you know zero essentially give or
take in you know, let's say twenty nineteen to you know,
seventy two, and then you know in twenty twenty four
to seventy six. Why do we seem to have a
bit of a plateau? And I'm speculating it has to
do with the SEC rules. People are now legally obliged
not just in you know, the United States, the Security
and Exchange Commission, They're legally not just in that jurisdiction,
but in other jurisdictions around the world. There's similar rules
around the world. If you have an incident that is
material that any reasonable investor would use as grounds to
buy or share or sell or value shares, you must
disclose it. But I have the sense that we are
seeing fewer disclosures because by law you're required to disclose
material incidents. And again because I speculate that because the
lawyers are involved, we are seeing fewer disclosures. You know,
they disclose the material incidents and they squash everything else.
Is the sense I have. But you know, you asked
about the numbers seventy six last year. Nation state attacks
are up from you know, there were two the year before,
there were six last year? You know, is this a trend?
It's still small numbers, who knows? And industrial control system
capable malware malware that understands industrial protocols and that is
apparently designed to manipulate industrial systems is up sharply with
there were three new different kinds of malware disclosed last
year or you know, found in the wild last year
that had that capability versus you know, seven in the
preceding fifteen years. Again, small numbers. Is it a blip?
Is it a trend? Is AI helping these people write stuff?
We don't know? So these are all sort of you
look at the numbers and you scratch your head and
you go, I wonder this. Doesn't you know what's going
on here? So that's that's the threat report in a nutshell.
There's other statistics in there, but those are sort of
the headlines that leads us into the topic of the show,
which is lessons learned from incidents. You folks do incident
response all the time. You know, can you talk to me?
What are you seeing out there? You know, is there
an incident or three that sticks in your mind. As
you know, Andrew, the most important thing I have to
tell you is and you know or the most recent
Where would you like to start?
Speaker 1: Okay? Sure? We have been doing OT incint response since
I've been here, and I can give you a few examples.
Last year in twenty twenty four, we responded to a
North American manufacturing company that had their OT network for
looking at a Purdue model. It's the third layer or
level three of the network was directly impacted by the
Akira ransomware game. And what had happened was an unknown
Internet connection was made by this third party who was
running the site. They had put in their own Cisco
ASA firewall, and it just so happened to be that
there was two critical vulnerabilities in that firewall at the time,
and the Cure ransomware game was targeting targeting those exposed firewalls.
So don't necessarily think this was a targeted manufacturing OT attack.
It's just ransomware gangs doing what they do, trying to
make money, and so they were able to log in
and get in through these vulnerabilities and deployed the ransomware
on directly on the OT network, which was flat and
every system but about five or six or seven were
completely encrypted, including their o T DCS vendors, And there
was multiple not pick picking on any one in particular,
but ge A, B. B Rockwell several others that were there,
and the backups server was impacted in the back up
of the backup server was impacted. They were all on
the same flat network. So this was a really tough
situation since the company manufacturing did not have any backups
that were offline. The OT vendors, like I mentioned for
I had to come on site to completely rebuild the
Windows systems, the Windows servers, the engineering workstations, the hmis,
all the things that were Windows and or Linux that
had to completely rebuild. Client didn't pay the ransom in
other words, and so the lessons learned here. Work with
your OT vendors and OEMs and even your contractors to
make sure that your Windows systems and Linux systems have antivirus,
make sure that you have OT backups that are segmented
from the main OT network, and keep offline backups and
test them on a at least a year basis. Backups
will get you out of a bad day, even if
it's an honest mistake at five o'clock on Friday. So
this is a basic win here having good backup strategy.
And then in the last case here we recommended they
eliminate this external firewall and leverage the existing itot DMZ
firewall that came in from the main owner of that site.
And so they had a back door essentially that this
third party contractor had installed in a new Internet connection.
So give away from the shadow it go back to
your normal itot dmz with jump box two, factory authentication
and all those things. But if you do the basics
and do them well, keep good segmentation, have backups, and
patch your firewalls on a regular basis, I think that
will go a long way, especially in this case.
Speaker 2: You know, I feel like I've heard of variant of
that advice that Chris just gave a million times, and
I don't work in industrial security, so you folks must
hear it all the time, or it must just be
such basic knowledge that you don't even think about it.
So are there really industrial sites out there that still
need to hear that you shouldn't be making an Internet
connection from your critical systems?
Speaker 3: Short answers, Yes, you know, people who do security assessments,
you know, not just incident response that we're talking here,
but security assessments come back and say they regularly find
connections out to the IT network and occasionally straight out
to the Internet. The connections to the IT network tend
to have been deployed by the the you know, the
engineering team or the IT team to make their lives easier.
You know, people with gray hair enough gray hair like me,
they talk about, you know, how the systems used to
be air gapped. This was a very long time ago.
We're talking thirty forty years. The systems used to be
air gaped, and you know, people with gray hair like
me might assume that's still the case. It's not. You know,
everybody who does audits reports these connections. The really disturbing stuff,
you know, yes, it's it's disturbing that there are connections
to the IT network that are poorly secured, that, you know,
But the really disturbing stuff is the vendors going in
and if you do an audit on a site time
and time again, I hear people saying, yeah, they discovered
three different Internet connections. The vendors stuck in there, and
you're going, well, wouldn't you notice if there was a
new Internet connection. I mean, no internet service provider gives
you a connection for free. You got to run wires,
you got to You got to pay for this thing.
Every month. It's showing up on your bill.
Speaker 1: No, it's not.
Speaker 3: You know, there's a lot of wires being run while
stuff is being deployed. You don't notice a new wire.
And the vendors pay month after month for the Internet connection.
Doesn't even show up on the bill of the owner
and operator. Because the vendors are providing a remote management
or vote maintenance service and they want to minimize their costs,
they want to maximize their convenience in terms of getting
into the site, so they deploy rogue DSL routers, they
deploy rogue firewalls to the site's Internet connection. They deploy
they might deploy rogue cellular access points where you know,
there's not even wires to run. It's just a box
sitting there that has a label on it saying, you
know important, do not remove. And of course God makes
it invisible to everybody who's looking at it says, oh what,
don't touch that one. Yes, it's very common. The advice
I try to give people is when you do like
a risk assessment or a walk through or an audit
of your site look for these rogue connections. Unfortunately, you're
probably going to find one or two of these contractual
penalties with the vendor help, but they're no guarantee. You
said that the victim decided not to pay the ransom.
You know, do you see victims ever paying the ransom
to recover an OT network, to recover the HMI, to
recover worse than that, the PLCs and the safety systems.
You know, why would does anyone trust a criminal to
take the tool the criminal provides and light it on
their safety system and you know, restore it because they
trust the criminal. Does anyone trust the criminal that far
as that does that happen?
Speaker 1: We have seen traditional I T systems where they pay
the ransom and get access back to these systems and
some that are OT adjacent such as colonial pipeline and hospitals. Right,
we do know that those systems and both of those
incidents or well those examples the are colonnal pipeline are
name a hospital breach. In some cases there were OTS
type data OT type critical information that was impacted and
so they of course paid and due to the fact
that there's if someone didn't trust these ransomware gangs to
do what they say they were going to do, they
would those ransomware games would be out of business. So
if you pay them, they don't, they decryptor doesn't work,
then it's no good in their ransomware gain job is
over with at least for this thing. But for OT
in this instance I talked about, they did not pay.
I don't have enough data to know if OT direct
on the control systems themselves, the Windows, HMI s, engineering workstations,
DCS servers, SKATAS servers, if they've paid in those instances.
But I'd say it's it's plausible, And it really comes
down to the business decision of the plant owner, the
CEO of the cut based on what the engineers at
the lower level, Hey, can we get do we have backups?
Can we get the vendors to come in? And so
I really don't have enough information to say about do
OT asset owners like a plant, like a site, like
a you know that directly operates h O T, if
they trust these ransomware games or not. It may just
roll up higher than them about that. Uh. Also, there's
usually an advice from a ransomware negotiator that's a third
party that specializes in negotiating with ransom so they may
advise to pay or not to pay, or to get
a reduced pain as well. So it's very, very complicated.
I know I didn't answer your question directly, but in
the instances we've seen, we have seen them not pay
and we have seen them pay and what's OT or not.
Speaker 3: So so coming back to our theme here, you know,
lessons from incidents. You know, the lesson from this incident is,
you know, get rid of that firewall, use the existing infrastructure,
and you know, look at your backups. I mean, if
the backups are encrypted, it's it's all over. That makes
perfect sense. What else have you got? What else you
know have you been running into lately? That that's that's
interesting and noteworthy.
Speaker 1: Yeah, I mean it's basically boths down to either ransomware
or commodity malware. So I've got I've got another example
about ransomware. Electric utility was impacted by ransomware on the
IT side, but they had a good incident response plan
and they severed the IT and OT connections there and
even down to the power plant type networks, and so
that was really amazing, and so that's a good story,
and we were able to actually verify the IT team
were able to verify that the threat actor, the ransomware
game Quantum, were scanning the OT DMZ, but they didn't
get a chance to get be let through. We did
do a full assessment of their DMZ and looked at
the domain control of the firewalls and even the domain
controller and the firewalls and others down inside the OT networks,
and we found that that they were actually pretty lucky
because they had some weakness in some of the firewalls.
So eventually do if they had enough time, if the
ransomware actor had persisted long enough, they could have gotten
through that firewall, made it to the DMZ And the
active directory had some weaknesses as well, and they could
have gotten domain access domain admin and pivoted to the
OT network. But the great thing is highlight again that
they had a good instant response plan. They were able
to segment quickly and then they were able to have
their OT vendor in this case it was Emerson Ovation
were able to go on site and they were able
to not only take the IOCs that we had from
the ransomware, but they were able to sweep because it
was their contract to do so, to look in the
POC logs, the OT workstations, endpoint protection and all that stuff.
So we all worked in concert together in this incident.
And then actually they hardened the firewalls, hardened the domain controllers,
hardened the workstation configurations before doing anything else. They did
all of that. When the ransomware was eradicated and hardened,
then they said, okay, now we'll reconnect everything back the
way it was. So that was a really great lesson
learned with another ransomware, and it wasn't a direct impact
to OT, but this is a great opportunity to leverage
that incident response plan that they had.
Speaker 3: So nate the concept of separating it from OT networks
in an emergency. This is the concept that I see increasingly.
I mean, I think we've reported on the show here
a few times that this is what the TSAD demands
of pipeline operators, petrochemical pipeline operators ever since colonial the
ability to separate the networks in an emergency so that
you can keep the pipeline running wild it's being cleaned up.
I haven't I'm told I haven't actually read the translated
the Danish law, but apparently in Denmark there's a recent
law in the last twelve months saying exactly the same thing.
You know, the TSA applies to pipelines and rails, in
Denmarket applies to critical infrastructure, and it says, in an emergency,
you have to be able to separate. They call it
islanding the industrial control network, and as Chris points out,
it can be effective, but it relies on really rapid
intrusion detection and rapid response because as Chris said, you know,
the bad guys had been testing the OT firewall. If
they had had just a little bit longer, they could
have gone through. So, you know it, even though it
it's you know, imperfect, it is a measure that I'm
seeing increasingly required of critical infrastructure operators and you know,
recommended to non critical operators as a as a measure
that helps, especially on the incident response side. Have you
got another example for us? I mean three is the
magic number you've given us, you know, sort of two
sets of insights. What what else have you got for us?
And in terms of lessons.
Speaker 1: Learned, yeah, there's lessons learned I can just name a
few other lessons learned from it, just about any attack, right,
making sure that you have these at least Windows systems
with antivise. In a lot of cases the OT network
didn't have anti just basic antivirse, not necessarily an agent
or EDR solutions. If you have those, great. If you
don't have any antivirse, that's you need to get at
least corner version of Windows or operating system and with antivirus,
having good backups, having good that good vendor support. Now,
this last incident we responded to was using a living
off the land attack, So we responded to electric utility
in Ukraine in twenty twenty two and it was a
distribution utility that the attacker came in through the IT
network deployed their typical wiper malware. This was the group
APT forty four or Sandworm team, which has been targeting
critical infrastructure around the world for quite a while, and
they were able to pivot to the Skater system and
used the feature of the Skater system to trip breakers
using a tool that was built in the skate of
system itself, So just giving it a list of breakers
to trip and calling that executable in the system to
trip those breakers on. Behalf of the attackers is the
too long did read of that incident? And so the
lesson learned here is targeted attacks. They're going to not
use malware. They're going to use the features or the
inherent vulnerabilities in an OT network stealing valid credentials like
an operator workstation or an engineering administrator account. And if
you can even spearfish an engineer or administrator network AM
on the IT network and you don't have good segmentation
of roles from IT to O T, then that's that
attacker is going to use every one of those tools
to evade detection, to bypass your normal detections because they're
they're coming in as a valid user. So the lessons
learned there is to limit the amount of administrative access.
And this is you know, role based authentication, right and
does the person that got promoted and now is in
a different department, does he still need ADMIN rights? You know,
does this person having enough control for just their area
only or their responsibilities too wide? And now we say, okay,
we need to reduce the amount of admin. Do we
require two factor authentication or even hardware two factor authentication
to really reduce the attacker down to an insider threat,
because remotely that's very hard to do to to to
to bypass hardware token based two factor authentication. And so
there's some there's some living off the land guides out there.
The US government DEWE has put out a threat hunting
guy for living off the land attacks after the volt
typhoon announcements last year. But I would also go as
step above and beyond that, learning good ways to detect
anomalous logins, even from your own folks. If it's out
of a normal time, out of a normal location, you're
really gonna have to have some tuning on some of
these detections. And the only way to really test those
is with a red team that's trying to be quiet
and not trigger your detections. And that is some of
the more advanced asset owners and end users they're using
leveraging red teams, hiring red teams like what we do
at Mandiant to come in and see if we can
do living off the land attacks to bypass their detections.
Speaker 2: Since Chris mentioned it but moved on before we could
actually define it, let me just for listeners, living off
the land is the process by which an attacker, rather
than using their own malicious tooling, would make use of
legitimate software or functionality of the system they're attacking to
perform malicious actions on it. It's been a growing trend
in recent years, I believe, because it's so effective in
that it is so difficult to detect. You know, you
could spot malware with certain kinds of tools, but can
you spot somebody doing things with legitimate aspects of Windows
or whatever you might be using. It sounds though like
Chris is talking about detecting living off the land tactics,
which seems difficult to Andrew.
Speaker 3: That's right, I mean, I you know, have been have
been following Living off the land to a degree. You
know the what's the right where the short The short
answer is you run an anti virus scan on a
machine that's been compromised by a living off the land
attack and it comes up squeaky clean, there's nothing nasty
on the machine. And what I heard Chris say is
that this is because the bad guys are using normal mechanisms,
especially remote access, to log into these systems as if
they were normal users and use the tools on the
machine to attack the network or you know, to wait
for a period of time until it's opportune, and then
attack the network. And you know what I heard him
say is that because it's a lot of a lot
of this is remote access, he says, you can detect
this by focusing hard on your remote access system. A.
You can prevent it by throwing in some hardware based
two factor. You know, that will solve a lot of
the problem, not necessarily all of it. There's always vulnerabilities
and zero days, but two factor helps enormously. It's way
better than not having two factor. But that's preventive. On
the detective side, he said, pay attention to your remote
access If normal users are logging in at strange times,
that should raise a red flag. If normal users are
logging in from strange places, the IP address coming in
is from China, Well is Fred in China this week?
No he's not. So you know what I heard was
one way to to you know, help detect living off
the land techniques is to pay close attention in your
intrusion detection system to the intelligence that you're getting about
remote users logging in. So one more question, you know,
we talked to a lot of folks on the on
the podcast, A lot of them are our vendors with
technology that we talk about, and you know, sort of
a consistent theme for most of these vendors most of
these technologies is operational benefits. Yes, the technology whatever it is,
is helping with cybersecurity, but often this stuff helps with
just general operations in sometimes surprising ways. We've been talking
about incidents and lessons, and you know, a lot of
what you do is incident response. Are there operational benefits
that you run into that people say, you know, I
did what you told me and everything is working smooth
more smoothly than not just on the security side. Can
you talk you do you have anything like that for us?
Speaker 1: Oh? Absolutely. And one of the things that I always
teut is looking at your network, looking at the packet
captures in the network can aid in not just cybersecurity benefits,
but these operational benefits. You can see things like switch
failures happening, TCP retransmissions happening. All this traffic may be
went like your Windows hmis maybe trying to reach out
to Windows Update, but it's blocked by the it OT
firewall or anything else. It may not have a connection
at all as trying to reach out all this unnecessary
traffic or indications of proper configurations, misconfigurations and things like that.
So just looking at your network with some of these
tools that are out there, free tools, pay tools, ICs
specific tools, or IT specific tools. It doesn't matter if
you look at If you take any one of those,
say just even wireshark and look in your OT network,
you can get an idea on what traffic doesn't need
to be there that you can eliminate it, make your
improvements to the system, and now I have better visibility
if there is an incident. I can easier to detect
if there's a cyber incident or if something's operationally wrong
like a switch failure or something, and so there's a
really great benefit there. Also helps improve reliability. We've done
an assessment at a company that had a conveyor belt
that they were having problems with. If the conveyor belt
wasn't timed exactly right, if they had too much latency
on the network, the conveyor belt would stop and all
the things on the conveyor belt would just go everywhere,
and it was a disaster. So we just looked in
the network, Oh, you got all these TCP re transmissions
is and you look at the map in the in
the software and say, oh, it's coming from these two
IP addresses. Oh, we know what those equipment is. And
we had the network person coming, Oh, I've been trying
to figure this out for weeks and just looking looking,
just using a duel like that. Uh, they were able
to find and fix the problem and they fixed their
latency issue because of that. So going back to you know,
incident response, having these having an incant response plan. A
lot of ot already has this because of disasters, fire, floods, storms,
it spills, air releases, safety issues, and that's all part
of their normal disaster recovery or incident response plans. If
you already have one of those, you've already done ninety
percent of the work to have a cyber incident response plan.
You just now have a cyber incident response added to that.
So that's the whole premise behind things like ICs or
ICs incident Command system for industrial control systems and having
that say, chief person in charge of cybersecurity for a
site for a paper mill, for a power plant, for
manufacturing facility, even though that's not your day to day
job all the time. If you say the lead that
and you say you have multiple plants, have multiple leads
for those plants. That still the every decision will go
through the plant manager, the general manager of the plant
or site. But at least you have someone that is
in charge of cybersecurity, just like you have a designated
firewatch person or anything else. So if you take safety
culture that we've known about for over one hundred years
and mold your cybersecurity culture to fit with that, things
will make a lot more sense. We've already we've already
invented this. We're not reinventing the wheel here now. We're
just including another paradigm of cyber security, network security, and
endpoint security into these things that we have been doing.
There's a fire, okay, let's put out incident response. So
and if you have a plan, that's great. If you
don't have a plan and you run around, that's not good.
So if you have a plan, you can at least
prepare for it, and sometimes that's the win. Being prepared
is better than not being prepared.
Speaker 3: Well, this has been tremendous. Thank you Chris for joining us.
Before we let you go, can you sum up for
our listeners what you know? What are the key points
we should take away here.
Speaker 1: Sure if you didn't listen to a single thing I said,
and you can listen to these three things. Collaborate, plan
and practice. So collaborate. You get your IT teams, talking
to your OT teams, talking to your manufacturers, and identify
the right roles within each of those and make sure
you get together and talk about these things. Have some
donuts and coffee. So collaborating. Knowing who is in charge
of what is half the battle. Knowing who to call
when plan, having an incent response plan, or including OT
security in your incident response plan and or engineering procedures.
That's gonna help when an incident impacts OT directly or indirectly.
And then practice even can start with a simple question, Hey,
what would we do in an incident or even going
to having a tabletop exercise collecting logs from a PLC
of security logs? How long does that take? How many
devices do we have? If the general manager says how
long is this going to take to pull all the
logs from all of our systems, you won't be able
to say I don't know. You'll have to say no,
this will take two hours and forty five minutes. Because
we've tested it so collaborate plan and practice. If you
need help with OT security or IT security, we do that.
At Mandian. We offer incidant response retainer that covers IT
and OT. There's no separate retainer. If you have an
IT incident and don't need OT, not a problem. If
you have OT only incident, not a problem. If it's
IT cloud and OT all at the same time, we
can help you around the world twenty four to seven.
And lastly, if you want to learn more about this,
you can reach out to me Chris's drunk at Google
dot com, my email LinkedIn social media blue Sky, and
we check out some of our blogs on the Google
Cloud or Mandian security blog. We have great content out
there that is actual actionable not marketing fluff. It is
actual actionable reports. The next M Trends Report is coming
out next week r s a time prime end of April,
so that's a free report. It's a great report to
look at and gain some insights on what we've been
responding to over the last year. And with that, I
appreciate it. Collaborate plan practice, Andrew.
Speaker 2: That just about concludes your interview with Chris Sistroke. Do
you have any final words to add on to his
to take us out with today.
Speaker 3: I mean Chris, Chris summed up collaborate plan practice. You know,
what I heard, especially earlier in the interview, was, you know,
do the basics, guys. They some people call it basic
hygien is. Basically do on an OT network as much
as you can of what you would do on an
IT network. Put a little anti virus in on the
systems that tolerated, you know, get some backups, get some
offsite backups so that if the bad guys get in
they can't encrypt the off site backups. There's somewhere else,
you know, look for the vendors leaving behind internet connections,
get rid of them, you know. And you know, on
the in terms of living off the land, you know,
he gave some very concrete advice that I've never heard before, saying, look,
these people are coming in as users. Get two factor.
Two factor will do a lot to breaking up living
off the land attacks. And in your intrusion detection systems,
look hard at what your remote users are doing, and
if it seems at all unusual, that's the clue that
you're being a tact And you know, in terms of
his collaborate, plan and practice. I really liked his his
his uh the fire warden analogy, saying, look, you know,
if you have an industrial site that is flammable, okay,
your firewarden does not just sit on her hands until
the place bursts into flames. Okay. The fire warden is
someone who's active in terms of actively h you know,
looking at managing, raising the alarm when they see dangerous
practices in this flammable plant. It's not you know, it's
not just a reactive position. It's also a proactive position.
And we need that for cybersecurity because basically every site is,
you know, in a sense, a flammable cybersecurity situation. So
it's not just that they sit on their hands until
there's an incident and then they're in charge. They are
actively looking around, just like a fire warden wouldn't say,
you know, we shouldn't be doing this. You know, my
job is not just to put the fire out when
it occurs, or coordinate putting the fire out. My job
is to help prevent these things. And so I love
that analogy. That makes so much sense. Anyhow, That's what
I took from the episode.
Speaker 2: Sure well, thank you Chris for speaking with us and
Andrews always thank you for speaking of it.
Speaker 3: It's always a great pleasure.
Speaker 1: Thank you.
Speaker 2: This has been the Industrial Security Podcast from Waterfall. Thanks
to everyone out there listen.