The facilities, guys put the username and password on a sticky note stuck to
the bottom of the monitor. Now, some of them get super sophisticated about
this and they put it on the bottom of the keyboard. Welcome listeners to
the Industrial Security Podcast. My name is Nate Nelson. I'm here with Andrew
Ginter, the vice president of Industrial Security at Waterfall Security Solutions. He's going
to introduce the subject and guest of our show today. Andrew, how are
you. I'm very well, Thank you, Nate. Our guest today is
Kyle Peters. He is a senior consultant at Intelligent Buildings and he's going to
be talking about safety and security and how it all fits together with IIC six
two four four three in building automation. Then, without further ado, here's
you and Kyle. Hello Kyle, and welcome to the podcast. Before we
get started, can I ask you to say a few sentences about yourself and
about the good work that you're doing at Intelligent Buildings. Yeah, thanks,
Andrew. So my name's Kyle Peters. I'm a senior consultant for Intelligent Buildings
and I primarily focus on cybersecurity for building automation systems. Which right now encompasses
me doing on site and virtual assessments of those systems, a lot of pre
construction document reviews and policy and creation guidelines. And I kind of got started
in this from the other side where I was a programmer of building automation systems
and moved over into this world, this side of things by way of seeing
problems that I was running into. And so now I get to help out
the guys doing what I used to do to better secure they're building automation systems.
Thanks for that. And our topic is everything from safety to six two
four four three in you know, cybersecurity for building automation. You know,
I understand that you do a lot of assessments in the space. Can you
walk me through one of your assessments? What do you find in these buildings
that you're looking at? Yeah, so primarily we'll do we we like to
follow the six two four four three framework and the CSMs that you'll find at
the end of part two one of the standard and that framework that walks us
through. You know, we get started on a project and we have a
high level assessment and so I do a lot we do a lot more of
those of the high level assessments, and that's where we would walk into a
site and visually inspect and do some very light work on the computer systems or
investigation on the computer systems, and we're looking for vulnerabilities or threats or risks
that exist within the building automation system. So I walk around and I might
look at I might find things like cellular modems that the vendor, the controls
company themselves put in place for them to more easily do maintenance. I might
find operating systems that are severely outdated. I might find network equipment that was
installed in the early nineteen nineties and is still running hopefully and probably covered in
about three inches of dust bunnies. So it's those kinds of things that we
look for, and that sets us up to move on down the line of
the program so that we can get a more in depth look and we can
start developing policies and doing those sorts of things to really take their their program
and implement countermeasures and those kinds of things to make their program stronger. Okay,
and you mentioned six two four four three dash two dash one. I
haven't read that in a while. You mentioned Appendix B. Can you can
you give us just a bit of background. What is two dash one and
what's Appendix B and how do you use it? Yeah, so two dash
one is uh, it's entitled the Establishment of an Industrial Automation and Control System
Security Program. So it's basically just how you get started and how you get
going with a security program within an industrial control space or in our case,
buildings, and Appendix B is the roadmap for that and it literally has a
diagram that shows where you're at. So we use that as our as our
diagram for our whole program that we get going, and specifically as it relates
to what we've been talking about with walkthroughs, that would be the second section,
the high level risk assessment, and so that helps us determine what risks
already exist within a facility, within a building automation system. And at that
point we're also going to start looking at what the target is that they're trying
to achieve, so that we know where the disparities are and we can help
the client develop their program from there into something that more closely reflects what they're
trying to achieve. So nay, you know, for anyone who hasn't looked
at the six two four four three series of standards in a while. I
mean, I'm most familiar with three DASH three, which is the one that
says, you know, you have to have antivirus here, you have to
have long passwords there, six I C six two four four three is the
you know, the whole family of industrial automation standards. One DASH one is
you know, concepts and terminology. It talks a lot about zones and conduits,
which are basically you know, subnets, its network segmentation. Two DASH
one is the one we're talking about here, which is getting started with an
automation and control system security program. Uh. Two DASH three is patch management.
Two Dash four has to do with, uh, you know, when
you're establishing a program, what are the requirements for the program. So two
DASH one is getting started. Two Dash four is you know all the rules.
Three Dash three is all the rules for you know, which controls to
put in. Three DASH two is doing risk assessments. Uh you know.
Four DASH one is UH secure product development. This is for the developers of
products. Uh you know. Four DASH two talks about uh, you know,
requirements for security programs. There's a lot in there, and what we're
talking about today mostly is the two dash one, which is getting started designing
one of these programs in the first place, as opposed to looking at at
individual measures like you know, password lengths. So that makes sense. But
you know, you said a moment ago when on your walk through you're finding
ancient gear, you're finding you know, dust and presumably neglect, it sounds
a little depressing, you know, when you compare what's there to what's in
you know, two dash one, you find gaps. I assume, you
know, is any of this changing? What's changing in this space? So
the biggest thing that that has changed recently and there in the last three to
four years. Obviously with COVID and work from home there's but it started before
that, but you know that timeframe really accentuated this. That remote access has
become a big thing, and I think that that is starting to drive more
awareness towards cybersecurity for these buildings that before this, the most common thing we
might hear is who, what's the worst that can happen? You know,
it gets warm in an office, And now they're starting building owners and property
managers are starting to see more of that risk because it's happening in other sectors
and they're realizing that they're online more now to to so that that risk is
heightened at that point, So remote access. I mean, you know,
I'm looking at the news just yesterday at you know, we're recording this here.
Just yesterday there was news that MGM had been breached. You know,
details are skilled. Apparently the attackers claim that they did some social engineering.
They made a ten minute phone call to the to the help desk and got
in. Now they didn't say remote access, but you know it, my
guess would be, I don't know that someone gave him a password. Again,
I don't know how credible this is. It's very early days, you
know. Do you have a take on what's happening at MGM. Yeah,
as you mentioned, it's it's hard to say at this time, but I
can envision bringing this over to the building automation side. If if I were
to call up and pretend to be the vendor, the programmer for their building
automation system, maybe I installed their Tritium system or something. I don't have
to actually done it. I just have to know that it's there and pretend
to be that guy and say, you know, I'm really trying. They
called they've got an issue I'm trying to help them remotely. Can you go
over there should be a sticky note. This happens. I see this all
the time that the building, the facilities guys put the username and password on
a sticky note stuck to the bottom of the monitor. Now some of them
get super sophisticated about this and they put it on the bottom of the keyboard
so that you have to turn the keyboard over to see it. But you
know, if I called up as you mentioned, if I call up helpdesk
and say, hey, you know, I'm trying to fix this forum,
can you just go look and tell me that what that says real quick so
that I can take care of that. That might be one thing. You
know. We can also if I on a call again, pretend to be
a vendor and figure out what systems they have, Then I know what protocols
they have, and I might be a short showdown search away from discovering where
their systems are located at on the internet, you know, finding an IP
address and perhaps getting into things very quickly that way, just from a conversation.
The latest numbers in the twenty twenty three Threat Report on ot cyber incidents
show that the threat environment has changed fundamentally. At the beginning of this decade,
OT cyber attacks with physical consequences have changed from a theoretical problem to a
very real problem, more than doubling every year. The new report is focused
on deliberate cyber attacks in the public record. These are attacks that cause physical
consequences in process industries and discrete manufacturing. Most of these attacks are ransomware,
though the fraction of activist attacks is growing, and the report's appendix includes a
complete list of all cyber attacks since Stuxnet that meet these criteria. To see
how today's OT cyber threat environment has changed, I invite you to download the
report, a joint effort between Waterfall Security and the ICs drive OT Incident Repository.
You can download the report at Waterfall dash security dot com slash twenty twenty
three dash Threat dash Report, or just go to the resources menu at the
Waterfall Security site and click on white papers and Ebooks. So, Nate,
as you and I recorded, it's a few weeks after we recorded the UH
the session with Kyle. More is known about the the MGM hack the UH.
You know, the reports in public suggest that what happened was there was
social engineering. The bad guys called up and uh, you know, persuaded,
they helped usk that they were legit, and you know, they had
the uh, the account name. They've done some you know, some research
on social media on LinkedIn, they found some employee names. They came in
impersonating one of the employees, uh said, you know, I've lost my
accounts messed up. Can you reset my two factor authentication? So they had
two factor authentication allegedly, you know, it's just these are news ports allegedly
enabled, and so they called in and got the all that reset so that
they could log in and you know, stole I don't know, the reports
I'm reading said unknown terabytes of information. So it was an information theft process
allegedly. You know, they were apparently eventually discovered, so they handed the
credentials over to another part of the you know, the the underground economy,
the ransomware ecosystem, who started encrypting everything in sight and encrypted apparently a lot
of servers and virtual machines and eventually impaired the gaming systems, the access control
systems, the reservation systems, and everything ground to a halt. Yeah,
you know, I think that last bit has to be the most surprising part
of this all for me that you could, as a general ransomware actor that's
just trying to lock up files and whatnot, end up affecting you know,
I don't know slot machines and doors and such. How could it be that
those systems are so interconnected. Short answer is, I don't know. In
this particular case. You know, MGM hasn't published their network architecture, and
I really don't know about the gaming machines. I just I don't know how
that part of the of the industry works. But you know, let's talk
about the door systems, you know, the when we talk about OT.
You know, I'm not sure I asked Kyle is but you know, is
the door lock system part of OT? Or is OT really the air conditioning,
the power systems, the sort of the hard OT. But you know,
we we Waterfall puts out a threat report last year. There were fifty
seven incidents worldwide that caused shutdowns of everything from buildings to uh, you know,
oil terminals and very commonly, now I don't have the numbers, but
it's very common that the ransomware group targets it, does damage on it,
and then operations has to shut down because operations depends on something in it,
and you know, it might be that the door lock systems were in it,
or it might just be that the door lock systems depended on I don't
know, active directory to log into an active directory was crippled, or it
might be that the door lock systems depended on some other system in it that
had been crippled. These dependencies seemed to be responsible for a lot of physical
shutdowns when it's really it's it systems that go down. But you know,
people haven't done the dependency analysis and it bites them. Are people waking up
to this? I think so yes. As we do more of these assessments,
that risk assessments that we've talked about, the eyes start opening a little
more. And you know, here at Intelligent Buildings we have a remote solution
that's uses as zero trust architecture and whatnot. That's one solution you guys,
waterfall, you have the unidirectional gateways, and I really do wish I saw
a lot more of that kind of thing as well within building automation systems,
not just in the industrial sector. So people are starting to take note.
I'm seeing less and less unsecure team, viewer connections and more there. And
there's other products out there too, you know, there's more, there's more
solutions coming up every day. So I'm starting to see more and more of
that. But as much as I say I'm seeing more, there's still a
long road to go. And as awareness grows, I think we're going to
see that percentage of unsecure internet access or remote access sites that number going down
hopefully. When we were, you know, talking about the possibility of this
podcast, I remember you used a buzzword that I wasn't familiar with. You
said that, you know, you do security assessments, risk assessments, he
said, you also do spec reviews. What's that? Yeah, So a
spec review, you know, the specifications that come out leading up to a
project. So before construction, be that a new construction at building coming up
out of the ground or maybe we're redoing a floor, we get the specifications
of what's going to be going in so design design documents and information about the
systems that a vendor is planning on installing, so we look at those before
they're built so that hopefully we can avoid building in issues for from day one.
There's and there's all kinds of things that we see there from specs that
call out the use of ancient technology, outdated operating systems, those sorts of
things. So we try to catch those issues when it's when it's most cost
effective to fix them, and that is before they are purchased, and then
give those results back the engineer reviews, they change the spec hopefully, and
and then we can help ensure that a building is built, designed and built
to meet the client's own cybersecurity policies and their goals for for being as CyberSecure
as possible. I guess it makes sense when you're when you're looking at U
spec, you know, you want to design the building to be sort of
modern and secure. What does that mean though, I mean, I'm guessing
that a bank needs a different kind of system than does like a parking garage.
Yeah, yeah, absolutely, you know, the risks are different,
and we've seen all kinds of this stuff. I've seen in doing assessments where
the bank needs to protect against nation state attackers that they're actually getting hit on
a daily basis, and their parking garage may not have much more than fans
and CEO or two sensors, and so they don't view the criticality the same,
so they set different targets for that so that they can put resources where
they have deemed that they're needed. So we use the six two four four
three standard to help get this program in line where they have their their security
levels of zero through four, where we say zero is essentially we don't need
to protect that system at all, and a four is the ability to protect
against nation state attackers or something extremely high level like that. And most buildings
fall somewhere in that one to two range where they need to be able to
be resilient. They need because the CO two sensor, for instance, that's
something that's critical in that space, but may not have quite the same impact
if it goes down or is becomes vulnerable as the cooling system for the data
center that keeps the whole bank running. So that's why they set different targets
for different systems and different buildings. Perhaps that's interesting, I mean, I'm
coming from sort of the heavy industry perspective. In heavy industry, safety is
always job one. If if a hacker gets into the CO two sensor and
reprogram ams it to say, you know, it's not three percent CO two
in the air that is is going to trigger the fans. It's ninety percent
CO two in the air. That's a safety issue. People in the garage
are going to get sick or worse. Should the CO two cents or not
be you know, really thoroughly protected, just like the bank's data center.
It's a good point, and yes it should be protected. We don't want
that system to be completely vulnerable. I would never put that as a at
a zero for instance. But as far as the risk maybe maybe it's you
know, depends on the construction of things obviously, and so we still want
to protect it. But do we need to put the amount of resources towards
that that we do other systems, and that is up to the client,
and that is up to what their risk tolerance is. As you mentioned,
that's thattra. It's getting into a life safety issue, which I think is
important. Uh So we would want to protect that, and maybe one of
our protections is that we don't have connectivity to that system. Maybe it's a
standalone system. I don't like. I don't like necessarily having the air gap
mentality as a firm way of protecting as they as someone might say, uh,
you know, philosophy of protection for a system but maybe we put that
as read only points. You know, they have to be hard coded in
or something, so we find countermeasures that make sense for the application that we're
looking at. This very issue is actually being discussed within a group called Building
Cybersecurity dot org. It's BCS dot org, and we're working on taking the
six two four four three standard and making it more applicable to buildings and safety
instrumentation. Systems that are very common within industrial controls are less common or not
common at all within building automation, and so this is still something that is
being debated on how to handle these things as this as this industry matures.
So Nathan, let me add here. You know, I'm watching what some
of the drafting teams are doing in six two four fourth three, not just
I'm not part of the Building Automation BCS dot org. The question of security
levels is being debated even more widely than BCS dot org. You know what
are security levels? They let me back up them all. They're basically four
levels that describe the the capability of an adversary that you have to defeat with
your security program. So you know, sl one says, I've got a
program is strong enough to defeat script kitties who know almost nothing but you know,
and download a tool, press some buttons and get in trouble. You
know. SL two in my recollection is something like, you know, insiders
who've got some knowledge, who's got some permissions. SL three is basically,
you know, they don't use the terminology, but I read it as organized
crime, and SL four I read as nation states. And so if you
say I need, you know, my network has to be withstand an SL
four attack, It has to withstand a really sophisticated kind of attack. And
safety systems you might ask, well, how should they be protected? Well,
A, that's being debated, and you know, b one of the
observations I make in you know, the book that I just released is that
it makes sense. It often makes sense to use different security levels for different
adversaries. And so if the ransom are group nowadays are using what used to
be nation state techniques, and you know they're they're trailing nation states by only
a few years, it really makes sense to take really sensitive systems like these
safety systems and protect them from nation state grade network attacks. But the other
controls, like the anti virus and you know, those controls really you know,
are passwords or you know, access management. Those controls really are relevant
to physical access to people, you know, who are our insiders, not
who are coming in across the network. And the insiders tend to be much
less capable. They tend not to be you know, to have nation state
attack tool capabilities and knowledge. And so you know, what I'm seeing people
start to do is using different security levels within the same network for different types
of security controls. The controls that are focused on insiders might be set at
an sl too, even for the safety systems, because you you know,
the insiders just aren't that clever, bluntly, whereas the security tools that are
focused against network attacks coming in from the outside are at a much higher level.
So yeah, it's something that's being debated in multiple places in the industry.
This whole question of I call it the question of how much is enough.
I'm going to use it as an excuse that your book is very new
and so I haven't got a chance to read it yet. But I guess
what I'm wondering is why you wouldn't otherwise just ramp up all of your defenses
as much as you're able to. Is it just a matter of resources because
in my head, when you say okay and then SATURA doesn't have a nation
state's capabilities, Well, what if a nation state plants somebody in a manufacturing
or wherever you're talking about. I know that that's a bit far off,
but why wouldn't you overestimate their capabilities rather than try to guess exactly who you
might be up against. Well, you certainly, you know in theory you
can protect everything to nation state level, but it gets very expensive, and
you know, the question is is it is it really needed? So for
example, if you have I don't know, if you're running something insane like
a nuclear generator, you have to have everything at the nation state level,
meaning you're even the security controls that you have deployed to protect against insider attacks.
You've got to consider the fact that a nation state might put a sleeper
or three, you know, a spy into your organization twenty years ago and
activate the spy today because conflicts are ramping up. You know, is it
really reasonable for a building, you know that, you know, an office
tower with a with a parking garage to take measures that are sufficient to detect
sleepers that other nations have put into their organization, you know, twenty years
ago, that's just overkill. So yeah, it's a cost thing. You
look at the you know, the obligation that all of us have who are
operating you know, dangerous equipment. The obligation we have is not to do
the most that is possible. The obligation we have is to do something reasonable,
to do what any reasonable person would do if they were in our shoes
and saying, I'm going to protect against you know, intelligence agencies planting sleepers
in my building that you know, you know, keeps I don't know,
keeps a retail store going. That's just not reasonable. And you know it's
it's a lot of money to spend on stuff that isn't reasonable. I take
your point, Andrew, and I agree. If you're operating a nuclear or
facility versus a building automation system, then you would apply different security controls to
those two situations. But if I understood correctly what you were saying originally,
it was that you would apply different grades of security to different kinds of systems
within one site, which is what I'm more curious about, Like, whether
it's building automation or a nuclear facility, why you wouldn't set all of your
security controls to a level four, a level two, or what have you.
That's a good question. So, you know, I answered the question
that certain security tools protect you against insiders versus outsiders, and outsiders nowadays tend
to be much more sophisticated than insiders, so there's some distinction that you make
across different kinds of tools within the same network. But you're asking, is
the whole network you know, fine, you decide that it's SL two for
insiders and SL four for outsiders, But is the whole network two for insiders
and four for outsiders, or you know, is it three somewhere? And
the answer is that in theory, you know, what's six two four for
three says? Is you know, every little network that has a slight the
different function you might give a different security level too. In practice that gets
really complicated and you start making mistakes about applying, you know, the wrong
security controls to the wrong networks, the wrong level of security control. So
in practice, what I observe people doing, yeah, is applying pretty much
the same set of standards, the same approach to security controls to entire networks,
just because you know, breaking stuff up into seventy three sub networks each
with a different security policy is just hard, but in theory you could do
that. There you go, So that's progress the industry wide. This has
been great Kyle, thank you for joining us. Before we let you go,
you know, can you sum up what should be be taken away here?
Yeah? You know, I think I think the biggest thing to take
away is that there is hope that things are looking up and the building automation
industry is kind of slowly but steadily working on catching up to the IT industry
and the ICs industries with regards to maturity in cybersecurity. As I mentioned,
groups like BCS dot org are doing great things to help push things along.
And my advice would be that, you know, if we're going to do
things like remote connectivity and remote management systems, don't be the bottom rung on
the ladder. You know, Let's let's start taking a look at this and
take cybersecurity seriously. And it's not just it's not just who would want to
attack, it's how do we keep our systems running no matter what happens.
Somebody spills coffee on the server, you know. I mean, those kinds
of things are our little things that we look at to keep systems resilient,
and you know here intelligent buildings like so we do the assess months, we
do manage services to help keep things going once they're operational, So things like
that. I think, I think we're moving in a positive direction and I'm
very excited to see where the future takes us in this industry. And I
love it. You know, it's just a great, great industry to be
in with some awesome people of keeping buildings running for the world to keep working.
Andrew, that was your interview with Kyle. Do you have anything to
take us all out with today. Yeah. You know, we've had a
couple of episodes on building automation before. I'm reminded. One of them I
think has in the title twenty thousand CPUs, and we talked about really how
how many you know CPUs in thermostats are scattered through a large building like a
skyscraper, and how exposed these systems are because you know, people can touch
the thermostats, they can pull them off the wall to get access to the
wiring. You know, they're they're exposed to attacks in ways that you know,
other systems just aren't. I remember an episode talking about destroying a three
hundred ton chiller by operating it too fast for a number of hours, the
blades that move the liquid coolant were moving too fast and there was vacuum cavities
forming behind these blades, tremendous vibration over a course of hour hours that you
just you destroy the cooler. And today we're talking about you know, BCS
dot org. Uh, the organization is debating security levels. It's basically asking
the question how much is enough? How much security is enough for different kinds
of networks. And you know, I observe that the I see that debate
in the larger ie C six two four four three standards community as well,
and you know, the larger community. In part, I mean, there's
many reasons to revisit this question, but in part, it's because the threat
environment's evolving. Uh. You know, tools and techniques that you know fifteen
thirteen years ago when when the standard I'm most familiar with the three to three
standard when that standard came out. The tools and techniques that nation states were
using that was SL four today are being used by ransomware which is SL three
adversaries. And so you know, how many of the security approaches. The
security controls that used to be appropriate to nation states at the SL four level
now need to be reclassified at the SL three level. All of this is
being debated because again, you know, threats continue to evolve, and you
know, I summed the whole thing up as with the question how much is
enough? How much security is enough? How high do we put the bar?
This is in a sense a constant debate, but in the standards community
it's being specifically debated in the last I think twelve months or so. Well,
then, thank you to Kyle Peters for bringing all of that to our
attention. And Andrew, thank you for speaking with me. As always,
it's always a pleasure. Thank you, Nate. This has been the Industrial
Security Podcast from Waterfall. Thanks to everybody out there listening.