So we started working on defensive measures. How can we automate that, just
as we automate engineering actunities. Welcome, everyone's the Industrial Security Podcast. My
name is Nate Nelson. I'm here as usual with Andrew Ginter, the vice
president of Industrial Security at Waterfall Security Solutions. Who's going to introduce the subject
and guest of our show today. Andrew, how are you? I'm very
well, Thank you, Nate. Our guest today is Yusuf's Jad. He
is the chief technology officer and co founder at Sivault and his topic is active
defense in ot how to make it work? And you know when I heard
about this concept, I was very interested. Common wisdom has it that you
cannot do this on industrial network, yet here's Cybald doing it all right,
then, without further ado, here's your conversation with Yusef. Hello, Yusuf,
welcome to the podcast. Before we get started, can I ask you
to introduce yourself a bit and tell us about the good work that you do
at Cybald. Well, thank you for having me, Andrew. My name
is Yusef jd CTU O T I C S cyber Defense and co founder at
Cybald, which is a division of pm SKADA cyber Defense, Originally an engineering
company. We deployed manufacturing plants and such substations and so forth. So as
part of my role, I'm leading cyber defense operations and novel R and D
products. Have over twenty years experience in osh so IoTs also blockchain and quantum
cyber applications when it's possible, So we're trying to be ahead of the curve
whenever it's it's possible. I'm also course consultants for fourteen ten companies delivered turn
key solutions to multiple organizations, including the HSBI. I've trained military units on
defense and offense exercises UM and to them. Also part of an same or
for ICs. For ICs which you might know, it's is the Incident's Command
system for industrial command control systems. It's a US initiative UM and I was
also a lead on the Want to Cry v too or hansomwhere leading the data
task force globally and as I wold, we serve mostly critical infrastructure including energy,
healthcare, smart series utilities, sometimes defense, and others. I didn't
know you were involved in in uh quantum. We'll have to get you back
on on the show again for for for a different topic. But you know
today's topic is active defense and I mean, you folks have a product that
does this in the OT space. And you know this was a surprise to
me. Um. You know, there's a there's a widespread perception, there's
common knowledge in the industry that you don't do active defense in OT. It's
it's you know, too risky. But you know, before we get into
the the the why and the how, um, and you talk to us
about what you've got in the active decent species you space. You've got a
cyball dome product. What what is it? What does what does that product
do? And then we'll dig into the details. Yeah. Sure. So
a few years ago we looked at the attack landscape and the otics industries,
right, and we saw that there is actually a need for for defending them
against this attack. So we started working on defensive measures. How can we
automate that, just as we automate engineering activities, and we created dome.
Dome came from you know, a dome principle and it's also a very familiar
Israeli irondome. Um. So we focused especially for otics, which was the
sector lacking these kind of defenses, so that it is not too late when
an attack happens or when it is initiated. M So, we created a
hybrid model. It's not a fully automated as anyone might might think. It's
a hybrid model where there is automated incident response actions and there is also manual
incidental response actions, and those we will provide through our DOME platform to our
customers or partners would be operators, owners and such, so that they can
execute those and then we support them along the way. So as a service,
it's industries first managed or co managed Cyber Defense Operations Center or o g
i CS, and it comes with technology that is software. We have appliances
that are for different environment We have ruggodized appliances that take action, provide visibility
into those environments and so forth. So we take care from the visibility standpoints
or stage all the way to defending actively in case of a confirmed attack.
Okay um, and you know where the beginning. Some of your some of
your some some attacks have manual responses, others are automatic. I'm especially interested
in the automatic responses. I mean common with is that you don't do this
because you know years ago, what I remember is people would take intrusion prevention
systems, which were basically firewalls or you know, stuff hanging off the side
of the network looking at packet traces. Basically, if you've got an intrusion
detection system that's looking at the network saying, oh hey, look that's an
attacking progress, and triggers a prevention action, which is usually I don't know,
a TCP reset pack it's saying, turn off that TCP connection that is
launching the attack. It's not there anymore. You wound up with, you
know, a signature update would come through and suddenly half of your connections between
the SKATA system and the PLC would be interpreted as attacks and interrupted and everything
would just break. And so the common wisdom became. You know, you
can do intrusion prevention very high in the architecture, you know, in the
IT network close to the IT network, you can do intrusion detection very low
in the architecture. But you never do prevention low in the architecture. Yet
here you are not sorry, you know, here you are responding to attacks.
As you know, an intrusion prevention system of all would have deep in
the architecture. How do you do that without risk? Two? Normal operations.
That's a very good question, Andrew, and yes, that's why we
created a sivault and we've done all these efforts. So our principle is being
automated as much as possible, and the first level is providing security controls that
are built for otics not that's come from it, and they are trying to
implement them in ot environments and adapt them to these environments and processes so that
they do not impact them. It's the right way from the beginning. So
this is for the security controls. The r aspect is we have Cybold Labs,
which includes partnerships with different engineering and vendors that provide engineering equipment where we
test and officialize our high fidelity response actions. So if an action for a
specific asset is confirmed hundred percent that it's not going to impact operations delivered by
that equipment, that goes into our automation actions that are part of that pool,
and of course the engineering engineers can confirm the the you know, if
that action could be implemented in their environment or not. And that's of course
collaboration between us and the plant resources and it's strictly blocking attack specifics as an
intern so it's not if you think of an IPS having all those rules and
signatures or behavior analytics turned on and then pushes updates like you said, Well,
that's not the way we're doing it. It's the reverse. So let's
take this IPS and then it's not enabled completely. It's enabled on specific attacks
that are happening in this environment, and that we confirm that those signatures have
been tested and could not impact the operations. So they're not going to block
a mod bus or an obviously away command. Our communication that is going through
that is all tested on our end. So any what I'm hearing here,
you know, it's let me back up a bit sort of general principles engineering
change control. You know, if you've got a safety critical network or a
reliability critical network, every change is a potential threat to safe and reliable operations,
so you have to study the change before you make it. All of
that is true, but sort of the next level of detail is that,
look, some changes on these networks are carried out on many networks are carried
out routinely. How can you do that? How's that possible? What's an
example of that is, you know, a new screen on the human machine
interface on the graphic user interface the twenty four by seven operator uses. They've
got one hundred thousand data points. They're managing the only hope they have of
You know, you can't see one hundred thousand data points on the screen at
once. And so you've got summary screens. You've got summaries of summaries.
You're constantly summarizing the information differently. There's new screens. You know, you've
got a thousand screens. You might invent a new screen every month or so
because you've discovered a new need in terms of seeing summary information. And it's
the plant operators who are the gurus on using these screens, because they do
this time. And so most sites, not all, but most sites,
the engineering team will study the process of how to safely define new screens on
the HMI and they will put together a procedure for the operator saying, when
you need to define a new screen, this is what you do, and
you go step by step through the procedure, and the operators follow the procedure
to change the screens on the HMI. And so this is an example of
a change that has to be done routinely, and so the engineering team has
studied it, put a procedure together, it becomes part of the system.
What I'm hearing that Yusuf is telling us is that his technology is being treated
the same way saying an interruption. You know, if there's an attack detected,
we need to interrupt that attack. That in principle is a change to
the operating of the system. And you know they're not saying an intrusion prevention
system from the IT network. Throw it on the OT network. Throw all
the signatures in there, cross your fingers that nothing bad happens, because you
don't know what these signatures are. They were designed for a different environment.
He's saying that his system was designed for the the engineering environment and has been
tested on specific vendors, specific assets, specific attack scenarios, and the vendors
and engineering teams have said, yeah, this is safe to do to this
device in this context, and it's only sort of approved changes, approved attack
interruptions on approved assets, you know, in scenarios that the you know,
the engineering teams understand thoroughly. It's only those approved changes that he'll carry out.
And this is why I think he's got a manual process. If you've
got anything else going on, you've got to have manual process. So this
was the deep insight that I got out of you know, the sort of
the first insight I got out of this this inter you, Um, it
is possible to do routine change provided it's been engineered, and this is what
they've done. You've got equipment into your labs, you've been testing. Uh
you know whether your your system would would interfere with normal operations? Um.
You know when you take your equipment and out into the field, You've got
equipment in the field that you've tested against. You've got equipment that you haven't
tested against. How fine grained you know? Ore is your your system in
the field? You know you can presumably tell it this asset is you know,
safe to to send messages to to interrupt the tax in progress because we've
tested it. You know that asset possibly not? Is it? You know?
Do you control the active response per asset or you know, per attack
per asset or purport on the asset? How how fine grain? How do
you control or know what is safe and what is not working With our customers,
we act only on assets that do not impact operations or safety. So,
for example, an engineering workstation that has no involvement in real time activities.
We could, for example, take an action that says isolate this machine,
so if we act automatically, that action is per asset and also on
those attacks specific signals, so would have criticality levels assigned to those assets working
with the operatures or the engineers in the plant or other environments, and then
we assign them criticality in terms of are they part of the real time process?
Could we automate these actions? Which actions should we automate or can automate?
And we we leverage things like the five DS, so we could defend,
which is automatic action of blockage. We can also delay or just go
revert to detect in some cases. So it really depends. But the concern
here is the automeric actions. So I understand that is going to be part
of our collaboration with the field. In the beginning of the industrial security revolution,
engineers were told to use it security principles, protect the information. We
were told. We knew this was a poorfeit, but it was all we
had. Today. The top security priority at industrial sites is safety. Don't
kill anyone, don't cause an environmental disaster, and the second priority is reliability.
Do not shut down our factory or infrastructure Today, safe and reliable operations
use unhackable protections from cyber risks, not just cybersecurity. For a deeper look
at the evolution of the revolution, we invite you to download Waterfall's report on
the Emerging Consensus for Industrial Security Engineering. You can access the report at the
Waterfall website Waterfall dash Security dot com, slash Engineering dash Consensus, or just
go to the resources menu and click on white papers and ebooks. So the
second insight that I've got here, and you know, he didn't quite say
it this way, but you know, he gave the example of sort of
an asset that's almost always safe to target, which is the engineering workstation,
which is a full Windows machine or a full Linux machine with a lot of
software on it and the ability to interact with just about everything to reprogram it.
It's a very powerful tool, and so of course it's a natural draw
for cyber attacks reaching into the OT system. And you know, there's other
what what's in an industrial network. There's a lot of you know, PLCs
and r to us and very low level devices that might have some horribly stripped
down Linux under the hood, or a real time operating system that has very
limited capability. You give these devices limited capability deliberately so that you can test,
you know, deeply, the limited software they have to make sure it's
safe, it's reliable. So there tend to be two kinds of assets in
industrial networks. Assets that are sort of full fledged Windows Linux powerful tools,
and assets that are much less powerful that have one job to do that are
extremely reliable at that one job. And you know, in my read of
the space, if an attackers coming in, they are going to want to
get a foothold on the most powerful machines in the space so that they can
use those machines to manipulate the sort of less intelligent machines in the space.
That machines are actually connected to the physical process doing stuff. And so what
I heard was that, you know, it probably makes more sense. You
know, I didn't hear this use of didn't say this, but my read
on it is that, you know, printers and engineering workstations and remote access
jump posts, all of these full fledged Windows Linit machines, these are the
ones that are likely to be the most tolerant of intrusion prevention actions sending them
a message saying, you know, interrupt this or turn that off or because
they're the most powerful, and they tend to be you know, the stuff
that's essential to second by second safe operation is well bluntly dumb. It has
one job to do, stay safe. And the stuff that is more powerful
tends to be the stuff that manipulates and programs these other dumb systems. So
every attack has two pieces. If you apply the intrusion interruption process to the
non essential machines that are attacking everybody else the dumb machines, that strikes me
as as brilliant. I thought, Hey, I never thought of that.
You know what a clever approach this is. This is the second thing I
got out of it. Even though I'm not sure he quite said these words.
I might be reading something into what he's saying, but it makes sense
to me. Can you give me a couple of examples? What what's an
example of an attack that you would most likely interrupt because it's it's all been
tested, and what's you know, maybe an example of an attack that you
would I don't know, maybe just report rather than get in the middle of.
Can you give us some some concrete examples here? Yes, So let's
take an example of a ransomware attack Let's say that engineering workstation as an example,
or a PLC that is not sending real time event so it's all depending
on real time activities. If that engineering workstation get infected by our ransomware,
even an unknown variant that we have never detected before, and now we confirm
that it is, so our actions will include isolating, for example, that
engineering workstation and then validating after that. What what are we going to do
more than that just that isolation. The other example where we're going to take
manual action is a PLC or an engineering workstation that is involved in real time
process. It's sending commands, it's sending changes, and those are almost real
time or real time, so those we cannot going to impact the operations.
So what we do is we send man response actions that can be executed by
the field resources, and then we're with them during that execution, supporting them
so we can minimize and you know, contain that attack and minimize the impact.
So that's interesting. You've used the word isolate a couple of times.
How do you isolate an asset? Do you like log into the windows and
tell it to drop the network interface or how do you isolate? I'm used
to interrupting communications with like I said, TCP reset packets of old what do
you do nowadays to isolate? So our device is a consider it an orchestrator
of the security controls that are already in place into viruses, machines that we
can interact with safely, like Windows prcs that we can communicate with, and
others. So it orchestrates all this incident response action at the network level.
Or if an agency is installing a machine, would it be a third party
agent or our own proprietary agent that we put in place for for insident response
purposes. In the network level, there could be a industry switch stick,
for example, a rugged gom swhich would take action on all an inty virus
that is already sitting on that machine that we would leverage. So this makes
sense. Um, you know you are are interrupting only those communications that are
a less critical and B you've tested for the impact on the physical process.
Um. But as I said, you know, in my experience, there's
a fair amount of resistance out there to any kind of active response because of
the experience you know ten years ago of the old intrusion prevention systems. You
know there's a lot of a lot of engineers are that old, they remember
those problems. How do you get you know, how do you overcome that
sort of resistance in the marketplace? How do you get buying from the engineering
team who are are very focused on controlling change, controlling everything? How do
you get them to buy into this. What we do is we start first
of all with visibility and detection, right, Um, that's the first step.
We don't come in and say automates everything. After that, we start
automating little by little, working with the engineering teams and the plant resources and
the field resources which are involved with us in defining what is critical, what
is not critical as assets, what actions could we take, what actions could
we not take at all? And there is sometimes where we have semi automated
responses. So the work is done with the engineering team. We do not
just rely on tech. Tech is just there to help an assistant facilitate,
but it's actually this collaborative work. So if we take, for example,
let's say the engineering change control, it's a very crucial process. So they
leverage thinks like automated changes or automated routine challenges. So we could take these
incidental response actions that are automated as part of engineering challenges and activities, and
that way, they are part of it. So we always look at this
from the engineering aspect, actually not from the IT cyber defense aspect, so
because we are very aware of the impact that it can cause if something goes
wrong. And it seems like in your question there you're referring to maybe some
specific personal experiences that you had some years ago. At this it wasn't so
much personal experience. It was sort of, as I said, Colin Wisdom,
in the industry, it was a long time ago. I don't even
remember specific examples. I know there were specific cases where, you know,
entire plants were shut down because of a malfunctional because of a miss you know,
a misfire of the intrusion prevention system. And you know, it doesn't
even take more than a couple of these incidents before you know, multiple industries
draw the conclusion that this is not going to work. So you know,
for example, if in uh, you know, in any industry um A,
you know, you put a let's say a an intrusion prevention system built
into a firewall. You put a firewall between the control system, the HMI
and the devices, the PLCs and that system conclude. You know, it
gets a new a new signature, download that and you know, and it
doesn't understand industrial protocols, and so it sees a new signature comes through saying
when you're when somebody, someone's targeting an attack on this port that you you
know they haven't targeted before, but that port is being used for a different
purpose in the PLCs, in the you know, the real time communications.
Now it says, oh, there's communications on this port and it matches this
crudely defined signature, you know, poorly tested signature. Bang, shut down
the communications to the PLC. Now the operator can't control half a dozen its
plc's or her plc's and has no choice but to shut the plant down because
the operators lost control of the physical process. This only has to happen a
couple of times before entire industries. I mean again, that's sort of a
generic example. I don't remember where exactly had happened, but I know it
happened. You know, these things happened. But you know, someone like
refining to turn around and looks at this and says, oh, if I
have to trip my plant, I process one hundred million dollars worth of product
a day. If I trip my plant and I've got to start from a
cold start to go back up to full production, it's going to take me
twelve days to get back up to full production. So I've lost, you
know, fifty percent of that. The average between zero one one hundred percent
is fifty percent fifty percent times six days. Do the math. At one
hundred million dollars, I've lost a half billion dollars of product. Here a
half billion, now five hundred and six hundred million dollars worth of product.
Never going to use one of those, you know, it just bang gone.
Entire industries groups of industries look at one of these incidents and say that's
unacceptable. So I don't remember the specifics, but you know, a couple
of them happened and everybody turned around and said not doing that ever, the
you know, it just costs too much. Then, Andrew, how,
you know, give me a sense how strong is the resistance, because his
use have mentioned, you know, he's working with engineering teams on this seems
like it's pretty amenable. I imagine there's a fair bit of resistance initially,
and you know, I thought his answer was interesting. He said Andrew.
We start with detection. I mean they are detecting attacks as well as interrupting
them. Start with detection, you know, show the engineers what's happening in
their network, and you know, show them presently develop a relationship. Now
now they're learning stuff from you. Whenever, whenever you turn on detection on
a network that you've been blind to before, you learn stuff about it.
So you're developing a relationship with these engineers. You're teaching them things about their
network, You're develop developing credibility with them. And you know, he's got
in his pocket, you know, endorsements by vendors, testing by vendors,
the same vendors that produced the control system that the engineers are operating in the
plant. You know, he's got the endorsements of the vendors saying it's safe
to do this, it's safe to do that. And the engineers are learning
about their networks. You know, they learn together. They they the engineers
are not resisting change for the sake of resisting change. The resistant change because
there's a risk. And once you persuade them, you show them, you
know what's going on, you show them the evidence. Yeah, they're absolutely
willing to you know, design, you know, include in their change control
system, you know, ways to carry out routine changes if there's a benefit.
And the benefit here is interrupting a taxing progress and that's a huge benefit.
So you know, you can't come in and drop it on them,
turn it on and walk away. That's they're going to turn it off and
walk away, thank you. But work with them. And yeah, these
people are reasonable, they want the benefits. You know, just just have
to make it happen. And maybe there's a generational component to it too,
you know, like the ways that people thought and the experiences that they had
ten years ago maybe aren't quite as relevant today to a degree. But engineers
new or old, nobody wants to trip the plant. And so yeah,
you know, the younger folks might not have suffered quite as many hard knocks
as the older folks. It's this is why you have, you know,
these engineering teams. We haven't talked about it. I should bring someone on
talking about this. But these engineering teams design succession plans into their planning.
They have like thirty year timelines they are executing too, and so yeah,
they deliberately mix up you know, young and old so that you know,
one can learn from the others, so that you know, each can learn
from the other. So yeah, there there is some of that. We
know. You also talked about manual incident response. Now you know, I'm
familiar with the term flyaway teams. Uh. You know, something goes wrong
at a an IT network and OT network and you know, incident response experts
get on the aircraft, fly out there with suitcases full of equipment and deal
with the problem. You're doing a bunch of this remotely. How does that
work? I mean, is it as simple as you pick up the phone
and call the engineer who's your contact and tell them what to do? Or
you know, is there is there more to it than that. Yeah,
so it's a combination. First, we are present in North America covering it
from Montreal and officially and also in Europe from France, and Africa is covered
from Morocco. So what we do, like you said, Andrews, most
of it is automatically is remotely done with the teams from from our from the
field resources. But part of it is we have experts that are ready to
go on site with their GEM bags, ready to go to take care of
these incidents, but still handled from a centralized crisis crisis unit part of the
So the manual actions that we take our part of our platform. So technicians
or engineers or field experts can log into the platform or have those playbooks,
men will be clarify to them where is the process that needs to be done,
what are the steps to follow. It's a step by step kind of
guides that are also tested in our laps unprovided part of this offering. I
mean, this is this is unfascinating. I learned something today. I thought
intrusion prevention on on you know, industrial networks was an experiment failed long ago,
but here you folks are doing it, and you know, you're looking
into the future as well. I mean this, you know is clearly an
evolution over what was available ten years ago. But I imagine you guys are
working on this into the future as well. What are you working on?
What can we look forward to? What's coming? Yeah, So, so
DOME is very crucial aspect of our business, and we've been working on this
for years and making it as perfect as possible for whatever our missions we provide.
So we're continuously simplifying the implementation. That's that's one of the aspects that
we have to concentrate on and also helping industry adoption of what we spoke about
today. For our dome offering, we're also involved with different entities, would
be vendors, partners, government entities to push this principle so that industry or
community cannot cannot be afraid. We don't. We don't have to act after
the attack happened, which is the case today for organizations that are not our
customers, where an attack has already happened, the substations are already blocked,
the grades shutdown. Well, now we need to revert back right. So
that's the first piece. We want to be as proactive and as active as
possible. We can do it. We're very confident we have done it with
different entities. The other aspect is we want to push more and more also
so well, what we help with is consequence driven cyber informed engineering so CC
or just cyberinformed engineering to help these engineers in the field engineer out most possible
cyber risks. And it's what we call in the community. I see as
unhackable mitigation so or mitigation aspects that are part of the engineering design. And
I like the example that that you give Andrew, which is very simple to
understand. It's we have a boiler that can explode, and you know if
we control the PLC and we hack it and now we control the parameters of
that boiler to make it explode and kill people. Well, as a simpler
way of countering that cyber risk is putting an overpressure valve which will automatically,
you know, mitigate that risk. So we're working. I'm working with all
these entities in our company also, so we're involved with the our labs and
Caesar and others to make sure, you know, we bring this active aspect
to the community and not be afraid and I'm sure everybody will appreciate it.
Before we let you go, can you sum up for our listeners what you
know? What should what should listeners take away from the episode here? Well,
the OT I see as cyber defense aspect is something where I'm calling everyone
to contribute and collaborate, you know, practitioners, vendors, service providers to
safeguard or critical infrastructure. This this is this is a community work and we
need to provide the best ways of doing it. How can we prevent these
attacks not just detect? Prevention is key. We cannot let these criminal gangs
win over us. We can do it, as as Cybald. We have
tested it, we have customers that already have our hybrid automated model and dome,
which is our offering. Is the first step in indefendedness. And I
thank you for inviting me today, Andrew. It was a pleasure being here.
Andrew. That was your interview with use of JAD. Do you have
any final thoughts to close us out today? Nothing new, I mean I'm
repeating myself here, but I was fascinated by by this technology because common wisdom
is you can't do this, and yet here they are doing it. I
mean, you know, innovation the world, the world continues to move forward.
Good job. Okay, Well, thanks to you so for speaking with
you, Andrew. And Andrew, as always, thank you for speaking with
me. It's always a pleasure, Nate, thank you. This has been
the Industrial Security Podcast from Waterfall. Thanks to everybody out there listening.