Staying still is moving backwards and then the security I think. Welcome listeners to
the Industrial Security Podcast. My name is Nate Nelson. I'm here with Andrew
Ginter, the vice president of Industrial Security at Waterfall Security Solutions, who's going
to introduce the subjects in guests of our show to day. Andrew, how
are you? I'm well, Thank you, Nate. Our guest today is
Jake Hawkes. He is a senior product manager at Aviva and his topic is
going to be doing product security for Aviva Enterprise Skata. And you know,
this is the product that he's the product manager for. And you know,
I know this product line for a long time as as the industry leader in
control systems for oil and gas pipelines and today I know they have many other
industy that they're involved with, but the oil and gas pipeline thing that was
in my recollection. How they how they got started? You know, fifteen
twenty years ago. They used to be called tel vent Oasis, but Aviva
bought Televint or at least bought the product line a'm week on the details,
and they renamed it to a Viva Enterprise Skata. So that's what we're going
to be talking to Jake about how they do cybersecurity for a Viva Enterprise Skata.
Then, without further ado, let's listen in to you with Jake.
Hello Jake, and thank you for joining us. Before we get started,
can you give us a few words of introduction about yourself and about the good
work that you're doing at Aviva. You bet yeah. I'm a senior product
manager at Aviva based here in Calgary, and I'm in charge of the Enterprise
Skat product which was formerly known as Oasis Skata and it predates Aviva when Aviva
bought Telvent. Essentially that's how they acquired this product. They acquired this so
Aviva acquired Enterprise Skater as part of the carve out of the software business from
Schneider Electric. Me myself personally, I started my skated journey about twenty three
years ago as an intern for a pipeline operator in Australia, where I was
first exposed to the Unix version of Oasis. Since then, I have held
positions in customer support, technical sales, proposal support, project management, and
now product management. Most of my career I've been in oil and gas,
but because our products are used in water and wastewater. I've spent some time
over there as well as transportation, agriculture, and then I took a brief
hiatus from oil and gas and did some advanced weather systems, still using Oasis
and using some of my computer systems engineering degree with a bit of hardware thrown
in for some fun. So I've come a full circle starting out as an
intern on this product for which I am now the product manager. So very,
very satisfying arc of my career. Thanks for that. Our topic today
is sort of the approach for cybersecurity that two folks are using in the Enterprise
Skat product. But you know, before we dive into security, you know,
I'm familiar with the product. You guys are here in Calgary. I've
sort of watched you from the outside for a very long time. But for
everybody else, can you say a few words about what is Enterprise Skata?
Who uses it this kind of thing? Yeah? Thank you? Yeah.
So previously it was known as Oasis, and it has been known as that
for longer, so perhaps your listeners know it by that name. When we
were Intellgent and before that val Met, We've had a lot of names over
the years. It is a SKATER system. So SKATA is an acronym supervisory
control and data acquisition. It is a computer system or a system of systems
really that does supervisory controlling data acquisition. Not to repeat myself, but that
is what it does. It acquires data from across the entire asset, and
it provides that situational awareness to the operator who is sitting in a control room
operating the asset twenty four to seven, and enables them to send commands to
the field to operate the asset. So their job is primarily to move product
through the pipeline, but secondarily to keep it all in the pipeline. And
so the system starts with bringing back the raw data and allowing controls from the
operator to the field. But then that's really its starting point. And then
on top of that we layer applications that make it easier for the operator to
manage and operate a pipeline. There are many SCATA systems on the market.
Ours comes out of the box with all of these heightened pipeline applications layered on
top of it and integrated into it. On top of that or next to
it, if you will, coming out of our same product group are other
pipeline industry applications. We sort of know these as the advisor application. So
we have Measurement Advisor, we have Gas day Advisor, Commercial Advisor. These
products then are ancillary products around this SKATA system and they bridge between the OT
space the control room with say the commercial aspects of buying and selling product from
your suppliers and customers, as well as then accounting for the product as it
goes through your pipe. So Measurement Advisor is a gas measurement system. We're
working on a liquid enhancement to that so that we'll be able to measure engls
and other things like that. But the gas measurement accounting system then is a
way for you know, the company to build based on an energy value,
not just a volume. So so there was an example of some of the
layered applications on top. But fundamentally, enterprise SKATA is a skater system.
Skater systems differ from DCS systems direct control systems mostly by the way in which
the communications is arranged, and dcs is are usually on site with the actual
field equipment, like at a compressor station or something like that, whereas a
skater system is meant to control the entire pipeline, and we'll often interface with
the DCS systems, you know, in the form of talking to the PLC
and so on directly. So yeah, so a skater system primarily, but
for us, it's the platform on top of which we layer other pipeline specific
applications. So just a little side note there to give you some insight into
the industry. Ate in my recollection, it was Valmet created Oasis, and
then Telvent a conglomerate bot Valvement, and then later on Schneider Electric bought tel
Vend and then you know, as Jake Relates uh spun off you know,
or sold off their software businesses to Aviva. So this product went to Aviva,
and very recently Schneider Electric bought Aviva back. So the product line has
has bounced into an out of Schneider Electric for a while. There's Schneider Electric
is is a behemoth. They've they've purchased a lot of stuff, including Aviva
recently. You know, it's a it's a truism of the industry that the
the industry is very fragmented there are you know, you ask, what is
what's the world's most best known I don't know relational database. Well it probably
Oracle, maybe SQL server, you know, maybe my sequel, which is
the free one that everyone uses under the hood of of you know big web
applications. Three of them. That's it. That's those those are sort of
your choices. Yeah. Yeah, there's other databases in the world, but
none of them have the market share that those three do. What's the top
industrial control system in the world. Don't know, highly fragmented market. Really
don't know what's the top. But you know, nowadays, because Schneider Electric
has bought so many other businesses, nobody knows what is the world's most popular
industrial control system. But we know that whatever it is, Schneider Electric probably
owns it. So that's that's the world we live in. Thanks for the
intro. I've as I said, I'm here in the same city. I've
been watching you folks for some time. I know you. I know you
know telvend and Oasis and now a Viva and Enterprise Skata. I know you
folks as pioneers in this space. I know you've been doing this this cybersecurity,
you know, in the product space for a very long time. You
know, I see you as leaders in the space from the very beginning,
Can you talk about I don't know if you want to go into the history,
but can you talk about the big what are you doing with you know,
cybersecurity in your in your product line right now? What approach are you
taking to that? Yeah, great question. Our approach is for sure not
one of complacence. You know, we we became you know, I mentioned
that my started back in the Unix versions. So when the big big switch
in the early nineties came where we went from Unix to NT, we did
it for many reasons, not least of which were the fact that it was
now just a single operating system to support the Unix flavors back then were different
enough that you know, it really posed some problems for us, but also
also the switch to NT, which was controversial back in the early nineties.
But we did it so that we could leverage active directory, curb barols,
authentication, and and other parts that the operating system would bring to bear for
us, so that we didn't have to and that has proved very smart in
hindsight. When we first started deploying NT systems with active directory, fully admit,
I don't think we did it right the first time, but we we
you know, learned more about active directory and started to use ATOM, the
Lightweight Directory Service or a d l DS that goes on top of active directory,
and that really helped our PSR our performance, stability and reliability. But
you know, prior to that, the security mindset at the time, I
think in the in our company and throughout the industry, even myself, was
you know, mostly security through obscurity. No one thought that a Unix server
behind locked doors and air gap for the Internet was a risk. And you
know, ironically today that actually might be true. Some old legacy Unich system
behind locked doors air gap from the Internet probably isn't really a risk anymore,
except it is, and I probably shouldn't have said that. But we we
now have a very robust security model that was ahead of its time back then,
but now is becoming the emerging standard in the ICs space. I've attended
a couple of Department of Homeland security working groups where they're looking at you know,
formalizing I guess or formalizing might not be the quite right word, but
they're essentially centralizing what would be the best practice for a topology and network topology.
And and it only took a couple of those sessions for me to realize
that, oh, they're kind of catching up to where we've been for several
years now. So we follow the Purdue model, you know, segmentation,
network segmentation, and you know, and you know, I could go on
and on about what our security model is, but our approach to security in
Aviva is interesting, I think. So as a product manager, I get
to decide what R and D does. Right, We have a fixed capacity
in terms of hours of development time we can do per year, and at
this time of year, frankly, here in November, we're looking at how
to spend it next year. One thing that I actually don't have much say
over is security. That capacity is sliced away from me, and it is
managed by a dedicated security team in Aviva who are constantly looking at the security
industry and the security landscape and are finding things to do and prioritizing them according
to a standardized score, and they decide what the R and D is going
to do with that percentage of the capacity. Then at the end of the
release cycle, when we're getting to release again, the security mindset in Aviva
is so prevalent that they actually have released veto Power on my product. So
if we don't meet the right security progressive so if we don't meet the right
security score, or if we haven't made sufficient progress, or if our internal
code scanning tools reveal a vulnerability that scores high enough, that's it. The
release is on hold until those things are resolved. And it is one area
where you know, there's always a little bit of give and take between the
business and R and D in terms of balancing priorities and capacities and pressures.
But this is one that is not off a debate, and I'm very happy
to have experts that are, you know, full time watching the the industry
and making sure that the product is as secure as it can be. I'm
you know, without trying to sound too immodest, my product moves a lot
of petroleum in the world, and it you know, I go to bed
with the surety that my product is not going to end up on the front
page of the newspaper in the following day. And I take that very seriously,
and we all do as well. It's a concerted effort the product group.
Obviously, when we put out our release, we package up our MSIs
and we're on But then that's the beginning of how our develop sorry, our
delivery group what they take it from there, and they deploy it in a
secure way, a secure process so that there's no chance of supply chain infiltration.
And then the last layers of security are the customers. And I think
this podcast is probably delved into that topic many times around the people, processes
and procedures that customers need to do to secure their system. You know,
I like to say that the system is only as secure as the last person
who touched it, and so it has to be a comprehensive and holistic approach
otherwise, you know, otherwise the cracks will form and then it's game over.
Andrew, I maybe I'm just making an obvious assumption here. I would
have assumed that in the kind of case that we're talking about here, management
tends to run the show. But it sounds to me, based on what
Jake is saying, that the developers have a lot more say in control in
this process. That is true, and it's it's not that unusual. I
mean, I've only worked in a handful of businesses in my career doing product
development, but where I've worked and a lot of this was sort of pre
cybersecurity. I'm thinking way back to the early nineteen nineties. You know,
when I joined Hewlett Packard pre security. You know, security wasn't the thing
back then, but quality was huge because we were producing control systems. And
you know, in Jake's case, the control system is controlling natural gas pipelines
and other infrastructure. In our case it was oil pipelines and power grids.
And you know, when we're developing the control system, we're developing new features.
We're adding tens hundreds, sometimes of thousands of lines of new code into
the product every release. Well, I'm sorry, people are huge. They
make mistakes. If you've added one hundred thousand lines of code, you've probably
added you know, five hundred or one thousand defects into the product as well,
and now you've got to go through and painfully clean them all out.
And so we had a quality decision making process that sounded analogous to what Jake
is talking about on the cybersecurity side. Yeah, the management team set the
goal it has to be, you know, this level of quality so that
we're not embarrassed when we release it, so that our customers don't you know,
scream blue murder because their control system is falling over dead every ten minutes.
And then it was up to the end. But you know, the
managers did not They weren't face down into the code all day long. The
engineers were, the software developers were, and you know, we were the
ones that had to say, okay, there's the bar that's been set for
quality. Have we met that bar yet? And if we came back and
said no, we're not there yet, didn't matter if we were late,
it didn't matter how much the manager screamed. They were not going to overrule
us because they knew that if they released, if they overruled us, the
business would be majorly embarrassed. Their next would be on the line. And
so yeah, the people who are close to the problem, you know,
very technically, staring at the security holes, staring at the quality defects,
they're the ones, you know, that have to assess whether you've met the
standard, you know, the requirement. Management can set the requirement, but
they generally don't want to vary that for schedule reasons because they're going to be
majorly embarrassed. So this is not that unusual. It's you know, it
sort of jives with my own experience in the space. A lot of people
in the supply chain world, you know, NERKSIP has standards saying are your
suppliers trustworthy? Did you buy software or hardware components from untrustworthy suppliers who might
have embedded a backdoor? That's not what you mentioned, you know, lots
of other people are talking about. Well, I embedded a library from a
trustworthy supplier in my product a year ago and released the product, and today
the vendor of that library has announced a vulnerability. Can I track that?
How do I get that out? That's another thing, you know, that's
sort of the s bomb topic, the software bill of materials topic that everyone's
talking about. Nobody is talking about what I see as the biggest problem,
which is the solar wind scenario, which is the bad guys get into your
system and tamper with the product under development. Yet that's the first thing you
mentioned, So I'm a little surprised. Can you go deeper on what you're
doing on supply chain and especially this the last element, how you you know,
sort of secure your development process? Yeah, yeah, supply chain it
can mean I think a lot of different things, So to take those in
random order as part of of our FSR Final Security Final Security Review. There
are a number of scans that get run against the codebase. One of them
is a third party library and open source attribution checks. So the open source
attribution checks are interesting because when I first learned about them, it included things
that I hadn't thought about. For example, we don't like to use open
source projects that aren't actively under development, so no abandon where Obviously we have
to attribute the open source licenses and our product, which we also do and
we know where they all are. But I thought that was interesting too that
only open source that are under active development, and then of course that code
gets scanned with our tools as well. The third party libraries is another thing
that is in the FSR process, and we have a hard rule that says
that third party libraries have to remain current and for exactly the reasons that you
mentioned. We have quite a few third party libraries in our product, and
if any vulnerability is found, it is up. We have an obligation to
our customers to take the updated third party library and spin it into a service
pack, well, bring it into the next service pack, and those rules
manifest for us in this way, which is that if you have third party
libraries that are not current. You have to update them when you release your
service pack. It's a non negotiable. Then. The last part of this
then is really I think driven by some of our customers who maybe have some
of the NERKSIP sensitivities, because not all of our customers do. And this
is you know, some people call it the double glove. Essentially, it's
how do I trust that the software, the MSI package, the ZIP file,
the VM image, whatever, how do I trust that I can accept
that into my secure clean zone if you will, I'm now the customer.
And because that's like a major vector for infection, right, is that I'm
now accepting some large piece of binary software through all of my firewalls and so
on. So some customers have taken an extreme approach where they don't want to
accept our vms and they don't want us to build their vms. So this
is where the double glove approach comes in, where instead of our project team
building vms, which is normal, you know, previously to this, we
used to bring hardware to our office here in Calgary, stage the customer's hardware,
put this, put the bespoke software on it, tear it all down
after fat ship at the site and reassemble it. With the advent of virtue
machines, that's basically gone completely by the wayside, and we're in that.
We regularly move VM images around, but some customers are saying, no,
we would rather that VM image be built from the ground up here in our
clean room, and we'd actually don't want you touching it at all. So
the double glove becomes you know, gloves behind glass, if you will,
where we sit there and watch the customer and instruct and mentor them and say,
you know, step them through the installation process that we would normally provide.
That's an extreme example. There was one customer that went even more extreme
to the point where we were sitting around with our lawyers scratching our heads like,
I'm not actually sure how we will ever get any software to you because
you've kind of closed every potential way in which we could deliver software to you.
I'm not quite sure how you would ever take it. So that was
an interesting negotiation as well. But yeah, it's an interesting concept in terms
of how to protect our code base. Obviously, we have code reviews so
with pool requests and so on, so they would have to they would have
to infiltrate you know, identity hack or something. Our developer, so Viva.
It has locked our environments down very very hard. We have multi factor
authentication for everything that we do. Some things are also behind VPNs as well.
You know, we take that very very seriously obviously, because if we
were to have been breached, then you know, it puts a lot of
I don't even want it. I'm coming out in hives just thinking about it,
but we would have a we have an instant in response that would kick
in at that point. We've I think we've only had to do it once
and it was a it was a it ended up being nothing, but it
boy, it was. It was a panic. That was quite a few
years ago now though I want to assure your listeners that it was a bit
of a false alarm. But since then, you know, we we'd take
that very seriously. H and we have regularly responded to the S bomb kind
of questions in RFPs and with customers. We have a very open relationship with
our customers with regards to security. So if they do a penetration test,
they want to know that we are, you know, going to be interested
in their results. And of course we are happy to say that the penetration
tests that our customers are performing are not turning up anything, or if they're
they're turning up some minor things that we're like, yep, that's safe to
ignore. That's that's reasons that you Yep, you can turn that off,
that kind of thing. And so, like I said at the very beginning,
it's a constant evolutionary process here where every time we put out a release,
we're always updating our GPOs from the Center of Internet Security and so on.
But I think I'm now straying off the topic of the of the supply
chain question. So yeah, suffice to say, it starts like I said,
it starts from the bottom layer, how we manage our code, how
we access our code, how we accept changes, all the way to how
we actually get binaries to cite for the customer. And this is just the
Aviva on prem experience, right. The cloud is a is a different beast
and it but it has the same kind of security oversight and more because because
of the nature of cloud. The point that you made in your question there
and then Jake's response are kind of interesting. I mean, the subject of
supply chain security is not new to our podcast. If I recall in Not
So Far Away episodes, we've been talking about s bomb a few times.
S bomb is just a way to account for what the heck kind of software
you're dealing with. And it seems like the point that Jake was making there,
among among others, is that maybe beyond just knowing what's in your product,
using only the kinds of software that you can hold to account that's continuously
updated, so that you know that all the components of your product are ultimately
just as securable and enforceable as all the others. Yes, the the the
thing is that you know, in my recollection supply chain is is like four
different things. Three of them in a sense are verifiable. The vendor can
prove to the customer that they've done it right, and the fourth one is
just hard. You know, the three that are verifiable are you know things
like did you buy your components hardware and software from trustworthy sources or you know,
did you buy them from band sources? Well, you can look at
the components, you can see the labels on them, you can look at
your contracts, you can you know, in the worst case, bring a
lawyer into review the contracts under non disclosure and prove that you purchase your stuff
from you know, trustworthy sources. Another sort of gotcha is did you buy
you know, even if you the stuff was manufactured by somebody trustworthy, did
you buy it from an intermediary who is criminal, who is you know,
taking some of the profits and funding terrorism or something horrible like this? And
again, you know you can prove with your contracts and with your paper trail
that you haven't done this. The third one is you're your your you know,
are there vulnerabilities in the libraries that you've used? And there are tools
that can scan the product that can figure out which libraries you've used and which
versions. They can verify the customer can verify that you know what you've advertised
in terms of your libraries and versions are the ones that are in the product,
can go and look to the cve the vulnerability database and prove to themselves
that none of these libraries have known vulnerabilities. You can prove all of this.
The thing that you can't prove is what I asked about, which is
uh and in a sense was Jake's first answer, that the thing that you
can't prove is that you know, the bad guys haven't snuck a sleeper,
a terrorist or you know, a spy into your development organization who is inserting
malware into the product as one of the developers, you know, how do
you prove that hasn't happened. That's really hard and you know what you have
to do to deal with that risk. You just have to be really paranoi
from one end of your development process to the other. It's just it's just
hard. And yet that's exactly the behavior. That's exactly the attitude that Jake
has described here. So you know, these folks have been doing this for
a long time. You know, they've wrapped their heads around the degree of
paranoia you need in your development process to assure that you know with a high
degree of confidence that the bad guys aren't sneaking something in under the hood,
so you know, good on them. So that's a lot. I mean,
it's you know, reassuring to hear a vendor with with you know,
such a what's the right word, A broad approach to cybersecurity and the product.
You know, we're again, you're the leading provider for at least natural
gas pipeline control systems and you know, active in lots of other space.
But when we talk, when we say the word pipeline, you know,
the elephant in the room is the Colonial incident. Do you want to talk
about that? What was sort of the the consequences of that of that incident
for for Aviva and for the whole industry. Yeah, you know, we
had the skated director from Colonial join us on stage at our recent pipeline summit
here in Calgary. I hosted a cybersecurity panel and it was the second time
we had done it. And uh, mister Warrenburger from Colonial, he he
had joined me previously last year in San Francisco at the Aviva World Conference and
for the same thing, a cybersecurity panel for our midstream user group, you
know, and when he when he agreed to volunteer for that panel, the
first question I asked him is, oh, are you sure? Do you
want to get up in front of everybody? And he's one hundred percent.
And the first thing he says is, We're sorry, but you're also welcome.
You know, we we kind of forced us all to become secure and
and you should you should be taking this seriously, and of course we all
are. I don't know you know, and of course you know, this
limits to what I can talk about, and there's limits to what I know
about what happened at Colonial, But our understanding is that the enterprise gated system
wasn't compromised and that the shutdown of the pipe was due to an abundance of
caution. You know, our system being air gap from the corporate network and
designed to operate independently, and everything is all well and good until your entire
business operations depend on applications that are not in the control room and cannot be
air gapped. So so that's you know, that's an interesting thing that the
industry is grappling with right now is how to how to survive an ongoing cyber
incident and not be fine for shutting down, which was sort of the Colonial
takeaway that surprised me. So that's very interesting is how resilient do we need
to be? How how does your disaster recovery or business continuity which is slightly
different, how does your business continuity now change given the idea that maybe you
can't you can't shut down now. Colonial may have been able to go to
manual operations, but some of our larger customers there there just might not be
enough people to send out to the field to operate manually right and in fact,
do you even have h a you know, when was the last time
you tested your manual operation procedure? So so these are these are some of
the interesting lessons learned. And like I said, you know, being very
transparent about this and and the corrective actions that we're making. It's it's extremely
important for the industry to to share this kind of knowledge back and forth.
So so as a result, the TSA rule came out that had a bunch
of guidelines and so on, and and what we were uh so we struck
a team, struck a little committee that met daily to discuss the progression of
these rules and to understand how are our customers are going to be impacted.
Happy to report that because of our topology deposit, because of our design and
our approach that I've detailed already, our customers didn't have to do much.
One thing that they did have to do was to cycle their system passwords,
which for the older versions of our product was a little bit labor intensive perhaps
and a little bit risky. But our but our technical support team was able
to to work with our customers to get those passwords rotated without causing downtime.
The later versions of our product we leveraged group Managed Service Accounts, which is
an active directory Microsoft Windows feature that rotates these system passwords automatically for you.
So so going forward again, our customers have to do nothing there to comply
with that rule. However, one other major change for me that affected me
as product manager here was was the how our products interacted with the various third
party security tools. So previously we had been quite prescriptive. It's a long
story, but we had got into the situation where we were uh in testing,
integrating, and certifying one third party security tool, and overwhelmingly our customers
said that, well, we don't want to use that tool because our IT
department is forcing us to use this other tool. And so just a note
on that IT OT convergence. Sometimes it's a swear word, sometimes it's the
answer to your problem. In this case, you know, the guidance that
we've been giving here is don't fight your IT department when picking a security tool.
You have to work with them because ultimately you need a holistic response to
the to the entire operations of your business, which includes IT and OT so
it is in your best interest to have an overarching response to this. It
doesn't have to be a single tool. You don't want to violate any you
know, any network security rules or guidelines or best practices. But to have
a common response and perhaps a common tool, if not just a single instance
of that tool, we think is probably the better way to go. And
so as a result, we announced that we were not going to be endorsing
any individual tools, nor were we going to be testing them, because there's
obviously too many and we can't test them all. So instead we pivoted and
we documented in great detail the elements of our product that you need to know
about when you're shopping for, configuring, testing, and operating a third party
security tool. So I'm talking anti virus, I'm talking allow listing, I'm
talking multi factor authentication, I'm talking host firewalls. So there's several chapters now
in our administration guide that step you through what you need to know about our
product so that you can pick those third party tools and then work with your
I T department to consolidate and collect aborate on on the tools, and then
the overarching processes that you need to be you know, to be safe and
to sleep well at night. So Nate, let me add just a bit
of background here. I mean, back in the in the early days of
industrial cybersecurity, I was working for a control system vendor, one of Jake's
competitors. But every vendor in the industry was facing the same problems, the
same demands from customers, the same sort of changing landscape. You know.
One of the big issues back then was that the customers were demanding that the
vendors support the customer's anti virus system of choice, the customer's whitelisting vendor of
choice, the customer's file system change tracking vendor of choice, the files you
know, the host firewall for whatever host, uh, the customers wanting the
software on, you know, any kind of network firewall the customer chose.
Because, of course, you know, enterprise security teams were dictating security choices
company wide, and they would dictate to the you know, the ot folks,
the engineering teams. You know, you want to use an antivirus,
you have to use this one. It's the company standard, which meant that
the control system vendors had to support everything. You know, back in the
day, the control system vendors were told you have to support antivirus. So
every one of us picked and one anti virus vendor, you know, and
you have to support firewalls. So we picked one firewall vendor, and we
tested our stuff exhaustively against that one vendor's products, and we documented our stuff
for that one vendors product so that the customer could get some some security going.
And the customers came back and said, no, we don't want your
vendor. We've already standardized on this other vendor. But you know, if
the control system vendors, you know, if we had to support everything,
there was enormous costs. I mean, were we supposed to buy one of
each antivirus? I mean buying the antivirus wasn't the cost. The cost was
testing against all of the anti virus vendors, all of the antivirus systems,
to make sure that nothing malfunctioned. You know, there were there were malfunctions.
I mean, if you run a full anti virus scan, everything slows
down and stops. And you can't do that with you know, a power
plant or a pipeline. You know, were we supposed to test our stuff
with one of every kind or all of every kind of control system? Security
potential product on the market, all the filesystem changed, tracking vendors, all
the white listing vendors, all the different firewalls, and if we get a
support called what are we supposed to do? You know, the engineers on
the other end of the line did not know how to operate the security technology.
Nine times out of ten we had to teach them how to operate therecurity
stuff because they hadn't taken training. You know, enterprise firewalls, you might
be used to doing a little bit of you know if if you have to
do something tricky on your home firewall, well it's got six screens, it's
not that hard. I'm sorry, enterprise grade firewalls, you need to take
training to figure out how to use this morass of screens too. So it
was a real problem back then, and over time, you know, everyone
had to change. The vendors had to change. Of Viva was one of
the leaders in you know, leading change in that space. But the customers
had to change. They had to learn, you know, the enterprise the
engineering teams had to learn that, they had to take training. You know,
the vendors had to learn that. We had to support everything, We
had to document everything, so that you could use the firewall of your choice
because we documented what ports you need. You know, the vendors had to
take training on how to operate their security gear. They had to take training
on how to test their security gear so that they didn't call us and say,
oh, your stuff is broken, when in fact they'd fumble fingered the
firewall configuration. And you know, all of us had to learn to cut
each other a bit of slack. You know, if the engineering team had
taken the training and still had a problem and we had tested the stuff and
our stuff still wasn't working, well, well, you know we had to
come together. So it was. It was a difficult time. Today,
you know, the leading vendors in the space support a lot more than they
used to back in the day. Maybe not everything. Everyone's learned to make
a few compromises, but it was it was a difficult period for a number
of years as we figured this out. Can I ask you one detail you
haven't haven't really touched on? The security directors out of the TSA talked about
shared trusts and about documenting these things. You know, in my understanding,
shared trust is code for active directory on the IT side. In a sense,
you know, controlling or you know, being having the power to create
users and manage permissions on the OT side, Can you talk about shared trusts
in the enterprise skate of product? Sure, yeah, we were. I
was really excited personally for the idea of single sign on from the IT domain
into the OT domain. It seemed like such an obvious user experience improvement.
But you know, to skip to the end, I would say that,
you know, trusts between domains is now not best practice, right, But
if I was to back up a little bit, you know, using using
active directory has I think we're largely over it at this point. But when
we first started to deploy active directory, you know, the IT OT relationship
with the customer would really come to bear. Uh, it's active directory,
therefore it should be in charge of it, right and in charge of that
domain. Now, I don't want to start like a debate amongst your listeners,
but you know, there clearly is ways in which you can bring it
into your OT zone to do this management. And obviously you know and take
advantage of the fact that they've got all the extra stuff. They may have
DBAs on stuff and so on, but you do need to give them that
OT training. They do need to become aware of the differences between OT and
I. So when we started using IT systems like Active Directory, you know,
we had to politely but firmly insist that they do not put our product
into an IT domain. We don't have that. We don't have that debate
anymore. And now in terms of you know, and we were also at
the time, you know, really that many firewalls, that many different domains,
really that seems like overkill. We we don't. We don't hear that
complaint anymore, you know now that it's become best practiced, so uh,
you know, trusts were seen as a great way of giving corporate users access
to the Decision Support System, which is that read only skater, a system
that sits in the DMZ between O T and I T. You know,
we thought, and it's still true, that it removes the burden from the
SKATE administrators for things like accounts and password resets. And then with single sign
on access to the historian in the d s S, they have access to
all the historical data they could ever want. However, you know that the
the the user persona of who uses the d s S is changing, and
I'm happy to talk about that further if you'd like, and and all.
But but you know, the the net net of all of this is that
without a trust between your I T domain and your OT even the OT D
S S domain, there is no single sign on. So the d S
S now is essentially out of reach from your your core users outside of you
know, pre defined reports and perhaps a well other things that we have in
our product, like a remote HMI that you can peer into it. So,
yeah, the future of the DSS is is an interesting one as a
result of the security landscape changing. Just a quick clarification, you've used the
word DSS a number of times. Decision support system. Is that sort of
the new branding or a superset of functionality sort of around the historian or is
the historian a different animal than DSS, So certainly a historian would be in
the DSS. The DSS Decision Support system is a you know, not to
be repetitive, but it is a system to help people make decisions. So
for us, this is a replica of the control system that is in the
secure zone, right, so the main scate system that the operators are using
to send commands to the field. We've implicate that into the DMZ network zone,
and we call it the DSS. The DSS has no abilities to send
commands to the field, so it's a read only system, but it contains
both historical data and real time data. So our product has a real time
side and on a historical side, our real time side is obviously what brings
back the data from the field and shows it to the operator and then allows
the commands to be sent. The SKATA administrator will configure which of those points
in the real time servers need to be historized, and so we will historize
that data into a smaller historian in the secure zone purely for trending and for
operator trending. Then all of that data is also sent to the DSS,
along with all the real time data, and the historian in the DSS will
then usually contain a lot more data, a lot older data. But now
and also now we're seeing that there is another historian outside of that zone,
in the corporate zone, and you know, Aviva having purchased OSIsoft, we're
seeing we're seeing PIE. I mean, we would recommend PIE and putting Pie
outside of the ot zones, be it Zone three or three point five to
use the produe nomenclature. Putting pie outside of all of that means it can
be the destination for all of your corporate data, not just data from the
field. So, getting back to the question, DSS, Yes, it
contains a historian, but it also contains a read only replica of the real
time allowing non operators to see operator screens without without the ability to actually do
anything other than navigate and see data. So the DSS then, because it's
was designed at the time with a trust for the corporate users, any corporate
user then could essentially pretend to be a skata operator with the exception of being
able to send commands and change configuration or whatever, just read only. But
because the trusts have gone away, the DSS now is inaccessible to them.
So we are needing to find a different solution for the DSS to make it
to get it back to what it needs to be to help people outside the
control room make decisions. Interesting. I mean, some of the features you
talk about, you know, giving the the anyone with access to the DSS
on the outside on the enterprise network, giving them the ability to see the
same screens that the operators would have seen if they if they'd clicked through to
them this. Uh, you know, I've heard the word digital twin.
You know, I've heard the word digital twin usually applied to a system in
the cloud that in some sense emulates the control system, you know, in
the in the OT network, or the physical process in the OT network.
Can you talk about you know, is this a digital twin? And can
you talk about the cloud? What? What is the future of the cloud.
Are we talking about operating the pipeline from the cloud? What? What?
What does the cloud mean security wise? Yeah? Straight for the jugular,
Yeah, skated in the cloud. So it used to be that making
reference to cloud in my user's presence would have me politely but firmly shown the
door. Time marches on and the cloud is not as scary as it once
was. The cloud, remember, is just somebody else's computer. So with
our product as it is today on prem on premise, you know, sometimes
the computers are not belonging to the O T. They're belong to the IT.
Sometimes the data center is and a different building across town that is a
cloud. It's just your private cloud. So I think that the concept of
understanding, you know, the risks and and so on of on prem software
versus the cloud. We're seeing a shift, so definitely digital twin. You
know, it's it's a bit buzzworthy, of course, but the DSS is
essentially like a twenty year old version perhaps of the digital Twins. So yet,
just like you say, the digital twin is supposed to be a virtual
representation of the entire asset. So I think, for example, that the
digital twin will have a big role to play in how we manage and reduce
the cost of point to point checkouts. If you were able to show to
a regulator that you have paperwork all the way from instrument to eyeball, then
that you can prove that you know where the changes have been made. You
ought to then be able to reduce the amount of effort it takes to do
a P two P a point to point checkout, and then that information,
you know, should be stored in a digital twin. But digital Twin is
a lot more than that, and Aviva is definitely investing heavily in this concept.
Aviva wants to be a cloud first SaaS business, but that would see
to be at odds with uh, you know the world's leading oil and gas
control system. So what what I have done then is we're gonna We're gonna
put our toe in the water with d s S in the cloud. So
with ds AS being almost useless now to non control room people, Aviva brings
to bear a bunch of products that replace what the on premise d s S
used to do and still does for those that are using which is everybody.
So products like Aviva Insight, which which allows you know, ad hoc user
access to analytics UH and UH and dashboarding, Aviva Reports which is Dream Reports
UH rebranded, and then Aviva Teamwork, which is a workforce workforce automation tool.
These are just the first three products that I think we want to bring
to the midstream industry, and all of it would be back by Aviva Data
Hub, which is the OSI soft Pie technology. But in our Aviva Cloud.
There are other regions in the world that are actually a little bit more
progressive with their thinking about cloud. I have several Latin American customers that are
thinking about backup as a service. So a backup control center would know only
be like a direct replica of your primary control center with you know, building
cooling, ups, rax power, internet supply, all of the rest of
it. They're there because of they're a little bit on the smaller side.
They're interested in in the in shifting that backup server to the cloud in order
to reduce their operating costs, and so I'm actively looking into what it would
take too to make it easier to run enterprice KATA in the cloud. But
I am not advocating for North americ and oil and gas customers to put their
primary control center in the cloud. In fact, I'm not quite sure what
the future will bring. I have some visions that I'm working on in terms
of the next twenty years or ten years for enterprise SKATA and the cloud,
but suffice to say, we will always have to have some sort of fallback
to on prem I think, if no other reason, then that's usually where
all of your communications infrastructure is, you know, originating from. But I
can see a future where the cloud has demonstrated its stability and security and reliability
to the point where there are customers that are are happy to run their SCAT
system in the cloud. Now, having said that, there we have many
SKA customers that are running our product in an infrastructure as a service, which
is a form of cloud. Right. The difference though, is that that's
still just vms I lift and shift approach. So so, yeah, the
cloud is the cloud is coming for sure. And you know, and even
within North America, we have users that are saying, we want to know
how you're going to move skated to the cloud, and then we have other
customers which are saying, don't say cloud in my presence, and then we
have some in the middle. And I would say that second one very few
now. In fact, at our pipeline summit, I mentioned it already,
but I'll say another anecdote from that event was someone come up to me and
say, yeah, your DSS in the cloud presentation was very interesting, but
I just don't think we'd ever be able to do it. And I was
getting ready for the usual list of reasons why we couldn't go to the cloud,
right security, latency, data, privacy, data, sovereignty, these
kinds of things, but instead the customer surprised me with the actual reason why
they would probably be hesitant to go to the cloud, which is that it
would require having to run proxies within the IT DMZ and they don't have a
healthy relationship with their IT and that really threw me for a loop. Right,
like Like I said, I was expecting a whole bunch of other pushback
as to why we couldn't move to the cloud. But the real reason is
because they don't have a healthy relationship with their I T department, right,
And the reasons behind that, I hope are clear. And obviously we have
several firewalls to transverse to get to the cloud and back, and so we
need proxies and and secure proxies and other things to live in these other network
zones outside of OT. I was kind of shocked to hear that that there
could be customers out there. These were largely and this was a large customer
too, that that still have such an unhealthy relationship with their I T department.
That to me is as alarming and as an industry, I think we
ought to be trying to close that gap somehow. Jake said a lot there,
but it sounds like maybe the overall point is that the cloud is complicated
or maybe just too complicated for me. Well, he did make the point
that, you know, the end of tree, the customers who are using
these control systems seem to be all over the map. You know, some
are saying over my dead body, read my lips you know, some are
saying, you know, let's do this. Everybody else is somewhere in between,
you know, and it is complicated, you know, relationship wise with
it. Reliability wise, Is the Internet reliable enough to do a cloud based
control system? Security wise? You know, is it wise to have your
control system that exposed to the internet by operating across the internet? You know,
something new that I heard in his answer that I'm still thinking about is
the possibility of a backup control center in the cloud, because you know,
these control centers, the physical buildings with you know, wiring coming into them,
computers throughout that sit there basically idle through the entire possible sometimes through the
entire life of the facility unless you're unless you're testing the backup system a big
investment. And if you can host your backup in the cloud, you know,
in the life of a facility, you might never use it. Does
you know, are you exposed? Can you design the backups so that,
you know, a cloud based backup so that you're not exposed to the security
problems unless you switch over, and you might never switch over. These are
all to me, these are all interesting questions that you know, I'd have
to think about that the idea of a backup in the cloud security wise,
what does that mean? You know, is something new and you know something
I'm certainly going to be going to be thinking about going forward. Thank you
for joining us. Before we let you go, can you sum up for
us what are the most important you know, lessons that that you think we
should be we should be taking away about, you know, the product security
and especially product security in you know, the your perspective, the way that
you folks do it. Sure, yeah, definitely. Security is as a
layered approach. Security and depth is essential, and it starts before you've even
written a single line of code, through your design, and then all the
way through deployment and then the the you know, the the last person who
touched it. So it is a collective exercise. It starts years before you
need it, and you have to invest in it, and you have to
continuously invest in it. You know. I mentioned how we pivoted from our
guidance on third party security tools. One of the things that we mentioned in
there is, you know, be sure to understand the resource requirements in terms
of human staffing, because if you'd buy a third party tool, test it
and deploy it, and then never check it. It's not there. It's
not doing anything for you. If you're not actively looking at the results and
chasing down the phone positives and and and so on and constantly improving that you're
you're not progressing. You know, staying still is moving backwards in the security
I think, So you have to keep on top of it. And then
I would say, you know, really, in terms of cloud, you
know, this is my chance, I guess to to uh to talk a
little bit about where we want to go with the product into the future.
Is I guess, you know, maybe introspectively look at some of the prejudices
that you have about cloud and really and really ask yourselves the kind of questions
that you're going to get for me if you challenge me in person, right,
which is latency and data privacy, data security, so you know,
the data security one. It's like, do you think that you have more
people on your security task force than Aviva does, because I can tell you
we have quite a few people looking at you know, at DevOps and and
the security landscape, as I've mentioned many times. So it's you know,
maybe have that have that introspection and challenge some of your internal prejudices. But
you know, security, like I said, it's it's extremely important, and
it's a it's a group effort. It needs to be a collective effort.
Uh and uh and yeah. And if you want to know more about how
Aviva is keeping the world secure, I guess is to reach out to your
account manager if you're already a customer, or hit me up on LinkedIn and
I'd be happy to start this discussion with you and put you in charge and
put you in touch with people who can who can you know, continue this
discussion with you. We're only a software vendor. You need to be having
this discussion with all of your vendors, your POC vendors, your your your
payroll vendor and so on. It's like it's it's no point locking only one
door of your house, right, you have to look at all of your
doors and that starts with even finding them. There are there are companies out
there that that will help you even just understand what your security footprint is before
you even start figuring out how to secure it. So so yeah, I
think it. But you know what, we have a great industry, lots
of fantastic people like for yourself that are promoting these kinds of security concepts.
It's extremely important that we all get on board and do have those conversations with
your IT team. Try to make them friends instead of enemies. No back
to back firewalls because you don't trust the other guys firewall right, and I
have seen that multiple times, which is you know, sad, but there
it is. But yeah, you know, our product is secure. Come
and have a look. We have our Aviva conference next year. We're going
to do our pipeline summit again in Calgary, I think again, so watch
out for that. And yeah, hit me up on LinkedIn and let's take
this conversation deeper. I want to know more about why we don't want to
go to cloud because I need I need to start formalizing a strategy for that.
So yeah, very interested. Thanks for having me on, Andrew.
Andrew, that was the conclusion of your interview with Jake Hawks. Do you
have anything else you'd like to take today? Yeah, I mean I was
impressed. I asked hard questions and I heard a lot of the right answers.
I mean, you know, deep transparent documentation so people can make informed
decisions about, you know, using the security tools of their choice. This
is, this is the right answer. You know, vendors used to push
back on this, and a Viva isn't anymore. You know, a security
budget for the development team is sounds really interesting. This is you know,
it sounds like the right answer. You know, if you don't have that,
the push for features, the push for schedule tends to muscle out security
investments and you can't afford to do that. So you know, you give
that decision making authority over to the development team. You take it out of
the hands of management in a sense deliberately because management wants security as well.
You know, paranoia is the right answer to assure the integrity of the development
process, you know. And he's right, you know, Aviva, he
at Aviva looks at one thing, the product, but you know his point
that that owners and operators have to have this security conversation with all of their
vendors, with all of their teams, with with their IT teams and their
engineering teams day. You know, it's it's a big picture and we all
need to be you know, talking to each other and doing the right thing.
So you know, again I'm very impressed. Well, thank you to
jay Hawks for all of that and Andrews always thank you for speaking with me.
It's always a pleasure. Thank you, Nate. This has been the
Industrial Security podcast from Waterfall. Thanks to everyone out there listening.