Hey, what's going on? Y'all? Welcome to another episode of Adventures in
dev Ops and joining me in the studio today my new routine co host,
Warren Parade. What's going on? Warren? Hey? Thanks? Well,
you know, at some point that's going to get old and I'll be the
old co host and we'll have to come up with a new tagline for me.
I know, I was just thinking that and I was like, damn,
I didn't come up with anything new to say, so I had to
go with the old, stale one. I'll work on. I'll work on
that before next week's episode. I promise you didn't have enough homework to do.
You've got one more thing now, right, right, It's good to
keep busy. But also joining us today we have the co founder and CTO
A Hero, Jonathan Elder. Welcome Jonathan, thanks for having me. Hey,
I'm excited to have you here, and I'm looking forward to this episode
because I've got two security experts here in the studio with me, and I
think security is one of those really really, it's a really odd specialty to
have because it's one that I feel like from a business perspective, like businesses,
let's just be honest. Here, businesses tend to prioritize customer facing features
and cheer about those much more than they do security work. But the end
customers themselves, I don't think they ask for Like the end users of my
applications don't ask for security related stuff. They just assume that I'm doing it.
And so from a developer's perspective, it's really hard to feel like security
gets the prioritization that it needs because everyone, I think everyone just assumes that
you're doing that, because that's what you do as a good steward of your
code. And so I'm interested to talk about that and how y'all view that
and get some actionable tips that help us all bring security up and like make
it the first class citizen that it should be. So, with that rambling
intro out of the way, Jonathan, how did you get to this point
where you're the co founder and CTO of a bureau? That's a long road,
Like professionally it's been twenty years starting in the idea in Israel, but
even before that, like there are many developers who are you know, tinkers
to like pick apart things, figure out how it works, trying to reverse
engineer stuff, whether it's you know, toasters or applications or back in the
day like I don't know, Prince of Persia or whatever, trying to figure
out how stuff works. So I think this was the start for me.
I've been working on many startups since then, quite a few, and it
took me twenty years to or maybe twenty minus five years since we started a
bureau to start as a founder with the down plotting my partner who's the CEO.
After we work together on a totally different thing, which is also security,
but in a different field altogether, because security is really really broad,
you know. Yeah, for sure, I think it's one of those that
it's easy to to get specialized in a specialized specialty. When you talk about
security. Yeah, there's there's a lot of niche like niches in in security
now now people are saying cybers like. There are a lot of terms.
There are people who are you know, working on ed R, people who
are working like in pentests, people who are working on AppSec for thirty years.
People. There are various fields that are everything is related, but there
are many specialties in the industry. So there are communalities like the cat and
mouse game with like good people versus bad people. It happens a lot,
and it's it's like it takes different shapes in the net work security domain where
it done. And myself we worked back in the day in Microsoft and UH
and in in other places, but also in apsec, and also in in
IoT and in very like diverse like sub cyber terms right on. And so
with a hero, you're working to to provide visibility and and actionable context to
the consumers of your platform. What does that what does that actually mean?
What kind of things do you help help someone do or accomplish? Uh,
there's a there's an evolution in AppSec, in application security. UH. It
started with like very like technology related products. There was at first there was
you know, SASS finding vulnerabilities in your code and it and it evolved,
and there there came os S security, s c A finding vulnerabilities in open
source, and then software supply chain security that finding stuff like this is more
develops proper right if the GitHub configuration allows you to do such and such as
an example and various other fields, right, secret detection, et cetera.
So it all evolved and now it's like got a proper abbreviation, a SPM
Application Security pasture management, which like takes everything into one single like UI or
one single source of truth where you can see what goes on in your applications
and it Uh, we've come a long way uh for Gartner to coin that
term and for companies to understand that having seven different like specialized tools is not
a good way to understand what goes on. There's a lot of reasons,
but I think that the main one is you don't have context. Right.
You have thirty thousand alerts, twenty thousand came from that SAS tool, five
thousand came from that OSS tool. And now what I have three people in
my A team covering thirteen thousand developers, We have twelve hundred poor requests a
day. What do I do so to answer your question like this is the
context maybe to answer your question like we look at it in like three layers.
First is visibility, knowing what you have. I always say, you'll
be amazed how developers, CTOs, CEO, CIOs, AppSec people they don't
have a clue what they have. As long as you have more than maybe
ten fifteen twenty developers, not to mention one thousand or thirty thousand, you
don't know. I don't know which database is, what technology is, how
many platforms for oh off you are using? How many languages are all this
you can't fathom too much. So you can go from there to what APIs?
Do I have? Do I have graph quel even? I don't know?
Does it go through some engeniques? What's the technology that says don't go,
go, no go? I don't know? So first visibility, like
what do I have? We call it an inventory? Right. The second
phase is, as I said, you have thirty thousand alerts, findings,
whatever. So prioritize. Tell me what should I focus on right now?
What will bring most value to my company? What will stop and attack,
what will be worth my time, what will save us the most money,
et cetera. So prioritization is the second thing after that remediation. And after
remediation, like help me find who's the right person to talk to because I
have again thirteen how much did I say, thirteen thousand developers? Who's the
right person who can say this API should actually go to the internet and download
such and such other Who I can say should I go and ask? So
this is remediation, and after that, like the advanced customers of a PIRO
do more of managing and prevention for such things, so they employ workflows that
automatically whenever a certain risk goes above a certain threshold, then they will start
a certain trigger, maybe appentest, maybe a code review. So this is
how we help people, like three people versus thirteen thousand people maybe try to
get a hold of application security right on. Yeah, So a big part
of that is just, well, like you said, just set in context,
like taking all of this information and figuring out what parts of it's relevant,
which parts are urgent, and just trying to turn it into something actionable
so you don't feel like you're overwhelmed and destined to never succeed. Yeah,
it's really like I think personally, one of the most frustrating things that I
faced in that context is people saying shift left, shift left, which is
like, it's a nice term, I get it, but if it says
all the burden on me, the developer or the develops engineer, then someone
else do the job, Well, why should I be the one to handle
everything? Right? So shift left is really important, but you have to
do it right. If you have six thousand findings from your automat like automated
certain process of static analysis, and you're shifting it left. You have to
really chew it up and spit out only the three, the five, whatever
actionable findings that I can handle. If you'll give me one hundred, it
will be deprioritized in my I have features to develop. But if you'll say
these certain vulnerabilities, it might have false positives, it's fine. But if
it's two out of three, I'm fine with it because I see the value.
I can see I'm helping my company be more productive. But if these
three a let's say vulnerabilities are in APIs that I know that are not in
test code. They are in some application that is facing the internet, right,
so it's exploitable. Perhaps maybe not, but it has a certain like
it might be exploitable, and I know the code right, it's like people
from my team has written the code. And more than that, I'm a
security champion, right because my team has one hundred developers. But the apps
gave that task to me because they know that I'm like well versed in security
stuff. Right. In that case, go on shift left. I will
make it right. If you have a false positive, I'll help you be
better. But I will fix it and I will accept the the two out
of three false positives. I got a hypothetical for you here, So like,
I know that there are these other platforms out there that are compliance driven
checklist aggregators right there for SoC two or ISO, and they collect a lot
of things from individual sources within your your company, your engineering organization to help
you achieve passing some sort of certification or regulation. But it feels like your
company, uh actually focuses on the value of security there right like actually,
rather than some sort of checkbox approach. It really is these things are important
to solve and not just to achieve a particular In my that's a great question.
As my head is a CTO. I have to go through sock to
type two. Of course I have to do that, and it's important too
write right like show our customers that we take it seriously, even if some
questions in those compliance frameworks are kind of mundane and seem like that they didn't
adapt to twenty twenty four maybe, but these two are important. So in
general, we do have like a module for questionnaires. There are a lot
by the way, for I'm not sure from the audience here, what's the
average size of a company that they work for, from independent contractors to one
of thirteen thousand developers maybe or thirty thousand. But for the larger companies,
they do have a lot of compliance going on, sock to type, sock
to ISO, and also like they have all sorts of controls for you don't
you can't change a certain area of the code base without answering like two hundred
questions. So this is an area that we also help our customers and do
it smart. Like if we can automatically answer, as you said, or
thirty seventy sixty percent of the questions because they are code related or developer related,
we can easily answer that. But for the rest of the forty percent
of the questions you can go and answer. Then you can automate the process.
I say, I mean, and I'm just really intrigued here, Like
it sounds like you could almost utilize a pro to replace whatever automation platform you
have for soalk two findings and utilize that to actually achieve it is that accurate
well talk to like let's say it might require you to present that you have
some sort of like a rule set of how you do back up and restore,
so this is an area that we won't help you with that. You'll
still you still also have to go through I don't know, some sort of
an audit vendor that will the stamp you have to do that. I won't
tell. I won't save you from that one. But for the real answers
like that are expensive in terms of time and money to together, right uh
uh. They will ask you show me evidence that developers, maybe statistical evidence,
but that developers are going through such and such process in order to to
trigger some uh some other let's say paintest or whatever. Okay, and all
of these things, like many of these things we can derive from what we
analyze. Because we analyze every commit in your guitar, bit bucket, git
lab, whatever you use. Perforce. We analyze every commit, every ProQuest,
every Gerra ticket, as a DevOps ticket, whatever. There's a lot
of data there. And the last thing I want like our customers to do
is go through tedious like uh like exports of things and go with their eyes
over a thousands of rows and figure out stuff by themselves. We want to
help as much as we can, hope I answer the question, yeah,
No, I mean you said something really interesting to me, and I think
maybe interesting for the listeners as well as like where do you hook into the
process? And I got this idea, maybe there's an integration that happens if
I change some database through terror form that I'm already a learned that this could
have security applications. That is that accurate? Yes? Yeah, so so
if you if you go let's say let's say that you're an advanced shop,
right and you have everything via their form. Okay, so this is a
good case. Uh we uh at at least hook up into your source control
manager. Okay. If you're more advanced, we will hook into your gra
maybe your Kubernetes environment in a w S or Google or asure. If you're
more advanced than that, then to your API gateway, a PI security tool,
and and and maybe your security scanner. Maybe you have sneak, maybe
you have menned, maybe you have like dozens of others. Right. Uh,
And we take all that data and crunch it up together. But at
the very minimum, we analyze your code. You can opt out, you
can choose what code to analyze and whatnot of it. But assuming that you
have terror form, then yes we will alert if let's say a certain application
shouldn't be connected to the internet, and maybe you hook it up into an
ingress in GK whatever that is connected to the internet, then you can automate
that process and put a guardrail. Right. So I should have mentioned maybe
before that we have like a concept that we called governance, which by default
it comes with a certain rule set, but you're free to change, to
delete rules, to add rules, to tailor it to your specific needs.
But you can say, you can define what's to you, what's a material
change, as we say, right, if someone added an API, it
might not be material to you. But if someone add then API that is
connected to the internet and it is exposing a certain type of data, then
maybe this is dangerous because whatever in your realm, this is what happens,
right, And then you can say, let's define a workflow. If such
and such happens, then open a ticket block. The PR don't allow a
developer to change the terraform module X, y or z in such a way,
right, and so you can control that. Usually people start with understanding
what they have and then they evolve, like their process evolves into putting more
and more prevention guardrails like that one. I think automated feedback to the engineering
teams when they're performing something that matches one of these governmance rules. I feel
like there's going to be a bunch of platform develops related people out there that
are jumping for joy over this functionality. Honestly, yeah, this is as
a developed like I have to people enjoy it, like people usually say,
eat your own dog food. Right, Yeah, it's an important concept.
So for us, it's like in the extreme, right, because our platform
is for APSEC teams and developers. Developers are are a bunch of people who
are really hard to please. Okay, let let's put it that way.
And that again, like developers are used to tools like compilers or IDs where
they get I don't know, five nines of accuracy. Right, the compiler
doesn't make mistakes. Usually it happens, but it's really rare. That's good
point. Actually, usually it's your fault as a developer if you see an
error, it's not the compiler. So again, going back to the example
from before, like going through six thousand alerts from a certain security platform is
not acceptable for a developer, So we we really make an effort to cater
the needs of developers where they get when we get to the developer like the
develops as you said, the develops engineer who or who is jumping out of
joy. We really we are working with our customers to make sure that the
process we will start with visibility. They understand, we understand what goes on.
And then when they let's say, automate jolting with electricity, the developer
who did such a bad thing, do it only when you're sure, right,
or let's say more realistically, like opening a ticket for a certain team
when such a thing happens, is only when they are certain right. There's
always mistakes, It's fine, but we really advise customers to evolve into it.
And if you see that, you can not ask your develop steam to
be always they will be alert. But don't go every day at five pm
go through all pull requests and make sure there is no such and such right,
don't do that. Automate that right. This is an easy thing,
let's say so yeah, with care, but as much automation as we can
is really the ROI is really clear for the customer. Yeah, definitely,
Yeah, And I think that's really a great approach as well, because by
making sure that you're relaying the information to only the people who can actually do
something about it. You're going to minimize the tendency that we have to just
turn off things that we we determined to be background noise, you know,
like you like you mentioned you know, if you get ten thousand security alerts,
we're going to turn it off as background noise, even though they might
not be. But if you provide specific information and and remediation steps for it,
and it's like, oh, that's that's something I can do, And
I think that actually lends a lot of long term stability to your overall security
process for sure. Also, there's a I think something that that is people
tend to consolidate certain tasks. Let's say there's a there's an app SEC team.
Let's say I get a report every day every week with a list of
vulnerabilities. I might say, hey, John, you're the you're the remediation
guy. H here's the list. Go every day again and again, go
through it, make sure it's fine. You might stick some boxes. You'll
say to the ISO person from the auditor, You'll say, I have a
process. Here's John, here's my vulnerability person. I'm good, and you'll
get You'll get the v there but like, at least in some cases,
not all the time, but this in some cases, the shift left here
is really substantial. Right out of the six thousand developed developers. Sorry vulnerabilities,
you might say, these four thousand are irrelevant because they are in tesco,
they are in old repositories that are not deployed anywhere. Forget about it.
Yeah and so and so, And you narrow it down and you get
to the twenty seven actionable vulnerabilities that like tending to will bring impact to the
company. And then you'll say, for each one of the twenty seven,
who's the right person? And here comes the concept that I like the most
here, which is code ownership. In appare we have a collective code ownership,
but when a certain problem arises, there's always the one person that knows
about this thing the most. And it's not, you know, naively,
the last person who touched that piece of code. It's more usually reality is
more complicated than that. So we make huge efforts in analyzing the behavior of
every developer, what code they are touching in which way do they like.
There's a difference between like the CEO doesn't know well enough and he does refractoring
and he makes the code cleaner because he wants to the difference between that guy
and someone else who went and replaced the authentication mechanism from user password to octa
integration datata. Right, there's a huge different difference between those two. I'm
not saying the latter is better, but that person is way more capable in
saying, hey, this vulnerability is real, it's actionable. This is the
way to solve it. Maybe some other person will solve it, but this
is the way, and this is how we can, you know, get
to do remediation much much faster and cheaper. That's super cool. So you're
actually building like a a historical audit of the actions of the people on the
platform, so you can identify what their expertise is and figure out who the
right person is without asking, without doing like an ad channel and a SLAT
group with thirteen thousand people in it. Mm hmmm, yeah exactly. And
we have a thing where we I feel more like, let's say it's maybe
a bad word, like more privileged than my previous self. When we did
in previous company, we did network security. We analyzed network traffic, so
we connected to switch in the server farm and we tapped to the traffic and
we analyzed it. But it started tequal zero. It started when we connected.
Right now we are privileged is an app sec company where we connect like
via API to let's say guitub and then we can analyze the entire history of
a Git repository from ten years ago. Right. So in that sense,
we have from when we start as we start to have a lot of information
about the code behavior, who's doing what, and we analyze the behavior ever
developer and we say this subset is what we call a security champion, right.
So some of our customers have actually using that. They have built like
a security champion program. So they have let's say three app sec people on
the team, like who's in charge of the entire company, and they have
one hundred and fifty development teams and they chose one or two people from every
division will be like their ambassadors into security. So when there's an issue with
product X, they can say, hey, team, we've talked before,
we've taught you about our processes, there's such and such problem, please advice,
right, And he's one out of seventy people, so they don't need
to go through proper channels with the management and and do like a like a
heavy thing. They have people they can talk to and it's not bureaucratic.
It's data driven, which is really like it proves helpful in many cases,
right, and that's super cool. One of the things you talked touched on
a little bit earlier was talking about the context and the visibility of it,
and you kind of mentioned that, you know, fixing these vulnerabilities gives you
the ability to say, hey, we prevented or we blocked this before it
was exploited. How much value do you think there is in that by by
showing the things the ability that you had to stop something before it happened and
kind of elevate secure already work to that of like future requests because everyone gets
excited over the new future requests. Do you think that showing the security vulnerabilities
you've re mediated helps bring bring some awareness to that? For sure? So
I think coming back to something I think you said maybe before we started recording,
security is a thing like everybody wants it to not exist until something hits
a fan somewhere, right, and then oh my god, we had to
do such and such before. I wish three months ago when we designed the
feature we would have thought about it. I wish we drink testing. We
would go through the list of things that could go wrong and find out about
it. So, but the reverse of it is everyone, everybody wants to
celebrate new features and sell it to customers. So uh, it's a hard
bargain sometimes to for security teams to prove their their worth to the organization because
nobody pays attention to quiet. Like the CSO can can come to a board
meeting, or the APPSC engineer can come to a certain meeting all hands for
the company and say, hey, for seven months no incident, everything is
quiet on my front. They might say, okay, let's reduce the team
right exactly. Yeah, life, life of security is really hard that way.
But I think I think that it's a game of numbers proving. Let's
say again, when people are used to false positives, and it's true both
for networks, security, intrusion alerts, whatever, DLP viruses, for everything
including appsect. When people are used to false positives, they're they're they're not
paying attention, like everything is noise. We have a we have a concept
of funnel where we are showing you have starting with thirty thousand alerts, and
via various steps you go to a lower and lower and lower number until you
get to a manageable amount of things that you can actually work on. And
if this is the focus, and in each team, it's a little bit
different. For some is like the is it connected to the is it exposed
to the internet? Just no, it really reduces. For some everything is
exposed to the internet, So there are differences. But as long as the
number goes down and and is it becomes manageable and real, then the teams
pay more attention. When you come as an absect to the dev team,
you're not wasting their time. They feel once they do like there's a I
think there's people underestimate developers or develops engineers in the industry where some say that
they don't care about security because it's someone else's concern. I don't think it's
true. Maybe that's just my opinion, but I don't think it's true.
And and if if like apseck came to developers with a small number of real
things, then the developers will be excited to deliver on it and they will
feel that they're building a stronger product. By the way, the same thing
is true also with performance and other things that are not like purely like features.
Performances a feature. Security is a future. I really believe in it.
So I think as long as the numbers are are manageable, I think
people will treat it the same way as features, and they can even celebrate
it. Yeah, for sure. I think especially like if you are just
trying to say, Okay, we got to get we got to get a
grip on our security, and you're working with a code base that's been around,
you know, for a year or more. The first time you run
that security scan, it's going to be overwhelming and that your first at least
for me, my first instinct is to look at the findings and go,
yeah, that's a problem for next sprint and then move on and next year
I'll have the same conversation with myself. So I think there's really a lot
of value in saying, Okay, let's just put this in context and here
are the key things that you can focus on this brand that'll actually make a
difference. Yeah. I mean, you're really talking about curation there, and
I think that's one of the things that a lot of tools in the Devil
of Space overall get wrong. They think it's like a lack of information.
They're all about exposing as many things as possible as if that's where the value
is. And I think the tool we haven't said but kept talking about here
is sort of like depend upon on GitHub, which you know spams every single
thing, and it's very difficult to know whether or not there's value in any
one of those alerts. So realistically, you want it to be curated.
That's where the value is, right, having not only information, but relevant
and useful and correct information, Right, that's prioritized. Yeah, exactly,
like this is our this is our claim to fame actually, because we're saying
like ESPM is the is the abbreviation, and we say deep SPM because we
really believe that just aggregating alerts, putting putting them, displaying them in a
nice table which streams nicely, uh, and putting nice graphs on top.
It's like, I don't want to sound disrespectful, but as you said,
right now, we're putting context uh into into that data is really crucial,
uh, in order to allow prioritization and remediation. So we when we say
deep I'll give you an example, when we say deep uh we uh we
analyze the code right, We analyze the gerry tickets, We read all the
fields. We analyze. We know what's an API. We know what's a
data model, and if it's safe to the database, if it's exposed via
that a PR or that I p I we know UH if a certain UH
like code element is in a certain module, which might be a part of
a repository. There might be a mono repo, might be a certain application
that has seventeen different modules from six different repositories, some in guitub and some
in bit bucket, and some I don't know from somewhere else. So when
we see a vulnerability, it might be a secret that we detected secret in
code. It might be a vulnerability that came from check marks or sneak or
someone else. We know a lot of context around it. Okay, So
if it's part of an API, we know a lot about that API.
If you connected to a COMI, then we can say, okay, that
API is exposed in that cluster and is getting bombarded right now in production.
So we try as as more and more things get connected, we add more
context into it. So it's really really important to inject that context into those
lists of vulnerabilities or material changes, because asking the developer to do the work
is not realistic. Right. We see we have customers with I think the
most has a forty something thousand repositories. Oh wow, that's that's a that's
a number that I don't think a person, a human being can fathom in
terms of like there's no intuition around it. So you have you have to
have that knowledge modeled somewhere. I mean I do, I do have some
intuition there that micro services went wrong somewhere, But you know from that,
I mean you bring abouts an interesting aspects here, which is information about the
likelihood of these vulnerabilities actually being a problem. And I think this is a
difficult area to answer. You can look at some code, even with the
context, and know that there is a security problem there. Maybe it's open
to ssr F, you know, service I Request forgeries or something on the
UI side that's open to cross site scripting, but it's you know, I've
been doing research and I just cannot find actual numbers on the number of say
data breaches there are, because the actually detecting that it's happened and knowing that
a change that you've made is going to has actually stopped. One is really
a challenge, and I assume that it's not even something that you would be
able to know about. Right. We do have some statistics for interesting,
but I can say easily that a state in terms of like level of hacking
might might uh utilize or take advantage of some some sort of nobility that could
be found in code and might help you to find your way through after four
months of an attack, like doing various things in order to get to a
certain crown jewel type of information. It happens, but many more times,
ah, some some person has connected the wrong wires and and and opened the
the I don't know, the some proxy server connected to the wrong the wrong
network and like maybe it's via terraform or click ups or uh, maybe a
certain a p I that is supposed to be under authorization is actually without because
like a bug in the code, these type of things are not immune to
bugs in code. And you might have like a huge wall, a firewall
and octa and whatever in front of your application that prevents the most brilliant hacker
to pass through. But your API doesn't require authorization. You can just approach
it and inject things into a database or pulled data from the database. So
I think what we see in various companies is evolving from the notion that we
have X vulnerability is let's solve every vulnerability right, and evolving into let's do
a threat model, analyze what I have in the most important applications or the
most sensitive applications, which is another area that we help with. But let's
focus on those, and in those, let's understand what's the attack vector,
and there we will say what types of actions we think we need to do.
Maybe make sure there are no exposed secrets. Maybe make sure that the
APIs don't do things that they shouldn't do. Maybe it's super duper critical.
Let's make sure we have zero verun abilities and do everything we can, but
not for everything. So this is really important in terms of time and effort
that the people put into it. Does that mean that you're actually also able
to investigate the sort of transitive dependencies of source code pack venders that are being
utilized? Yeah, actually yeah we do so, uh open sources. I'm
assuming you're in open source, but not necessarily but yeah, I mean right,
I mean they could be internal packages that another team has published or you
know, obviously code that comes from outside for sure. Yeah, okay,
so uh yeah, we do. We have a distinction between external and internal
packages, whether you publish it via an internal Nougat or Maven or something else
inside your organization, or even if you have a package that you copy and
paste it into some other area. I think, for me, the most
the more interesting thing in terms of like developing a product is that you have
a certain module that is shared between various applications. Right So you have the
classic example is an infrastructure package that has I don't know, utilities or ways
to connect to you or central user service or something like that, and it's
being used in like three different versions of that are being used in seventeen different
applications. So what's the impact of Warren going and changing a certain threshold of
I don't know password, the strength in that package. What's the impact If
it's on seventeen applications, it might be more substantial, right so, and
it could go through more than one hop of analysis. So we're doing really
interesting things in that area. And it's hard that computationally if you think about
it. Yeah, for sure. So Warren, you're the CTO of a
security off company. How do you like obviously security is like front of your
company. How do you approach this whenever you're building and deploying your product?
Oh, definitely don't ask me that question. Sorry, Should we have the
editor cut this part out? No, I mean it's totally fine. I
mean realistically, we are definitely looking for products that, like you know,
ton Scott here to review. I think a lot of the problem is that
many products in the space fall into this sort of uncanny value where they're either
unnecessary alerts that don't really tell us anything. We've definitely gone through the snakes,
sneaks and dependent bots code QLs of the world, and as well as
look at the ones that review stuff for a Soalk two and ISO, but
they're mostly checkbox driven security and don't really focus on actually finding real problems.
So I think realistically it's on us to do this investigation most of the time
manually, I mean obviously using the tools that are available to make these decisions.
Now, I think we have a lot of benefit over some other companies
given the size that we are at much smaller, so it's not like we
have forty seven thousand repositories that we've got I think the number is under a
thousand, and most of them are not services. Yeah, well, I
mean there's just an amount of overhead as far as cognitive overload that teams will
have to deal through. So more repositories can only be managed by more people,
and that breaks down as size increases. So I mean, realistically,
there are things that you just don't want to have to deal with. Now
in my position, what I've really focused on is what are the platforms or
tools that are out there that help eliminate a whole class of problems. So
classes of problems can be eliminated by I mean me personally, you know,
i'd say get away from Kubernetes, but really virtual machines in general. While
it may seem like it's easy to spin them up, the number of security
concerns from my standpoint is just massive. There just patch schedules, et cetera.
Even for things that are zero day, right, it's so much easier
to delegate that to a potential cloud provider. How to get around those.
Another thing that we're leading with I'm not sure when this is going to be
available in production, is allowing our customers to sign the data that they're sending
to us. That means that we have a fidelity over you know, whether
or not that data has actually been changed, not necessarily access, but specifically
since we are in the AWE space, we do care about when it's been
changed, and so there are things like this that we consider novel, but
realistically, it's about finding those tools that help us elevate what we're utilizing to
another level. Yeah, and I think that's like a unique problem because you
do have like a limited number of people on your team, but literally an
unlimited number of security problems. You know, Yeah, for sure, you
know that that's always going to be a trouble. I think I think this
goes for everything. Focus on your core competency, whatever that is, and
find a way to eliminate or at least delegate those concerns elsewhere. So every
time one of those comes up, find a company that helps you eliminate the
broadest class of them, or you know, transition to a different stack or
something like that where you don't have those issues, because you will never solve
every problem. And I think often you may hear things like, well,
we can't depend on another company offering a critical component for us, because what
happens if it's down, what happens if there's an issue there? I mean,
like you should trust the security experts to get that right more so than
I mean, there's no different between the set of engineers that they're hiring and
the ones that you can hire. And they're probably focused on this, right.
You know, we're focused on hiring people that do care about that security
aspect, just as you know Tom's focused on this as as well. I'm
sure, right, you know, you hire people that understand this part of
platform management and security to go forward, and you're not going to be able
to do it as effectively unfortunately. Time and resources, Yeah, for sure,
we at Apparel, we do have an infinite amount of time and resources
that that's their usp there right right there. You know you heard it there
first, right, So a lot of times with sort of continuing on what
you were saying there, Warren, you know, there's like a there's three
models of solving a problem. There's do it yourself, there's tell get someone
to tell you what to do, or there's just hire someone to do it
for you. And so, Jonathan, what are the what's the like entry
level do it yourself version of handling this like obviously your apiro is like the
Hey, we're gonna collect all the data and and put it in context and
prioritize it for you. But like, what's the entry level version of that
that probably most of us are actually trying to do but unsuccessfully. That would
tell us, man, there's got to be a better way. You mean,
you mean, how how the company, like a software shop would do
what we do manually like without yeah, uh so so like people more people
essentially? Uh yeah, so so we see that, uh and I understand
the like like when when you're facing a certain amount of work, then you
can either uh focus the team on it and do it like one person for
a small shop or one team for a larger shop that handles like the load
for everyone. And there's the companies that uh distribute to work with the load
on the developers. I think the most important thing to understand to maybe a
marker that might tell you that this thing got out of hand is if like
hopefully you're asking your developers like how they are doing and if they they feel
that their job is going well. So if they say twenty percent, thirty
percent, seventy percent of my time is going after things that I don't see
any value coming out of this is a good marker that you might want to
change the process. Either either use it like a platform, like a peer,
or maybe something else. Either either that or don't do it. Like
sometimes it's better to take take the lead and not go through a certain process
save like ten percent of your one hundred developers time, that's ten ten developers.
Ten developers can bridge the gap and bring more either more value, more
features, or more security. So either use the right tool for the job,
or maybe do less of it of the of the noise. Sometimes it's
better then, like bringing everything to a halt. That's probably a really great
point there, actually using the right tool for the job. And I don't
think that could be said enough because you know, to answer a little bit
more of your question, Well, i've seen the let's build our own sort
of internal strategy for automatically updating versions of packages for instance, right, And
it's like, well, you're building this yourself, Like why are you building
the tool versus using something that's way more intelligent at actually finding where the security
vulnerabilities are and focus on that one. And I think there's the other one
I've seen, which I also am not a fan of is the replicate all
the external packages in the world locally in your own package repository, because unless
you have a team that is expert in reviewing that or a tool that's actually
going to do it, you're not really doing anything other than having out of
date versions that are available. So you're actually almost hurting yourself more than you're
helping by using one of these because it's not solving a problem of security,
it's solving a problem of anything reliability or where the single point of failure is.
Right. It's not a highly reliable package manager out in the world that
has security teams dedicated to finding vulnerabilities. It's some tool off the shelf that
you're deployed in your own infrastructure which is out of date. Yeah. I
think also this is a great example of sometimes not understanding the trade off usually
most things, maybe in life, but in our field, there's a trade
off. Like if you're saying, don't like I have a certain certain version
of package X that is not vulnerable, then let's freeze it in time and
not never ever upgrade it because there might be a vulnerability. There's a trade
of there, right, both functionally and maybe maybe it does have a vulnerability,
but it's unknown. Now I'm not saying like for everything, newer is
better, but there's always a trade off. So yeah, I think that
they like what what you said is really really true. If you're doing it
yourself and like inventing things that are not your core business, then be really
careful because you're probably not the best of breeding usually. No, I mean
no, no, it's for true. I mean I think there's another problem
there, which is you don't know whether or not you're doing it right.
So you're going through the motions of things right, you know, you're even
if maybe maybe updating the versions is the right thing, but it could just
as easily be the case where it's not, and you're actually having a negative
impact overall. So you're not doing the research to understand what the best practice
trends are, and you're using your own sort of heuristics from your own experiences,
which they're still valid, right, but just in a small niche compared
to what the collective wisdom says, and going about that, you know,
in that way, which may lead you to, oh, yeah, we
need to automatically roll all versions of all packages all the time, or we
need to freeze all of them, and I think both of them are equally
problematic in some way. Yeah. One of the other things you know,
you mentioned higher or use the people who are best at that. I think
one of the things that I see common with that is whenever you go to
like you say, Okay, I shouldn't be writing on authentication package myself.
That's a solved problem that's been solved by people way smarter than me. So
you go out and you start looking at available packages. Like part of that
research is understanding who the expert really is and digging in, like dig into
their GitHub repo. Is it being updated frequently? How many committers are working
on this? Because if you find an authentication package that was updated three years
ago by one person and hasn't been touched since, that may not be an
improvement over your current authentication layer. Yeah. Sure, I mean i'd be
curious what you know, Tom, actually if it has any information about this,
Like are there heuristics about update frequency or the type of packages that lend
themselves, especially in the open source world, to being more vulnerable or more
secure? Yeah? So really interesting, especially right now, so in Apiro
you can see a lot of like interesting properties about open source packages. Okay,
so I think will you mentioned some of the most important ones, like
if it's maintained or not maintained. There are other things as well, but
right now, we just published, like I think a week ago, like
our research team found a lot of malicious code in many guitab repositories and some
of the guitub repositories. Like it made a lot of waves I heard about
some of some of the guitub repositories are really things that you would like at
first glance. You you wouldn't say there's something fishy here, right, A
lot of maintainers, it's being maintained really st regularly. So I think not
to be too cynical, but I don't think you can trust anything right pully
with you. Actually, I was just thinking about this, like whatever a
metric you're going to use, just think about how it's going to be potentially
abused, right, you know, it's maintainers or updates or issues. You
could easily clone an existing package out there on GitHub, have the same number
of issues, copy everything one to one and then make just a small little
malicious change in the package somewhere, you know, post install, or just
for a production run time and also publish it to a new get MPM,
you know, wherever, and you would never tell the difference unless someone actually
investigated that source code. So those sorts of heuristics and may not even be
valuable. Yeah, yeah for sure. But you know, the same way
is it's hard to get like a good grasp over forty thousand repositories in your
organization. Yeah, which there's not a lot of eyes going through that code
base, but in like let's say React, JS and Internet there's a lot
of people like going through. But even that repository is really large, and
there are dark places within that repository that people like for years didn't pay attention
to. So when you get big data, it's really like it's easier to
to to hide in plain sight sometimes then if there's a two file repository and
to take hold of that, so you really need the tools the machines to
do the work for you. No, I mean that that actually gives me
some flashbacks to I think it was a few years ago there was a bunch
of academic centers that were intentionally making malicious changes to like Linux Foundation source code
for the kernel as well as other things just to prove that you could get
malicious code into an open source project under all pretenses and so like that's pretty
ridiculous. Yeah, yeah, I remember that. That was kind of wild.
Yeah, So nothing not things off the table when it comes to where
problems could be. And you know, if you look at the lots of
issues that show up in public repositories, like they're not necessarily people that know,
you know, fundamentally every single aspect that's going on there, Uh,
poll requests that get opened are you know you don't. No one has full
understanding of how all the packages interact with each other, so you know,
who is even able to validate that? I think at this point you pretty
much need to rely on some sort of tool to do this for you.
Right on. Cool, So I want to ask both of you this question,
because you both have strong security backgrounds. For someone who's just starting to
put focus on their security footprint, what's your best piece of advice on where
to start start with? We'll start with Jonathan. Okay, So, without
qualifying if it's a small company or a large company or like one developer,
I think the most important thing to get a grasp on is like inventory,
what I have. I think this is the fundamental part here. By the
way, as a developer, you might want to reduce it as much as
you can get rid of what you don't really need, and in large scales
it's even more important. But knowing what you have, I think this is
the most fundamental thing to really really understand before I do anything advanced. Know
what you have and plot twist. Apiro can help you with that, right,
I wouldn't I wouldn't imagine saying anything that won't lead you to say that.
All right, Warren, what's your thoughts? Yeah? I mean I
think actually Anton mentioned this earlier and I know I've become a broken record for
this, but it's start with your threat model. If you build or approach
any problem without first thinking about why you're doing it, not only can you
what you're doing isn't going to nessly be useful or relevant. It can even
be harmful as far as your overall approach goes. So you know concretely what
like, what do we need to do, why are we doing it?
And then start looking at okay, what are our options for implementing solutions?
Because otherwise, I mean, you'll end up with the six thousand dependent on
alerts and not knowing what to do, and the result may be developing your
own platform that automatically increments, you know, package versions, versus knowing that,
hey, we're getting lots of spam on our endpoints in the form of
WP dash admin and we're not running any PHP stuff. You know, it's
not a DUS attack, but it's someone's looking for an exposed WordPress instance or
Mango dB or elastic search, and if you've got that exposed, then you're
going to have a problem. So, as Youzon said, the the inventory
of what your services are and how they're exposed to the Internet is equally valuable
for sure, right on excellent. Any any other closing or parting thoughts on
security before we move on to picks, I'd say, alluding to a previous
point that you made before, that security is a feature and and uh,
if if you do the right thing, then then people will appreciate it.
If it's sometimes even more than doing the features. No, I mean,
I love that approach actually, because often people see it as the like a
cost center to prevent something and the best case scenario is that you nothing happens,
and so the ROI you know, is a hard justifier, But I
think the missing part there is that actually the way you've implemented security has a
lot of value that for your product, for your end users, potentially a
better experience with s oh and log in is it is it defining factor for
companies, especially enterprises, who want to be secure likewise goes having the latest
functions and features from your dependencies, being able to utilize them and not be
stuck on legacy versions for fear of a security issue that just on my gets
pulled in. Like using a platform like a bureau you know, helps you
get that, helps you achieve that without getting stuck in in fear land or
dedicating your resources in an area that's not your core competency. Yeah. I
think that's like one of the common things that I see with teams or departments
that are traditionally seen as cost centers is changing that perspective that you're not a
cost center, you're just bad at marketing, you know, because there's like
it there is like it's everything comes down to politics at some form or another,
and so part of getting your team to not be is a cost center
is by marketing your team and showing hey, this is these are the security
vulnerabilities that we closed down that we were exposed to? Or from a DevOps
perspective, hey, here's how implementing this platform allowed the engineering teams to go
from fifteen deployments today per day to fifty deployments per day in a safe and
secure manner. So I think really learning to market your team helps give you
mind share across the leaders of the company and brings additional focus. And with
additional focus, you get additional resources. I mean, last time we talked
about product management, and I think this is closely connected to it. Like,
you know, you can even start before that, right, Like,
if you're going to be rolling out features or functionality or a platform to your
organization, what do the other engineering teams actually think about? You know,
do they care about even being secure? Because if they're not already on that,
then it's the same challenge as marketing to external customers. You what problems
are they trying to solve? And if security is not part of it,
then you're really starting with an education approach, right Like, here are the
sorts of things that could happen because of how you set up your services and
if the organization hasn't aligned incentives actually encourage them to be secure. Then it's
going to be an uphill battle wait for them to get hacked and then go
oh oh okay, so now y'all are concerned about security. Oh. I
mean there were definitely there were definitely some suspicious companies out there that were offering
like DEDOS protection and as part of their actual operation, I mean they were
they were hugely malicious to start out with, but uh, as part of
their operations they would go out and actually just dds customers and be like,
hey, you know you got d dos Because you don't, you're not using
our product. And I mean it's a little bit suspicious if after our security
incident you immediately get someone showing up your door saying, you know, hey,
we could help you solve that problem. Though, it's like the mafia
style approach they send and then go so, hey, it looks like you
need some protection. Right on. So let's do some picks as we round
this episode out, Warren, I've been picking on you, so I won't
pick on you this episode. I'll actually start it off, and much to
everyone who's a frequent listener, surprise, my pick this week is going to
be platform con coming up this June the five day virtual conference on Platform Engineering,
where they will have some folks talking about security topics too as they relate
to platform engineering. And then my favorite part of the conference is at the
end, I am going to be interviewing some of the speakers in a live
Q and a session to close out the conference. And as I mentioned last
week, if they're specific people you want me to talk to, or specific
questions you want me to ask, be sure and hit me up on Twitter
or send me a DM or any other contact method and I will filter and
aggregate those and get those questions to ask for you. So platform con dot
com if you want to check that out, and that's my pick. So
Warren, what have you got? Yeah, Actually, this is timely.
It's going to be security related. I just got back from a conference in
Germany last week where I was actually talking about the security journey we took at
authors and I got asked, you know, well, how do I get
into security? How do I become an expert? Is it just the do
I am? I either completely don't know anything or do I have to know
everything? And the truth is, you know, you just learn things over
time from different sources and eventually, one day you get to stand up on
stage and start giving a presentation about security, even though you believe you don't
really know anything. But I will say that there are a couple of great
newsletters out there that I read myself. They may be too technical for some
others may be way beneath them, but I highly recommend the TLDR newsletters.
Specifically, they have one Security and there's also another one called the cloud Secklist,
which is oriented towards cloud security infrastructure, and they've been great. They
talking touching on a lot of different topics, and you can go deep with
the links that are in there, and I can take you a long way,
especially at the beginning of your security journey. Right on Awesome, how
did you talk? Go by the way? Well, you know, like
usual, I'm sure I did terrible, but then everyone tells me how great
it went, so I have to I'll have to watch the video once it
comes out online, which will probably be posted on the author's blog somewhere Right
on awesome. All right, Johnathan, what you got for us for picks?
Okay? So I had something else, But Warren, I think I
can maybe connect to what you what you said because something that I feared for
a long time happened at my family yesterday, where my son was a twelve
year old came to me and said, Dad, I managed to hack the
Google Dinosaur game and and made it such that I can never lose. It's
not that advanced. It's I think it came to him through YouTube or Instagram
or something like that where you just override the game over function or something like
that. She's really nice. Uh so, so it came to me as
a shock, but uh it got me thinking and and I like alluding to
when you said how to start in the security business. Maybe uh so.
I think my piak is is when it comes to kids, maybe uh start,
start from the fundamentals, not not. It might be cool to show
your friends in class that you can hack a game, but I I since
then, I since I tried to show him the fundamentals. What goes on
there? What does it mean a function? What does it mean to change
it in run time? Like, uh, maybe you should prevent that maybe
not like like really have him understand what goes on there. I think it's
really important for kids that can see YouTube or whatever TikTok, they're they're seeing
really advanced things in life, not only in security or coding, but they're
really pushed to go fast and like having them understand the fundamentals, understand for
for a non native English speaker, understand English, understand math, understand the
like the concept, and then go and hack the Pentagon or No, we're
not recommending that anyone does anything. I do not condone that sort of no.
But that's a really great point actually, because there's security is so vague
as an individual idea that really defining for yourself what it even means to go
into security, right, Like are you interested in hacking things or even depending
against attacks, which I think, you know, we talked a lot about
here, but or our products that help you build up tools and whatnot.
You know, these are all different, fundamentally different areas for sure. Yeah,
And I think there's like just going beyond that. I think there's huge
understated value in just having a basic understanding of how code works in a modern
technology life that we live in now, because once you understand what's going on
under the hood, I think you get a lot of perspective in how you
interact with different companies, you know, whether it's through their website or their
application or their call center or whatever. Like. If you have that basic
fundamental understanding, I think you start to approach problems differently, even if you're
not in a technical role. I think when you're talking to a sales rep
of a customer support agent of a company and you get an error on your
screen, that's as like you know, on unhandled exception, and you're trying
to explain to them how their website is broken, right exactly, all right?
Cool, Jonathan, Thank you so much for being on the show.
This has been incredibly insightful, so I really appreciate you taking the time to
join us. Yeah, thank you so much for inviting me. Yeah.
Yeah. Happy to have you back on at any point, so let us
know if that's of interest to you. Qaarren, thank you again for joining
me here and co host to miss with me, and to all the listeners,
thank y'all for listening. Love having y'all out there, and we will
see y'all next week.