Hey everybody, and welcome back to another episode of Adventures in DevOps. This
week, on our panel, we have will Button. What's going on everybody?
So I have Jeffrey Grumman. Hey there, I'm Charles Maxwood from dev
chat Do TV. And we thought, well, we had a guess not
show up, and then Jeffrey's telling us about all the horrible but fascinating things
that are going on on the Internet these days, and so we're going to
talk about more breaches and concerns and stuff. Is going to freak me out,
but I'm kind of curious in a morbid way to hear about so I
guess, I guess let's just dive in. So, Jeffrey, what's broken
in the world? Oh? Man, I gotta tell you. It's been
a crazy week. I think that started out last weekend and a piece of
software called Case, which is used by different managed service providers, was found
to be carrying malware in terms of ransomware. And so basically any managed service
provider who uses this case to like manage their customers or the client environments,
all of a sudden were just locked up. And that means that they can't
you know, that means they themselves are locked up, and that means that
they can't service their clients. So basically you're like, you know, if
you're using an MSP that gets that it uses cassea, you're basically like you
have no service, Like you have no IT service provider anymore because they're just
their host. So that's how we started the you know, the week coming
out of the long weekend, the holiday weekend. And then Microsoft, I
think a couple of researchers found a really crazy remote code remote code execution RCEE
vulnerability in Microsoft Windows Principuler. So there's a lot there. Maybe it's just
worth us a couple of minutes in talking about what that means. So if
you're not familiar with Microsoft Windows, going all the way back to the beginning
of time, one of the biggest things that people used to do was print,
and if you weren't printing off of your own machine, you were printing
off of a print server, which is very common if you're not familiar.
Printers are these big, oblong plastic devices that spit paper out from time to
time or toning. Right. It's funny because these are like this is like
legacy stuff, but it still is, right, there inside of every Microsoft
Windows whether it's like the desktop version or the server version, they're all running
the print spooler. It's a service that runs by default and it just has
all this legacy software in it because it's just been around for so long.
And so some researchers have been spending time looking at that and they found this
again, the remote code execution. What does that mean? That means that
I don't have to be connected to the machine to be able to like literally
like logged into the Windows guy to be able to exploit that vulnerability, And
the exploit of that vulnerability gives me what's called system privileges. A system is
basically like a service account version of like your local administrator. So it's an
elevated privilege that certainly could be used for all kinds of nefarious purposes. Interesting
thing about it, though, is that that I think, if I remember
right, or if I understood this correctly, you have to have a legitimate
user account on the box to be able to take advantage of this, Which
makes sense because the Prince Spooler is typically not something it's not like a web
server that any anonymous connection can use utilize do something make requests of et cetera.
The Prince booler is only available to users of that system, so you
have to have a Windows user account already. So that was something that you
know, when you think about like the risk side of this. Okay,
how do I prioritize or what do I think about this vulnerability? Okay,
Well, if it means that somebody has to have local access or you know,
an account of the box already, then maybe some of my systems I'm
less concerned about. But maybe if I have a Windows system that is exposed
to the Internet, I might be more more concerned about that, especially if
it's let's say, like a SharePoint server right that's exposed to the Internet,
where somebody out there might have a user account on my box for whatever reason.
So then I'd probably be more concerned about it. But yeah, through
this and it went on for days off, I'm just trying to figure out
how we fix it. Microsoft was trying to figure out workarounds. Researchers were
trying to find out workarounds. Can you just shut down the print spooler service?
But that's it wasn't enough. That actually doesn't stop the vulnerability from being
exploited. I think finally, today I saw an update that Microsoft provided an
out of band patch, out of band meaning not part of their like patch
Tuesday, which is I think a monthly cycle, so dead it's out of
band to that they just released a patch for I think for basically every version
of Windows. So if you haven't gotten that, if you're not aware of
this, you can certainly download it from Microsoft. But yeah, that's basically
it. Then nuts all though, I think there's a lot there to talk
about. So it sounds like if you just delete all your user accounts though,
then it can't use to exploit anything, right, that is correct.
You know we used to talk about you know, this is way back in
the bill before the bill gates famous letter about writing secure software. We used
to joke that the best way to secure a Windows server was to fill the
footprint with cement and then bury it in the ground. So deleting every user
account on the device is similar to that approach. It is it is no,
yeah, no, no, yes, we're paid on results. Yeah.
Yeah. Apparently it's been according to Microsoft and SIS, which is there's
a government agency called the ci A and they sort of track these vulnerabilities across
all vendors and provide bulletins and that sort of thing, and they claim that
these have been exploited or there have been exploits seen in the wild. I
haven't heard of any specific breaches based on this, but clearly like something this
big and this that's like sort of easy to exploit vulnerability is going to be
you know, there's going to be folks out there that are basically going to
be testing, you know, every system they can to see if they can
exploit it. So so yeah, because because some of the people who are
vulnerable to this are used are relying on the MSPs to provide their IT support
if you can't do that because they've been locked out my ransomware right right,
It is the perfect storm. So again we're back to deleting all the user's
accounts is really our only option. Absolutely absolutely, you know, going back
to pen and paper forms in triplicate, if you still have them in a
box in your closet, you might want to bring them out stock in carbon
paper this afternoon. I think that's where we're going. Yeah, I didn't
know you mentioned before the show, Will that you you went you were paying
attention to some of the I guess recount stuff in Arizona, and they somebody
mentioned that they should go back to paper ballots. Yeah, And I've been
to a number of political I'm fairly involved in Utah politics, and yeah,
there are a few more than a few people we've gone to electronic voting,
especially during COVID right where we couldn't actually actually get together and vote with paper
or anything that looks like paper. And there are so many people that have
brought up I just want to go back to paper because I know you can
count it, and not to get political and not to go into any of
the implications of how that can go wrong, right, But it's it's interesting
that yeah, Chads my friend ha, Yeah, well yeah, nobody on
the outside can hack the paper, but that there are humans counting it.
So, I mean, it's just it's really interesting that a lot of this
really comes down to what users do and how they approach it, and at
some level there's trust, right, And I think that's the point is delete
the user's account, is we're kicking all the people off that we don't trust.
Yeah, yeah, I mean it's interesting. Well, and I think
just to round out that the discussion about voting, I mean, I once
in a while, I will hear somebody talk about, hey, we should
make it even easier to vote. Let's why why aren't we doing it?
Why aren't we letting people vote over the internet or over their mobile phone?
And if you hadn't thought about it in the past, like these two examples
by themselves, like of the vulnerabilities and just how easy it is to manipulate
or to break in, should give anybody pause about thinking about having something like
voting, something as critical to society as a whole, to put it online
or to put it on a mobile device or which is also online. But
you know what I'm saying, Like, it's just it's so difficult to try
and protect these things, especially when you talk about third party software. Right,
you're running in a window, or you know, you're running your business
on a piece of third party software, and you don't know, you don't
know how well that company. I mean, listen, when we're talking about
Microsoft, they have and they now have a long track record of, you
know, really trying to be on top of the security, and I think
they've but how many lines of code are in you know, Microsoft Windows.
I mean, we're talking about it enormous, right, oh yeah, chunk
of work there. So it's and and we don't know, like it's that
that really is the challenge, Like, you know, you try to think
about the risk to your business and how to quantify risk, and when you're
talking about putting a piece of software and running your business on that piece of
software, like you have no idea what that risk really is. Well,
and you talk about you brought up the voting and just the security of all
the things involved, but even down to the user devices, I mean,
how many of those are compromised that people just don't know about. Yeah,
it's it's so fascinating to me just to see all the different levels that this
goes to and yeah, what what it effectively boils down to, right,
I Mean it's always talked about security versus you know, sort of the usability
the convenience of it, and security always will get in the way of usability
and convenience. I mean we see that in a walks of life, you
know, whether it's physical security where you have to go through a security checkpoint
or something like that and you've got to take off your shoes or you know,
whatever it is that they're making you do. There's always those inconveniences,
but you know, depending on what you're doing. I think that's the important
part for anybody running a business, is what you know sort of what due
diligence are you doing or what how are you thinking about the convenience versus the
usability or versus the resilience, right, convenience of just buying some piece of
software or downloading a piece of software versus the disruption the potential disruption to here
the entire business, and that has to be well thought out, I think
today, especially when we're doing so much more online. I mean, remember
the days when it was not that unlikely, or there wasn't that uncommon that
you go into a store and the credit card swipe device thing was offline because
the phone line was down or remember that, And so they would take out
from under the counter, they'd take out the dow hiki that goes back and
forth over the carbon right, And everybody still have those today. That happens
in your store. I was in a home depot the other day and they're
like the manager was like running around the aisle saying sorry, sorry that we
can't can't check anybody out right now. I don't know why there's no work
around today. People don't really have that sort of thing. So, yeah,
I was at an if A country store and which is just kind of
a local farm store here, and what they were doing is they were they
were basically grabbing They jot down your credit card information and tell you that they
were going to charge your card later, which didn't necessarily make me feel good.
But then they gave you an invoice, a copy of the invoice right
right, which was also fascinating in its own way because it was like,
I don't really you know, and yeah, then yeah, the charge came
through later, and they're like, yeah, don't leave that invoice anywhere because
if you do, it's going to be a problem, right, And yeah,
what do you do? Yeah? And again I I just don't.
I don't. I think that's that's just sort of a short, shortballower or
just something that has not been really thought through in many businesses, many you
know, many companies, of just what happens if you know something some key
piece of software that we rely upon, and sometimes it's not even something you
you even think about or realize. Like let's go back to the solar winds
example from right from January or or last December where it wasn't that wasn't a
ransomware that was linked back to somewhere some group within Russia. And you know,
it seem like they were going after very specific companies and government agencies,
but still very disruptive to your business. And you didn't even think about Solar
Winds. Most people didn't even think about didn't even have an idea like who
is solar wind Who haven't even heard of this company? But it's a piece
of software that so many IT shops are using to manage their own infrastructure,
servers, desktops, network gear, all that stuff, which made a prime
candidate for being for hacking into it and putting basically embedding your malware into that
piece of software, because it's just it's ubiquitous, like so many companies use
that. So if you're sitting in the business office, a business side of
things, you're not even going to think about some piece of software that your
IT people use and how that can really disrupt your entire business or whatever it
is that you do. If you're a government agency or your entire agency to
where they basically take themselves offline and investigate what happened and figure out what's going
on, and you know, it's just so disruptive to your entire IT organizations,
everything else you're trying to do or the IT folks were trying to do
gets put on hold until you can fix this problem investigated or do what you
have to do. Yeah, but yeah, that's the I mean, it's
the it's a situation that we're in today, and I think it certainly plays
into when we think about like DevOps or dev zecops. It certainly has to
be part of what we're doing. Right. So, if you're building software
and you're building it on top of open source libraries or connecting to you know,
somebody else's API or whatever else, and it's probably several other scenarios,
But what are you doing, like what how are you thinking about the security
of that piece of software or that service that's being provided, whether it's a
SaaS service or what have you. What happens if it's down? That's probably
one of the most most obvious scenarios. But what happens if they they themselves
get penetrated by these threat actors? Then what how would you detect it?
Do you have the ability to detect it? How would you write? How
would you respond? To this, And I think the first step is just
sort of thinking through that process, because the truth is is that you can,
you know, you can detect these things and you can respond to them,
but not if you're not prepared. Right. It's if you know,
the first time you even hear about it is because the FBI is calling you
and saying, hey, you've got solar winds and we've tracked that the bad
guys are in your environment. It's too late to try to figure out how
do we detect it at that point, right, it's now you're just all
hands on deck. It just behooves organizations. So really start to think about
this, especially with just the number of these situations. Ransomware is. It's
just a perfect example. It's so lucrative. It's not going anywhere, no
matter how many times Biden sits down with Putin and talks about how ransomware has
got to you know, they've got to put a stop to ransomware. I
mean Obama tried to do that with with she you know, back in the
day in Shinna several years ago. And you know, it doesn't stop,
and it's not going to stop. So I just think these are examples of
that everyone's got to take the heart and figure out, Okay, what is
where is the risk to my organization and what are the kinds of things that
we can be doing to mitigate that risk. Yeah. I think one of
the things you can do that I don't know of a lot of companies that
do this is implement a full on disaster recovery and preparedness plan. You know,
I've worked at a few companies in the past where we would you know,
we would actually take our backup tapes off site and go to restore them
onto this is when we still have physical data centers and restore them onto new
service just to verify that we could bring things back up. And a lot
of that practice came from whenever I was in the Navy, because that was
a large part of what we did there, Chris, the stakes were a
little different, but the principle's the same. You really don't know what the
missing pieces are until you go out and try to do that, and that's
that's the case. I think. I think that's a situation that a lot
of these businesses find themselves in is whenever something happens and they are down,
that's when the conversation starts, Oh what do we do now? Well,
the first was to have this conversation before, right before the first step to
getting out of a burning building is to plan your exit before the building's on
fire. Yeah, it's so true. I mean, quote to that to
that point. How often have you seen people that had an entire backup strategy
never tested their backups, have an issue happened right sert of those down or
whatever you got to restore to the new system, and then they find out
that the backup actually didn't work, or the restore process doesn't work, or
whatever they thought was on the tape doesn't exist or right. I mean,
it's yeah, it is so common. Yeah, I've seen it a lot
and been guilty of it a lot myself. Even you know, even after
like the first time is like, oh that was a painful lesson. I'm
gonna I'm gonna learn this time, and now I don't, because every it's
always there's always like a unique little best. You know, last time you
were missing a certain set of files. You know, this time you had
it stored in a location that you couldn't access or you know, so it's
like you can't just do it once. You have to do it repeatedly because
the environment's always changing. You know, maybe last time you restored from a
particular server that now no longer exists, or you've switched to a different provider
for some particular service. But it's and it's hard. I think it's really
hard for IT teams and DevOps teams to get the buy in to spend the
time and effort to do that because a lot of the pressure is on build
new features, push features to production, increase traffic to the servers, increase
the conversion rate, you know, and do disaster recovery planning and execution.
Does nothing to increase the revenue of a business. One of my favorite analogies
to that I use it all the time, is interesting. It just resonates
well, especially here in America. But you think about football, right,
so you know, and it's the same sort of principle like if all your
focus is on your offense, because that's what's you know, that's sort of
the analogy of the revenue, right, If all your focus is on offense,
because that's where the points could score, and you can't win. If
you don't score points, right and you forget about your defense, you're going
to be in big trouble because you're most likely not going to be able to
score enough points to overcome whatever deficit the lack of a good defense brings you.
And then if you think about it, so then you let's say the
analogy want to step forward and you say, well, what does a good
defense mean? It means focusing on all the little things, right. A
good defense means that if my guys can't consistently do open field tackles, we're
going to be in trouble. If I don't have good coverage, we're going
to be in trouble. Right, It's the basics. There's nothing fancy,
there's nothing like exciting about. You know, you're not scoring a quick six
or anything like that. But if your folks aren't doing that and practicing,
and the only way that you are able to consistently do an open field tackle
or provide good coverage or all those other little things that you've got to be
able to do is if you're practicing them every week. And like you said,
you're going back through tapes and like, which is the same idea,
right, It's like exercising it and looking at, well, what happened last
time? Why did we fail? Or why did this go wrong last time?
You know, you're reviewing tape, reviewing the videos and saying what happened
and how do we do better next time? And if you're not doing that,
then you know, you think about it a professional football team, like,
think about all the support staff that they have on both offense and defense.
These coaches, the trainers, the equipment, all of that stuff that
goes into both sides of the field. And if you're not doing that,
you're not going to win. And I think the analogy of the business is
if you're not focused on both and really working both, you're just putting yourself
at huge amounts of risk for something like this to happen, and then you
know you're just going to be in big trouble. Yeah. Response has to
be contextual too, because you know, using that analogy, Okay, we
need defense to prevent the other team from scoring points, so we're going to
send the defensive team out on four wheelers because they'll be more effective that way.
It's like, well, wait, no, you can't really do that.
And I think the same thing goes with your testing, your disaster,
your preparedness plan. You know, it has to be contextual. It has
to be when you propose that, you've got to be mindful of the fact
that there is still an offense and you want to minimize the disruption to them,
if not completely avoided as well, because I think there have been instances
where the security team is like, oh, that's that's insecure. You know,
we have to completely close that off to the point where it starts to
the security starts to hamper the business then, and so you really have to
find out what that fine line or that balance between the two extremes there is.
I think that's that that's a really important point is that if you if
your security team is operating, you know, in a black hole, as
the old joke that you know, the security team is always sort of in
the dungeon behind locked doors, no one else can get there, and it's
that sort of thing. That's that's the worst scenario ever. It really is
if your security team is not embedded in the business to where the security folks
understand the business and understands revenue. I mean, because let's face it,
what is the biggest risk any business faces the lack of cash flow? It's
Microsoft vulnerability, right, It's not the biggest risk you face is a lack
of cash flow, right, because that impacts my paycheck. That's everything,
right, I mean, that's that's the you know questions is the business open
tomorrow or not? That's that's what defines it. So if the business isn't
in line with that, I'm sorry with the security team isn't in line with
the business with where revenue comes from. And you know, I couldn't care,
Like, if you've got a business process that is where the cashlow is
coming from, then it doesn't matter like what the risk of the cyber risk
is, Like that's the biggest rish you've got, right, And it's to
your point that that's really what's important. I see that too often that security
teams are really just divorced of what's going on in the business. They don't
understand business processes, they don't understand how the business makes revenue. And then
those kinds of things that you said, like oh no, sorry, all
those ports are turned off, firewalls locked down, and nope, sorry,
you know, nobody gets administrative access to X, Y and z, And
you know that's what that's when that happens because all we're thinking about is best
practice and we're not thinking about risk and what's going on with the business,
and you cannot divorce the deal or else you're just yeah, you're just setting
yourself up for even more problems. So so what do you do? I
guess if you find that you're a victim of one of these breaches. I
mean, we talked about this a little bit with Solar Winds, but I'm
just curious to reiterate some of this. It's like, Okay, crap,
we got to turn it off before we get stewed into next year or whatever.
Right, So that is really probably one of the biggest challenges is what
do we do now? And it depends on what the situation is. So
for something like Solar Winds, where they weren't the attack itself wasn't disruptive because
thread actor, that wasn't their goal. Their goal wasn't to disrupt people's environments.
It was really to steal information, right, So there the disruption to
you is you've got to investigate what happened, because if you find that you
are a victim of something like a Solar Winds style attack, you've got to
try and figure out what it is that the thread actor accessed, what data
that was You may have your own disclosure laws that require you to disclose it.
Let's say you found out that they stole information that included personal information or
protected health information or whatever else sort of falls under legal liabilit things. They
in Europe you have privacy laws at the GDPR under in California you've got the
CCPA, and now other states are following suits. So it's these privacy laws
where if sensitive information which is defined under those you know in those laws,
gets disclosed by unauthorized parties, which would include you know, one of these
threat actors, then you've got to disclose. So you've got to figure that
out. You've got to find out what they did, and that means investigating
it, which means having somebody come in and help you with the forensics to
figure that out. I mean some companies, bigger companies obviously have big security
teams with forensic people on board and all that sort of sort of thing.
If you don't, that means you're out there trying to you know, hire
inside response firm to do that for you to run that investigation. So that's
you know that that's one possibility. So you know, getting back to like
what Will was saying earlier, how do you be best prepared. If you
don't have that kind of a sophisticated security team on staff, then you should
have a retainer in your back pocket for an instant response team that you know
that you can pick up the phone and say, hey guys, I need
your help, like I'm in over my head. You've got to, you
know, and you've got to have that practiced and exercised, and obviously the
contract signed and all that, all that should have happened before, you know,
before you have an incident. But if you if you're the victim of
ransomware, in some ways, it's a lot worse because now your entire business
or at least parts of your business are disrupted or offline because this ransomware is
on all your systems and you can't access anything. So now you've got to
meet this decision of do I pay the ransom which could be very very expensive,
and it's questionable about whether they would whether paying the ransomware is going to
actually make it any faster for you to recover, or do you just try
to recover on your own not pay the ransom, you know, and that's
a big decision that you know clearly is going to depend on context and what's
what's been disrupted and what you have in place to be able to recover from.
And you've got to answer all those questions. But again it's why you
should have something practiced of. Have you be tried how would you recover from,
you know, some kind of a disruptive attack like that? Like what
happens if like let's say you're a company of a few hundred or few thousand
whatever, Well, what happens if seventy five percent of your machines ever and
somewhere on them or fifty percent of your machines twenty five percent? I mean,
it's it's going to be a big number. It's not like, you
know, for most companies, it doesn't you know, you're not going to
be able to just sort of like replace all these machines and get back up
and running. It could be a matter of having to like basically wipe and
reinstall everything back on a whole bunch of machines, including data. So how
are you going to do that? Do you have the feet on the boots
on the ground to bit to do that, or similar to what we talked
about earlier, do you need to have a retainer in place with some outside
services organization that can help you recover. And this case of this the MSP
is what happens if your MSP happens to be the one that gets hit,
then what you know, do what would you do at that point? These
are these are tough situations, but I mean as sort of as disruptive as
our world has become, these are scenarios that you know, people used to
probably call this like one hundred year storm or something, and they're not anymore.
You really got to be prepared for these types of storms that can take
down your business very very quickly and for prolonged you know, a prolonged period
of time, you know. And then getting back to what we said earlier,
your biggest risk is, you know, running out of cash cash flow.
What happens if your computing environment is offline for several days? Do you
have the cash flow to survive that? You know, you've got to Those
are that also enters the equation. So those are the things you've got to
be prepared for. And there are ways to prepare for this, Like it's
you know, this is not insurmountable any stretch of the imagination, but unfortunately,
most companies that especially the ones you read about in the press are the
ones that we're not prepared for it and are really struggling when it happens.
Yeah, I think one thing that you kind of touched on that I want
to elaborate on is whenever you call in the forensics team you know, or
the experts to help you understand what the impact was, their answer is most
likely going to be I have no idea unless you've got logging turned on inside
of your system, right because you know that someone got into your network.
But unless you have like audit logging turned on between different devices. I've been
into a lot of companies that once you're inside the network, there's no tracking
anywhere to determine what you may or may not have done. Yeah, that's
a really good point. It really So here's and we could we could spend
another entire episode on this, but here in lies I think maybe the next
step. And I'm glad you sort of helped unpack this because I just quickly
said, and make sure that you've got some forensics firm on you know,
on a retainer. But they're not all created equally. They have different tools
that they use, and they have different capabilities and some groups will some teams
will just sort of come on site and they're going to use whatever you've got
set up. And if you don't have logging set up, and if you
don't have all the tools available for them, you're going to be in a
that's gonna be a difficult situation. There are companies though, that what they
will do, and so'll come into your environment and the first thing they're going
to say is we're going to ship equipment to you and you have to install
it. That's step one. Because you don't have what we need to be
able to pull the artifacts that we need to be able to pull. So
that's another approach, and depending on again, depending on your situation, that
might be the better approach for you. If you don't have a strong security
team and you don't have all these tools and processes in place, finding a
FORENDS ex firm that will simply bring all their tools with them and and then
they'll be able to pull it pull those artifacts themselves might be really necessary for
you. It's it's just one of those things you have to sort of shop
for that and know in the back of your mind what it is that you're
looking for, right what where, where are your limitations, what are you
you know, what is your organization capable of performing? And what are you
really hiring out for. You've got to be really clear on that what exactly
are you looking for in terms of that service, clear that you are in
that them that they I think that the clearer it will be, like who
are the right one or two or three firms that you want to talk to
and decide on, you know, which one do you do a retainer with?
And on that subject. Just maybe just one last thought on that is,
you know some of the retainers are you know, you you pay up
front for them, and then you know that you've got to guaranteed service level
agreement that they'll respond and you know, on time for But you might want
to have a second one, you know, sort of backup just in case.
And there are zero dollar retainers as well, where you just sign the
contract but you don't put any money up front. Obviously you don't get an
SLA with that, but it may not be a bad idea to have another
one in your back pocket because you just never know, Like this is it's
just the nature of the world that we're in right now, is you just
don't know I guess. The other thing that I'm wondering because we're talking about
Okay, you know, you'll get some forensic team to come in, you
know, make sure that you've got things set up so they can get with
the information that they need. I'm assuming you can consult with these companies ahead
of time to make sure that you have everything logging what it needs to log.
Right. This just seems kind of obvious to me, I guess.
But the other question that I have is I'd like to be proactive and not
have this problem in the first place, right, And I recognize that if
I'm leaning on another company to provide me a lot of these security services and
they get breached kind of like Solar Winds or some of these others, you
know, they're performing a function within my network, may or may not be
able to mitigate that because you just don't know who's going to get hit next.
My question is what can I be doing, Like, what proactive steps
do I need to be taking in order to I guess offset the easy the
easy stuff right where some kid with a script he downloaded off of four Chan
comes at my postcrist server, right, And the next thing I know is,
well, I forgot to update it last week and so now they're in.
Right, are there things that I can be doing to mitigate sort of
the script kitty easy? Yeah stuff? Yeah? Absolutely, And you know,
I think you know good examples of that is, I mean, how
how often you don't even read about it anymore, don't even like it just
doesn't even open people's eyes anymore. No one's even surprised when I see it.
But if you, you know, go back a couple of years,
remember how often we would see cases of like open s three buckets, right,
it had all kinds of sense of information or proprietor right information, right,
I mean it's still happening all the time. Or somebody set up like
a Mongo dB box and EC two and it's just world you know, just
you know, Internet face readable, Yeah exactly, And those are just full
of vulnerability. I mean, Mongo dB was never meant to be Internet facing,
right that you know, So how do you find that stuff? How
do you how do you defend yourself against that? And I think one of
the easiest ways, and it's it's not expensive, it's easy to do,
is to have vulnerability scanners. The two biggest product companies out there that do
this retenable and qualless, and they both do it from the cloud. You
don't even have to like install infrastructure in your environment. And the first thing
you do is just have them, you know, subscribe to it not that
expensive, and have them scanning your IP addresses, whatever external IP addresses you
own, whether it's in the cloud, whether it's in your own data center.
They're scanning it continuously and you're actually reading the report. That step two,
you have to lead their work. If you don't, then you're not
going to do anything about it. So you know, like if you've got
that Postcriss database that's that's unpatched, or this Mango deb that pops up in
EC two and it's in it's world Internet facing, the scanner finds it and
if you know, and it says, wow, here's all these critical vulnerabilities
that you've got. You see that and right away you fix it, right
you fix it before anybody else finds it. So that's a really easy way
to deal with It's just continual vulnerability scanning of all of your systems. It's
just you know, it's it's that blocking and tackling we were talking about earlier.
It's not hard. It's not expensive, but it requires a litle bit
of discipline in terms of making sure that you're always scanning all of the IP
addresses that are Internet facing that you own. Ideally, you're even scanning the
stuff that's inside of your network because you just don't know, right, You
don't just assume that because it's behind the firewall, it is perfectly risk free,
because that doesn't exist, right. It's really you should be scanning everything.
But means that you're doing that, and you've got the discipline to make
sure that the scanner is operating and it's still running, and you're seeing the
reports and you're acting on them, and it's got all of your the latest
IP addresses. So when somebody turns up a new VPC in Amazon, that
that set you know that that new network that Slash twenty four or whatever it
is that you're using is now being scanned as well. Right, that's really
critical. I mean, I would say that's step one absolutely. Step one.
Step two is making sure that you've got some kind of more sophisticated anti
virus. I hate that term, but you know, basically there's something more
right because we all know how good antivirus is but realian works well after the
fact that they add in the signature for whatever it is that you got hit
worth, right yeah, but the other stuff still floating around out there,
right right, So you have to have something and the idea is that you're
also testing it. So and even if that means bringing an the firm and
saying, hey, help me test this. What happens with this actually protect
me against ransomware? And if so, like how well does it work?
What process would I you know, what happens in one system against ransomware?
Is does this protect me against it? Not like proliferating through my network?
You know that's sort of going to work these through and do a lot of
testing and exercising. Like we were talking about with the you know, the
analogy of the football team, Like it's not like the football team is you
know, sitting back binge watching Netflix for six days a week until it's game
day. Right. They're practicing, They're watching film, they are busy,
they're you know, the most successful ones are are I mean, it's it's
a grouling schedule and that's what your folks got. I mean, they have
the grooling schedule, but they need to be practicing and exercising and all that
kind of stuff, and if they're not, you're going to get hit by
something that surprises you. Yeah, I want to I want to add a
little bit to that. When you do the port scanning, I would recommend
aggressively questioning every exposed port. For example, if you do have the postgrass
port exposed for your database server, you know, question why because and I've
seen this quite a bit in the last couple of years, people who are
using database as a service companies like, hey, sign up with our service
and we'll give you a hosted Mango database and you don't have to do any
of the maintenance on it, and then they give you a publicly exposed URL
for it, and it's like, dang, dude, you know that,
like every script kitty on the planet is just beaten on that thing, like
a rental car all day long. You know, in your car. There's
there's a better way than this, you know, because no matter you can
apply to patches within seconds after they come out, but eventually someone's going to
get through. Yeah. So, like my golden rule is no port should
be exposed except for Port eighty and its only purpose is to redirect traffic to
port or for three for your website and a port for VPN access. And
then if you happen to be hosting your own email services, you know have
port was it twenty five open to receive email, and that really should be
it. Everything else should just be highly scrutinized. And then since we deal
a lot with development, one of the other things you can do is a
big place for us to get in trouble as developers is unpatched vulnerabilities in the
packages that we use. So you know, if I'm writing code and no
JS and I install an NPM package, well that NPM packages built with dependencies
on other NPM packages, which is built with dependencies on other NPM packages.
So it's almost impossible for me to know what's actually installed on my MPM server.
But in the CICD pipeline, we can install a tool like Snake and
it will look through the manifest every time we push code to master and check
for vulnerabilities, and then it has a capability of failing to build if there
are if the vulnerabilities exceed whatever you define threshold is. And so that's a
really good way to make sure that you are checking and updating your software in
an automated process, because if it's a manual process, you're going to get
busy, you're going to forget whoever's doing it is going to be on vacation
or whatever. You know, there's all these reasons that mano processes fail.
But if you can automate that as part of your CICD pipeline, then it
just happens as a course of doing your natural daily activities. I totally agree
with that. And just to hit that one home, the Equifax breach back
in twenty seventeen, that was that stemmed from an Apache struts vulnerability, and
Equifax had patched Apache struts in other applications and they missed it in the one
that was compromised. And this all this is part of the congressional hearing.
Why I bet somebody feels dumb, right, how did we miss that?
We hadn't solved? How did we miss it? Well? Yeah, and
that CEO is no longer there and yeah seriously, But but you know,
you use something like Will is talking about, and you're not manually trying to
figure out why what was built on a patching struts I don't remember, I
mean that was a built twenty years ago. I have no idea, Right,
it doesn't matter. You have an automated process for finding that because all
those that equifacturers breached something like two or three months after the patch came out,
so there's plenty of time for that to be patched. And that brings
me to another point that I want to bring up, and it's a very
unpopular opinion, so this is the perfect format to do it. I tell
everybody I'm a moron, and here's why. I'm just kidding. I do
that all the time. Yeah, I mean it's you know, it's what
being an influencer is all about. Yah. Yeah, you pick the reasons
why people hate you. Yeah. But no, like the Equifax breach is
a perfect example. You know, they were they were vulnerable for months and
compromised millions of people's data and the final fine payout was like what, seventy
five bucks a person. It's like, oh, that's that's appropriate, Which
brings me to my whole point of this is, whenever you're using a third
party service, you know, a SaaS service, you outsource a part of
your business to them, is I think it's really important to question them on
what their obligation is when they are breached. You know, it's not if
they're going to be breached, it's when they're going to be breached. What
is their obligation to my customers? And in ninety nine point nine percent of
all SaaS agreements there is no obligation. So they can leave your customers out
high and dry, ruin your business reputation, and if you're lucky, you'll
get mentioned on a tweet from the CEO of that company whenever he apologizes publicly
for it. Okay, over, no, it's it's true. And I
don't again, you know, that's one where I don't I don't know what
the solution is because I've worked with many companies that try to do a good
job of their due diligence when they are vetting third parties right, thirty third
party relationships, But how do you vet like you know, you're doing business
with an Equifax first, they're not going to give you the time of day
anyway, because they're much bigger than just met everybody, right, But how
do you know how good their security processes are? How do you know whether
they're going to find that Apache struts vulnerability In twenty nine out of thirty of
their application servers, right, I don't know. And I think that's that's
an example of where you want to make sure that your insurance is covering you
because that's a risk you don't want to take on yourself. You can't mitigate
it, so you have to transfer it to you transfer it by buying insurance.
And you know, this is I guess out of scope for you know,
DevOps folks, But just as a thought, I mean, that's how
I mean, there's again, there's a solution to that problem, but it
is it's I think that's by making sure that you are insured properly for that
risk. Yeah, And I'm not one hundred percent certain it is out of
scope for DevOps because I think they DevOps is probably one of the few places
in the business that has enough irons in the fire in different camps to be
able to see those bigger picture things. Yeah. That's a good point,
you know, because your legal team, your legal team is not going to
know who your third party SaaS providers are. No. No, Well,
it's interesting that you bring this up though, because, for example, when
I'm dealing with like sponsors and stuff for the shows, a lot of times
the yeah, they are involved in the process of Okay, you know,
here's the contract, here's the here's what we expect, here's what we're going
to get. And I've seen companies do this with their vendors as well,
right where they do scrutinize the terms of service, and they they do scrutinize
this. But typically it's the larger companies that are going to push for more
favorable terms. Right know, you are actually going to help us with these
things when they occur. You are going to be involved at this level.
It is going to be your fault when it's your fault, right, Yeah,
you know, shared liability agreement start. Yeah, exactly. If you're
big enough you can demand those yeah. But if Yeah, for the little
guys like you and I, I mean, you're kind of stuck with whatever
they're gonna do for you, or go find another vendor that's going to do
it for you, which you may or may not be able to find,
yeah, or do it yourself, which don't have the resources to either do
or hire to do yep. Yeah. But I mean I think the bottom
line though, is that there's there really is a lot that you can be
doing to protect yourself. And I want that message to be to resonate that
it's not hopeless. This isn't an insurmountable problem. Unfortunately, It's just that
many companies are just not spending enough time and focus on the security side and
making sure that security is just part and parcel what everybody is doing in their
day to day jobs. It's not just some you know, security team off
to the side dealing with it. It's everybody, and you're practicing it,
you're exercising it. It's just sort of constant vigilance. And if all that
fails, you can always fall back on the Y two K bunker in Idaho.
That's right. If I just get a new job, we're in technology,
there's other jobs out there, I guess. I guess That's another thing,
right, is what if it's not your vendors, right? What if
it's what if it's your coworkers? At what point do you look at it?
I'll give you an example. So the company I work for, we
have this process where we take the data that we've gathered and there's a group
of business folks involved, the QA the data. Right, they make sure
that the data makes sense based on what we know about the market and things
like that. That we're gathering it from. And then what they were doing
it was there wasn't a good interface for managing that, and so they would
actually and they set all this up before I got there. I have to
disclaim that because I'm embarrassed by it. But they would export it to an
Excel sheet and then mungge it up and then make us check it back into
the codebase and run a script on it in order to import it back in.
And yeah, I had a fit and put my foot down right when
I found out about it. I was like, no, we're not doing
this anymore. This was after the cycle had ended. I said We're not
doing it again this way. And I got some looks and I got a
little bit of ribbing, a harsh treatment from it. But what point do
you look at these situations as the technical person and say this isn't secure,
or this isn't the best way to do this, this is a really dumb
idea. Yeah, well, I told them I'm not going to be liable
for this data, so you better find another way to do it. And
they took me seriously enough to where we're sitting down and actually having a conversation
about it now and they're going to need it in like two or three weeks.
But yeah, realistically, what if they told me to go jump on
a like what can I do? Do? I just do? I quit?
Do? I? I work at a company that is large enough to
actually have a security team, so I could report it, right, But
yeah, I mean, what do you do? And how serious does it
have to be before you go this just really isn't worth it? Yeah,
I think there's all of the options are on the table, and it's important
before you decide which option is for you to fully understand like the whole scope
of the thing. Not saying that you didn't, but like, just as
advice to someone listening who says, oh, I'm in this position, have
the full conversation, to sit down with whoever you can, which is probably
gonna be multiple people, and say how did it get this way? Because
most of the time those types of things, in my experience, have come
from just like tribal knowledge, and it's been decades in the process, and
at each step of the way, no one invented this. They only changed
one little piece of it, you know, and then over time you've changed
enough of the pieces where it no longer resembles the original thing that it was,
but since we've all been doing it that way all along, nobody really
picked up on that until someone comes in from the outside and gets introduced to
it for the first time and they're like, whoa, wait, what is
this? You know, so, I think it's important to have that context
of how it got to be that way, and then try to articulate the
concerns that you have about it and weigh those concerns and risks against the cost
of the cost of rectifying that. And then once you've exhausted all of those
options, now you're at the point where you have enough information to make a
decision as to whether you report it to the security team or say no,
it's this isn't the right place for me and pack your bags. Yeah.
Well, and it's interesting too write because I had the conversation with a number
of people. A couple of the people, yeah, I mean that was
effectively their response was, oh, wow, I didn't realize it had gotten
that bad, right, and so you know, no, there was no
malicious intent or anything, right, But at the same time, it was,
yeah, we definitely need to fix that, But nobody was making out
a priority until I actually put my foot down either, and so I think
there there's some trade offs and some conversations, and obviously it requires some tact
which I do not and never have possessed, but somehow we made it through
anyway, and so yeah, I think it's worth pointing out that, Yeah,
you have to have the conversations, right And at the end of the
day, I put my foot down, and I put my foot down with
my boss, you know, who's a dev manager I think is effectively his
stated title, and then the project manager. And so they went back to
the business people and said, your dev team is not going to move forward
on anything else after a while until this is solved. So if you want
to be able to use this process, we've got to come up with a
way for you to do it that they can implement for you, because the
way we have been doing it isn't going to happen. And right, and
so those conversations did happen and it did go the way that it needed to.
But yeah, I just want to add to that, Yeah, have
the conversations. I probably could have been a little more tactful in my approach,
but at the end of the day, I think at some point you
got to put your foot down and just say, look this just this opens
us up to all kinds of problems, and we're either going to do this
the right way or somebody else is going to be doing it the wrong way,
because it's not going to be me. Yeah, but yeah, I
did want to reiterate your previous point. Yeah. A couple of people said,
oh, I didn't realize that it had gotten to that point, right,
Yeah, And that's just been my experience, you know, is it's
somebody creates this thing, sets it loose in the wild. It's like this
old story. I can't remember where it happened, but they put these monkeys
in a room with the banana on top of the ladder, and every time
a monkey went up to get the banana, they posed it down with a
fire hose, and so over time, whenever one of the other monkeys would
go up, the other monkeys would drag him back down. Then they started
replacing the monkeys one at a time. The time they had replaced all of
the monkeys, and all the monkeys knew if anyone goes for the banana,
to drag that monkey off the ladder. Although no one knew why anymore,
right, And then the summary to that is that's how corporate policy gets created.
Yeah, I mean we are talking to people though within organizations. So
how do you start having the conversations about this stuff? Right? Not necessarily
the kinds of things that I'm talking about, but maybe more along the lines
of setting policies and setting up automations and things like that that you guys have
brought up. If they're not doing it, how do you go to them
and say, no, we need to start doing this, or how do
you start pushing them to start doing things that they've never done before, or
pushing people on your own dang deam. It's tricky, right, because that
comes down to like political skills for lack of a better term, you know,
and what type of politician are you? Are you Teddy Roosevelt where you're
going in with a stick and beating them into submission, or you you know
in Abraham Lincoln that can can vince them with words and you know, sell
them on their own virtues. So you've got to know what your own personality
and your own strengths are. But either way, it starts with communication,
you know, and highlighting the problem getting getting you have to understand what their
perception of that is, and then you have to be able to articulate to
them what your perception of the risk is, so that everyone has the common
ground. Yeah, that makes sense, and it's a lot easier said than
done. I rolled that out in about sixty seconds, but in reality or
in practice, that could take weeks or months. And if you've never done
that before, you can expect to fail the first couple of times, which
leads to its own set of frustrations, because then you're like, damn,
I went and tried what that dude said, and now nobody implemented my solution
and they think I'm a jerk. Yeah, I will definitely add to that,
though. It does help to know what your strengths are. It sounds
like Jeffrey's trying to chime in. So I'm just gonna say what I was
gonna say. I'm kind of a blunt object and I know that, and
so I know that my approach at some point, relatively quickly is going to
devolve into no, we need to do this, or I'm gonna quit my
way or the highway. Hey, it works works a lot. I was
going to say that I think it's that's the situation where you sort of realize
you'll find out really quickly what the culture is like in your organization. Right.
For instance, like one of the tenets of DevOps, right is the
idea of being a learning organization, right, a continually learning at organization.
And if that's really the culture, then you if you bring up an issue
like this, that's gonna work, right. I mean, you know there's
gonna be mechanisms already for you to be able to do that. For the
vast majority of us who don't work for organizations like that, it's more of
a challenge. But I think it's also perhaps an opportunity to help your organization
and say, listen, here's a here's a problem. And oh, by
the way, I'm only finding one problem. I'm sure there's others lurking our
culture. Really we should be encouraging people to bring up these kinds of issues
and finding better ways of doing things so that we are a so we can
become a learning organization and we can continually do better. Yeah. Well,
and it's and for the most part I found that most people if you can,
if you can explain why, then most people will at least hear you
out, and so it's only come down to we're doing we're not doing this
this way, or we are doing it this way, or I'm quitting.
It's only come down to that once or twice ever in my fifteen year career.
Right. Most of the time, you give them a good reason and
people are going to go, yeah, yeah, we don't we don't want
to have that problem, and so they'll right, they'll figure it out,
yep. And if it really does come down to that, I also just
want to point out that, yeah, you don't have to accept the liability
for those issues. You can go find another place to be. Yep.
It's one of the fortunate things about working in tech these days is there's a
lot of jobs. Yeah, yep, yep. Absolutely, all right,
Well, I think we've kind of exhausted our time, and then some is
there anything that we should make sure that we include that we didn't talk about
before we're you know, so yeah, I just want to leave something out
and then be like, oh, and make sure that you say this when
you I don't know, right, This is my secret weapon is every time
I bring donuts in, when I know I'm going to have the hard conversation,
I don't know anyway. By the way, donuts really do work there
miracle food. But yeah, let's go ahead and do picks then, Jeffrey,
do you want to start us off? I had a feeling you were
going to say that will Do you want to start us off? Man,
I'd love to start us off. I've got this pick today. We're just
going to seem out of the ordinary because normally I say really profound stuff,
but I'm going to open this one up with As humans, it turns out
we have a dependency on oxygen. And so this book I've been reading is
called The Oxygen Advantage by Patrick McEwan. So this is actually pretty cool.
I mean it sounds like, based on what I just told you is like,
really this is where you're going, But it's actually kind of cool.
I'm a certified scuba diver, a certified free diver. I rode twenty four
hour mountain bike races for a number of years and competed in that, and
so I spent a lot of time focusing on my breath. And I just
got this book a couple of days ago, and I'm just plowing through it
because what the guy is going through here is talking about how your body utilizes
oxygen. And since we breathe by default kind of out of necessity, we
never really focus on improving our breathing. And because our habits have changed over
the last thousands of years where we don't really do a lot of physical labor
or not on the move a lot anymore, we actually don't breathe in line
with the way that our bodies should be. And as a result, people
who are breathing what turns out to be way too much. And so he's
got these exercises in here to help you lower your oxygen intake, increase your
oxygen utilization, and increase your carbon dioxide, which improves the efficiency of your
muscles. So I think it's been a pretty fascinating read for me. I
haven't finished it yet, but I was so excited I wanted to make that
my pick this week. And I think it applies to everyone who whether you're
if you're an athlete, or you're doing a lot of physical activity, it'll
be specifically beneficial to you, But even if you're not and just thinking that
you want to be more active, it's got some tips and tricks in there
that will help you focus and refinething there as well. Awesome, I'll have
to check that out. Well, I'll talk about on my picks, Jeffrey,
do you have some picks? Yeah. So just following up similar to
to Will's. I read a book a while ago called micro Resilience, and
it's sort of like the idea behind it is that we sort of understand the
idea of macro resilience of having like an exercise routine and that sort of thing
that's sort of like long term, like these are the things that sort of
keep me in shape and keep me active and keep me healthy and all that
sort of thing. But the idea of micro resilience was the idea that,
like, what are the things that we can do sort of during our day
that are not necessarily to try and you know, develop muscle tone or aerobic
exercise or something like that. But you know, there's just those moments during
the day when we just feel exhausted or we've just feel overwhelmed or whatever it
is that just and a couple of things that she brings out in the book
is one it's like, you know, sometimes you feel hungry during the day,
and her gument is a lot of times it's not that you're hungry.
It's actually that you're thirsty. And many times if you just drink, you
know, just ta take some time and like drink down big glass of water
or something like that, that it actually sort of re energizes you. And
I have personally found that that really helps instead of getting you know, sort
of looking for like the quick energy bar or something like that, just drink
some water. It's huge. Or you know, shows talking about like just
standing up from your chair and not just like standing up, but actually do
some physical activity, like you know, moving your arms around, moving them
up up above your head, that sort of thing. Again, just sort
of getting your blood flowing, and it just sort of re energizes you,
sort of rejuvenates you. Like another example was, you know, sometimes you're
on the road. I've been on the road much in the last year and
a half. But if you're on the road, you're staying in a hotel
and they have those like little swimming pools. She's like, you know,
even if you just like it starts your day off in the morning by just
jumping the pool doing a couple laps. Again, it's not for endurance or
anything like that, but it just gets your blood flowing and you know,
it sort of helps you just start your day off. So I think there's
a lot of really good ideas of just how to sort of deal with those
sort of lulls that you get during your day and rather than grabbing, you
know, a chocolate bar or an energy buyer or something my dad like,
finding other ways that are probably healthier that will help you sort of get your
energy levels back off. I love it. And to be perfectly honest,
that's one of the things that I've done lately to help mitigate some of the
tension headaches I've had is just drinking water. I don't know what it was,
but I cut back on the energy drinks and sodas I was drinking and
just started drinking more water. And that's made a huge, huge difference for
me. So yeah, yeah, I'm going to throw in a few picks
of my own. So what I was going to say on Wheel's pick about
oxygen was just that, and I think I've mentioned it on the show before,
but one of my goals is to complete an Iron Man, and so
I've been just getting out and swimming and running and biking, and to be
perfectly honest, it's been interesting. I was a swimmer in high school and
did a little bit in college, and so you know, kind of my
breathing patterns kind of stem from that, even when I'm running your biking,
but I have some friends who are who are runners all their lives, right,
and then got into triathlons and started swimming, and just the cadence of
movement and swimming and stuff has anyway, it has changed the way that they
breed and the way that they exercise and things like that. And so it's
been interesting to me to just see how all that comes together. And I'm
interested to see within this book as far as picks go. I found so
my swim coach, because I'm on a swim team now. In the morning
I go to swim practice. I had an equipment issue with some of my
fins. Apparently she wanted me to have longer fins, which are more work
to swim with but also make it go faster and help keep you on top
of the water. She sent me a link to some fins. I'll put
a link in the show notes. But they're kind of a little bit longer,
but they're not like the really really long scuba fins. They're more like
snorkel fins, and so I'm going to pick that just because they're nice to
swim with, but they're not as big time, heavy duty as some of
the scuba fins. And then I think I may have mentioned this last week,
but I'm still reading Atlas Shrug and I'm really digging it. It's funny.
I picked it on JavaScript Jabber and one of the other hosts basically said,
don't take it too seriously, but I find that I agree with a
lot of the opinions in there. So I'm going to pick that just because
I've really really been enjoying it. And then I've picked up a new book.
I've only read the forward and a little bit of the first chapter,
but it's already appealing to me, and it's called The Ruthless Elimination of Hurry
by John Mark Comer, and just talks about getting what you want from life
and what you can do to eliminate a hurry from your life and some of
the stress that you have around some of the stuff that you're probably trying to
accomplish with life. So anyway, I'll put links to all of those in
the show notes. Cool good stuff, all right. Well, with that,
I guess we'll wrap up this was We've had some really great conversations lately.
I've really enjoyed these, and especially just being able to sit and chat
and go through some of this stuff related to some of these concerns and breaches
has been great as well. So anyway, we'll just wrap up here and
until next time, folks max out