1
00:00:14,359 --> 00:00:18,679
Hey everybody, and welcome back to
another episode of Adventures in DevOps. This

2
00:00:18,719 --> 00:00:22,719
week, on our panel, we
have will Button. What's going on everybody?

3
00:00:22,839 --> 00:00:26,000
So I have Jeffrey Grumman. Hey
there, I'm Charles Maxwood from dev

4
00:00:26,079 --> 00:00:29,480
chat Do TV. And we thought, well, we had a guess not

5
00:00:29,559 --> 00:00:34,119
show up, and then Jeffrey's telling
us about all the horrible but fascinating things

6
00:00:34,159 --> 00:00:37,240
that are going on on the Internet
these days, and so we're going to

7
00:00:37,320 --> 00:00:42,200
talk about more breaches and concerns and
stuff. Is going to freak me out,

8
00:00:42,320 --> 00:00:47,600
but I'm kind of curious in a
morbid way to hear about so I

9
00:00:47,719 --> 00:00:53,840
guess, I guess let's just dive
in. So, Jeffrey, what's broken

10
00:00:53,840 --> 00:00:56,439
in the world? Oh? Man, I gotta tell you. It's been

11
00:00:56,439 --> 00:01:00,039
a crazy week. I think that
started out last weekend and a piece of

12
00:01:00,079 --> 00:01:07,000
software called Case, which is used
by different managed service providers, was found

13
00:01:07,040 --> 00:01:11,799
to be carrying malware in terms of
ransomware. And so basically any managed service

14
00:01:11,840 --> 00:01:15,439
provider who uses this case to like
manage their customers or the client environments,

15
00:01:17,040 --> 00:01:19,359
all of a sudden were just locked
up. And that means that they can't

16
00:01:19,519 --> 00:01:22,920
you know, that means they themselves
are locked up, and that means that

17
00:01:22,959 --> 00:01:26,519
they can't service their clients. So
basically you're like, you know, if

18
00:01:26,560 --> 00:01:30,159
you're using an MSP that gets that
it uses cassea, you're basically like you

19
00:01:30,239 --> 00:01:34,599
have no service, Like you have
no IT service provider anymore because they're just

20
00:01:34,040 --> 00:01:38,439
their host. So that's how we
started the you know, the week coming

21
00:01:38,920 --> 00:01:42,480
out of the long weekend, the
holiday weekend. And then Microsoft, I

22
00:01:42,519 --> 00:01:49,799
think a couple of researchers found a
really crazy remote code remote code execution RCEE

23
00:01:49,319 --> 00:01:55,959
vulnerability in Microsoft Windows Principuler. So
there's a lot there. Maybe it's just

24
00:01:56,000 --> 00:01:59,159
worth us a couple of minutes in
talking about what that means. So if

25
00:01:59,159 --> 00:02:02,840
you're not familiar with Microsoft Windows,
going all the way back to the beginning

26
00:02:02,879 --> 00:02:07,040
of time, one of the biggest
things that people used to do was print,

27
00:02:07,319 --> 00:02:10,719
and if you weren't printing off of
your own machine, you were printing

28
00:02:10,759 --> 00:02:15,719
off of a print server, which
is very common if you're not familiar.

29
00:02:15,759 --> 00:02:21,560
Printers are these big, oblong plastic
devices that spit paper out from time to

30
00:02:21,639 --> 00:02:24,840
time or toning. Right. It's
funny because these are like this is like

31
00:02:24,960 --> 00:02:30,960
legacy stuff, but it still is, right, there inside of every Microsoft

32
00:02:30,400 --> 00:02:35,479
Windows whether it's like the desktop version
or the server version, they're all running

33
00:02:35,599 --> 00:02:39,400
the print spooler. It's a service
that runs by default and it just has

34
00:02:39,439 --> 00:02:43,759
all this legacy software in it because
it's just been around for so long.

35
00:02:43,960 --> 00:02:46,400
And so some researchers have been spending
time looking at that and they found this

36
00:02:46,639 --> 00:02:51,120
again, the remote code execution.
What does that mean? That means that

37
00:02:51,159 --> 00:02:55,400
I don't have to be connected to
the machine to be able to like literally

38
00:02:55,479 --> 00:03:02,599
like logged into the Windows guy to
be able to exploit that vulnerability, And

39
00:03:02,800 --> 00:03:07,800
the exploit of that vulnerability gives me
what's called system privileges. A system is

40
00:03:07,840 --> 00:03:13,319
basically like a service account version of
like your local administrator. So it's an

41
00:03:13,360 --> 00:03:20,680
elevated privilege that certainly could be used
for all kinds of nefarious purposes. Interesting

42
00:03:20,719 --> 00:03:23,159
thing about it, though, is
that that I think, if I remember

43
00:03:23,199 --> 00:03:28,120
right, or if I understood this
correctly, you have to have a legitimate

44
00:03:28,199 --> 00:03:31,719
user account on the box to be
able to take advantage of this, Which

45
00:03:31,759 --> 00:03:37,080
makes sense because the Prince Spooler is
typically not something it's not like a web

46
00:03:37,120 --> 00:03:43,439
server that any anonymous connection can use
utilize do something make requests of et cetera.

47
00:03:43,719 --> 00:03:47,639
The Prince booler is only available to
users of that system, so you

48
00:03:47,800 --> 00:03:53,439
have to have a Windows user account
already. So that was something that you

49
00:03:53,439 --> 00:03:54,759
know, when you think about like
the risk side of this. Okay,

50
00:03:54,800 --> 00:03:59,800
how do I prioritize or what do
I think about this vulnerability? Okay,

51
00:04:00,000 --> 00:04:03,199
Well, if it means that somebody
has to have local access or you know,

52
00:04:03,520 --> 00:04:08,560
an account of the box already,
then maybe some of my systems I'm

53
00:04:08,599 --> 00:04:12,639
less concerned about. But maybe if
I have a Windows system that is exposed

54
00:04:12,639 --> 00:04:15,759
to the Internet, I might be
more more concerned about that, especially if

55
00:04:15,800 --> 00:04:20,920
it's let's say, like a SharePoint
server right that's exposed to the Internet,

56
00:04:21,199 --> 00:04:27,879
where somebody out there might have a
user account on my box for whatever reason.

57
00:04:28,199 --> 00:04:30,560
So then I'd probably be more concerned
about it. But yeah, through

58
00:04:30,600 --> 00:04:33,480
this and it went on for days
off, I'm just trying to figure out

59
00:04:33,480 --> 00:04:38,759
how we fix it. Microsoft was
trying to figure out workarounds. Researchers were

60
00:04:38,800 --> 00:04:42,920
trying to find out workarounds. Can
you just shut down the print spooler service?

61
00:04:43,240 --> 00:04:46,680
But that's it wasn't enough. That
actually doesn't stop the vulnerability from being

62
00:04:46,720 --> 00:04:51,839
exploited. I think finally, today
I saw an update that Microsoft provided an

63
00:04:51,879 --> 00:04:57,040
out of band patch, out of
band meaning not part of their like patch

64
00:04:57,079 --> 00:05:00,079
Tuesday, which is I think a
monthly cycle, so dead it's out of

65
00:05:00,160 --> 00:05:05,360
band to that they just released a
patch for I think for basically every version

66
00:05:05,360 --> 00:05:08,759
of Windows. So if you haven't
gotten that, if you're not aware of

67
00:05:08,800 --> 00:05:13,040
this, you can certainly download it
from Microsoft. But yeah, that's basically

68
00:05:13,120 --> 00:05:15,800
it. Then nuts all though,
I think there's a lot there to talk

69
00:05:15,800 --> 00:05:19,040
about. So it sounds like if
you just delete all your user accounts though,

70
00:05:19,120 --> 00:05:23,839
then it can't use to exploit anything, right, that is correct.

71
00:05:23,959 --> 00:05:27,240
You know we used to talk about
you know, this is way back in

72
00:05:27,279 --> 00:05:31,240
the bill before the bill gates famous
letter about writing secure software. We used

73
00:05:31,279 --> 00:05:34,920
to joke that the best way to
secure a Windows server was to fill the

74
00:05:35,000 --> 00:05:41,759
footprint with cement and then bury it
in the ground. So deleting every user

75
00:05:41,800 --> 00:05:48,199
account on the device is similar to
that approach. It is it is no,

76
00:05:48,399 --> 00:05:53,759
yeah, no, no, yes, we're paid on results. Yeah.

77
00:05:54,120 --> 00:05:58,000
Yeah. Apparently it's been according to
Microsoft and SIS, which is there's

78
00:05:58,040 --> 00:06:03,279
a government agency called the ci A
and they sort of track these vulnerabilities across

79
00:06:03,279 --> 00:06:08,120
all vendors and provide bulletins and that
sort of thing, and they claim that

80
00:06:08,399 --> 00:06:12,879
these have been exploited or there have
been exploits seen in the wild. I

81
00:06:12,920 --> 00:06:16,800
haven't heard of any specific breaches based
on this, but clearly like something this

82
00:06:16,959 --> 00:06:21,720
big and this that's like sort of
easy to exploit vulnerability is going to be

83
00:06:21,879 --> 00:06:25,160
you know, there's going to be
folks out there that are basically going to

84
00:06:25,199 --> 00:06:28,399
be testing, you know, every
system they can to see if they can

85
00:06:28,439 --> 00:06:32,120
exploit it. So so yeah,
because because some of the people who are

86
00:06:32,399 --> 00:06:38,959
vulnerable to this are used are relying
on the MSPs to provide their IT support

87
00:06:39,000 --> 00:06:44,600
if you can't do that because they've
been locked out my ransomware right right,

88
00:06:45,439 --> 00:06:47,720
It is the perfect storm. So
again we're back to deleting all the user's

89
00:06:47,720 --> 00:06:51,040
accounts is really our only option.
Absolutely absolutely, you know, going back

90
00:06:51,079 --> 00:06:56,439
to pen and paper forms in triplicate, if you still have them in a

91
00:06:56,480 --> 00:07:00,199
box in your closet, you might
want to bring them out stock in carbon

92
00:07:00,240 --> 00:07:05,000
paper this afternoon. I think that's
where we're going. Yeah, I didn't

93
00:07:05,040 --> 00:07:10,240
know you mentioned before the show,
Will that you you went you were paying

94
00:07:10,279 --> 00:07:14,319
attention to some of the I guess
recount stuff in Arizona, and they somebody

95
00:07:14,319 --> 00:07:17,279
mentioned that they should go back to
paper ballots. Yeah, And I've been

96
00:07:17,319 --> 00:07:21,800
to a number of political I'm fairly
involved in Utah politics, and yeah,

97
00:07:21,879 --> 00:07:25,759
there are a few more than a
few people we've gone to electronic voting,

98
00:07:26,000 --> 00:07:30,279
especially during COVID right where we couldn't
actually actually get together and vote with paper

99
00:07:30,360 --> 00:07:33,399
or anything that looks like paper.
And there are so many people that have

100
00:07:33,439 --> 00:07:35,920
brought up I just want to go
back to paper because I know you can

101
00:07:35,959 --> 00:07:40,519
count it, and not to get
political and not to go into any of

102
00:07:40,600 --> 00:07:45,560
the implications of how that can go
wrong, right, But it's it's interesting

103
00:07:45,639 --> 00:07:50,600
that yeah, Chads my friend ha, Yeah, well yeah, nobody on

104
00:07:50,639 --> 00:07:55,560
the outside can hack the paper,
but that there are humans counting it.

105
00:07:55,720 --> 00:07:58,959
So, I mean, it's just
it's really interesting that a lot of this

106
00:07:59,040 --> 00:08:03,800
really comes down to what users do
and how they approach it, and at

107
00:08:03,839 --> 00:08:07,680
some level there's trust, right,
And I think that's the point is delete

108
00:08:07,680 --> 00:08:11,720
the user's account, is we're kicking
all the people off that we don't trust.

109
00:08:11,839 --> 00:08:16,160
Yeah, yeah, I mean it's
interesting. Well, and I think

110
00:08:16,439 --> 00:08:20,279
just to round out that the discussion
about voting, I mean, I once

111
00:08:20,319 --> 00:08:22,480
in a while, I will hear
somebody talk about, hey, we should

112
00:08:22,480 --> 00:08:26,480
make it even easier to vote.
Let's why why aren't we doing it?

113
00:08:26,680 --> 00:08:28,720
Why aren't we letting people vote over
the internet or over their mobile phone?

114
00:08:30,040 --> 00:08:33,000
And if you hadn't thought about it
in the past, like these two examples

115
00:08:33,039 --> 00:08:39,120
by themselves, like of the vulnerabilities
and just how easy it is to manipulate

116
00:08:39,279 --> 00:08:45,000
or to break in, should give
anybody pause about thinking about having something like

117
00:08:45,120 --> 00:08:48,360
voting, something as critical to society
as a whole, to put it online

118
00:08:48,440 --> 00:08:52,080
or to put it on a mobile
device or which is also online. But

119
00:08:52,360 --> 00:08:54,639
you know what I'm saying, Like, it's just it's so difficult to try

120
00:08:54,679 --> 00:08:58,039
and protect these things, especially when
you talk about third party software. Right,

121
00:08:58,080 --> 00:09:01,159
you're running in a window, or
you know, you're running your business

122
00:09:01,240 --> 00:09:05,840
on a piece of third party software, and you don't know, you don't

123
00:09:05,879 --> 00:09:09,480
know how well that company. I
mean, listen, when we're talking about

124
00:09:09,480 --> 00:09:13,519
Microsoft, they have and they now
have a long track record of, you

125
00:09:13,559 --> 00:09:16,960
know, really trying to be on
top of the security, and I think

126
00:09:16,000 --> 00:09:20,600
they've but how many lines of code
are in you know, Microsoft Windows.

127
00:09:20,639 --> 00:09:24,360
I mean, we're talking about it
enormous, right, oh yeah, chunk

128
00:09:24,399 --> 00:09:28,039
of work there. So it's and
and we don't know, like it's that

129
00:09:28,039 --> 00:09:31,559
that really is the challenge, Like, you know, you try to think

130
00:09:31,600 --> 00:09:35,679
about the risk to your business and
how to quantify risk, and when you're

131
00:09:35,720 --> 00:09:39,360
talking about putting a piece of software
and running your business on that piece of

132
00:09:39,360 --> 00:09:41,919
software, like you have no idea
what that risk really is. Well,

133
00:09:43,279 --> 00:09:46,080
and you talk about you brought up
the voting and just the security of all

134
00:09:46,120 --> 00:09:50,559
the things involved, but even down
to the user devices, I mean,

135
00:09:50,559 --> 00:09:54,120
how many of those are compromised that
people just don't know about. Yeah,

136
00:09:54,200 --> 00:09:58,399
it's it's so fascinating to me just
to see all the different levels that this

137
00:09:58,519 --> 00:10:03,360
goes to and yeah, what what
it effectively boils down to, right,

138
00:10:03,559 --> 00:10:07,639
I Mean it's always talked about security
versus you know, sort of the usability

139
00:10:07,720 --> 00:10:13,480
the convenience of it, and security
always will get in the way of usability

140
00:10:13,480 --> 00:10:16,279
and convenience. I mean we see
that in a walks of life, you

141
00:10:16,279 --> 00:10:20,399
know, whether it's physical security where
you have to go through a security checkpoint

142
00:10:20,600 --> 00:10:22,600
or something like that and you've got
to take off your shoes or you know,

143
00:10:22,639 --> 00:10:26,279
whatever it is that they're making you
do. There's always those inconveniences,

144
00:10:26,279 --> 00:10:30,039
but you know, depending on what
you're doing. I think that's the important

145
00:10:30,039 --> 00:10:33,600
part for anybody running a business,
is what you know sort of what due

146
00:10:33,639 --> 00:10:37,159
diligence are you doing or what how
are you thinking about the convenience versus the

147
00:10:37,440 --> 00:10:43,679
usability or versus the resilience, right, convenience of just buying some piece of

148
00:10:43,679 --> 00:10:50,679
software or downloading a piece of software
versus the disruption the potential disruption to here

149
00:10:50,759 --> 00:10:54,279
the entire business, and that has
to be well thought out, I think

150
00:10:54,320 --> 00:10:58,200
today, especially when we're doing so
much more online. I mean, remember

151
00:10:58,240 --> 00:11:03,120
the days when it was not that
unlikely, or there wasn't that uncommon that

152
00:11:03,200 --> 00:11:07,080
you go into a store and the
credit card swipe device thing was offline because

153
00:11:07,360 --> 00:11:11,840
the phone line was down or remember
that, And so they would take out

154
00:11:11,840 --> 00:11:16,519
from under the counter, they'd take
out the dow hiki that goes back and

155
00:11:16,559 --> 00:11:22,080
forth over the carbon right, And
everybody still have those today. That happens

156
00:11:22,080 --> 00:11:24,399
in your store. I was in
a home depot the other day and they're

157
00:11:24,480 --> 00:11:28,720
like the manager was like running around
the aisle saying sorry, sorry that we

158
00:11:28,799 --> 00:11:35,200
can't can't check anybody out right now. I don't know why there's no work

159
00:11:35,240 --> 00:11:39,480
around today. People don't really have
that sort of thing. So, yeah,

160
00:11:39,039 --> 00:11:43,080
I was at an if A country
store and which is just kind of

161
00:11:43,080 --> 00:11:46,480
a local farm store here, and
what they were doing is they were they

162
00:11:46,480 --> 00:11:52,080
were basically grabbing They jot down your
credit card information and tell you that they

163
00:11:52,080 --> 00:11:54,639
were going to charge your card later, which didn't necessarily make me feel good.

164
00:11:54,840 --> 00:11:58,120
But then they gave you an invoice, a copy of the invoice right

165
00:11:58,320 --> 00:12:01,919
right, which was also fascinating in
its own way because it was like,

166
00:12:03,159 --> 00:12:07,159
I don't really you know, and
yeah, then yeah, the charge came

167
00:12:07,159 --> 00:12:11,879
through later, and they're like,
yeah, don't leave that invoice anywhere because

168
00:12:11,919 --> 00:12:15,480
if you do, it's going to
be a problem, right, And yeah,

169
00:12:15,559 --> 00:12:18,639
what do you do? Yeah?
And again I I just don't.

170
00:12:18,799 --> 00:12:22,639
I don't. I think that's that's
just sort of a short, shortballower or

171
00:12:22,720 --> 00:12:26,519
just something that has not been really
thought through in many businesses, many you

172
00:12:26,519 --> 00:12:30,159
know, many companies, of just
what happens if you know something some key

173
00:12:30,159 --> 00:12:33,919
piece of software that we rely upon, and sometimes it's not even something you

174
00:12:33,919 --> 00:12:37,480
you even think about or realize.
Like let's go back to the solar winds

175
00:12:37,799 --> 00:12:45,200
example from right from January or or
last December where it wasn't that wasn't a

176
00:12:45,320 --> 00:12:50,120
ransomware that was linked back to somewhere
some group within Russia. And you know,

177
00:12:50,120 --> 00:12:54,960
it seem like they were going after
very specific companies and government agencies,

178
00:12:54,240 --> 00:12:58,360
but still very disruptive to your business. And you didn't even think about Solar

179
00:12:58,399 --> 00:13:01,000
Winds. Most people didn't even think
about didn't even have an idea like who

180
00:13:01,080 --> 00:13:03,879
is solar wind Who haven't even heard
of this company? But it's a piece

181
00:13:03,919 --> 00:13:09,840
of software that so many IT shops
are using to manage their own infrastructure,

182
00:13:09,120 --> 00:13:13,200
servers, desktops, network gear,
all that stuff, which made a prime

183
00:13:13,519 --> 00:13:20,960
candidate for being for hacking into it
and putting basically embedding your malware into that

184
00:13:20,000 --> 00:13:24,639
piece of software, because it's just
it's ubiquitous, like so many companies use

185
00:13:24,759 --> 00:13:31,159
that. So if you're sitting in
the business office, a business side of

186
00:13:31,159 --> 00:13:33,240
things, you're not even going to
think about some piece of software that your

187
00:13:33,240 --> 00:13:39,799
IT people use and how that can
really disrupt your entire business or whatever it

188
00:13:39,840 --> 00:13:43,320
is that you do. If you're
a government agency or your entire agency to

189
00:13:43,360 --> 00:13:48,480
where they basically take themselves offline and
investigate what happened and figure out what's going

190
00:13:48,519 --> 00:13:52,759
on, and you know, it's
just so disruptive to your entire IT organizations,

191
00:13:52,879 --> 00:13:54,840
everything else you're trying to do or
the IT folks were trying to do

192
00:13:56,360 --> 00:14:01,519
gets put on hold until you can
fix this problem investigated or do what you

193
00:14:01,600 --> 00:14:05,679
have to do. Yeah, but
yeah, that's the I mean, it's

194
00:14:05,759 --> 00:14:09,600
the it's a situation that we're in
today, and I think it certainly plays

195
00:14:09,600 --> 00:14:13,480
into when we think about like DevOps
or dev zecops. It certainly has to

196
00:14:13,480 --> 00:14:18,799
be part of what we're doing.
Right. So, if you're building software

197
00:14:18,799 --> 00:14:24,480
and you're building it on top of
open source libraries or connecting to you know,

198
00:14:24,519 --> 00:14:28,519
somebody else's API or whatever else,
and it's probably several other scenarios,

199
00:14:28,879 --> 00:14:33,159
But what are you doing, like
what how are you thinking about the security

200
00:14:33,240 --> 00:14:37,679
of that piece of software or that
service that's being provided, whether it's a

201
00:14:37,720 --> 00:14:41,200
SaaS service or what have you.
What happens if it's down? That's probably

202
00:14:41,360 --> 00:14:46,919
one of the most most obvious scenarios. But what happens if they they themselves

203
00:14:46,919 --> 00:14:50,799
get penetrated by these threat actors?
Then what how would you detect it?

204
00:14:50,879 --> 00:14:54,440
Do you have the ability to detect
it? How would you write? How

205
00:14:54,440 --> 00:14:58,000
would you respond? To this,
And I think the first step is just

206
00:14:58,039 --> 00:15:03,279
sort of thinking through that process,
because the truth is is that you can,

207
00:15:03,519 --> 00:15:05,559
you know, you can detect these
things and you can respond to them,

208
00:15:07,000 --> 00:15:09,159
but not if you're not prepared.
Right. It's if you know,

209
00:15:09,399 --> 00:15:13,320
the first time you even hear about
it is because the FBI is calling you

210
00:15:13,360 --> 00:15:18,279
and saying, hey, you've got
solar winds and we've tracked that the bad

211
00:15:18,320 --> 00:15:20,960
guys are in your environment. It's
too late to try to figure out how

212
00:15:22,000 --> 00:15:24,480
do we detect it at that point, right, it's now you're just all

213
00:15:24,480 --> 00:15:28,080
hands on deck. It just behooves
organizations. So really start to think about

214
00:15:28,120 --> 00:15:33,960
this, especially with just the number
of these situations. Ransomware is. It's

215
00:15:33,000 --> 00:15:37,519
just a perfect example. It's so
lucrative. It's not going anywhere, no

216
00:15:37,559 --> 00:15:41,279
matter how many times Biden sits down
with Putin and talks about how ransomware has

217
00:15:41,320 --> 00:15:43,799
got to you know, they've got
to put a stop to ransomware. I

218
00:15:43,840 --> 00:15:48,320
mean Obama tried to do that with
with she you know, back in the

219
00:15:48,399 --> 00:15:50,879
day in Shinna several years ago.
And you know, it doesn't stop,

220
00:15:52,000 --> 00:15:54,120
and it's not going to stop.
So I just think these are examples of

221
00:15:54,600 --> 00:15:58,080
that everyone's got to take the heart
and figure out, Okay, what is

222
00:15:58,279 --> 00:16:00,600
where is the risk to my organization
and what are the kinds of things that

223
00:16:00,639 --> 00:16:04,519
we can be doing to mitigate that
risk. Yeah. I think one of

224
00:16:04,519 --> 00:16:08,360
the things you can do that I
don't know of a lot of companies that

225
00:16:08,480 --> 00:16:15,120
do this is implement a full on
disaster recovery and preparedness plan. You know,

226
00:16:15,200 --> 00:16:19,759
I've worked at a few companies in
the past where we would you know,

227
00:16:19,759 --> 00:16:25,840
we would actually take our backup tapes
off site and go to restore them

228
00:16:25,879 --> 00:16:29,759
onto this is when we still have
physical data centers and restore them onto new

229
00:16:29,840 --> 00:16:33,840
service just to verify that we could
bring things back up. And a lot

230
00:16:33,840 --> 00:16:37,960
of that practice came from whenever I
was in the Navy, because that was

231
00:16:37,159 --> 00:16:40,840
a large part of what we did
there, Chris, the stakes were a

232
00:16:40,840 --> 00:16:44,000
little different, but the principle's the
same. You really don't know what the

233
00:16:44,000 --> 00:16:47,759
missing pieces are until you go out
and try to do that, and that's

234
00:16:48,159 --> 00:16:49,759
that's the case. I think.
I think that's a situation that a lot

235
00:16:49,840 --> 00:16:55,799
of these businesses find themselves in is
whenever something happens and they are down,

236
00:16:56,120 --> 00:16:59,600
that's when the conversation starts, Oh
what do we do now? Well,

237
00:16:59,639 --> 00:17:04,960
the first was to have this conversation
before, right before the first step to

238
00:17:06,359 --> 00:17:10,599
getting out of a burning building is
to plan your exit before the building's on

239
00:17:10,680 --> 00:17:15,440
fire. Yeah, it's so true. I mean, quote to that to

240
00:17:15,519 --> 00:17:19,319
that point. How often have you
seen people that had an entire backup strategy

241
00:17:19,799 --> 00:17:25,000
never tested their backups, have an
issue happened right sert of those down or

242
00:17:25,000 --> 00:17:27,720
whatever you got to restore to the
new system, and then they find out

243
00:17:27,759 --> 00:17:32,000
that the backup actually didn't work,
or the restore process doesn't work, or

244
00:17:32,039 --> 00:17:34,079
whatever they thought was on the tape
doesn't exist or right. I mean,

245
00:17:34,119 --> 00:17:38,880
it's yeah, it is so common. Yeah, I've seen it a lot

246
00:17:38,920 --> 00:17:42,640
and been guilty of it a lot
myself. Even you know, even after

247
00:17:44,119 --> 00:17:45,400
like the first time is like,
oh that was a painful lesson. I'm

248
00:17:45,400 --> 00:17:52,680
gonna I'm gonna learn this time,
and now I don't, because every it's

249
00:17:52,680 --> 00:17:56,519
always there's always like a unique little
best. You know, last time you

250
00:17:56,559 --> 00:18:02,079
were missing a certain set of files. You know, this time you had

251
00:18:02,119 --> 00:18:06,960
it stored in a location that you
couldn't access or you know, so it's

252
00:18:07,240 --> 00:18:12,000
like you can't just do it once. You have to do it repeatedly because

253
00:18:12,039 --> 00:18:17,440
the environment's always changing. You know, maybe last time you restored from a

254
00:18:17,480 --> 00:18:21,720
particular server that now no longer exists, or you've switched to a different provider

255
00:18:21,839 --> 00:18:25,599
for some particular service. But it's
and it's hard. I think it's really

256
00:18:25,640 --> 00:18:32,799
hard for IT teams and DevOps teams
to get the buy in to spend the

257
00:18:32,880 --> 00:18:37,799
time and effort to do that because
a lot of the pressure is on build

258
00:18:37,880 --> 00:18:42,039
new features, push features to production, increase traffic to the servers, increase

259
00:18:42,079 --> 00:18:48,279
the conversion rate, you know,
and do disaster recovery planning and execution.

260
00:18:48,920 --> 00:18:52,799
Does nothing to increase the revenue of
a business. One of my favorite analogies

261
00:18:52,839 --> 00:18:57,240
to that I use it all the
time, is interesting. It just resonates

262
00:18:57,240 --> 00:19:00,720
well, especially here in America.
But you think about football, right,

263
00:19:00,799 --> 00:19:03,680
so you know, and it's the
same sort of principle like if all your

264
00:19:03,680 --> 00:19:07,519
focus is on your offense, because
that's what's you know, that's sort of

265
00:19:07,559 --> 00:19:11,880
the analogy of the revenue, right, If all your focus is on offense,

266
00:19:11,920 --> 00:19:14,680
because that's where the points could score, and you can't win. If

267
00:19:14,759 --> 00:19:18,839
you don't score points, right and
you forget about your defense, you're going

268
00:19:18,880 --> 00:19:22,960
to be in big trouble because you're
most likely not going to be able to

269
00:19:22,000 --> 00:19:29,599
score enough points to overcome whatever deficit
the lack of a good defense brings you.

270
00:19:30,119 --> 00:19:32,160
And then if you think about it, so then you let's say the

271
00:19:32,200 --> 00:19:33,720
analogy want to step forward and you
say, well, what does a good

272
00:19:33,759 --> 00:19:37,160
defense mean? It means focusing on
all the little things, right. A

273
00:19:37,200 --> 00:19:44,880
good defense means that if my guys
can't consistently do open field tackles, we're

274
00:19:44,880 --> 00:19:47,960
going to be in trouble. If
I don't have good coverage, we're going

275
00:19:48,039 --> 00:19:49,799
to be in trouble. Right,
It's the basics. There's nothing fancy,

276
00:19:49,839 --> 00:19:55,640
there's nothing like exciting about. You
know, you're not scoring a quick six

277
00:19:55,720 --> 00:20:00,680
or anything like that. But if
your folks aren't doing that and practicing,

278
00:20:00,720 --> 00:20:04,519
and the only way that you are
able to consistently do an open field tackle

279
00:20:04,720 --> 00:20:08,000
or provide good coverage or all those
other little things that you've got to be

280
00:20:08,000 --> 00:20:11,920
able to do is if you're practicing
them every week. And like you said,

281
00:20:12,240 --> 00:20:15,720
you're going back through tapes and like, which is the same idea,

282
00:20:15,799 --> 00:20:18,200
right, It's like exercising it and
looking at, well, what happened last

283
00:20:18,200 --> 00:20:22,000
time? Why did we fail?
Or why did this go wrong last time?

284
00:20:22,480 --> 00:20:26,000
You know, you're reviewing tape,
reviewing the videos and saying what happened

285
00:20:26,039 --> 00:20:30,119
and how do we do better next
time? And if you're not doing that,

286
00:20:30,240 --> 00:20:33,119
then you know, you think about
it a professional football team, like,

287
00:20:33,200 --> 00:20:37,720
think about all the support staff that
they have on both offense and defense.

288
00:20:37,799 --> 00:20:41,400
These coaches, the trainers, the
equipment, all of that stuff that

289
00:20:41,400 --> 00:20:45,480
goes into both sides of the field. And if you're not doing that,

290
00:20:45,519 --> 00:20:48,039
you're not going to win. And
I think the analogy of the business is

291
00:20:48,519 --> 00:20:52,200
if you're not focused on both and
really working both, you're just putting yourself

292
00:20:52,200 --> 00:20:57,079
at huge amounts of risk for something
like this to happen, and then you

293
00:20:57,119 --> 00:21:00,680
know you're just going to be in
big trouble. Yeah. Response has to

294
00:21:00,680 --> 00:21:04,519
be contextual too, because you know, using that analogy, Okay, we

295
00:21:04,559 --> 00:21:10,759
need defense to prevent the other team
from scoring points, so we're going to

296
00:21:10,839 --> 00:21:15,000
send the defensive team out on four
wheelers because they'll be more effective that way.

297
00:21:15,559 --> 00:21:18,880
It's like, well, wait,
no, you can't really do that.

298
00:21:18,920 --> 00:21:22,519
And I think the same thing goes
with your testing, your disaster,

299
00:21:22,559 --> 00:21:26,640
your preparedness plan. You know,
it has to be contextual. It has

300
00:21:26,680 --> 00:21:30,000
to be when you propose that,
you've got to be mindful of the fact

301
00:21:30,039 --> 00:21:36,000
that there is still an offense and
you want to minimize the disruption to them,

302
00:21:36,079 --> 00:21:40,519
if not completely avoided as well,
because I think there have been instances

303
00:21:40,519 --> 00:21:44,720
where the security team is like,
oh, that's that's insecure. You know,

304
00:21:44,799 --> 00:21:48,119
we have to completely close that off
to the point where it starts to

305
00:21:48,279 --> 00:21:52,160
the security starts to hamper the business
then, and so you really have to

306
00:21:52,319 --> 00:21:59,799
find out what that fine line or
that balance between the two extremes there is.

307
00:22:00,200 --> 00:22:03,319
I think that's that that's a really
important point is that if you if

308
00:22:03,359 --> 00:22:07,160
your security team is operating, you
know, in a black hole, as

309
00:22:07,200 --> 00:22:08,880
the old joke that you know,
the security team is always sort of in

310
00:22:08,960 --> 00:22:14,480
the dungeon behind locked doors, no
one else can get there, and it's

311
00:22:14,839 --> 00:22:18,720
that sort of thing. That's that's
the worst scenario ever. It really is

312
00:22:18,319 --> 00:22:23,079
if your security team is not embedded
in the business to where the security folks

313
00:22:23,200 --> 00:22:26,880
understand the business and understands revenue.
I mean, because let's face it,

314
00:22:27,319 --> 00:22:33,039
what is the biggest risk any business
faces the lack of cash flow? It's

315
00:22:33,720 --> 00:22:38,200
Microsoft vulnerability, right, It's not
the biggest risk you face is a lack

316
00:22:38,240 --> 00:22:44,359
of cash flow, right, because
that impacts my paycheck. That's everything,

317
00:22:44,680 --> 00:22:47,599
right, I mean, that's that's
the you know questions is the business open

318
00:22:47,640 --> 00:22:52,160
tomorrow or not? That's that's what
defines it. So if the business isn't

319
00:22:52,200 --> 00:22:55,519
in line with that, I'm sorry
with the security team isn't in line with

320
00:22:55,559 --> 00:22:59,759
the business with where revenue comes from. And you know, I couldn't care,

321
00:23:00,160 --> 00:23:03,720
Like, if you've got a business
process that is where the cashlow is

322
00:23:03,759 --> 00:23:07,759
coming from, then it doesn't matter
like what the risk of the cyber risk

323
00:23:07,960 --> 00:23:10,559
is, Like that's the biggest rish
you've got, right, And it's to

324
00:23:10,599 --> 00:23:14,960
your point that that's really what's important. I see that too often that security

325
00:23:15,000 --> 00:23:18,079
teams are really just divorced of what's
going on in the business. They don't

326
00:23:18,160 --> 00:23:22,160
understand business processes, they don't understand
how the business makes revenue. And then

327
00:23:22,200 --> 00:23:25,039
those kinds of things that you said, like oh no, sorry, all

328
00:23:25,079 --> 00:23:27,200
those ports are turned off, firewalls
locked down, and nope, sorry,

329
00:23:27,400 --> 00:23:30,839
you know, nobody gets administrative access
to X, Y and z, And

330
00:23:32,319 --> 00:23:36,079
you know that's what that's when that
happens because all we're thinking about is best

331
00:23:36,119 --> 00:23:40,319
practice and we're not thinking about risk
and what's going on with the business,

332
00:23:40,599 --> 00:23:44,640
and you cannot divorce the deal or
else you're just yeah, you're just setting

333
00:23:44,680 --> 00:23:48,839
yourself up for even more problems.
So so what do you do? I

334
00:23:48,880 --> 00:23:52,400
guess if you find that you're a
victim of one of these breaches. I

335
00:23:52,400 --> 00:23:56,200
mean, we talked about this a
little bit with Solar Winds, but I'm

336
00:23:56,240 --> 00:23:59,079
just curious to reiterate some of this. It's like, Okay, crap,

337
00:23:59,359 --> 00:24:03,319
we got to turn it off before
we get stewed into next year or whatever.

338
00:24:03,960 --> 00:24:08,519
Right, So that is really probably
one of the biggest challenges is what

339
00:24:08,559 --> 00:24:11,480
do we do now? And it
depends on what the situation is. So

340
00:24:11,599 --> 00:24:18,119
for something like Solar Winds, where
they weren't the attack itself wasn't disruptive because

341
00:24:18,720 --> 00:24:22,680
thread actor, that wasn't their goal. Their goal wasn't to disrupt people's environments.

342
00:24:22,720 --> 00:24:27,720
It was really to steal information,
right, So there the disruption to

343
00:24:27,759 --> 00:24:33,400
you is you've got to investigate what
happened, because if you find that you

344
00:24:33,559 --> 00:24:37,319
are a victim of something like a
Solar Winds style attack, you've got to

345
00:24:37,319 --> 00:24:41,480
try and figure out what it is
that the thread actor accessed, what data

346
00:24:41,599 --> 00:24:47,359
that was You may have your own
disclosure laws that require you to disclose it.

347
00:24:47,440 --> 00:24:52,359
Let's say you found out that they
stole information that included personal information or

348
00:24:53,039 --> 00:25:00,880
protected health information or whatever else sort
of falls under legal liabilit things. They

349
00:25:00,319 --> 00:25:06,559
in Europe you have privacy laws at
the GDPR under in California you've got the

350
00:25:06,599 --> 00:25:10,920
CCPA, and now other states are
following suits. So it's these privacy laws

351
00:25:10,960 --> 00:25:15,920
where if sensitive information which is defined
under those you know in those laws,

352
00:25:17,440 --> 00:25:21,759
gets disclosed by unauthorized parties, which
would include you know, one of these

353
00:25:21,759 --> 00:25:23,880
threat actors, then you've got to
disclose. So you've got to figure that

354
00:25:25,000 --> 00:25:27,079
out. You've got to find out
what they did, and that means investigating

355
00:25:27,119 --> 00:25:32,160
it, which means having somebody come
in and help you with the forensics to

356
00:25:32,200 --> 00:25:34,960
figure that out. I mean some
companies, bigger companies obviously have big security

357
00:25:34,960 --> 00:25:40,039
teams with forensic people on board and
all that sort of sort of thing.

358
00:25:40,160 --> 00:25:42,759
If you don't, that means you're
out there trying to you know, hire

359
00:25:44,000 --> 00:25:48,519
inside response firm to do that for
you to run that investigation. So that's

360
00:25:48,759 --> 00:25:52,079
you know that that's one possibility.
So you know, getting back to like

361
00:25:52,119 --> 00:25:56,359
what Will was saying earlier, how
do you be best prepared. If you

362
00:25:56,519 --> 00:26:02,079
don't have that kind of a sophisticated
security team on staff, then you should

363
00:26:02,119 --> 00:26:06,759
have a retainer in your back pocket
for an instant response team that you know

364
00:26:06,799 --> 00:26:08,319
that you can pick up the phone
and say, hey guys, I need

365
00:26:08,359 --> 00:26:11,119
your help, like I'm in over
my head. You've got to, you

366
00:26:11,160 --> 00:26:15,640
know, and you've got to have
that practiced and exercised, and obviously the

367
00:26:15,680 --> 00:26:19,880
contract signed and all that, all
that should have happened before, you know,

368
00:26:19,920 --> 00:26:22,960
before you have an incident. But
if you if you're the victim of

369
00:26:23,039 --> 00:26:27,359
ransomware, in some ways, it's
a lot worse because now your entire business

370
00:26:27,400 --> 00:26:33,359
or at least parts of your business
are disrupted or offline because this ransomware is

371
00:26:33,400 --> 00:26:37,119
on all your systems and you can't
access anything. So now you've got to

372
00:26:37,160 --> 00:26:41,359
meet this decision of do I pay
the ransom which could be very very expensive,

373
00:26:41,680 --> 00:26:45,880
and it's questionable about whether they would
whether paying the ransomware is going to

374
00:26:47,160 --> 00:26:51,119
actually make it any faster for you
to recover, or do you just try

375
00:26:51,119 --> 00:26:53,799
to recover on your own not pay
the ransom, you know, and that's

376
00:26:53,839 --> 00:26:59,240
a big decision that you know clearly
is going to depend on context and what's

377
00:26:59,720 --> 00:27:03,240
what's been disrupted and what you have
in place to be able to recover from.

378
00:27:03,359 --> 00:27:07,000
And you've got to answer all those
questions. But again it's why you

379
00:27:07,000 --> 00:27:12,720
should have something practiced of. Have
you be tried how would you recover from,

380
00:27:14,000 --> 00:27:15,359
you know, some kind of a
disruptive attack like that? Like what

381
00:27:15,480 --> 00:27:19,839
happens if like let's say you're a
company of a few hundred or few thousand

382
00:27:21,000 --> 00:27:25,119
whatever, Well, what happens if
seventy five percent of your machines ever and

383
00:27:25,200 --> 00:27:27,359
somewhere on them or fifty percent of
your machines twenty five percent? I mean,

384
00:27:27,359 --> 00:27:30,720
it's it's going to be a big
number. It's not like, you

385
00:27:30,759 --> 00:27:32,920
know, for most companies, it
doesn't you know, you're not going to

386
00:27:32,880 --> 00:27:36,079
be able to just sort of like
replace all these machines and get back up

387
00:27:36,079 --> 00:27:41,720
and running. It could be a
matter of having to like basically wipe and

388
00:27:41,759 --> 00:27:48,319
reinstall everything back on a whole bunch
of machines, including data. So how

389
00:27:48,440 --> 00:27:51,079
are you going to do that?
Do you have the feet on the boots

390
00:27:51,119 --> 00:27:52,559
on the ground to bit to do
that, or similar to what we talked

391
00:27:52,599 --> 00:27:56,480
about earlier, do you need to
have a retainer in place with some outside

392
00:27:56,599 --> 00:28:03,000
services organization that can help you recover. And this case of this the MSP

393
00:28:03,160 --> 00:28:06,119
is what happens if your MSP happens
to be the one that gets hit,

394
00:28:06,519 --> 00:28:10,640
then what you know, do what
would you do at that point? These

395
00:28:10,640 --> 00:28:15,839
are these are tough situations, but
I mean as sort of as disruptive as

396
00:28:15,960 --> 00:28:21,559
our world has become, these are
scenarios that you know, people used to

397
00:28:21,599 --> 00:28:25,880
probably call this like one hundred year
storm or something, and they're not anymore.

398
00:28:26,240 --> 00:28:30,039
You really got to be prepared for
these types of storms that can take

399
00:28:30,079 --> 00:28:33,960
down your business very very quickly and
for prolonged you know, a prolonged period

400
00:28:34,000 --> 00:28:37,880
of time, you know. And
then getting back to what we said earlier,

401
00:28:37,960 --> 00:28:41,240
your biggest risk is, you know, running out of cash cash flow.

402
00:28:41,799 --> 00:28:45,240
What happens if your computing environment is
offline for several days? Do you

403
00:28:45,279 --> 00:28:48,880
have the cash flow to survive that? You know, you've got to Those

404
00:28:48,880 --> 00:28:53,119
are that also enters the equation.
So those are the things you've got to

405
00:28:53,119 --> 00:28:56,440
be prepared for. And there are
ways to prepare for this, Like it's

406
00:28:56,559 --> 00:29:03,039
you know, this is not insurmountable
any stretch of the imagination, but unfortunately,

407
00:29:03,119 --> 00:29:06,559
most companies that especially the ones you
read about in the press are the

408
00:29:06,599 --> 00:29:10,920
ones that we're not prepared for it
and are really struggling when it happens.

409
00:29:11,200 --> 00:29:12,880
Yeah, I think one thing that
you kind of touched on that I want

410
00:29:12,920 --> 00:29:17,640
to elaborate on is whenever you call
in the forensics team you know, or

411
00:29:17,680 --> 00:29:22,039
the experts to help you understand what
the impact was, their answer is most

412
00:29:22,200 --> 00:29:27,279
likely going to be I have no
idea unless you've got logging turned on inside

413
00:29:27,279 --> 00:29:30,920
of your system, right because you
know that someone got into your network.

414
00:29:33,160 --> 00:29:37,400
But unless you have like audit logging
turned on between different devices. I've been

415
00:29:37,440 --> 00:29:42,039
into a lot of companies that once
you're inside the network, there's no tracking

416
00:29:42,240 --> 00:29:48,079
anywhere to determine what you may or
may not have done. Yeah, that's

417
00:29:48,119 --> 00:29:52,599
a really good point. It really
So here's and we could we could spend

418
00:29:52,599 --> 00:29:56,680
another entire episode on this, but
here in lies I think maybe the next

419
00:29:56,720 --> 00:30:02,039
step. And I'm glad you sort
of helped unpack this because I just quickly

420
00:30:02,079 --> 00:30:06,240
said, and make sure that you've
got some forensics firm on you know,

421
00:30:06,359 --> 00:30:11,200
on a retainer. But they're not
all created equally. They have different tools

422
00:30:11,400 --> 00:30:17,319
that they use, and they have
different capabilities and some groups will some teams

423
00:30:17,319 --> 00:30:21,319
will just sort of come on site
and they're going to use whatever you've got

424
00:30:21,720 --> 00:30:23,359
set up. And if you don't
have logging set up, and if you

425
00:30:23,440 --> 00:30:27,839
don't have all the tools available for
them, you're going to be in a

426
00:30:29,400 --> 00:30:33,680
that's gonna be a difficult situation.
There are companies though, that what they

427
00:30:33,720 --> 00:30:36,640
will do, and so'll come into
your environment and the first thing they're going

428
00:30:36,720 --> 00:30:38,160
to say is we're going to ship
equipment to you and you have to install

429
00:30:38,200 --> 00:30:41,960
it. That's step one. Because
you don't have what we need to be

430
00:30:42,039 --> 00:30:45,680
able to pull the artifacts that we
need to be able to pull. So

431
00:30:47,440 --> 00:30:51,319
that's another approach, and depending on
again, depending on your situation, that

432
00:30:51,440 --> 00:30:55,119
might be the better approach for you. If you don't have a strong security

433
00:30:55,119 --> 00:30:59,440
team and you don't have all these
tools and processes in place, finding a

434
00:30:59,599 --> 00:31:03,640
FORENDS ex firm that will simply bring
all their tools with them and and then

435
00:31:03,680 --> 00:31:08,160
they'll be able to pull it pull
those artifacts themselves might be really necessary for

436
00:31:08,200 --> 00:31:11,599
you. It's it's just one of
those things you have to sort of shop

437
00:31:11,720 --> 00:31:15,839
for that and know in the back
of your mind what it is that you're

438
00:31:15,839 --> 00:31:19,319
looking for, right what where,
where are your limitations, what are you

439
00:31:19,319 --> 00:31:25,160
you know, what is your organization
capable of performing? And what are you

440
00:31:25,240 --> 00:31:29,319
really hiring out for. You've got
to be really clear on that what exactly

441
00:31:29,319 --> 00:31:32,759
are you looking for in terms of
that service, clear that you are in

442
00:31:32,839 --> 00:31:36,880
that them that they I think that
the clearer it will be, like who

443
00:31:36,920 --> 00:31:38,640
are the right one or two or
three firms that you want to talk to

444
00:31:38,920 --> 00:31:41,680
and decide on, you know,
which one do you do a retainer with?

445
00:31:42,359 --> 00:31:45,599
And on that subject. Just maybe
just one last thought on that is,

446
00:31:47,160 --> 00:31:49,039
you know some of the retainers are
you know, you you pay up

447
00:31:49,039 --> 00:31:53,079
front for them, and then you
know that you've got to guaranteed service level

448
00:31:53,119 --> 00:31:57,599
agreement that they'll respond and you know, on time for But you might want

449
00:31:57,599 --> 00:32:00,079
to have a second one, you
know, sort of backup just in case.

450
00:32:00,359 --> 00:32:04,880
And there are zero dollar retainers as
well, where you just sign the

451
00:32:04,920 --> 00:32:07,839
contract but you don't put any money
up front. Obviously you don't get an

452
00:32:07,920 --> 00:32:10,359
SLA with that, but it may
not be a bad idea to have another

453
00:32:10,400 --> 00:32:14,519
one in your back pocket because you
just never know, Like this is it's

454
00:32:14,599 --> 00:32:16,279
just the nature of the world that
we're in right now, is you just

455
00:32:16,319 --> 00:32:22,400
don't know I guess. The other
thing that I'm wondering because we're talking about

456
00:32:22,400 --> 00:32:25,799
Okay, you know, you'll get
some forensic team to come in, you

457
00:32:25,799 --> 00:32:29,599
know, make sure that you've got
things set up so they can get with

458
00:32:29,640 --> 00:32:32,559
the information that they need. I'm
assuming you can consult with these companies ahead

459
00:32:32,559 --> 00:32:37,440
of time to make sure that you
have everything logging what it needs to log.

460
00:32:37,519 --> 00:32:39,119
Right. This just seems kind of
obvious to me, I guess.

461
00:32:39,240 --> 00:32:44,079
But the other question that I have
is I'd like to be proactive and not

462
00:32:44,160 --> 00:32:49,279
have this problem in the first place, right, And I recognize that if

463
00:32:49,319 --> 00:32:52,440
I'm leaning on another company to provide
me a lot of these security services and

464
00:32:52,480 --> 00:32:55,480
they get breached kind of like Solar
Winds or some of these others, you

465
00:32:55,480 --> 00:33:00,680
know, they're performing a function within
my network, may or may not be

466
00:33:00,680 --> 00:33:04,119
able to mitigate that because you just
don't know who's going to get hit next.

467
00:33:04,680 --> 00:33:07,839
My question is what can I be
doing, Like, what proactive steps

468
00:33:07,880 --> 00:33:14,519
do I need to be taking in
order to I guess offset the easy the

469
00:33:14,559 --> 00:33:19,359
easy stuff right where some kid with
a script he downloaded off of four Chan

470
00:33:19,559 --> 00:33:22,400
comes at my postcrist server, right, And the next thing I know is,

471
00:33:22,720 --> 00:33:25,559
well, I forgot to update it
last week and so now they're in.

472
00:33:25,720 --> 00:33:30,200
Right, are there things that I
can be doing to mitigate sort of

473
00:33:30,200 --> 00:33:35,519
the script kitty easy? Yeah stuff? Yeah? Absolutely, And you know,

474
00:33:35,960 --> 00:33:37,599
I think you know good examples of
that is, I mean, how

475
00:33:37,720 --> 00:33:42,240
how often you don't even read about
it anymore, don't even like it just

476
00:33:42,279 --> 00:33:45,160
doesn't even open people's eyes anymore.
No one's even surprised when I see it.

477
00:33:45,160 --> 00:33:47,240
But if you, you know,
go back a couple of years,

478
00:33:47,279 --> 00:33:53,079
remember how often we would see cases
of like open s three buckets, right,

479
00:33:53,640 --> 00:33:59,359
it had all kinds of sense of
information or proprietor right information, right,

480
00:34:00,000 --> 00:34:01,880
I mean it's still happening all the
time. Or somebody set up like

481
00:34:01,920 --> 00:34:07,320
a Mongo dB box and EC two
and it's just world you know, just

482
00:34:07,760 --> 00:34:13,199
you know, Internet face readable,
Yeah exactly, And those are just full

483
00:34:13,239 --> 00:34:17,679
of vulnerability. I mean, Mongo
dB was never meant to be Internet facing,

484
00:34:17,760 --> 00:34:21,760
right that you know, So how
do you find that stuff? How

485
00:34:21,760 --> 00:34:23,599
do you how do you defend yourself
against that? And I think one of

486
00:34:23,639 --> 00:34:28,239
the easiest ways, and it's it's
not expensive, it's easy to do,

487
00:34:29,119 --> 00:34:34,559
is to have vulnerability scanners. The
two biggest product companies out there that do

488
00:34:34,599 --> 00:34:37,960
this retenable and qualless, and they
both do it from the cloud. You

489
00:34:37,000 --> 00:34:42,480
don't even have to like install infrastructure
in your environment. And the first thing

490
00:34:42,519 --> 00:34:45,880
you do is just have them,
you know, subscribe to it not that

491
00:34:45,000 --> 00:34:51,000
expensive, and have them scanning your
IP addresses, whatever external IP addresses you

492
00:34:51,079 --> 00:34:52,920
own, whether it's in the cloud, whether it's in your own data center.

493
00:34:53,440 --> 00:34:58,719
They're scanning it continuously and you're actually
reading the report. That step two,

494
00:34:58,800 --> 00:35:02,800
you have to lead their work.
If you don't, then you're not

495
00:35:02,840 --> 00:35:07,679
going to do anything about it.
So you know, like if you've got

496
00:35:07,679 --> 00:35:13,320
that Postcriss database that's that's unpatched,
or this Mango deb that pops up in

497
00:35:13,400 --> 00:35:16,800
EC two and it's in it's world
Internet facing, the scanner finds it and

498
00:35:17,079 --> 00:35:21,960
if you know, and it says, wow, here's all these critical vulnerabilities

499
00:35:21,960 --> 00:35:24,480
that you've got. You see that
and right away you fix it, right

500
00:35:24,519 --> 00:35:30,480
you fix it before anybody else finds
it. So that's a really easy way

501
00:35:30,519 --> 00:35:35,079
to deal with It's just continual vulnerability
scanning of all of your systems. It's

502
00:35:35,119 --> 00:35:37,639
just you know, it's it's that
blocking and tackling we were talking about earlier.

503
00:35:38,239 --> 00:35:43,519
It's not hard. It's not expensive, but it requires a litle bit

504
00:35:43,559 --> 00:35:47,960
of discipline in terms of making sure
that you're always scanning all of the IP

505
00:35:49,039 --> 00:35:52,840
addresses that are Internet facing that you
own. Ideally, you're even scanning the

506
00:35:52,840 --> 00:35:55,440
stuff that's inside of your network because
you just don't know, right, You

507
00:35:55,519 --> 00:36:00,599
don't just assume that because it's behind
the firewall, it is perfectly risk free,

508
00:36:00,639 --> 00:36:04,239
because that doesn't exist, right.
It's really you should be scanning everything.

509
00:36:04,519 --> 00:36:07,079
But means that you're doing that,
and you've got the discipline to make

510
00:36:07,119 --> 00:36:09,599
sure that the scanner is operating and
it's still running, and you're seeing the

511
00:36:09,679 --> 00:36:14,519
reports and you're acting on them,
and it's got all of your the latest

512
00:36:14,559 --> 00:36:17,920
IP addresses. So when somebody turns
up a new VPC in Amazon, that

513
00:36:17,920 --> 00:36:22,000
that set you know that that new
network that Slash twenty four or whatever it

514
00:36:22,039 --> 00:36:27,719
is that you're using is now being
scanned as well. Right, that's really

515
00:36:27,719 --> 00:36:30,679
critical. I mean, I would
say that's step one absolutely. Step one.

516
00:36:30,920 --> 00:36:37,199
Step two is making sure that you've
got some kind of more sophisticated anti

517
00:36:37,320 --> 00:36:39,119
virus. I hate that term,
but you know, basically there's something more

518
00:36:40,559 --> 00:36:45,000
right because we all know how good
antivirus is but realian works well after the

519
00:36:45,079 --> 00:36:47,199
fact that they add in the signature
for whatever it is that you got hit

520
00:36:47,239 --> 00:36:52,480
worth, right yeah, but the
other stuff still floating around out there,

521
00:36:52,519 --> 00:36:57,199
right right, So you have to
have something and the idea is that you're

522
00:36:57,199 --> 00:37:00,679
also testing it. So and even
if that means bringing an the firm and

523
00:37:00,719 --> 00:37:05,199
saying, hey, help me test
this. What happens with this actually protect

524
00:37:05,199 --> 00:37:08,440
me against ransomware? And if so, like how well does it work?

525
00:37:08,519 --> 00:37:13,400
What process would I you know,
what happens in one system against ransomware?

526
00:37:13,519 --> 00:37:16,599
Is does this protect me against it? Not like proliferating through my network?

527
00:37:17,159 --> 00:37:20,480
You know that's sort of going to
work these through and do a lot of

528
00:37:20,559 --> 00:37:22,840
testing and exercising. Like we were
talking about with the you know, the

529
00:37:22,880 --> 00:37:27,079
analogy of the football team, Like
it's not like the football team is you

530
00:37:27,159 --> 00:37:31,159
know, sitting back binge watching Netflix
for six days a week until it's game

531
00:37:31,239 --> 00:37:35,880
day. Right. They're practicing,
They're watching film, they are busy,

532
00:37:35,960 --> 00:37:38,559
they're you know, the most successful
ones are are I mean, it's it's

533
00:37:38,559 --> 00:37:43,519
a grouling schedule and that's what your
folks got. I mean, they have

534
00:37:43,599 --> 00:37:46,320
the grooling schedule, but they need
to be practicing and exercising and all that

535
00:37:46,440 --> 00:37:49,840
kind of stuff, and if they're
not, you're going to get hit by

536
00:37:49,840 --> 00:37:52,239
something that surprises you. Yeah,
I want to I want to add a

537
00:37:52,239 --> 00:37:54,800
little bit to that. When you
do the port scanning, I would recommend

538
00:37:55,639 --> 00:38:01,360
aggressively questioning every exposed port. For
example, if you do have the postgrass

539
00:38:01,480 --> 00:38:07,199
port exposed for your database server,
you know, question why because and I've

540
00:38:07,199 --> 00:38:12,840
seen this quite a bit in the
last couple of years, people who are

541
00:38:13,039 --> 00:38:17,239
using database as a service companies like, hey, sign up with our service

542
00:38:17,280 --> 00:38:21,840
and we'll give you a hosted Mango
database and you don't have to do any

543
00:38:21,880 --> 00:38:23,840
of the maintenance on it, and
then they give you a publicly exposed URL

544
00:38:24,199 --> 00:38:27,039
for it, and it's like,
dang, dude, you know that,

545
00:38:27,199 --> 00:38:32,239
like every script kitty on the planet
is just beaten on that thing, like

546
00:38:32,360 --> 00:38:37,000
a rental car all day long.
You know, in your car. There's

547
00:38:37,119 --> 00:38:42,519
there's a better way than this,
you know, because no matter you can

548
00:38:42,559 --> 00:38:46,480
apply to patches within seconds after they
come out, but eventually someone's going to

549
00:38:46,559 --> 00:38:53,119
get through. Yeah. So,
like my golden rule is no port should

550
00:38:53,119 --> 00:38:59,440
be exposed except for Port eighty and
its only purpose is to redirect traffic to

551
00:38:59,639 --> 00:39:05,440
port or for three for your website
and a port for VPN access. And

552
00:39:05,519 --> 00:39:08,960
then if you happen to be hosting
your own email services, you know have

553
00:39:09,480 --> 00:39:15,679
port was it twenty five open to
receive email, and that really should be

554
00:39:15,719 --> 00:39:20,519
it. Everything else should just be
highly scrutinized. And then since we deal

555
00:39:20,679 --> 00:39:22,960
a lot with development, one of
the other things you can do is a

556
00:39:22,159 --> 00:39:29,079
big place for us to get in
trouble as developers is unpatched vulnerabilities in the

557
00:39:29,199 --> 00:39:31,360
packages that we use. So you
know, if I'm writing code and no

558
00:39:31,559 --> 00:39:38,960
JS and I install an NPM package, well that NPM packages built with dependencies

559
00:39:39,000 --> 00:39:44,199
on other NPM packages, which is
built with dependencies on other NPM packages.

560
00:39:44,760 --> 00:39:49,639
So it's almost impossible for me to
know what's actually installed on my MPM server.

561
00:39:50,360 --> 00:39:53,400
But in the CICD pipeline, we
can install a tool like Snake and

562
00:39:53,679 --> 00:39:59,599
it will look through the manifest every
time we push code to master and check

563
00:39:59,679 --> 00:40:04,400
for vulnerabilities, and then it has
a capability of failing to build if there

564
00:40:04,599 --> 00:40:07,639
are if the vulnerabilities exceed whatever you
define threshold is. And so that's a

565
00:40:07,679 --> 00:40:14,320
really good way to make sure that
you are checking and updating your software in

566
00:40:14,400 --> 00:40:16,760
an automated process, because if it's
a manual process, you're going to get

567
00:40:16,800 --> 00:40:21,599
busy, you're going to forget whoever's
doing it is going to be on vacation

568
00:40:21,800 --> 00:40:24,039
or whatever. You know, there's
all these reasons that mano processes fail.

569
00:40:24,079 --> 00:40:28,679
But if you can automate that as
part of your CICD pipeline, then it

570
00:40:28,840 --> 00:40:34,320
just happens as a course of doing
your natural daily activities. I totally agree

571
00:40:34,360 --> 00:40:37,440
with that. And just to hit
that one home, the Equifax breach back

572
00:40:37,519 --> 00:40:45,599
in twenty seventeen, that was that
stemmed from an Apache struts vulnerability, and

573
00:40:45,039 --> 00:40:52,320
Equifax had patched Apache struts in other
applications and they missed it in the one

574
00:40:52,360 --> 00:40:55,440
that was compromised. And this all
this is part of the congressional hearing.

575
00:40:57,079 --> 00:41:00,719
Why I bet somebody feels dumb,
right, how did we miss that?

576
00:41:00,800 --> 00:41:01,920
We hadn't solved? How did we
miss it? Well? Yeah, and

577
00:41:02,000 --> 00:41:07,119
that CEO is no longer there and
yeah seriously, But but you know,

578
00:41:07,280 --> 00:41:10,800
you use something like Will is talking
about, and you're not manually trying to

579
00:41:10,800 --> 00:41:15,280
figure out why what was built on
a patching struts I don't remember, I

580
00:41:15,360 --> 00:41:17,320
mean that was a built twenty years
ago. I have no idea, Right,

581
00:41:17,440 --> 00:41:22,920
it doesn't matter. You have an
automated process for finding that because all

582
00:41:22,000 --> 00:41:28,760
those that equifacturers breached something like two
or three months after the patch came out,

583
00:41:29,039 --> 00:41:31,760
so there's plenty of time for that
to be patched. And that brings

584
00:41:31,760 --> 00:41:35,599
me to another point that I want
to bring up, and it's a very

585
00:41:35,760 --> 00:41:39,079
unpopular opinion, so this is the
perfect format to do it. I tell

586
00:41:39,159 --> 00:41:45,360
everybody I'm a moron, and here's
why. I'm just kidding. I do

587
00:41:45,519 --> 00:41:50,199
that all the time. Yeah,
I mean it's you know, it's what

588
00:41:50,360 --> 00:41:54,599
being an influencer is all about.
Yah. Yeah, you pick the reasons

589
00:41:54,639 --> 00:41:59,800
why people hate you. Yeah.
But no, like the Equifax breach is

590
00:41:59,800 --> 00:42:02,679
a perfect example. You know,
they were they were vulnerable for months and

591
00:42:04,400 --> 00:42:12,239
compromised millions of people's data and the
final fine payout was like what, seventy

592
00:42:12,320 --> 00:42:16,360
five bucks a person. It's like, oh, that's that's appropriate, Which

593
00:42:16,400 --> 00:42:20,760
brings me to my whole point of
this is, whenever you're using a third

594
00:42:20,800 --> 00:42:22,559
party service, you know, a
SaaS service, you outsource a part of

595
00:42:22,599 --> 00:42:28,199
your business to them, is I
think it's really important to question them on

596
00:42:28,440 --> 00:42:31,119
what their obligation is when they are
breached. You know, it's not if

597
00:42:31,159 --> 00:42:34,760
they're going to be breached, it's
when they're going to be breached. What

598
00:42:35,000 --> 00:42:39,400
is their obligation to my customers?
And in ninety nine point nine percent of

599
00:42:39,559 --> 00:42:45,840
all SaaS agreements there is no obligation. So they can leave your customers out

600
00:42:45,000 --> 00:42:51,159
high and dry, ruin your business
reputation, and if you're lucky, you'll

601
00:42:51,239 --> 00:42:55,360
get mentioned on a tweet from the
CEO of that company whenever he apologizes publicly

602
00:42:55,480 --> 00:43:02,840
for it. Okay, over,
no, it's it's true. And I

603
00:43:02,880 --> 00:43:06,679
don't again, you know, that's
one where I don't I don't know what

604
00:43:06,760 --> 00:43:10,519
the solution is because I've worked with
many companies that try to do a good

605
00:43:10,599 --> 00:43:16,199
job of their due diligence when they
are vetting third parties right, thirty third

606
00:43:16,199 --> 00:43:20,480
party relationships, But how do you
vet like you know, you're doing business

607
00:43:20,519 --> 00:43:22,159
with an Equifax first, they're not
going to give you the time of day

608
00:43:22,159 --> 00:43:25,800
anyway, because they're much bigger than
just met everybody, right, But how

609
00:43:25,880 --> 00:43:30,760
do you know how good their security
processes are? How do you know whether

610
00:43:30,800 --> 00:43:36,039
they're going to find that Apache struts
vulnerability In twenty nine out of thirty of

611
00:43:36,159 --> 00:43:39,800
their application servers, right, I
don't know. And I think that's that's

612
00:43:39,840 --> 00:43:45,119
an example of where you want to
make sure that your insurance is covering you

613
00:43:45,159 --> 00:43:49,840
because that's a risk you don't want
to take on yourself. You can't mitigate

614
00:43:49,880 --> 00:43:52,960
it, so you have to transfer
it to you transfer it by buying insurance.

615
00:43:52,199 --> 00:43:55,079
And you know, this is I
guess out of scope for you know,

616
00:43:55,599 --> 00:43:59,800
DevOps folks, But just as a
thought, I mean, that's how

617
00:44:00,199 --> 00:44:01,960
I mean, there's again, there's
a solution to that problem, but it

618
00:44:02,079 --> 00:44:07,079
is it's I think that's by making
sure that you are insured properly for that

619
00:44:07,199 --> 00:44:08,599
risk. Yeah, And I'm not
one hundred percent certain it is out of

620
00:44:08,639 --> 00:44:14,280
scope for DevOps because I think they
DevOps is probably one of the few places

621
00:44:14,400 --> 00:44:19,000
in the business that has enough irons
in the fire in different camps to be

622
00:44:19,039 --> 00:44:22,519
able to see those bigger picture things. Yeah. That's a good point,

623
00:44:22,920 --> 00:44:24,159
you know, because your legal team, your legal team is not going to

624
00:44:24,239 --> 00:44:29,320
know who your third party SaaS providers
are. No. No, Well,

625
00:44:30,039 --> 00:44:32,679
it's interesting that you bring this up
though, because, for example, when

626
00:44:32,760 --> 00:44:37,519
I'm dealing with like sponsors and stuff
for the shows, a lot of times

627
00:44:37,360 --> 00:44:42,800
the yeah, they are involved in
the process of Okay, you know,

628
00:44:42,920 --> 00:44:45,559
here's the contract, here's the here's
what we expect, here's what we're going

629
00:44:45,599 --> 00:44:50,159
to get. And I've seen companies
do this with their vendors as well,

630
00:44:50,280 --> 00:44:53,559
right where they do scrutinize the terms
of service, and they they do scrutinize

631
00:44:53,599 --> 00:44:58,559
this. But typically it's the larger
companies that are going to push for more

632
00:44:58,639 --> 00:45:01,480
favorable terms. Right know, you
are actually going to help us with these

633
00:45:01,559 --> 00:45:05,840
things when they occur. You are
going to be involved at this level.

634
00:45:06,079 --> 00:45:08,400
It is going to be your fault
when it's your fault, right, Yeah,

635
00:45:08,480 --> 00:45:12,480
you know, shared liability agreement start. Yeah, exactly. If you're

636
00:45:12,519 --> 00:45:16,119
big enough you can demand those yeah. But if Yeah, for the little

637
00:45:16,159 --> 00:45:20,280
guys like you and I, I
mean, you're kind of stuck with whatever

638
00:45:20,280 --> 00:45:22,719
they're gonna do for you, or
go find another vendor that's going to do

639
00:45:22,760 --> 00:45:25,079
it for you, which you may
or may not be able to find,

640
00:45:25,639 --> 00:45:30,599
yeah, or do it yourself,
which don't have the resources to either do

641
00:45:31,079 --> 00:45:36,000
or hire to do yep. Yeah. But I mean I think the bottom

642
00:45:36,079 --> 00:45:37,639
line though, is that there's there
really is a lot that you can be

643
00:45:37,760 --> 00:45:43,719
doing to protect yourself. And I
want that message to be to resonate that

644
00:45:44,199 --> 00:45:49,000
it's not hopeless. This isn't an
insurmountable problem. Unfortunately, It's just that

645
00:45:49,159 --> 00:45:54,079
many companies are just not spending enough
time and focus on the security side and

646
00:45:54,239 --> 00:45:59,119
making sure that security is just part
and parcel what everybody is doing in their

647
00:45:59,199 --> 00:46:01,679
day to day jobs. It's not
just some you know, security team off

648
00:46:01,679 --> 00:46:06,239
to the side dealing with it.
It's everybody, and you're practicing it,

649
00:46:06,320 --> 00:46:10,280
you're exercising it. It's just sort
of constant vigilance. And if all that

650
00:46:10,360 --> 00:46:14,400
fails, you can always fall back
on the Y two K bunker in Idaho.

651
00:46:14,679 --> 00:46:16,880
That's right. If I just get
a new job, we're in technology,

652
00:46:16,920 --> 00:46:22,239
there's other jobs out there, I
guess. I guess That's another thing,

653
00:46:22,360 --> 00:46:23,679
right, is what if it's not
your vendors, right? What if

654
00:46:23,679 --> 00:46:28,320
it's what if it's your coworkers?
At what point do you look at it?

655
00:46:28,639 --> 00:46:30,639
I'll give you an example. So
the company I work for, we

656
00:46:30,719 --> 00:46:37,639
have this process where we take the
data that we've gathered and there's a group

657
00:46:37,679 --> 00:46:42,000
of business folks involved, the QA
the data. Right, they make sure

658
00:46:42,039 --> 00:46:45,360
that the data makes sense based on
what we know about the market and things

659
00:46:45,440 --> 00:46:50,880
like that. That we're gathering it
from. And then what they were doing

660
00:46:51,159 --> 00:46:55,760
it was there wasn't a good interface
for managing that, and so they would

661
00:46:55,800 --> 00:47:00,840
actually and they set all this up
before I got there. I have to

662
00:47:00,920 --> 00:47:07,440
disclaim that because I'm embarrassed by it. But they would export it to an

663
00:47:07,480 --> 00:47:10,880
Excel sheet and then mungge it up
and then make us check it back into

664
00:47:10,920 --> 00:47:14,840
the codebase and run a script on
it in order to import it back in.

665
00:47:15,239 --> 00:47:19,360
And yeah, I had a fit
and put my foot down right when

666
00:47:19,400 --> 00:47:21,880
I found out about it. I
was like, no, we're not doing

667
00:47:21,960 --> 00:47:25,519
this anymore. This was after the
cycle had ended. I said We're not

668
00:47:25,599 --> 00:47:30,159
doing it again this way. And
I got some looks and I got a

669
00:47:30,239 --> 00:47:35,239
little bit of ribbing, a harsh
treatment from it. But what point do

670
00:47:35,320 --> 00:47:38,880
you look at these situations as the
technical person and say this isn't secure,

671
00:47:39,199 --> 00:47:43,360
or this isn't the best way to
do this, this is a really dumb

672
00:47:43,440 --> 00:47:46,079
idea. Yeah, well, I
told them I'm not going to be liable

673
00:47:46,159 --> 00:47:49,880
for this data, so you better
find another way to do it. And

674
00:47:50,280 --> 00:47:52,599
they took me seriously enough to where
we're sitting down and actually having a conversation

675
00:47:52,679 --> 00:47:55,519
about it now and they're going to
need it in like two or three weeks.

676
00:47:57,239 --> 00:47:59,840
But yeah, realistically, what if
they told me to go jump on

677
00:47:59,880 --> 00:48:01,079
a like what can I do?
Do? I just do? I quit?

678
00:48:01,440 --> 00:48:04,800
Do? I? I work at
a company that is large enough to

679
00:48:04,800 --> 00:48:07,119
actually have a security team, so
I could report it, right, But

680
00:48:07,559 --> 00:48:10,119
yeah, I mean, what do
you do? And how serious does it

681
00:48:10,199 --> 00:48:13,719
have to be before you go this
just really isn't worth it? Yeah,

682
00:48:13,920 --> 00:48:16,360
I think there's all of the options
are on the table, and it's important

683
00:48:16,360 --> 00:48:23,440
before you decide which option is for
you to fully understand like the whole scope

684
00:48:23,519 --> 00:48:28,280
of the thing. Not saying that
you didn't, but like, just as

685
00:48:28,320 --> 00:48:31,280
advice to someone listening who says,
oh, I'm in this position, have

686
00:48:31,480 --> 00:48:37,079
the full conversation, to sit down
with whoever you can, which is probably

687
00:48:37,119 --> 00:48:42,159
gonna be multiple people, and say
how did it get this way? Because

688
00:48:42,199 --> 00:48:45,599
most of the time those types of
things, in my experience, have come

689
00:48:45,760 --> 00:48:51,519
from just like tribal knowledge, and
it's been decades in the process, and

690
00:48:51,880 --> 00:48:57,039
at each step of the way,
no one invented this. They only changed

691
00:48:57,079 --> 00:49:00,800
one little piece of it, you
know, and then over time you've changed

692
00:49:00,880 --> 00:49:04,559
enough of the pieces where it no
longer resembles the original thing that it was,

693
00:49:05,320 --> 00:49:07,519
but since we've all been doing it
that way all along, nobody really

694
00:49:07,559 --> 00:49:12,480
picked up on that until someone comes
in from the outside and gets introduced to

695
00:49:12,599 --> 00:49:15,519
it for the first time and they're
like, whoa, wait, what is

696
00:49:15,639 --> 00:49:19,800
this? You know, so,
I think it's important to have that context

697
00:49:19,880 --> 00:49:25,400
of how it got to be that
way, and then try to articulate the

698
00:49:25,559 --> 00:49:31,599
concerns that you have about it and
weigh those concerns and risks against the cost

699
00:49:31,800 --> 00:49:37,920
of the cost of rectifying that.
And then once you've exhausted all of those

700
00:49:38,039 --> 00:49:42,280
options, now you're at the point
where you have enough information to make a

701
00:49:42,320 --> 00:49:45,440
decision as to whether you report it
to the security team or say no,

702
00:49:45,719 --> 00:49:50,400
it's this isn't the right place for
me and pack your bags. Yeah.

703
00:49:50,880 --> 00:49:53,480
Well, and it's interesting too write
because I had the conversation with a number

704
00:49:53,480 --> 00:49:57,599
of people. A couple of the
people, yeah, I mean that was

705
00:49:57,639 --> 00:50:00,360
effectively their response was, oh,
wow, I didn't realize it had gotten

706
00:50:00,440 --> 00:50:05,519
that bad, right, and so
you know, no, there was no

707
00:50:05,599 --> 00:50:07,599
malicious intent or anything, right,
But at the same time, it was,

708
00:50:07,719 --> 00:50:10,639
yeah, we definitely need to fix
that, But nobody was making out

709
00:50:10,679 --> 00:50:15,320
a priority until I actually put my
foot down either, and so I think

710
00:50:15,360 --> 00:50:19,760
there there's some trade offs and some
conversations, and obviously it requires some tact

711
00:50:20,280 --> 00:50:23,519
which I do not and never have
possessed, but somehow we made it through

712
00:50:23,559 --> 00:50:29,400
anyway, and so yeah, I
think it's worth pointing out that, Yeah,

713
00:50:30,000 --> 00:50:32,039
you have to have the conversations,
right And at the end of the

714
00:50:32,119 --> 00:50:36,400
day, I put my foot down, and I put my foot down with

715
00:50:36,519 --> 00:50:39,159
my boss, you know, who's
a dev manager I think is effectively his

716
00:50:40,079 --> 00:50:44,280
stated title, and then the project
manager. And so they went back to

717
00:50:44,320 --> 00:50:46,840
the business people and said, your
dev team is not going to move forward

718
00:50:47,079 --> 00:50:51,719
on anything else after a while until
this is solved. So if you want

719
00:50:51,760 --> 00:50:54,320
to be able to use this process, we've got to come up with a

720
00:50:54,400 --> 00:50:57,960
way for you to do it that
they can implement for you, because the

721
00:50:58,039 --> 00:51:00,199
way we have been doing it isn't
going to happen. And right, and

722
00:51:00,320 --> 00:51:04,760
so those conversations did happen and it
did go the way that it needed to.

723
00:51:05,639 --> 00:51:07,920
But yeah, I just want to
add to that, Yeah, have

724
00:51:08,039 --> 00:51:12,480
the conversations. I probably could have
been a little more tactful in my approach,

725
00:51:13,079 --> 00:51:15,280
but at the end of the day, I think at some point you

726
00:51:15,320 --> 00:51:17,559
got to put your foot down and
just say, look this just this opens

727
00:51:17,639 --> 00:51:21,840
us up to all kinds of problems, and we're either going to do this

728
00:51:21,880 --> 00:51:23,719
the right way or somebody else is
going to be doing it the wrong way,

729
00:51:23,760 --> 00:51:27,039
because it's not going to be me. Yeah, but yeah, I

730
00:51:27,320 --> 00:51:30,599
did want to reiterate your previous point. Yeah. A couple of people said,

731
00:51:30,639 --> 00:51:31,960
oh, I didn't realize that it
had gotten to that point, right,

732
00:51:32,360 --> 00:51:35,760
Yeah, And that's just been my
experience, you know, is it's

733
00:51:36,239 --> 00:51:39,559
somebody creates this thing, sets it
loose in the wild. It's like this

734
00:51:39,719 --> 00:51:44,280
old story. I can't remember where
it happened, but they put these monkeys

735
00:51:44,480 --> 00:51:47,440
in a room with the banana on
top of the ladder, and every time

736
00:51:47,519 --> 00:51:51,360
a monkey went up to get the
banana, they posed it down with a

737
00:51:51,400 --> 00:51:54,119
fire hose, and so over time, whenever one of the other monkeys would

738
00:51:54,119 --> 00:51:57,760
go up, the other monkeys would
drag him back down. Then they started

739
00:51:57,800 --> 00:52:00,320
replacing the monkeys one at a time. The time they had replaced all of

740
00:52:00,360 --> 00:52:05,280
the monkeys, and all the monkeys
knew if anyone goes for the banana,

741
00:52:05,400 --> 00:52:07,559
to drag that monkey off the ladder. Although no one knew why anymore,

742
00:52:07,880 --> 00:52:12,599
right, And then the summary to
that is that's how corporate policy gets created.

743
00:52:14,440 --> 00:52:17,199
Yeah, I mean we are talking
to people though within organizations. So

744
00:52:17,920 --> 00:52:22,400
how do you start having the conversations
about this stuff? Right? Not necessarily

745
00:52:22,880 --> 00:52:25,239
the kinds of things that I'm talking
about, but maybe more along the lines

746
00:52:25,360 --> 00:52:30,960
of setting policies and setting up automations
and things like that that you guys have

747
00:52:30,039 --> 00:52:32,840
brought up. If they're not doing
it, how do you go to them

748
00:52:32,920 --> 00:52:37,320
and say, no, we need
to start doing this, or how do

749
00:52:37,440 --> 00:52:39,679
you start pushing them to start doing
things that they've never done before, or

750
00:52:39,719 --> 00:52:43,840
pushing people on your own dang deam. It's tricky, right, because that

751
00:52:44,000 --> 00:52:49,199
comes down to like political skills for
lack of a better term, you know,

752
00:52:49,519 --> 00:52:52,800
and what type of politician are you? Are you Teddy Roosevelt where you're

753
00:52:52,800 --> 00:52:58,239
going in with a stick and beating
them into submission, or you you know

754
00:52:58,360 --> 00:53:02,280
in Abraham Lincoln that can can vince
them with words and you know, sell

755
00:53:02,280 --> 00:53:06,519
them on their own virtues. So
you've got to know what your own personality

756
00:53:06,599 --> 00:53:09,880
and your own strengths are. But
either way, it starts with communication,

757
00:53:10,079 --> 00:53:16,719
you know, and highlighting the problem
getting getting you have to understand what their

758
00:53:17,679 --> 00:53:22,360
perception of that is, and then
you have to be able to articulate to

759
00:53:22,559 --> 00:53:28,639
them what your perception of the risk
is, so that everyone has the common

760
00:53:28,760 --> 00:53:32,559
ground. Yeah, that makes sense, and it's a lot easier said than

761
00:53:32,679 --> 00:53:37,440
done. I rolled that out in
about sixty seconds, but in reality or

762
00:53:37,480 --> 00:53:40,639
in practice, that could take weeks
or months. And if you've never done

763
00:53:40,679 --> 00:53:45,079
that before, you can expect to
fail the first couple of times, which

764
00:53:45,159 --> 00:53:47,920
leads to its own set of frustrations, because then you're like, damn,

765
00:53:49,000 --> 00:53:52,920
I went and tried what that dude
said, and now nobody implemented my solution

766
00:53:53,239 --> 00:53:59,239
and they think I'm a jerk.
Yeah, I will definitely add to that,

767
00:53:59,400 --> 00:54:02,320
though. It does help to know
what your strengths are. It sounds

768
00:54:02,320 --> 00:54:06,599
like Jeffrey's trying to chime in.
So I'm just gonna say what I was

769
00:54:06,639 --> 00:54:10,119
gonna say. I'm kind of a
blunt object and I know that, and

770
00:54:10,320 --> 00:54:15,599
so I know that my approach at
some point, relatively quickly is going to

771
00:54:15,679 --> 00:54:21,199
devolve into no, we need to
do this, or I'm gonna quit my

772
00:54:21,360 --> 00:54:24,480
way or the highway. Hey,
it works works a lot. I was

773
00:54:24,559 --> 00:54:30,119
going to say that I think it's
that's the situation where you sort of realize

774
00:54:30,199 --> 00:54:36,000
you'll find out really quickly what the
culture is like in your organization. Right.

775
00:54:36,480 --> 00:54:40,119
For instance, like one of the
tenets of DevOps, right is the

776
00:54:40,239 --> 00:54:45,079
idea of being a learning organization,
right, a continually learning at organization.

777
00:54:45,239 --> 00:54:50,920
And if that's really the culture,
then you if you bring up an issue

778
00:54:50,960 --> 00:54:53,760
like this, that's gonna work,
right. I mean, you know there's

779
00:54:53,760 --> 00:54:58,840
gonna be mechanisms already for you to
be able to do that. For the

780
00:54:58,960 --> 00:55:02,280
vast majority of us who don't work
for organizations like that, it's more of

781
00:55:02,320 --> 00:55:07,599
a challenge. But I think it's
also perhaps an opportunity to help your organization

782
00:55:07,679 --> 00:55:09,320
and say, listen, here's a
here's a problem. And oh, by

783
00:55:09,400 --> 00:55:14,159
the way, I'm only finding one
problem. I'm sure there's others lurking our

784
00:55:14,239 --> 00:55:16,880
culture. Really we should be encouraging
people to bring up these kinds of issues

785
00:55:16,960 --> 00:55:22,840
and finding better ways of doing things
so that we are a so we can

786
00:55:22,920 --> 00:55:27,840
become a learning organization and we can
continually do better. Yeah. Well,

787
00:55:27,920 --> 00:55:30,159
and it's and for the most part
I found that most people if you can,

788
00:55:31,000 --> 00:55:36,559
if you can explain why, then
most people will at least hear you

789
00:55:36,679 --> 00:55:40,639
out, and so it's only come
down to we're doing we're not doing this

790
00:55:40,800 --> 00:55:43,920
this way, or we are doing
it this way, or I'm quitting.

791
00:55:44,280 --> 00:55:46,599
It's only come down to that once
or twice ever in my fifteen year career.

792
00:55:47,000 --> 00:55:50,519
Right. Most of the time,
you give them a good reason and

793
00:55:50,559 --> 00:55:52,920
people are going to go, yeah, yeah, we don't we don't want

794
00:55:52,960 --> 00:55:55,760
to have that problem, and so
they'll right, they'll figure it out,

795
00:55:57,039 --> 00:56:00,760
yep. And if it really does
come down to that, I also just

796
00:56:00,840 --> 00:56:02,320
want to point out that, yeah, you don't have to accept the liability

797
00:56:02,360 --> 00:56:05,880
for those issues. You can go
find another place to be. Yep.

798
00:56:06,039 --> 00:56:08,280
It's one of the fortunate things about
working in tech these days is there's a

799
00:56:08,360 --> 00:56:12,559
lot of jobs. Yeah, yep, yep. Absolutely, all right,

800
00:56:12,679 --> 00:56:15,000
Well, I think we've kind of
exhausted our time, and then some is

801
00:56:15,079 --> 00:56:20,760
there anything that we should make sure
that we include that we didn't talk about

802
00:56:20,800 --> 00:56:27,199
before we're you know, so yeah, I just want to leave something out

803
00:56:27,239 --> 00:56:29,760
and then be like, oh,
and make sure that you say this when

804
00:56:29,840 --> 00:56:36,079
you I don't know, right,
This is my secret weapon is every time

805
00:56:36,159 --> 00:56:38,079
I bring donuts in, when I
know I'm going to have the hard conversation,

806
00:56:38,199 --> 00:56:43,400
I don't know anyway. By the
way, donuts really do work there

807
00:56:43,519 --> 00:56:49,599
miracle food. But yeah, let's
go ahead and do picks then, Jeffrey,

808
00:56:49,639 --> 00:56:51,400
do you want to start us off? I had a feeling you were

809
00:56:51,440 --> 00:56:52,800
going to say that will Do you
want to start us off? Man,

810
00:56:52,840 --> 00:56:57,480
I'd love to start us off.
I've got this pick today. We're just

811
00:56:57,519 --> 00:57:00,559
going to seem out of the ordinary
because normally I say really profound stuff,

812
00:57:00,639 --> 00:57:05,920
but I'm going to open this one
up with As humans, it turns out

813
00:57:06,039 --> 00:57:10,559
we have a dependency on oxygen.
And so this book I've been reading is

814
00:57:10,679 --> 00:57:15,679
called The Oxygen Advantage by Patrick McEwan. So this is actually pretty cool.

815
00:57:15,719 --> 00:57:17,719
I mean it sounds like, based
on what I just told you is like,

816
00:57:17,840 --> 00:57:21,719
really this is where you're going,
But it's actually kind of cool.

817
00:57:22,000 --> 00:57:28,079
I'm a certified scuba diver, a
certified free diver. I rode twenty four

818
00:57:28,159 --> 00:57:31,360
hour mountain bike races for a number
of years and competed in that, and

819
00:57:31,440 --> 00:57:35,119
so I spent a lot of time
focusing on my breath. And I just

820
00:57:35,199 --> 00:57:37,079
got this book a couple of days
ago, and I'm just plowing through it

821
00:57:38,159 --> 00:57:45,320
because what the guy is going through
here is talking about how your body utilizes

822
00:57:45,440 --> 00:57:52,440
oxygen. And since we breathe by
default kind of out of necessity, we

823
00:57:52,599 --> 00:57:59,760
never really focus on improving our breathing. And because our habits have changed over

824
00:57:59,800 --> 00:58:04,400
the last thousands of years where we
don't really do a lot of physical labor

825
00:58:04,559 --> 00:58:07,760
or not on the move a lot
anymore, we actually don't breathe in line

826
00:58:07,880 --> 00:58:12,519
with the way that our bodies should
be. And as a result, people

827
00:58:12,519 --> 00:58:15,519
who are breathing what turns out to
be way too much. And so he's

828
00:58:15,559 --> 00:58:22,639
got these exercises in here to help
you lower your oxygen intake, increase your

829
00:58:22,679 --> 00:58:30,360
oxygen utilization, and increase your carbon
dioxide, which improves the efficiency of your

830
00:58:30,440 --> 00:58:34,719
muscles. So I think it's been
a pretty fascinating read for me. I

831
00:58:34,760 --> 00:58:37,360
haven't finished it yet, but I
was so excited I wanted to make that

832
00:58:37,480 --> 00:58:43,079
my pick this week. And I
think it applies to everyone who whether you're

833
00:58:44,000 --> 00:58:46,800
if you're an athlete, or you're
doing a lot of physical activity, it'll

834
00:58:46,880 --> 00:58:51,559
be specifically beneficial to you, But
even if you're not and just thinking that

835
00:58:51,679 --> 00:58:53,960
you want to be more active,
it's got some tips and tricks in there

836
00:58:54,079 --> 00:59:00,519
that will help you focus and refinething
there as well. Awesome, I'll have

837
00:59:00,599 --> 00:59:04,239
to check that out. Well,
I'll talk about on my picks, Jeffrey,

838
00:59:04,280 --> 00:59:07,920
do you have some picks? Yeah. So just following up similar to

839
00:59:07,159 --> 00:59:13,320
to Will's. I read a book
a while ago called micro Resilience, and

840
00:59:14,440 --> 00:59:19,119
it's sort of like the idea behind
it is that we sort of understand the

841
00:59:19,119 --> 00:59:23,840
idea of macro resilience of having like
an exercise routine and that sort of thing

842
00:59:23,920 --> 00:59:27,320
that's sort of like long term,
like these are the things that sort of

843
00:59:27,440 --> 00:59:30,960
keep me in shape and keep me
active and keep me healthy and all that

844
00:59:31,360 --> 00:59:37,000
sort of thing. But the idea
of micro resilience was the idea that,

845
00:59:37,559 --> 00:59:38,920
like, what are the things that
we can do sort of during our day

846
00:59:39,000 --> 00:59:44,760
that are not necessarily to try and
you know, develop muscle tone or aerobic

847
00:59:44,840 --> 00:59:47,480
exercise or something like that. But
you know, there's just those moments during

848
00:59:47,559 --> 00:59:52,760
the day when we just feel exhausted
or we've just feel overwhelmed or whatever it

849
00:59:52,880 --> 00:59:55,239
is that just and a couple of
things that she brings out in the book

850
00:59:55,360 --> 00:59:59,480
is one it's like, you know, sometimes you feel hungry during the day,

851
01:00:00,000 --> 01:00:02,360
and her gument is a lot of
times it's not that you're hungry.

852
01:00:02,559 --> 01:00:07,679
It's actually that you're thirsty. And
many times if you just drink, you

853
01:00:07,719 --> 01:00:10,679
know, just ta take some time
and like drink down big glass of water

854
01:00:10,960 --> 01:00:15,639
or something like that, that it
actually sort of re energizes you. And

855
01:00:15,719 --> 01:00:20,679
I have personally found that that really
helps instead of getting you know, sort

856
01:00:20,679 --> 01:00:23,199
of looking for like the quick energy
bar or something like that, just drink

857
01:00:23,280 --> 01:00:28,079
some water. It's huge. Or
you know, shows talking about like just

858
01:00:28,280 --> 01:00:31,119
standing up from your chair and not
just like standing up, but actually do

859
01:00:31,360 --> 01:00:35,840
some physical activity, like you know, moving your arms around, moving them

860
01:00:35,920 --> 01:00:38,679
up up above your head, that
sort of thing. Again, just sort

861
01:00:38,679 --> 01:00:43,000
of getting your blood flowing, and
it just sort of re energizes you,

862
01:00:43,159 --> 01:00:46,519
sort of rejuvenates you. Like another
example was, you know, sometimes you're

863
01:00:46,559 --> 01:00:49,880
on the road. I've been on
the road much in the last year and

864
01:00:49,920 --> 01:00:52,599
a half. But if you're on
the road, you're staying in a hotel

865
01:00:52,599 --> 01:00:54,159
and they have those like little swimming
pools. She's like, you know,

866
01:00:54,280 --> 01:00:58,840
even if you just like it starts
your day off in the morning by just

867
01:00:59,000 --> 01:01:01,159
jumping the pool doing a couple laps. Again, it's not for endurance or

868
01:01:01,199 --> 01:01:06,519
anything like that, but it just
gets your blood flowing and you know,

869
01:01:06,559 --> 01:01:09,239
it sort of helps you just start
your day off. So I think there's

870
01:01:09,239 --> 01:01:14,440
a lot of really good ideas of
just how to sort of deal with those

871
01:01:14,480 --> 01:01:17,760
sort of lulls that you get during
your day and rather than grabbing, you

872
01:01:17,800 --> 01:01:21,920
know, a chocolate bar or an
energy buyer or something my dad like,

873
01:01:22,000 --> 01:01:24,519
finding other ways that are probably healthier
that will help you sort of get your

874
01:01:24,639 --> 01:01:28,760
energy levels back off. I love
it. And to be perfectly honest,

875
01:01:28,920 --> 01:01:31,119
that's one of the things that I've
done lately to help mitigate some of the

876
01:01:31,239 --> 01:01:35,239
tension headaches I've had is just drinking
water. I don't know what it was,

877
01:01:35,559 --> 01:01:38,760
but I cut back on the energy
drinks and sodas I was drinking and

878
01:01:39,440 --> 01:01:43,719
just started drinking more water. And
that's made a huge, huge difference for

879
01:01:43,840 --> 01:01:47,000
me. So yeah, yeah,
I'm going to throw in a few picks

880
01:01:47,039 --> 01:01:51,440
of my own. So what I
was going to say on Wheel's pick about

881
01:01:51,519 --> 01:01:53,559
oxygen was just that, and I
think I've mentioned it on the show before,

882
01:01:53,639 --> 01:01:58,159
but one of my goals is to
complete an Iron Man, and so

883
01:01:58,280 --> 01:02:01,920
I've been just getting out and swimming
and running and biking, and to be

884
01:02:01,960 --> 01:02:07,480
perfectly honest, it's been interesting.
I was a swimmer in high school and

885
01:02:07,719 --> 01:02:12,159
did a little bit in college,
and so you know, kind of my

886
01:02:12,280 --> 01:02:15,199
breathing patterns kind of stem from that, even when I'm running your biking,

887
01:02:15,800 --> 01:02:19,440
but I have some friends who are
who are runners all their lives, right,

888
01:02:19,519 --> 01:02:24,559
and then got into triathlons and started
swimming, and just the cadence of

889
01:02:24,840 --> 01:02:30,280
movement and swimming and stuff has anyway, it has changed the way that they

890
01:02:30,719 --> 01:02:34,559
breed and the way that they exercise
and things like that. And so it's

891
01:02:34,559 --> 01:02:37,360
been interesting to me to just see
how all that comes together. And I'm

892
01:02:37,400 --> 01:02:42,159
interested to see within this book as
far as picks go. I found so

893
01:02:42,320 --> 01:02:45,760
my swim coach, because I'm on
a swim team now. In the morning

894
01:02:45,920 --> 01:02:50,639
I go to swim practice. I
had an equipment issue with some of my

895
01:02:50,719 --> 01:02:52,960
fins. Apparently she wanted me to
have longer fins, which are more work

896
01:02:53,039 --> 01:02:58,400
to swim with but also make it
go faster and help keep you on top

897
01:02:58,440 --> 01:03:00,880
of the water. She sent me
a link to some fins. I'll put

898
01:03:00,920 --> 01:03:04,360
a link in the show notes.
But they're kind of a little bit longer,

899
01:03:04,400 --> 01:03:07,280
but they're not like the really really
long scuba fins. They're more like

900
01:03:07,360 --> 01:03:13,440
snorkel fins, and so I'm going
to pick that just because they're nice to

901
01:03:13,480 --> 01:03:15,360
swim with, but they're not as
big time, heavy duty as some of

902
01:03:15,440 --> 01:03:20,920
the scuba fins. And then I
think I may have mentioned this last week,

903
01:03:21,800 --> 01:03:28,000
but I'm still reading Atlas Shrug and
I'm really digging it. It's funny.

904
01:03:28,039 --> 01:03:30,360
I picked it on JavaScript Jabber and
one of the other hosts basically said,

905
01:03:30,840 --> 01:03:35,039
don't take it too seriously, but
I find that I agree with a

906
01:03:35,079 --> 01:03:38,159
lot of the opinions in there.
So I'm going to pick that just because

907
01:03:38,159 --> 01:03:42,000
I've really really been enjoying it.
And then I've picked up a new book.

908
01:03:42,199 --> 01:03:45,320
I've only read the forward and a
little bit of the first chapter,

909
01:03:45,559 --> 01:03:50,199
but it's already appealing to me,
and it's called The Ruthless Elimination of Hurry

910
01:03:50,800 --> 01:03:53,599
by John Mark Comer, and just
talks about getting what you want from life

911
01:03:53,800 --> 01:03:59,880
and what you can do to eliminate
a hurry from your life and some of

912
01:03:59,880 --> 01:04:02,280
the stress that you have around some
of the stuff that you're probably trying to

913
01:04:02,360 --> 01:04:06,800
accomplish with life. So anyway,
I'll put links to all of those in

914
01:04:06,840 --> 01:04:12,280
the show notes. Cool good stuff, all right. Well, with that,

915
01:04:12,400 --> 01:04:15,920
I guess we'll wrap up this was
We've had some really great conversations lately.

916
01:04:15,000 --> 01:04:18,800
I've really enjoyed these, and especially
just being able to sit and chat

917
01:04:18,880 --> 01:04:23,840
and go through some of this stuff
related to some of these concerns and breaches

918
01:04:23,880 --> 01:04:27,800
has been great as well. So
anyway, we'll just wrap up here and

919
01:04:27,920 --> 01:04:29,760
until next time, folks max out