On, y'all, welcome to another episode of Adventures and dev Ops. I'm
your host, Will Button joining me in the studio, my co host back
on a streak making Tom Brady look like a slacker, Warren Parade. Welcome
Warren, and thanks for letting me come back for an of my hopes up
that it will keep on going. I'm Warren, I'm the CTO of author
has just to reintroduce myself. Yeah, I mean, I like how this
is going so far, and I have no plans to coward my way out
of the future right on. I'm excited to hear that, because otherwise it's
just me and the guest and always lead to trouble. Speaking guests joining us
today Pete Fritchman sre infrastructure staff infrastructure engineer over at Observe, Inc. And
Pete has joined us today to talk about observability in those pesky internal applications that
we all have. And I don't know, it might be a love hate
relationship there, but Pete, welcome to the show. Hey, thank you
for having me right on. So tell us a little bit about your background,
because you have You've got the staff engineer title, which takes a while
to get to and is also I think a little bit uncommon on the infrastructure
side. You know, it's pretty common on the software engineering side, but
I think it's a little more rare to see staff engineers on the infrastructure side.
So tell us how you got to that point. Sure, yeah,
I mean I've been doing this for a long time. I had in the
computers as a kid, which I guess is more common these days, maybe
less so in the nineties. But I knew early on as a kid that
I wanted to do computer stuff, and I wasn't exactly sure what computer stuff
was. Something with programming. I enjoyed that kind of you know, make
the computer do what I want aspect of things. I was really lucky to
land an internship at my local ISP in like eighth grade, the summer after
eighth grade, before ninth grade. Oh no, I had a great mentor
there, and I haven't kind of. I had done a mentorship like my
seventh grade summer doing Linux stuff, and I bought a laptop and ran Linux
on it. And running Linux on a laptop is a great way to become
one with Linux, you know, hate it, love it, whatever.
And working at this ISP and having a really great mentor George kind of like
I realized, Okay, CIS and Men is the thing I'd like to do,
Like I enjoy debugging these problems and building things and writing automation. So,
you know, I did the high school thing. I worked all through
high school. I'm in a workaholic forever you for better or worse. I
worked at this ISP all through high school. Went to college for a year.
Wasn't my thing. It was fun socially, but the school part was
just not. I don't know, I just lacked the focus. I just
really wanted to work right. So I had contributed a bunch to FreeBSD in
my high school days. I was a ports committer and ports are like the
packages in previous D. And through that I landed a job and a really
awesome group at FedEx doing system administration on everything Internet facing. And it was
a small group, and looking back, like that was the group to be
in at FedEx for doing Unixie stuff. They were definitely ahead of their time.
Everything was automated. They wrote their own tools, and I thought this
was very normal for like a two thousand and two shop, which I you
know, now we know maybe was so the whole automation. First thing has
always just kind of been how else would you do it? Kind of thinking,
and then I was lucky enough to land and that's every gig at Google
after that in two thousand and five, and you know, they were like,
hey, we should automate things and I was like, well, yeah,
how else would you do it? Right? And then just from there
it's been a whirlwind of startups. And I tried the banking world for a
little bit, had some fun there, has some not fun there. I
ultimately decided to go back to the startup world because I think that's my true
you know, that's where the for me, that's where the most fun is
the most fun. I had a bank was kind of a startup in a
bank, and that's very hard to find, oh for sure. So yeah,
cool, Yeah, I mean there's definitely a huge It's it's almost completely
two separate, completely careers working at large enterprise organizations versus startups, Like you
have to have two different mental models to be successful at each of those.
It's almost two different skill sets. I mean the base technical skill set of
course, and then you know at a startup you kind of have to pick
up the pieces and lead the way with what you have. And then at
a big enterprise you have all the resources, but you also have all the
politics, and so you have to you know, as much as you know,
everyone I know hates that you have to play. You have to make
friends in other organizations and figure out how to influence people, and it is
hard and stress. I enjoy the stress of tech, and the stress of
that is just it's a lot. Yeah, it's a lot of political engineering
versus softer engineering laying right, for sure. So when you started work at
the ISP, like are we talking back? Like dial up modem is P.
Yeah. I sat in the after room. I had a I had
like PM two e's next to me and those are quiet at least, and
then I had a rack of like fifty six k's that were just you know,
I'm pretty sure I could whistle my way into a fourteen and four connection.
Back then, we were like the regional ISP, So we did t
ones for businesses and they're there. Big thing was they would put the I
forget what Cisco it was, but there was something where you can take the
t one and like, oh, I want to take some of my parents
and use them as phone lines, and some of them as data and that
was like a revolutionary in ninety seven, so that yeah, for sure.
Yeah. I was working in the telco industry right around then and nice.
Yeah, and then we got you know, d slams and until DSL thing
came through, and it was just it was a fun way to get a
lot of exposure to a lot of different things. I got to go to
pops and put stuff in, I got to build Unix boxes, I got
to kind of do the whole gamut of things. So you got the benefit
of dealing with Y two. Yeah, you know like it. Yeah,
it was. It was such a non event that people at work weren't even
worried. You know, everyone was very much a realist there. They were
like, every whatever, everything breaks, we just live in the world or
nothing works, and you know, we passed ourselfware. What more is there
to do. I knew people. I didn't know them like very well or
personally, but I knew people pre Y two k that the area I lived
in at the time, they built underground bunkers and stopped them and they were
going like late December they were going underground and saying we're not coming out for
ten years or something, and I've never seen any of those people again,
so I'm like insanely curious. Were they of January? Are they still in
Yeah? Yeah, yeah. I didn't to have like the you know,
I didn't know Kobal. I mean, there was a whole you know,
crew of people that were just making insane money and projects. I'm just watching
ka prep needed or not. I mean, I feel like we'll hit that
again in twenty thirty eight with thirty two bit. I t like that that
one is actually a little scary. I hope to be retired by that,
right, no doubt. That seems like too much cool. So so talking
about internal platforms and observability, Yeah, like that's that was whenever this came
across as our topic, I was like, you know, that's that's just
brilliant because all of these little internal apps, and sometimes not little internal apps,
but like the things that the company uses to make decisions about are we
doing the right thing or not? They often are just like little pet projects.
So what's what's your experience there and how do you get those recognized as
the valuable assets that they are? Yeah, Well, often it happens on
its own, and but it's the worst case of there's a catastrophic failure at
the worst possible time, and everyone goes, oh, that was really important.
I worked somewhere at a public company, and the system that did the
closing the books every day, you know, like very barring, right,
Well, it wasn't working and it was taking you know, thirty hours to
close twenty four hours worth of books, and obviously that doesn't work so well.
And they kind of didn't notice it or take any action until, like
they were a week before they had to do some sec filing quarterly thing.
I don't know all the details, but it became, you know, suddenly
on Friday afternoon, it was we need all hands on deck to fix this.
But it's everything from that to not just necessarily apps, but like internal
infrastructure, right CEI and developer experience, and everyone's got these you know,
shell script infrastructure that dev's run to run the local cluster on their laptop,
and there's you know, any company you can probably sit down and find a
million of them. I just think they're perpetually under you know. The sexy
part is production, right, people want to build SLOs for users and show
awesome graphs and look at this great incident management process. We have. Then
they have this like Jenkins instance, it's barely holding on internally, you know
that everyone hates but no one really talks about because it's just, oh,
yeah, that's life, you know, that whole side of things just need
more love. Yeah, And so I think part of that, you know,
like you mentioned there, especially with like around Jenkins and things, it's
people. It's like people who build tools to solve specific problems that they're having,
and then it almost has like this organic growth of other people see it
and I'm like, oh yeah, I need to use that too, and
then it grows in its role in the company. So how do you identify
those before that catastrophic event? Yeah, well, you know a lot if
you think about So I do a lot of thinking about like how to compare
it to production. So in production, if you say I want to launch
this new micro service or I want to consume this two AWS service, you
write the sign bocs, you have launch review. I mean hopefully right,
ideally you have these things, you have this whole like rigmarole and process for
it. But eternally it's like, oh, I'm going to fire up a
EC too and run this new tool I downloaded hey, two weeks later,
it's important you have to apply that same same principle. And part of it
can be a staffing thing, right, Like, you know, you might
have this team of a million people working on PROD and then you got the
two you know, the IT guys working on infrastructure stuff that doesn't work very
well at all. Yeah, So I mean, I think you have to
put some process to it, and I mean I think that I'm a big
fan of the whole infrastructure management, incident management, post mortem process. Like,
I think that that is a great way to drive, you know,
out of just happen. And we all have to accept that nothing's one hundred
percent, but you should get the most out of every adage. So when
your internal tool does blow up and cause a you know, company visible,
everyone is like, hey, I don't really know what this was, but
we couldn't do business for a day. Make the most of that, right,
Hey, this thing broke. By the way, there's five other services
we've identified that are in the exact same state and could all blow up tomorrow
and we'd be on the same call. And so that's your that's your you
know, entrance point to getting everyone to have attention on that thing. And
it's unfortunate that, you know, things have to fail first sometimes, but
that's how it goes when you're especially at a startup where you're kind of have
conflicting priorities. Oh for sure. Yeah, And I think it's a really
good good point there is, like outages happen, so let's just make the
most out of that learning experience. Absolutely, which is that's been a cultural
change for us as an industry over the last I don't know, i'd say
ten to fifteen years. I feel like Etsy published that blame you know,
the famous blameless postmortems. It feels like that's ages ago now, but it's
still so relevant every day. Yeah. Yeah, things broke, who cares
how they broke. Let's just have it not break in the same way again,
right, And if you can do that, that to me is like
the health of a necessary team, right, it's not. Yeah, I
really hate places that measure health by oh there were four outages last quarter,
okay, like where they repeat root causes, Yes, okay, maybe there's
a real problem. But if they weren't repeat root causes and you're solving root
causes and writing good postmortems, O just are great. Yeah, yeah,
for sure. And I think it's I think that's one of the things that
it's really hard to convince people of is outages aren't as bad as you think
because where most of us work like we can recover and and I say that
coming from a background where sometimes when we had outages, people's lives were at
stake. And so if you can walk out of an outage and say no
one died, like we're going to be all right. Yeah. Well,
plus it ties into SLIS SLS air budgets, right, people a lot of
times before the SLO concept became very popular, like it's five minutes of downtime
bad? What does that mean? They have this air budget and you can
go, well, you know, we're a three nine service committed to we
have forty four and a half minutes a month. Okay, five minutes isn't
great, but it's the fine. You know we can. Yeah, but
if you have you know, hey, I've gone past it and we're writing
checks or you know, giving SLA credits like okay, now you know we
have this quantifiable not how people feel number to talk about things and yeah,
so do you for internal apps, do you take them that far where you
give them assign them slis and SLOs and slas for performance. I've been starting
to so this is actually my first, my first gig out of many where
I've really kind of focused on internal stuff. I've often been the problem of,
you know, the person focusing on proud and you know, on a
different team that now my team is kind of you know, wearing all the
hats thing here, and yeah, so I'm we're putting SLOs on internal services
and internal workflows and trying to treat them with the same we have that in
production, of course, and we're trying to give it the same, uh,
the same level of you know, thought and execution. I've seen a
huge shift in the industry over We're all like where some teams maybe they're called
platform teams, have sort of been excluded from thinking about what a product is
or how they do product management or even product ownership. And I feel like
that's really been turning around, first with the DevOps movement and now just I'll
look at how we build services, micro services all all together and no one
sort of excluded or have a different process of internal teams. Doesn't matter.
You're still offering a real service to someone. Just happens to be your customers
or within the same customer or same organization. That's exactly how I sell it
is. Yeah, you still have customers, they just happen to be coworkers,
right. Yeah, you can still do the same stuff. You can
build user journeys. You can figure out what does a customer expect, what
makes a customer angry. It's a lot easier to figure it out because you
can ask them and slack instead of guessing, like, man, what are
my customers? What's the threshold at which my customer complains? You can just
straight up ask them when they work for you know, I don't think they
necessarily have good, like nice answers though, Like you're a customer, you
find there's a variety of personalities, right, people, the moment of Jenkins
job doesn't passes in four minutes. They're like, ah, Jenkins is graph
It's terrible, but at least you can get it. You know. That's
how customers are. You know, are in general right you have so you
get the whole gamut of emotions. But you can tell when there's an outage.
You know, how loud people are. Internal people will tend to be
louder internally that in your customer I mean, that's what I mean, Like
you could definitely like it's still a problem in some regard, but like how
do you temper the difference of perspective? Like I feel like users on the
outside are more quiet in a lot of ways, like it's difficult to pull
information out of them, and internally, as you mentioned, you know,
it's like everyone screaming as soon as something goes even a little bit. I
think that if you can show them that a there are graphs, right,
like there should always be a graph, you know, like generically I work
for AGRAD that always sad, where's the graphere's the graph? And it was
annoying at first, but you're like, okay, maybe there should always be
a graph so people know that You're, hey, people are actually watching Jenkins.
People are actually rather computers are watching Jenkins, not people, right,
And if you show like, hey, we had this outage and yes it
was a really terrible day and no one shift any God, but here's this
great postpartum everyone can read and that's got a reasonable trigger and root cause and
follow ups and timeline and we're actually closing the follow ups. Like it's very
visible and I feel that should be the same way with customer outages. It's
very visible, right, nothing, there shouldn't be anything to hype. Yeah,
So one idea I've been working on because I work a lot with startups
and like, the whole thing about a startup is odds are the product that
you launch is not going to be the product that you're successful with, and
you're going to try a lot of different things before you become successful as a
company. So how do we measure that so we don't spend any more time
than it's absolutely necessary working on the wrong problem. And so I've been working
on this idea of success criteria, like what does it take to make this
application successful? And a lot of my background is in the infrastructure for mobile
apps, and for mobile apps, you can measure it as you know,
we need ten thousand monthly active users spending forty five minutes per week in the
app or something like that, measure engagement. Yeah, yeah, yeah,
So like because then if you if you tie in, like you know,
our our total cost of acquisition to get a new user as x amount of
dollars, and our cost of infrastructure is x amount of dollars per user.
You know, you can do some pretty simple math there to find out how
many users you need for this to be a profitable product. And so trying
to figure that out and get that in the early stages of an application so
we know when to either double down on this application or shelve it as soon
as possible. I'm wondering if like that same thing applies to internal tools,
and how do you how do you define what that looks like? Yeah,
that's a good question. And I think I had like a good you know,
where we deployed the like a code analysis tool, and yeah, I
don't it's a good I don't know how to quantify it for stuff like that.
But again, we have some of the very active developers that will give
feedback of like, hey, this then gave me a bad analysis and which
we can and make it better. It's like, okay, that means they're
actually looking at it and you know, making code health better. And I
don't know, I think the like, I think you're on the right track
with you know, having a number like mus and that kind of thing.
It's tough internally, Yeah, yeah, well, and it's interesting because yeah,
well, I think there's two classes of internal products, right there's internal
products that people are very opinionated on. Code analysis is one, right,
Like, I think this gives me bad code analysis. I don't like that.
But CI right, I think largely people don't actually care what CI product
you're running. They care do my prs get approvals fast? Is master green?
The deployments happen at the pace that we think they should at our company,
and if that works, who cares what the product is? Right,
they're like, And so for stuff like that, you can just be super
results oriented, right, Like, hey, is are the s lives?
These our solos we're trying to go for we're making them, we're not making
them. We're a d inking shop now, and we're considering alternatives. And
of course, you know, you ask people what we should. There's a
million things, but no one actually cares what we run, right, you
know, if they can declare their jobs in some declarative way that's not too
terrible and it works, everyone's happy. Yeah, I'll tell you what'll make
them care. If they have to be the ones to convert your drinking jobs
to the new platform, then a lot of people will be like, ah,
you know, Chinkin's is probably okay. There's a lot of GitHub actions
talk because a lot of people have done stuff there. So I mean part
of you know, part of that is that's the politicals or well, the
human side of it is, Hey, maybe it's if all of the products
can do the thing, maybe we pick the one people have the most experience
with just to make life easier in the transition. That's a good point.
I mean, will you ask question, which is, how do we know
our thing is going to be successful at a startup level? And if we
take the perceived public metrics on ninety percent fail are what a companies do with
teams that are part of that ninety percent? Do they let them go?
Do they reposition them? I think that's a huge struggle and it's scary to
know that there are some metrics associated with your successful even within a larger company
that you don't have job security in. Yeah, I can tell you just
in my experience with startups, that's like when you identify the success failed criteria
answers that question because if you can identify it early, then you take those
people and you put them on what your next idea is. But in many
cases, if you wait too late to identify that this product is not going
to be successful. You've already been bleeding too much cash for church along that
you've got to salvage what's left, and the first thing that goes when you're
trying to cut costs is your staff. I wonder if there's a lesson to
be learned that can be pulled to internal apps. Though in the startup market,
there's this idea of letters of intent, right getting signatures from potential customers
even before you built anything based off of that idea, and maybe maybe there's
an idea here of how to transition this to even internal teams. Yeah,
I think, yeah, I like the idea of formalizing it. I mean,
I think what I often do for these kind of things is like you
find, uh, you know, hey, you might have ten teams and
six teams really care about this thing you're building because they're big use of it.
You find like a champion on each team that is, you know,
opinionated and is willing to talk to you about what's good and bad. And
maybe not a letter of intent, but you write a design doc and then
you make sure that they've read it and give you feedback, not just to
check, but you know, if anyone reads a design doc and has no
comments, they didn't actually read it. Like I've never met an engineer without
an opinion on a design. So so you get you know, you get
feedback from people on these teams, and it's maybe not as formal as a
letter of intent, but you kind of have the you know, social buy
in, right, Yeah. Yeah, And that was one of the things
that we whenever I worked at Active, I started with them super early and
we just accidentally got this right. But we built this platform for all the
engineers to build and deploy their software, and the way we built it,
we we had such great collaboration between an infrastructure team and an engineering team that
anytime the engineering team wanted to expand the capabilities of that platform tool, most
of the time their request came in the form of a poor request to just
add that feature to it. Yeah, and I've never been able to duplicate
that since then, but that had such a significant impact on how I think
about platforms that that's my goal every day. Yeah. I like that.
It's kind of the open source approach, right, Yeah, Yeah, that's
I mean, I think a lot of that is the buy in, right,
because if you're using a platform or a language that no one else wants
or knows or likes. They're not going to learn rusk to send you a
poll review to fix a thing they don't like. But if it's already in
go and the rest of your codes and go, yeah, I just show
up one morning. Maybe there's a corollary here as well to the total economy
in the startup world, where you know your customers are somewhere along the spectrum
of innovators, early adopters, early minority, all the way to laggards,
and it's really who are you talking to? You know, what does the
rest of your organization look like? Because I feel like, well, you
would need innovators and early adopters there who not only need that functionality but have
a huge stake or care about how it's implemented, and not just people who
think it's table stakes or just belongs there or have opinions that they just want
it done their way. Yeah, yeah, true, yeah, And that's
I think you hit that. I think you hit it right on the head
there that early stage startups require that innovation mindset. For sure. They're not
going to get very far if only a few people are thinking about how to
push what do you bring their product is forward. Yeah, I really a
great and I want to clarify that there. You know, there's there's a
certain type of person who just wants to show up for work and do their
assigned job and they want to do that for twenty or thirty years till they
retire. And there's absolutely nothing wrong with that, just that an early stage
startup is not the right environment for you to be successful. Yeah, you
definitely see that. You can find out the bigger companies. You have these
classes of employees, right, there are some people that you can tell either
came from startups or are going to go to startups afterwards. And then like
you said, there are some people that just you know, close tickets and
do the work. And that's yeah, completely fine. Yeah, I sometimes
wish that I could be that person. It'd be a lot lessful. And
yeah, I don't know, maybe I will be in five or day.
Who knows, right, people change, But yeah, So let's talk a
little bit about the types of internal platforms, because we've talked about you know,
like CICD, what other tools are out there that we should be looking
for as internal tools. One that pops up immediately in my mind only because
I've been bitten by this one at every single company I've ever worked at for
the last three decades is the internal data analytics team. Like those guys have
monster infrastructures and are just doing what it takes to generate the reports that the
business wants to see. And whenever you get a hold of it, like,
oh wow, I'm not even really certain how this thing is working.
So what are there to break? Internal tools are out there. Well,
some people run their own vcs, you know, get get lab on prem
h we run Garrett so that's its own you know, Carrien feeding and that's
the loos. I think one thing people actually miss is the monitoring infrastructure themselves.
You know, if you run Prometheus and Elastic or if you don't have
a vendor, right, and even if you have a vendor, you have
some component that is shipping metrics to them, right, you have to monitor
that. If that stuff all breaks, you're flying blind and you need to
know it. I think that often goes neglected, right. I worked somewhere
where someone came to us once and they were like, you know, they
run a really noisy on call rotation, which is its own problem, and
they knew things were broken because they had I haven't gotten a page duty in
ninety minutes, and that was their escalation and I was like, well,
it's actually broken, and this is so sad that this is how we found
out about the problem. On many levels, right one, we didn't know
too. You expect to be paged at least once every ninety minutes, but
it's it's truly I think the monitoring infrastructure, it seems like it should be
obvious, but it's not necessarily obvious that do you think that's Do you think
that's a mess from like existing observability tools that they aren't able to have internal
metrics on what the expectation is on getting logs, Like I feel like we've
used cloud Watch for a while and one of the things it does in AWS
is you can alert on missing data. Well that's a hard like in Prometheus,
it's very hard to alert on missing data right end of alerting on like
the uptime series being zero. But then what if you have a problem that
generates your targets and there is no uptime series right, Like there are a
lot of these different things. Yeah, I think that it's not Yeah,
I don't think it's first class of you know, watch the watchers. I
think the data is there, though, and you have to kind of do
it. I worked somewhere where we had a relatively large Prometheus and Grafana infrastructure
for metrics and a really large Blunk infrastructure for law, and both of them
had their problems and they're run much two different teams. And we got together
and said, hey, let's make a deal. Right, We'll write a
Probert for Splunk and have an export metrics and monitors Spunk and Prometheus. You
write a prober for Prometheus and have it writ logs and monitor and Splunk And
that's not perfect, but you make with what you've got. And that was
a company where it was a large place that was going to be hard to
bring in a third party tool to help us, and you know, but
we may do with it. And it was super successful. We found lots
of problems with you know, if they both died at the same time,
Okay, yeah, that's the end of the world. But the world is
probably already ending if they both died. So do you think there are third
party vendors that actually solve this in a reasonable way? Well, I mean,
I think it's hard to say, right, It depends what you're looking.
If you're using a third party vendor for monitoring, you know, you
need to look at your metrics of shipping. Is your data there with them?
Right? So you have to set something up in their system to do
it. But then you do you want to back up? Uh? You
know? Do you want to know if they're up? You may have to
run a little side monitoring infrastructure too to watch them because it might not be
anything you can do about it, but you may want to at least be
aware that, Hey, the thing that normally sends me alerts is not going
to send me alerts. Maybe we should all be go back to the knock
days and you know, stare at some things for a couple of hours.
I mean, that's exactly what I don't want to have to think about,
Like I don't have to think about my vendor being down in some way that
requires me to monitor them. Like I feel like, you know, if
I'm paying money out for that, I should I should be getting that by
default. I don't know. Maybe I think that's just my pessimistic brain.
Yeah, for sure, everything will break. Everything will break, So whether
you're building it or not, it's going to break and you either want to
know about it or you know, if you think something doesn't break, you're
just not measuring it. Yeah, that's kind of mine. Yeah, yeah,
I have two examples where I think that's a really like a lot of
effort has gone into that Core Logics as a logging and monitoring platform, and
they have anomaly detection, and so if you'll if you have a service that
normally spits out, you know, one thousand log entries per hour, if
it stops, it identifies that as an anomaly, and we'll trigger that and
say, hey, nothing is like technically alerted, but something's changed here.
And then another one that we just recently started using as a tool for monitoring
our infrastructure spin called clouds zero, and it has a really cool anomaly detection
as well, says, hey, this project, their current spend rate has
been this, but today it changed to this, which is you know,
not necessarily a problem, but cool that it acknowledges that and you're like,
yeah, why did that change? I think the correlation analysis like that is
really good for not necessarily paging someone itto a, but reports and debugging like
hey, I have a problem started two days ago, what else changed two
days ago? Right, Hey, you also started spending less money on this.
It's like, yeah, she doesn't mean causation, but I'm certainly going
to look there first. Now. I think people underestimate that really though,
like just looking out, when does the problem start? What else happened there?
And I feel like it's so obvious to say that, but it's usually
one of the first things that's missed. Yeah, for sure. It's been
like a pet project of mine for a very long time. I have a
super stale, busted repo trying to do this. Back when, uh remember
CEP continuous event processing, it was all the hype and like maybe it wasn't
all the hype, but it was somewhat popular in the late two thousands,
I was trying to do the uh whole Winters forecasting, Like the thing that's
built in is that you know already tool added that thing again ages ago,
dating myself. You could, you know, put the prediction line in front
of your graph, And I always thought like, yeah, let's just do
that, like let's predict every time series. Like it may not be great
signal and I'm not going to look at it unless something's broke, but hey,
I've got twenty thousand metrics. I'm not going to look at all of
them, right, So when the site car starts getting slow, it'd be
really interesting if I could also see, hey, i'll wait, time on
the database server went up. It's okay. Are we sending queries that a
disc go bad? Oh? Hey, look errors on this disk spindle started
going up too, Like maybe there's something here. And I think it solves
another monitoring problem that people still make is people monitor root causes and not what
they actually want out of a system, right, like sending me an alert
that my CPU is high at too in the morning, Like, cool,
we're using the CPU we paid for, like comes up back to bed right,
awesome capacity planning. But if it's making the site slow, page me
and say the site is slow, right, don't use me and say don't
page me and say job is doing GC pauses? Well, yeah, you're
running travel That's what it does all day. If they're too long, you're
going to fail a latency monitor and page me on the thing that matters.
So like, I want these root causes to be the correlation engine to kind
of help me figure out. Okay, well eighty thousand things go wrong,
and if you put an alert on all of those. You're back at that
team that notices you don't get paid for ninety minutes. Right, Yeah,
if you look at every team that has page fatigue, this is exactly the
problem. Yeah, most people don't even know why they get paid for ninety
percent of the things. It's organically grown over the life of the team.
Hey, this one time, you know, the database backups did this thing
terribly, and now we page on it. And then you have so many
of those it's like, well, there's eight thousand alerts and most of them
resolve themselves, and we just thrug our shoulders. And I don't think it
starts from a good place. Like I think there's this idea that just having
dashboards is valuable in itself, and it sort of propagates from there. It's
like, oh, you know, what are all the metrics we could be
collecting, and let's show so. And sometimes there is an organization that thinks
that there is some inherent value, like oh, this is how many requests
we're getting, Let's use it to get more headcount because we have to support
such complex things. Or I remember a previous company that thought it was really
cool to have a geolocation dashboard of where things were happening all over the world
related to their thing, And I'm like, you know what, I did
need to drive that with real data. I could have just randomly pinged a
spot on the world on a flat, you know, two dimensional map and
be like, look it's happening. I didn't need to know the lights coming
up all over Yeah, for sure. It's just so totally unnecessary. And
so I've been on this maybe personal vendetta here to make them be actionable.
So you know, what is the business impact? But more than that,
you know what will you do with that information? Is the site being slow?
Will you actually take an action here to make it faster? Or is
there like a run book or something that we can go and actually execute on.
Well that's another interesting part of it all is like the term run book
has been so ruined by so many places. Like if your run book is
log into the server and run the script, like yeah, it's just not
automated, and like I think our run book should be these are the places
you should look, like, if it's ever a thing we know that breaks,
fix it. Like, yeah, I talked to someone once that was
saying they were very proud. I mean I felt that or they built this
great thing and like we had this service the crashes all the time, and
we built this great thing that automatically restarts it. I was like, okay,
I see right. How about collecting stack traces and sending it to developers
and fixing the reason the service crashes? All like, yes, you should
restart the service when it crashes, no argument, But like, that is
not the end of your journey. That is the first ten percent of your
journey, right, you know, I you know, I love that because
it actually happened at one of the previous companies I was in this. The
name of the service was the Service Monitor Monitor, and it did actually do
this. The root cause, though, and maybe you've got some infinite wisdom
here, is they were using a library for math operations that had a memory
leak in it, and so this would have required actually contacting a third party
company to get their open source software actually fixed, which I think was under
a proprietor use license. So sometimes you have to do some ridiculous things.
Yeah, I mean, I guess if you're in a world where you have
to use a vendor library you can't fix, you may just have to you
know, live with the workarounds and acknowledge the terror. Yeah, no,
for sure. The problem is when you start expending that to every problem that
looks similar, rather than knowing that it's the right answer. Right the Nightly
Jenkins restart. Yeah, well go ahead. I would say, like,
you know, a lot of some bank places I've worked, we're like,
hey, we should just you know, we're twenty four by five point five,
right, you know the Japan and US markets and they're closed, you
know, Friday night. Let's just reboot everything every Saturday. I'm like why,
Like why not? And I was like, I think I asked the
question first, like like yeah, we should. We should reboot things when
we upgrade the kernel, and we should use that time to apply upgrade.
But just doing a bunch of stuff every Saturday because we can because there's no
services. It's like it feels like we're just making busy work and you know,
finding things to break on a Saturday and ruin our weekends. Yeah,
but there's also really hard arguments to argue against, even though you know there's
somehow fundamentally wrong. Yeah, we didn't do it. Regress, Yeah,
someone you know, there was a change set out to put a reboot in
a crown job, and I was like, that is no, that's something
is going to be terrible there, and I don't want to that's how to
ruin a weekend one on one. Yeah, totally. And it was already
like that because I don't know. I think twenty four by seven is a
way better environment to have fundamentally good operations than twenty four or five five.
There's just too many bad habits in twenty four or five five. Oh,
we can always restart this by turning off the database and running you're turning off
the clients and running an alter. Eventually you're going to have to run an
alter mid week during trading. Yeah, you're not gonna like it, but
it's gonna happen, and you're gonna have to take an out us to do
it because you haven't figured out how to do it. I mean, there's
two obvious failure modes from now, which are, well, what happens if
something changes on the restart, like you know, a new upgrade or something
right now you're triggering that at an unpredictable time, or just straight database replication
crashing and losing out on whatever what was in the journal at that moment,
that's not the thing you want to actually have happened. Yeah, do it
in your lab to figure out how to deal with it, but maybe not
proud. Yeah. So You've mentioned Jenkins and Garrett, and so I feel
like I'm picking up on a trend here that you're running a lot of services
in house, whereas other companies may choose to use SaaS providers for those Do
you have a particular opinion on that? Well, I mean I feel like
my whole career, I've been at all angles of the build versus by debate.
It was funny some of the banks I've worked that you really any of
the big orgs you see, like the historical cio CTOs whoever they are,
or one wile will get hired and there we have to build everything. Then
then you can you know, you have ten years of this right, and
then you look back and you have all these disparate things of like, oh,
this must have been built during the build era of two thousand and nine.
I mean, I think the answer is, really it depends on the
staff and what their expertise is, and should you run Jenkins in house?
What can you run Jenkins in house? Right? Like do you have are
you going to dedicate the resources to do it? Right? But just outsourcing
something isn't always as easy as just paying somebody, right we mentioned before,
it's porting stuff there, it's operationalizing it, and actually, like taking taking
advantage of a SaaS service to do provide value is sometimes just as hard.
The challenge isn't running the infrastructure, it's using the infrastructure effectively. Right,
Having Jenkins doesn't help. Having jobs that do meaningful things are important, and
you have that problem in any CI infrastructure. So I think some of the
things that doesn't matter, you know, observability, right, it's really hard
to run high availability observability. It's also I'm mouthful to say so, I
mean, like I'm a fan of I mean worker reservability company, but I
mean I'm a fan of outsourcing some of that stuff. I think there's a
certain scale where you run it internally, I think very few people are at
that scale. Yeah, very few people are actually running for four nines proper,
Like that's I don't know if it's a triple digit number, but it's
a small amount of companies and actually a lot of people think they're doing it,
but not a lot of people actually do it, right, It's interesting.
I mean, we're we're five nines on our core competency service, which
is like identifying stuff, but yeah, it's huge. We're not we're not
running the monitoring observability stuff ourselves like we are, like we've found we're using
our cloud provider or actually we're still in the process of trying to find a
vendor that actually works with us. And I think that's going to fuel my
next question. I'm curious whether or not you see companies get the build versus
buy decision, right. I know that they put a lot of effort into
comparing vendors, but then I feel like it's sort of there's this gap on
actually being able to correctly identify what the total cost of ownership is if they
do actually go and build or run something themselves. You know that that is
a tough one, right. TCO is so hard because it's easy to do
the this is my AWS bill, this is the vendor bill. How do
you quantify the adages and the people and the stress and the upgrades? And
yeah, I haven't seen a good answer for that. I mean, of
course many you know, any vendor will try to do that for you because
they of course want to help. Yeah, good, but that one is
tough, I mean, but seeing it done right. I think the most
common mistake people make when they're doing the evaluation is writing a requirement stock is
surprisingly hard. Yeah, like writing a requirement stock for CI and don't use
a single product name right right. People say, like, my requirement is
an Envoyd proxy. It's like, there's absolut no way. If your requirement
is an on Boid proxy, what have you done? Right? If your
requirement is a thing that speaks XDS, maybe envoys the only answer. But
write down the thing you want, the outcome you want. This is the
same root cause versus slow thing, Like it's a it's the x Y problem
as well. I don't know if yeah yeah, people say I want to
do this thing is like, what are you really trying to accomplish? And
then let's figure out this is what we could build and accomplish that. This
is what we can buy and accomplish that or not accomplish that, and maybe
the decision gets more obvious, but it's just so hard to frame it in
terms of completely agnostic to the tool, the thing you want to do.
You get to be the bad guy. You you have to say, well,
you were going to do an easy job, which is just pick a
tool. Uh, and now you're forcing them to go back to the drawing
board like well, why you know, really really look right? Why do
you want your builds to succeed? Yeah? Exactly, But it's a good
question, like in the CI case, like why do you want your bills
to succeed? Because I want to merge code faster and ship code to production
faster. Okay, so your metric is time from commit to development, time
from commit to production. Okay, we came up with really great metrics from
asking guard. I think it usually it seems kind of like, I don't
know, weird at first, and you know, contrived, but I think
it actually leads to like, oh, okay, yeah that makes sense.
No, I'm totally what do Yeah, it makes me think that there's like
some Freudian stuff here where I just want to sit here with a pipe and
go. But why do you want your bill to succeed? Well? Is
it because of unresolved issues with your mother that you feel your build business succeed?
I mean when you pull individuals into an organization, their personal values do
impact what that organization drives as important and sometimes I've known software engineers to be
uh, quite illogical and driven by their emotional state to you know, it
has to be like this, it's so much better, and you know,
you joke, but there is something there. No, that's definitely I mean,
I'm probably guilty of that. Yeah. So you work for a observability
company and you're observing your internal tools. Do you use your own product for
that? We do? Yeah, Yeah, trying to dog food all the
time and say we measure SLOs internally. One cool thing we've done is we
have a bunch of you know, it's not great, like you know,
some some shell scripts stuff that maybe shouldn't be shell scripts, but everyone's got
a pile of that somewhere and it's pretty important in the normal workflow. And
it was having a lot of problems, and uh, you know, we
just found a better way to measure like what is the success of people that
say I want to run a local cluster? How often does that fail?
Turns out it was failing a lot more than we thought. And then the
buggy. It's really hard because you're like, okay, well could you go
put a set dash X in this file? And run it again in the
mouth, and no one wants to like, no one wants to do that,
right. People just want to say, hey, I think broken,
I can't you just fix it? Right? So we built all of this
in even to our shell right, so like we have like a tracing view
I think of micro service tracing. Like there's a request I D and you
know it hits tny services, Like you run this one shell script that actually
is running like you know, ninety three shell scripts underneath it for better or
worse or worse, but it's happening, and we have like a trace graph
of you ran this and like so the people come and say, man,
it's took nine minutes to start this thing and it normally takes six minutes.
This sucks. Why is this? So we can go in, look up
their user name, find with a nine minute run, click it, look
at the trace, and go, oh, yeah, there was this bug
pulling from ECR or whatever it is, right, and just doing that helped
us pin down. It turns out all these problems with the tool were like
systemic to one or two things being wrong, you know, poor assumptions being
made, and we fix those and we're kind of off to the races on
it, and now we're building a prober for it, so we'll have like
a graph. You know, again, everything has a graph, right,
So the same way we have an solo for ingesting an observation in a certain
amount of time, we'll have an sol for Hey, this thing can create
a environment locally and it happens in less than this amount of time. And
then we'll have variants of it, like does it work for people that run
it over and over again on their box? What about a new engineer that
logs into a fresh box and runs it, because that's always a different you
know, there's some you know, that's where always the goblins live in these
things. Well, my terraform works fine? Oh man, I destroyed it.
Have to run it from scratch, you know, does that work?
So kind of trying to measure all those different things from that using our tool.
Is there like an eBPF integration here that you plan on utilizing in the
future to understand what the requests are or how the script is running fundamentally on
the machine. Yeah, I think I think we're looking at that, and
I haven't dealt with it too much myself, but I think that that is
kind of the ultimate for this, I would I think marry the two right
VPF for the raw like just show me everything that's running and helped me and
then my injected data. So yeah, I think that is in the future
for your internal tools, Like once you identify them, you're like, Okay,
we need to bring this up to being like a part of our we
need to treat it like it's part of our core infrastructure. How do you
socialize that across engineering so that everyone knows that this is a supported tool.
This is a preferred path if you were thinking about going and building something on
your own for this one use case, we already have you covered here.
How do you communicate that email to the whole list? Pending your Slack message?
What could you go wrong? Right? Channel? Baby? You can
write documentation all day, but like I think we all know how much our
documentation gets read, right, I think I think the way to do it
is to this you know, goes on the layer aight problem we're talking about
earlier, but like have to build a good relationship with all these teams and
have them come to you sooner in the process, like the sooner infrastructure and
s E. Folks can be involved in anything, like before code is written
would be super ideal, so that them come to you and say, hey,
I wrote this really cool thing, but it uses mago dB. It's
like, we don't do that here, right, Like maybe there's a great
reason that does that, but you know, but it's much harder once they've
written code, right, Like, you know, it's the poker pot committed
things like while I call for the fluster on the flop, I'm putting all
my money in on the tourna matter what. It's like, let's do the
math. Not great, get to them, you know, pre flop,
right, Like should you even be in this pot with mango dB? Right?
Maybe? Yeah, but that's not easy to do, right, You
have to There's not a technical answer there. That's just build relationships, make
your team available, make people you know, had interactions people like you know.
I think it was back to the sort of product management thing earlier that
we were talking about, where if you are at the point where you need
to tell people about the thing that you're working on, like maybe you didn't
approach the situation necessarily in the best way rather than driving it from how you
would a startup, which is okay, you know, what do our customers
pain points look like? And you know, as users, what do they
want? And they're coming to us and saying, hey, when are you
done with this thing that we asked for? And the implementation details are of
course what you're picking because you know that best. But fundamentally it's just a
matter of pinging them on whatever RSS feed that they're looking at. Yeah.
I worked at a large company before and we had an infrastructure PM and it
was amazing. It was just so it felt like it was easy mode,
right, Like someone else is going to go gather requirements I don't have.
Yeah, I just I'll go do some work. That's that's fine. And
then they show up, you know, prioritize on a list. It's like,
well this is this is awesome, this is what it's like on the
other side, right, you just need to plug that into chat GPT and
get the answer out as well, and then you can just you know,
stop doing all the work altogether. What do my engineers want? That's interesting?
Get a slack bot that uses chat GPT to act as a PM role.
I was just the other way around. Have the pms just answer,
like answer the question of what they want to have built, and then it
will automatically build it for them. This is what the hive mind Internet wants
to build. Probably not so bad if everyone wants to. I don't think
you'll get to five nines. I think it's more like five two's will be
in the front in there somewhere if you carry it out to enough decimal places.
There's some nines. You know, non repeating imager and doesn't really good
cool, So what else should we be thinking about? For internal tools?
That's your big takeaway piece of advice. Apply all the same riggor Like you
know, when PROD breaks, you go and declare an incident using your incident
management tool. You have a communications role, you have you have the whole
thing or you got to run, but hopefully if you don't, you should,
And then when you're done, you write a post mortem. You might
even publish. The customers publish it internally, right, Like why shouldn't an
engineer be able to read about why Jenkins broke? If production breaks, you're
going to file a bunch of follow ups. You're going to prioritize it over
other works because we don't want PROD to break again. Same thing for internal
tools, Like it's really I think that's the big takeaway is there's really not
much of a difference. I mean, and people, well, if PROD
is down, our customers that pay us can't do work. Okay, great,
if CI is down, how are you going to ship a fix when
PROD is down? Right? Like you're like, oh, I brought this,
I wrote this really great infrastructure where changes can only go through CI and
not handmade. That's a great thing. But if your CI isn't as of
the ball has production three nine CI four nine is production. You can only
make changes through CI. You know, not a math major. But we're
going to have a problem here forty minutes out of the year. You know
it's going to be an issue. So I think that treating it, I
think people just underestimate how important that stuff really is and the impact it can
have. You don't have to wait till it. You know. The worst
case is production is having a problem. You're monitoring is broken and you can't
see it. Your CI is broken and you can't ship a fix for it,
whether that fixes can FIG or code, I mean, and then that's
a really terrible postportum to have to send out to a customer of well,
we knew what was wrong, but we had to fix. We couldn't build
the dockor image that had to fix because you know our two thousand and four
era Jenkins decided it's a crash and I'll start it. Yet, Yeah,
that's not It was not a good look for anybody. I mean, you
identified it. If there's value in doing this activity for some of your services
because of what the users look like, then there's probably value in doing it
for other services that just happen to be internal. I think that's a big
part of post mortems, right, Like if you find a production problem where
oh, hey, we had this bug in our database connection pool and we
had this reconnection issue and this thing happened. Where else do you have connection
pools? Right? That should be the logical question. You ask the same
thing internally, Right, we had this neglected service. Where else what else
is flying under the radar that's going to bite us? And those are the
tough post you know, some of the post mortem actually like fix this bug
with a reproduced case. Okay, that's like a day of work, right,
you know, when you're done because the test passes, it's easy.
These are much more ominous, like project the follow ups. But if you
don't do them, you're going to pay the price. Is there like some
obvious pitfall that a lot of companies or maybe even everyone seems to get wrong
in this area sort of besides the stuff we've been talking about, well in
post mortems, people are extraordinarily bad at distinction between root causes and triggers.
Right, Pete type this command and took the site down? Right, root
cause is not Pete sucks? Right, that's right? Like the root cause
is why was there a command to where are the where are the seatbelts?
Where are the gates? Where is the code review? Where's all that stuff?
Or you have to really really I think the five wise thing is interesting.
I don't think you actually have to write down why five times and a
documented filled out I think that's a little bit you know, the meaning and
not great, but the philosophy of like, really come to the root cause
of the problem, right, Like root causes aren't well? Aws had an
availability zone die? Like, what can we do? You can run more
than one availability zone? You can do this, You can do that right,
like, and maybe you choose not to at this point, but you
should at least identify it and say Okay, like we know the root cause,
and we've chosen that this is a risk, and this is why we're
a three nine service or a four nine service. And maybe someday will make
it better, maybe we won't, but at least being honest with yourself about
it. Ah, that's huge right there. I want to highlight that because
like, just because you identify the root cause doesn't mean you have to do
anything about it. Because I've seen multiple instances where companies build infrastructure that is
far beyond their budget and their actual requirements because they're focused on that. And
the analogy I like to use is like, whenever I go to work every
day, the fastest, most efficient way for me to get there is buying
my own jet copter, But my budget really says I should stick with my
eighty seven Toyota Corolla, you know, and so you have like balance those
two things, right, right, Maybe the fix is leave ten minutes earlier
instead instead of mind Yeah exactly. Yeah. Well I think also it comes
back to SLOs of like, Okay, we had this big problem, but
ay was it a big problem? You know? Like it's hard back to
the emotion thing, right, Like if a big customer is impacted, it
gets a lot more priority, but you have to quantify at the end of
the day, like, hey, this is how many nines we have.
This is our error budget. We used it during this incident. Maybe that's
not good, but that's this is par for the course, and we don't
need to suddenly become multi cloud, multi region, you know, all the
things load balancing, complexity, because that's the other thing is looking at.
You know, you can add nines with complexity, but complexity can also reduce
nnes. And you have to be careful over engineering and response to things.
And I hate to use the term like you know this is always the escalator.
Well, if there's an act of God that you know, this managed
service goes away, it's like, well, what's an act of God?
Is that any going down? I don't know, that's just expected, right,
Is it a tornado hit the Virginia area and all of the US's to
one one away? Okay, there's an act of God we don't have to
plan for. But if you're trying to be a five nine service, you're
not running in one region anyway, So you know, it kind of all
has to that has to make sense together. You know, the whole story
has to just kind of flow. And that's where I think some people get
off. They they write an SLO, but they don't have a story to
back it, or they write a really complex story where they don't need for
an solo that's simpler. Yeah, point for sure. But it's an art
so it's you know, there's no right or wrong answer. I always say
solo everything is so technical and solos or this like walfty, what is right?
What is wrong? Who really knows? You just kind of have to
do it. People always we were rolling out SLOs in a company. How
do I know my slo's right? I was like, well, you know,
it's not. It's your first SLO. So you build it and you
have post mortems and you adjust it as you go. You know, were
you at you know, did you have an outage? Yes? Did your
solo show it no cool? Make it more aggressive? Did you not have
an outage? And your SLO says you had a outage? Make it less
aggressive? Right? Like it sounds simple, but that's just the feedback loop.
And if you're doing it twelve months later, you probably have a pretty
decent setup and have a much better idea of what your customers consider an outage
or not. You hit on something really interesting there. Actually, So if
SLA is your you know, contracted amount and the and the I is whatever,
your indicator is always just an objective. So it does seem like it's
it must be subjective in every way. How do you sort of pick that?
How do you know what? First off, if you have an A,
the O must be more aggressive, let's hope, right, right,
But let's assume for a moment you don't have a contractual you know, I
say, you know slas are like SLOs with lawyers, right, it's really
kind of a yeah, you have to guess, Like I mean, I
think that's the unfortunate of it. And that's where you know, hire somebody
with experience. They'll be able to guess more accurately maybe, but they'll at
least also know when they're wrong. So I don't know. It's a process
where you just have to iterate and sometimes you have a major miss You're like,
oh man, we really thought we were measuring this service and we had
this massive outage in our dashboard. Was like everything's green, nothing's wrong,
and users are entre It's like, okay, well we missed this super critical
part of the picture, and then you do the postmortem thing and you figure
out, Okay, well I made this common mistake. What other of my
solos have this common You know that I make this mistake more than once.
Would it be fair to say that it should be meaningful so that if you're
violating it, then you're taking some action as a result, and if you're
not, then you don't do anything. So maybe it's about finding that sweet
spot where it causes the right thing to happen in your organization. Absolutely,
I mean I think that. I think to get there right, you need
to have some kind of alerting and reporting around it. I think alerting on
solos is like a very hard problem. Like the Google book will talk about
burden rate alerting. Have fun implementing that, right, that's very hard.
But if you can get there or have some approximation of it and report on
it, like I'm a big fan of getting you know, it doesn't work
right away, but eventually maturing to a point where every alert has a ticket,
right and the ticket is either fixed the thing that caused the alert.
You know, this needs to be more resilient, This needs more replicas or
this alert page mean nothing was actually wrong. We should fix the alert and
if you do that over time, like that's how you can get solos and
cause change, and when they really are broken, the post mortem loop is
the real fix. Like the trick with SLOs is you kind of I don't
know. There's that perpetual battle with infrastructure and product right You're like, hey,
we need you guys to write more stable code and products, like we
need features, we need this needs to be a different color, right,
which is fine, like that needs there's a balance there. But if you
get everyone to agree on SLOs and you're really user focused, like, hey,
users are happy when the latency you know p ninety nine latency is this,
and you just are sad when it's over that, and everyone everyone,
like business engineering management, agrees on that. Your argument is a lot easier
when you have an outage to say, our users weren't happy, Like objectively,
our users weren't happy, So we either fix the thing or we decide
that our threshold was incorrect on what is a happy user. But one of
the two has to We can't do nothing. We can't just write features and
say, well, users might be unhappy again, because we know this thing
will blow up, and I find that that is a good you know,
you almost trick the trick product into it and like, oh yeah, so
those these are great, and then later it's like, oh, yeah,
I guess we have to fix this now, so you know, we'll give
we'll give you a sprint or you know, whatever it is. You know,
I did the same trick a long time ago with okayrs. Very similar
thing, where you know, once they agree to the okay ares, you
try to set the mindset up what does this actually mean? You know,
why are we setting it? And you set the so we can know what
to do when we know what to do the right thing, and then later
you can just point back to it and be like, hey, you know,
we decided what the right thing was going to be in this situation.
Now it's time to execute on it. Or you know, you have to
make a trade off. As you said, Yeah, it's a tough.
You know, you can't sell people problems with tech, but you try.
That'll be the million dollars million dollars you know thing. But I don't know,
I feel like data is always helpful. You know, emotions are bad,
you know, everyone has emotional arguments, but data is hard to People
still argue with data, but if I feel like if you're out the side
of data, you at least have a fighting chance of pushing for change,
or how do you know, like you ever feel like you're in a situation
where you get scared of survivorship bias, where you even if you're doing root
cause analysis and you're finding really what the underlying problem is that even though you're
going on and fixing it, that you're missing some other side of the iceberg
that is waiting out there to come and crush you totally. But I mean
I think that I don't know. I just accepted it, right. It's
a pestimistic, sary brain, Like everything will break. Everything I fix will
also break when I leave a company, everybody will blame me when it breaks.
And that's whatever, you know, Like I've just I've internalized it and
accepted it. And it used to stress me out a lot more and now
it's just like, Yep, this thing I'm rolling out might break, and
you know, at least you know that if it breaks, you're going to
write a good postmartum and learn from it. And that's kind of the Constellation
prize of like, yeah, it sucks to write a big Postmartum after a
big adage. But also you know, that's that's the gig, and like
we shouldn't have hired Pete, that's that's the R. Well, you know,
like the first place I ever with FedEx, right, there was this
meeting every week, the R squared Meeting, the Redundancy and Reliability Meeting interesting
and it was like how good of a week is the VP having? Right?
And there was always this rumor that like, oh, somebody was fired
once at this meeting because they had an outage, and like no one can
actually tell you that person's name, what year it was. I'm pretty sure
it was, you know, it was trying to hype you up to prepare
for it. I think it was. I don't know if they intentially did
this or it was just a grow but like, that's such a terrible you
know. I think the blameless culture is big people can't be a problem,
right, But I think often it's not the person that made the change that
is actually the problem. It is someone's reviewing code, right, Like code
review should be first class, not a checking a box. I feel I
feel really bad when to change I reviewed cause an adage that happened recently.
I was like, who approoved has changed? Oh, never mind, it
should have done better due diligence, And yeah, I mean it goes back
even further than that though, because if you do think that it is one
person's responsibility that caused the problem, you can look at, well, you
know, what was the culture we had that allowed them to make the mistake?
Or you know why were they even hired? Right? Were they a
good fit for the role in the first place? And you can definitely go
back up somewhere else or a different chain to really dive in there some of
those things you might not write. It's like a MetaPost mortem of the Yeah,
for sure, how do we get here? Right? How do we
end up hiring people that understand networking for a networking product or whatever? You
know it? Right? Yeah, I think that's suff's important. And then
then change the process, right, Okay, these are a new criteria for
this for sure. Yeah. Yeah. When I was in the Navy,
we had this process for getting your certification for whatever job you were doing.
That I've tried to bring into poor request reviews that it takes a cultural shift
to get fully implemented. But in the Navy. It was set up so
that, like I was a nuclear engineer, so you had to learn all
these different skills to operate the power plant. And so you would go around
the power plant and work with the existing engineers and show them that you knew
how to do something, and if they felt like you understood it, they
would sign it off in your book. And this was way back pre computer
stuff, so you actually had a physical book that you carried around, but
you signed it off in that book. And then if at any point in
your career you ever screwed that task up to a point where your skills were
called into question, they would open up the book to see who signed it
and go back to that person and say, hey, why why did we'll
screw this up? And and so it was that like that. It did
put that sense of pressure on you so that before you would sign off on
anyone's book on anything, you wanted to make sure that you were reasonably confident
that they actually knew what they were doing. And so I pre poor requests
the same way like if I approve a poor request and it breaks something,
I don't consider that to be a fault with the person who submitted the poor
request. I consider it to be my fault for not catching it in the
review. This is like the Hurdos number corollar ate the uh, there's the
you know, you chase it back all the way up the chain, like,
well, you know, who did that person? You know, what
does that person's book look like? The reviewer? Right? No, I
think that's a good, good way to look at it, and it's just
good for accountability and you know, yeah the meta issues. Yeah, and
like a one off instance, you know, is not that big a deal.
But over time, you know, if like everyone that I signed off
on this particular skill is having problems, that's going to point back to the
root cause being me not actually understanding either what the skill is or how to
evaluate that skill. Yeah. I've been places where the root cause has just
been fatigue. It's like the call rotations are too insane. This was at
the end of this person covered someone else's on call. They were on two
weeks of twenty four by seven on call. They had had a bunch of
major incidents overnight, they were running on no sleep, and they just simply
did the wrong thing. You can't fault the human for that. Why do
we put them in this grinder like every other industry. Airline industry is a
great one, right, you can only have limit limits on how much you
can fly and be responsible over people's lives. I mean, obviously when lives
are instakes, like you said, you have to be more rigorous. But
it doesn't have to be that rigorous for you know, being on call.
But you know, we have an informal policy on our team is if we
take overnight pages, someone will offer to cover the next day, the next
night so that person can catch up. Yeah, there's nothing worse than having
a week of terror where you're losing sleep every night like you're just you're dead
by there, and then a major incident. You know, the worst timing
always happens in these things, right, Like the worst adages are never one
thing. It's a confluence of events. Right, So that terrible outage is
going to come Friday when you're running on fumes and your brain isn't isn't there,
and you're going to make mistakes, you're not going to see problems and
that's not your fault. But that's tough to in a startup world like oh
yes, you're meant to grind, but there has to be some reasonableness of
Okay, the people with responsibility are keeping the sight up need to be aware
and awake because we're not just running run books where we copy and paste off.
It's the run book is use your brain. Yeah, I think we
really realized over the at least the most recent decade that the grind is not
helpful. Even like doing more hours of work, especially in knowledge work industry,
does not translate to an additional value. And so if you do have
outages every incidence, every day that people are on, I like, it
seems like fundamental that you would intentionally rotate them off so that someone else is
there because and you know, I think there's like a pride issue here where
the engineer just wants to stay on because you know, it's their rotation and
they don't realize that it's actually harming the company. Like they should speak up
for the benefit of the company. It's not about them necessarily. Right.
Heroics shouldn't be I mean, it's great sometimes heroics just have to happen,
and they happen and it's good, but that shouldn't be the goal of like,
oh man, I want to be a hero of the Saturagye, please
know, like it was really fun early in my career, and now it's
like, oh, there's heroics happening. This is awful. Yeah, it's
it's the cult of the hero because then you you fulfill that hero role and
then you know, you get at mentioned in slack and you know, everyone's
like, oh wow, that was such a tough effort, you know,
and the worst yeah, the worst word to use the NHR rockstar. Yeah.
Absolutely. I don't want to be a rock star at work, man,
Like, please know, I want to do I'm going to be a
rock star. I want it to be on Motley Cruz, fueled by cocaine
and hookers. Let me be a rock star, like being a rock star
and infrastructure is like, yeah, where's my hotel room to tear up?
Yeah, I mean at the beginning it was oh they described me as a
rockstar, that must mean I'm doing great. And now it's like, oh,
now it's like what your mouth? Yeah, don't call it that.
But I think I think the culture is Yeah, I don't know. It's
it's a tough one to just you don't want to actively discourage you. You
don't want to say long off of the home, like if you want to
work. I don't know. My thing is like I'm a workaholic, and
I always say I'm happy to work long hours if it's what I want to
do. If I want to work on Saturday, awesome. If other people
want me to work on Saturday, that kind of falls apart from me.
And the social engineering trick is that, well, just to keep work that
he really likes and it's passionate about, and he'll work on Saturday. That's
a fine. Okay, that's a good trick. It works on me,
but I don't think people have used that. And some weekends I game all
weekend. Some weekends I work all weekend. But usually it's it's my choice.
The point about that, I think part of that is that we are
creative in the work that we do, you know, using our creativity to
solve problems. And creativity doesn't doesn't show up at nine am when you hit
the time clock, you know, and if it's something that you're excited about,
you you get this, you know, that creative bush like, oh,
I got to go do this, which is what leads you to you
go in and work on Saturday. Most of my best work is done yeah,
after midnight or on the weekends correct, Like it's exactly that is.
You get the idea and you're like, man, you're laying in bed and
you're like, I can build it this way, that way. It's like
past draft to build it right, Like I know. It always fueled me
was there was some really annoying problem that I did, like some other problem
I didn't want to have to solve, Like someone was asking ridiculous things on
how to fix Jenkins and the solution was something I didn't want them to do,
and so I felt the need motivated to just have this problem completely go
away. And that's when I would really work on things like non stop wast
nerd sniping. Right, someone can't be done see you Monday. Right here
it is, but I don't think. I think the other problem with that
is it has to be clear on the team that that's happening. Like some
people just don't ever work weekends. I think that's great, Like you know,
that's that should shouldn't be a problem. So it shouldn't be a peer
pressure of like, oh Pete work of the weekend, everyone else should.
Like I think that is a terrible message to send, so you have to
be careful with it, like I've learned not to send too many emails.
Well, I don't. Luckily the company I work at it's not an email
company, but I like big companies where email is life. I'm always very
careful about setting emails on the weekend because you're implicitly setting expectations for other people
that are watching, and I feel like that is trouble. Yeah, I
mean that's another avenue realistically, Like, I think there is a thing about
just like you want the load on your systems to be constant. You don't
want to see spikes because they're incredibly hard to deal with the same goes for
teams that are putting out work. Right, If some engineers are incredibly spiky
on load, then you're unpredictable and what you can deliver and how much,
and the reliability or quality of that. So it's you're not necessarily doing anyone
a favor by one day going out and solving a problem. If that's your
pattern. Yeah, I think it's a difficult lesson for a lot of people
to learn. I struggle with it a lot. I often find myself like
I'll work a weekend because I want to, and then Monday, I'm like,
oh, man, I don't want to work today, Like, yeah,
well that's fine, right, because then you're then you're still having the
same amount of work that you're sort of putting out and productivity for the team,
but you're not overburning them because someone will has to review that, right,
someone that's still creating followup work down the road, and depending on the
statement, maybe that creates incidents as well. Yeah, now, you're right,
it's a very tough balance to figure out that. Yeah, I'm still
trying to figure out what that is. I would say my whole career is
pretty spiky over There are some places I work at where I I'm always you
know, I can't help the work aholic at me. But there are some
places where I chill a little more in some places where I'm like eighty hour
weeks insanity, And I don't know. For me, I've learned it's a
healthy cycle. Like every couple jobs, I do the insane grind because that's
just where I'm at, and then I have a couple of years of like
less insane grind, kind of relax and find other things to do, and
then bright back to it because I miss it, you know. I mean,
it doesn't matter what your pattern is. I mean, if you are
spikey, it's fine as long as it's somehow consistent. Right. You know,
if every couple of weekend, you know, every other weekend, you
do extra work, then you sort of expect that into the realm of things
and how that's going to play out. But if it happens and it's unpredictable,
then you don't know what the impact is on the team. You may
think that the team can have more work done than is reasonable, and so
a new big project comes out and now it's taking even longer or unexpected because
he's not pulling those weekends anymore and doing the a real job that has a
lot of value, but messed up some sort of prediction or timelines or deadlines.
I think engineering time prediction is like the hard I just throw that out
the window, especially with infra. Right, well, it's two weeks of
infra work. Oh so it'll be done in two weeks, maybe like two
weeks, And for work sometimes takes two months with outages and interrupts, and
it's hard to explain sometimes. Yeah, it's like Einstein's theory of relativity.
This is real the fast where we go this is different where we go no.
But I think going back to your you're talking about working in spikes.
I think that's like, that's how we've worked as humans for you know,
forty thousand years. Like you go out and you do the big grind to
you know, to to hunt the animals, and then you go back and
you just your rest and you relax for a while. Or you go out
and you work all summer to plant the crops and harvest the crops and then
store them for the winter, and then you ride the winter out. So
I think that behavior is actually something that has been native to us for a
long long time, and to try to break that in the course of a
three decade career is going to be difficult. I mean you're on as a
there definitely what are you called it? Art? Right? So you look
at the Renaissance artists, famous ones like you know, see what they did?
What are other artists doing before? And even today? You know,
how are they working? Because that's the same of expectation you can have for
any knowledge work, which is very similar to a creative process. You have
to have the right motivation and sometimes that's really hard to figure out. I
don't know what that is sometimes, right, some weekends, I'm just I
want to be at and stare at a screen. And some weeks I wake
up early on a Saturday and I'm like, let's write some code, let's
do stuff, And yeah, I have no idea what what drives it?
Sometimes I know, but often it's just I don't know. It's hard to
say how I'm feeling or will feel until I actually wake up. And that's
what to do for sure. I think if we figure that out and can
define it and reproduce it, we've got our next multi billion dollar startup.
Seriously, awesome. Is there anything else we should talk about for internal platforms
of infrastructure? I think we covered a lot. I think that Yeah,
all the things I want to talk about awesome. Cool. Let's do some
picks. Warren have been picking on you for picks the last couple of episodes,
but I gave Pete the heads up before we started recording, so I
know he's preps. So I'm gonna put Pete on the spot. What'd you
bring for us? Pete? I'm a gamer, and I like, uh,
I don't know. I like games that are really hard. I like
games that are kind of like a second job, and grindy and I don't
know, I'm go utton for punishment. I guess this is the summary.
Some people play games just to have casual. I have some casual you know,
hang out with the boys and play some games kind of thing. But
I like spreadsheet on the second monitor and gaming on the first monitor kind of
thing. So I really like aarpg's you know, hack and slash anything where
you can mind max. It's just interesting. Up my alley. A new
game came out. I don't know. I think I went full release ten
days ago. It's been around for years, and like betas and Alpha,
it's called last Epoch. It's like a you know, Diablo ish ARPG thing.
And there's Diablo. I played a lot of Diablo. There's Poe,
which is like the ultimate. Like I don't have enough hours to I'm also
like a very addicted personality, so I have to choose my gamescre I don't
play a factorio because I know that I would just stop working. Like Last
Epoch is this nice middle of It's a lot more complex than Diablo. There's
a lot more things that can contribute to your final build that you can nerd
out on trying this and trying that there's a lot of RNG, so there's
the grind aspect of it. It just checks all the boxes. And for
me, I know a game is good when I lose track of time constantly.
Oh it's a very am I should go to bed now. That's been
like the last week and a half as the game came out, And so
that is my pick. If you like ARPGs or min maxine games or things
like that, this game just checks all the boxes for me. Written by
gamers. Sometimes you play a game when it's pretty clear that the people that
wrote it don't actually play the game they've written. It's actually way more common
than you think. And you realize because there's no quality of life features and
it's like, well this is awkward and I have to do it every two
raids and this sucks this game just like oh I have to do this thing.
Oh wait, this is really easy to do, Like okay, just
like someone plays the game wrote the game. So yeah, I enjoy a
game like that. It's a smaller the company. So is there an online
multiplayer mode for it? Yeah? So it's like a whole way do you
go to the common areas there's a bunch of other people there, and there's
trading and stuff, and there's you know, public chat where half trolling,
half people asking questions. But you can party up with your buddies and tackle
content together. And so I have a small group I play with and we've
always played ARPGs, so this is our current one that we're all just kind
of grinding up characters on and figuring out what the best builds are and what
synergizes with each other and things like that. All right, So the follow
up question is do you want to share your gamer tag so that you're listening
for the show can jump on and talk trash. Yeah, I'm pdf backwards
the TEP all right, someone out there has p F places. So I'm
a yeah, you said difficulty. I thought for sure you were going to
bring up Lost Souls or Dark Souls, and so I that kind of game
I if I was good at I would play. I don't think anyone's good
at it the whole, like the the you know those kind of games where
it's like jumping mechanics and stuff. I don't know, I'm that's stuff.
He's not a platformer. That that's it? Yeah, I mean really twitchy
games. Yeah, I'm totally with you. I played Ninja Guide in a
lot in the past, and like that was very there are some parts that
were incredibly challenging. Blessed Podcast. Enough of that, Like some of the
boss fights are legitimately challenging. You know, we just one dude we bought
probably forty times in a before we beat him, but now were really good
at it because we yeah, we post bared them each death. Okay,
you can't stand in the middle if really where games are worth right, Like
it's I think that's a real lesson. You know, you really have to
take root cause analysis and post mortems to your your private life, and you
know in your friend group when something goes wrong, you really need to investigate.
I often joke that, like many things about me are really good for
work and really terrible for personal relationships, but they work out really well in
gaming. So we're causing relationship problem is usually a bad idea. We're causing
why you died to a boss. Apply it where you can fair enough,
all right, Warren, would you bring this week? Yeah, so a
couple weeks ago I mentioned this already, but I'm gonna plug it again.
On Friday, there's a decompiled conference in dressed in Germany, UH which I'm
actually giving a talk on about our journey at authors and adding security. But
there are some interesting talks there, like there's one that I really want to
go to that's about migrating from kubernetties to server lusts, and I think there
are a bunch of other ones that are really interesting. I'm looking forward to
nice, excellent, cool. So my pick is going to be no surprise
to anyone who's been listening to the last few episodes. I am picking platform
Con coming out in June. It's a five day virtual conference about platform engineering,
so check it out totally free, and there's going to be tons of
great talks there and specific to me. At the end of the conference,
I will be doing a live Q and A session with some of the speakers,
so they're finalizing who the speakers are going to be, and then once
that's done, I am going to try and turn this into a Q and
a session that you actually want to listen to. So I'm going to go
out on X and start asking people which speakers do you want me to interview
and what questions do you want me to ask them, so that it becomes
the interview that you want to hear, and yes, I know that by
going out on X and asking you what questions to ask, some of you
are going to ask some of the wrong questions. So I'm just gonna be
honest here with you. I'm not going to ask that, but thank you
for listening to the show, you sick little pervert. And so whenever the
Q and A session comes up, I will ask the questions, some of
them because we know what you're going want to ask, but I will ask
some of them and then you will be able to say, hey, I
heard about this on the X, which takes it full circle because I did
go through that whole setup just to walk all the way through and plug his
zz Top song in my pick for today. So all of you listeners who
are Zezytop fans out there, that was all done for you. Thanks for
listening. Cool, So I think we got an episode here. Awesome.
Yeah, Pete, thanks for joining us. This has been a great talks.
Fun to have you on the show. Yeah, thank you for having
me again. This was great. Yeah, anytime. Warren, thanks again
for joining me as a his co host here. Of course, I love
having you on the show. It's been a lot of fun. Look forward
to seeing you next week. Yeah, and for all you listeners out there,
and we will see you all next week too. Thanks everyone,