Hey, what's going on everybody? I am your host for today of the
Adventures and DevOps podcast, and today is super exciting because if you are a
frequent listener of the show, you know that Jonathan and Jillian both have moved
on to higher callings, and you know, at this point, I've just
got to say that it can't be me. I mean, like, I'm
the only hosts that's here that's stood the test of time. But surely it's
not me. It must be them and not me. But we'll find out,
because today I have a new co host. I have Warren Parade.
You might remember him from episode one forty seven when we were talking about open
source software. He's the CTO of Author and he is now the new co
host joining me on the podcast. Warren, welcome, Oh thank you.
Well. You know, I'm surprised you remember the number, because I for
sure did. I'm not gonna lie. I totally looked it up before the
show started. It was actually interesting that you that that one was on open
source because I actually just came back from foss Dam UH and Brussels, and
there's a lot of parallels there. I felt like it's like the largest open
source conference in the world. Yeah, how was that? That seemed like
it'd be a cool show to go to. You know, I think it
definitely. It definitely cemented a lot of the stereotypes that you can imagine about
both like the open source community. Uh. You know, I go to
a lot of conferences because I speak at them, and so like I try
to pick the ones that I feel like I can have the most impact on
either sharing my experiences are whatever. Have you given my my role? So
he said, I'm like, I'm the ct of AUTHORESS Security Application Specialist really,
so anything that falls in that space. Actually, it was really surprising
when I was there because there wasn't a security track and it's the first year
they even had anything related to APIs. So like my room was like totally
packed with people. But it really goes to show you that either it's still
a relatively new area and sometimes I forget that for sure. Yeah, I
think that's pretty much a rule for our industry as a whole, as it
is new, you know, if you consider it too, even like airplanes
or trains or boats, Like, we're just getting started here, and I
when I talk with a lot of people who are just starting their careers.
You know, they feel like, oh, it's too late, you know,
I've missed the boat, and granted there is a lot to learn,
but it's it's it's definitely not too late. And one of the things I
try to emphasize with that is like when I started my career in the nineties,
you know, we just didn't know anything, and we've learned a lot
since then, and a lot of that is like foundational knowledge for someone entering
the career. So I kind of consider at this stage in my life one
of my roles is to relay that information, you know, and make sure
that people just starting their career don't have to spend thirty years learning the things
that I've learned over thirty years that I'm going to distill it, digest it,
relay it to them, and let them build on top of what I've
learned. Yeah. No, I'm totally with you. I think the numbers
rolled up for me and I'm like, oh, wait, Like I remember
when I started applying for our positions and seeing high numbers of years of expectations
for experience, like even like five or seven. I'm like, I got
to get there at some point, probably not. And now, like I'm
way past that number, and I'm like, oh, like I'm supposed to
know things. But I mean, I think it's a good reminder though,
too, because there's this bias, even especially in our industry, where we
think that I think it's experts everywhere believe that the common person knows way more
about the subject, and even within software, there's so many different disciplines that
I'm often talking to people about things that I think a majority of people know
already and they're just like, wait, don't I don't. I have no
idea what you're talking about, Like, all right, that's I totally can't.
Maybe it's that surprising. Yeah, No, I think that's been a
shift in the decades that I've been doing this too. Like whenever I first
started, if you said I don't know about that technology or I don't know
how to do that, it was it was a bad thing to say,
and it could be detrimental to say you were expected to know everything. And
we've kind of broken that down over the years where it's okay to say I
don't know, and and it has to be the vast number of things that
we covered, like you're not going to know everything. Yeah, I think
coming, like actually acknowledging that imposter syndrome is a thing and that it's okay
to have has has been a huge shift that's only just happened recently, like
even in the history of software engineering. I mean, I think it's permeated
to other knowledge work related paths that they're they're really as you point out,
there's just way too many things that you could ever know. Right, I've
too good and stuck because like I keep want to get it into more into
AI and I feel like, you know, that's that's its own can of
worms. But every time I look, I'm like, Okay, but if
I learned something there now, it will just totally be different in five minutes.
So maybe I should I should hold off and just not try to get
down that back, right, let it simmer down for a few years,
then check it out. Yeah. Right. And that's the other thing too,
Like some of these things like if you invest time and effort into them
and then a couple of years from now, it's not even a thing,
Like dang, there's three years of my life I'm never getting back. Well,
you know, it totally pays off because I feel like my experience in
software engineering is filled with the situations that I definitely regret, but they all
turn into great stories and experiences. I feel like whereas and that you can
pull forward in some way. I mean I had to be careful about that,
like not getting not staying somewhere where I feel like I've learned everything there
is to learn and there's nowhere for me to go and it's not necessarily get
promoted or move up, just like am I using technology or skills that I'm
now trying to focus on or grow? For sure? Yeah? So on
that note, how did you actually get started in this field? You know,
I don't know if I should go forward or backwards here, so maybe
maybe at least I'll put the goalpost. So I'm the CTO of a company
that does authentication and authorization as an API. What does that even mean?
I feel like those words are complicated that aren't immediately straightforward, so like user
identity and access control specifically for businesses with complicated user roles and resources, multi
tenantcy anything in that domain. So it's been a long journey for me to
get there. I moved a lot around in a lot of different industries and
locations, so I actually didn't even go to school for software engineering. I
went for my undergraduate degree in engineering is electrical and computer and that means I
wanted to build Spice satellites, but that actually didn't work. Yeah, I
know, that was my you know, it was that or go work for
NASA or the equivalent. And interestingly enough, I you know, sort of
got an experience of doing that later. But when I graduated, I couldn't
find a good job. I'm by good, I mean like the year before
I graduated there was a huge crunch sort of similar to what's happening now in
technology, so I wasn't I didn't feel very optimistic about my opportunities when leaving,
and so like I didn't get that many and I ended up in healthcare.
It in the middle of nowhere in the United States. And let's just
say my hobby programming practices of since I was like fourteen years old, I
think that's about when I started. We're more sophisticated than what this company had
going as far as like software development processes and source control, which they didn't
have at all. So that was, you know, quite an initial shock
for me, and I really didn't want to stay there very long. I
think what was interesting about that job too, is that I didn't have experience
in hard software engineering skills, but I did a lot of classes that I
think are still valuable for me today. So like, I don't care if
someone went to a university when I'm hiring them or not, but there is
some background that's super useful, like algorithms and structures and understanding what that is,
like if I say hash map or dictionary that sort of comes with some
expectations. I don't care if you know how virtual page file system works,
because I still can't figure that out for the life of me. Yeah.
So this again actually wasn't even software. I was doing technical services, which
was pretty much doing custom software development for the customers of the company, but
not building the product up. And that was actually really working with customers directly,
our users to help them understand how the technical product worked. Super complicated,
lots of configurability, so much so that customers could put in their own
code into the system, which is sort of ridiculous that you would go to
production and make changes there directly in the operating system level, and like that
scared me that anyone would do that, but that that was the thing.
So that also wasn't wasn't pure software, and that was great because I feel
like I got experience with Okay, what we're doing has an impact to a
user somewhere, like there has to be a reason for doing this. Yeah.
I think healthcare is one of those interesting places to work in technology because
you have like all of these different constraints. You know, you have just
like the whole world of medicine is huge, but then you have it's implemented
differently in different regions, and then you have like the legal aspect of it
with hipper requirements and stuff. One of the jobs I had was it was
an This was before the term SRE was around, but it was kind of
an SR type role where we worked directly with the staff and trauma centers and
so whenever they would call us and say, hey, this thing's broken,
right, they had somebody in the emergency room on the table that they couldn't
give medical care to because our stuff was broken. And so you were talking
with you know, doctors and nurses, and there were there were many occasions
where you're just tell them, hey, look, just put me on speakerphone,
go do what you gotta do, and I'll shout if I need anything,
and I wish I wish I sort of had that experience, because that
seems like one of those that you know, falls in the category of I'm
glad I did it in my past, but it's not my day job anymore,
right. I Actually I actually worked in claims and like billing primarily,
so I got to deal with all the problems of configuration in this system and
also how to deal with those contracts with third party integrations to insurance companies or
clearinghouses. Yeah, insurance companies are just a joy to work with at every
level, right. I mean, this is like after a standards body came
together and said we need things like hl SAT then to perfectly identify programmatically how
to communicate, and then the hospital organizations would go out and sign contracts with
these insurance companies that violate the data that's allowed to be put in the programmatic
communication and the interface and their APIs or EEDI interfaces depending on where you're from,
and like that was really a struggle to communicate where like, hey,
you know, our software doesn't do this because it's against the spec like this
field, it takes a Booleian true or false, and you can't put a
three in there, and part of it was to protect the patients, like
what information should be allowed to be sent to the insurance company that wasn't relevant
for paying out whatever diagnostic or surgery was assigned in that claim. So I
did remember one time, and this is where I'm glad, I'm not I
never was in the sor repath directly that one of the data centers that one
of the hospitals that I was supporting had an incident and their like main systems
were all offline like totally, which you know, same to do your case,
and what they had us do was remote into the systems, not using
SSH that would be cool, but using some other technologies that I don't want
to name, and like three different levels, like we had to first we
got the beeper go off, you had to get on call giant phone room
where then you got a sign credentials in order of access. Then you could
get in and then SSH to another system and then to another system to even
get into production to run a script basically to verify that the database was integrity
was still you know, okay. I didn't like that. I think one
of the big takeaways for me from working at that job was just putting things
into perspective, you know, especially since I still deal with production systems a
lot in incident management. You know, after that, my approach is,
oh, we've got an outage, Okay, how many people are going to
die? Zero? Okay, Well, let's just simmer down a notch here
then, and let's just make conscious focused decisions here because everyone's going to be
okay. Yeah, No, I totally get that. I think this is
one of the most frequent advice that I end up giving inexperienced like new engineers
the field is really just take a look at that risk profile here of what's
going on, Like I need help immediately. I'm like, do you?
I mean, your production deployment isn't working, but the site is working at
this moment, and it also doesn't serve critical usage, and also you don't
have an ongoing marketing campaign and maybe you have five users, right. I
think I saw an XKCD on this. I don't remember if it's that or
someone else, and there was like, is this is like posts like my
SQL or postcress or something. Is my database configuration will work for having one
hundred million concurrent users and the responses how many current users do you have oh
we have five? Oh yeah then yes, yeah, right, I mean,
because there's a lot of truth to that. It's just like you you
don't need that scale at all. I was actually talking to someone today and
they were doing they were almost exact same question, but they're like, we
don't want to use an identity provider that's proven hardened or whatever. And I'm
like that's okay. They're like, oh yeah, it doesn't scale for us.
And I'm like, how many use it? Like ten thousand users?
And I'm like, okay, you know that's just like such a small number,
right, not even here. I'll let you borrow my raspberry pie that
work, I know. Right. The thing that scared me is the difference
between it being like a commercial application for personal use and when that goes to
a business. Because as soon as you bring in business as if people are
asking these questions, I start to get concerned, like, Okay, what
is the data privacy that you're thinking about? You know, what is your
data redundancy or disaster recovery situation? Because if you're worried about spending ten bucks
a month on running your app, I feel like there's a promise here to
your customers. That's more than just that could go down and having a backup
strategy just it's like number one for me. If you're in the business space
over you know, who cares if you lose some music files from your Spotify
account? Right? And I think that ties into something that is very specific
to devlops but also applies to other areas of technology too, is like it's
not always just about the technology, Like you have to understand the business as
well and understand like what your promise to the customers is and how it impacts
their life and even like in in some cases, you know, you have
to understand the finance aspect of it too, because we've build all of this
infrastructure and now the finance team doesn't really know what to ask. But if
you go to them and say, hey, some of this infrastructure is op
X and some of it is CAPEX, and if you work with us categorize
it correctly, it can result in a huge amount of tax savings for the
company. And so you get into all of this non technical aspects of a
technical business. I love that. I love that you brought that up because
I think that understanding even the difference between those two is such a foreign concept
too. And sure, in general, I think we get lots of customers
that are like, oh, it costs ten thousand bucks a month for your
service. That's too much. I'm like, let me explain to you,
you know, the savings here. Also, you probably have capital expenditures that
you're currently paying in and saving those resources that are depreciating over time, versus
paying a SaaS provider to you know, deliver some of that. And if
I look at their use and it's like peanuts, like a couple bucks a
month, it's like you're literally wasting more time trying to make a decision on
which statas provider to use for something than you are to pick a random one
and paying the cost to transition later. It's really sad, honestly how often
I have that conversation. Yeah, you have to you have to know.
And I think this just comes from experience of knowing when to dig into the
weeds and when to take a step back and see are we even in the
right part of the forest here. Yeah, No, it's interesting because actually
you know to Can you continue on my career past story. I worked at
a large global manufacturing company that wrote their own software and they actually did chargebacks
really well and expose the cloud costs eventually to the individual team so that they
could take ownership over how much they were spending on some things. And you
know, I think that's a great a great first level, like giving that
exposure rather than locking it behind another team or even a different part of the
organization like one quote unquote finance, because you know, that was great to
see, Oh yeah, you know, if we' spin up this infrastructure,
it's going to cost us two thousand dollars more a month. But then it's
great to also hear that other teams are like, well, well, whoops,
we just spent two hundred thousand on the others sasas provider. I think
I think I read it it was like in the ten day of like Coinbase
or something that they had spend sixty five million dollars last year or two years
ago on data dog or something like that. That's actually really easy to do
with data Dog. I mean that's always been my concern. I mean,
they're not the product for us, given where like the sort of data that
we want to collect and the sensitivity of that data. But it's also really
amazing that there aren't any product that really fit the space for us. I
wish someone would come to us and be like, you know for metric usage,
you know, here's a perfect thing, but you know, we still
haven't found that. Actually, So we actually because in my full time job,
I'm a staff engineer and the DevOps team for Polygon, and we actually
just rolled out a product called cloud zero that has been super cool. It
does, it collects the it connects to whatever billing APIs for different providers that
you use, pulls that in and then gives you insights on it, and
it's really really insightful. Like the very first day, it's like, hey,
here's a bunch of net gateways. You're using an AWS, I don't
have any traffic. You can save you know, five thousand dollars a month
just by turning these off or and it's like daily emails. It's like,
hey, over the last twenty four hours, the spending on this GCP product
or GCP project went up one hundred and twenty five percent. You're like,
wow, that's a jump. And so that's that's been really cool just to
manage those those kinds of things, and it's been very I'm not trying to
plug Cloud zero or turns into or anything. We just had good experience with
it because they've just they've brought up some stuff that we would have not seen,
you know, very easily, and the integration was that was super easy.
I mean, I think I think that's the truth is like those sort
of tools have like significantly evolved recently. I think at the time, which
was almost eight years ago now, we were using like I think it's called
cloud and area or something like that, and it was it was not it
was not particularly good at the time, but yeah, being able to see
those trends in not an amateurized cost way was was helpful and actually driving alerts
and whatnot, which at the time AWS didn't really support. AWS doesn't really
have motivation to support it. Well, you know, I think that's sort
of the sad story of SaaS products is I mean really everything in general,
that once you find your core customer, if they're not going to pay marginally
more money for your same product, even if you make it better in some
way, and that's just like the law, like I pretend that doesn't exist,
Like I want to happily live the rest of my life believing that if
you make it better, people will appreciate that there's a brand impact for us.
Like a very common conversation I have with my CEO is about how well
the quality of whatever we're doing before we start going down another path, because
I think it does really reflect well users bringing in more customers, et cetera,
rather than secretly complaining about it behind closed doors. Yeah, I would
agree with that because I you know, you obviously, with a good internet
based marketing campaig, you can reach a huge audience. But there's something to
be said for just delivering a great product and having satisfied customers, and then
when they go to their next job, they're like, Hey, I used
this before and really liked it, And I think there's it may not have
as big of an impact as a funny marketing campaign or whatever, but I
just think there's something professional about that. Well, it's interesting you say that,
because I'm not convinced that in the business space that even viral marketing campaigns
have any real value there for sure, if we're talking about personal products,
like right, yeah, definitely, But there's a whole There was a whole
initiative by the search providers AD marketplaces quite a few years ago that tried to
actually figure out whether or not the traffic that was being sent to your website
or app actually converted for you. Right, obviously that's a big importance.
But if it didn't convert, you almost got dinged because you basically lied about
your advertisement, right, you know, traffic when to you user didn't get
what they wanted. So I think it's I think, you know, originally
the original version of this is like sex cells, and I think the truth
is, well, no, sex just makes people click the link. But
after that then they then they drop off because you know, it's not actually
what they wanted. And so at least for our content that we put in,
like our knowledge base or our blog, it's like it's got to be
super relevant because while there's lots of things that we can say that are really
interesting from a technology engineering blog sort of way, those don't really help get
customers in the door brand. I mean, Brandon Winness is one thing,
right, you know, you must your brand all you want, but that's
not going to really result in direct sales, but it will result in this
concept known as oh did I forget the name. It's like positive reinforcement.
Yeah, yeah, sort of along the lines of like someone has to hear
your name five times before they ever recognize your company name. Oh, that's
a high number. I didn't want to hear that. Don't don't quote me
on that number. There's a rule that marketing teams use that people have to
see your ad or whatever a certain number of times before they start to recognize
your company name. But I don't remember if the number is actually five or
what the value is. I mean, I'm sure the unfortunate truth is it's
more than you'd like it to pay, right, right, that's the takeaway
there for sure. So looking back over your career, what are some of
the things that you some of the career development decisions you made that when you
look back, you're like, ah, that was just pure brilliance. Well,
I don't know I have an answer to that's question, or like,
Okay, that wasn't completely dumb. Okay, let's set the bar a little
bit lower there. Yeah. I think I made a couple of realizations while
I was working on different things. So one of them was like, the
first DevOps related thing I had ever done, which was not called DevOps at
the time, was helping the organization I was in to replace the need of
a different organization, which was called Release Engineering, and the third organization that
wanted to help us do that told us the solution was Puppet. And if
you don't know what Puppet is, it's a convention based Ruby set proprietary packages
really, which is just the most ridiculous thing that you install agents on all
of your machines to deploy and upgrade the software. The thing is is that
you pretty much to do the equivalent of writing your own terrorform provider for your
own software in order to deploy it. There wasn't anything common, really,
and what was common didn't really work out of the box. And like you
will, I was the only one left on this project by the end,
from three teams that hadn't been swapped out in some way, like twelve people
left it over the time I was trying to get this done. But the
point that I bring this up is that repeatedly it was said, usually by
me, oh, if we wanted to do it that way, maybe we
shouldn't be using Puppet. And at the time I was literally I was so
under leveled. I was literally the lowest level that an engineer could have at
this company at the time, which its own sort of travesty, which should
teach you something that your ability doesn't really mean anything when it comes to the
level that you put decide if I had been a higher level, I would
have really after the fact that saying those words to myself or even out loud,
is an indication that you should do something about it. And I mean,
in this case, if I left the company, But that's something I
created. This rule. I mean, I call it the rule of three
really, but like if I hear the same problem in my vicinity three times,
then I know that it's on me to solve it. That no one
else is going to come up and just magically resolve this issue. And I
mean it's not like a hard and fast rule or anything like that, but
realistically, no one else may be privy to it as much. Right,
Like I may have three different sources that are all coming to me or talking
about it in my vicinity. So I usually watch out for those things because
they help a lot. So like this was one I should have realized,
you know, after the tenth time I said, maybe we shouldn't be us
puppet like, we should have picked a different one. The sad truth is
I think that company still has my like the system now I had helped build
there to integrate with Puppet being run by like out of the three organizations that
were set up to manage this infrastructure. Basically there's like a team of two
or three people now that are handling like all three components, which is just
ridiculous really when you think about it, because this is the wrong technology no
matter how you look at it. And they don't have the skills to build
to even manage the rappers or abstractions that were built to manage this, and
they don't have the experience or understanding or knowledge to do it either, So
it's really really unfair. But the biggest lesson there, I think is don't
use Windows. Yeah, I mean, so this is a manufacturing company.
At the time, there was four manufacturing plants all over the world, and
they had a monolith. There was something like one hundred and twenty Windows Services,
which amounts to like it was like eleven hundred c sharp projects that were
being built and managed and deployed in a monolithic fashion. So like they had
a real need for strategy strategic change of how they were doing deployments, which
and I think this is a good story if you wanted to make a new
service, which like this is still in the days of say, like the
term micro service didn't even exist yet. You had to contact this second organization
and say, I want a new micro service. Except you could only do
that. You needed a three week lead time for them to create the infrastructure
for your service. And given when you had to do that and the lead
time it would take for you to actually release your new thing, you could
only do it once every five weeks, and it was one day of the
week that you could do this, so literally every once every five weeks.
I'm like, oh, there's that. Like I couldn't believe this when I
figured it out. It was like they have a process, and we have
a process and together that eliminates every possible day with it a one plus period
except for this one specific day because we had code freeze and you had and
you had to introduce the service and then have a handoff and then other nonsense.
I mean, that's why we were going to puppet in the first place,
which was supposed to be the revolution never really made it to be right,
it was going to automate everything for you. Oh yeah, for sure.
The claim to fame there, like the interface I provided for other engineers,
which I thought was absolutely fantastic. Was a single file that could merge
in lots of different version changes from individual parts of the monoliths changing at any
time, And so each line of the file was a because the new organization
said that ivy was the way to go, which is a Java resolution sort
of package. I mean, it's not even a package manager. It's a
JAR dependency manager, which is a sort of own atrocious thing. This was
a HTML file well that had service and version as h al attributes. And
to make it worse, we get every other line was a comment that said,
do not remove this comment because because if you're not aware of this when
you make a change and get, if there is no change on the line
above or below, then you can merge very easily. But if there is
a change on the preceding or following line, then there's going to be a
merged complex so unnecessary line. So yeah, I mean there was a comment
between each one of these that said, you know, do not remove this.
But I thought this was a great interface, and for sure, you
know, I would never never do it that way ever again, but it
was the right thing at the time, for sure. Yeah, And that's
one of those things where, like you you know, it's not the permanent
solution, but it's a foundational step in building a permanent solution. I mean,
I wish someone had said those wise words to me at that moment,
because it is certainly the belief that I had that this would be the only
thing that was ever necessary for the rest of the life of this company.
I mean, I guess the fact is still running is some truth in there,
no, But I think that is a good point worth elaborating. Is
like everything we do is, you know, without trying to sound like philosophical,
everything we do is temporary, but just meaning that technology continues to grow
every day, and so whatever we do today is really just a stepping stone
to whatever is beyond that. And if we just like do like a really
high level history lesson you know, we had servers in closets at the office,
and then we had data centers or we would go wrap our servers,
and then we figured out how to run virtual machines on the servers, and
then we got you know, cloud providers like AWS where you could run virtual
machines, and then containers and and so like all of those were just building
on top of the lessons learned from the previous decisions. I mean, I
think that's an interesting point. I think it was like, not too long
ago, I was reading a paper that suggested the persistent media storage that we
had over time was becoming more ephemeral, and like the replacement speed of things
like the time between SETE cassettes and I don't know what else CDs and then
VHS tapes and DVDs to Blu ray to you know, USB sticks slash drive
was getting shorter over time. Ah, And it's interesting like our is our
technology that we're building now more ephemeral, transient than what we had in the
past. And I think what really missed out in this article was thinking about
the lack of stability of volatility of the previous generation. I mean, the
I'm gonna get this wrong. The US launched deep space probe that contains the
gold DVDs with is this Voyager? I know, I don't want to I
could believe it's called Voyager, you know, I am not. I am
not the exit export name recognition here. But yeah, right, like we
put it on gold discs, right, would we still like, I mean,
I have no idea is that better? Than you know, some other
less ephemeral flash system that we have now, Like, would we still use
that same strategy? I mean probably not. I'm sure someone dream up it's
like some sort of antimony composite or otherwise, but it's still like is it
metal with like physical embeddings in it? I don't know, So I thought
this was interesting. I would just product tape a USB stick onto the satellite.
I mean, I think the realistic is like you just put fifty USB
sticks in the satellite, like its all as like one of them needs to
survive. But then you like you figure out like Hoffman coding and error correction
and so like it's okay if like half of each one of them survives because
you can put the data back together. Just load them all with a multi
petabyte ZIP file bomb just for laughs. Some aliens find it and it just
blows up their computers. Yeah. Well, I mean it's like running commands
is always weird, I guess, right, because you have to know what
the language is in order to even execute that thing. Otherwise it's just you
know, nonsensical. But I think that that's really that's really the rick roll,
right, like just just put literal nonsense on the disc and set that
up, you know, a society. Yeah, I remember this an outage
I was on many years ago, and it was storage related where a vast
majority of our production systems just had no data. And so we're on this
call troubleshooting it for hours, and I get the storage team on the call,
you know, and they're like, yeah, well we had some disc
failures and so we swapped out the disc and like, okay, well that
seems reasonable, and how's the rebuilding going. Yeah, the rebuilding seems to
be stuck. Okay, And we talked this out over a course of a
couple of hours and found out that nearly every day they would come in and
a disc would fail, and so they would swap out the disc. But
then after a week or so, in the twenty four hours between the failures,
that wasn't enough time to rebuild the disc. But then they would come
in and swap out another disc. So after a week or so they'd swapped
out enough discs, we're in this storage array where none of the dis had
enough information to do it, so they lost everything. Wow. I mean
I wow, that's you know, and then you go back to okay,
well, let's let's restore from tape. When's the last time you've verified your
tape backups? Yeah, and you get crickets like all right, well we're
going. I mean it's like this is like one on one running fire drills
on your that's the recovery process. Like, yeah, forget verification, Like
did you even like you literally need to load the date in bload the data
and be like yeah, we're switching over to it for real. Otherwise you
might as well not happen. Yeah, And I think that's in my experience,
that's been a hard thing to prioritize. You know, you just essentially
have to say, look, this is what we're doing. You may not
like it, but there will be a day in our future where you are
happy that we spent this time because you learned so much in trying to do
that and you recognize all of the things that you thought you had covered that
you don't have covered. Yeah, no, for sure. I mean I
think that's one of the areas that's really undersold by the cloud. Like so
often I used to do before my current thing, I was doing some fractional
engineering CTO advising consulting, and it's too expensive for us to move to the
cloud because we looked at our current infrastructure that was on some on prem data
center or delegated and we just calculated the exact number that it would be if
we tried to run those exact things in AWS. And the ridiculous part is
like they're not accounting for the actual value add of not having your tape backups,
you know, right array failure issues, because that is like if you
care about that, and you probably should actually care about that, it's non
trivial to have that working correctly for sure. Yeah, that was whenever I
got out of high school, I joined the Navy and became a nuclear engineer.
And one of the things that they did there was they just really drilled
home the whole disaster recovery thing. You know, you would be operating the
power plant at two am, and you know, some master chief who was
up and couldn't sleep, he'd go walk down into the power plant and just
turned something off just to watch everyone respond, you know, and it really
reinforced the disaster recovery drills so that whenever something did happen unintentionally, just those
skills we were just muscle memory at that point. Yeah, that really Also
it seems like an exercise and root cause analysis, right, because which is
not practiced enough realistically. I mean, there's a lot of I don't know.
I think there's a lot of companies out there that talk about post mortems
and one on, but having been in quite a few in my day,
I don't usually see them run in a way that actually has meaningful outcome there,
like actually resulting in better processes or practices in general. I think the
number one thing that comes out as well, we should have better tests,
right, we should have more tests. That's the solution, more tests,
And that's usually what happened, more tests, usually by adding some sort of
rule to your CICD platform that ensured that your code coverage was at above eighty
percent, which is always just great whenever you delete code and it goes down
because of course you deleted code with tests on it, because that's usually the
code that's tested the most worthless not necessary. So and I'm just like,
I don't like I have stopped working contributing to certain open source projects when that
happens. I'm just like, I don't know what to do here, Like
I don't know enough about this service or SDK library to add tests somewhere else
because obviously they're not there for a reason. It was deemed too difficult to
add them. And now you're asking someone who doesn't know what's going on here
to fix a bug by deleting code and you can't. You can't do it,
yeah, for sure. And every test carries an equal amount of technical
debt with it because the test has to be it has to be actively maintained
and evaluate like is this test still testing the right thing? And when it
fails, did it fail for the right reasons? Yeah? Yeah, no,
for sure, for sure. And I think there's a whole, this
whole h extra hours of uh content here that may not be directly related to
DevOps that you go on with. I mean I worked with a lot of
originally QA but eventually QUI and just selfware engineering teams trying to turn it all
into the DevOps mindset, because that was a new term, I guess during
my tenure at different companies, and not one that was always so practiced readily,
for sure. So what were some of the things in your career If
for someone who's just getting started or part way through their career, what were
some of the things that you did that didn't seem to have any value or
were perhaps negative to your career, Oh for sure. The one, My
favorite one is telling other people how their solution won't work. Yeah, that's
rare, my favorite. I thought, you know, I was doing the
right thing early on. Hey there's this thing that doesn't work. I mean
I not so long ago. I actually did a presentation about how the human
psychology doesn't respond togative to terrence or even really any sort of constructive feedback very
effectively. I mean, there's a whole psychological safety aspect and desire to change
of which most people you know don't fit there. Most people care more about
being told that they're right or they're doing a good job, which says a
lot for positive reinforcement that you can get people to do the right thing,
to be motivated even if you congratulate them, if you tell them how they're
doing good job, give them rewards. Just don't put this the carrot in
front of them too often or explicitly, because it will. It can derail
where people get personal satisfaction and motivation from. But it helps way more than
anything else. Like, I think there's this common misconception that's widely acknowledged that
performance improvement plans work in some way. I mean, it's a great example,
right, you know, hey, we're going to fire you if you
don't get your act to get other never really fix anyone's career specistically, it
did help, right, you know, those were the words that someone needed
to hear. Yeah, yeah, but you know that was a big one.
I think another one for me, and I was lucky that I had
a couple of mentors that really shared this with me specifically, is that it
doesn't matter how good of a job you do. Realistically, no one cares.
They care about the outcomes that are important for them, and so figuring
out what actually is important for them. And it's not like some them,
which is some vacuous, morphous blob somewhere. It's you know, put points
to your finger at whoever is responsible for giving you a promotion or who I
mean, realistically, whoever cares about your future development, which you should know
who that is and how they're going to help you get to the next level,
and ask what's important for them today, right now in the future.
And that's the answer, Like everything else, totally irrelevant. I mean,
do it makes you happy? Right? Like you shouldn't go to your job
and be unhappy, so you know, find a way to work that in,
but don't for one moment believe that doing a good job. And this
was like some lie that I had told myself, and I remember thinking it
so many times. I know there are these things called politics. I'm not
going to get involved. All I care about like I will get promoted by
doing the best work I possibly can. And I just stop using the word
politics because it doesn't like it doesn't really land. It didn't land for me,
It doesn't I don't think it lands for other people. Really, just
think about what you want and then draw the steps to how to make that
happen, which are usually convinced someone else that you did a good job.
And that just involves a conversation for sure. Yeah. Yeah, there was
a quote and I can't remember it exactly, but someone said, just because
you choose not to play in politics doesn't mean it's not a political game.
H Yeah. I mean I think I think for me, there was the
like this way, you still twist it for yourself, where like, okay,
there is still a game, but I'm going to try to get around
it in some way. And I think I think sometimes assigning those labels really
hurts. Like another one I think really wasn't great for me in my career.
It was people calling I mean, I don't know if you can believe
this, but apparently early in my career I lacked what are called soft skills,
you know. I think that goes along with telling people that they're wrong,
you know their solution won't work, and then also be like I'm very
to the point and very blatant about it. So, you know, a
couple of those things together, and sometimes people's feelings got hurt, which of
course was never my intention, which is totally different than all those people that
do it with this sort of negative or malicious connotation to it, like it
wasn't about you, it was about like, literally, this solution doesn't work.
I think for me, what really helped to stop calling them soft skills,
stop calling it Paul politics, and focus on what it really means,
which is you can work way more collaboratively with other people if they like working
with you, right, And when you phrase it like that, I feel
like it much more helps people who maybe it were in my position or similar
positions who maybe don't go along by default or different cultures engage with each other
to take a step forward than it would if your manager just says, hey,
you know you're missing this manager buzzword that doesn't necessarily connect. Yeah,
for sure. I think you know you mentioned having mentors. I think that's
been really I've really taken a completely different view of that over the last few
years, because you know, like I grew up in the middle of Texas
in the seventies and eighties and can honestly say, I don't know that I
ever heard the word mentor until I was in my twenty and probably never used
it in a sentence till I was in my thirties. But now you know,
it's a very common word. And I used to think, well,
I guess I didn't have any mentors, But then I the more I thought
about it, the more I realized that I did. I just didn't know
what the word was for them, and it wasn't a negotiated arrangement. And
I think that's still true. You know, like you can have mentors without
having a formal conversation. I have this contract here, would you like to
be my mentor? It's very low offer and low commitments, easy out.
You know, it's not that type of thing. Like a mentor can be
anyone who who says anything to you, whether it's helpful or hurtful, and
it doesn't have to be like on your terms. And what I mean by
that is like, you know, people have their way of communicating, you
know, Like I said, grow up in Texas and then I went to
the Navy. And this was back in a time whenever being in the Navy,
your performance was judged on how many bar fights you got into without going
to jail. So it was, you know, it was a different environment.
So whenever, like your master chief told you, hey, you ever
pulled that shit again, I'm gonna kick your ass. You know, that
was a form of mentoring. You know, it wasn't it wasn't a nurturing
relationship, but it was a mentoring relationship. And so I think that was
one of the things I've learned over the years is that mentoring comes in all
forms and it's not a predefined, negotiated relationship. Yeah, no, that's
I think it's even more effective when it's not. Like I hear of companies
trying to start some sort of mentorship program, and it's usually like my company
is starting a mentorship program? How do I make this be effective? And
I'm like, well, why are they starting it? Like you're getting in
on the wrong foot almost, And you say that, and I'm thinking back
to my own career and I feel like every job I was in, there
was always someone who I gravitated towards and because maybe they were more experienced,
but it was never like, hey, can you be my mentor? It
was me asking them questions when I when something came up, trying to get
additional information or understand how they did things. I think that that definitely helped
makes like going to the right people potentially that I think I would definitely classify
as as mentorship. So you asked about things to do early on in your
career, you know, find people that can teach you something, and they
will absolutely teach you those things. I hate to quote Steve Jobs, but
he has an example where he called out and asked an executive for feedback to
help them and they're like, yeah, sure, I how can I do
that? I think it really is that people will I think Adam Grant says
the majority of people are matters that most people will reciprocate un asked. Really,
if they think there's a good engagement for them or could be a good
relationship or whatever. It doesn't have to be like something static like put a
label on it, like it's not necessarily required. Yeah for sure, for
sure. All right, so there you go, Welcome to the podcast.
Yeah, I mean, you know, even here I I I before this,
I was asking you, you know, personal questions about how you how
you run this, you know, as a good as a good example,
you know, come ready, how do I do this effectively? Just because
you know, I want to do the best job I can and I'm not
the super expert in the area. Obviously I pull skills from what I have
in the past, but looking for people that have even just slightly more experience
than I do. Yeah for sure. So what do you think should we
move on the picks? Yeah, I'm trying to. I'm trying to think
if I have any you know, most most important thing that I you know,
ever ever wanted to share here. And I feel like I have so
many things, but probably not super relevant at this point. So yeah,
let's let's do the That's the great thing about picks is it doesn't have to
be relevant at all. Just be like, oh man, this is so
cool, you know. And so we were talking about movies before we started
recording, and during that conversation I learned that, Okay, you obviously are
a movie guy. You have your own IMDb account where you save and rank
your movie, so I know you've got something there. He wants he wants
to hear my favorite movie of all times, all time favorites, you know.
And that's the thing where I'm looking at this list, and I have
twenty two movies rated at ten stars, which is the maximum, and so
I feel like even picking one of these is just such such a challenge.
Realistically, It's like picking a favorite kid. You can do it as long
as the other kids as you have, don't hear you. I mean,
you know, and I like them. I think they liked them all for
different reasons. That's the interesting thing. Like I have ones that are film
noir, which I as a genre I really like, But then I have
ones that just for whatever reason I liked and had some emotional component to it.
For instance, you know what this one I think I saw recently,
So I'll say this. I think you know, it's not my top one,
but you know, this is the one I'm going to pull out this
contact. It's if you haven't seen it, it's with Jodie Foster and I
think they're the sad part. It's about about making contact with aliens. Maybe
she has sort of a moment in building a machine. It's about the SETI
project where they discover, of course, signal from outer space and t USB
drives to the satellite to contact. No, actually they the communication around our
space was a machine in ructions for building a machine which they could build.
And of course, like you know, she spent her whole life doing this,
you know, through adversity, and she doesn't get to be the one
to sit in the chair to go contact the aliens. Initially, I feel
like the thing that really hit me is there is it's such sort of like
a I don't even know it was intentional, but such a parallel to some
of the struggles of diversity in the tech industry, because I feel like she
was the perfect pick in every single one of these situations and kept getting gone
over for whatever reason. And like the Catharsis, at the end of the
movie, she actually does get to meet the aliens and then no one believes
her, which is just like, I mean, I like space movies,
and I like that one had me just I felt so bad, like hearing
that, like just seeing that story, which of course is not real as
far as I know that, Like, I'm sure there are people out there
who said that, you know, they've they've met aliens, right, Yeah,
there's been a few, but none of the interviews I've seen have the
credibility of Jodie Foster just say yeah, what's your what's what's your peck?
So my pick for the week is going to be for the platform Con conference.
And we had the guy who created it, Luca. He was on
this show. I don't remember that episode number, but he was on last
year talking about platform engineering. But platform Con is a virtual conference that's five
days long, coming up here in a few months. But it's it's really
well done. And the reason I'm picking it is because at the end of
the conference there's going to be a live Q and a session where I'm going
to interview some of the keynote speakers at the end of the conference. So
I'm looking forward to that, looking forward to to talk with those people and
seeing if I can uncover some things that didn't get brought up in the conference,
and that seems pretty awesome. Yeah, Yeah, it's a cool it's
a cool conference, and the best part is it's virtual, so you can
just throw it on the background while you're doing whatever it is you're doing and
you don't have to try to argue with your boss to get you know,
to get approval to go to it. But check it out at platform con
dot com. And you've got a few months to get into to get signed
up for that, so plenty of time I start. I'm going to put
it on my list because I'm sure there's something there I'm interested in. Yeah,
they cover a pretty good range of topics, you know, not just
obviously not just DevOps related, but platform related and platform really is I'm like
in the word platform because I think it's more representative of all of the things
that we cover versus DevOps. Yeah, for sure, So that's going to
be my pick. So for anyone who's listening to the show and wants to
reach out to you about it, either about authors or just career or just
to say hey, when can you get that will do fired so that we
can go on with a decent podcast? How can they contact you? I
think for sure? I accept LinkedIn connections if you start a conversation, I
will I will I kick people out of my connections if they don't talk to
me on LinkedIn. I think knowing who your network is that is valuable to
you is valuable, and it's difficult to go through your rolodex if you don't
know who you can actually talk to. But I think fundamentally the best place
is on discord. I'm in a bunch of different Discord communities. I think
we can get the link in my host profile or something like that for sure.
Yeah, And that's definitely going to be the best way. And I
happily talk about whatever you want to talk about. Their cool right on.
My wife spends a lot of time on Facebook. Not a lot of time,
but she's on Facebook. But she has a similar rule where she only
allows fifty people as her Facebook friends. So if someone else wants to be
her friend on Facebook and their number fifty one, she's like, I got
to kick someone else out before I can be friends with you because fifty is
the limit. So there's two really good points about that. First one,
I'm doing the exact same thing on Facebook. I mean, if we think
about Dunbar's number, which is not a fixed number for the record, it's
just not possible to have that many close connections, so realistically, you should
keep people in the loop that are relevant for you. It's way more effective
and you'll be happier when you are focusing on your connections that matter. On
LinkedIn, it's slightly different. I don't put a fixed number there because there
is a advertisement effect for my brand. It's sort of like Twitter. The
more people that hear me, the better. But I'm not on Facebook and
I almost never use it other than I you know, I do need to
talk to one of these really close people, and I don't really use Facebook
at all besides that. Yeah, agreed. Cool, All right, Well,
Warren, I'm excited to have you here and I'm looking forward to many
many episodes with you, and I think this is going to be fun.
Yeah, me too, I'm I'm excited right on. Cool. Well,
thanks for listening to everyone, and we will see y'all next week with another
episode. And I'll stop the podcast as soon as I find my mouse there
we go, all right, see everyone. Bye.