Hey, what's going on everybody? This is another episode of Adventures and DevOps
and I'm your host for today. Will Button joining me today, I have
Corey O'Daniel, CEO and co founder of mass Driver, but also from Let's
See. You're a principal software architect at the Real Real Cloud Solution, architects
at Container Heroes prior to that, staff engineer at Click Trips, and most
importantly, taco afficionado. So two things jumped on my mind right away.
Number one, I need to know what your definition of the perfect taco is.
But also I want to talk about your background because it looks very similar
to mine in terms of the amount of time you spend at each company,
and I think that's an important thing to share with everyone, especially people who
are just starting their careers. But we'll get to that meanwhile, Corey,
thank you for joining me on the show today. Yeah, thanks for having
me and opening with a taco question. Oh okay. I think my favorite
taco, like my go to is just like a classic like Carne Sato Street
taco like that is my happy place and I've found the place in La that's
like one of the better ones I've had in my life. So that's that's
very convenient. It's a place called Cacau, Mexico, Testa, and the
thing is how you pronounced the second word in Eagle Rock. It's amazing,
and their molat is delicious right on. I would I would be I would
be really really sad if you told me that you were living in LA and
couldn't find a good taco spot. I mean the thing that it's like I
always feel I mean, of course La is to have a can taco spot,
right, But like I've I lived in Central America, Mexico under okay,
the place were all So I feel bad saying that. But I mean
like there's like this like interesting thing that happens in La. There's a whole
bunch of people here, like like California is like what thirty plus percent Hispanic,
right, and so like there's a whole bunch of people here, and
then all the rich people are bougie and expect the best, so like all
food has to be fantastic, right, So like I feel like those two
things combined for like just it's like like La is like the best food mall,
like everything's delicious. So I say that and I always feel bad.
So I'm like, I've had some really great tacos in Mexico City, but
like one of the convenience of driving tenants definitely impacts it. Oh for sure,
absolutely, yeah. Yeah. Cool. So let's talk about your your
job history real quick, and I'm going to be real clear, I'm not
trying to call you out on it, because, like I mentioned mine,
it looks very similar where a lot of your your positions you were there for
you know, a year and a half, two years, three years,
and I, you know, like I'm older and like my parents were,
like, you get a job and you work there for forty years and your
golden and I've never held a job for more than three years, and I
think that's just I think part of that is just the new norm because after
three years, Like the way I approach it is that if after three years
of working with you, I haven't accomplished what you hired me to do,
then I have failed. What's your take on that. I've got a couple
of takes on this, but a lot of hot takes, So apologies and
advance to anybody, but uh, I think one we have a lot If
you think about our job is software engineers. Can you just classified kind of
all of us under the software umbrella, whether you don't know. So some
people don't feel like software engineers because they're like DevOps engineers and themal engineers,
but you are, like there's a different abstraction layer, different level, like
different expertise. Like if you think about like us in the grand scheme of
things, we're an interesting group of people where our job is being a really
good tool for the most part, right, Like if you think about where
software started, where it was people that were professionals and they use software to
do their job. Right, slowly software development became its own profession, right,
So I see what we do a lot of time is like like like
what's the best way of phrasing it? Like we are brought in to help
business people code five things, right, Like that's really what we're doing.
There's people that have an idea of how to make money and they don't know
how to do the work that we do, right, and so we're not
always invested in the business idea necessarily that we're writing software for, right,
And like you can see this is plenty of companies, Like I know one
of the companies I work on very early it's not on my resumes because I
was a consultant at the time was honest company for sending baby diapers to your
house. There was a bunch of single men there that built the software.
None of them were like, wow, I can't wait to get some diapers
doing that, Like that's the first thing. Is like like you can love
what you do but not necessarily love the thing that you're building, right,
and that has interesting trade offs, like you might get to the point where
it's like, okay, like all the ideation and collaboration with the business book,
which is kind of like what I really like about software is like talking
through the ideas and figuring out like how to make that code, not necessarily
typing it. You to a point where it's like, Okay, we've done
the ideation, We've done the interesting things. Like I feel like I'm really
in a maintenance mode. I don't love the business, so it's like,
okay, well what is here for me? Right? And a lot of
a lot of careers we see people working ten, twelve, thirteen, fourteen
years, they don't have that choice. I'm a lawyer. If I don't
like being a lawyer anymore, I'm kind of shit out of lot. Right,
and so I think that's one part of it. But we're also we're
all hobbyists, right, Like we all it's oh, it's five o'clock,
it's time to get off work. What am I going to do? I'm
going to continue to sit at my computer and write some open source software.
I'm going to go play with something, right, Like we love to play
and learn new things. I think like those two things, especially for me,
like I have a almost like a wonderlust, like I have to be
doing something more, doing something interesting, engaging myself when I get to a
place where I'm like, okay, like there's just not really anything here that
rewards me. And guess what, the businesses aren't either. The best way
If there's any juniors listening to this, like, sorry, guys, here's
here's the news. The best way to get a raise is to quit your
job and go someplace else. Absolutely a three percent raise, which means you're
losing like eight percent of your salary over the last couple of years with inflation.
The best way to jump twenty percent is get a new job. And
at the end of the day, like I like writing software, but you
know what, I really like getting paid well, right, and so it's
like if I got to the point where it's like there's something that I can
say I did, there's somebody at that company that I can use as a
reference to, like Corey Rocked, and it's like, Okay, I've gotten
like some like interesting raises and maybe I'm like in line with inflation, but
I'm not I'm not making more money. It's like, what is the growth
that I have here if I'm not invested in the business, if I'm not
learning anything new about the software, and I'm not getting paid more money,
Like I'm just degenerating in every way, right, Like, so it's time
to go and what's really interesting? And well this doesn't offend anybody, But
like when I see a resume and somebody's like I worked there for fifteen years,
it's like I don't want to work with you, Like no things is
like you you are so ingrained in like your Google you like I worked at
Google for fifteen years. It's like, well, guess what, Like you
didn't work in the real world, like you worked in like the perfect utopia
of like building software around other software engineers for software engineers like that. Like
so like seeing that person that's like they have enough tenure where you know that
they weren't just getting fired from job to job. They had that experiences,
different teams, different stacks, different products where they can bring a lot more
than just being able to type. Like that's that's kind of the value I'm
looking for. And I'm looking at a resume. I'm terrified if somebody just
sat someplace forever, he just work on like the banner of Amazon for eight
years, Like, what did you do? Right? So I know there's
probably some really great time. Orry. I didn't mean yeah, I hear
you though. That's me in a nutshell. Welcome high everyone right now.
I had a boss that I worked for a long time ago, and I
I interviewed for another job and he found out about it was a smaller town,
and you know, I was terrified that he was just going to fire
me on the spot. And he said, well, did you get the
job? And I said, well, they're waiting, are you? Are
you mad? He said no. He said, if if no one else
wants to hire you, why would I want you working for me? Damn,
that's pretty savage. The funny thing is, I like, I've seen
this. The number of times is like a hiring manager, where like you'll
make an offer to somebody, right they come in the interview, go al,
I really like that person, Like I interviewed one hundred people or whatever,
and that was the one. I'm like, that's my person, and
they're like my company counter and I'm like, yeah, if it took you
leaving for your company to recognize your value, then you accept that, like
I don't, I don't know, Like I mean, you take it,
definitely take it because it's more money, but you should immediately reshot that out
and find something better, Like take the money while you're there, and then
go find the person that appreciates you for who you are and wants to give
you a good salary, not just because you're gonna walk right. Yeah.
And the counterpoint to that is if you do stay with them, Like all
companies go through down cycles, and whenever they hit that next downcycle, they've
already set precedent that they're only going to do the bare minimum to keep you
around, and so whenever they have to lay off people, they're likely to
think, well, hey, this person has already been trying to leave anyway,
so they need to be first on the top and block to go yeah,
especially if like you've like if that salary is no longer convincerate with the
value that you have. You just got a twenty percent raise because you were
going to quit. Like now you're the biggest line item on the team,
right, But now I do think of this analogy just hit me. I
kind of feel like what we do are like the wagon train masters of the
Old West, Like our job is to get the settlers and their wagons out
west, and once we get them out west, we're done. Like you
never hire a wagon master and say, well, we're going to use you
to escort our wagon train for the next fifteen years. No, get us
to this point and then go away. Yeah, it's kind of like it
suffers, like we hump if you wanting eighteen eighty three, is that where
this came from? Where you're just like a fore Gun trail fan. It's
just a side quest finished eighteen eighty three is so good? Yeah, No,
it's funny. It's like, yeah, nobody's ever like, yeah,
we'll get out the Oregon. We'll keep the wagging guy around in case we
want to go to like Montana or Alabama or something. Maybe we'll want to
go back to hudos, right, and like that's what you're just like,
Okay, well I'm here until like we have another direction we want to go,
we to change all this stuff. Yeah, that's an interesting one them.
I was talking to somebody the other day, a buddy of mine that's
working on an AI code product, and I will call them out because I
think the big disagreements on it. But like this whole notion of like we've
had this like for like twenty five years for anybody who's newer to the industry.
But like when Macromedia dream Weaver came out, I remember people being like,
we'll never be like business people are gonna build all the web pages and
it's like, ah no they aren't. But like as like as right,
because at the end of the day, like we are tools for business,
right, Like so this idea of like no code and AI it can help
this is people like build software Like that is a great idea, and like
I'm sure that that will happen eventually, But in the meantime, us being
relegated to being code reviewers for a machine like that feels like the most miserable
intersection of how this did go right. It's like the creativity part, and
like thinking of how to like abstract like business logic like business needs and logic
into code is part of the fun part. And it's like, that's just
a computer and I'm just like, ah, I don't like the way you've
spelled that variable, Like that's a that's a sucky life for software engineers.
So but yeah, wagons AI, it's all the same. That could that
could be our new startup wagon train dot a the bafest path to organ that
ended up. But there is there is like one of the things you mentioned
earlier, you know about the the alignment of you as an employee versus the
company. Like the founders of the company are vested in the product. Me
as an employee, I'm vested in providing a really good living for my family,
putting food on a table, putting a roof over their head, giving
them the things that make them feel like they're having a quality life. And
so we have different goals there. But jumping around like we do also is
a benefit to those companies because those people, like the diaper company, they're
really passionate about how to get diapers to young families. But if I've worked
for a bunch of different industries. I can bring perspective to them and help
them build a product that meets that need better because I know the roadblocks that
they aren't familiar with and can't see because they're focused on that the diapers on
the doorstep thing. Yeah, and it's interesting because, like you know,
in those like as a company grows, especially from startup to so you say
series V and beyond, like the product also changes, right, So,
like one of the things that might bring you into maybe new e commerce companies,
the person had experienced a previously e commerce compani It's like, oh,
we needed somebody who knows inventory systems and you know, billing and all that
stuff. Like, let's bring that person in. But then as like this
company evolved, let's say three years, we're doing like a lot of video
options. It's like, why don't necessarily need like this person's deep knowledge and
e commerce systems anymore? I know they need somebody to news how to do
concurrent video streams to hundreds of thousands of people that might be bidding on this
item, right, and so like our business has changed, the people that
we need involved change, and sure like that person that's been there for three
or four years could go learn how to do like video streaming and you know
all that stuff. But it's like, also you could go Hoch somebody from
you know, Prime or something like that. Right, So I think like
that moving, I think is what makes us really good tools. Again,
like if you sit in this narrow channel for fifteen years doing one thing for
one product with one team, that may have people swapping in and out,
like you were very uniform, right, and so like now when you're coming
into another environment, I feel like it's harder to figure out like how to
how to necessarily fit you in, right. I feel like it was kind
of like limits where you I mean, I guess when you have the prestige
of a big band company can kind of go wherever you want. But like
if you were like, oh, that person was at Macy's for seventeen years,
you know like, well we got to get them in here, right,
But it's someways like that it was a Google for seventeen years, You're
like, oh, how much do how much money we have to live in
a bank. Let's give it to this person, give them the stock.
Right. So when we were chatting over email before the podcast, you mentioned
that DevOps is pure bullshit. So I don't want to care about this.
Yeah, I'm gonna call you out on it. This is fine. I
love talking about this, so, like I have to say that, this
is one of those moments where, gosh, you can't remember who said the
quote, like being able to hold two opposing ideas in your head at the
same time. Who was that? Uh huh. I'm not really good at
quotes, and this is live, so I can't google it. But ultimately
it's like a definission of being crazy. So I was like mesuring Franklin or
something. It's like, oh, that's a sign of intelligence. It's like
also inside of being nuts. But it's really I think it was it.
It was one of these much smarter and successful to me said it, so
you got to but it it is and it isn't, And I think it's
one of those things that's interesting interesting. It's like death ops is what it
is based on, Like how much your zoom dinner is zoomed? Out?
Right? You could watch somebody who's building an application. Let's say it's twenty
thirteen and somebody's building a Ruby full stack app monolith and they deploy it on
Heroku. They are they doing DevOps, I would argue, yeah, they
are. They're they're operating their software happens to be running on a platform.
It's abstracted a lot away from it, but they're operating at the abstraction that
they've been given, right Versus Okay, I'm an engineer. I'm sitting around
three hundred micro services. I'm configuring the Kubernators cluster and all the service meshes.
Like am I doing DevOps? Like yeah, you are too, You're
doing like the dev and the ops like sure, So like it can exist
at different levels and you can create abstractions where people can still do DevOps.
And I think that where we are today and how we like handwave a lot
of the definition of DevOps. Like there's people that are in like it's a
team camp. There's people like it's a collaboration camp. And there's the people
that are like you do everything, like you build it, you run it,
like you have to do it all. Ah, It's like okay,
well, like we kind of like three definitions so like that right, there's
a definite bullshit Like it gave defines, right, but like I have three
of those like don't happen for most people, And I have proof. We
love the Dora report and DevOps, but when you go through it, fifty
percent of people that respond to the Door Reporter like everything's fucked here and it's
just like numbers, like it takes us we really software once a quarter and
it's like okay, and it's like an outage lasts up to two days,
Like fifty percent of the Door report respondents report pure chaos. So like while
walk like we're on Hacker News, We're like, haha, at my company,
we do DevOps great. And it's like, okay, you're a soul
CTO who deploys on verse cell, Like what you're doing it right, like
your hands in fine, And then people will be like, we do devlops
great. We have eighty thousand micro services like you also have ten thousand ops
people who've built an internal platform that makes it where you can do a DevOps
with the abstraction it's given to you, right, And so I feel like
I feel like like it is plausible. It is an idea that makes sense,
and you do want your developers to have a baseline understanding of the abstraction
that they're given. But I think the bullshit is in like like like are
we giving people the right abstraction, And I would argue, like, that's
where it really gets super complicated, And the answer is also like we probably
aren't right. So what's interesting is looking at another report. State of CD
report is another interesting one. The most recent state of CD report found that
only twenty seven percent of companies are using I see infrastructure's code, right,
and so we're all DevOps engineers. We automate everything, do we? If
only twenty seven percent of companies are using infrastructure as code and I ventured,
I bet most of them that marked yes and that aren't using it one hundred
percent throughout the company. How are those other seventy three percent of companies that
are doing DevOps automating stuff? If you weren't using antibal, chef care form
or blooming, are you using Selenium to automate clicking through the AWS dashboard?
Like I'm confused here because we're all doing DevOps, but you're not automating any
of your infrastructure management. That is where a whole other chunk of bullshit comes
in infrastructure management. Like some people consider it a part of DE but it
didn't exist really, I mean it was like CF engine, but like there
wasn't much infrastructure management or even cloud infrastructure when like the DevOps term was pointed,
right, And so this whole idea of cloud infrastructure, which is this
thing that ran our software that over the past ten years has slowly become our
software, and it's going to get worse with AI and LLLM models, Like
a lot of our software is no longer ours. Right in two thousand and
eight, my code was my code. In twenty twelve, it was like,
well, all my cues and background processes are actually done by this sqssn
S thing, Like I've got a lot less code to manage I call people
other people's APIs and now it's like step function in land. That's like my
code's getting smaller and smaller and this cloud service area is growing. And then
you look at llms and it's like I don't have any code. I just
shove hashes in somebody else's database and magic comes out of it. Right,
Like we're getting a lot less code, a lot more consumed cloud services.
And I don't think we ever tradition like went back and like include infrastructure management
like in that loom, right, And then when you do, whose responsibility
is it? Because if you say, all of aws's the responsibility of all
the services that you may or may not use in AWS is your engineer's responsibility.
You're out of your fucking mind. Like their responsibility is to build business
value if they have to go read the twenty seven pages of how to monitor,
configure secure YadA YadA, YadA, YadA, YadA dada postgress. When
it's like locally, I just like Docker run postgress, It's like that's the
abstraction that they should see. They should say, I need postgresses, and
they should be able to assume that whatever's happened on the other side, I've
got secure compliance, scalable, cost effective thing. I shouldn't have to think
about costs, and I shouldn't have to be sidelined on it three months later
when the cfo's pissed off or we're spending thirty grand a month on a misconfigured
post resistance. Oh sorry, So that's why I think it's bullshit. But
here's the catch is, like I'm also one of these people. It's like,
hey, platform engineering, you got to do this right, which I
will admit platform engineering is a subset of DevOps, but the catches, like
most people don't have the budget to actually do it or the talent to actually
do it because their DevOps team or operations team is in drowning and technical debt,
and like trying to ticket ops for their developers and people that just have
developers and don't have an OPS team, they're probably better served by the past
and going directly to the cloud until they get to the point that they need
that scale. So that's me defending the DevOps as bullshit. So it was
like a therapy session. Yeah, I'm gonna get your insurance details after the
podcast here that. Yeah, so I look forward to comments after the show.
No, I agree with you. And the irony here is I actually
have the title at my current employer of DevOps Engineering, and I agree,
like that should not be a thing. And I think part of it just
become to in my perspective is we've we've we're just kind of like that's where
we've drawn the line in the sand, like, oh, we're doing DevOps,
damn it, no matter how hard it is. And it's kind of
like, you know, being a forward guy or a Chevy guy. You
know, if you if you buy a Chevy, you're a Chevy guy.
Now you you know, it's like I can't buy a Ford, or they're
going to kick me out of the line dancing club or whatever, you know,
and we have this failure to adapt with the way that our our industry
has changed. And you know, because you can't, like you mentioned,
you can't ask a software engineer to be proficient in writing code and go,
but you also have to be an expert in Postgress database and uh, an
expert in Docker and kubernetties and security gateways and cost management. There's just so
much context switching there that you'll never get anything better than mediocre across the board.
Yeah, and it's funny because, like I feel like I might be
wrong here, I probably am. Okay, Like I feel like two things
have kind of got us into the mess that we're in with develops, right,
because if you think about just like an engineer, like the amount of
stuff that we will stack as a responsibility on engineers, right, why does
this happen? Right? So? Yeah, I'm a software developer, right,
software? Okay, I use a database. I picked postgrats. Okay,
well this is an Oracle nine I it's Postgress. So I'm assuming that
you're the database administrator. Also software developer, you're going to tune your indexes
right like a decade then fifteen years ago, like we had DBAs, but
like you're seeing them become less and less prevalent in businesses except for like massive,
massive businesses because like, oh, it's like that person will figure it
out, and then so that happens. We stack that responsibility on people,
and I think it's twofold. I think one's business people. I don't understand
like where the lines are. And we're again we're hobbyists. So it's like
okay, like, hey, yay, will do you know how like LLM
building works? And you're like no, And you're like, well, you've
got a better chance of figure it out than me over here. It's working
on the old it's right. And so you're like, if you give a
software engineer like time to go discover something, they will do it. Now,
will they become a master of it? No? Will they get something
that works enough to go into production? Yes? And that's where our product
managers are content, like they're not like, oh, Corey, did you
master building elements before you've lobbed this thing into a kuberinetes pluster. No?
But it works right, And that's all matters. Like I made the button
click faster, I sold more of the things. Like at the end of
the day, that's what matters to the business, not necessarily that it's perfectly
secure. Do you care about that business? Do you care that it's perfectly
secure? Or are you going to buy an insurance plan from Equifax to cover
people's credit when it breaches. It's much much cheaper to buy the insurance plan
than to spend the engineering resources to make it secure. Yeah. Yeah,
as just ask my real estate and investment company that I'm working with, because
I prompted them on that. I'm like, how much time you spill security?
Are things perfectly secure? Well, then we do investing. There was
a breach and like we leaked your social Security number and all your financials.
I was like, I knew my credit's locked already. It's fun. But
so I think, like, you know, that's that's it's really interesting,
Like we'll just stack stuff on people. And I think that's kind of where
it's easy for burnout to happen, right, Like you think you're going to
be focusing on this problem, and you are, You're trying to solve the
problem, but all these other things are kind of tangential to it, and
if you don't have the people that know that thing, your company is looking
around like okay, who has either the bandwidth or who do we think is
like the best engineer that can go figure it out? And like now they
have another technology and other responsibility that kind of lands on their plate. If
you're a small company, you're starting to grow and you're like, okay,
like I'm doing a lot of this cloud stuff, like we should hire our
first DevOps engineer, and so now it's eight people that don't have like this
deep experience in the cloud. They are like, we should hire somebody that
does. Then you start that job, you're the death first DevOps engineering.
You're like, okay, I had a lot of work for like three months
and I was stressful, But now I'm sitting around like five hours of work
to do a month. Right. Then that's where it gets weird, because
like you can do two things at that point time. You can either sit
back and be like I have thirty five hours of drinking coffee, or you
can try to do like real DevOps how do I make this easier for those
developers? How do I make things more efficient? How do I make things
more resilient. How can I start practicing some SRI principles since I have this
bandwidth to help make sure their software is staying up. And that's where I
think I feel like a lot of DevOps people, particularly if you look at
our slash DevOps, so that's a group of people not saying it, but
like the amount of times people that are like I have nothing to do,
Like I've been working here, like I work like ten hours a week and
like everything's like kind of messy still, but like nobody's given me, like
the DNS project to work on. And it's like, well, you're supposed
to be talking to your you're supposed to be doing that collaboration part, talking
to your engineers for how you can make their life better, Like that is
your job, and like that devlops role is like continue to make the software
better, the business better, and the engineering dev experience better. And I
think that like once you start going down that path, like that is a
good place for DevOps to be. But where we're just tossing stuff over a
fence to a person and we're like, oh, well, the devlops team
they configure the cloud things for me, Like that that's that's that's just two
thousand and five. I'll just pay for sure. Yeah, I take the
approach that as a as my role in develops. The software engineers are my
customers. So my job is to make it to like provide the tools that
they can consume that let them do their job more efficiently and putting in putting
in guardrails so that they can do what they need to do. But then
I have guardrails in place so that they don't accidentally do something that they may
not want to do, like you know, expose root access to the public
Internet or something like that. Yeah, I think those guardrails are important.
Like that's actually a term that we use pretty pretty frequently and like our sales
calls and even in our content, Like I feel like again like our software
is becoming more and more cloud services that we're consuming and less of our code
a lot of used to B calls, and I feel like when you're using
the cloud, like there's so many services like how do you do how do
you how do you do event driven systems on AWS? That's an SQUS,
is it event bridge? Is it POFKA, Like like I've had three or
four options just in AWS, right, and then let's say I pick it.
Okay, let's say I pick my solution. Looking through some of the
AWS like models when you like post your thing to make a new thing,
Like, some of them are insane. Like one of the most crazy services
at AWS is that's three. That's three, like one hundred and twenty configuration
options, right, And so there's a lot of stuff that is getting exposed
to your engineers. So if you're like, oh, I did DevOps because
like I installed terra form and I get of action and now they can write
whatever, Like you didn't you just you just gave everyone a massive footgun.
You didn't make their life better. Sure they can do whatever they want,
but like should they be able to do whatever they want? Right? And
I think that what we need to be doing as an industry DevOps, platform,
engineering operations, whatever your company calls it, is like enabling those developers
to do what they need to do, but also shrinking the surface area of
the cloud. Right. So, like if you have an established operation scheme
that understands all the cloud services, being able to have like a catalog of
like here's what we use for a venture and systems, it's COFA. If
you don't want to use COFT, that is your choice, but it's now
your operational burden. Like make an RFCs to why we should also support e
vent bridge and like we will take on owning a vent bridge. But like
if you're just gonna loan range it, like go for it, right.
And also even when you pick a service, like you need to service,
you need to shrink the surface area of that service, right, Like the
cloud's goal is not to be easy. AWS isn't We're going to be easy.
If they're easy, they would only have lights fail. Their job.
Their job is to be capable. Their job is that you can cut this
pizza in any shape. You want to make your party happy. You want
squares customers, you want we cut wedges, You want to do a weird
little zigzag, you want all edge, I don't know whatever you want to
do, Like we should be able to do it with AWS. And here's
all these things, right, But the reality is is like a lot of
those configuration settings don't matter for your use case high Aurora, like you can
configure Aurora. There's so many ways that I'm gonna go provision postcress and I'm
gonna turn on this thing. That is this field that I can turn on.
It's like that's only for my sequel auror it's like, oh, well,
then why was it exposed to me? I only use Postgress. I'm
not going to randomly switch from postgress to my sqel one day, but I
might randomly change a field name right, or like a field value right,
So it shrinking that service area is important? Like why why? So?
I would love somebody in the comments tell me why I'm wrong. Why is
turning encryption off for anything an option anymore? Yeah? I agree with you,
and I think there's like a certain amount of responsibility. I actually just
had this conversation the other day. I think there's a certain amount of responsibility
from the cloud providers to do a better job here. Like if if I'm
as a cloud provider, they advocate infrastructure's code, least privileged security on all
those kinds of things. So why in their documentation does it say, oh,
if you want to push to the artifact repository, you need artifact repository
dot admin privileges, Like do I really need admin privileges or was there a
better way you could have phrased that to help me adhere to the principles that
your marketing team is advocating. Yeah, and again like I don't, I
don't. Yeah, it's like I don't. I don't think if their goal
is to be easy, it's just like it's it's just to have those capabilities
and right like I AM is like a whole other just when you can see
it, like it's funny, like I feel like some of people like what
are you telling me I AM. It's like I AM is cloud blocking.
Like everything we're using Dynamo CloudLock and it's like, no, you can use
Scala. It's like S three. We're locked in. It's like I just
throw Minio in front of GCS or whatever. Like plenty of ways to get
around cloud locking except for I AM. Like if you invest heavily, if
you go beyond the asterisk, Oh my god. That is that is like
everybody's im is just like completely different. You can you can tell how like
hard it is, like there's I AM engineering roles now in large companies.
It's just like, oh, like I can't like it's important. Oh for
sure, I can't imagine. I personally can't imagine what that job. And
hear you saying with like pushing a container. It's like, oh, you
know, I need right. It's like getting to those places where you're like,
my sole role is to figure I am for this place, but the
cloud doesn't give me the ability to make maybe maybe a particular service as secure
as I want, right because of the granularity Google. Yeah, I like
software. It might not sound like it. I actually love it, but
I just yeah, sorry, No, I think that's part of it.
I think that's one of the common characteristics of software engineers is there's a really
really tight balance of which we like more. Do we like writing a code
more, or do we like bitching about writing the code more? And and
it's it's a tight race bitching about it. I coulde. I still do
a lot of development, probably more than I should, but I switched between
Go Go Lang and a lickxer that's where our app's built in pretty frequently.
And you know, whatever language I'm in, I'm just like, oh,
I hate this language. I wish it's if I'm going to rewrite this in
language acts, then you get it done and you're like, that was stupid.
Yeah. I had one of those yesterday. Ask him about four hours
working on something, and I was just like, I finished, I was
like, why did I? Why did I? I'm had a purpose when
I started, it got way out of hand, like I still do it,
and like I over engineered the ship out of it, and I was
like, I don't. I don't know why I spent four hours on that,
But I gotta go do some real work now, right, Yeah,
yeah, I'm gonna do something that I can actually claim because I can't show
tomorrow and say that's all I did. If you feel that way, people,
it's hard to shake. This is this is This is me years into
being a CEO, and I will still go over engine or something if I've
got a spare of time. Oh for sure. Yeah. I actually for
my side projects, I have a group of guys that I hang out with,
and we all know that our sole job as part of that group is
to call each other out on that specific thing, like are you chasing a
shiny object? No? Well okay maybe yes? All right, fine,
I'll go back to what I'm supposed to be doing. I'm gonna start the
antithesis of that group. I'm gonna start like a Rube Goldberg engineering group.
I think you've made that more complicated and like throwing some more technology into It's
like, you know what I could, I could I'm gonna spend a couple
of hours this week and doing that an LLM that generates Bash scripts. Oh
man, that would be just go. I'm afraid of I'm like the next
five years. I'm like, oh, I'm in like like a I think
I'm in the trophic disillusionment around AI. Like people keep asking us, we
did our fundraise this year. We raised eight million dollars and we did it
without putting I mean, it was a hard year to raise. We did
it without putting the words machine learning, data science or AI in our deck.
And we've stayed away from it in like our materials just because like I
don't I don't want to build our product off of like some like trend that
maybe not go where we hope it goes. And seeing some of the other
people in the OPS space like ah AI ops and You're like, Okay,
I'm going to give this a spin, and I'm like, I would like
a Hippa and sock two compliant Ubernetti's cluster. And I will not mention the
name of the other cloud provider, but I shit you not. The code
that it generated was some type scripty codeink wink. It was all the code
to like initialize an KAS cluster, and then the comments I kid you not,
I kid you not. The only code was like some import statements.
It was like stree equals new EKS cluster, and then in the comments it
was like configuration for stock too and nip A compliance here. And I was
like, yeah, the intelligence is pretty fricking artificial if you ask me,
but like I don't know, like the thing that's really freaking about it.
I kind of got it, couldn't even imagine it for bachelors be terrifying,
But like this is this is again a part of our problem, like we
don't have enough opps experience in most of our engineering team, and so the
idea kind of like business people looking at engineers and not being able to tell
like where the lines are the same thing happens when you go and ask something
to write software for something you're not familiar with, if you aren't familiar with
running Gubernetes, or maybe it's even just configuring some serveralist stuff, you know,
step functions or whatever, and you're like, oh, I need some
terrorform. It'll do all the step function stuff, make it for my computer.
It's like okay, like cool, you might run that and it might
apply it may or may not do what you intended it to do. But
like now you have a bag of code in front of you, you don't
understand it's doing something. You don't know if it's doing it right. And
like chat GPT was confidently Hey I did it right. I think like you
have no idea, so like are you going to go Google every single one
of those lines of code that isn't the resource block to figure out like,
oh, is this is this what I need to set for this? Like
no, you're like just like the product manager looking like that works right right?
Yeah, I got step pumption. I'm moving on. It's like that's
that's freaking. It's like I would rather have you just gone on, Like
I would just please go click it in the AWS console, like turn on
a macro reporter, install the Chrome extension that led like records your mouse clicks.
Go do that and just use the Chrome extension as like your I C
tool at that point in time. I'd rather do that than run something through
AI and hope it right. Like that's great. I've been I've been struggling
with this for a while is for people who do click ops, how do
you how do you document and track that? But I never thought of just
using the click recorder and Chrome. That's brilliant. The problem solved. Yeah,
yet we've got these We've got these video files. Just watch them and
you'll see how I was built. Well, if I'm a lot of the
I'm not sure I'm a lot to plug myself. But you're looking for something
that feels like clickops through engineers, but you want all that operational maturity and
security. No, I do want to talk about that, which is one
of the reasons why I was looking forward to having you on the show here,
because I like I'm at the point where I advocate strongly for infrastructure's code
because six weeks from now, whenever something is different, I want to be
able to look at a get history or poor request history and see what has
changed, or run terraform and see that whatever's running out on that provider doesn't
match what terraform says it should be. And that's been one of the big
challenges I've had with the platform engineering tools, that some of them are just
clickops on top of clickops, you know, and it's like if I just
want to click ops, I would just use AWS. So how do you
approach that? Yeah, so I think that. Okay, So I'm gonna
tell you a couple of things about me. I'm a software engineer, so
I understand like I understand vendors, and I have to struggle as like a
CEO and a founder and also a developer where like I don't want to be
the annoying vendor. That's why I was like, I don't know want the
plug like I want to talk about like I want to sell this thing,
but at the same time, like I don't want to annoy people. But
uh but yeah, So what's interesting is so the second thing about me,
I'm going to say something and I'm going to be I'm not going to be
the founder that says this and is disingenuous. I think we're going about enabling
platform engineering much differently than a lot of the other tools out there. Right,
So what's really freaking me out about this is I think platform engineering is
important. Actually, I make a note of like the clickops, how we
do it different. I'm gonna come back to that in a second. Do
I haven't have time to set way? He said, we can go on
forever, right, I can, Yeah, there's there's there's no time on
it. So here here's why this is important. I spend a lot of
time like reading surveys. I like it. It's one of my weird things.
And I feel like a lot of times, like when you get a
survey at the door report, there's like a couple of like pull quotes and
some numbers, and you're like, oh, we're all doing much better.
But then when you like get to page like thirty stuff and you're like,
holy crap, everyone's gonna die. Looking at the stack overflow survey for the
past four or five years is really interesting. If you look at cloud operations
experience, every every administrative role, operations role in the stack Overflow survey has
steadily declined in the amount of people doing it over the past five years.
DBAs, SRES, DevOps, operations, all of them are going down.
Now are we retiring beast? Maybe we have a lot of grave yards that
could be the reason. But also we're just making engineers past. We're making
engineers faster than we've ever made engineers ever. Remember seventy years ago, there
was like one software developer. Now there's more software developers on the planet than
there are people in Australia, Like you met an entire, huge ass country
full of kangaroos and software developers. Like that's something we could do, would
be really weird, but we shouldn't do it. But like right, so,
like where are these people coming from. They're coming from boot camps.
Like we're still on a lot of people coming out of universities. We're printing
people out of boot camps, and that's not a bad thing. Some of
my favorite engineers I've hired are out of boot camps. There's a personality type
that I look for when I see somebody coming from a boot camp, like
a fresh grad no history, that's eighteen. That's a person that was really
those probably doing software, got very excited, went straight to a boot camp
because like I want to get in the job. I don't want to sit
four years learning about algorithms to build a database, like I want to build
a website. That person, right, the person that worked at a power
plant for fifteen years and they're like forty three, that decides I'm going to
give up on this career that I've been doing all this time to go in
software, Like that person has grit. I love that person. I think
that boot camps are an important part of our society. Why how do you
attige boot camps to like civilization? Everything's moving to the cloud. Everybody is
a software company. We're becoming more and more dependent on this stuff. We
need more software developments. But what's happening is boot camps don't teach the cloud.
Boot camps don't teach DevOps. They teach you to build react code.
They teach you to build rails. They teach you to add business value as
quickly as possible, which most businesses don't see. DevOps is business value.
Right. So it's like you come out of this boot camp for twelve weeks,
you got your twenty six thousand dollars tuition with like eight percent or whatever,
and you're like, I'm ready to write some reacts. I'm ready to
rails, generate an active record model, like let's go. And so those
people come in and they don't have any DevOps experience. The don't have any
operations experience, right, And I think that's where we're seeing this, like
like this steep drop is like some people are retiring because like we do have
a lot of gray beards, especially on might be in the data centers and
more on the administrative like Linux to step inside. Those people are going away
for sure. We're also just printing a bunch of people that are very much
closer to the top end of the stack, right. And the thing that
sucks with OPS is we don't teach it in universities either, because if you
talk talk Kubernetes. Actually, if you if you rewind four years and you
taught people how to deploy in the cloud, what would you be teaching them.
Probably not Kubernetes four years ago. Today it'd be Kubernetes, but four
years ago maybe the two instance. Yeah, maybe, Like nothing is really
I'm gonna say something bold here. Nothing's really changed about our software in the
past seventy years. LOOPS variables ifs sure, you've got different syntactical sugar on
it. But in the last twenty years, everything about the way that we
package and ship software to the cloud has changed four times. We had data
centers, then we had vms, then we had everybody started going the servi
list, and then containers came around, and then like big Container orchestration,
then servilest containers, now llms, like we've changed the actually llens might be
the first thing that's happened in like the way we write software that's fundamentally changed.
Besides just like if look, you're not because they're all just jam and
hashes into somebody else's database. Now sorry, So so that is sorry.
So that was me explaining like why we're doing things a little different, Like
the the amount of operational knowledge on this planet is going down in relation to
the amount of software developers we have, and those software developers are only going
to learn OPS on the job. That's a problem, right, Yeah,
are they going to learn ops from? Are they? Are all of your
DevOps engineers going to turn into You're going to make a fifty room campus so
you can teach all your software developers like how to do a DevOps You have
brown bags every single day. Like you're not going to transfer that knowledge from
your DevOps team to your engineers in an efficient way, right, So you
need to build the substraction. You need to lower the surface area so they
can do as much as they have access to without having to like jump over
that wall. Right as soon as you reach around that wall, you're like,
hey, I was able to deploy everything myself, but like it's down
Okay, we they gotta be able to troubleshoot themselves, right if they have
to come around to you and be like, hey, I deployed this thing
that build a container. It went up and that is just fucked Like if
they come around to you like that, like they need to have that knowledge
themselves, like we need to be building those abstractions they can solve these problems
themselves. And also at the same time, I think as a platform and
we do this, and we hope that our users do this for their users
do this, but like as platforms and people this knowledge, we need to
figure out how to get it as efficiently as we can into developers because we
do need more people understanding how this stuff works. And I love when one
of our customers even is like, hey, how does this work? Like
under the hoot, It's like, yeah, you should you should be asking
me that, like you should want to know that, but like if you
don't, you should be able to do your job effect And so what I've
so this is a long lead up to me saying I think we do things
a bit differently, which every startup founder sets but the thing that made it
different for us is everybody's talking about platform engineering, and the first thing they
jump to is who there's like Kubernetes. Oh, guess what, I love
Kubernetes. I develop on Kubernetes locally. Like that's how much of a family
I would be, Like, if you want to use my product, you
have to use Kubernetes. I just made a choice for you, right,
Like you might be perfectly fine running on Lambda, right, but we want
to use that forever. So I think that, and I think one of
the things that's a little scary about this idea of platform engineering is like we're
leaning too heavily towards Kubernetes and not accepting the fact that there are other things.
I mean, not that anybody's going to run out and run masive sphere,
but like there is nomad, there are other run times, there are
things that are outside of Kubernetes. Right. It's very plausible that your software
teams in your company operate their software differently. And so the idea that you're
going to come in and be like we have one platform and everybody, nine
percent of you are going to fit your code into this one platform. It's
like, Okay, that sucks for the jam stacked people. That's suck inc.
So what are we doing? Are we gonna run Are we gonna run
llms on versell? Are you gonna run jam stacked on open Ai? Like?
How are we gonna fit this all on the one platform? The reality
is like much like organizations are different, like your teams are different, and
so you can't have a platform that does everything. Those engineers need to be
able to have the agency to get what they need from the cloud without constantly
bothering another person or reading a white paper, and the business needs to know
that that software is secure and compliant. That's hard to do when most companies
are abysmally failing at DevOps. Fifty one percent of you based on the door
or report, So if that's not you, one of your friends is lying.
And also like a lot of these companies, like you can see a
company with like fifteen engineers and like nobody really has ops experience, right,
So it's like we just don't have enough of this, and so our approach
has been to focus on operations engineers and DevOps engineers that are either coming to
this realization that they cannot scale themselves or their company is kind of mandated,
like hey, Gartner said, eighty percent. Companies are gonna have a platform
team by talking about the wrong side platform team by twenty twenty six, and
it's going to make DevOps more efficient. So DevOps Guide, you're a platform
engineer now and does that mean right? And so our approach is we should
be able to work with So like our product's number one user is operations,
they don't use our site, they use the tools that they know. So
mass driver today works with Terraform, open TOFU and helm and we're currently working
on CloudFormation and Gloomy support. And so the idea is you as an operations
team can decide what cloud services you want to support and then you limit that
API of those cloud services using Terraform, Helm, open Too, Food,
et cetera. And so you just publish okay we or Kafka for events,
we support postgresses or transactional database. We run on Kubernetes. We also support
land n API gateway for the services team, and you essentially write Terraform modules
with your whether it's OPA or bridge crew, whatever inside of it. You
create these modules that are your businesses, security compliance and practices spotified into the
naming conventions, et cetera. You publish them into mass driver instead of like
the Terraform registry or whatever. And now when your developers come in, what
they see to state instead of seeing all the services of a WS, they
see the services that my operations team supports. Now what's really cool is they
just draw infrastructure. So like, okay, like we run on Kubernetes,
Like, throw my Kubernetes cluster up there. I have a whole thought,
I have a whole I have a whole rant. I can go on about
how many Kubernetes plusters you should have. It should be more than one.
Cattle cattle versus pets. People, you got all these all these applications or
cattle cluster is certainly a pet. Is that you're afraid to change it.
But we named it and everything we actually had, sorry i'm gonna say something,
and a little side tangent. We had a cople of company flip trips
that we were at. We had essentially two Kubernetes clusters of van production.
And what we would do is every time that we were doing a Kubernettes upgrade,
we would do it too essentially our standby, and then we'd roll applications,
cut traffic to it. And so like that's how we kind of determined
that, like okay, upgrading at one point two seven or whatever, it
worked. And both of those clusters were pets ish we created them with cub
admin or cub no cops, but they both had names and it was because
Siri could never get Kubernetes right, so we had cuban Eddie and Kubernetti.
But anyways, still like our our approaches, all right, I'm a I'm
a brain map to follow me, but so approaches like make sure the operations
people don't have to learn a new tool chain, right, like use the
stuff that you already know, Like there's not enough of you. I can't
as a business, have you spending more time learning a bunch of new stuff
when engineers are already depending on you for stuff, demanding stuff from you.
Your team is already stressed out, and now we're saying, hey, we
want to make we want to scale you to make it more efficient for them
by adding more work to your plate on top of the other stuff that's going
to happen. It's like okay, well, or was if we're also kind
of like helping with outages and monitoring, et cetera, like Are we just
not doing that anymore? Why we go build a platform, right? Are
we all going to turn into React engineers so we can build backstage plug ins?
And it's like no, Like so our approach has always done an operations
engineer, Like we wanted to build something that worked with the tools that your
team knows. You publish it in there. Your engineers just draw the infrastructure,
and so as they're drawing, what happens is they drag let's say they
drag an API gateway out or a lambda. There's these additions that you can
make to your infrastructure's code we call we internally we call them artifacts. But
it effectively lets you create a diagram of your terraform. So you can say,
like my application run time like terrifle module or healm module depends on Kubernetics
and so now like when you drag that into the campus, that will automatically
connect like the nearescubernets cluster. You can move it around like your engineers don't
have to think about it. So it's like, Okay, we're building this
service for the first time we wrote it. I'm going to use the helm
chart here for running going applications. I drag it on. I set my
Docker repository and it just connects to kubernets cluster, I hit deployed and I
get back to work, right, And so they have this limited view of
the cloud. It's the services that your team supports. But then also your
terraform variables are also when they come into mass Driver, are become heavy documentation.
So it's not just like oar name like. You can actually mass driver
supports like super rich and like cross validation, so you can say, like
if the name is this, then this other field has to have this specific
validation on it as well. So you can start to codify a lot of
constraints into the system. So you can say, okay, we have an
Aurora postgrest bundle and it can run. If anybody's a familiar with Aurora,
you too. There's an instance type called serverless. It's very weird, but
so technically you can switch your Aurora back and forth between servilists and server full.
Like you experience a downtime while it's happening, but you can switch it
back and forth. Right, But the configuration changes wildly depending on if the
instance type it is serverless, not if it's a servilists Aurora the instance type
of servilist, right, and so like internally we have an Aurora Postgres bundle
that we use and we use servilists for all of our not like our preview
environments, and then we use to reful for our production environment. And so
that configuration like it opens on the side, panels your terraform variables and then
as you change values, the configuration changes it, it'll actually expose and hide
fields based on this. So now's the operation scheme. You can say,
you know what, I know that this my sql bin log stuff of Aurora
doesn't apply to Postcrest whatsoever. So those fields that there's no those variables are
even in here, you can and worry about thinking about configuring your resource.
And you can also say it when they you have this men in max scale
for your instances, when somebody selects a servilist instance, those fields are hidden
so they don't accidentally say I want ten serviless and instead they show it shows
you acus right, and so you can start to codify a lot of convention
there for them. And so now your engineers they don't have to learn a
new tool leaver, they don't have to go copy and paste a bunch of
Terraform modules and do a directory, set up some GitHub actions. Literally,
the only thing to do is they put our CLI in there and they says,
Okay, I want to deploy my production database or I wanted to pull
in my production application, and it's pretty much just like the name of the
resource and mass driver and it deploys. You can put adjacent file in there
that it'll get applied to either HELM or Terraform or flu Me to configure it
based on the constraints that you've defined as an operations engineer. And now what's
happened is you're able to scale this operations team. You can do platform engineering
effectively. Mass Driver we're a platform orchestrator if you're familiar with the seven hundred
terms that people have invented in this space, and each of your projects and
mass rivers effectively its own internal developer platform. But the cool thing is it's
not developers having to operations team doesn't have to go and talk to developers and
make an MVP and make a giant form to survey them on what they like
and don't like, and collaboration back and forth. You say, this is
what we support as a business. This is all that we know. This
is what we're using today, and if you need a new service, we
get hub issue and we'll build the module for it. It'll be a mass
reever. You can drag and drop it. Right Like, at the end
of the day, all of your modules and mass rivers get up repositories where
your code is like that's how we kind of do the collaboration. And then
your ops team didn't have to learn anything new. Your engineering team didn't have
to go learn terraform or polloomy or gett actions or anything like that. They
can focus on writing code. They just drag something onto the canvas when they
need it. You as the ops person, get an audit log, get
deployment histories, you can roll back. We have automated costs and metrics built
into it, right, so trying to give that full agency to the developer.
As soon as you can figure something a mass driver, a day later,
costs show up. So if you can figure this postpress citizence, you're
like, holy shit, it's three hundred dollars like the next day right Like,
okay, I'm gonna change my configuration here because I can also see utilizations
three percent, so I don't have a mad engineering manager mad CFO three months
later when they realize that cloud goes high, I, as a software developer,
have this agency to control my costs. Do the same tool that I
provision it in the same place that I can see my metrics. If we
turn your dash your canvae, you're essentially your your network diagram is also your
dashboard. So you can say, okay, I want all these metrics from
cloud watch to show up alongside this instance and set some alarms for them.
So now when you get an alarm goes off at two am, because that's
when grimlins use the internet, right, I think, So it's like two
thirteen, it's like, what in the hell is going on? Who is
a Crown job schedule right now? So now, like at two am,
when you get an alarm in an email, it's like you would like Peter
do right. You think about where we are today, It's like, oh
pat patre alert and you're like, oh fuck, I hate that sounds two
thirteen in the morning. Click on it, uh Ari zero alarm and data
Dog. You go to data Dog and it feels like you just walked into
Vegas at three am, just like crash all over the ground. You're like,
Okay, some something's broke. I can see the things going up this
way. It's red. That's not good. But like, what what's broken?
Okay, it looks like I'm out of database connections on my application?
Well, shit, like our traffic's no different than normally is I din't the
application hasn't been deployed in three weeks because it's pretty stable app. We're not
like actively working on Like what's changed? Like if the traffics the same,
the application hasn't changed in weeks, Why am I out of database connections?
That's super weird. Well, the reason you're out of it is because Tony
on the ops team, he got in trouble because the post residence was too
expensive. CFO was mad. They're like, we got to control our costs.
He scaled down the post for instance, and it can't handle as many
connections now. So now this change that happened that's not in your purview as
a developer, and a different git repo has caused an outage on your application,
and you're sitting there trying to figure out like, why why can't my
application pack the postgress? If you don't have access to that, get repo.
You have no idea if you don't have a dashboard for that Postcriss sentstence,
you have no idea. If you haven't seen the cost to change that
post, for instance, you have no idea. Like we don't have agency
as developers through chaos. Right, So now you're sitting there for hours trying
to figure out what it is. You hop in the slack and you're like
at channel like, has anybody changed anything about the data it's And Tony's like,
yeah, yeah, wait, there's some costs of things. It was
sitting at like ten percent CPU all the time, so we just scaled it
down to some T three micros or whatever. And they're like, oh,
there's your problem. Right, And so with mass drivers with the story's a
little different. Is you get your alert, you click on the link,
we take you to your infrastructure diagram. We highlight the thing that is broken,
and from there you can see the metrics and you can see the change
history of everything. It's the dependency of your applications, is your cluster.
You see your databases, you see your cues, and I can see right
there. I'm like, oh, Chris made a change to the database yesterday.
What did he change, you click on the database you visually diff will
show you the difference to actually keeping track of the changes in your terraform Holme
et cetera. It's like, since yesterday, this is what's changed the instance
type. And you're like, there's your problem, and now I can fix
it. From right there, I can go back to bed and in the
morning, I'll call the CFO and be like, that instance is going to
be expensive again, right to my app to deal with that, maybe if
thro a PG bouncement, then the mix, like the ability to like get
through and resolve an issue like this was never like our intention to the platform.
Our attention was like letting people get infrastructure securely and safely. But what's
happened is like it's just much easier to understand what's going on when you're looking
at a diagram, right, Like we all want diagrams. We draw it
on a whiteboard, we go to our desks and we do some crazy shit
and terraform. And then somebody comes along six months later and they're like,
to diagram this, They terraform, visualize it and get that horrendous resource bag,
and then they try to back that out to like a lucid chart or
like cloud Craft, you're like, hey, we've diagrammed it and nobody updates
it for six months, and then six months later somebody someone breaks and you're
like, this isn't what our system looks like at all, Like we want
it to do. Anybody who says we don't want it to be visual,
do you've ever seen touch a whiteboard or cloud craft or lucid charts, You're
a liar, Like, we want it to be visual, but like there's
not a way of doing it. So I think that's what's fundamentally different about
us, Like we don't want to just throw a portal on top of Kubernetes
and be like you can see all your services. Like it's like we wouldn't
change the way that you think about managing it and change the way to think
about troubleshooting it without making you change all of your tooling. Like developer in
this scenario just writes code and your engineering team or your ops team says,
hey, we support on AWS, we support ekas, and LAMB does great.
We have two separate platforms to land the platform in a Kubernetes platform and
mass driver. I use the one that my team needs if I'm on the
back end team and we're like, actually, you know what, I'm making
the service. I don't know that need a full instance. It is very
small. I could be able to run on LAMB, but I don't have
to go and try to get the ops team or the platform team to like
build out this new functionality for me. I just said, we already have
API gateway and Lambda. The front end team uses it for some jam stack
stuff. I'm going to drag this LAMB over, push a small function up
in there, and call it a day, right, Like, like,
why should I have to engage with a product team and spend time trying to
convince them of what my needs are waiting for them to build an MVP survey.
The rest of the team's like I need to get some work done,
right yeah. So yeah, because then your your ops team becomes the bottleneck
and the limiting factor to your business. And that sucks because like I feel
like we're always the bottleneck. Yeah, they're always the bottleneck. I mean
nobody's ever like, oh my gosh, I was going to go ask the
ops team for something and they already imagined it and did it before I got
No matter what even in the best scenario, you're like you're waiting a bit,
like we're always a little bit of bottle. You bring us in early
on a project and you're like, Okay, we're working on this project.
We're we're trying to figure out what the cloud infrastructure is. It's like you
can do that, But I also feel like when you're very very early in
a project talking about what it could be, like you don't know how the
software and requirements you're going to change, right, and so like the end
of the day, in reality, it's like any given terraform resource or collection
of resources isn't months of work for us, right, It's like trying to
figure out what you need, what abstraction to expose to you, and then
I cobble up some terraform, put in some practices, right my OPA rules,
and like off we go, Like I don't need three months to do
that for a couple of services you're going to use, So like when is
the right time to bring me in? Probably like some percent of the way
through the project, Like once you have something like viable running, maybe you're
using local stack to like figure it out locally, or you're using mass but
as you can, but so like like when do you bring us in,
Like we're always a little bit of a bottleneck, and the reality is is
like maybe Google or not because there's ten thousand OPS engineers, but like the
average company where you're out ratio, I think we say like one to seven
is like the good ratio of OPS to engineers. There's not one to seven
on the planet. Yeah, but like if you look at the if you
look at the stack Overflow survey and you look at the numbers, they're like
it's it's not there. Like if we have twenty seven million software developers,
we do not have two points. Sorry I'm not good enough, but we
don't have seven million operations engineers, right, Like, there's not that many
roles. There is that many roles and they're all sitting open, right Like
we have one of the longest roles that sit open. We also have one
of the shortest tenures. Why is that? Why does it take forever to
place with us? And we quit all the time. We're also one of
the highest paid people on the team. It's because we bounce those jobs constantly,
get those raiss Like we are always going to be a bottleneck until we
start doing something that helps us scale, and even in the platform engineering scenario,
like we still are a bottleneck when somebody starts to move outside of the
platform, right, right, we get that good balance of like the platform,
like I'm never a bottleneck to you because the platform starves what you need
and then when you kind of go outside of like the norm, you go
off that golden path. Now we might be a bottle that now we're talking
about being a product. Okay, great, Like what do you need,
Like, how can we do this for you better? Okay, great?
We need to support ECS because there's some ECS task integration with redshift or something.
I don't know. I don't think that actually tips Like let's say there's
like a very specific use case like okay, cool, let's put together a
real proof of concept terraform module that has ECS. I'll push it up to
you. You can throw it onto your canvas, connect it to the redshift
module we already have, and like, let's let's iterate on this and get
it good enough so that we can have like market like production quality for any
other team that might need to use in ECS, right, And I think
that gets into the social aspect of DevOps too, Like if you can socialize
to the teams that you support, these are the products, like, these
are the platforms that we've built for you to use, and you can like
define the rules ving engagement. Then they know when they're venturing outside of those
rules engagement of engagement, and then they need to come back and talk to
you about that. But without doing that, the engineering teams see DevOps as
everything being one off. They're like, oh, I'm going to deploy this
thing. I gotta go talk to the DevOps guys, and it's such a
painful process that they wait till the last minute because they just don't want to
deal with our bullshit. And so I think, just like creating that catalog
for lack of a better term, like the here's the things you can choose
from the catalog, and we do custom orders. We just need a little
lead time. But if you can like get get them to that point where
they feel like they understand what you do, then you can socialize that conversation.
And I feel like those conversations are always hard, right because like I
feel like a lot of people again like maybe not everybody that's listening, like
has this experience, but like when you go to an ops person, whether
it's a ticket or you walk over to their desk and like, you need
something that's like out of the ordinary, right, Like let's let's let's imagine
that you had this magic tool where you can actually find all your terraform modules
that your company supports, right, and you go to that, you go
to that repo and you're like, ah, like Mango at list isn't there.
We need Mongo outists, We need a document store. Maybe they don't
have maybe they don't have postcrests and they don't realize that you just use postcrusts.
Sorry, I need Mongo at lists but it's not here. Now,
I got to go talk to the ops team. Ticket got to figure it
out myself. Maybe if that's our if that's our culture, and that's fine.
But you go to the ops person you say, hey, like we
need Mongo out lists with a new project, and it's like we do you
need Mongo at lists because they send you some marketing materials? Or do you
need mongo because we can throw Mango and a container for you? Like what
do you like? Why do you need at lists? And also like important
question like why not why not document dB? Yeah? In aws? Right,
like and the catches is like it's really funny. Like people anyone out
there is use document dB. I know, I know, I know,
chill out. I'm trying to make an example here. I'm I'm not a
document dB fanboy, but like reality is like there's a like document dB like
can have a pretty similar experience to Mongo Like there are places that like obviously
Mango's going to move faster in feature development and like the Mango space, yes,
but if you're not using you're just using some basic document store, like
you might not need everything that Mango has. And if it is, I
think Mango even publishes like a last time I checked, it was like either
like a list or tool that could show like what document dv didn't support.
Right, They're like, hey, like use us because this doesn't do this
one thing. It's like you don't need that operation now and you can't see
the chance you need that operation or configuration option in the future. Does it
benefit us to spend the money to have an enterprise account or whatever Mango atlas
where we'll bind to our vpc. We want to go through the effort of
peering our VPC with Mongo aut lists, Like do we want to go over
the public internet to hit them. Like there's a lot of questions in the
mind, and so like the gut reaction from the person is like why do
you need why do you need Mongo Outlets, which like it's very hard to
ask that question without sounding like you're criticizing the person that's made that decision,
right, And it's like, I'm not saying you're wrong, I'm just trying
to understand, Like I don't know if you know all the options that are
available, and I'm trying to figure out, like what is the right service
for us? Right, so, like can we can we try document dB
so it doesn't create a ton of work for us because it's if you need
document dB, I can throw together that terriform module. I'll skip lunch if
you do this favor. But if you need Mongo Atlas, when we have
to go sign to enterprise account, talk to procurement and do some VPC bindings,
maybe we have maybe we've made the dumb choice of having you know,
that that side or something that that makes it impossible for us to buy into
other ords it to appear to other people's networks, right, you don't know,
Like you don't know like how well or chaotic our VPC is, right,
So like there's there's other things that come to mind when we start talking
about using these. They says services that sit outside of AWS, and so
I feel like, like that that's just like a problem that exists like in
our in our collaboration, and I feel like that's another thing that like we
just need to be able to simplify, right, like being able to say,
like, these are the things that we support, you can quickly find
them, right, and be able to get new services into people's hands quickly
even to experiment. But like how do we get the time for that?
Right? Like I've had cases where people have asked or something like I could
throw together a document dB. Like this is a real scenario. I'm parroting
from a previous job. It's like I could have put a module together for
them in maybe an hour or two for them to try out document dB to
see if it's good enough. But I had too much work to do,
Like I didn't have enough time to do a little experiment to see if I
could save myself a bunch of time, a bunch of time, right,
And so it's just like okky, like they need this thing, and it
always seems like they needed now right. It's like, oh, like why
didn't this come up. It's like, well, we didn't know we need
document databases when we started the project. You were at that meeting. I
was at that meeting. I wasn't at subsequent ones because like there wasn't anything
that was needed for me or using postgrass got to the point where like somebody's
like, oh, we actually need Mango for this, and someone brew installed
Mango or they added it to their doctor composed file and they didn't think,
oh shit, somebody else over there is going to have to do work right,
So now I'm a bottleneck again. It's like I'm not a bottleneck.
It's that whole like, you know, lack of preparation on your part.
It's not an emergency on my part. Like an asshole for saying that,
right and sure, so like that's the thing that just kind of sucks,
like that collaboration is really really hard to do well for most organizations. Can
we jump to that thing that we were talking about earlier before the call started?
I want to share this story. Can we talk about do we have
time to talk about some IT stuff some some IT help est data stuff.
Yeah for sure, because we both we both started our careers in the help
desk. So help desk corse stories. Put take one, go for it
the one I wanted to share with you earlier. So my career. I
was originally a physics and an architecture major, but I was working at a
hospital and I was working in data entry. I'd been doing software development.
Was like fifteen or sixteen. I wrote a little program that did my job.
I did my data entry, so I just like read from Excel docs
and like input them into this like terminal system used for explanation of benefits for
the hospital I worked at. And when I automated my job, I wasn't
a software engineer, so I didn't sign like a PIA forum. I didn't.
They did not have rights to my software. I wrote it at home
like it was mine and say buy it from me, which is exciting.
So how I got into help desk was they offered me a thousand dollars for
my little program and a job on the help desk team. And I was
like, what, dude, pay me nine dollars an hour and give me
one thousand bucks. I'm eighteen. I'm like I'm rich. I literally I
took that thousand dollars, Like did you not I put a down payment on
a wave Runner like I was imagine I didn't have a place to park it.
I did not think this through. I lived in a dorm. I'm
like, what the hell am I gonna do with this wave Runner? But
I was just so stoked. I was on the help desk team, and
wow, like help desk is like we give those people so much shit,
Like I think of every support person at police. It's like it's so hard
not to be like especially like especially calling help deskers anything technical as a software
engineer, like yes, I know, I understand how to log and just
yeah, I know, like right, like and so it's frustrating. But
like those those like we're frustrated with them, but like their job is really
hard, like they do run into a bunch of people that are just not
familiar with computers all day long, and so they have to assume like they're
meeting more people like that than people that are right. Right, we had
seventeen offices and my second or seventeen offices across Florida, and so like,
aren't my my domain was pretty big, like to drive places all the time,
I had a couple of like data centers, like it was the first
one was in the building I was in. We have second one up in
uh, like Zephyr Hills, and so I'm in Fort Myersporters. So anybody
in Florida, Fort Myers or Zephyr Hills, imagine driving this distance for help
desk golf. So next time out that person there talking to you like you're
dumb, like this is this is their life. This is a multi hour
drive. And effectively what has happened is like there, gosh, I can't
remember what it was. Something was frozen on their on their computer, and
every time they rebooted it was frozen, and it was just like okay,
that's a heart and the troubleshoot, like I can't even like remote desk into
it, like because they rebooted it. And this is, by the way,
this is two thousand one or two thousand and two, so we didn't
have the great tools we have today. So I drive two hours to Zephyr
Hills. I go into the office and like, show me this computer,
and sure shit, I sit down at it the screen, I grabbed the
mouse and go to click to log in. The mouse I hit the tab
button and it tabs into the username field and I'm like, it's not frozen.
That's interesting. So that I put in like one of the AVON passers.
I log in and I'm like, okay, I grab the mouse and
move it again. Nothing and I'm like. The first thing I do is
I look under there's this big tower. The mouse is unplugged. My ve
driven two hours for an unplugged mouse, and I have to drive two hours
home. And I looked down and I'm just like and so I plug it
back in. It works, And she was like, how did you fix
it? I'm like, the mouse, the mouse, and this is this
isn't like a USB mouse is like one of those old like DIN connector things
like real hard to yank this thing out. So what I was able to
tell from looking at it as well was it was pulled out with some force.
So the assumption is it got wrapped around her rank ankle and pull out
of the machine some of those sliding things right here, but it slid out
with like the keyboard right in the mouse right like there was some serious force
pulled this thing out. Someone must have known, like I feel like a
trick must have happened. But yeah, that was that was one of my
that's one of my favorite ones. That was a little fun. My favorite
one that was not fun. This is second second. I ended up kind
of focusing in healthcare once I changed careers. So my master's is in healthcare
information system, so work for another company. And we had these two offices
that were gosh, I don't want to give away, Well, I guess
I have to give it away. These two offices that we're connected to the
data center here and literally across the street was one of our heart surgery centers.
And this is two thousand and three or so, and like we're pretty
far along, Like we've got Wi Fi all over the place. We've got
a like a digital pills system for dispersing like pills in our pharmacy. Like
we've heavily invested in technology. We had a really great engineering budget. We
had badges blogging the machines, which is a nightmare for him. We got
to find so much from doctors just leaving their batch's places by like a hip
a shopper who came in and like the doctor left the card and they logged
in and just accessed a bunch of shit. It's just like kill me.
But my boss was, this is like in the days when you could haze
new engineers. I don't know if you ever do you ever get hazed?
Yeah time or two? Yeah, progressing from help desk into like the network
operations, like working on data center. And so we had these two sites
and it was going to cost us something insane like one hundred and twenty thousand
dollars or something wild to run internet like under the road to our other office.
And we didn't want them like using dial up or anything like that to
like dial in heart surgery center, Like we need this thing to be online.
And so we did is we installed microwave dishes on the roofs pointed at
each other, and we had point to point microwave internet between between these two
offices. And they started having outages and they were very intermittent. They're very
random, and they're very brief, but like brief enough that it would like
stop like an upload or you know, like some sort of scan upload they're
scanning back to like our medical record system. And so we thought it was
birds. We thought birds had gotten on the on the things. So my
boss is like, like, we we like how we can fix this.
I'm gonna throw some of those little bird things on there, like stop them
fronts. I don't know what putting those on a microwave dish actually would have
done. And he's like, what's confirm that birds? First? I'm like,
what are we to ask them? Like? He's like uh no.
And I was like, okay, well, what are you gonna do?
And he's like come here and he walks out to his truck and I should
you not. He's got one of those umbrellas with a stand on it,
a lawn chair, and then he had to walking talkies and I'm like,
he's like, you're gonna this is Florida the summer. He's like, you're
gonna sit on the roof and when they have an outage, I'm gonna call
you on the walkie talkie and you're gonna tell me what you see. Plus
So I'm sitting up there for like six hours and all of a sudden,
walkie talkie, hey h surgery centers down what's going on? And I'm looking
at I'm looking at the thing and I'm like, dude, there's no birds
on this thing. Like I'm looking at like there's there's no birds like I
don't know why. I still there's no birds, and so like out my
little like fox hound thing, I can still see that there's like signal going
through the wire. There's just out and then all of a sudden it's back
up, and I'm like he's back up. I'm like, I'm stumped,
Like I'm stump, like this was not in the net plus book. This
wasn't in the A plus book, Like I got it, I got an
MC. I see this was not on the test right like, And so
I'm sitting there for like a while, and maybe like two or three hours
later, goes out again, and I'm like, maybe it's a burn on
the other side, Like maybe it's a burn on the receiver. That didn't
even think about this. I don't know if everybody else is like, it's
a burn on the other receiver, you idiot. And so I look across
the street and I don't see the satellite, but I see as a semi
truck parked on a restante. Like the other building was a house that was
converted into a surgery center, and we were in this essentially this strip mall
that had been converted into our data center and like our primary care physician's office.
So this thing's like thirty forty feet tall, and this house is you
know however tall houses, and so the beams just hitting the side of this
truck probably packed it tons of stuff, and so with longer polls. Yeah,
meanwhile, the driver who had frozen pizzas in his truck shows up with
fully cooked pizzas at the destination of just like blast and I don't I don't
even remember how faster that was with was the one hundred megabit I think back
then for like a network. I don't know what. I don't know what
the satellite, a little satellite this thing did. But yeah, he's like
I got fourteen cooked pizzas and a brain tumor or anything else, uh helped
us. I don't know, Like I kind of want maybe when I retire,
I'll just go back and work in it for a year where it's like
it's probably a lot less stressful when you don't need to do it to pay
your bills, Like I bet it becomes pretty fun. Then it's just like,
okay, like what kind of weird shit do y' all have for me
today? Right to see how far you can push the envelope? Oh my
gosh, but do you have a good help desk color story. Yeah.
So the first help desk job I had was working for a manufacturing company that
had a huge sales department selling their product. And you know, this was
this was like nine two thousand era, and so they the salespeople were just
coming out of the stage of having their roll of decks where they would pick
up the phone and dial and then write down in a notebook what they were
doing. So they were trying to get them to you CRM software, and
at the same time they had to build computer skills, you know. But
I get this call from this sales guy. He's like, hey, I'm
out of disk base. I can't save anything on my computer. And I
had the ability to check their disk base and I looked and I'm like,
now you've got You've got plenty of disk space. He's like, look,
I'm telling you I cannot save anything on my computer. I don't know what
your little, your little magic thing down there does, but it's wrong because
I'm telling you I cannot save anything on my computer. Like, all right,
I'll come up. So I'll go up there, up on the third
floor. I go up there, find him in this sea of cubicles,
like, Okay, what's going on? And he's like look, and he
shows me his desktop, and on his desktop he had icons for every saved
document he had sent to a customer, lined up from left to right,
and there was not room to put another icon there. He's like, I
can't save it. There's no place to put it. Yeah, he was,
he was. He was geographically out of space, That's what he was
trying to say. Right, I was writing down what I thought it was,
and that was not it. You guess. I was guessing he had
the hard drive you're seeing that, but he was trying to save something to
a floppy diss Yeah, it was too big, but it definitely wasn't the
grid on Windows ninety eight. Oh my gosh. It is fun. It
is fun, but it is also frustrating. So be nice to your support
people. Yeah. Absolutely, Like with a grain of salt, understand like
who they might have been talking to just before they got on the phone with
you. It's so hard. It's hard, like too when it's just like,
oh man, like like making it. I always felt bad when somebody's
like I feel dumb, and it's like, I mean, like I don't
know I'd feel pretty dumb if I was doing open heart surgery, like I'd
fucked out, like like you shouldn't have to know. But uh, oh
gosh, that's funny. Cool. Well, Corey, it's been great having
you on the show Man. This has been a blast. I have had
a great time. This is super fun cool. Well, I'm happy to
have you back on anytime, because I'm sure we've got more stories, especially
healthcare stories. We could go down the healthcare path. I worked as as
a support operations engineer for a teleradiology company, and there's just so many stories
coming out of that that are just too funny. And I think it's been
long enough now where like any like anything I could get sued for saying has
past, so I could tell those stories now. Oh my gosh, I
have I have one that can't be told on camera. That is right mind
boggling. I don't know if, like if you had this experience. But
doctors, doctors all think they're God, or at least most of the ones
that I met, Yeah, when they're doctors that like own like that.
I work with a big practice and the doctors were all like board members and
shareholders of the practice, like ooh, that ego is gets big, but
we had one where his ego was big and his sense of humor was brutally
messed up. Yeah I'll hear that story once, but not streaming live anymore
because it is not it is beyond rated. Are right on, And that's
a segue for our new podcast, Things we can't say on the air,
So, oh my gosh, listen. No, sincerely, thank you.
I've had a blast. I appreciate you coming on the show, and thanks
to everyone who hung out for us this entire time. I hope it was
helpful, entertaining, and I hope to see you all next week. Yeah,
if it wasn't helpful, I really hope it was entertaining. That's like
it was over in life and life. But they're always competing, and yeah,
if you could just hit one of them every once in a while,
all that's that's winning. If I give somebody a ball, alright, alright,
see everyone, Hm