Speaker 1: Yeah, they changed the goal life button.
Speaker 2: It's just.
Speaker 3: I know the UI designers feel like.
Speaker 1: They're contributing, but sometimes, man, sometimes I question it.
Speaker 4: Where we have some on the show.
Speaker 1: If you're a UI designer and you swapped the order
of the okay and cancel buttons, give me a call.
You can talk.
Speaker 3: Anyway, Warren. What's going on?
Speaker 2: Man?
Speaker 4: Well, you know, thanks for welcoming back.
Speaker 5: I know the last couple of weeks I've been actually
out of commission traveling in Berlin.
Speaker 4: But I have an interesting.
Speaker 5: Fact for today the Adventures of DeVos, and that's in
the last three weeks there have been two different TOTP
mobile app providers who have both.
Speaker 4: Been popped in some way and significant vulnerability has been discovered.
Speaker 5: One of them just straight out sold to a let's
just third party. So it's a good reminder that passwords
are always insecure and mobile apps don't make it any
more secure.
Speaker 4: It's really amazing.
Speaker 1: Actually, I'm just going to say, a revenue stream is
a revenue stream is a revenue.
Speaker 3: Stream, right on? Yeah, I agree. I wish that more
more providers supported.
Speaker 1: Ub keys because I don't like those a lot, and
I hate the ones that say hey, we'll text you
a code. My great, that's not secure at all, but
thanks for making it harder.
Speaker 3: To use your app anyway.
Speaker 1: Joining us today, Marcus Thurner, Marcus, how are you man?
Speaker 2: Yeah, thanks a lot for having me. Exciting to be
in a podcast.
Speaker 1: Yeah, it's going to be an excellent podcast as I
understand it. And you are the director of technology at
Vista right, yes.
Speaker 2: One of them. I'm booking in order.
Speaker 3: Management right on.
Speaker 1: And then we're just going to completely bypass that and
derail the show entirely because it says on your LinkedIn
profile you are a professional chocolate taster.
Speaker 3: Is that a real thing?
Speaker 2: That was a real thing. It's not a thing anymore.
It's just like, yeah, my wife had a startup for
four years in that being to bar chocolate making business,
and you gotta do what you gotta do as a
husband and taste chocolate and enjoy their mind. It was
a good time. But I think it's still good to
keep some fun stuff on a LinkedIn profile instead of
like it starts to be too serious O advice.
Speaker 3: Right, Oh, I agree one hundred percent.
Speaker 1: There was a time early on in LinkedIn days where
you could just indorse anyone for anything, And so some
of my friends endorsed me for professional gift expert or
something like that, because about ninety percent of the time
I'll reply with a gift or a meme instead of
actual text. And so for long, the longest time, I
was endorsed for that on LinkedIn, and then they decided
to take that seriously and removed it. And now you
can only get endorsed for authorized roles, but no professional Yeah, right,
like professional chocolate taster.
Speaker 3: I mean I like the fact that it was.
Speaker 1: For your wife because your vowels were for better or
for worse, and I mean that's just the job.
Speaker 2: Absolutely absolutely.
Speaker 5: I wonder if there's a connection though, like have you
been able to find a way to take the lessons
learned from startup chocolate making to back to the technology world.
Speaker 4: Has there been any overlaps?
Speaker 2: I think I think yes. I mean running a business
is actually you learn a lot more than you're a
single woman or you know, just a very tiny business
just trying to survive. You do everything right, you do
everything for your customers, you do everything for every cent.
And I think you know, I didn't live it personally.
I've worked in a tiny company before, so I did
a lot of that myself, so but I think that
is what comes daily into a software engineer's job. I mean,
you're not nestily getting tasked of, right these three functions.
You gain tasks of solving a specific problem and then
it comes in and sometimes the problem is solved by
writing an email and not writing any code. So so
there is this sense of I want to do something
good for the customer and for the business or ideally
the mix of both. And yeah, and I think it's
a different drive what you have and what type of
skills at what you learn. And I think it goes
back to university projects where everyone does it in some
form or shape, and you know, motivation there and just
trying to be trying to make it the best what
you can at that point. But this older naivity what
you have, I don't know, fifteen year old in school
or twenty five year old in university. And so.
Speaker 1: Yeah, I think there's a tangential relationship there to freelancing
as well, because I did freelancing for a number of
years and I talk with a lot of people who
look at the rates that freelancers charge and they're like,
oh my god, that's amazing, and so I try to
caution everyone and say, okay, that's that number is big,
but you're not always going to bill forty hours a
week like you would, So like divide that number by three,
because for every hour that you work billing a client,
you have another hour of work for accounting and bookkeeping
and all of the stuff with running your business. And
then there's an additional hour on top of that where
you need to do sales and marketing and prospecting and
finding your next gig. So even though that hourly rate
looks big, you have to divide it by three to
realistically put it into.
Speaker 3: Scope of what your actual job is.
Speaker 5: I think you're under selling the stress that comes along
with it as well. Right, you're not just doing those
parts of the job that you normally think about. There's
every other aspect of it. And then there's getting away
from it and going back quote unquote home and still
thinking about your business because it's your livelihood, and how
to market in sales outside of that even when you're
not making anything for that particular customer. So I can
imagine there's a lot that goes into that for a
very small startup, especially one in the chocolate tearing space.
Speaker 2: I think it definitely also depends on even though now
in software engineering or some technology aspect is I you
more into short term projects. There's just like a lot
of noise and overhead and phone calls to just make
I don't know this week, this week's gig works worth
it and that as much be Oh, I'm basically just
like a loaner and heavier project assignment. There is you know,
the economics change, but also you get a lower rate.
So I think in the end then market tries to
optimize for that. But like, yeah, you definitely learn also
back to those skill sets and this drive too. I
need to justify that hour, right, And I think that's
the beauty of external people. They show up, they need
to justify every single hour. And if external people are
used the right way, independent on whether they come from
a big company type of consultancy or as a one
person is they don't have the ow ahead of a company, right,
And that's that's I think important as you collaborate with
those external people and making sure this hour needs to
value differently than an internal one. The internal one you
train that person to be excited in five years to
grow internally versus for the other one like, this is
the task in a week you on, this is how
I can can actually get the value out of it,
and also, hey, I paid you that, but you also delivered.
I think that's really interesting dynamics, but often an oversight
for those who are on the other hand, so not
the freelancer, but actually receiving the freelance work and actually
make that, make that a thing, and make make it
work and make it valuable.
Speaker 3: For sure.
Speaker 1: So let's shift gears a little bit and talk about
engineering excellence because that is our topic for the day.
So Marcus, give us, like your your high level thoughts
on what is engineering excellence and why it's important to you?
Speaker 2: Mm hm. So I think it potentially builds up a
little bit on where you started us here on you're
just just having to drive on figuring out what's best
for the business and the customer in the end, right,
and you just like it's not fulfilling a chop description,
but it's just whatever needs to get done. But that said,
let's let's focus more on the excellence part of the
engineering part and probably start with what is excellent, right,
there's probably variety of definitions when you look up on
the dictionaries and like it's wake, but some of those
sentence has definitely resonated to see you it's something like
greatness or being the very best, and you know there's
millions of soft engineers in the world. Not everyone can
be the very best. Not everyone can be you know,
these star singers or star sports and athletes and so on.
But nevertheless, you know, look, look look at other disciplines
like sports. You might not become, you know, the next
world champion of whatever sports, but you can look at them,
you learn from them, and you kind of strive to
do better even if you know you're never getting there.
You know, you might have your lab times on money
and then you just might want to be ten seconds
faster in half a year and you're proud of that
and you're striving for this incremental improvement. So anyway, so excellence,
I think it obviously depends on where you're coming from,
but generally it is about the relentless drive to become
to envision that greatness and embrace it and make iterations
towards that. And I think it's a little bit harder
to find what does this mean in a job that's
not so you know, you don't get so many medals
by being the best. You don't get you know, you're
not this key champion who is the fastest. You're not
the golf of who uses the least amount of things.
You're not now in the you know, Couper America and
the European World CHAMPIONIP, there's just one team winning it.
But nevertheless, those who are already out of the race,
they they're part of the excellent strive as well, So
let's not forget that. So it's definitely very inclusive of
a lot of people here. But nevertheless, I think all
of those who are on that playing field, they are
they're just like doing one more thing than anyone else.
So anyway, so how does that translate a little bit
into engineering? And I think one of the interesting talks
probably that that many might know is Lenis Tobo's defining
good taste software. And he did that in a TED
talk something like ten years ago and showed like, hey,
this is how computer science teaches you to remove an
item of a single lecklist or something like that, and
it had this if condition in it. He must not
lessly water by what he is taught by computer science,
But why these if conditions. It's just like there might
be a bug in this thing, and there's another if
condition added, and it just adds this complexity versus really
trying three layers deep of what's the thing, how can
you solve it? And how can you solve it in
a way that is just like, that's the thing, what
it does. There's no discussions about it. There's basically no
way of an alternative pass if the first one and
the last one or the middle one and so on,
Like all of a sudden, you get into no, that's
the thing, no matter which item in the in this
list it is. And I think that one is an
interesting one in writing code and and I think now
these days is a lot of that is a lot
of us are working in the cloud or in some
other form of you know, computing environment. And I think
one of the interesting phase one I want to put
out here, and it's well, let let's let's start go first,
one step back. I think there are are two different things.
One is kind of on agreed baseline, and you can
set that in your company or for yourself. You can
set that at whatever level you actually want. Hopefully that
level of that baseline increases over time. But I think
those are you know, you can look at classical engineering
KPIs how quick are you to recover from a failure
in production? How quick are you able to in what's
the velocity in the team, what's the error rate? You know,
there's the Dora metrics out there, there's a bunch of
other ways to measure that. Whether you really measure that
or just like gut feeling type of measurement almost doesn't matter.
But like, that's the baseline security Laaren. You had an
example at the beginning, like you know, okay, how do
we you know, instead of increasing your password rotations, can
we actually decrease the amount of passwords we have in
our systems? Right, It's a different type of approach. How
you can think of that, And that's I think there
is I don't think there is a standard in the world,
but there's agreement of those argued things, right, And I
think then there's this other part which is not this baseline.
It's it's very drives innovation. They drive to try out
something new, and trying out something new in your environment
might be I've never worked in a cloud, but let's
run the first service in the cloud. Even though that's
a ten plus year old saying it might be oh,
this is how we do things here on running a
specifically workflow, and let's try and alternative way. You're not
getting into childscript front and frameworks because you try another
one every other week. I'm still ten years ago. Who still?
Speaker 4: I mean, you be careful.
Speaker 5: You gotta be careful there because I'm sure someone's still
using like ember or knockout or something like that.
Speaker 4: That's that's still around.
Speaker 5: I mean, there's an interesting question here of of how
do you even measure an engineer to be better and
whether or not the ext whence happens at an individual
level or at a team level. You know, I know
that I am a terrible individual software engineer, and I
shut her to think that someone will be evaluating me
on like the number of lines of code that I
turn out, because I know that number is incredibly small.
Speaker 3: Yeah, and even if that's the right metric.
Speaker 1: So I think that's one of the things that I
think about when I think about engineering excellence is defining
how we're going to define excellent because I think it
varies from team to team. You know, each team is
going to consider different things important, and you can't have
everything important. So I think everyone just has to have
that conversation, you know, to say, these are the things
that we're going to consider important. And you know, maybe
that's going to change over time, but this is where
we're going to start, and everyone's on the same page.
Speaker 2: There.
Speaker 5: There is this underlying assumption though, that it's somehow not
directly related to the business, right because you know, otherwise
you would say somehow engineering excellence means delivering the right
business outcome or I mean maybe there's some business.
Speaker 4: Metrics already, you know, why not use that term?
Speaker 5: So I don't know, at least for me, it does
sort of spark this idea of what are we innovating,
what are we what are we improving.
Speaker 4: In some way?
Speaker 5: What is that really? You know, what is the meaningful outcome?
And I think the linus forbodes coding good taste is
an interesting aspect here because how you write that function?
Speaker 4: How does it impact the business? Does it?
Speaker 5: I mean maybe it does, maybe it doesn't. So what
are we even evaluating there and how do we even
do that?
Speaker 2: Absolutely? And I think you know, going to the invo
versus team, in the end, it counts what the team delivers, right,
it's the same. But you have a team sports, you
can have the start plays and one star player often
can have a pretty bad team because everyone cages around
that star player and typically doesn't have much of a
chance against the team that's just very cohesive, especially I
mean catering for the start player as a cohesive team
that's very strong. But if they are fights behind the
scenes doesn't work anyway. So that sets back to the
sports andology. But I think it's it's really what is
the team coming out in terms of and I think
will do To your point, I think there's a few
things that are true almost no matter where you are.
I think you don't necessarily want to ship bugs to production, right,
so less bugs and production is probably a metric that
everyone agrease.
Speaker 5: Controversial thought there. I don't want bugs of production. Tho's
got to disagree with you somewhere, Marcus, I'm sure.
Speaker 2: And to be honest, I think what is a challenge
if you're and think of my role, I mean this
is this the print is a sizeable business, and you
know order management. You don't necessarily want to give away
or just for free because your payment didn't work, or
you want to block things that check out. So it's
a serious thing. You need to move very slow at
some point. And for example, for mine team, it is
like an interesting thing. Oh this is a new thing.
Don't work like you work in your usual environment. Just
like have a hecka on type of experience, or have
one person go off on basically an island for we
can come back and have this zero to eighty percent
solution out there. You're not going to break anything in code,
you're not going to make any customers unhappy. But it
allows you to move fast. So basically a little bit
of an unconventional approach. So yeah, shipping bugs into production,
it might actually there might be some use cases around
you know, prototypes, early things. Then you haven't figured out
the exact park market fit. That's it. I still argue
that you always scale down on features and have the
right quality because otherwise, when you ship something of these
little features that are all buggy, you're not going to
make product market fit.
Speaker 5: So you're as a director of engineering, you've got like
four or five ten teams reporting to you. I'm not
sure that number is totally relevant. So when you're saying
sharing excellence, like you're actually putting these expectations on your
individual teams too.
Speaker 4: Is it deliver faster?
Speaker 5: Is it to deliver more of the right thing related
to the business, Like, how are you actually evaluating your
teams on this? Is it a metric you're even using
to evaluate or is it just is it some idea?
Speaker 4: What is that for you?
Speaker 2: I'm definitely the person who is more in the idea
than I mean, I'm very much data driven in terms
of wants to figure something out. I want to want
to chase that metric, and obviously, you know, I think
reasons because like chasing one metric, you're ignoring the other ones,
so you always need to counterbalance. But you know, there's
this this interesting discussions of not sure if I want
to go there, but like if you're doing the second
or third like even when you run AWDs elastic kumunated service,
but then you need the kuminat is out of lifetime, right,
they need to move to the next version. You can
close your eyes and hit that upgrade button in place
upgrade and hope for the best and you might be lucky.
But it's a little bit like I don't feel confident.
You know, there's there's order is coming in every other second, right,
I mean, this is is not something what you want
to play with. And once you have start having those discussions,
like there's other solutions out in the world where you're
not even in that problem, those other solutions might or
might not be a fit for the team. Right. It's
not saying we should go elsewhere, but we're definitely having
active discussions around that because we're our customers are not
not buying a comment adies upgrade from us. You're buying
business cards, bottles, t shirts, all those type of things
in our case or in other cases might be different things.
And and so it needs to come with this reality
check on you know what's I mean, what's this total
cost of ownership? Is the old style version of saying
it probably or very mechanical way and then it come
up with your business cases. But I think this is
things come up constantly, and that could be this service
is constantly making problems and creating noise. You want to
dive deep. Is it engineering excellence? It could also be
that other partners in the business are you know, not
aware of what they're causing. Like you know, think of
some configuration elsewhere three layers down in the system and
you have no idea, or they might be disconnected from
what actually that is causing. It might call cause customer
complaints and it's very very indirect and very very slow
feedback loop. And so sometimes it's not you know, the
problem might not be technical, it might be a process problem.
But generally this is also part of excellence, but they'll
be getting a little bit further into excellence of a team,
excellence of a pride team, excellence of an organization. And
I think, you know, yeah, I'm definitely more qualified to
talk about engineering excellence.
Speaker 4: I mean, I like the Kobrinties example.
Speaker 5: So if we say bad engineering excellence is just closing
your eyes and clicking the upgrade button, or maybe closing
your eyes and just letting a WUS charges more money
because it's on the extended lifetime support strategy, what what
is good engineering excellence in that in that situation? Like
what would you expect there on that spectrum?
Speaker 2: Generally, I think, you know, maintenance should be low or
reasonable for the value you create as a business or
as a team. And you know, what's the value what
you create? Sometimes it is more tied to a revenue
stream and other times it's more tied to something a
little bit that decided. You know, it doesn't necessarily be
a dollar metric, but it needs to you know, not
not sure what would other companies and teams generally have
in their mind, but I would argue, you know, thirty
percent is stuff that engineers need to do to maintain
their status core and then you know there's this admin overhead,
but then the majority of it should be to move
the product forward, to move the projects forward. That comes
with coding, that comes up with his non coding activities,
and really doing good for the customer. You want to
increase that share of the you know, the maintenance activity.
So maintenance comes with there certain you know, in my
environment's a little bit of large companies, certain things that
are must do by a given date, certain upgrades, certain
compliance aspects and so on, and those you know, those
covering you know, ten inch percent, and they might come
up with some spikes and some lows and so on,
and the rest is I'm interested in the team rather
spending this thirty percent time in figuring out a new
service or new workload version to run, or a new
technique or documentation other thing versus. Yeah, this is what
we do. Twice a year. We just do it right
and think, think of something simpler than thencuminated is password rotation.
Probably want to get into rhythm or rotating your password.
Initially you pro might say, yeah, it's just three, let's
do it manually. And at some point you have fifty
and some point like, oh, this is a problem, right,
and sure give this too. Junior person to onboard to
launder systems and they go into fifty places and update those.
At some point someone comes in and does some automation
around this, and at some point you think, why do
I even have those passwords? So I'm so there is
very Yes, I'm going a little bit by feeling it
by noise, but then I'm fortunately have to have access
to a lot of data of thinking how many what's
the average password age, what's the rotation of it, what's
it over the timeline, and the timeline is very strong,
like if it goes up and you actually want to
go it down or the other way around, like you
can still acting before it explodes. And I think that's
a really really important thing. So thinking of you know,
this cuminating thing, it's not the worst thing what I
have in my team by far not. It's actually serving
a lot of value work. Those were things right from
a real engineering perspective, right, But nevertheless, it is distractions
and you're having you know, fifty plus seventy plus services
and each of that needs kind of a redeployment to
a new cluster if you want to move it over.
And then like even if you bring down the hour,
like let's just assume one hour per service, that's fifty
seventy hours per year, which is a week, giving you
thirty plus engineers working on the whole thing for the
year and doing that once or twice a week, it's
an okay thing. But not having that in the first place,
that's an even better thing. With thirty hours, you can
do quite a bit of other things that might actually
drive forward. So that is, for example, we have not
yet decided whether we want to get off of it
or not. But what I'm excited about the team now
looks at should we do something else, And very specifically,
a is ECS is a very you know, natural follow
up in staying on a WS. But you know, different
cloud providers will have very similar real cloud and options
and more managed versus less managed services.
Speaker 5: Maybe this is an unfair interpretation of engineering excellence, and
maybe will we'll give me a thumbs up here or
down on how oft I am. I think maybe another
way of looking at this is challenging the status quo
in a way which may identify that what we have
could be categorized as tech debt. And I don't like
the word tech debt a lot, but maybe what you're
saying is, you know, what do we have right now?
Is it actually the systems that we want in play?
Actually the technology we want to be using? The stack,
the programming language frameworks, you know, are they actually the
thing that we want to be using or should it
be something else? And I think the point you made,
Marcus is that none of these things have anything to
do with delivering the business value on a short term timescale.
But there's like a long term implication here and an
expectation that the teams are asking themselves, Hey, you know
this technology you're using, should you even have to rotate passwords?
You know, this thing that you're doing constantly that you
may be baked into tradition doesn't necessarily make sense.
Speaker 4: I don't know.
Speaker 2: I don't know if I'm right there absolutely, And I
think this is I think challenge status crul of what
you're doing constantly. I mean, look look at look at
the world what it was ten years ago, and what
services did your favorite cloud provider have out there? I mean,
by now, you're just exploding in services. And sometimes it's
just like kind of a fake service on top of
three of their other services. But all services are also
getting better, right, so I think that is and they
find their very own product market fit. And just look
at something like S three what you were able to
do ten years ago versus now, how much of an
important pillar that is, for example for data lakes and
other things like who would have put the data and
do sequel statements on top of a three bucket? Nobody?
But today that that that's how the big data warehouses
actually make money, right and and like this this is
how technology was look at you know, we may chat
jokes about front and frameworks, look at where they where
they came from, and what are they doing today and
like including things like you know those server side versions
of it on how you can ventor things, and and
I think it's it's we should challenge basically every day,
what are we doing is this really necessarily and think
of how could we do things differently at the same time,
I think there is a beauty of not classifying something
just because it's old as tech dev but saying, look
at you're milking the cout here right, You haven't invested much,
but it still works and and I think there might
there might be a servileized function out there. Yes, you
might need to upgrade I don't know the library version
once a year, but it still does this thing. It's
not massive tech there. Would you would you design the
system exactly like that from scratch? Potentially not because the
world has changed, but it's still good enough and not
worth changing. Like you know, identifying those tipping points when
is it actually a value or an asset and one
when does it start to become a liability is an
interesting thing to think of. But I would not classify
any programming language or any past decisions as lessli tech there.
But you know, when you think of where's the world
going in terms of you know, relational databases were that thing,
and yeah, ten years ago it was still their thing.
No sequels started to be other things. But these days
when you hire someone off the university, they don't know
how to write sequel statements anymore because that's not being
taught in school they only learn no sequels and blog
storage and all those things. I think it's the right
thing to teach to them. But knowing sequel it's actually
good skill to have. So, you know, does my team
have a bunch of relational database yes? Is it worthwhile
changing that? I don't think so it is it worthwhile
maintaining it to something? Very say? No, you are satisfied.
Is the cost, performance, backup failures, all the whole criteria. Yes, absolutely,
we need to tame the beast if you want right,
but not all workloads require things on you know this
horizontal scalability and other effects. What you can actually get
out of no sequel databases. So that doesn't mean just
because you have this and you need to get off
of it, but you need to identify at some point
it might be, oh, now it's the right time to move.
It might come as a project, it might come with.
Speaker 5: Something I'm wondering about, the who who who spends the
effort doing this? And I think maybe this is where
maybe the sort of thing that drove forward the mindset
in the first place, the movement towards DevOps uh, the
idea that more attention needs to be on evaluating the
excellence of our of the engineering that we're doing, and
maybe there's a cultural component to it that not everyone
is has the sort of mindset to look at what
we're doing today and actually improve it or fundamentally having
to change what we're doing. Uh that that can be
scary for lots of people as well. So I feel
like there's an aspect there which could have driven the
DevOps mindset, which was maybe a different team is responsible
for infrastructure, and maybe the sort of product teams say, hey,
other teams in our company are fundamentally responsible for doing
this innovation, deciding which version of Kubernetes, or whether or
not we're using Cerblus functions, et cetera. Ah Is is
there an aspect where every team should be doing something
in their own domain or do we think we've moved
the industry has moved in a direction where a lot
of these decisions are we've already identified which team is
fundamentally responsible for making them.
Speaker 2: Well, I think I think it's at the engineer's hard
to actually make improvements in their own realm, right. I
think there's companies that say, heybe have a central team
that you know, manageerain things like but we provide a
link to a get up repository and magic happens, right,
And this is this is the script what you need
to put there, and here's some conflict what you can
you know, how it scales. That's a very valuable approach
to do it. And I think the approach today will
also be different than the approaches were done you know
ten years ago. Is basically sis ops and database ops
and managing deployments manually and sailors and you know, local
local things versus cloud cloud versions and son uh and
there is that. The way is your your ability to
run it as a basically as a Pride team or
as a small development team of you know, two pizza
teams type of size, right the Tannish people in total ah,
But no matter what model you are, you have different
responses with your task is really going into these whole
depthops from from you know, prop typing to delivering testing
to also running it production including your you know, on
call schedule, or it's a little bit more you only
you know you depending there might be some handover for
for monitoring. But in both ways you have areas where
I wanted to improve it could be on you should
definitely be interested, like when we look at very technical
metrics error rate, response rate, latencies, certain things. You might
have a little bit more leverage in one model versus
the other, but you should be interesting to understanding your craft,
which is in the end it is could be two ways, right,
One is really just writing the code and writing that
in a way that scales in terms of features, but
also in terms of performance and probably many many other angles,
right and you know, not testability and all sorts of things.
So that that's definitely want to but there is I
think when we go a little bit you a build it,
you run it model, you also have this you architect
potentially differently because running workloads in the cloud, especially when
you go into a little bit more of these ACYNC
processing type of things. Right, do you have an event
and something can happen in the back background, So think
of an order being taken in into the systems. A
bunch of things that are happening, but the customers don't
need to wait for that. They you don't need to
wait until this thing get shipped out while the browser
spends for two weeks. That's not really really right. So
there is a lot of these aspects, but like, how
would you architect something from scratch given the latest accourages
that the cloud providers provide. That requires constant challenging because
like five plus years ago it was SNS and SQS
on Amazon, but now with a vent bridge, it's just
like so many more features. And then how do you
do that between teams? Do you still go through either
arrest or graph Well or so API or do you
actually hook up some teams internally through some of those
you know, cloud native event systems. There's many ways to
get there, and like always can lead to excellence in
the very context. But I think just saying yeah, we
made this decision five plus years ago is yes, but
you're missing out on you can delete these three services
then just moving data around, which is solved by this
configuration over here in the cloud provider now. So I
think if you're not doing these type of things and
then assuming cloud deployment now and not any other type
of deployments, like yes, it keeps your team busy, it
keeps potentially team efficient. They might even have engineering excellence
within that code, but not owning that code at all,
but moving that to configuration and let Amazon who is
or Google cloud or asia scale that for you and
manage that for you. That's really where scalability and excellence
also it definitely comes in.
Speaker 5: You said something really interesting about the amount of time
a team should be dedicated to engineering excellence.
Speaker 4: I think you just sort of dropped it in there.
That's thirty percent number.
Speaker 5: And I'm curious, you know, as another perspective here, maybe
will you've got one like how much time do you
do teams feel like in and around you, how much
time they actually have to invest in this? Because I know,
even at smaller company scale or larger ones, I know
there's like some expectations that go around, but then you
have maybe some product managers that come in and say, no,
we have to just deliver.
Speaker 4: You know, how do you do do do.
Speaker 5: Teams actually get that time to do this or is
it about cutting out that time and actually saying no, right,
you know, even though we have all this feature work
to do, we still need to spend this amount of
time really looking at what we've got in questioning our
status quo, like our engineer is actually being given that
capability or is this just you know a nice thing
to say and doesn't happen in practice.
Speaker 2: Yeah, it definitely happens in practice for my environment. It's
more or less depending on teams. And I think it's
you know, to response to my direct responsibility to enable
that in the team. And you know you're working with
you know, alongside product and alongside ux, so it's it's
it's my thing to defend that time, right Otherwise, I mean,
it's just one part of my job. But you know,
the company also values engineering practices a lot, so there
is a bunch of that filter filmed by already central things, right,
mandatory upgrades, mandatory things. Also, what we're doing as part
of quarterly planning is you're showing, Okay, how much do
you spend on things that someone else tells you to
do on the feature side, things that you actually want
to do for your customer because you understood that problem
really trying to do that, and then things basically in
this whole engineering excellence it's on even so I think
that it's probably the wrong world we want to call that,
you know, this maintenance moving forward. So I think there
is this excellent part which I think isn't is not
necessarily reported out there, but I think that is something
as the engineering leaders they are definitely responsible to get
that into the teams and the mindset of the teams
and the inn wilds right every individual there's forty hours
a week. Sometimes it can be someone is passionate about
the following block podcast like this one, and it comes
out with a cool idea to actually start. Others are following,
you know, some news, and others are actually interested in
reading the documentation on the cloud provider or SaaS provided
whatever and saying hey, there's a new technique, let's try
it out. I encourage really everyone to do that. That's
way harder to do and get that into the mindset
of everyone versus the otis is a lot of more,
you know, leadership driven and saying hey, by the way,
we need to do this for the long term health.
And it comes a little bit as yeah, the expectation
is that you're looking to that promise space.
Speaker 1: I think that there's a lot of so let me
say this, if your definition of excellence is defined correctly,
I think a lot of the time and effort it
takes to achieve excellence just happens natively. And I use
an example of like a sports team. You know, every
member of a sports team has a different definition of excellence,
but each each team member's definition of excellence supports the
definition of excellence above that. So if you start it
like the franchise owner, their definition of excellence is to
sell as many tickets to their games as possible.
Speaker 3: But then you get down to the coach, and the coach's.
Speaker 1: Definition of excellence is to hire the best quality of
players for the team that creates a well performing team
that sells the most number of tickets possible. And when
you get to an individual team player, that person's job
maybe to you know, catch the ball and move the
ball down field, or to defend against the ball being moved,
or whatever their specific role is for that sport so
that their team performs better, so that they have the
support of the fans, so that they sell more tickets
to the games. And so each one of those people
have has a different definition of excellence, but they all
support each other going all the way up. And then
whenever you get down to that individual team player, they
know that they have to catch the ball or defend
the ball or whatever, and that's their metric that they
measure against. And so it's easy for them to make
decisions that either support or don't support that definition of
excellence because it's clearly defined with what they do and
how that supports the overall organization.
Speaker 5: I mean, I feel like you too, just keep picking
up the football analogy and just the ticket in me
because both my teams have been kicked out of their
associated competitions. Right.
Speaker 1: Well, I'll be honest, I'm a Dallas Cowboys fan for
American football, So I have no idea how to define
excellence because clearly I don't understan and the game.
Speaker 2: So I think where you were going to is to me,
it's a little bit the business metrics, right Basically, the
KPIs that are are very outcome driven, and I think
the excellent part in that relation to sports will be
how how do people show up day to day on
that training, not leastly on the match, but almost more
on the training. How do they challenge each other to
become better? What's the unique training plan that they have
that is or it could be also you know they
bring into college and say, by the way, now we
know what the competing team does, so we can actually
outperform them through through the brain instead of through the body. Basically,
so I think that is that to me, their verre
it starts to be really, yes, everyone does that. Everyone
wants to sell more tickets, Everyone wants to have the
best coach, everyone wants to have the best players and
basically get the best performance out of the value invested.
But I think this the excellence is not achieved by that.
That's really achieving it by management, by you know, there
is a lot of value and that's needed in a corporation.
That's needed in the corporational e sports and a corporation,
you know, like this brand, but this excellence around like
how can I how can it be innovative in my training?
How can I be innovative in my nutrition? How can
I be innovative in in the tactics during the game.
That's I think where versus the next level? And and
also you can follow schoolbooks, you need to come up
with your own approaches and more un convention or buttn't
you're not necessarily your own, but at least you need
to read through those unconventional approaches, try them out, and
adopt them to your needs. So that that's a little
bit this type of how I would distinguish that where
it's engineering excellence, This is I think I was conflating
those a little bit before as I went to do
the other topic. But I think there's very good like
to draw a line on their excellent starts and they're
you know, engineering practices or you know, other things are
there to just like establish the baseline.
Speaker 5: I think you really touched on an interesting aspect here
because I do remember reading a paper about how in
order to actually get better, we need to do that
thing called practice. And if we only do the execute
production thing, you know, show up to the games all
the time, we won't get.
Speaker 4: That practice then.
Speaker 5: And the practice is where we have the opportunity to
really change up what we're doing or learn more.
Speaker 4: And I do see the industry moving sort of at
the moment more away from that and forgetting that we
need to do the practice.
Speaker 5: Maybe the engineering excellence really is what do you do
during your software development your engineering practice sessions?
Speaker 4: You know, what are you doing there? How are you evaluating?
Speaker 5: And there was a bunch of years ago there was
actually a documentary soccer documentary that was recording neymar and
the Brazilian soccer player, and they used.
Speaker 4: Brain diodes to.
Speaker 5: Identify how much he's thinking during a game versus practice.
And the interesting thing was that the actual amount that
you're thinking during production is very low, like almost no
brain activity at all when measured, whereas during practice there's
a lot of thought that goes into it into that practice.
Speaker 4: And I don't know if there's a direct.
Speaker 5: Corollary here, but I mean it really made me think
of this that maybe the engineering excellence is challenging how
we're practicing day to day so that we're ready to
do the actual production work.
Speaker 4: And what does that even look like in the software
engineering domain.
Speaker 2: I don't know that. I think you have the production right, yeah,
but it's like the deploduct production button is such a thing.
Can you do that? Calm me? Or can you only
do that you know under stress? Right? And that the
practice in.
Speaker 4: This case, I mean, that's always stressful for.
Speaker 2: Me, testing the automation, the error detection, the roullterback you have,
but the actual the result is that you have come
when you pushed it button.
Speaker 1: Yeah, that's really that's a that's got that's got me
thinking a lot Warren where it feels like in some circumstances,
like our job, every day we're showing up to game
day and there is no training time.
Speaker 4: Doesn't it.
Speaker 1: Yeah, So it's like, how and I'm thinking about that
twofold like one, Yeah, it would be super cool to
have training time too, just like at a very fundamental level,
go do a test to compare E C S T
e KS.
Speaker 2: And like.
Speaker 1: Get enough experience to have an informed opinion as to
war or not. We should be making that migration between
the two. But from a business perspective. From a business perspective,
you know, do I want to fund my business in
such a way so that my employees have time to
go and do that stuff, or do I want them
gain day every day? Like if I'm just purely focused
on the numbers of my business, I want them I
want every day to be game day, regardless of the
implications on them for that. So there's like a there's
a trade off there that I can't quite wrap my
head around at the moment.
Speaker 5: There was this joke, I don't know how based in reality.
It was about a manager talking to the I don't
know head of engineering saying, hey, you know, our engineers
need to be trained better.
Speaker 4: You know that takes a lot of money and effort
and time. You know, can we just utilize them? What
if they're not good enough?
Speaker 3: Right?
Speaker 4: You know what I mean, No one's perfect.
Speaker 5: As we pointed out, they don't maximize the number of
lines of code or business value, and so we do
need to train them.
Speaker 4: And the concern is what if we train them and
they leave?
Speaker 5: And the manager's response is what if we don't train
them and they stay? Right, Marcus, you're gonna say.
Speaker 2: Something absolutely, But I think, I mean, training is a
hard one to really make. You know, there's some people
who are just like embracing it, uh, and others it's
just like hard. And also I see it in my
daily schedule. It's really very hard to get it in.
I mean, what works well for me is going running
over lunch and listening into a podcast. That's one form
of media consumption that works. And it's not just like
the doom scrawling type of value, but it's actually you know,
building up some some more thoughts and it's also have
to space time there. But I think, you know, in
terms of where you you might have been build is
I think there is a space where it's just like
there's constantly noise around a team. That's when you will
potentially even management will come in and they typically send
That's when the consultants show up, and when management shows
up and you know, weird reporting and someone looks at
how many metro crasts did they do yesterday, that's a
really bad sign. That's when you have underinvested. I think
there I'm talking to is already like, yes, you're bringing
up the point. Does is it valuable to do this
or not? Right? So we tried to sneak in those
type of things at the side a little bit. I
think there's certain things that are definitely of high value
of you know, think of not having passwords or it's
just like way simpler to manage, and it's also possible today.
Five plus years ago it was very very hard to
do and these days the systems have evolve. So I
think this is again like adjusting to huries the world
today and you know, the world has just innovated AI
in the last I don't know, it feels like cloud
three days and there's this whole whole hype cycle. Yeah,
the world is moving fairly fast, right, or you know
the cloud didn't exist ten years ago, that's only a
decade well, or you know, depending on what's the starting
point of the cloud, that's probably IBM saying the cloud
exists for the last forty years and their own world
did this different definition of that. But it's really really
moving fast, right and and not keeping up with that
is I think it gets you into a situation where
at some point a company might say, we need to
stop and we need a year to overhaul our accounting infrastructure.
And that's a really really hard job to do from
a team perspective, from a moral perspective, from actually can
you achieve that outcome? And you know, how can you
keep the customers happy? And so I think this is
the balance of that. Nevertheless, I think after that, even
if you don't, if you say, okay, scale down on
this bucket of making things better for the sake of
making it better is I think as an engineer, you
just want to have that. You want to have this drive,
you want to show off. And I think this is
also the opportunities that in the worlds have in corporations,
no matter what role they have and what level it is,
to showing off and saying I understand this technique, here's
a documentation, and you know it might become the de
factors standard how things are being done in the company
because it comes with the context of the company. It
comes with that. This is how we do deployments, this
is how we run things, this is the responsibility. These
are in the incentives. And if someone is able to
document it in a way or to create a create
some shared code that's easy to reuse and so on,
I think that's really really powerful and allows also the
junior engineers to shine. They might they might actually the
biggest druggle for them might be to put it out
into the world. Especially when it's an internal world. It's
almost harder and putting out there in the world and
nobody might see it, that's easy. But putting it out
internally and you have an intelligent purpose for that, it's
like you need you need some cheerleading from you know,
the basically the directors of engineering or you know, your
team lead to others to actually put it into a
shape that is not going to annoy the others, but
it's actually being appreciated. And that is really company and
political context that you need to fully understand to make
sure that you can do that. But it's I think
it's a big opportunity for everyone, and especially for more
junior or just like very technical savvy person who don't
want to deal with that, but actually would be interesting
to have that better excellence across all the teams and
not just for themselves.
Speaker 1: So is that part of your definition of engineering excellence
is promoting promoting internal things within the teams and supporting
them and like mandating that feedback loop to encourage excellence.
Speaker 2: Not necessarily mandating, but this is what I encourage you
to do this damage. Yeah, right, yeah, I think I
think I ut put that in and here we go.
Letten to the excellence part to go more into how
do you manage the people and how do you grow
the people? Is I want to encourage everyone on the
opportunities are endless, and yes, day to day this hard
first to identify the right thing for the right person
and so on, but like just simply the topics that
are on my plate. It's around security, it's about infrastructure.
It's about front and parts, it's about back in parts.
It's about you know, potentially libraries, it's about upgrades, about
deployment techniques. It's about challenging how we do run things
in the cloud. It might be a SaaS provided you're
using and oh be using that suboptimal and so on.
It is endless, and I think as you're an engineer.
You are seeing these things and I'm here, I'm not
even talking about the bugs you're seeing in your own
system and the and the error reports and other things. Right,
this is another part where you hopefully, hopefully that's your
first home. We'll just try make sure that the customers
are happy. Right, That's that's the first part where you
want to use the the data in terms of oh,
we had forty seven errors in this timeframe, we should
probably bring it down or no, this is really kind
of the maximum. How we like we just decided in
the team with with you know, senior input to know
this this this is good. Right, this is really world
class where you can be. But then all the other
topics it is just as you and you will be
welcome because a lot of those topics might not be
so passionately done by many much best female as an
admin task or I do the minimum versus those are
really unbound and you can have an impact that is
way across teams, way outside of your sphere of directly
influencing day to day. So it's both in the day
to day and how do you make yourself and your
team better and your softly you're directly supporting, as well
as the opportunity out there of if you do things
slightly more generic, share it slightly further, or you know,
raise your arm on you know, becoming you know, the
security expert in your team, and you'll learn all that,
and then you start looking at the data yourself and
to start crashing why are you doing it this way?
Start improving it into your own team, then starting sharing, Hey,
this is what it is. Those are the three steps.
It's that easy, Like, oh, once it's easy, others will
also do it. Versus this neverless thing of you know,
we might need to invest some technical spike time at
some point and it's in the backlog or the back
of our minds of people and it's never getting to
versus simplifying it, paving the path so others can just
walk it, and then hopefully if they're paved path, you're
not trying to reinvent the bill. You're just walking there
because you trust your colleagues. And the combination of those
also like appreciating this is one way of doing it
might not be the perfect way, but it helps you accelerate.
So it's both ways. It's not only sharing, it's also
taking what's there.
Speaker 4: It's weird.
Speaker 5: I think that we put so much emphasis on needing
to train up sort of inexperienced or junior engineers, but
then when it comes to the teams as a whole
or more senior engineers, I think there's a tendency to
forget that there's an opportunity there to spend time actually
reviewing what we're doing and like how we're working or
how we're doing it effectively, even at an individual level.
And I'm not really sure where that comes from. Maybe
it's like once you reach senior engineer like that's it.
You know, companies like that's it. You know you're as
successful as you need to be for us, and junior
engineers as long as you learn our systems, you're as
successful as you need to be for us. But there
is still that learning aspect there, and I'm wondering if
it's enough to just say, spend take thirty percent of
your time to learn new things, to listen to the
Adventures in DevOps podcast, or you know, use new technologies
and new UI frameworks, or whether or not there's something
that we need to directly encourage to make that happen effectively.
Speaker 1: I think it has to be a much more active role,
because it's far too easy for individuals, especially junior individuals,
to see tickets in their backlog or work that's assigned
to them and prioritize that over learning and education. Even
whenever we say, hey, you should be spending time on
learning and education, I don't think you can take a
passive role in just telling them that and then assuming
that they're doing it. I think it's something that you
have to actively manage and track. Whether that's you know,
telling them, hey, open a ticket and log your time
when you're listening to the podcast or you know, or
whatever the metric is. I think it has to be
something that's that's active and measurable.
Speaker 5: I mean, they're not actually gonna know either, right Like
they're in a you're in a spot, your team is
in a spot where even if you said, hey, you know,
spend a lot of your time on this, like what
is the right activities that you should be doing or
the right knowledge you should be gaining?
Speaker 2: Uh?
Speaker 5: And maybe that's why we I think as an industry,
we fall back into like watch these YouTube videos or
go to some conferences as if that's you know, the
right way to just automatically make this happen. But I
feel like that maybe even a little bit too passive
as well. Right, you know, there's a lot of information
floating around that doesn't necessarily convert into actual con crete
learning that can be applied back into what we're doing
day to day.
Speaker 3: Yeah, for sure.
Speaker 1: I think from my perspective, like whenever I first started
going to conferences, I was like, oh my god, this
is going to be amazing.
Speaker 3: I'm going to learn so much.
Speaker 1: And now I look at conferences as a way to
completely screw up my inbox where I'll never get a
valid email again because of all the mailing lists I
get subscribed to by going to the conference.
Speaker 5: Minor LinkedIn connections are totally like I just got back
from Germany last week, and like, I don't know who
these people are that are connected with me.
Speaker 4: Like I try to, you know, approve all of them,
and I'm like, I don't remember you. I'm sorry.
Speaker 5: Sure I did learn something though, as a speaker, I
actually learned something really interesting, and that's it amazes me
how many companies have on premise data centers or they're
renting a data center from a third party provider where
they're pretty much getting bare metal that they're putting stuff on,
and they don't have a need for this, like they're
not at a.
Speaker 4: Level or scale or industry that requires it.
Speaker 5: It's really quite amazing that there's just still so many
in this field, and I wonder if it's a lack
of engineering excellence that has left them there.
Speaker 2: Is the thing, right, right?
Speaker 1: Yeah, I think there's been a resurgence in going to
the data center over the last few years, and I've
known quite a few people who are using the cost
of AWS and GCP as the driver for that, But
I'm not convenient completely convinced on that. For like, if
you know what your workload is and it's pretty consistent
and well defined, I think if you had a good
data center provider that could send someone out to swap
failed hard drives and replaced nick cards.
Speaker 3: And all that kind of stuff, it might work. But
I don't know. That's I mean a lot of faith
and there.
Speaker 4: I mean for sure.
Speaker 5: I mean you're in an interesting area though, Will, because
like I feel like you deal with a lot of
companies at Polagon that are in this weird domain that
aren't necessarily cloud cloudish public cloudish optimized, right I mean
consensus networks and cryptocurrency companies, et cetera.
Speaker 4: Like I get that, but there are just so many like.
Speaker 5: Companies out there that are startups or even larger companies
that are spending sixty million plus dollars euros francs a year.
Speaker 4: Who thinks there are a benefit there? Marcus, you we're
going to say something.
Speaker 2: Yeah, I think, I think you. It's a nice segue
to one of the thoughts I haven't. And I was
talking about, Hey, there's this baseline, but then comes to
innovation in it, and innovation is a lot of different things,
Like we talked about education, we talked about you know,
you know, figuring out and taking on additional responsibilitiness on
But one of those odds that have is basically, and
it comes exactly where it's, oh my god, cloud is
so Expensive's cost per compute is kind of a term
that I came up and made up myself, but it's like,
what's the cost you're spending for the compute you're using?
And you know, basically, even in the cloud, you run
things on a virtual machine. By definition, you don't want
this to be more than ten or twenty percent CPU,
and you want to have at least I don't know,
fifty percent memory free. So you're paying paying the money
what you should have in your pocket, and you do
your team's bonuses into the bonuses of the cloud providers. Right.
This is they're charging for CPU which they are actually
not using. It's all these shared hosting environments. So that's
their game, right, and that's their margin. And yet it
is relatively attractive. So I think this is also very
this innovation comes in. So how do you actually bring
I think if you optimize cost per compute, so not
purely cost, because purely cost of organization is just like
turn everything off and you're done right of the business.
You will never ever have costs again. And I think
the alternative if it's just like run everything on AWS,
LANDA or whatever your serverlized function is, might also not
be the answer. That also doesn't say you cannot use
relation database. You must use down B or Mango, the
B or whatever it might be. It comes way more nuanced,
but cenarly, if you look at your cloud spend, cloud
spend is very easy because you get your statistics over
time and everything. You can analyze and tag things and sounds,
so it's quite you can have this this engineering hard
and start with the data and be data driven. Just
trying to iterate there. But look at the top five.
They're probably consuming fifty eight percent of of your money. Right.
Some of that is justified because that's actually the workload
that's running there. Others such just like wait a bit,
you know it's waiting. It's it's mostly the margin on
over provisioning. This is you know, relation databases you need.
There's no way to scale up and scale down at
least not you know, there is a way hope. By
the way, we have a peak season for two months,
so we scale up for the next instant and scale
down again or increase the distance. So slow moving scalability
can do. But those even during the days are very
hard to do and if it's more spiky, almost impossible
to do. But you're paying for your for your risk.
And got feeling a little bit almost like I mean,
obviously hopefully you look at data, look at some requests
died some low testing and everything, but like, yeah, there
might be this one customer coming along and put some
load on it. It's like I want to be prepared
for that, right, and you need to be prepared as
a business for that, and you need to be over
provisioned versus the more you go to basically really pay
per use. If there's nobody coming, it's zero cost. If
someone is coming, when you pay what you use. This
again has at some point it has a limit, and
it might be do my own thing is cheaper, right,
So at some point, you know, running certain things on
a huge scale is a central team that's expertise there
might be the better thing for some companies or running
again back to bear Metal, I mean there's just a
little bit of a hype of all we run this
thing on bear Metal. There are those workloads, but they
typically don't come from a startup. They don't come from
you know, smallish two pizza sized teams running their own thing,
even if they have a lot of requests. Those recreess
are typically you know, request then do some processing, call
of database, return the thing. And and like I think
cost per compute is a nice proxy to say, do
I have a spot where my top five or top
three things? And that's just looking at the pure costs,
right there comes to operational costs and other things that
you need to encounter. But at least this is hard data.
What you can look at and trying to optimize for
that and some point I'm good, Now I feel like
the costs spent these lum and I think, you know
a lot of companies might not have done that or
might not have incentivized the team, and so oh my
cloud costs are so expensive. But then you know, really
running that very same workload, it's the very same properties.
People can just spin up things and other things. You
paint that on the rack. Now, I mean community serves
need to have in there to actually have for those spikes,
because like scaling up and down doesn't make sense anymore.
Just like put it on full load and that's what
you're what you're you're having in there.
Speaker 5: I think there's some inspiration from the web three space
that sort of coin the term gas fee, right, the
the actual cost of that execution compute to the line
of execution for per instruction. I don't I don't know
what that means, but I mean, I think there is
an interesting aspect here where when we deal with the
physical goods and we have a capital expenditure, you buy
a thing and then you depreciate the value over time
because the metal rusts or the components get out of
date or you know, whatever it is about those, however
it's manufactured or being consumed.
Speaker 4: We sort of lost that as we switch to from
capital or cap.
Speaker 5: X expenditures to operational expenditures, which I think we're rewarded
for having. Hey, you know, don't upfront by this, because
there's you know, it's going to depreciate over time. You know,
take a car for instance, you buy a car, it
gets worse over time, It's worth nothing after you drive
it off for the lot, and then you drive for
another ten years and then it rusts and it's gone right.
Whereas you know, if you're renting the car, you always
get the newest, latest version, and if you never drive anywhere,
then it's you don't pay anything, you know, conceivably. But
I feel like there is a middle ground here where
a lot of companies end up in and they forget that,
even things like easy Too or Kubernetes.
Speaker 4: There there is a capital expenditure.
Speaker 5: It's just not on the bar metal, it's not on
the virtual machine potentially, it's it's this other aspect. You know,
it's slightly higher. It's not all all backs or all
cap backs, but somewhere in between and because we've gotten
away from that. I feel like we've lost people paying
attention to how much we're actually expending on the part
that's not fully operational, that that's fully Like you said,
the total cost of ownership is one aspect to pull
on here. And I mean, I don't know if there's
a question here, just sort of an interesting tangent.
Speaker 4: I guess no.
Speaker 2: I think you know, just paying one thousand dollars every
month is yeah, relatively easy to understand and beta, wait,
this is what my business costs versus every three years
paying thirty thousand, fifty thousands, Like, wait a bit, I'm
not spending that money, right, there's projects undying on just
like trying to do that, right, that's just like the
small numbers now, but now, and I think you of
not using one thousand but using a million, right, A
million a month is one thing to pay, but then
have this fifty million dollar built every three four years
is a different type of discussion. And and I think
this is as you're optimizing the cost, you're not bringing
the cost down by ten dollars, one hundred dollars, one
thousand dollars, ten thousand dollars, whatever your workload might be
you're bringing that down multiplied by I just multiply the
to a year, right and look at the yearly cost, right,
because at some point, you know, things change so much
that it's very hard to for long over the long term.
I think it's probably basic finance here, but also it's
important that numbers team members understand, like, no, you're not
bringing down this workload. You know, fixing a bug for
one user. No, you're fixing a bug for one user
day which is tween a sixty five users a year,
or you buy I don't know whatever it is, and
it's like, no, this is This is a little bit
where capital investment in terms of software engineering comes in.
And you know, some of that is on a product side,
but I would argue some of that is on engineering
side of saying if we fix these three things, that's
the impact you have to the customers, to the business,
to the cost to whatever. And that's you know, if
you invest an hour, now like you can basically count
how many days or how many weeks it takes until
the return of investment is There are very simple things
and there's very more complicated models. But like get the
back of the Napkain calculation into everyone's head, like just like, yeah,
make those decisions based on that. And if you're really
good at this type of investing your time where there
is a return, and preferably return is on better customer
service and better customer value because that really feels your business, right,
So like old engineering excellence aside, but where you can
invest and you know, prove improved things from an engineering side.
And it has also either direct so sometimes it's indirection.
How do you measure downtime? Right? If the team was
down once and once a month is pretty bad, right, Uh,
all of a sudden it's not down anymore. It's probably
a good thing that happened. I don't know X months
in the past, but it probably didn't happen or the
work change incidentally, right, it probably happened because someone's yeah,
right the thing.
Speaker 3: Mm hmm.
Speaker 5: I mean, I you come for the engineering excellence and
get the uh you know x maybe startup entrepreneur here
giving you financial advice on how to run your cloud.
Speaker 4: I mean, I I love the topic. I feel I
fear it's a it's another whole hour, uh which you're
going off which no, I mean it's super interesting. I
mean I I I really love the idea.
Speaker 5: I actually, you know, I'm hoping maybe we can find
someone to come on to actually uh discuss that in detail,
because I think there's, as you pointed out, really interesting
overlap with uh, the complexity of managing cloud resources or
I'm sorry, if you're on prem.
Speaker 1: If they're on prem, they'd never heard the podcast because
they're crawling underneath the floor running network cables or something.
Actually I am curious about that's if that's still a thing,
like if you are running on prem, are you still
for people who are running on prem or they still
jumping in the car driving out to the data center
swapping hard drives, racking servers.
Speaker 4: I mean, I can imagine it's a separate job.
Speaker 5: And there are the third party bare metal providers out
there that are trying to sell that. So I think
something that the even larger cloud providers offer or how
they work is they end up renting like locked in
space inside physical buildings that.
Speaker 4: Are called cages that you can't get in.
Speaker 5: So who's doing that activity though, I'm that is a
good question, right, I mean, because you don't want the
data center owner to be accessing your physical machines and
you know, pulling out the HSM that you've got installed there,
like that's that's a security vulnerability. So you've got to
have your own operation engineers that you're employing to perform
that activity. Otherwise, you know, you might as well be
using a public cloud. But you know, if someone knows,
I think they should ping us on discord and actually,
you know, give us a topic of conversation here. I
think that'd be super interesting to have someone with an
expertise in on premise data center management.
Speaker 3: Oh absolutely, I'm one hundred percent on that one.
Speaker 2: I think that woo especially interesting for those that are
you know, medium company size, not those very big players.
They might have other economics and other reasons to do.
Speaker 4: That, yeah for sure.
Speaker 2: Also not those who have to do it for compliant reasons,
because they're because it's the thing and that's the reason
why we do it. But really, like you know, those
have a decent engineering team, but those are not super
so so I think a lot of people could actually
relate and so, so, yes, would be a fantastic topic.
Speaker 1: Yeah, I would like to have somebody from well, yeah,
because there's the email provider hey dot com that was
started by I don't remember what his full name is.
He goes by the initials Dha. She's like the founder
of Rubion Rails or something. But they have left the
cloud completely operate all their own physical hardware and then
just recently announced that they're ditching Kubernetes for their own
orchestration platform that they have created, and like they're just
like completely going off and inventing their own stuff. And
I think that would be a really cool conversation to
have and say, just you know, how's that working out
for you?
Speaker 2: Yeah? I think I think what's interesting is is you
know where there's the value of kuman it is or
these type of things, right, these big frameworks in these
solve I mean kum that came out of this Google project, right,
and they run a few more service than I do, right,
and and and I think it goes down a little
bit too often many companies do either Hey, they have
just like say, some CP needs, or they have classical
that break cress type of needs to the database, or
they have some aazing processing kind of these servilized pipeline
type of needs. I feel like if they were running
and I'm guessing here is krumen it is really in
the cloud in terms of how potential is a cloud
provide to sell it and the potentially the options that
were there. I think that happened a three years ago
when they announced it. It's different than oh, like you
go all in on the cloud and figure out how
can you you know, costper compute, how like cost per
compute we're not good for them? How can you squeeze
it out? I think there might have been a cloud
way to do it. And those are you know, top
notche engineers. I'm pretty sure they looked at a bunch
of things. But this is also one of the arguments
like why you know, I was you know, looking at
I don't want the krumanides upgrade a year, but if
it's there, it's not the worst thing, right, But there
are other things like kumunides. You know a lot of
people put the stamp on a LinkedIn profile because they
want to have the stamp. It's the wrong incentive, right,
and it's but it's also a huge complexity to learn.
I mean, learning cumanities is equally challenging us learning ITWS
and learning Google Cloud or whatever it is. Right, It's
a huge thing and really really mastering that and then
scaling it to your needs and getting the right cost
performance security profile. That's another undertaking versus most teams might
actually just like, yeah, we have the kubanities in the
cloud and fully managed and these are the seven settings
we're looking at. And if you're doing that, it's probably
like something like you know, a more cloud native saying
that it is more or higher higher managed service like
NAS has easy as it's just a cluster, has a name.
That's all you can configure, and then you'll say, oh,
here's this service, here's the doctor instance, and that's giving
me so much CPU and memory and then hopefully that
matches the wallet you have and the workload you're thrown against.
It has the lot of right balance, but it managed
this deal all the scaling for you, Like okay, you
need to configure that right, but it's like it does
all the magic and the nine thing about it. In
five years, if you never touch the code, okay, your
own code might run into security volunabarities, but the underlying
hardware has changed seven times, runs faster by then, has
been gone through innovation cycles by the cloud provider, and
while doing nothing, so your code gets more secure and
faster while doing nothing. This is a really really beautiful
thing to have then using managed services, and I think
that's something that's undervalued. And often like you know this
on ITA, that's not sure how much we can go
deep there versus out like there is you back it
by your own virtual machine and you're only easy to
instance versus by fargate, which is basically easy to fleet
managed by WS so you don't see them is looking
at a one to one comparison. Obviously back by easy
two is cheaper because you manage it, you have the responsibility.
But they also know there's so much free capacity there
what they can basically what they are charging you because
you have to manage the capacity versus you're really only
paying what you're needing. So again it's yet one level higher.
So depending on where you are on the managed services,
go one level higher and you get a bunch of
things for free on maintenance, security, upgrades, maintainability and like
again strive there, like what's your next player? Then after that?
What's your next player? After that? At some point I
think there are two stopping points. Some point Okay, I
don't understand this thing anymore because it's too abstract. There's
services out there, drop this thing, and we kind of
run it right, and you want to have a little
bit more control depending on your needs. Great for startups,
great for prototypes, great for the amoor wasn't great for
certain workloads, but not lessening everything. So that's one stopping point.
And I think the other stopping point is when you
don't it's not a managed service, but the manage service
manages you kind of think it is. At one point,
at some workflow characteristics, they might become expensive. A very
classical one is the NAT Gateway. Just put it up.
It's very cheap, but once you actually hits traffic at
some point you think, should have run my own, right,
And those are things that I you know, I'm there
going back to the Hay conversation, is my yeah, should
I run not their mental arcy, but should I run
my own here? And it's a slippery slope, right, I
mean it's from a cost performance cost professor, absolutely, it's
probably also not that hard to run it. But then
comes the whole security maintenance, and nobody will have a
clue of how this works because there you need to
go deeper into networking and to routing tables and all
the other things. What you need to understand. I don't
necessarily want to spend that time there, but I fully
understand companies that don't run this nap game because it's
just like hitting a limit where they want to spend
the money. Yeah. So it's just one example, right, So,
so I think even managed services they are not this
is the way to go. There is orders, what you
need to set and hate by they here's here's where
we stop or here. Excellence actually means not doing the
thing that sound.
Speaker 5: Yeah, no, I think that's a good a good Uh,
you know, circle back around to the beginning of the
conversation really, like, you know, figuring out what decisions you
want to make, uh, and even what the is to
make those decisions is.
Speaker 4: What we're calling excellent, our engineering excellence.
Speaker 3: I fear that.
Speaker 5: We may be taking up too much of your time,
and I have this thought that maybe we should move
on to picks.
Speaker 4: If there is no one last thing that you wanted
to share.
Speaker 2: Oh, I feel like I've talked so much. Thanks for
having me, so just go on. Yeah.
Speaker 1: I do want to chime in with one last thing
and say that I think that net Gateways is an
individual line item on Amazon profit and lost balance sheet.
Speaker 2: It is it is right, it's not storage. Storage is
not a one.
Speaker 1: Yeah, Like, we sold five hundred million coffeemakers last year
and it almost made us as much money as net gateways.
Speaker 2: Something like that. Oh my god, you could go into
that direction. But let's let's.
Speaker 3: Right cool, let's do picks. Warren to bring a pick
this week.
Speaker 2: I did.
Speaker 5: I know, usually it's a book, but I just bought
this thing. It's a Legion from Lenovo. It's a tablet.
I loved my Google Nexus for forever. I think I
had it like ten years or so. Never let me
down up until last year, where I just totally froze.
For the most part, I blame the apps that I
have to run, like they just take up more and
more memory and upgrades from Google. But this thing a
little pricey, but still, I mean it's it's been perfect.
It's been absolutely fantastic. Everything I want in a tablet
for going on vacation with.
Speaker 3: So what are the common things you do with your
tablet that make this a great tablet?
Speaker 2: Oh?
Speaker 5: Yeah, I mean I'm not a gamer, but it's a
gaming tablet, which means the battery life is incredibly great,
and so I get on like fourteen hour flights when
I'm going to conferences on the other side of the world,
and so I needed to last that long. So content
for uh, you know, watching movies, et cetera. I read
a lot of books. So storage space it has a
removable flash drive, making it really easy to transfer stuff
from my desktop. You know, those are really my core ones.
I mean, every once in a while I need a
game to actually entertain me on a fourteen plus hour flight,
which I don't recommend to anyone unless there's something really
good in it for you. So that that's really my
my core requirement there, which is why I've gone for
this instead of something with like E yanch or some
or on that approach.
Speaker 3: Right now, cool, all right, Marcus, would you bring for
a pick?
Speaker 2: I thought of bringing that podcast, like a little bit
of competition to you. I do a little bit of sports,
and it's one of the way off of actually getting
some information. But it's it's Farnham straight Shane Perrish, and
he interviews just like top business leaders, top sports and
coaches and other things. And I've gone through wellow one
hundred series of I think he's at a one hundred
and ninety podcast by now does it every week or
every second week. I think every second week, and I'm
following along almost everything, and I think it's it's just inspirational.
What you know, talk about engineering excellence. It is what
are other disciplines doing right? Sometimes it's someone in psychology
some other science feels investing sports everything and like just
gives you a different perspective learning from the best. What
they have already figured out is what his tagline is.
And I think I really really like that. And I
can only recommend if you are short of time. I
don't know how much I should go into recommend recommending
other people here, but it's clear. Thinking is the book
Shane Heart brought up half a year ago. It's summarizing
about the whole community what he has built up over
the years and the consistency definitely worth a read right on.
Speaker 3: Very cool cool.
Speaker 1: So my pick for the week is there's a song
that's actually an older song now from a band called Metallica.
The song is called one, And it feels so weird
to say that that's an old song now because that
just reminds me that I am old now. But there's
you know, there's a lot of people who haven't heard it,
And so there's a YouTube reaction video from.
Speaker 3: A YouTuber called the Vocalist. She's like a.
Speaker 1: Classically trained vocal instructor, and so she watches the music
video for Metallica's song one and reacts to it. And
it's cool because there's just a lot of layers into this.
The song one itself is about a movie from nineteen
seventy three called Johnny Got His Gun, about a soldier
who's injured in World War One, and.
Speaker 3: So the song is about that.
Speaker 1: And then so there's clips from the movie in the
music video. But then the orchestration of the song itself
is just really well done. And then her reaction to
that as she watches it and she's trying to comprehend
the movie that the song is about, plus the orchestration
and the instrumental work of the song, Like it's just
all these layers in one short video. That's super cool,
definitely worth checking out, whether you are a Metallica fan
or not. It's just there's entertainment there for people of
all types. So that is YouTuber The Vocalist and her
reaction of Metallica song one, and that's all we have
for the podcast.
Speaker 3: So for everyone listening to the podcasting, thank you for listening.
Speaker 1: For all the people watching the live streams on twitch,
YouTube or whatever other streaming platforms we're on, thank you
for watching. And Marcus, thanks for joining us. This has
been a cool conversation.
Speaker 2: Thanks a lot for having me. Was an honor and
was also a fun conversation. Definitely some challenging ideas and
thoughts right on.
Speaker 1: Warren, as always, thanks for joining me and co hosting
the show.
Speaker 3: We'll see everyone next week.