Okay, what's going on? Everybody? Welcome to another episode of Adventures in
DevOps. I'm your host, Will Button and back after attending it was the
DevOps conference in Zurich last week, right Warren? Yeah for sure. And
you know, I actually listened to a little bit of last episode and I
felt like you introduced me even though I wasn't there. So I feel like
I have a free you know, I don't need to introduce myself this episode.
I will say that I owe everyone an interesting fact. And this time
when we were at the conference, we specifically went out and hosted an open
space where I talked with a group of experienced develops practitioners and we were really
trying to define what a platform even is. I think this word is thrown
away around everywhere, and after an hour we really came to a great conclusion.
It's something horrioral and that's I think there's an aspect here where it doesn't
all doesn't offer value to the end user. I think is an important thing.
So one definition we had was everything other than the product, which is
interesting to think that the platforms we build aren't aren't value add but I,
you know, I there's some interesting and deep dive there. But yeah,
no, it was great. It was great. Confidence. I'm back.
I'm happy to be back though, right, I'm cool. Happy to have
you back. And then also joining us in the studio the co founder and
CEO of Kurtosis, Galen Marquetti. Galen, welcome, thank you, thank
you for having me, and hello everybody. Yeah, I'm excited to have
you here. You know, we've talked a little bit in the past,
and then we've also I've also been using critosis a little bit and it's very
basic level. But tell us just give us like the high level overview of
Kurtosis, and then we're going to dig into what led us to this point
in life. So the highest level overview is that Krotosis makes it easy to
get development environments and testing environments set up for back end systems. So I
think something like doctor composed or helm charts with more parameterizability, more flexibility,
and some things that give you greater portability guarantees. Gotcha. So when we're
talking about dev environments, as a developer working on a product, I need
to test the code that I'm writing, but that means I need to run
the application. But then I also need all the supporting micro services, and
then those are completely worthless without the data that gives me some kind of real
world or pseudo real world scenario to test against. Is that right? That's
about right, And I think a good way to think about the pain that
kind of motivated the whole thing is that part about running other people's code.
We're running your own code. Usually people have a handle on that one.
It's when you need six or seven other micro services plus the data. Now
you have to dig into how do I set these things up? And a
lot of the time that I get stuck at that phase for an in order
amount of time, and that's kind of the heart of what we're doing.
Yeah, for sure. I can think back of different scenarios where I've addressed
this in the past, and some of the approaches I've used are the you
know, like the seven thousand line Bash script. There's the golden machine or
the golden VM that no one's allowed to touch because it has everything on it.
Right at the moment, there's trying to bribe your security team to let
you connect to the staging environment across the firewall. So those are some of
the approaches I've used in the past. Yeah, it's exactly right. It's
exactly right, and it's what a lot of the times people get a good
system set up for kind of a golden path they do all the time,
but then there's this like long tail of things that you always do a long
tail thing maybe once a week, but it's a different long tail thing,
and it's that set of issues and security and you have to touch the golden
and VM for it, it break it, and it just adds up for
sure. Orren, what's what's your approach in solving this problem? Oh,
well, you know, I think we only have like an hour on this.
There's something here where if you have really high separation, low coupling between
your individual micro services, you can just spin up the one that you've got
on your machine. We use local stack for our infrastructure, but I think
that's a very extreme perspective and it takes a while to really get there,
and we focus on having that at the beginning. And if you come from
a monolith world or one where the architecture wasn't really figured out, there's this
sort of dead zone in the middle where your services are still unfortunately coupled in
some way and you really can't get the value out of them without having some
of the ancillary pieces also spun up and working at the same time. Right
on for sure. So Galen, obviously there was a set of circumstances in
your life that led you to say, this is a real problem and I'm
going to I'm going to give it my best shot at starting. And it
led to a company. What what kind of scenarios were you working in that
made you realize this is what you want to do? Yeah, I mean
there's a couple. The earliest instance of this innate pain it came at my
senior year of college. My senior year of college, I was doing a
research project on distributed databases. And this was right before Ethereum launch, which
is a cryptocurrency, and I was doing a research project to explore different consensus
mechanisms, so like low level protocol comms between notes. I wanted to spin
up three nodes of this network just to experiment with my different consensus mechanism.
Nearly impossible to do that at the time, and a lot of my project
was oriented around simply getting the things spun up in the first place. Of
course, the code worked on the developers' laptops that worked on the DevOps.
Laptops didn't work on my laptop. And it wasn't about it wasn't about building
a particular note. It was about getting the cluster of the system set up
correctly where there's really no good abstraction that gets you that very consistently. You
can get it in certain architectures in certain ways decently consistently. But that was
the first instance. It was very fresh trading, like why won't this thing
just spin up? It's the simplest application, just run it as it is.
Yeah, that was the very first one. I saw it later in
my career as well as working at a company that had had a internal tool
that spun up a full micro service deck you can think maybe two hundred micro
services, and it was used for a variety of purposes between dev demoing uat
so like a high level just testing user workflows to see how they would work.
That took a lot of effort to get that system spun up, and
at the essence of it, it's the same issue. It's I just want
to run the system and a configuration that makes sense for the work while I'm
doing and it can be remarkably hard to do that. So this these sets
of issues. It's kind of the feeling of like, this shouldn't be this
way. It should be decently easy to just run software. It should be
hard to like build software that's that's that's amazing, but it should be easy
to run it. It's kind of the feeling of the way the world should
be that kind of motivated the entire start into pretosis, right, And yeah,
and I think there's I think maybe just part of that is we build
these complex systems, but for most of us, once we build that,
like once we build our production environment, the people who really have the skills
to build those, they don't build a second production. The part where this
pain has really felt is in the development teams where they're trying to you know,
build and tear these down to test different features and add new features to
it. Yes, and that's one of the main themes that I've learned while
going through this journey is it's a lot of the times these development environments they're
about getting us close for production as possible but safe and you can do the
workfload you need to do. But the main value access is how close can
you get to production? So, like Warren mentioned, local stack, you
can run a tows services on your laptop. It's one approach to it,
bring production locally or bring production to your development environment. The other approach is
the opposite. You put your development environment into the cloud, where you then
basically fork everything that you have in production and you have your cloud services are
sitting in the exact same environment where your development code is another approach to it.
But basically, everyone who's putting a lot of investment into getting these great
development environments does one of those two things, or some combination of both,
bringing that bringing production locally or putting your local into production in the sense of
into your cloud environment. Right right, what's the barrier to intrig for using
something like critosis, Yeah, right now. The biggest barrier to entry is
defining the setup logic for your environment. So we have a definition language.
We use Starlarc, which is the language. It's a Python dialect that Basil
uses to define the build rules. We use it for similar similar properties that
it has for the build system. It's you can determine that the program is
going to halt because it's not a full computing language. But you need to
define your set up logic in this language. And once you do it once,
everyone else can reuse it. But what happens a lot of the time
is who knows how to do that? You have to It's especially in an
environment where there's a lot of different open source tooling being used to varying degrees
of maturity. The person who has the expertise to understand how to spin it
up in a different right situations, that person might not be maintained that project
anymore, even so track that down or relearn it somehow, for sure.
Yeah. And then I think there's a related piece to that is not only
building it, but then making sure like reverse engineering, I guess, for
a lack of a better term, to make sure that what you've built for
your local dev tool actually mimics what's really going on out in production, right,
because there's a lot of different ways you can make all of these different
components talk to each other, and you've got to try to replicate that.
Yes, yes, and that's that's a lot of work, and it's that's
one of the reasons why a lot of people do opt to forego that whole
process and just put their development code directly into a production like cluster in the
cloud as as possible. Because they're like a localist, never going to be
close enough to production. It's going to take a lot of never ending effort
to get it close. So let's just get to it a stage in cluster
as fast as possible, let things soak there, and then go to abroad.
It's another approach to it. There's benefits and cons to each one,
for sure. Yeah, what are some of the big Well, first of
all, how long have you been building critosis? Yeah? We've been doing
this full time since December of twenty twenty, So what is that? Okay,
three years and change? Yeah, yeah, yeah, cool. So
what are some of the how is how is your thought process changed in those
three plus years for building development environments? I mean, I'm sure you've learned
lots of lessons and tried a lot of things that didn't quite work. Absolutely,
absolutely. The couple of things. First, we started out, we
started out our company selling a basically binaries. So one thing is, how
do developers and development teams want to consume developer tooling? So we started out
saying we're going to sell a compiled binary that you get to run as a
license fee, and their first couple of clients we operated that way and It
was fine in the very beginning. We were solving a very important problem for
those clients. It was acceptable at the time, but we very quickly learned
that this is it's not the way that developers want to and for really good
reasons, want to be interacting with software. You do have to have the
source code available, You do have to have a free license, you have
to have a free period of trying it where they know, because they're integrating
at a deep level with their tooling, if they're going to try it out,
if they're going to invest into it, I do want to know it's
not going to get yanked out from under them, and that's totally fair.
So we change our licensing entirely, made the reboot open, and that change
changed everything about the rate at which we were learning, the rate of the
exposure to different developers, people able to try out to too long gate different
perspectives. In retrospect, it was pretty obvious. Maybe a lot of these
learnings are pretty obvious in retrospect, but you know, that was that was
maybe the biggest change that we had in the business, was opening a tool
like that. No, I think you're in good company. The past few
weeks months I feel like there's been a huge shift and focus on the podcast.
It keeps coming out. Wills bringing up this point about selling your idea
within the devlof space as a product management challenge, So really understanding what your
teams need that are your consumers. It's difficult but also really interesting and can
help you build a great a great product for them. Yeah, and it's
developers are they're amazing users to work with, but it's a very different dynamic.
I think then products that maybe are selling the sales folks or business focused
folks, you're the way that products are integrated and interactive with is quite different
and as different dynamics. But once you're solving a problem that the developers really
really care about, you really do develop really deep relationships for users, especially
with like the community based approach with like an open product. So it's it's
been game changing having that kind of a relationship working for us. Yeah.
I think that's a really cool point there, because you're talking about who your
target market is, and so whenever you think about like production deployments, you
know, deploying my application to what my target customers, that's a very different
process than deploying my stack to the local development team so they can build on
it. It's a different, different customer and they have different goals and objectives
like the customers, the consumers of my product want high availability, up time
and fast response, whereas the developers want more granular access and the ability to
tweak and turn the knobs across the environment to see how they can solve their
problems. Is that a fair statement. Absolutely? Absolutely. In fact,
there's even three different profiles to consider. There's the developers that are the end
users of the tool. There's the we call it maybe the platform team DevOps
team in for a team, they're really the ones that are responsible for the
implementation and configuration of the tool. A lot of the times they're the ones
getting access to the GitHub repos they're doing the high level configuration administration of the
installation, and they're concerned about the security posture and all that. And then
ultimately you also do have like the director or head of engineering, whoever approves
the budget for whatever you're charging for an enterprise version. So there's three.
There's three different concerns to worry about, and you have to think about each
one differently. It's it's like what when you think about like a minimal valuable
product, Like how how batting of the concerns at each of the three parties
you really have to do to like get something off the ground. Uh yeah,
so it's a multi dimensional problem. No one envies a startup pounder,
I mean they do. They think you get to build your your dream,
but then you deal with these quite challenging situations where your buyers are not your
users that you are you know, really focused on. I have a question
on that. Actually, I feel like and I know I'm gonna put myself
in some bucket somewhere. I assume like everyone's using open TOFO today. I
mean, I know that's not the case, but you know, maybe there's
there's a huge audience there. Especially if we talk about people that are in
the quote unquote develop space. If they are familiar with this terminology, they
probably are using some sort of infrastructure as code solution. We use extensively CloudFormation
ourselves. What is the how does that interact with krotosis here? Like where
the touch points? Like if we've already defined our infrastructure in unfortunately HCl or
worth in a programming language that doesn't really well match to declarative nature, what
do we do to spin up a dev environment absolutely, so right now,
krtosis operates after you apply your open tofu or your terraform scripts to spin up
the cloud services that you would need for your development environment or your testing environment.
You can think of it really as a abstraction layer over Kubernetes and Docker.
So if you have can use your terraform scripts open tofu to spin up
your Kubernetes clusters and provision everything required there and then deploy onto the clusters with
grotosis, I see gotcha, And so then that so like that, that's
like a way to leverage some of the tools that you're using to deploy into
production for building that so you're not duplicating a lot of effort, right so
you can still leverage the learnings from terra farm or open to tofu or any
other production grade tools that you're deploying with exactly exactly. And then the cloud
provisioning side of things is an area that we are now exploring, and it's
not an area that's covered in the current like open tool and like the direct
aspaciates of the open tool where it's hey, can you also handle maybe four
K my production environment, and its like as look as you could fork an
environment, can you handle clone everything necessary there? Right now? That's a
lot of what's happened in people's information and terraform scripts is something like that we
haven't That's an area that we're exploring right now learning about. You know,
what's important when you're when you're looking at designing those those scripts. Yeah,
for sure when it comes to documenting, like because that's a I think that's
a would be an important artifact of this process is we've discovered how our production
infrastructure is working, reverse engineered it. How do you approach documenting that so
that when the next development team comes along, maybe the learning curve is a
little bit uh, not so sharp for them. Yeah, right now,
the document I guess there's two ways to think about it. One is one
is how much can you write? How much can you have your users write
code that is some self documenting, so like, can the environment definition be
the documentation and as much as possible. I like having code that is extraordinarily
readable and it seems self documenting, and the way that our tool is set
up, you tend to get that in the code. It tends to be
very composable, it tends to be very readable. It tends to be very
intuitive, So that's one function of it. Another function of it is the
functionality that we offer, so we have high level environment parameterizability. What that
means is when you write your environment definition, you can expose very easily how
to modify it in a sensible way. So you start it and you say,
well, maybe I want to modify it this way, this way,
this way. We don't need documentation for it because when you want it with
that configuration, you're going to get it out of the box. So you're
trying to reduce the need for documentation as much as possible with the design of
the definition language. But you can only dose it so much so that after
that it's it's traditional documentation. It's you got to rely on your users to
write the readings, to write the docs for those specific systems. That it
reminds me of a quote that I heard a long long time ago, back
when everyone used to have wikis for documentation, and I worked with this.
One guy said, yeah, the wiki is where the documentation goes to die.
But I think like one of the cool things I think that's happened is
if you've ever used Jupiter notebooks. I love the way that they let you
mix, you know, different different syntax in there, so you can have
like a markdown block and then a code block, and so you can kind
of integrate the documentation alongside the code. And for I think for iterative or
interactive type environments like that, it works really well because it then lets you
keep everything in one place, because when you try to update multiple places,
it just never happens in reality. Yeah, and I love Jupiter Notebooks the
experience, Like, it's one of my favorite That access in the world is
like spinning up and hacking something together in a Jupiter notebook. And the dream
is to feel that way about all of our development work. I'm not sure
realistic, but I love that experience. I mean, that's an interesting thread.
What would what would the world have to look like that regular development could
follow this sort of path here? Is it a matter of what we're working
on or can we already sort of start doing that today? Yeah? I
mean what's great about it? Like, one thing that's great is how fast
you see your changes work. It's like extraordinarily fast. You write it,
run run and run, run, run, and you get the feed back
right away. That's the build deploy like how the build deploy loop. It's
like, how fast does it take for you to see on real stuff off
what your things look like? Uh, usually in Jupiter notebooks you're using.
I mean, like a lot a lot of the fun use cases for it
are running on relatively small data sets so it fits in memory. So you
do that fast feed toploy build a play loop, and that's just tricky to
get. Like you can get it with like hot reloading into containers that are
running in your in your environment. You can get decently close to it,
but you're still stuck at the level of like what is this What is the
smallest build increment that you can that you can like define for your background system.
I know a lot of people spend a lot of effort getting their their
basal build rules so well defined that they're going to like minimize the build a
play loop as much as possible, But that takes a lot of effort.
You really have to look into it to get that experience. I mean,
I'm not sure that there's a really good way to get that now, No
for sure, for sure not but I think you know, tools, tools
like yours you know definitely help reduce the loop here, which is like the
most important aspect because no matter how much time you spend, really every second
you cut off your loop allows that feedback to happen faster, which improves the
time. I think it's a hackneyed response now that realistically people waiting for software
development, is there what makes them unhappy? Right? If you have to
wait for the response? Will said, You know, he's all about the
instant gratifications. Nothing better than you know, hitting F five and having your
program instantly work or tell you what you did wrong, which is you know,
I don't think it's just me. I'm convinced that we are a TikTok
generation. You know, whether you're a TikTok user or not, we've all
gotten addicted to short form content. You know, if you're going to do
anything that requires an attention span of over sixty seconds, you really have to
engineer for that. And I we've been digging a lot into this this little
that little feedback loop there and one thing that did the dream world. There's
kind of two really important things. One is getting that build deplay loop super
super fast, so you're coding the change, accoding to the change. The
other one is when you see the change, how close is the change that
you're seeing to what would happen in production? And I can you get both
of those things maximized, so like you're getting a really really fast build to
play loop, but also the deployment is safely basically a production when you're doing
it. That's kind of the dream the dream world. It's it would take
a lot to get there, I think all the way, but you kind
of like, roughly, what's happening. I think with the more and more
sophisticated like dev infra along the software development bicycle, roughly what's happening is you're
getting like more and more optimized on both of those axis to the point that
eventually it kind of feels like you're doing that like the limit of what's possible.
I think there's a sort of a conundrum here where the providers that are
offering production infrastructure or platforms, the amount of money that you're paying into them
is so large or so vast for the environment you're running for production that it
never makes sense. It's not something that is ever justified for them to make
it faster to run or get the loop closed because they don't care. It's
you know one like how much is your test staging, fake feral environment cost
in actual dollars, euros and pesos whatever compared to production It's like what one
percent less? Even so like Amazon, you know, Google, GCP platforms
of the world, they're not going to make anything there, and so I
feel like there it would be on them to really change that because that's sort
of like the canonical interface that you have to abstract or make the same as
much as possible. Yes, absolutely, and basically everyone's working against that limitation
as far as I can tell. It is like that's the that's the blocker,
the only way to like you're you're basically just trying to design more and
more like innovative solutions to get around exactly that obstacle. Yeah, because let's
be just like completely transparent here. The gold standard is them on the production
server. Yes, you said that. I feel like that's where I started
my career actually right for sure, Like I was working at a company where
you did software development by sshing into another service somewhere and there was an environment
running there which then you logged into and you literally changed production objects by opening
a text editor there. We didn't it wasn't VIM that would have been slightly
better than what was being used, uh and uh, and then changing them
and then you had to record the changes you made in an overgrown internal system
that would somehow get those changes applied to actual production when there was a release.
So, you know, I really do feel like what was old will
be new again, and I honestly don't know if I want that world.
My introduction to programming was doing something very similar to that, and the guy
who showed me how to do it, he's like, okay, when you
get here, you want to run the LS command and you could see all
of the files there. But then it was like, you know, for
any given file, you know, like just say like Maine dot c,
there was Maine dot c dot one, Maine dot C dot two, Maine
dot c dot three. Is like, so you want to look and see
what the biggest number here is and copy Maine dot c to the next number
so that if you screw anything up, you've got a backup for it.
And that was our version control system. Oh no, I'm just like,
I think people figured it out date based, right, You don't. None
of these are the fancy tools. All you need to do is append the
timestamp of the last time you edited the file there and then you'd be good.
Yeah, I mean it's the VIM on the production server. At least
for me, this is like clearly the best debt X Like I love it.
I love it. It's just dangerous, it's super dangerous, and like
staying up all night to think there of an outage is miserable. So that
experience is not great. But the feeling of the VIM directly to production is
is is the ideal, So like to what degree can you mitigate risk to
get as close to that as possible? I mean, you said it right,
you know, things like kurtosis, you know, shortening the gap there
on time to make a change so that rather than VIM on the production server,
it's VIM on your machine or you know, using one of those better
tools that's out there, and then making that change and just immediately that gets
to a clone of production that is almost exactly this, I mean exactly the
same, and you can see that change running as if you had actually applied
it and then get all the benefits without having actually the danger of making a
production change. Yes, yes, absolutely, that's that's that's the that's theer's
the dream Yes, So what is the learning curve for someone who's just they've
they've done the seven thousand line bash script and they're looking for something different and
wanting to check out critosis. What's the learning curve or the amount of time
it takes to be productive with this so as framed very low. And the
reason for that is because that person did the seven thousand best script part.
They figured out how it all works together, they figured out who needs to
be scripted, they figured out all of this hard stuff. That learning curve
is extraordinarily low. I mean, in Krytosis, you're using a Python like
dialect. Most people are familiar with Python. The set of I guess functions
and tools you have are very are very basic. Half a day to just
rewrite it and something that's uh, that's cleaner, and then it's going to
take you a lot further. It's probably about what you're going to get.
Now. The part that's you know, frustrating for us and we're trying to
figure out how to make this better is what if you haven't written the seven
thousand mind mass script, then what do you do? You say, Hey,
someone else wrote the seven thousand mind mass script. I don't understand it
yet, but I want it to be. I want to get all these
benefits that I can get from protosis. Now we have a really big now
we have a really big verded curve, but the learning curve is mostly inside
of the side. Interesting for sure. And then can you give us like
some some success stories of like different problems that customers have solved. I always
think that's kind of cool because then it, you know, it gives us
something that we can like resonate with and be like, oh, yeah,
I remember when I had to do that. Like, what are some cool
things that you've seen people do with krotosis? Well, I feel like your
whole work experience history here, you know, just recount those things and it's
like, yeah, yeah, my career could be a like a Forrest Gump
story in tech. Yeah. I Mean the biggest and like the most proud
use case that we have for cryptosis is what the Ethereum Foundation is doing with
cryptosis. That's probably the thing that we're most it's the most incredible thing.
So for folks who aren't like deep into the blockchain ecosystem, the Ethereum main
net, so it's like the production deployment of Ethereum is now extraordinarily complicated.
There is like twenty different open source components that have to get splat up together.
There's no centralized DEVLOPS team for this, so it's all different people running
their own different service with moose communication between them. What krotosis, what they're
using Katosis do. The Ethereum foundation is to package all of that together in
a one line run with high level configurability, so you can say, hey,
I want to run it with this topology like these clients and these clients,
and it works remarkably well. It works remarkably well as it's it's a
large degree of complexity. The Beast. I don't know what it would look
like if we like unfolded it into the Beast script scenario that you described,
it would be very remarkable. But now this thing it runs, if it
runs the UH, you know, you can one click replicated Thethereum main that
on your laptop or inside of one of your cloud boxes or Recubernators cluster at
scale. UH, so you can kind of hit the dev all the way
to late stage testing performance testing. UH. There was a team that built
h They built it's basically like a sad path testing on top of this tool,
so like a path network perturbin in lots of different ways, as you
call it, chaos testing on top of this, and it takes you all
the way almost to deploy the proud if you're going to make like a big
change in the deep layers of the network. So that's that's the most That's
the thing I'm the most proud of when it comes to like what are tools
been able to do. There's a large degree of DevOps complexity encapsulated in that
project, and then there's a bunch of people using it to do some really
cool things on top of it. I feel like that makes a lot of
sense there as well, Like when you have a lot of different testing scenarios
that need to be captured and the configurability for them is also determined sort of
by the constraints of the test, then you have a huge dimensionality of the
data and configuration that you need to specify to actually execute it. There.
Whereas I feel like, you know, some other environments that may not be
total like always the case, but in this one, like, yeah,
for sure, I can totally imagine, especially given the fact that it's there
is no you know, official production environment where you know cloud provider where THEOEUM
is running on exactly exactly, And it happens in cases where you're doing a
lot of different testing things. And it also happens when you're I guess you
could call it like a feature branch environment. It's it's these situations where you're
you're like pre merge and you want to spin up a representation environment that's specific
to you're not really deving on it because it's not inside it. You're like
build and run loop as you're in your IDE but it's kind of close to
like your death because they're not ready to put it into CI yet. That
situation as well as one where inside of an organization or an open source community
you might have hundreds of those a week of slightly different configurations, so it's
either easy to configure or it's not. But if it's not, like this
is a a great way to do it. This gives me the sort of
PTSD flashbacks to the previous environment that I was working in where there was like
different production environments that we had because they were for different customers or different internal
business units, and then we had copies of them that were test environments that
we're supposed to match the production one because realistically, given the ridiculous amount of
configuration, you almost needed to run your tests in every single sort of clone
of production however it could be possibly configured, and then of course it was
a problem where like the actual switches from production didn't match where you're in your
testing environment, and so you had not only did you have to run your
test in all of these fake versions of production, but also make some changes.
That for sure was quite the nightmare. And there's a mode of thought
that I'm honestly a fan of this movement. There's a mode of thought where
you just say, can we forego all of that and just make shipping the
production as safe as possible and do all of it like test in production.
That motive thought is like sometimes possible, I think. I think that's like
Honeycomb's big big model. They're an observability tool like test and production. We're
just going to give you really good monitoring tooling so the moment something goes wrong,
you can catch it right away. For an impacts your entire customer base
rollback like some some some products I guess can do this. Otherwise a tiny
little bug is a disastrous like medical tooling. I don't know, Like I
mean, I want to do it for sure, sign me up. I
mean we're in the at authors with the five nine SLA where for sure we
can't let stuff go through. I think the biggest challenge is depending on what
your business is actually like, what the business logic looks like. The fear
I've always had with these sort of automatic rollbacks in a lot of cases where
what happens if the problem is compounded by changing the database version? And I
always keep getting back to the Night Capital four hundred and forty million dollars loss
in forty five minutes, kind of like the ridiculous thing is they had the
wrong version of the software only running on one machine, and they made it
worse by restarting the cluster. Like if they hadn't restarted anything, it would
have been fine. And they assumed that the problem was with the rollout,
so they rolled back everything and then also compounded the issue. And it's those
things that really scare me, and I like, I would so totally before
if the automatic rollback could calculate whether or not the you know, the database
is that fault and also include that, And I feel like I'm not going
to see that in my lifetime. I don't know. To me, it
just sounds like we're making excuses to keep from giving people VIM access to the
production server. I mean, you know you're I think in control of that
actually had poligon right well, like you could just go and turn that on
and let that happen and it could be like it could be it could be
the real case study. You could do that right shop, whether it was
a success. You know, published that blog post. You know that you're
doing this, why it's so great, and you know you could get on
board with it, and then next week's episode will be reviewing my resume because
I'm looking for a job. I'll never forget that article that was published by
get lab about how they you know, accidentally deleted their production database. Uh
it was like a decade ago. Now. I was like, oops,
Like I ran this command and I thought I was on the replication uh server,
and I was actually in production. I've done that to do. Yeah,
it was in AWS, and I thought that I was on the on
the staging environment and uh just dropped an entire database cluster. Oh that that's
not that bad, right, I mean, you just restore that from from
the one that you you know, got snapshotted from an hour ago. Like
it's much worse when you, like, you know, write that update statement
in sequel and forget the select that comes after it, right, you know,
people say you should write that first, like write the from statement first
and then change your select to an update. Yeah, it's just it's ridiculous
that those mistakes can be made. But I think at every company I've been
at, there's been a special name for that, like, oh, don't
eighty four the database, because like that's just happened to be what the update
was, where someone didn't configure the restriction correctly and now there's some column with
this wrong value in it and it's so noticeable too, right, Like it's
not like, oh, database is gone, let's fix that. It's oh,
when did someone change the name of all of our customers to eighty four?
I mean that's kind of like a badge of honor though, like if
you were if you were the person who implemented eighty four, like, would
you put that on your LinkedIn? So yeah, I'm that guy. I
think that was my biggest fear when I started my engineering career was doing one
of these things like the terrified of touching anything that could look and feel something
like deleting a duction database. And I think it is about your honor.
It's the moment when you realize, like, once you do something like this,
hopefully to a smaller scale, you kind of learn like this is a
process issue, like this should have this should have never I never should have
been able to do it. Shouldn't have been a risk that I would do
this. I don't. But when you start out, you feel you feel
I'm gonna get fired if I need one big wrong. Yeah, and it's
one of those things. I don't think you can communicate that. I think
the only way to learn that is to to earn that badge. This is
a failure everyone has to go through in your life. You know, suffer
that trauma. You'll live with the PTSD from that moment on about how the
more the larger system, because you know you've gained more experience at this time,
you're in a more important position. Your impact to the company is obviously
going to be higher. You'll never make that same mistake again. Yeah,
it's like riding a bike, Like you don't really understand the pros and con
of riding a bike until you face plant into the side of your house.
You're like, oh, gotcha, breaks, it was a mailbox for me.
Cool. So let's talk about like we talk about mimicking the production environment,
and production environments are always changing with you know, updated code and updates
to the database and sometimes breaking changes, especially like since we've been talking about
databases, like breaking schema changes with cartosis. How do you make sure that
those production changes get integrated back into the local dev tool environments. Yeah,
right now, that's something that's something the end user has to configure. We're
looking into ways to get this more to provide like a product doze, way
to automatically get this. But right now, when it comes to reduction change,
my dev environment has to update to it. You got to. You
gotta have a process in place to check out updating the configurations. Ideally,
what you do is you any time that you make a production change, it
goes through your krotosis set up first. That's the ideal scenario, but it's
not always going to happen that way, of course. Yeah. No,
I mean I imagine that you're making the database changes in your source code anyway
to run a migration, if it's schemed or schema less you're making the code
changes that are associated, and you would just you can just refresh your dev
environment, like you know, blow it away and start fresh, because the
migration should just not even need to be run potentially and it would just be
the same as production. Well at least that would be my hope. Yes,
yes, it's it gets goofy when people do things like like hot fixes,
where like other people hot fixed stuff and you never looked at it.
It can get goofy. But happy past scenario, it's not that hard to
get it back up to. You know, these were these experiences are important.
I remember having to do a production fix and at the time that meant
that a different team created a SPN feature branch for us. It was a
hot patch of production, and I ended up with this problem where there was
code that should have worked in production, but it didn't play nicely with what
was there, and you know, I compared it to what was actually in
and Devin, it didn't match. And what I had found out is that
there could be multiple production fixes that had to be done at the same time,
and merging in SVN didn't really work that well. And what happened is
is you would get merge conflicts there, and this team that was not the
development team or knew how the feature should actually work, was going and actually
doing the merge conflict resolution themselves and pushing that out. And that was a
very horrifying learning experience for me. I think I'd rather drop the production database.
I mean, you don't even have the code to look at to know
what the issue was, because like I had to go find these other branches
that weren't merged to production and like merge them into my code and see like
it still wasn't right, Like where is this? Where is this? Wrong?
Code just didn't exist anywhere. And then of course you have like the
incident response team coming to your desk because in these days, of course you
had to be in the office and like asking you, why are we losing
one hundred thousand dollars every minute because of this problem? I'm like, I
have literally no idea. Yeah, that's always cool too, That's just another
fun experience of being able to quantify the cost of downtime, like, oh
wow, this is serious. I feel bad enough to work like twenty four
hours a day, seven days a week, and you'll never be able to
wait from your computer in case something happens there. Right, that's the solution
four seven with VIM on the production server. Yeah, it's just going to
keep you know, if you keep bringing that up like I'm gonna, I'm
going to write down this in my in my notebook, you know, see
when Will got this product ready to go, where we'll let you you know
it uses maybe s s H under the hood or your ID you know,
just you called VIM and production you can even you know it even it lends
itself a name VIP, right, and we'll see where that goes. Right,
I'm just going to keep campaigning until they let me give a talk at
like platform con about how to do this. Well, you know, how
to get accepted to those conferences. What you do is you first just say
you've done it, and and then if they get accepted, then you'll have
to come with the whole product and actually try it out and I then present
it were full of useful tips today. I mean I really I really like
this idea of getting a tool in the mix here for helping build the the
the environment for real. Yeah, and if you like to help illustrate the
scale of this, you know, with your success with the Ethereum foundation,
that's a monster project with completely open source, you know, and so I'm
assuming that like if you want to if like, if I wanted to contribute
to the Ethereum project, I could laugh and Warren, I think I'm going
to ask for access to, you know, update this tool, and I'd
be like, well, you know, will you know, making changes to
it? I'm not. I'm not sure I know. But if I wanted
to to contribute to Ethereum, like I could use crytosis to build my local
div environment which helps me write the code, test it, and then create
a poll or open a poll request that's going to be more likely to meet
their contribution guidelines. Is that a fair statement? It's I mean, it
probably is the easiest way to do that, Yeah, yeah, versus the
alternative prior to because I worked with Ethereum before they started at adopting cretosis,
like prior to that, you know, it was like twenty or thirty steps
to get this thing running, you know, where you were downloading from all
these different repositories, and then you were compiling it, and then you had
to figure out in the config files which parts you had to change to make
these different things that you just compiled, but you weren't really sure what they
were talk to each other and so yeah, this like having this type of
solution makes it easier, and that could be that could be another benefit for
you, especially if you're an open source project, making it easier for people
to contribute to your product. There's actually a good rule of thumb for when
an open source project or community would would benefit from from using for dosis,
and it's when you go to the docks and it says how to run locally
if it's like the longer that thing is, the better you're going to get.
It's kind of the general rule of thumb is how long is that thing?
So yeah, for the ethereum, it was, it was huge.
There's other general rule of thumb is that that that that's pretty much how it
works, uh and and it is help work for contributing. It's most of
the people, most of our implementations are open source entirely, so it's you
know, it's it's people contributing back to those uh, people that use them,
putting things back, getting more options available. And that really really helps
because our team doesn't have time to handle all of the different scenarios. UH.
When we start building examples and everything. So having people contribute back is
I think the the Ethereum package, it's called a package, the definition of
how to spin up an environment, the Ethereum package. It really is a
remarkable work of open source contributions coming back. They're very, very engaged users.
I think you touched on this a little bit earlier on about first run
the licensing model and then and then you change off to that. Like if
someone's going to get started today with this, is it I install a CLI
and write a configuration file. Is there a hosted version? Like on these
packages available and you'll deploy them automatically to my cloud environment. What does this
look like? There's actually two there's two versions right now. The standard way
is download the Clyde the a homebrew or app get or whatever is the preferred
installation thing. Most people who start, they're starting because they see a package
they want to run. So so most people they're just running a package when
they start to see how it spins up, and then they start modifying that
package if they want to modify it, or they're like, hey, this
is cool, I'm going to brite my own my own system, and it's
all all happens on your own laptop and you're own hardware, and you just
you just run with it and go. And then we do have a cloud
right now that we host where Hey, let's say you don't want to down
the CLI and you don't want to be using your own compute to be running
the environment, you can just button click. It's like a no code interface
to run any existing package. There is also no code interface for modifying the
package to a certain degree. That's evolving, so you can get more and
more power over modifying it. And then once you're excited about it, you
can write your own package and then you can deplay that there as well.
And then the packages are those the ones that end up showing up in the
Kurtosis catalog. Yes, yeah, because the Curtosis catalog is actually really cool.
Specifically, you know, if you going back to the Ethereum example,
you know, you can just grab that from the catalog and it runs for
you, right, and then there's there's actually quite a few out there in
your catalog. One that I happened to know of is the Polygon CDK as
of last week, just went up in the catalog. So if you want
to run that, you know, just grab it from the catalog and there
you go. Interesting. Yeah, they're right there. And and what the
I mean the way the catalog works right now is it scans public GitHub.
So if it's an open source package, it's going to get detected and populated
into the catalog and then you can just run it. Because it's an open
source, it's an open source tool. So the Polygon CDK, for example,
the Polygon team put a lot of work into to get into the package
very in a very nice format, and now it's there and you can just
button click run it. And one of the most exciting things about having made
this step of opening everything is that we're seeing stuff happen without us knowing it
or tracking it too closely, so like, oh, look like look at
what the Polygon team has done. This is amazing And that's very exciting.
Yeah, for sure. And I think that's because y'all are completely open source
as well. Right, So the tool is actually the BSL license, okay,
source available. You can do everything except for recreate our right on it,
which seems like a fair constrained exactly. Yeah, no, but I
think that's really cool having the catalog there, especially with the open source communities,
and it seems that Kurtosis is really resonating there to make it easy for
people to just run and then contribute and then improve your product and in a
consistent format, you know, because I think that's one of the biggest challenges
of running an open source project is verifying or vetting out the contributions that you
get to make sure that they have done in some cases just tribal knowledge.
Yes, yes, yes. And it's one thing we were noticing, especially
as we work with more and more people who have cloud dependencies, is the
more that your system is is composed of open source components, the more this
problem is important to you because you're you're dealing with a launch of different software
components built by different organizations, where where you can run that entire software component
yourself. That's the whole point of the open source So you don't have the
issue of you know, cloud provider run a production environment has zero incentive to
like make you able to like modify it or move it. In the open
source communities, not the case, you actually can do it entirely. So
the more open source your tool is in terms of like the percentage of services
that are involved in your environment which are open source. The more the more
you get this benefit of like how do we verify that the contribution is going
to work and makes sense? And you do kind of have to run it
all to see it. Yeah, I would even take it further, honestly
that most open source solutions out there that aren't using a strategy or have a
strategy for easy for contributors to come in and run the software would benefit a
lot from a strategy like this, because realistically, if you go out there
and find an open solution, no matter what it is, even one used
by thousands of organizations, it is quite the challenge to get that up and
running. Even if it's a darker container, even if it's you know,
in a registry like it's still there's a lot that goes on with it.
I remember it was it was even recently that you didn't need a zookeeper anymore
to run Kafka, Like it was always this sort of mess that if you
ever wanted to run Kafka yourself, you needed three different services running on your
machine with multiple nodes and clusters at the same time in order to even test
this out. And I'm like, what, yes, yes, it's a
big issue, and it's and it's for the most part, in my opinion,
it's a bit of a silent issue because if you're contributing back to an
open source project, and a lot of the time you're doing it out of
almost like a hobby mentality. It's like, I want to help this project,
I'm interested in it. If there's a really big friction bar for me
to do that, I'm not getting paid to contribute back to it. That's
the whole idea. So I'm not going to put an inorder the amount of
effort into knocking down the door to say, like make my testing environment better,
like make my environment better, I'm just going to disappear. So like
the amount of open source contributions that you're missing because because people just got frustrated
in the first five minutes and say, you know what, like someone else
is going to handle this contribution at some point hopefully is I can just imagine
the velocity that you're losing there. I mean, I hope that's really the
case, that that's even where they're falling off. I find that in Unfortunately,
the state of the open source today is like you find an issue out
there, or you want to even make a change and you may open an
issue or open that poor request and just never hear back. And with stuff
like this, you know that the team has invested in what that experience is
on building stuff up. So it could be a badge of honor really to
say this open source project is using kurtosis or another tool that makes this experience
better because you know that you thought about it, that they care about getting
those contributions. Then, so, is there is there a thought pattern behind
you, Like when we talk about production servers with the whole DevOps movement,
we kind of came up with this philosophy of servers are cattle not petch,
you know, referring to the fact that they should be short lived and you
shouldn't have emotional attachment tool them. You shouldn't be like hand curating them.
But I feel like dev environments maybe didn't fall under that umbrella. Is there
like a thought process or something that you've noticed through critosis where you kind of
see people shifting away from the long lived development environments because it's a little easier
and standardized to bring them up. Now, it's I don't know if I
would say that it's a shift away from it. It's in some ways it
is for certain use cases, like so how could I describe this? What
one thing crtosis does is if you run your integration tests in CI and cider
crtosis, it's quite easy to rerun that locally in your laptop. So if
you want to debug or failing integration and the end test, rather than recreating
it in a remote cluster that's like a shared cluster, you would just rerun
it on your laptop. And in that case there's a shift towards ephemeral environments,
like you're just for depth, like you're just running vision. But there's
also people who are like, I'm going to use kryptosis to set up a
persistent development environment because the other deaths of my team it's just significantly easier for
them to ping an endpoint that's sitting there stable and to get into that.
Then even to like download a tool and then re run some commands every single
time, and and then you get into the problem of like data management.
So if you've invested in your dev environment, having a particular data set behind
it that's useful for that dev project, and then investing the extra effort into
cloning and moving that data for an ephemeral use, it often doesn't make sense
you just use the same thing. So I think it really is It really
is dependent on the use case. But there definitely is a meaningful difference between
how you treat your dev servers and your broad servers. So like the cattle,
the cattle not pets thing. I think it's totally fine for your dev
server to be a pet. Like it's there's there's there's times when it makes
a lot of sense. The moment you want to see that it looks like
braud, you do kind of want to turn it into a cattle just to
see how it works. But there's a lot of telling reasons to treat it
like a pet, especially if you are debugging an issue or trying to figure
out the way things kind of work. You want to have that state sitting
around, and you want to really preserve that because it's going to take you
some time thinking and creative time to work with it. And if it's a
cattle and you lose it halfway through your work, it's miserable. You got
to restart, right And I think I heard you say that there is a
there's a use case here for using something like critosis to create a persistent dev
environment that provides a common set of services for your dev team to access.
So not everyone has to run that locally, but there's one that's centrally available
for them that maybe has a specific configuration or has a something unique to it
that everyone needs to be thinking about or addressing in their local development efforts.
So you just provide that as a shared service for everyone. Yes, and
that's useful for a lot of development workflows. But then there's another set of
development workflows where you're going to say, I actually do want to isolate for
an ephemeral purpose a particular set of micro services or a data set, and
I want that to be my sandbox that nobody else is touching. And that
happens a lot as well, because it's like, hey, what if three
different teams want to use the dev cluster and they're all hitting each other,
and then you have to go on Slack and say, hey, wait a
minute, like, let me use it for the next two hours, you
guys, when you blow away their work, and it's miserable, But that's
only sometimes. So the best world is you have the choice. When you
need to get the isolation, you can get it quite easily. When you
don't need it, you can use other people's work. And that's the direction
that we're trying to go towards, is how do we provide we like high
high level. You want it to feel like you're in Harry Potter and you
have a wand and you're just like that, you have the figure need Like
that's the highest level user experience that you want to get to. It's like
cool, Like, how do you design the want such that you can do
that? And I think you just want to have the configurability to be able
to pick which in this scenario, I want the shared duftbuster or I want
the isolated if I'm one right on, So, continuing on that thought,
what does the future of krotosis look like? What are the in addition to
that, what are some of the things that you would like to see getting
implemented? Absolutely, so I said before, it's right now. Crotosis operates
to you give your terraform scripts or your open TOFU and it deploys your provisions,
your hardware so to speak, and maybe your virtualization later like the communities
Buster or Ductor Engine, and then Crotosis deploys on top of that. We
want crtosis to move in the direction where you no longer have to do that
first part. You no longer have to think about and dig into like how
do I how do I split my my cloud assets? Over how do I
provision my hardware? We want to get better and better and better at providing
tools to not have to worry about that, so that that's where we're going.
You know, the more that the folks in the blockchain space they use
a lot of open source tools where it's just all can be containerized and you
get everything in the container. The more that you're working with folks who have
alliances on like a to bus services or manage authentication services and this kind of
thing, the more that that abstraction is not so useful anymore because those are
the issues, the issues of the cloud stuff. So the more that we
can build an instruction there that captures that stuff in a way that's useful,
that's where we want to go, and that's where we're doing all this research,
and that's where that's where this thought experiment of like, well, to
some degree you're limited by the cloud provider allowing you to easily get like everything
worked over to some degree, have those limitations. So then, given that's
the world we live in, how do we get as close as possible?
And that's kind of the design work that we're looking at right now. Well,
I think there's totally a gap here in the IAIC world where it doesn't
feel like any of the offerings are super compelling. There's I mean, I
think anyone that assigns themselves a DevOps title has a unique perspective on complaining about
literally everything that exists. So you know, of course the tools that we
have, you know, are not as good as they should be. And
I think there's always a space for someone to say I wrote a better terror
form or whatever, because I feel like that's sort of the abstraction layer that
exists over the cloud providers. And I'm not saying that anyone should go and
create a new is C tool. I'm sure someone just got that idea right
when you can do that, you absolutely can do that and wrap the SDKs
out there. But I do I do feel like that there's always there is
definitely a room here to grow in that direction. Yeah, and then that's
it's kind of a maybe a fork in the road. But the two flip
philosophies is like create that layer so that it's very easy to do those forks.
The other one is, I guess it's you call it like the honeycomb
or the launch Darkly philosophy, where it's like, can you build tools that
make production just safer, like feature flags, so now just don't worry about
testing or like like so much. Just get into production under feature flag and
then and then just turn it on for a certain set of customers and then
you'll figure out how things work or like deserveability thing. Instead of focusing so
much on like fourteen year staging and testing environment, you just go into production.
We're going to monitor everything so closely that we're going to minimize any kind
of damage. And there's I think there's gaps on both sides, and there
the different engineering leaders and platform leaders have different opinions about about which way they
want to go, so it's kind of a lots of different picks on.
I mean, I definitely have an opinion on that. Realistically, it would
be nice if we could get to that world. I don't think the cloud
providers offer us the sort of control over the environment that would allow us to
perform like one out of every million requests. Yeah, go and changes.
I just feel like we're much better off pulling production down to another environment to
validate that the code that we have is exactly executing as we want, and
we can do it, you know, creating synthetic requests from our production environment
and replicating that against the ephemeral, dynamically created dev one. This is a
perfect example of what we're seeing is Warren who is like, let's get things
synthetically mocked, Let's get everything running locally, Let's get it as close as
possible, make sure things work, and then we have Will who wants to
film Upbraud team VIM right here. I think valid approaches about valid approaches.
People have different appetites what they want to do. No, I mean,
there's still definitely things where I'm a huge proponent of pushing stuff to testing in
production, and we do that under certain circumstances where we make sure that we
wrapped. Like parallel execution is a huge thing for us. When we're changing
some of the business logic of how something works, we'll usually execute two different
paths actually simultaneously in production and validate whether or not the outcome is exactly as
it should be, and before turning off the legacy one and when you do
that, When you do that, does that output still go back to like
your fork traffic or do you know it's in process? Like it'll basically the
equivalent of having two threads. You know, you call this method and it
does two things at the same time, and then we throw away the new
result until we like we validate it, and then we throw it away and
if we get any invalidations, then we you know, that's an alert that
someone goes and looks at and comes back and says, Okay, you know
what happened here. And if we don't get any alerts after a period of
time, then that sort of level of testing has been completed and we can
just delete the old code and own and actually return the new stuff. I
love that setup. I think that's I think that's great. I think I
think that's like decently. I mean, it's it's a far cry from like
VIM and prade, but like it's almost a step towards that. Right.
It's like like you're at least able to get something into production more safely,
even though it's I imagine there's constraints on like you wouldn't be able to do
that if there was like very secure data that was going into the response that
you're like, you shouldn't see. You wouldn't be able to nidate it that
well, well, I mean we we sort of had that because we're an
identity provider federated and aggregation and no one sees the data. It's just you
know, hey, there's something here that doesn't match, and you know,
you can sort of revalidate that and realistically the actual who the user is doesn't
really matter in those circumstances. You can just sort of see what it looks
like. If for some reason there is some concern. We encrypt the logs
that get written even before they leave the production environment. We hash the data
that we'll get seen automatically by any of our engineers before it leaves. So
I mean, very rarely it's the case that that actually, like what someone's
last name was was relevant for us. That's a very cool setup. I
like that. So that's some discipline there, though, Like that's very that
sounds to me like that sounds like a very disciplined approach because you're being very
deliberate about what you're testing, writing that second path for it, and then
you're having to specifically define and measure what the success criteria for that is.
So that's a charitable way of looking at it. You know, maybe that's
my na You don't do that at all. I mean there was nobly a
time in my life where I'm like, you know, you're releasing that code
to production. Why is there an arbitrary try catch block you know around that?
But there is another aspect here when if you're if you care about reliability
and that your stuff don't really go down and production and the code you're changing,
like there are too many unknowns there. Realistically, you didn't test literally
every case, and sometimes it's not so critical that you have tested everything,
but you know, validating something happens in production, like oh, we don't
know if a customer is using this. Well you can at least create a
log message that says, uh, hey, customer is using this thing,
don't turn it off yet and run that for a bunch of weeks so that
you have the data to validate. Uh yeah, I mean you laugh,
but you're not if you don't take that first step, Like this is absolutely
the advice that someone could be utilizing to improve their decision making apility, not
like some spreadsheet with a list of features that are on or off per customer,
but really, like what is happening in your production environment, and this
is just the next level after that, which is release the code, have
multiple versions running at the same time, measure performance, reliability, you know,
consistency, whatever that is. This is just how we've happened to be
doing and I feel like it works really well for us. Unfortunately, so
you know, we're not going to migrate to something else. I don't know.
I haven't heard anyone else doing something that you know really changes this.
I mean there's blue green deployments right where you're siphoning traffic off from your new
version. It sometimes it doesn't really make sense a lot when you have a
lot of features going to production over time, where the features especially aren't interacting
with each other. Having them weigh in a queue to get through this and
then deciding to roll back or not is a huge expensive process. As you
said, you know, feature flags can work here, although that's again a
switch. So really parallel execution has been the only thing I can think of
that really made sense for us early on. And how do you think about
how that compares to like a canary environment. I mean, this is sort
of the same thing, like when you if you have millions of requests that
are that are coming into your service all the time, then some number of
people would be affected by any change, no matter how small. And if
you have the budget, which is you know, one of the key terms
that we talk a lot about in observability, then you know, go for
it, right, you know you don't need to take some extra steps.
But if it's just a matter of you know, changing a bunch of parameters
somewhere and you want to see how it validates it, especially when you have
all the business logic on your side already, right, you know you're not
calling a third party service, you're not doing some other things like if you
can really test it all in your service in production at runtime, why not
do this to validate? Oh was that hard to set up? I mean
you really, like, you know, think of you're changing a method,
you know, any any engineer out there. You're changing the internals of some
method. All you need to do is duplicate that method down to a second
block, call it V two and have the caller execute both of those things,
wrap the other one in a trycatch block, and then validate the outputs
in your code and if it doesn't work, you know, log a message
and that's it's not It's honestly not a lot there. You can create some
structure and around that with a framework where you can wrap you know, parallel
execution and then pass it into methods and have it automatically do it. It's
not a lot of extra overhead if you feel like you're doing this very very
often. I think this happens, like one out of ten features that gets
implemented goes in like this. It also gives us some security around it,
so I wouldn't say it's complicated. I think a lot of engineers know that
this is a great a way that you can test and this helps I think
junior engineers, you know, they navigated this right, you know, throw
a try catch around there. I'm scared about this affecting production. I think
bringing that fear back in it, you know, helps us. You know,
thinking that our code that we write is invincible is the problem here?
Yes? Yes, So can you just share your screen for us? Warren,
I'm kidding it's really boring right now. I mean, when I'm doing
the podcast, I literally have a notes page and my audio equipment uli up
in running and that's it. There's nothing else on my desktop right now.
We are. We're completely different. I never I never closed windows, and
never reboot. That's exactly what somebody who wants to be proud would say.
Well, I saw a meme yesterday and it cracked me up, and it
said, damn, I just accidentally closed that browser tab that I've been meaning
to read for two years. It was very careful about my usage there that
control shift T brings that up or the previous window that I've closed, you
start all my tab Yeah, for sure. That. Like I was using
x marks on Firefox back in the day before they bought into it, because
it's just so critical for me that I can actually get that. The worst
one I have is emails in my inbox because when you click archive accidentally or
delete like at that email was months old. Good luck finding what that was.
We have a security polow see that automatically deletes our emails after a specified
period of time. So email email management is automated for me. Cool.
So what well on critosis, have you found an area where it's not a
good fit. Yes, definitely crotosis with where we are right now, if
you are able to very adequately express your system with a doctor composed file or
a HELM chart with a relatively small helm chart in the sense of like your
helptot rebo doesn't have some dozen lines in it, it's got like, you
know, the same number of configuration settings. If you have those two things,
you're you're doing great. Krotosis is not going to be great for you.
It's the situation where those things have to be augmented by the seven thousand
of my mast script, the bunch of different documentations on how to configure these
things. If your doctor post file has or your home charts have a bunch
of lines commented out that say if you want this behavior, please uncomment these
lines, and it's all over the place. That's where that's where you should
think a look at prtosis uh, and that's where we're going to provide a
lot of value at the moment. Right on excellent. So you mentioned doctor
compose and HELM So I think I can safely infer there that you're a big
favor of containerization. Yes. Is that specific or is that a result of
like this is what you're you learned from your customers or from the beginning,
where you just like containers all the way I mean, I would love to
say it was entirely deliberate. Uh, and from like the very beginning.
Definitely in the beginning it was like, hey, this is the standard that
people are using to ship these things. I think it's a great abstraction.
I think containers. I think it's wonderful to embrace the world of containers.
I think right now, container right now, When people think about containers,
they often also think about the Docker run time, the Docker build process,
and so like the of a container, the thing about not only like the
idea of the container, but also like what does it look and feel like
when you build everything inside of the Docker file, which is like separate from
having a container. So like what I love is the container abstraction. I
don't so much love. It's very easy to bloat the Docker file. It's
very easy to blow the container so that it's like significantly bigger than it really
needs to be. There's a lot of stuff to improve there. But when
it comes to like how you wrap things up, the ideal world is like
containers as lightweight as possible, wonderful. The only issue is it is pretty
It is pretty annoying to get in and out of containers and deal with the
like is this a port that is exposed inside of the container, or is
it on my laptop or is it like like which port? Is it?
At what level of networking? It's pretty annoying, and it can be.
It can be also difficult to map between filesystems and different containers to your own
file system. When you have mounts mapped or not, that stuff can be
tricky. A lot of the abstraction that we provide in our like run time,
tries to make that stuff better when it comes to interacting with the def
environment. But I think really the the reaction folks have against containers when it
comes to environments, a lot of the time it comes from them being perceived
as heavyweight, which is often true. If you're running a doctor it often
is true. It's like there's a lot of there's a lot of stuff going
on there. But a lot of the time it doesn't have to be heavyweight.
It's just that the way we build containers now tends to lean towards a
more heavyweight approach. No, I think you nailed it there. Realistically,
it's like this, there's no hobb load automation. When you make a code
change, how do you get the necessary like auto compiling and run like you
just want to sort of debug it as soon as possible. I think it's
a small like that's suset of things. And then there's this other set which
you for sure mentioned like what if you're changing something at the container level right
like to You're definitely going to make some changes and you're going to need to
test that in some way. I don't think I know anyone that wrote a
Docker file and it was right the first time or the second time or the
third time. I did learn about this thing that you can like now make
a builder out of it, and that can really help reduce the size of
your containers by not storing ninety percent of what's necessary there. Like, I
still feel like that's really esoteric, and most people are not aware that the
overcon container initiative supports that sort of reduction. Are you talking about like using
like a multi stage Docker build where you run your build process and then store
that as an artifact and then launch a new Docker in it. It was
actually in the door one thing, right for sure? In multi stage No,
there's actually a in the darker file you can specify multiple images and one
of the images you can say is this is my builder, and then have
the tools in there, and then another image lower down that says, this
is the actual source image I want, because there's just so many run time
versus execution time problems that come up, where in the previous world we were
in a state where they're the same thing and it's really expensive, but now
we're much better off. Like you can get rust containers down from gigabytes to
really megabytes because the builder is big and expensive, but the actual built image
is just pretty much a wrap binary. Yeah, that's the crazy thing,
especially in the area of like build optimization. There's so much low hanging fruit
everywhere, but it it's all different low hanging fruit, and you have to,
like you have to do some degree of looking into it to figure out
that, oh, you can grab the thing and put it in here,
and all of a sudden, I forgive your buys the megabytes. But it's
so common to hear that, oh, I brought my build time down from
ninety seconds to nine seconds, or I brought it down from five minutes to
one minute, and it's these are massive improvements and you build all the time,
but they're right there in the city there, but there's no it's not
often connected, right, Yeah. I think a lot of For a lot
of people, there's just they've got they've done the build process and it took
a long time, and they just like, oh, I guess that's the
way it is, not realizing that there are a lot of knobs you can
tweak to change that. Yeah, And I think I mean, depending on
where you are on the build process. If you're if you're on your laptop
in your building and you want it to be like super fast, and if
it's two minutes, it's going to be a bit annoying. But if you're
if you're merging your pr and you're waiting for it to go to a staging
environment, like if it's if it's five minutes, you're probably okay with it.
You're probably like, Okay, this is fine, and try to get
it from five to two minutes. It's probably like not that big of a
deal for you because you don't do it that often. So then like when
would you ever want to optimize that? But across all the debts and all
the mergers, that's it adds up. It's a lot of time, but
you could have gone faster for sure, well, sweet, is there anything
else we should talk about? I feel like we kind of just ran the
gauntlet here we did. So if folks are interested in learning more about Kurtosis,
what where do y'all hang out at online discord? Well, first of
all, I recommend going to our GitHub. The GitHub is the place for
you enter. So like GitHub, it's got our code, it's got our
docks there, it's got the reading me. So if you want to learn
my business, go right to the getthub. And then if you want to
chat with us, we're always chatting with the community, So join our discord
and we'll be there. You can DM us, you can chat with us
in the general channel. It's fun right on cool. Yeah, and I've
been chatting with y'all a little bit. Y'all are you really are super active
and super responsive online, which is nice and it's refreshing to see. It's
the fun part. The fun part is chatting with the users. See,
that's just so anti tech right there, Like, like I would love tech
if it weren't for the damn users. I feel like there's the difference of
you know, the outward appearance versus what you're thinking when they're actually saying things.
And it's also the users that you're talking with too. You know,
when you're talking with with people who are doing the same thing, you know,
whether they're software engineers or whatever. I think that's a different experience than
trying to talk mod through getting her dial up modem connected and clearing out her
Internet Explorer cash or whatever. Yeah, I mean, I think that's definitely
difference between support right for something and real engagement with your customers. Yeah,
for sure. You know, I usually come away from them like, oh,
no, we should have that feature. I can't believe we didn't think
of it. I mean, one of the fun things about having a developer
tool is it's buy engineers for engineers. There's a lot of empathetic connection across
the the product user barrier, where maybe if we weren't the exact same profile,
you'd have to you have to bridge the gap somehow. Well. You
know, I've heard many times that that's what makes us successful startup is whenever
you're solving a problem that you personally have. I totally agree with that.
Yeah. Cool, Well, Gayalen, thanks for being on the show man.
This has been a really cool talk. Thanks for having me. Cool
Warren thanks for joining me, welcome back, Glad to have you back,
kind of show, glad to be here. And for all of the listeners,
thank you for listening. Really appreciate it. Hope you found it useful.
Oh right, yeah, yeah, picks, all right, you brought
it up. Kick us off oring. I just you know, I know
some listeners were going to be mad if you know they didn't hear on the
report. Yeah, So mine this week is Turn the Ship Around by David
Marque, which is a book that, irrelevant of your experience and technology or
why you're focused on either core engineering or somewhere else, can really help take
your understanding of how to work effectively to the next level. It's really about
leadership in every way, starting from the bottom all the way up, and
how to think about it. And I think it's really it's really interesting to
read, and if you feel like you can't commit to doing that, you
can find ten minutes to watch the shortened video online. It's something of greatness.
I can never win the name of the title. It's a YouTube video.
You can find it, which is it's a hilarious and it's really good
content right on. Awesome Galen putting on the spot. What'd you bring for
a pick my pig? It can be it can be a music Oh for
sure. Yeah, so I have been very into jelly Roll lately. I
don't know if you guys know jelly roll country singer. Oh yeah, yeah,
yeah, he's he's uh the country post Malone. Yes, yes,
that's hilarious. I've never heard it before, but as soon as you said
it, I knew that you were thinking of the right person. Yes,
Country post Malone. Jelly Roll is amazing. I've just been It's rare that
I find an artist. And then I just listened to a dozen of his
songs on a regular basis. He hit me with, hit me with all
of them. So if you haven't checked out jelly Roll, if you're into
country, if you're into hip hop, it's a really nice combination of both.
So for sure, Yeah, I last year I stumbled across hardy and
which hardy is like country meets heavy metal or or hard rock. And then
at some point he did a collaboration with jelly Roll, which took me down
that path. Yeah, so I'll agree with you on that one. That's
pretty cool and it's you know, it's definitely not what you would expect from
country music. It's it's quite a bit different. I'm not sure Hank william
Senior Senior would approve of it, but here we are cool. So I
brought two picks this week. The first one is going to totally destroy my
nerd credibility because I have never read the book Dune or seen the movie Dune.
So I am starting to read the book Dune, and dude, maybe
one of the best books I've ever read. I mean, the things that
that Herbert does in the book is just unmatched by other stories realistically, like
that you can tell us by the number of new concepts and things that were
just made up. It's not like a world that just easily imaginable, right,
And I'm looking forward to it and I've I've heard from multiple people.
You know, it's a common saying, oh, the book's better than the
movie, but I've heard from from multiple people like, no, No,
the book is better than the movie on many, many levels. So I'm
looking forward, looking forward to getting into that. The other pick I'm going
to bring up is I kind of just lost I kind of fell out of
love with Chrome as my browser, and I wanted to see what else was
out there, Chrome, It's not you, it's me. I just felt
like to see other browsers across the ARC Browser, and I think I completely
locked up here there you I did here may be back. I got completely
kicked out for talking trash about Chrome, didn't I. I'm going to go
through with it anyway. The ARC Browser has been really cool. I've been
using it for about a week now, and there's so many cool, subtle
little features that I didn't realize belonged in a browser. Like my favorite one
is when you're in a Google meet, you know, and you navigate off
to another tab, and then somebody asked you a question. You can never
find which tab is the one for meat, right, So then you got
this awkward silence and you gotta admit that you were screwing off and not paying
attention to the meeting. Well, with the ARC Browser, when you leave
the tab, it just puts a little thumbnail in the corner of your screw
with that video meet still going. So when you never have to go back
to it, it's just down there in the corner, you click into it
and it pops you right back up. And I just thought that was super
cool. So that's my second pick for the week, and with that,
I think we're done here. Galen. Thanks again, Ben, this has
been pretty cool. I've enjoyed it. I hope the listeners enjoyed it.
Thank you for listening. We appreciate your support, and yeah, I hope
see you all next week.