Speaker 1: Hello everyone, and welcome back to another episode of Adventures
Speaker 1: in DevOps. And I'm joined today again by my co
Speaker 1: host Amy Knight. How you doing, Amy?
Speaker 2: Pretty good? Walking in?
Speaker 1: She likes walking, that's uh.
Speaker 2: I gotta get my steps in.
Speaker 1: I don't have a good strategy for that yet. Yeah,
Speaker 1: so today I have to be completely honest. The topic
Speaker 1: is in and around finnops. But I don't really know
Speaker 1: what finnopps is. But if it's anything like DevOps, I'm
Speaker 1: guessing it means firing all your business analysts and hiring
Speaker 1: some additional software engineers to do a job they don't understand,
Speaker 1: and then, after your years of failing, reappropriate the name
Speaker 1: to mean something else and claim success. Well, Amy, you're
Speaker 1: already smiling, so maybe you know something I don't know.
Speaker 3: I don't actually laughing because it's very entertaining.
Speaker 2: And I also.
Speaker 3: I think, like last week, I realized your title on
Speaker 3: LinkedIn a tech entertainer.
Speaker 2: So that was a very entertaining intro.
Speaker 1: You did a great job, well, thank you. So we
Speaker 1: pulled an expert from the field, chief strategy officer at
Speaker 1: cloud Bolt, Yasmin Rajabi, and I've noticed here so you
Speaker 1: have a history of product management, and so I'm really
Speaker 1: excited to have you on the show.
Speaker 4: Welcome, thanks for having me excited to be here.
Speaker 1: How completely off base was my thoughts about what finof says.
Speaker 4: I mean, not so far off base.
Speaker 5: Any time you put like an ops inside in one
Speaker 5: of the words, you're obviously trying to force some things together.
Speaker 4: And it's funny.
Speaker 5: A question I get asked often is like, okay, so
Speaker 5: how are finnops teams and DevOps teams coming together? And
Speaker 5: the OPS in pinnops is supposed to be for DevOps,
Speaker 5: so technically like they're supposed to already be together. They're
Speaker 5: not supposed to be separate teams. But there's always the
Speaker 5: ideal and then the reality of if you take financial
Speaker 5: analysts and you take engineers and you want to bring
Speaker 5: them together, they speak different languages, they come from different worlds,
Speaker 5: So there are folks that are trying to use software
Speaker 5: to bring that together. I'd put us in the mix
Speaker 5: of that, but it's still like people from completely different
Speaker 5: worlds and trying to get them to think and care
Speaker 5: about the same topic.
Speaker 3: Okay, I was going to add too, we also need
Speaker 3: to add in is it greenops. So it's like getting
Speaker 3: environmentally savvy as well within the realm of finnops.
Speaker 5: Okay, well, I realized that I need to reduce the waste,
Speaker 5: but I want to be able to measure that from
Speaker 5: a like my carbon footprint standpoint. And it was interesting
Speaker 5: I was talking to a Phinnops person that is using
Speaker 5: those metrics to drive people to care inside their organization,
Speaker 5: because the people in the organization is like, Okay, well,
Speaker 5: it's not my money, I'm not spending it. But when
Speaker 5: you actually tie it to the environmental impact, the kind
Speaker 5: of human nature part of people helps kind of get
Speaker 5: people a little bit more passionate about reducing the waste
Speaker 5: inside their inside their organization.
Speaker 3: As far as the step, I did kind of lap
Speaker 3: it off before.
Speaker 2: But I saw.
Speaker 3: Something recently about like as I're putting more and more
Speaker 3: data centers up, like the water shortage that's happening to
Speaker 3: the communities around them.
Speaker 5: Some cities are literally like their grid cannot handle any
Speaker 5: more data centers, data centers coming online, and like they're
Speaker 5: starting to buy in different area, Like it'll be interesting
Speaker 5: what things look like in ten years.
Speaker 3: Like I saw like these poor families like literally like
Speaker 3: they can't run their appliances as they could like a
Speaker 3: couple of years ago.
Speaker 2: Yeah, anyway, Okay, So I.
Speaker 1: Think that's a really interesting topic because we've had a
Speaker 1: lot of our previous guests on the show, either they've
Speaker 1: been AI focused in the last few months or they've
Speaker 1: had a unique insight into data center operations. One of
Speaker 1: the ideas that keeps coming up is that it's actually
Speaker 1: not a struggle to get energy reservations for the data centers.
Speaker 1: But it's interesting that you bring up the water impact
Speaker 1: because that's a new thing that I'm not familiar with.
Speaker 5: Where it kind of interplays with water is probably less
Speaker 5: of my expertise and background, just the fact that you know,
Speaker 5: you need the water to cool down the data center.
Speaker 5: The more the more power it's using, then it becomes
Speaker 5: it's just like a cycle.
Speaker 4: Though I would say their.
Speaker 5: Ability to use liquid cooling and the data centers reuses
Speaker 5: the water instead of an endless supply of water. The
Speaker 5: recycling the water that exists has at least helped make
Speaker 5: some of an impact. What's interesting is people are pulling
Speaker 5: these metrics out and trying to overlay them on their
Speaker 5: usage as well. So, Okay, I'm writing an application I
Speaker 5: deploy it. I never think about these things, but as
Speaker 5: I can start to pull some of those metrics out
Speaker 5: of either my cloud provider as they start to provide them.
Speaker 5: I think Microsoft started to pull in like carbon data
Speaker 5: and other metrics that you can pull up or if
Speaker 5: I'm running my own data center. Then being able to
Speaker 5: pull those metrics out then kind of overlays and says, hey,
Speaker 5: here's the impact of what you're doing, Like make sure
Speaker 5: you're setting these things correctly. Otherwise there's there's a lot
Speaker 5: bigger impact than just financial.
Speaker 1: What's a good example of needing to pull in accounting
Speaker 1: or financial operations into software.
Speaker 5: So I'll say for most of my background in this
Speaker 5: space is on the Kuberneti side, and when you're deploying
Speaker 5: a Kubernetes app, you need to set requests and limits,
Speaker 5: one from a reliability standpoint to make sure you actually
Speaker 5: have the resources, but two typically what people do is
Speaker 5: just like set them really high so they don't have
Speaker 5: to think about it, and then deploy the application. And
Speaker 5: it's really hard to understand the impact of that, especially
Speaker 5: because that's at the pod level and you pay for
Speaker 5: nodes and you're like, okay, well, I don't really know
Speaker 5: how my pods translate into my nodes. So I'm paying
Speaker 5: some amount of money, but how do I actually know
Speaker 5: the impact? And so being able to pull in both
Speaker 5: the billing data and then actually allocate it correctly, because
Speaker 5: one of the challenges in Kubernetes is, Okay, how are
Speaker 5: you going to allocate your costs? Do you just look
Speaker 5: across name spaces and say like, okay, I'm going to
Speaker 5: divide it by name spaces and then that's what you get.
Speaker 5: But what if you have a larger name space or
Speaker 5: you have how do you split out like cube system
Speaker 5: and all of anything on the control plane. That's some
Speaker 5: of the challenges that people have. And if you're what
Speaker 5: you need to be doing is pulling in that billing
Speaker 5: data to say here's how it's costing you, here's how
Speaker 5: we can split that out if we want to split
Speaker 5: up the pod level, and then you can be able
Speaker 5: to tie that back to the actual nodes that you
Speaker 5: pay your bill on.
Speaker 1: So would it be accurate to say that Historically maybe
Speaker 1: software engineering teams haven't been held accountable for how much
Speaker 1: they spend in.
Speaker 5: The cloud for sure, And often what happens is the
Speaker 5: developers are setting those requests, and then the platform engineering
Speaker 5: team are the ones that are like, Okay, well we're
Speaker 5: managing the cloud environment, we're managing those bills. They get
Speaker 5: asked to do something about it, and I don't know
Speaker 5: the app. What do you want me to do? If
Speaker 5: I touch something, I'm going to get a phone call
Speaker 5: from probably not a phone call, slack message from a
Speaker 5: developer of like, hey, what do you do to my system?
Speaker 5: It's down, it's having challenges. You are the last one
Speaker 5: to touch it.
Speaker 1: Amy having experienced there.
Speaker 3: Yeah, this is a very real problem. I would say,
Speaker 3: like the requests and the mins also, and I think
Speaker 3: like if a scale gets large enough, you also need
Speaker 3: to understand like traffic patterns and scaling appropriately for that,
Speaker 3: because you know, if you're an early stage company, you
Speaker 3: can just kind of scale at like a you know,
Speaker 3: relative level and stay there. If the app really grows,
Speaker 3: which hopefully it does, you're going to be hit with
Speaker 3: a like, very realistic problem. It's going to cost a
Speaker 3: lot more money to be running your app at you know,
Speaker 3: nine am versus midnight if it's a global application, to
Speaker 3: get into having this spread apart things and allocate resources
Speaker 3: appropriately versus like the distribution of the request coming in
Speaker 3: and things like that. So it's a very complicated problem, and.
Speaker 1: Most of the time the people that have been sort
Speaker 1: of on the chopping block for figuring this out have
Speaker 1: not been the engineering teams who understand what the application
Speaker 1: is doing. So how do you get that back, Like,
Speaker 1: how do you align the incentives appropriately so that if
Speaker 1: the amount of money you're spending on your cloud provider
Speaker 1: or on capital allocation for on prem data center resources,
Speaker 1: how do you align that back with the software engineering teams.
Speaker 5: I think a little bit of it is reporting, just
Speaker 5: giving them visibility. Some of our customers call it shame
Speaker 5: back reporting. I'm like, yeah, I get it, but maybe like,
Speaker 5: don't call it that. But I think on the other hand,
Speaker 5: it's also just giving them the tooling and the insights
Speaker 5: that they know, they have the confidence in the systems
Speaker 5: that are kind of handling this for them. I really
Speaker 5: don't think you're going to get far if you go
Speaker 5: to a developer and you're like, hey, you need to
Speaker 5: change your configuration settings. This is costing us a lot.
Speaker 5: They're almost never going to be incentivized to do anything
Speaker 5: about it because their goal is deploy new applications. Move
Speaker 5: the business forward. And so I think I would say
Speaker 5: it's harder to incentivize them to do the work than
Speaker 5: it is to give them the tooling that allows the
Speaker 5: work to be done for them, while they can get
Speaker 5: input into the tooling. So like, if you say, hey,
Speaker 5: I'm going to do this for you, you have no insight,
Speaker 5: no ability to override anything, no ability to constrain it.
Speaker 5: They're gonna be like, no, you can't do that. For example,
Speaker 5: you can take in their concerns and codify that into
Speaker 5: the system that's handling doing the fixes for them, then
Speaker 5: you're there're a lot more incentivized to kind of move
Speaker 5: this forward because it's less work for them and they
Speaker 5: get the better outcome.
Speaker 1: So I can see like one of the core problems
Speaker 1: here is that maybe the team that's fundamentally responsible for
Speaker 1: managing the application like building it and running at the
Speaker 1: development team DevOps team, how how do they get the
Speaker 1: knowledge for what should be reasonable as far as spend goes,
Speaker 1: I do s disconnect, like do you have any I mean, it.
Speaker 5: Definitely depends on the organization of like how much they're
Speaker 5: spending because scale's different, right, Like a very very large
Speaker 5: bank is going to be spending a lot more than
Speaker 5: a ten person startup with not always because sometimes, especially
Speaker 5: with AI tools, ten person startups can be spending a
Speaker 5: lot of money. But the metric that we usually look
Speaker 5: at is waste percentage, So of everything that you're doing,
Speaker 5: how much of it are you actually using? And it's
Speaker 5: usually a better metric to go to folks and say,
Speaker 5: do you know you're like wasting seventy percent of the
Speaker 5: resources that you are paying for? So we don't go
Speaker 5: in and say, like, you should be spending less here
Speaker 5: and there, because sometimes you want to be spending more.
Speaker 5: It's just are you efficiently using the dollars that you're spending.
Speaker 1: Do you think that may optimize for the wrong thing though,
Speaker 1: that if you encourage teams to have high capacity utilization
Speaker 1: for their pods, nodes, or if they're using something else,
Speaker 1: let's say virtual machines, just like, oh the CPU is
Speaker 1: only at five percent, get we should use a much
Speaker 1: smaller machine to achieve the same thing. Optimizing at that
Speaker 1: level maybe optimizing the wrong part of the application, Like
Speaker 1: maybe the issue isn't that the utilization is too high
Speaker 1: or too low, but the thing that they've built is
Speaker 1: just wrong altogether.
Speaker 5: Yeah, Yeah, oftentimes it can also be just the app itself,
Speaker 5: the way that it's architected. And even sometimes when you
Speaker 5: write size, then you end up with a smaller but
Speaker 5: higher number of pods, and all of those pods are
Speaker 5: going to be making network calls. So, like, it's a
Speaker 5: lot more complex than just looking at resources. You do
Speaker 5: have to look at the whole system, and so you
Speaker 5: have to look at the actual application itself, how it's
Speaker 5: consuming metrics, Maybe it needs some changes within the application.
Speaker 5: What can you tweak in tune there there's no like
Speaker 5: one size fits all, here's what's going to save all
Speaker 5: your problems. But it's what can you start with and
Speaker 5: what's the lowest hanging fruit? I mean, honestly loose hanging
Speaker 5: fruit for most people is just shut off systems you're
Speaker 5: not using. And it's surprising how often that is the
Speaker 5: majority of the waste in an organization.
Speaker 1: That's that's actually surprising to hear, because I feel like
Speaker 1: Corey Quinn, if you're familiar with Internet personality, often jokes
Speaker 1: that the highest cost for you know, utilization is your
Speaker 1: backup buster or your you know, your database backup. You
Speaker 1: know you're never using that, And yeah, there's you know,
Speaker 1: a huge cost associated with that. I mean, there is
Speaker 1: something to be said like understanding your utilization patterns to
Speaker 1: be able to eliminate the waste in that way. Is
Speaker 1: that really, like, are we don't going to like fifty
Speaker 1: percent of the cost is just you know, right sizing
Speaker 1: and then you're done.
Speaker 5: Honestly, typically like sixty seventy percent of the waste is
Speaker 5: just it like sounds kind of a little embarrassing because
Speaker 5: you're like, it's just like set your CPU requests at
Speaker 5: your memory request.
Speaker 4: We're talking about two settings.
Speaker 5: And I feel like in the VM world that was
Speaker 5: at least a little bit easier to say, go set it.
Speaker 5: But in the container world, you now have sometimes millions
Speaker 5: of containers, and each individual container has different resource needs,
Speaker 5: and so it becomes more of a scale and like
Speaker 5: toil problem than then it's anything else. So I do
Speaker 5: think a lot of the waste, or at least what
Speaker 5: we see is a lot of the waste is just
Speaker 5: in right sizing. But honestly, there's also so we track
Speaker 5: you mentioned storage, We track kind of unused storage and
Speaker 5: backups that you haven't touched in ninety days, and the
Speaker 5: amount of pocs that we come across and we're like,
Speaker 5: just just turn this on and see what happens. People
Speaker 5: are like, oh wow, like I got a lot of
Speaker 5: waste there that is just easy. Go remove it, delete it,
Speaker 5: turn it off. It's surprisingly more often than I would
Speaker 5: have expected.
Speaker 3: Oh no, I'd say, I'll add to another problem I
Speaker 3: have seen. So it's also not just like I said
Speaker 3: it and forget.
Speaker 2: It kind of thing. I have seen.
Speaker 3: Issues where you have like a portion of the app
Speaker 3: that is maybe not touched as often as application grows,
Speaker 3: Like if they're only deploying it every six months, suddenly
Speaker 3: what you've requested like if you do a redeployment, but
Speaker 3: if you don't kind of babysit that you can run
Speaker 3: into problems too because it changes.
Speaker 5: Yeah, if you don't have policies in place that are
Speaker 5: continuously looking at this like okay, great, you were able
Speaker 5: to either write size or remove or downscale once, but
Speaker 5: then your application is going to change. And you mentioned
Speaker 5: like apps that change every six months, some change daily,
Speaker 5: so how are you going to stay on top of that?
Speaker 5: And one of the other things you had mentioned earlier
Speaker 5: was just horizontal scaling of you know, you need more
Speaker 5: resources at nine AM. Tools like the HPA, like CATA
Speaker 5: are great for that because they see requests are increasing. Okay,
Speaker 5: I'm going to give you more replicas allow you to
Speaker 5: scale horizontally, and that's awesome because it'll always help you
Speaker 5: keep up with traffic demand. But if you're not scaled,
Speaker 5: if you're not at your configurations correctly, you're just duplicating
Speaker 5: waste across the board. And it's very good from a
Speaker 5: reliability standpoint, but not very good from a waste management standpoint.
Speaker 4: So you kind of have to.
Speaker 5: Take the vertical approach alongside the horizontal.
Speaker 1: I had a train of thought, and I guess I
Speaker 1: got stuck on this a little bit. There. There is
Speaker 1: this aspect of I know, especially engineers in this area
Speaker 1: tend to want to optimize everything and and part of
Speaker 1: that there's a lot of products that come out. So
Speaker 1: I can imagine that someone wrote like some sort of
Speaker 1: AI based product that like you deploy into you Kubernetes
Speaker 1: cluster and automatically tries to write size all of the
Speaker 1: CPU usage and odd sizing, you know, whatever needs to
Speaker 1: be allocated. Like this seems like a genius idea, right.
Speaker 5: Are you describing storm Forge or look us up ahead
Speaker 5: of time, so we we don't use everyone's like, oh
Speaker 5: I use AI.
Speaker 4: We use machine learning, thank you, it's math.
Speaker 5: It's like that I can't do. It's not some fancy
Speaker 5: AI like we and we develop it in house. We
Speaker 5: have PhDs that work on this stuff. It's like patent pending.
Speaker 5: Depends on the audience sometimes like yeah, okay AI machine learning,
Speaker 5: but like it's not the same. And I feel very
Speaker 5: passionate about just making that differentiation because I mentioned this
Speaker 5: is a scale problem. When you're looking at the different
Speaker 5: settings at the scale, then that just means it's a
Speaker 5: really complicated math problem. And so we use the math
Speaker 5: to help solve that problem. We do look at the
Speaker 5: trends in the data, like usage patterns, HPA, scaling patterns,
Speaker 5: all of that, and then we pull the metrics straight
Speaker 5: from Kubernets.
Speaker 4: It goes into a Prometheus.
Speaker 5: Database, and then our algorithms look at those scaling behaviors
Speaker 5: and come up with what those right configuration settings should be.
Speaker 5: You could say like okay, cool that anyone can do
Speaker 5: that and come up with the configuration settings. I think
Speaker 5: what makes us unique is two things. One, the horizontal
Speaker 5: scaling behavior we talked about before where people usually use
Speaker 5: the HPA KETA. Those are the most common tools when
Speaker 5: it comes to Kubernati scaling and have to work with
Speaker 5: those tools. You can't work against them because if you're
Speaker 5: going to go in and vertically right size a container,
Speaker 5: for example, and it's horizontally scaling, then the HPA is
Speaker 5: gonna be like, oh, well, you're using more of your
Speaker 5: downsize so you're moving using sorry, using more of your resources.
Speaker 5: I need to go give you more replicas. And then
Speaker 5: the VPA or any other vertical right sizing solution would
Speaker 5: be like, well, okay, now you need more resources. So
Speaker 5: you get into that turn cycle. And one of the
Speaker 5: things we look at is your horizontal scaling behavior in
Speaker 5: addition to your vertical right sizing requests, and then set
Speaker 5: those together so that you can continue using the tools
Speaker 5: that you are using that are open source, that like
Speaker 5: I would never want to ask anyone to rip out,
Speaker 5: and then also kind of reap the benefits of vertical
Speaker 5: right sizing and then the other pieces we've been talking
Speaker 5: about developers and how to incentivize them and input into
Speaker 5: it is important for that confidence to be able to
Speaker 5: actually do this automatically.
Speaker 1: You say that and now I'm just wondering if this
Speaker 1: isn't a solved problem, like why is it that, say
Speaker 1: that JVM or Kubernetes pods or even Docker containers, Like
Speaker 1: why am I even serveralist solutions if I'm running on
Speaker 1: VERSL or aws lambda functions? Why am I specifying like
Speaker 1: timeouts and CPU and memory usage? Like why is it
Speaker 1: just not automatically determining based off of I don't know
Speaker 1: the last ten years and daily load, but how it
Speaker 1: changes over week by week, month by month where holidays
Speaker 1: are based off geographic regions, Like why not just take
Speaker 1: all this stuff and just automatically figure out what the
Speaker 1: scaling capacity needs to be? Like I guarantee you as
Speaker 1: when I was an engineer, I definitely could not have
Speaker 1: figured out like how many requests I was going to
Speaker 1: get a week from now, let alone a month from now,
Speaker 1: or how is that going to change based off of
Speaker 1: whatever marketing campaigns are being run, et cetera.
Speaker 5: Yeah, So I mean the VPA that is open source
Speaker 5: comes with kubernatores allows you to do just basic like okay,
Speaker 5: look at my past, I think week and usage and
Speaker 5: come up using P ninety five with what my request
Speaker 5: should be and for like basic deployments that works right,
Speaker 5: look at what I did in the last week and
Speaker 5: then use that and set my request for me. I
Speaker 5: think some of the challenges are the VPA if it's
Speaker 5: scaling on CPU or memory, doesn't work well with the
Speaker 5: HPA what I mentioned earlier. So you're like, if you're
Speaker 5: going to choose between something that's horizontally scaling you or
Speaker 5: vertically scaling you, You're gonna go horizontal because it's more
Speaker 5: about reliability and how do you keep up with your
Speaker 5: traffic patterns versus like vpas more about how I reduce
Speaker 5: the waste.
Speaker 4: And then if you if you're an organization.
Speaker 5: That maybe has more cyclical patterns, things change over time.
Speaker 5: You want to be looking at like a quarter's worth,
Speaker 5: a year's worth of data, where you're going to store
Speaker 5: that data, How are you going to look into that data?
Speaker 5: It all costs money. So then that's where kind of
Speaker 5: vendors come into the mix. To be honest, if you
Speaker 5: wanted to build something yourself, you could, But do you
Speaker 5: want to be into the business of kind of building
Speaker 5: systems that manage your own systems or do you want
Speaker 5: to be in the business of building applications that move
Speaker 5: your business forward.
Speaker 4: So that's kind of where where where we see it.
Speaker 5: And sometimes I talk to companies that I've built something
Speaker 5: themselves and it works for them, and it's like, great,
Speaker 5: if it works for you, it works they could.
Speaker 1: I'm gonna ask a really embarrassing question here, because I
Speaker 1: am a self proclaimed not Kubernetes expert. What is VPA
Speaker 1: and HP.
Speaker 5: HPA's horizontal pod auto scaler. So HPA looks at the
Speaker 5: target utilization. It looks at your CPU and then you
Speaker 5: can set like a target utilization, say, if I'm using
Speaker 5: more than seventy percent of CPU, I want more pods,
Speaker 5: So give me more replicas copies of my my deployment
Speaker 5: if and it'll scale for you horizontally up as you
Speaker 5: need more, and it'll.
Speaker 4: Go back down as you need less.
Speaker 5: The VPA is a vertical pod auto scaler, and that
Speaker 5: looks at resources in a vertical manner of like, okay,
Speaker 5: you have x amount of CPU, so like you have
Speaker 5: maybe one hundred mili core, you need two hundred mili
Speaker 5: core or you need fifty mili core. Does the same
Speaker 5: for memory, both of them, of course, but that is
Speaker 5: what allows you to scale vertically. These are both tools
Speaker 5: that are open source, so anyone using Kubernetes can be
Speaker 5: using them. Should one hundred percent be using the HPA.
Speaker 5: There's almost never use cases where you shouldn't be using
Speaker 5: the HPA, though there are some, but they're tools available
Speaker 5: for folks and almost everyone. I think like data Dog
Speaker 5: did a survey now three four years ago where over
Speaker 5: half of their customers or prospects people that did the
Speaker 5: survey we're using the HPA, like one percent, we're using
Speaker 5: the VPA. Since then, the number's grown, but it's still
Speaker 5: majority HPA. Because it's very easy to use. It makes
Speaker 5: sense to be able to scale horse montally, and.
Speaker 1: So one of the things that I think could be
Speaker 1: interesting here is you said, basically the number one thing
Speaker 1: that should be done is maybe time based load management,
Speaker 1: so reducing your capacity scaling down when you don't actually
Speaker 1: need these resources. I think this resonates with a lot
Speaker 1: of cases, especially not that long ago, and I'm sure
Speaker 1: there are still some companies that are doing this.
Speaker 3: They have.
Speaker 1: Managed action runners for GitHub or workspace runners for for
Speaker 1: get lab or whatever they're using for CICD, and they're
Speaker 1: not Those aren't being used when no one is doing
Speaker 1: software development. Obviously in this more remote world, that's harder
Speaker 1: to predict over time now, I think. But if that's
Speaker 1: number one, and I think that's sort of the obvious one,
Speaker 1: like what's number two in number three of what people
Speaker 1: could be doing or you frequently see as problematic or
Speaker 1: you know, high cost and could easily be cut out
Speaker 1: or maybe not too easily.
Speaker 5: Yeah, So we talked about AI earlier, and just like
Speaker 5: you know, everyone's doing something AI, a lot of people
Speaker 5: have a bunch of data jobs, like they're using Spark,
Speaker 5: and so it's fairly easy to deploy jobs on Spark.
Speaker 5: And what I've seen is people will have a lot
Speaker 5: of waste in kind of similar gethub runner, like it'll
Speaker 5: go run, go do a bunch of stuff. How long
Speaker 5: does it take to go run and go do that stuff?
Speaker 5: How many resources does it take? Typically not an area
Speaker 5: you're going to go optimize because you're like, okay, well
Speaker 5: last maybe a couple of minutes, if even maybe it's
Speaker 5: under sixty seconds. But when you start to do that
Speaker 5: at scale, very similar to Spark jobs, that becomes a
Speaker 5: challenge because now you're running tens of thousands of these
Speaker 5: data jobs that are very compute intensive when they're running
Speaker 5: and then they go away, and how often are people
Speaker 5: pruning them making sure these are jobs that are actually
Speaker 5: still relevant to the application. What I've found is when
Speaker 5: it comes to AI related things, people are less thinking
Speaker 5: about how am I going to optimize this? More just okay, spence,
Speaker 5: spence bend, I need to just run it. Like GPUs
Speaker 5: come up often too, and people will ask us like, oh,
Speaker 5: are you optimizing GPUs? And my next question will be
Speaker 5: would you actually optimize your gp like do you have
Speaker 5: any initiatives? And they're like no, I just know it'll
Speaker 5: be a problem in a few years. And I'm like, okay, yes,
Speaker 5: but right now.
Speaker 4: But when people get R and.
Speaker 5: D budgets, it's just give me the fastest, most powerful
Speaker 5: system that I can put together. They're not really thinking about, Okay,
Speaker 5: is this actually cost effective?
Speaker 1: I think it's a really interesting path to go down
Speaker 1: because I know, historically, at least in my career before
Speaker 1: I moved slightly outside of hands on software engineering, the
Speaker 1: joke amongst a lot of my leadership was there's no
Speaker 1: reason to optimize costs in any way because the amount
Speaker 1: that we're spending in the cloud is just completely dwarfed
Speaker 1: by whatever else thing we're paying for. Usually it's engineering
Speaker 1: salaries by like factors of magnitude. But as we get
Speaker 1: closer to having companies with smaller size and more hardware requirements,
Speaker 1: especially given the hardware requirements for running mL workloads or
Speaker 1: data jobs or specifically running models open source or otherwise,
Speaker 1: there is a huge increase in cost there per engineer. Right,
Speaker 1: And I don't think the mental I think the mentality
Speaker 1: has actually gotten worse. Right, Like it used to be
Speaker 1: much more conservative when like which tools we use, which
Speaker 1: cloud writers will use, you know, what will we write?
Speaker 1: And I feel like now it's gotten way more liberal,
Speaker 1: like no, go as fast as possible, pay as little
Speaker 1: attention to this. Are you still seeing companies be like okay, no,
Speaker 1: we actually need to think about this methodically, like we
Speaker 1: know it's going to cost a lot to run our
Speaker 1: models or is it really just a VC race?
Speaker 4: It's definitely a race for sure.
Speaker 5: Like every time I open up LinkedIn, I get these
Speaker 5: posts about like these companies that had two employees or
Speaker 5: ten employees and achieve this much, And I'm like, yeah,
Speaker 5: I wonder what they're spending on the public cloud providers.
Speaker 5: But we do get the question, so it's rarely are
Speaker 5: people like I actually want to do something about it.
Speaker 5: But we talked, you know, we kicked this off talking
Speaker 5: about finnops and DevOps and bringing that together. The finnops
Speaker 5: teams are the ones that are asking the question. When
Speaker 5: I was at fops X a month or two ago,
Speaker 5: I got the question about GPU optimization often of like
Speaker 5: how do I manage to spend here? And the thing
Speaker 5: I didn't want to say is like, honestly, your AIML
Speaker 5: team is just going to spend whatever they want, because
Speaker 5: is anyone giving them that guardrail of don't spend. But
Speaker 5: it is great that the phinops team is thinking about
Speaker 5: it because they know this is going to be a challenge.
Speaker 5: And it starts with visibility because as they are building
Speaker 5: the case of hey we should maybe maybe put some guardrails.
Speaker 4: I'm not saying go go.
Speaker 5: Tell the team they can't spend any money, but like
Speaker 5: some guardrails, it's okay. Getting the visibility is the first
Speaker 5: step to that.
Speaker 1: Is it just ACYNC background jobs run by quote unquote
Speaker 1: data teams that are, you know, trying to make the
Speaker 1: business run or I think it's definitely.
Speaker 5: A mix I'll even just like talk internally about cloud Bowl.
Speaker 5: So we talked a lot before about the machine learning
Speaker 5: that we do, but we also do have like we're
Speaker 5: working on a chatbot where you can ask questions about
Speaker 5: your infrastructure, and I get this question off and from
Speaker 5: people it's like I don't want to put them into
Speaker 5: the reports and have them figure it out. I just
Speaker 5: want them to act. Now everyone has learned how to
Speaker 5: interact with the chatbot that we get our main practitioner
Speaker 5: being like, I just want my end users to ask
Speaker 5: a question of what was the AWS bill for the
Speaker 5: last month and how to that map to what we
Speaker 5: had put in our budget. And so we are actively
Speaker 5: kind of developing based on a lot of.
Speaker 4: The libraries that are out there and exists today.
Speaker 5: And so it is a mix of Okay, I need
Speaker 5: the infrastructure to be able to deploy, and what I
Speaker 5: do my training on is different than when I do
Speaker 5: my inference on. So there is like i'd say, because
Speaker 5: we have some history in building the tooling, we are
Speaker 5: a little bit more cost conscious than other organizations would be.
Speaker 5: But we use that to then say, Okay, if I'm
Speaker 5: an end user, what is the information I would need
Speaker 5: to be.
Speaker 1: Able to look at it sounds like it's not necessarily
Speaker 1: fin ops, which needs to happen now it's been mL
Speaker 1: OPS is the primary focus of cloudbal to be a
Speaker 1: much smarter version of the VPA and HPA. Or is
Speaker 1: that the set like you know, hyper focus in this
Speaker 1: area or is there like extended integrations.
Speaker 5: Yeah, so cloud Bowl there's a portfolio of products, and
Speaker 5: the Kuberneti's focus comes from the Stormforge acquisition. That's like
Speaker 5: kind of my background. That's probably why I talked about
Speaker 5: a little bit more, and that is Kubernetes right sizing,
Speaker 5: so the ability to actually look at your traffic patterns,
Speaker 5: use machine learning to come up with what your right
Speaker 5: configuration settings are. And we're currently building that into the
Speaker 5: cloud Bolt platform, which is a is a whole sleoth
Speaker 5: of things. And the cloud Bolt platform pulls in billing
Speaker 5: data from AWS, Azure GCP and it does all that
Speaker 5: with the focus back. So the Finnops Foundation came up
Speaker 5: with the focus back to basically be like, hey, everyone
Speaker 5: needs to talk the same language.
Speaker 4: Like it is.
Speaker 5: It's hard enough to give people the right insights. Now
Speaker 5: they have to know what do I what is storage
Speaker 5: and AWS compared to Azure and when you're talking to
Speaker 5: Finopps folks, not all of them know all the technical terms.
Speaker 5: So how can we abstract that we rebuilt our data
Speaker 5: platform to be focused native, so we can automatically pull
Speaker 5: that information and give you reporting and visibility across all
Speaker 5: your public clouds or anything private like VMware, Openstaff, new Tannics,
Speaker 5: all that with that one language, one abstraction layer, and
Speaker 5: then you can interact with chat thought that we're still
Speaker 5: working on. So I'd say it's in beta, it's not
Speaker 5: like publicly available yet, but we are using it internally.
Speaker 4: Is pretty cool, and interact with your data there.
Speaker 5: You can do your cost allocation to say okay, these
Speaker 5: different departments are often like, we have a lot of
Speaker 5: MSP customers and so it's like the cloud Ponzi scheme.
Speaker 5: Someone's buying cloud from someone then selling it to someone
Speaker 5: else and they're selling it and every layer you need
Speaker 5: to understand what are the but the discounts that get applied,
Speaker 5: what are the margins? I want to add to this,
Speaker 5: how do I make sure that third tier customer is
Speaker 5: only seeing what they should be paying for, not what
Speaker 5: I'm actually paying for? Because I could get a little hairy.
Speaker 3: For me ask a question like technically, how does this
Speaker 3: actually work? Like you said, we're looking at like traffic patterns,
Speaker 3: Like how is it actually intercepting and reading those traffic patterns.
Speaker 5: Yeah, there's an agent that sits in the cluster. It
Speaker 5: pulls the metrics from like se advisor, pull like cube
Speaker 5: state metric type metrics. It's a couple pods that sits
Speaker 5: in the clusters, not a damisut or anything, and then
Speaker 5: pulls that information out and then ports it into our
Speaker 5: our database. And that's one source. So we also have
Speaker 5: an agent that pulls metrics from like VMware for example,
Speaker 5: looks at that utilization data and then with the focus
Speaker 5: spec we can pull overlay information from like your billing reports,
Speaker 5: and then it put that on top of like all
Speaker 5: the utilization metrics.
Speaker 2: Okay.
Speaker 3: I always get nervous when we talk about like adding agents,
Speaker 3: so the agents are not sitting like within like a
Speaker 3: current deployment or something like that, because then it's like, Okay,
Speaker 3: now I have like more research in quests than I
Speaker 3: need to account for on top of to solve this problem.
Speaker 4: Yeah, yeah, totally.
Speaker 5: That's the people are very sensitive to agents. The word
Speaker 5: I mean soot not.
Speaker 3: Agent agents, but agents living inside your clusters.
Speaker 2: Yea.
Speaker 5: And what the storm Force software actually does is we
Speaker 5: use our own software to right size the agent so
Speaker 5: that we can make sure that the agent is just
Speaker 5: using the resources required for that cluster. And there are
Speaker 5: two lightweight pods at least in the kubernet space. When
Speaker 5: people have agent based tools, a lot of times of
Speaker 5: damon sets and so they take up a lot of space.
Speaker 4: On every node.
Speaker 5: And we've taken a deliberate approach to have a very
Speaker 5: lightweight agent that is just two pods that script and
Speaker 5: metrics and send them up.
Speaker 1: Is it a security concern or like a capacity utilization
Speaker 1: issue of our resources in the cluster.
Speaker 2: I've actually seen it as a capacity issue.
Speaker 3: So we've I've seen people like deploy agents for various
Speaker 3: things and suddenly of capacity issues with because they're you know,
Speaker 3: everybody's fighting for the same resources. But if these are
Speaker 3: like you're saying, sexic deployments, I feel like that's less
Speaker 3: of a less of an issue.
Speaker 1: At least you have some like previous trauma there.
Speaker 3: Trauma it seems like, well, is.
Speaker 1: There something that we can shame specifically?
Speaker 3: And I actually probatuct.
Speaker 1: Well, let's just start like this is there is there
Speaker 1: a frequent concern with pulling an open source you know,
Speaker 1: definitions into your into your cluster without really evaluating how
Speaker 1: much how resources they're going to consume compared to something
Speaker 1: that building yourself, and is that the source of most
Speaker 1: of the problems or you know, because these things aren't
Speaker 1: necessarily built for like I can I always complain about
Speaker 1: the security of open source not libraries, but full services
Speaker 1: that you deploy within your cloud environment or a cluster,
Speaker 1: because they're not designed for scale and reliability. They're designed
Speaker 1: to usually funnel you into their paid product or licensing,
Speaker 1: so you know, I it's like always a concern from
Speaker 1: that standpoint. So I can imagine an equivalent issue where
Speaker 1: they're not really designed to have low impact from a
Speaker 1: cost standpoint either.
Speaker 3: Yeah, I mean, I'll time in I guess a little bit,
Speaker 3: but I'm also curious that you have in the thoughts.
Speaker 3: But at least just what I've seen is is more
Speaker 3: of it's less of a security issue but more of
Speaker 3: a resource issue.
Speaker 5: I mean on our end, because so a previous version
Speaker 5: of our same product did actually use a lot of
Speaker 5: resources inside the cluster and customers would be like I'm
Speaker 5: using your tool to reduce my waist, but I also
Speaker 5: am wasting resources to use your tool. And so when
Speaker 5: we kind of went through a re architecture, that was
Speaker 5: a very important piece for us also because I mean
Speaker 5: it speaks to our credibility. We're supposed to be reducing resources,
Speaker 5: then our own bits should also be using limited resources.
Speaker 5: So that is why we take a very lightweight agent approach.
Speaker 5: And it's actually not an open source agent. We've gone
Speaker 5: back and forth of like should we release the code.
Speaker 5: It's not like it's anything secret, it's just it's always
Speaker 5: been a private agent. But you know, people can see
Speaker 5: the helm chart, they can see what resources will deploy
Speaker 5: with like default requests, so that obviously everything's capped. But
Speaker 5: then people will use the software to then write size
Speaker 5: it to make sure it is literally just using the
Speaker 5: minimal required resources and its size to the cluster too,
Speaker 5: so you use a small cluster, you're not spending the
Speaker 5: same amount of resources then if you had really like
Speaker 5: a large cluster smart I have sometsd from agents because
Speaker 5: I a lot of time at Puppet and there's a
Speaker 5: lot of agent wars at the time, to the point
Speaker 5: where I like hated the word, but.
Speaker 1: You know, of well, you know, I guess I could
Speaker 1: say maybe I'm glad that you have some you know,
Speaker 1: PTSD from working at Puppet, because I definitely have some
Speaker 1: from using Puppet. But that was a long time ago,
Speaker 1: and it's been quite some time since I've had virtual
Speaker 1: machines even to run stuff on. Now I feel like most,
Speaker 1: well I say it most. I've read some statistic like
Speaker 1: still fifty percent of the world uses some PHP backed
Speaker 1: builders for websites, So I mean, there's still a huge
Speaker 1: amount of you know, people that aren't even using any
Speaker 1: sort of infrastructure as code solutions. And I'm over here saying,
Speaker 1: you know, isn't everyone pretty much using some icy and
Speaker 1: deploying their their stuff directly and spinning up containers. Amy's
Speaker 1: already shaking her head. Either mean she has a really
Speaker 1: great story to share or a really great story she
Speaker 1: doesn't want to share.
Speaker 2: I just I.
Speaker 3: Don't know. It's a lot of people will try to
Speaker 3: do the right things, and then other people are just
Speaker 3: gonna try to do the quickest, what seemingly work solution,
Speaker 3: which is just click it and go. But I see it.
Speaker 2: I see it so often itrks me I mean.
Speaker 1: There is a huge aspect here where it's about aligning incentives.
Speaker 1: So I think most people, uh most people in general
Speaker 1: have a logical desire to do something. They see some
Speaker 1: initial conditions, and then they follow that their path as
Speaker 1: best as they can to go up with the action
Speaker 1: they want to take. And part of that input is
Speaker 1: whatever incentives they have to go forward. And sometimes those incentives,
Speaker 1: you know, aren't aligned from one team member to another one,
Speaker 1: or from one team to another one, Like you give
Speaker 1: certain incentives to your develop your engineers, and then you
Speaker 1: give different incentives to not engineers or release folks or
Speaker 1: engineering folks, product management or uh we talked about having
Speaker 1: financial operations folks you know, have responsibility of actually reducing
Speaker 1: the costs, but separate from not actually making sure that
Speaker 1: the products that you're building are a have low footprint, right,
Speaker 1: and so you immediately have a different of incentive there. So,
Speaker 1: you know, I think there is a lot that goes
Speaker 1: into into that sort of poor decision making process where
Speaker 1: you get some optimal outcomes when your organization as a
Speaker 1: whole doesn't have this included. And I don't remember the
Speaker 1: last organization I was in or the last company I
Speaker 1: was talking too. I was like, yeah, you know, and
Speaker 1: our objectives and key results are okay, ours, we actually
Speaker 1: have like reduced the total cost associated with our infrastructure.
Speaker 1: Although I feel like everyone's gonna be like, yeah, of course,
Speaker 1: of course you need that. Of course you you have
Speaker 1: to counter your new development. But I don't think a
Speaker 1: lot of companies are doing that, are they.
Speaker 3: No.
Speaker 5: It's funny when I ask people, I'm like, oh, like,
Speaker 5: how important is like reducing costs, Like it's a top priority.
Speaker 5: I'm like, oh, okay, can you rank like your top
Speaker 5: priorities And it doesn't even make the top five and
Speaker 5: you just said reducing costs is important, Like, well, yes,
Speaker 5: we have the security thing we need to do with
Speaker 5: this new application we're trying to get out, and it's
Speaker 5: always compete. That's why I think like any solution is
Speaker 5: going to need to take minimal time and just work
Speaker 5: inside a system so you can be like, Okay, this
Speaker 5: is low hanging fruit. I can go tackle it. Even
Speaker 5: if it's six on my list, it's still in the
Speaker 5: top ten. But it's like, it is really interesting every
Speaker 5: time I talk to an IT leader and they're like, yeah,
Speaker 5: this is so important for us to costs and then
Speaker 5: I'm like, okay, rank your priorities.
Speaker 4: Where does it actually sit.
Speaker 3: You are mature enough engineer, you kind of treat this
Speaker 3: as like just your bread and butter is like part
Speaker 3: of your job, so like, yes, it helps to have
Speaker 3: the incentive there. In my opinion, I feel like if
Speaker 3: you're going into that role, like this is not something
Speaker 3: like management should tell you like, oh, we need to
Speaker 3: reduce costs, Like this is just like a given of
Speaker 3: like as part of the reliability metric that people measure,
Speaker 3: Like the cost metric could be something that you just
Speaker 3: go to your management team on a quarterly basis and
Speaker 3: be like, here's the cost.
Speaker 2: We reduced or if here's what we did. I like
Speaker 2: that you shouldn't have to be told to do it.
Speaker 1: If you're an experienced engineer or even a senior engineer,
Speaker 1: you may have your own internal metrics for say when
Speaker 1: to pull out a public package or you know open
Speaker 1: source solution, like oh yeah, I go to GitHub and
Speaker 1: I look and I see there's fifteen thousand stars. That
Speaker 1: means yeah, sure, I'll use it, no problem. You know
Speaker 1: there are ratings there that that are and I feel like,
Speaker 1: you know, as you ascend the sort of competency ladder,
Speaker 1: and you get to a staff staff plus principal engineer,
Speaker 1: you're not like you need your own sort of internal metrics.
Speaker 1: And I feel like you're increasing the scope of your
Speaker 1: delivery at those higher levels and there has to be
Speaker 1: a trade off, like how do you make make sure
Speaker 1: you're doing the right thing? And I totally agree with
Speaker 1: you Amy that part of it is like, well, how
Speaker 1: much am I spending to make sure that this technology
Speaker 1: that we released to have a business impact is costing
Speaker 1: us the right amount to actually have a positive ROI
Speaker 1: for us and not and not be negative.
Speaker 3: I would say like, just like if you enjoy your
Speaker 3: job and you're looking at your runway, like this should
Speaker 3: be something that is consistently top of mind. And I
Speaker 3: mean through no fault of like more you know, newer engineers,
Speaker 3: this is just not something they think about. It's not
Speaker 3: something I definitely didn't think about my first two years.
Speaker 3: I was just it's like, oh my gosh, I need
Speaker 3: to get this speech. You're out.
Speaker 1: I think there is definitely a cliff function here, and
Speaker 1: maybe either of you can correct me, but I feel
Speaker 1: like there's like there's a switch where early on in
Speaker 1: people's careers. They assume that the money that a company
Speaker 1: is spending is their own money, like, oh, this price
Speaker 1: tag is like you know, we get customers all the
Speaker 1: time like, oh, this is like going to be ten
Speaker 1: thousand a month. That's too expensive. I'm like, how, like,
Speaker 1: how much are you paying your engineering team to maintain
Speaker 1: that solution that's not even working? And how much are
Speaker 1: you paying your cloud provider to run that solution for you?
Speaker 1: I assure you it's like at least ten times how
Speaker 1: much you'll be paying us. And then there's a switch later.
Speaker 1: I feel like where they you know, they're on one
Speaker 1: of these two sides of extremes, right, like the money
Speaker 1: doesn't matter or the money matters too much, And I think,
Speaker 1: just like everything, I think the number one answer staff
Speaker 1: plus engineers love to give is you know it depends, right,
Speaker 1: you know, how much would we spend on our stack? Well?
Speaker 1: You know, well what are we running? You know, how
Speaker 1: important is it? What's the reliability necessary? I do want
Speaker 1: to ask you a question, yesman, though, is that Let's
Speaker 1: say I am in one of the roles, and while
Speaker 1: we all collectively agree that you should have the responsibility
Speaker 1: and accountability for either running some sort of solution that
Speaker 1: handles auto scaling, pulling in the appropriate sasas, or opens
Speaker 1: our solutions to help you. What if I'm not giving
Speaker 1: given that flexibility, Like what can you know on the
Speaker 1: ground engineers do to help make a case to say
Speaker 1: their leadership or their teams to have a more mature
Speaker 1: perspective on how to approach the problem.
Speaker 5: The first question I usually ask people is like, what
Speaker 5: what do you think your cluster utilization is? And they'll guess,
Speaker 5: and usually they always guess higher than it actually is.
Speaker 5: Then they'll go look into whatever monitoring system they have
Speaker 5: and they're like, oh wow, it's like twenty percent. And
Speaker 5: I'm like, okay, what if you just doubled that? Right? Like,
Speaker 5: I'm not saying go to eighty percent, I'm just going
Speaker 5: to go from twenty to forty percent and setting that
Speaker 5: goal to be something that's achievable. Typically, I found that
Speaker 5: is something people can do on their own, without tools
Speaker 5: and just get an understanding of how bad of a
Speaker 5: problem do I have and how much waste is there?
Speaker 5: Because when when you go to people and you're like, hey,
Speaker 5: we could remove waste, then it's like the language you're
Speaker 5: removing something there's there's a risk in that. But when
Speaker 5: you say, what if you could improve your cluster utilization
Speaker 5: from just double it from twenty to forty percent, you
Speaker 5: still sixty percent headroom, Like there's still a lot of
Speaker 5: waste there, but you've doubled your impact. It's a pretty
Speaker 5: good metric that you can feel good about.
Speaker 1: I do see this fhere of like I don't want
Speaker 1: to touch something I don't understand because I don't want
Speaker 1: to break it, Like maybe it's twenty percent right now.
Speaker 1: But I think the other thing that we hear frequently
Speaker 1: on the show is like, how as an experienced engineer
Speaker 1: do you get to some sort of depth? I get
Speaker 1: asked a lot when I'm giving toxic conferences like how
Speaker 1: did I become a security expert? I'm like, I have
Speaker 1: no idea. I guess I just spend a lot of
Speaker 1: time doing things. And I think this is one of
Speaker 1: those areas like if you see the capacity is really
Speaker 1: like utilization is really low, Like maybe ask the question like,
Speaker 1: well why is it so low? Like what should it be?
Speaker 1: Like you don't have to change anything, but maybe you
Speaker 1: should be able to answer, like, well, if I wanted
Speaker 1: to change it, what steps should I take how should
Speaker 1: I know what it should be?
Speaker 2: Like?
Speaker 1: What information should I pull out? I think it's really good.
Speaker 5: I went back when I was an engineer. I learned
Speaker 5: everything by breaking things. Made sure I broke them in dev.
Speaker 5: But like, you don't learn until you try. And so
Speaker 5: as long as you have a safe space to try,
Speaker 5: and you have a cluster in dev that doesn't matter
Speaker 5: and it has really well utilization, you can practice different
Speaker 5: ways to improve that on a cluster that doesn't matter
Speaker 5: until you kind of hone your skills and go deploy
Speaker 5: it to other environments.
Speaker 1: You can definitely say no comment here if you're not
Speaker 1: comfortable answering this question. But like, surely people were concerned
Speaker 1: about the cost of the cloud before moving there, or
Speaker 1: their cost of their you know, container management solution or
Speaker 1: you know lead of virtual machines that they were using
Speaker 1: vSphere on VMware, Like weren't there like solutions, Like weren't
Speaker 1: people already talking about and doing this before?
Speaker 5: Yes, but there was an expectation that moving to cloud
Speaker 5: would be cheaper, and then there's a reality that like
Speaker 5: you have to put in work to actually make sure
Speaker 5: the cloud is cheaper, and most people didn't put in
Speaker 5: that work, so now they're paying for it.
Speaker 1: Do you think that's like a huge driver for the
Speaker 1: let's move off of cloud because it's too expensive And
Speaker 1: you're like the voice of reason here, It's like, well,
Speaker 1: it doesn't have to be too expensive. It can be
Speaker 1: the right amount of expensive or even cheaper if you
Speaker 1: just think about what workloads you're running and what they
Speaker 1: are capacitily utilization, and how to scale effectively.
Speaker 5: Yeah. I think like everyone wants this ideal world where
Speaker 5: they don't think about day two problems, but doesn't matter
Speaker 5: where you're deployed, you have to think about day two problems.
Speaker 5: And the reality is for some people, probably moving back
Speaker 5: to the data center is the right thing, and that's okay,
Speaker 5: Like there is this oh, you have to be on
Speaker 5: the cloud to be cool, which like I think people
Speaker 5: are starting to get that you should be on the
Speaker 5: cloud for the right reasons and if you are there,
Speaker 5: you need to be thinking about your day two challenges,
Speaker 5: like making sure your configurations are set correctly, like you're
Speaker 5: reducing wasts, you're doing the right sizing, you have policies
Speaker 5: to go clean up unused backups, just you know, basic things.
Speaker 1: I just want to totally disagree with you. I mean,
Speaker 1: I think it's a very mature approach to say, you know,
Speaker 1: there are reasons that you may want to go to
Speaker 1: an on prem data center, but like my, my justifications
Speaker 1: for doing that are just drying up one after another one.
Speaker 1: Like I think about just the the depreciating assets for
Speaker 1: on prem, Like it used to be the case for
Speaker 1: sure that you could buy some you could buy a
Speaker 1: data center or even run a data center and throw
Speaker 1: your own server blades in and those would be usable
Speaker 1: for a decade or even twenty years. And now I
Speaker 1: feel like, especially if you're doing something cutting edge technology
Speaker 1: and your run and you want to do some data
Speaker 1: train you know, evaluation, or build some models or run
Speaker 1: the models and do inference, the technology to maintain a
Speaker 1: competitive advantage actually requires frequent hardware upgrades. I don't know.
Speaker 1: I actually would love to sell someone to tell me, like, no,
Speaker 1: actually here is this area where it makes sense to
Speaker 1: actually be on prem. But the more simple, the more straightforward,
Speaker 1: the more common what you're doing is like everyone else,
Speaker 1: the less of a reason there is, like it should
Speaker 1: it really should be cheaper being in the cloud for me, I.
Speaker 5: Think for most startups that are also like deploying to
Speaker 5: cloud native applications makes sense, you still got to think
Speaker 5: about day two otherwise you're just going to spend like crazy.
Speaker 5: But like you still have retail companies that have been
Speaker 5: around for a while and have like oms systems that
Speaker 5: are not built to run on a public cloud. So
Speaker 5: it's you're forcing your shoehorning something that's just going to
Speaker 5: cost you in the long run. And because I mentioned
Speaker 5: our history of like being an infrastructure automation platform for
Speaker 5: the last fifteen years, we see a flavor of every
Speaker 5: type of infrastructure you can imagine, like things that are
Speaker 5: still running in a data center somewhere on tapes, like
Speaker 5: it still exists, and not everything makes sense to move
Speaker 5: because if you're just going to lift and shift it,
Speaker 5: then you will be paying more.
Speaker 1: It's interesting because it was never for me that moving
Speaker 1: to the cloud was actually a cost reduction. I mean,
Speaker 1: I believe it, it's a reduced cost you build the
Speaker 1: right thing directly, but like my personal motivations were always
Speaker 1: like high reliability, getting cutting edge features and support the
Speaker 1: infrastructure platform whatever you're utilizing, and being able to you know,
Speaker 1: do quick have quick development cycles, you know, be more
Speaker 1: agile and I feel like in the agile world with
Speaker 1: just in time technology, there's a higher there is a
Speaker 1: cost associated with that, so like I do see it
Speaker 1: as paying for the advantage. I also like the mentality
Speaker 1: of like, well, it actually doesn't have to be more
Speaker 1: expensive if you, you know, utilize what the cloud provider
Speaker 1: is offering. It's interesting to see that, you know, you're
Speaker 1: still engage with companies that are are running off prem
Speaker 1: and it just will, like forever, always be cheaper.
Speaker 5: That fifty percent metric honestly doesn't surprise me. It probably
Speaker 5: did before we got acquired because I've been living in
Speaker 5: the Kubernetes world for so long. But the more and
Speaker 5: more I talk to different customers and prospects that there's
Speaker 5: still a mix of there's a healthy mix of on
Speaker 5: prem and even honestly within Kubernetes, people running Kubernetes on prem, I.
Speaker 1: Used to trust them. I mean, so I'll say where
Speaker 1: this is from. Like I think it was like every
Speaker 1: year WordPress would come out with a number of like
Speaker 1: the percent of the Internet that was running on WordPress,
Speaker 1: and like, I never liked PHP, so I wasn't particularly
Speaker 1: inclined to trust the numbers coming out. But what after
Speaker 1: every you know, local recent scandal. I even trust those
Speaker 1: numbers less. But whatever that number is, it's still higher
Speaker 1: than I would be comfortable with, you know, as a
Speaker 1: human society, that we're you know, overutilizing this. So there
Speaker 1: are tons of uh, you know, non tech zero one
Speaker 1: or two person companies that before the web flows and
Speaker 1: levables of the world bubble eyos, there there was like
Speaker 1: you needed to get your website up and running and
Speaker 1: GeoCities and whatever Yahoo had, you know, just Google sites
Speaker 1: you know, was not cutting it. So you did do
Speaker 1: this and with a number of themes and plugins like
Speaker 1: I do get it. But that means like they can't
Speaker 1: utilize any of this, and the costs for those is
Speaker 1: of course really high, but they have no value in
Speaker 1: transitioning to you know whatever. Think of like your local hairdresser,
Speaker 1: like they're not going online. I mean at this point though,
Speaker 1: I'm sure they're using some SaaS product to manage all
Speaker 1: their scheduling and LMS to actually do the hair dressing itself.
Speaker 4: That'd be great.
Speaker 5: Ask a chatbot like how should I cut my hair?
Speaker 5: I guess you could never thought of it.
Speaker 1: Well, I just assume you're not too far out from
Speaker 1: asking the chatbot, like handing it to the scissors and
Speaker 1: telling it to do what it thinks is best.
Speaker 3: I actually saw a video on the other day of
Speaker 3: one of Musk's robots doing haircuts.
Speaker 1: Yeah, we're not going to go into that topic otherwise
Speaker 1: I think I'm gonna actually have to cut it out
Speaker 1: of the podcast episode.
Speaker 3: So I'm just stating the facts. YEA.
Speaker 1: Well, you know, I think for some people the facts
Speaker 1: are even up for debate.
Speaker 2: So, yes, it was a video.
Speaker 3: I'm like, did an Ai just make this video or it
Speaker 3: is just like a realistic video.
Speaker 5: I just I don't even can never tell honestly, every
Speaker 5: video I watch, I'm like, is it real?
Speaker 4: I don't know.
Speaker 1: There was there was something I'll say that was really interesting,
Speaker 1: totally unrelated. Uh, there's a British comedy show called QI
Speaker 1: Quite interesting and every season they would say facts and
Speaker 1: they by like the thirteenth season, what they had done
Speaker 1: is they actually calculated the lifetime of the facts, so
Speaker 1: what they said earlier on became less correct the later
Speaker 1: that season would go so like it was like the
Speaker 1: previous season, like n minus one eighty six percent of
Speaker 1: the facts were still correct, but like the first season,
Speaker 1: by the thirteenth one, there was like something like sixty
Speaker 1: percent of the facts were like just no longer accurate,
Speaker 1: like science had like whatever we had decided different now,
Speaker 1: And I think the canonical one is like what the
Speaker 1: food pyramid? Like, what is the food pyramid supposed to be?
Speaker 1: Every year there's a different change to it, and it's
Speaker 1: it's never never accurate. I think I think before we
Speaker 1: go too much down another tangent, maybe maybe it's a
Speaker 1: good time to switch over to picks. Maybe mostly nods here. So, Amy,
Speaker 1: what did you bring for us today?
Speaker 3: I debated on this because I don't want a lot
Speaker 3: of people to go out and buy it and then
Speaker 3: they have like a they're no longer able to fulfill
Speaker 3: the oilers. But it's really good, so I feel like
Speaker 3: I should share. I'm a big health person, hence why
Speaker 3: I'm walking on the treadmill, and my pick is going
Speaker 3: to be like a specific protein powder. So when I
Speaker 3: was like new on my protein powder journey, I didn't
Speaker 3: quite care. I was like, what does it taste like?
Speaker 2: And what's the price?
Speaker 3: So this stuff is a little of pricey, but it's
Speaker 3: like super clean ingredients and it's the brand's called the clip.
Speaker 2: The other good thing.
Speaker 3: They have a protein I feel really embarrassed, like maybe
Speaker 3: other people are not into this, but in case someone is,
Speaker 3: they also have a protein bar now, which like, if
Speaker 3: you travel for work frequently, I'm always like, if I
Speaker 3: have to go to conference or something like, I need
Speaker 3: something that is going to not like make me feel
Speaker 3: like trash when I'm there, because sometimes the food doesn't
Speaker 3: always I don't know, it's not always the best. So anyways,
Speaker 3: they have a protein bar that just came out. I
Speaker 3: had to order like a month in advance. That's how
Speaker 3: special this prostin bar is to me.
Speaker 2: And I just got it. And it's still like this and.
Speaker 1: You know you did you did Peloton last last episode,
Speaker 1: so you know, we clearly on a healthcake here.
Speaker 2: I guess I just want to think of something else
Speaker 2: next week.
Speaker 1: You don't have to. You can definitely bring high. I
Speaker 1: don't live a healthy lifestyle picks every single week. I think,
Speaker 1: you know, especially for those of us who sit at
Speaker 1: our computer, like literally sit at our computers all day long,
Speaker 1: I think there's a something good to be said for you.
Speaker 3: Actually, here's something practical about it. It's not just healthy,
Speaker 3: but a lot of people who are like trying to transition.
Speaker 3: I hear like a lot of my coworkers are like,
Speaker 3: I want to get healthier, and they switch to a
Speaker 3: protein powder. A lot of protein powder is way based
Speaker 3: and that can upset people's stomach. So this one is
Speaker 3: actually made from beef protein, which you might think, Like, initially,
Speaker 3: I was like, this sounds disgusting, but it actually tastes
Speaker 3: really good.
Speaker 1: That's a that's a pretty good cell there. Okay, Osmin,
Speaker 1: what'd you bring for us?
Speaker 5: So initially I think I had more of a fun
Speaker 5: fact that I learned this week than a pick. But
Speaker 5: since we're on a health pick topic, I've been having
Speaker 5: bone broth recently. A friend got me into it, and
Speaker 5: I've been drinking like beef bone broth, chicken bone broth.
Speaker 5: There's one on Amazon that I buy by like nineteen
Speaker 5: ninety snacks. It's like a orange container. I'm sure they're
Speaker 5: all good, but uh, it's like actually really good instead
Speaker 5: of a tea or just lunch snack. If I'm like, oh,
Speaker 5: I don't have time, I'll just heat it up. And
Speaker 5: I think maybe it was a thing that everybody else
Speaker 5: knew about that I didn't know about. But yeah, I've
Speaker 5: been really getting into that lately.
Speaker 2: They're also good for travel.
Speaker 1: I've never heard of it, and I'm I don't think
Speaker 1: I'll ever have it, so.
Speaker 3: You should try it.
Speaker 4: You should just try it.
Speaker 3: Yeah, really, Okay, here's an idea to use it too,
Speaker 3: Like if you don't want to try to drink it,
Speaker 3: Like if you ever have some sort of like pasta,
Speaker 3: you can cook your pa in it.
Speaker 1: So I absolutely go to the store or actually a
Speaker 1: lot of butchers will throw away bones. You can go
Speaker 1: in just ask them for the bones. And you don't
Speaker 1: need to buy beef bullion or chicken bullion at the
Speaker 1: store or vegetable stock. Honestly, just boil the bones and
Speaker 1: some vegetables. I usually use caracter celery in a bay
Speaker 1: leaf in a pot for like an hour or two,
Speaker 1: and then use the liquid for whatever else you're making.
Speaker 1: And it's just so much better.
Speaker 2: I like your idea better.
Speaker 5: Just try drinking it. Just try it, even if it's.
Speaker 1: I don't know, Like I find for me that the
Speaker 1: best broths are like usually when the meat is a
Speaker 1: little bit fatty. So if you use like chicken thighs
Speaker 1: on set or wings like instead of breasts and uh,
Speaker 1: pork or beef bones, Like I don't have that much meat,
Speaker 1: but like there's a lot of flavor here. I still
Speaker 1: think it's like too fatty for me, Like I don't
Speaker 1: like the sensation of having like a fatty drink like
Speaker 1: that that's the idea is warming up to me. But
Speaker 1: it's it's a lot, it's a lot of extra effort
Speaker 1: to go out and do this. But I suppose if
Speaker 1: you know you, if you live close or you you
Speaker 1: buy whole animals to to cut up and consume, then
Speaker 1: you have probably a lot of extra bones. And if
Speaker 1: you're not utilizing them to also make drinks, I guess
Speaker 1: here's your here's your opportunity. Okay, So I actually wasn't
Speaker 1: sure what I wanted my pick today to be, so
Speaker 1: I went and I grabbed this article called lions and
Speaker 1: dolphins cannot make babies, And it sounds pretty obvious on
Speaker 1: the surface. I mean, obviously they're in different families in
Speaker 1: the taxonomy for animals. But the article is about basically,
Speaker 1: how we got to this point in technology required a
Speaker 1: synonymous relationship, symbiotic between whatever the hardware we're utilizing and
Speaker 1: the software that we're building. And I think this is
Speaker 1: true in a lot of things. It's known as an evolution,
Speaker 1: as red Queen's race. So you think of like bacteria
Speaker 1: and viruses competing for resources, or even any sort of
Speaker 1: parasitic organism and the host evolving. And it goes into
Speaker 1: a little bit about how we probably will not have
Speaker 1: any more innovation in AI unless we really take a
Speaker 1: huge giant leap in some of the hardware technology available
Speaker 1: to us. We can't just build better software and fix
Speaker 1: some of the problems. I mean right now, I think
Speaker 1: one of the limitations we have is everything that's built
Speaker 1: in the last six years was based off of the
Speaker 1: paper written by Google on transformer texture, and like, sure
Speaker 1: there are some shortcomings, like it can't figure out patterns
Speaker 1: or do math, but we'll extract some characters from you
Speaker 1: can ask it like where is the math, and then
Speaker 1: like rejects that and then pass it to any sort
Speaker 1: of solver engine and get that. So you know, you
Speaker 1: start to solve some of those problems. But fundamentally, to
Speaker 1: get to the next level, we'll need something different, and
Speaker 1: it's not just writing new software. I think the idea
Speaker 1: here is fundamentally that we're stuck until we actually understand
Speaker 1: that we may need a different kind of hardware to
Speaker 1: power this. I don't know if quantum computing is sufficient
Speaker 1: to running this, but really just something that's slightly different
Speaker 1: from our current computational engines.
Speaker 4: I'd love to read that.
Speaker 1: It's not very long, so I think it maybe overseelling this,
Speaker 1: but it'll be in the link to the in the
Speaker 1: podcast episode. In the show notes, thank you so much
Speaker 1: for our kind guests to show up today and tell
Speaker 1: us all about it off and off, So thank you
Speaker 1: for coming. It's been fantastic, And thank you to all
Speaker 1: of our listeners for another great episode of adventures and
Speaker 1: DevOps and I hope you're all back for the next one.
Speaker 3: The st