WEBVTT 1 00:00:08.080 --> 00:00:11.080 Hello everyone, and welcome back to another episode of Adventures 2 00:00:11.080 --> 00:00:14.320 in DevOps. And I'm joined today again by my co 3 00:00:14.400 --> 00:00:16.760 host Amy Knight. How you doing, Amy? 4 00:00:16.879 --> 00:00:18.039 Pretty good? Walking in? 5 00:00:19.839 --> 00:00:22.239 She likes walking, that's uh. 6 00:00:22.440 --> 00:00:23.559 I gotta get my steps in. 7 00:00:25.879 --> 00:00:28.280 I don't have a good strategy for that yet. Yeah, 8 00:00:28.039 --> 00:00:31.839 so today I have to be completely honest. The topic 9 00:00:32.000 --> 00:00:34.759 is in and around finnops. But I don't really know 10 00:00:34.799 --> 00:00:37.679 what finnopps is. But if it's anything like DevOps, I'm 11 00:00:37.679 --> 00:00:40.280 guessing it means firing all your business analysts and hiring 12 00:00:40.320 --> 00:00:43.200 some additional software engineers to do a job they don't understand, 13 00:00:43.520 --> 00:00:46.200 and then, after your years of failing, reappropriate the name 14 00:00:46.240 --> 00:00:48.880 to mean something else and claim success. Well, Amy, you're 15 00:00:49.039 --> 00:00:51.119 already smiling, so maybe you know something I don't know. 16 00:00:52.039 --> 00:00:54.640 I don't actually laughing because it's very entertaining. 17 00:00:54.759 --> 00:00:55.679 And I also. 18 00:00:57.039 --> 00:01:00.119 I think, like last week, I realized your title on 19 00:01:00.159 --> 00:01:02.399 LinkedIn a tech entertainer. 20 00:01:03.039 --> 00:01:04.680 So that was a very entertaining intro. 21 00:01:05.280 --> 00:01:08.599 You did a great job, well, thank you. So we 22 00:01:08.680 --> 00:01:11.680 pulled an expert from the field, chief strategy officer at 23 00:01:12.040 --> 00:01:18.159 cloud Bolt, Yasmin Rajabi, and I've noticed here so you 24 00:01:18.480 --> 00:01:20.400 have a history of product management, and so I'm really 25 00:01:20.400 --> 00:01:21.519 excited to have you on the show. 26 00:01:21.760 --> 00:01:23.959 Welcome, thanks for having me excited to be here. 27 00:01:24.159 --> 00:01:28.480 How completely off base was my thoughts about what finof says. 28 00:01:28.719 --> 00:01:30.359 I mean, not so far off base. 29 00:01:31.319 --> 00:01:35.079 Any time you put like an ops inside in one 30 00:01:35.079 --> 00:01:38.239 of the words, you're obviously trying to force some things together. 31 00:01:38.439 --> 00:01:39.159 And it's funny. 32 00:01:39.159 --> 00:01:41.959 A question I get asked often is like, okay, so 33 00:01:42.040 --> 00:01:45.640 how are finnops teams and DevOps teams coming together? And 34 00:01:45.799 --> 00:01:48.519 the OPS in pinnops is supposed to be for DevOps, 35 00:01:48.599 --> 00:01:51.359 so technically like they're supposed to already be together. They're 36 00:01:51.400 --> 00:01:54.120 not supposed to be separate teams. But there's always the 37 00:01:54.560 --> 00:01:57.159 ideal and then the reality of if you take financial 38 00:01:57.239 --> 00:02:00.840 analysts and you take engineers and you want to bring 39 00:02:00.840 --> 00:02:05.079 them together, they speak different languages, they come from different worlds, 40 00:02:05.079 --> 00:02:08.719 So there are folks that are trying to use software 41 00:02:08.759 --> 00:02:10.840 to bring that together. I'd put us in the mix 42 00:02:10.879 --> 00:02:15.199 of that, but it's still like people from completely different 43 00:02:15.199 --> 00:02:17.240 worlds and trying to get them to think and care 44 00:02:17.319 --> 00:02:18.319 about the same topic. 45 00:02:18.680 --> 00:02:21.199 Okay, I was going to add too, we also need 46 00:02:21.240 --> 00:02:25.759 to add in is it greenops. So it's like getting 47 00:02:25.840 --> 00:02:29.080 environmentally savvy as well within the realm of finnops. 48 00:02:29.080 --> 00:02:31.800 Okay, well, I realized that I need to reduce the waste, 49 00:02:31.840 --> 00:02:33.319 but I want to be able to measure that from 50 00:02:33.319 --> 00:02:36.639 a like my carbon footprint standpoint. And it was interesting 51 00:02:36.680 --> 00:02:38.680 I was talking to a Phinnops person that is using 52 00:02:38.759 --> 00:02:42.599 those metrics to drive people to care inside their organization, 53 00:02:42.680 --> 00:02:45.000 because the people in the organization is like, Okay, well, 54 00:02:45.039 --> 00:02:46.800 it's not my money, I'm not spending it. But when 55 00:02:46.879 --> 00:02:50.199 you actually tie it to the environmental impact, the kind 56 00:02:50.199 --> 00:02:54.479 of human nature part of people helps kind of get 57 00:02:54.520 --> 00:02:56.960 people a little bit more passionate about reducing the waste 58 00:02:57.039 --> 00:02:59.240 inside their inside their organization. 59 00:02:59.360 --> 00:03:01.479 As far as the step, I did kind of lap 60 00:03:01.520 --> 00:03:02.240 it off before. 61 00:03:02.319 --> 00:03:03.680 But I saw. 62 00:03:03.639 --> 00:03:05.960 Something recently about like as I're putting more and more 63 00:03:06.039 --> 00:03:08.960 data centers up, like the water shortage that's happening to 64 00:03:09.000 --> 00:03:10.360 the communities around them. 65 00:03:10.319 --> 00:03:13.479 Some cities are literally like their grid cannot handle any 66 00:03:13.520 --> 00:03:16.520 more data centers, data centers coming online, and like they're 67 00:03:16.639 --> 00:03:19.680 starting to buy in different area, Like it'll be interesting 68 00:03:19.680 --> 00:03:21.319 what things look like in ten years. 69 00:03:21.360 --> 00:03:23.560 Like I saw like these poor families like literally like 70 00:03:23.560 --> 00:03:26.000 they can't run their appliances as they could like a 71 00:03:26.000 --> 00:03:26.680 couple of years ago. 72 00:03:26.840 --> 00:03:28.039 Yeah, anyway, Okay, So I. 73 00:03:28.000 --> 00:03:30.840 Think that's a really interesting topic because we've had a 74 00:03:30.879 --> 00:03:33.960 lot of our previous guests on the show, either they've 75 00:03:33.960 --> 00:03:36.879 been AI focused in the last few months or they've 76 00:03:36.919 --> 00:03:39.919 had a unique insight into data center operations. One of 77 00:03:39.960 --> 00:03:42.000 the ideas that keeps coming up is that it's actually 78 00:03:42.000 --> 00:03:46.520 not a struggle to get energy reservations for the data centers. 79 00:03:46.879 --> 00:03:49.680 But it's interesting that you bring up the water impact 80 00:03:49.680 --> 00:03:51.719 because that's a new thing that I'm not familiar with. 81 00:03:52.280 --> 00:03:55.240 Where it kind of interplays with water is probably less 82 00:03:55.280 --> 00:03:58.280 of my expertise and background, just the fact that you know, 83 00:03:58.319 --> 00:04:00.159 you need the water to cool down the data center. 84 00:04:00.639 --> 00:04:02.879 The more the more power it's using, then it becomes 85 00:04:03.080 --> 00:04:04.199 it's just like a cycle. 86 00:04:04.719 --> 00:04:07.240 Though I would say their. 87 00:04:06.560 --> 00:04:10.120 Ability to use liquid cooling and the data centers reuses 88 00:04:10.159 --> 00:04:13.479 the water instead of an endless supply of water. The 89 00:04:13.719 --> 00:04:17.000 recycling the water that exists has at least helped make 90 00:04:17.040 --> 00:04:19.959 some of an impact. What's interesting is people are pulling 91 00:04:20.000 --> 00:04:23.839 these metrics out and trying to overlay them on their 92 00:04:23.959 --> 00:04:27.199 usage as well. So, Okay, I'm writing an application I 93 00:04:27.319 --> 00:04:30.079 deploy it. I never think about these things, but as 94 00:04:30.079 --> 00:04:32.240 I can start to pull some of those metrics out 95 00:04:32.319 --> 00:04:37.079 of either my cloud provider as they start to provide them. 96 00:04:37.199 --> 00:04:39.800 I think Microsoft started to pull in like carbon data 97 00:04:39.839 --> 00:04:41.920 and other metrics that you can pull up or if 98 00:04:41.920 --> 00:04:43.759 I'm running my own data center. Then being able to 99 00:04:43.759 --> 00:04:46.199 pull those metrics out then kind of overlays and says, hey, 100 00:04:46.199 --> 00:04:48.480 here's the impact of what you're doing, Like make sure 101 00:04:48.480 --> 00:04:52.040 you're setting these things correctly. Otherwise there's there's a lot 102 00:04:52.199 --> 00:04:53.680 bigger impact than just financial. 103 00:04:53.920 --> 00:04:56.759 What's a good example of needing to pull in accounting 104 00:04:56.879 --> 00:04:59.920 or financial operations into software. 105 00:05:00.879 --> 00:05:04.560 So I'll say for most of my background in this 106 00:05:04.680 --> 00:05:08.160 space is on the Kuberneti side, and when you're deploying 107 00:05:08.160 --> 00:05:11.360 a Kubernetes app, you need to set requests and limits, 108 00:05:11.399 --> 00:05:14.519 one from a reliability standpoint to make sure you actually 109 00:05:14.519 --> 00:05:17.800 have the resources, but two typically what people do is 110 00:05:17.839 --> 00:05:19.279 just like set them really high so they don't have 111 00:05:19.319 --> 00:05:21.879 to think about it, and then deploy the application. And 112 00:05:22.800 --> 00:05:25.439 it's really hard to understand the impact of that, especially 113 00:05:25.480 --> 00:05:28.079 because that's at the pod level and you pay for 114 00:05:28.160 --> 00:05:30.360 nodes and you're like, okay, well, I don't really know 115 00:05:30.399 --> 00:05:33.240 how my pods translate into my nodes. So I'm paying 116 00:05:33.319 --> 00:05:35.199 some amount of money, but how do I actually know 117 00:05:35.279 --> 00:05:38.199 the impact? And so being able to pull in both 118 00:05:38.240 --> 00:05:42.199 the billing data and then actually allocate it correctly, because 119 00:05:42.240 --> 00:05:44.600 one of the challenges in Kubernetes is, Okay, how are 120 00:05:44.600 --> 00:05:46.839 you going to allocate your costs? Do you just look 121 00:05:46.879 --> 00:05:48.959 across name spaces and say like, okay, I'm going to 122 00:05:49.399 --> 00:05:51.360 divide it by name spaces and then that's what you get. 123 00:05:52.319 --> 00:05:55.120 But what if you have a larger name space or 124 00:05:55.879 --> 00:05:59.199 you have how do you split out like cube system 125 00:05:59.319 --> 00:06:02.040 and all of anything on the control plane. That's some 126 00:06:02.079 --> 00:06:04.759 of the challenges that people have. And if you're what 127 00:06:04.800 --> 00:06:06.800 you need to be doing is pulling in that billing 128 00:06:06.879 --> 00:06:09.839 data to say here's how it's costing you, here's how 129 00:06:09.879 --> 00:06:11.199 we can split that out if we want to split 130 00:06:11.279 --> 00:06:13.079 up the pod level, and then you can be able 131 00:06:13.079 --> 00:06:16.079 to tie that back to the actual nodes that you 132 00:06:16.839 --> 00:06:17.879 pay your bill on. 133 00:06:18.160 --> 00:06:21.040 So would it be accurate to say that Historically maybe 134 00:06:21.319 --> 00:06:23.920 software engineering teams haven't been held accountable for how much 135 00:06:23.959 --> 00:06:24.560 they spend in. 136 00:06:24.519 --> 00:06:27.079 The cloud for sure, And often what happens is the 137 00:06:27.120 --> 00:06:31.480 developers are setting those requests, and then the platform engineering 138 00:06:31.519 --> 00:06:33.639 team are the ones that are like, Okay, well we're 139 00:06:33.680 --> 00:06:37.399 managing the cloud environment, we're managing those bills. They get 140 00:06:37.439 --> 00:06:39.560 asked to do something about it, and I don't know 141 00:06:39.600 --> 00:06:40.959 the app. What do you want me to do? If 142 00:06:41.000 --> 00:06:42.759 I touch something, I'm going to get a phone call 143 00:06:42.800 --> 00:06:45.600 from probably not a phone call, slack message from a 144 00:06:45.680 --> 00:06:48.600 developer of like, hey, what do you do to my system? 145 00:06:48.839 --> 00:06:50.920 It's down, it's having challenges. You are the last one 146 00:06:50.920 --> 00:06:51.360 to touch it. 147 00:06:51.720 --> 00:06:53.360 Amy having experienced there. 148 00:06:53.360 --> 00:06:57.279 Yeah, this is a very real problem. I would say, 149 00:06:57.360 --> 00:07:01.319 like the requests and the mins also, and I think 150 00:07:01.399 --> 00:07:03.959 like if a scale gets large enough, you also need 151 00:07:03.959 --> 00:07:09.199 to understand like traffic patterns and scaling appropriately for that, 152 00:07:10.439 --> 00:07:12.759 because you know, if you're an early stage company, you 153 00:07:12.800 --> 00:07:14.759 can just kind of scale at like a you know, 154 00:07:14.879 --> 00:07:17.720 relative level and stay there. If the app really grows, 155 00:07:17.720 --> 00:07:19.959 which hopefully it does, you're going to be hit with 156 00:07:20.040 --> 00:07:22.439 a like, very realistic problem. It's going to cost a 157 00:07:22.480 --> 00:07:24.839 lot more money to be running your app at you know, 158 00:07:24.959 --> 00:07:28.600 nine am versus midnight if it's a global application, to 159 00:07:28.639 --> 00:07:32.319 get into having this spread apart things and allocate resources 160 00:07:32.360 --> 00:07:35.040 appropriately versus like the distribution of the request coming in 161 00:07:35.120 --> 00:07:38.360 and things like that. So it's a very complicated problem, and. 162 00:07:38.319 --> 00:07:41.079 Most of the time the people that have been sort 163 00:07:41.079 --> 00:07:43.160 of on the chopping block for figuring this out have 164 00:07:43.279 --> 00:07:45.959 not been the engineering teams who understand what the application 165 00:07:46.079 --> 00:07:47.839 is doing. So how do you get that back, Like, 166 00:07:47.879 --> 00:07:51.079 how do you align the incentives appropriately so that if 167 00:07:51.240 --> 00:07:53.519 the amount of money you're spending on your cloud provider 168 00:07:53.680 --> 00:07:57.120 or on capital allocation for on prem data center resources, 169 00:07:57.160 --> 00:07:59.439 how do you align that back with the software engineering teams. 170 00:07:59.519 --> 00:08:02.839 I think a little bit of it is reporting, just 171 00:08:02.879 --> 00:08:05.600 giving them visibility. Some of our customers call it shame 172 00:08:05.680 --> 00:08:07.839 back reporting. I'm like, yeah, I get it, but maybe like, 173 00:08:07.879 --> 00:08:10.279 don't call it that. But I think on the other hand, 174 00:08:10.319 --> 00:08:13.199 it's also just giving them the tooling and the insights 175 00:08:13.360 --> 00:08:16.519 that they know, they have the confidence in the systems 176 00:08:16.560 --> 00:08:18.800 that are kind of handling this for them. I really 177 00:08:18.839 --> 00:08:20.120 don't think you're going to get far if you go 178 00:08:20.199 --> 00:08:21.879 to a developer and you're like, hey, you need to 179 00:08:21.920 --> 00:08:24.519 change your configuration settings. This is costing us a lot. 180 00:08:24.800 --> 00:08:27.839 They're almost never going to be incentivized to do anything 181 00:08:27.879 --> 00:08:31.959 about it because their goal is deploy new applications. Move 182 00:08:31.959 --> 00:08:35.360 the business forward. And so I think I would say 183 00:08:35.399 --> 00:08:38.320 it's harder to incentivize them to do the work than 184 00:08:38.360 --> 00:08:40.960 it is to give them the tooling that allows the 185 00:08:41.000 --> 00:08:43.080 work to be done for them, while they can get 186 00:08:43.080 --> 00:08:44.960 input into the tooling. So like, if you say, hey, 187 00:08:45.000 --> 00:08:47.080 I'm going to do this for you, you have no insight, 188 00:08:47.200 --> 00:08:50.639 no ability to override anything, no ability to constrain it. 189 00:08:50.679 --> 00:08:53.200 They're gonna be like, no, you can't do that. For example, 190 00:08:53.279 --> 00:08:56.919 you can take in their concerns and codify that into 191 00:08:57.000 --> 00:08:59.759 the system that's handling doing the fixes for them, then 192 00:08:59.799 --> 00:09:02.919 you're there're a lot more incentivized to kind of move 193 00:09:02.919 --> 00:09:04.960 this forward because it's less work for them and they 194 00:09:05.039 --> 00:09:06.039 get the better outcome. 195 00:09:06.240 --> 00:09:08.080 So I can see like one of the core problems 196 00:09:08.080 --> 00:09:12.320 here is that maybe the team that's fundamentally responsible for 197 00:09:13.000 --> 00:09:16.000 managing the application like building it and running at the 198 00:09:16.039 --> 00:09:20.200 development team DevOps team, how how do they get the 199 00:09:20.360 --> 00:09:23.240 knowledge for what should be reasonable as far as spend goes, 200 00:09:23.559 --> 00:09:26.919 I do s disconnect, like do you have any I mean, it. 201 00:09:26.879 --> 00:09:29.200 Definitely depends on the organization of like how much they're 202 00:09:29.240 --> 00:09:32.559 spending because scale's different, right, Like a very very large 203 00:09:32.559 --> 00:09:34.360 bank is going to be spending a lot more than 204 00:09:34.440 --> 00:09:37.720 a ten person startup with not always because sometimes, especially 205 00:09:37.759 --> 00:09:40.720 with AI tools, ten person startups can be spending a 206 00:09:40.720 --> 00:09:44.000 lot of money. But the metric that we usually look 207 00:09:44.000 --> 00:09:48.519 at is waste percentage, So of everything that you're doing, 208 00:09:48.559 --> 00:09:50.879 how much of it are you actually using? And it's 209 00:09:50.919 --> 00:09:53.480 usually a better metric to go to folks and say, 210 00:09:53.480 --> 00:09:56.600 do you know you're like wasting seventy percent of the 211 00:09:56.639 --> 00:09:59.759 resources that you are paying for? So we don't go 212 00:09:59.840 --> 00:10:02.080 in and say, like, you should be spending less here 213 00:10:02.120 --> 00:10:04.399 and there, because sometimes you want to be spending more. 214 00:10:04.480 --> 00:10:07.639 It's just are you efficiently using the dollars that you're spending. 215 00:10:07.879 --> 00:10:11.639 Do you think that may optimize for the wrong thing though, 216 00:10:11.679 --> 00:10:15.360 that if you encourage teams to have high capacity utilization 217 00:10:15.639 --> 00:10:18.279 for their pods, nodes, or if they're using something else, 218 00:10:18.759 --> 00:10:21.159 let's say virtual machines, just like, oh the CPU is 219 00:10:21.200 --> 00:10:23.600 only at five percent, get we should use a much 220 00:10:23.600 --> 00:10:28.559 smaller machine to achieve the same thing. Optimizing at that 221 00:10:28.679 --> 00:10:32.200 level maybe optimizing the wrong part of the application, Like 222 00:10:32.240 --> 00:10:36.279 maybe the issue isn't that the utilization is too high 223 00:10:36.399 --> 00:10:39.080 or too low, but the thing that they've built is 224 00:10:39.559 --> 00:10:40.519 just wrong altogether. 225 00:10:40.799 --> 00:10:43.799 Yeah, Yeah, oftentimes it can also be just the app itself, 226 00:10:43.840 --> 00:10:46.399 the way that it's architected. And even sometimes when you 227 00:10:46.440 --> 00:10:49.159 write size, then you end up with a smaller but 228 00:10:49.320 --> 00:10:51.720 higher number of pods, and all of those pods are 229 00:10:51.720 --> 00:10:54.240 going to be making network calls. So, like, it's a 230 00:10:54.240 --> 00:10:57.399 lot more complex than just looking at resources. You do 231 00:10:57.519 --> 00:10:59.320 have to look at the whole system, and so you 232 00:10:59.399 --> 00:11:02.679 have to look at the actual application itself, how it's 233 00:11:02.919 --> 00:11:05.879 consuming metrics, Maybe it needs some changes within the application. 234 00:11:06.480 --> 00:11:08.600 What can you tweak in tune there there's no like 235 00:11:08.679 --> 00:11:11.320 one size fits all, here's what's going to save all 236 00:11:11.320 --> 00:11:13.200 your problems. But it's what can you start with and 237 00:11:13.240 --> 00:11:15.639 what's the lowest hanging fruit? I mean, honestly loose hanging 238 00:11:15.679 --> 00:11:17.879 fruit for most people is just shut off systems you're 239 00:11:17.919 --> 00:11:22.240 not using. And it's surprising how often that is the 240 00:11:22.279 --> 00:11:24.399 majority of the waste in an organization. 241 00:11:25.120 --> 00:11:27.720 That's that's actually surprising to hear, because I feel like 242 00:11:27.759 --> 00:11:31.759 Corey Quinn, if you're familiar with Internet personality, often jokes 243 00:11:31.759 --> 00:11:35.559 that the highest cost for you know, utilization is your 244 00:11:35.679 --> 00:11:38.480 backup buster or your you know, your database backup. You 245 00:11:38.519 --> 00:11:40.519 know you're never using that, And yeah, there's you know, 246 00:11:40.559 --> 00:11:42.399 a huge cost associated with that. I mean, there is 247 00:11:42.440 --> 00:11:45.519 something to be said like understanding your utilization patterns to 248 00:11:45.639 --> 00:11:48.080 be able to eliminate the waste in that way. Is 249 00:11:48.080 --> 00:11:49.799 that really, like, are we don't going to like fifty 250 00:11:49.799 --> 00:11:51.759 percent of the cost is just you know, right sizing 251 00:11:51.799 --> 00:11:52.440 and then you're done. 252 00:11:52.879 --> 00:11:55.840 Honestly, typically like sixty seventy percent of the waste is 253 00:11:56.000 --> 00:11:59.440 just it like sounds kind of a little embarrassing because 254 00:11:59.440 --> 00:12:02.120 you're like, it's just like set your CPU requests at 255 00:12:02.159 --> 00:12:02.759 your memory request. 256 00:12:02.759 --> 00:12:03.879 We're talking about two settings. 257 00:12:04.200 --> 00:12:06.120 And I feel like in the VM world that was 258 00:12:06.159 --> 00:12:08.559 at least a little bit easier to say, go set it. 259 00:12:08.879 --> 00:12:11.759 But in the container world, you now have sometimes millions 260 00:12:11.759 --> 00:12:16.679 of containers, and each individual container has different resource needs, 261 00:12:16.879 --> 00:12:19.440 and so it becomes more of a scale and like 262 00:12:19.519 --> 00:12:22.840 toil problem than then it's anything else. So I do 263 00:12:22.919 --> 00:12:24.480 think a lot of the waste, or at least what 264 00:12:24.519 --> 00:12:26.440 we see is a lot of the waste is just 265 00:12:26.600 --> 00:12:30.960 in right sizing. But honestly, there's also so we track 266 00:12:31.440 --> 00:12:36.440 you mentioned storage, We track kind of unused storage and 267 00:12:37.080 --> 00:12:40.080 backups that you haven't touched in ninety days, and the 268 00:12:40.200 --> 00:12:43.279 amount of pocs that we come across and we're like, 269 00:12:43.399 --> 00:12:46.080 just just turn this on and see what happens. People 270 00:12:46.120 --> 00:12:48.039 are like, oh wow, like I got a lot of 271 00:12:48.080 --> 00:12:51.799 waste there that is just easy. Go remove it, delete it, 272 00:12:51.799 --> 00:12:55.240 turn it off. It's surprisingly more often than I would 273 00:12:55.240 --> 00:12:55.879 have expected. 274 00:12:56.000 --> 00:12:58.399 Oh no, I'd say, I'll add to another problem I 275 00:12:58.440 --> 00:13:02.080 have seen. So it's also not just like I said 276 00:13:02.080 --> 00:13:02.960 it and forget. 277 00:13:02.639 --> 00:13:05.279 It kind of thing. I have seen. 278 00:13:06.559 --> 00:13:09.120 Issues where you have like a portion of the app 279 00:13:09.159 --> 00:13:12.279 that is maybe not touched as often as application grows, 280 00:13:12.279 --> 00:13:14.480 Like if they're only deploying it every six months, suddenly 281 00:13:14.639 --> 00:13:17.039 what you've requested like if you do a redeployment, but 282 00:13:17.320 --> 00:13:19.600 if you don't kind of babysit that you can run 283 00:13:19.600 --> 00:13:21.080 into problems too because it changes. 284 00:13:21.120 --> 00:13:22.799 Yeah, if you don't have policies in place that are 285 00:13:22.840 --> 00:13:26.360 continuously looking at this like okay, great, you were able 286 00:13:26.399 --> 00:13:30.240 to either write size or remove or downscale once, but 287 00:13:30.279 --> 00:13:32.480 then your application is going to change. And you mentioned 288 00:13:32.519 --> 00:13:35.320 like apps that change every six months, some change daily, 289 00:13:35.639 --> 00:13:37.080 so how are you going to stay on top of that? 290 00:13:38.240 --> 00:13:40.759 And one of the other things you had mentioned earlier 291 00:13:40.879 --> 00:13:43.639 was just horizontal scaling of you know, you need more 292 00:13:43.679 --> 00:13:46.879 resources at nine AM. Tools like the HPA, like CATA 293 00:13:47.000 --> 00:13:50.559 are great for that because they see requests are increasing. Okay, 294 00:13:50.559 --> 00:13:52.240 I'm going to give you more replicas allow you to 295 00:13:52.240 --> 00:13:55.600 scale horizontally, and that's awesome because it'll always help you 296 00:13:55.679 --> 00:13:59.080 keep up with traffic demand. But if you're not scaled, 297 00:13:59.320 --> 00:14:03.080 if you're not at your configurations correctly, you're just duplicating 298 00:14:03.120 --> 00:14:06.519 waste across the board. And it's very good from a 299 00:14:06.559 --> 00:14:09.600 reliability standpoint, but not very good from a waste management standpoint. 300 00:14:09.639 --> 00:14:10.399 So you kind of have to. 301 00:14:10.360 --> 00:14:13.559 Take the vertical approach alongside the horizontal. 302 00:14:13.840 --> 00:14:16.759 I had a train of thought, and I guess I 303 00:14:16.960 --> 00:14:19.399 got stuck on this a little bit. There. There is 304 00:14:19.440 --> 00:14:23.279 this aspect of I know, especially engineers in this area 305 00:14:23.399 --> 00:14:27.039 tend to want to optimize everything and and part of 306 00:14:27.039 --> 00:14:28.480 that there's a lot of products that come out. So 307 00:14:28.480 --> 00:14:30.159 I can imagine that someone wrote like some sort of 308 00:14:30.639 --> 00:14:33.600 AI based product that like you deploy into you Kubernetes 309 00:14:33.639 --> 00:14:36.200 cluster and automatically tries to write size all of the 310 00:14:36.240 --> 00:14:39.879 CPU usage and odd sizing, you know, whatever needs to 311 00:14:39.919 --> 00:14:43.240 be allocated. Like this seems like a genius idea, right. 312 00:14:43.639 --> 00:14:48.159 Are you describing storm Forge or look us up ahead 313 00:14:48.159 --> 00:14:51.720 of time, so we we don't use everyone's like, oh 314 00:14:51.759 --> 00:14:52.279 I use AI. 315 00:14:52.559 --> 00:14:54.600 We use machine learning, thank you, it's math. 316 00:14:54.759 --> 00:14:58.200 It's like that I can't do. It's not some fancy 317 00:14:58.279 --> 00:15:02.039 AI like we and we develop it in house. We 318 00:15:02.080 --> 00:15:04.399 have PhDs that work on this stuff. It's like patent pending. 319 00:15:04.480 --> 00:15:07.440 Depends on the audience sometimes like yeah, okay AI machine learning, 320 00:15:07.759 --> 00:15:10.159 but like it's not the same. And I feel very 321 00:15:10.960 --> 00:15:15.440 passionate about just making that differentiation because I mentioned this 322 00:15:15.559 --> 00:15:17.679 is a scale problem. When you're looking at the different 323 00:15:17.679 --> 00:15:19.799 settings at the scale, then that just means it's a 324 00:15:19.799 --> 00:15:23.919 really complicated math problem. And so we use the math 325 00:15:24.039 --> 00:15:26.200 to help solve that problem. We do look at the 326 00:15:26.799 --> 00:15:30.399 trends in the data, like usage patterns, HPA, scaling patterns, 327 00:15:30.480 --> 00:15:32.720 all of that, and then we pull the metrics straight 328 00:15:32.759 --> 00:15:33.519 from Kubernets. 329 00:15:33.559 --> 00:15:35.080 It goes into a Prometheus. 330 00:15:34.720 --> 00:15:38.679 Database, and then our algorithms look at those scaling behaviors 331 00:15:38.720 --> 00:15:41.360 and come up with what those right configuration settings should be. 332 00:15:41.480 --> 00:15:43.600 You could say like okay, cool that anyone can do 333 00:15:43.679 --> 00:15:46.159 that and come up with the configuration settings. I think 334 00:15:46.879 --> 00:15:50.279 what makes us unique is two things. One, the horizontal 335 00:15:51.399 --> 00:15:54.360 scaling behavior we talked about before where people usually use 336 00:15:54.399 --> 00:15:58.000 the HPA KETA. Those are the most common tools when 337 00:15:58.000 --> 00:16:00.679 it comes to Kubernati scaling and have to work with 338 00:16:00.720 --> 00:16:03.840 those tools. You can't work against them because if you're 339 00:16:03.840 --> 00:16:06.879 going to go in and vertically right size a container, 340 00:16:06.919 --> 00:16:10.519 for example, and it's horizontally scaling, then the HPA is 341 00:16:10.519 --> 00:16:13.200 gonna be like, oh, well, you're using more of your 342 00:16:13.279 --> 00:16:17.159 downsize so you're moving using sorry, using more of your resources. 343 00:16:17.799 --> 00:16:20.080 I need to go give you more replicas. And then 344 00:16:20.240 --> 00:16:23.919 the VPA or any other vertical right sizing solution would 345 00:16:23.919 --> 00:16:26.360 be like, well, okay, now you need more resources. So 346 00:16:26.600 --> 00:16:29.519 you get into that turn cycle. And one of the 347 00:16:29.519 --> 00:16:32.080 things we look at is your horizontal scaling behavior in 348 00:16:32.120 --> 00:16:35.600 addition to your vertical right sizing requests, and then set 349 00:16:35.639 --> 00:16:39.039 those together so that you can continue using the tools 350 00:16:39.039 --> 00:16:41.120 that you are using that are open source, that like 351 00:16:41.759 --> 00:16:44.159 I would never want to ask anyone to rip out, 352 00:16:44.799 --> 00:16:47.399 and then also kind of reap the benefits of vertical 353 00:16:47.480 --> 00:16:49.879 right sizing and then the other pieces we've been talking 354 00:16:49.919 --> 00:16:52.960 about developers and how to incentivize them and input into 355 00:16:53.000 --> 00:16:55.840 it is important for that confidence to be able to 356 00:16:55.879 --> 00:16:57.200 actually do this automatically. 357 00:16:57.440 --> 00:16:59.039 You say that and now I'm just wondering if this 358 00:16:59.080 --> 00:17:01.519 isn't a solved problem, like why is it that, say 359 00:17:01.519 --> 00:17:05.160 that JVM or Kubernetes pods or even Docker containers, Like 360 00:17:05.319 --> 00:17:07.640 why am I even serveralist solutions if I'm running on 361 00:17:07.799 --> 00:17:12.119 VERSL or aws lambda functions? Why am I specifying like 362 00:17:12.319 --> 00:17:15.960 timeouts and CPU and memory usage? Like why is it 363 00:17:16.039 --> 00:17:19.240 just not automatically determining based off of I don't know 364 00:17:19.319 --> 00:17:22.359 the last ten years and daily load, but how it 365 00:17:22.440 --> 00:17:25.680 changes over week by week, month by month where holidays 366 00:17:25.680 --> 00:17:28.119 are based off geographic regions, Like why not just take 367 00:17:28.119 --> 00:17:30.359 all this stuff and just automatically figure out what the 368 00:17:30.359 --> 00:17:33.119 scaling capacity needs to be? Like I guarantee you as 369 00:17:33.240 --> 00:17:35.319 when I was an engineer, I definitely could not have 370 00:17:35.319 --> 00:17:37.440 figured out like how many requests I was going to 371 00:17:37.519 --> 00:17:40.559 get a week from now, let alone a month from now, 372 00:17:40.759 --> 00:17:42.519 or how is that going to change based off of 373 00:17:42.519 --> 00:17:44.640 whatever marketing campaigns are being run, et cetera. 374 00:17:44.839 --> 00:17:47.279 Yeah, So I mean the VPA that is open source 375 00:17:47.559 --> 00:17:50.880 comes with kubernatores allows you to do just basic like okay, 376 00:17:50.880 --> 00:17:53.000 look at my past, I think week and usage and 377 00:17:53.559 --> 00:17:56.039 come up using P ninety five with what my request 378 00:17:56.079 --> 00:18:00.000 should be and for like basic deployments that works right, 379 00:18:00.759 --> 00:18:02.680 look at what I did in the last week and 380 00:18:02.720 --> 00:18:05.920 then use that and set my request for me. I 381 00:18:05.920 --> 00:18:08.640 think some of the challenges are the VPA if it's 382 00:18:08.839 --> 00:18:11.799 scaling on CPU or memory, doesn't work well with the 383 00:18:11.920 --> 00:18:14.920 HPA what I mentioned earlier. So you're like, if you're 384 00:18:14.920 --> 00:18:17.400 going to choose between something that's horizontally scaling you or 385 00:18:17.480 --> 00:18:19.759 vertically scaling you, You're gonna go horizontal because it's more 386 00:18:19.799 --> 00:18:22.279 about reliability and how do you keep up with your 387 00:18:22.279 --> 00:18:25.160 traffic patterns versus like vpas more about how I reduce 388 00:18:25.200 --> 00:18:25.720 the waste. 389 00:18:25.799 --> 00:18:28.640 And then if you if you're an organization. 390 00:18:28.240 --> 00:18:32.200 That maybe has more cyclical patterns, things change over time. 391 00:18:32.240 --> 00:18:34.720 You want to be looking at like a quarter's worth, 392 00:18:34.759 --> 00:18:37.039 a year's worth of data, where you're going to store 393 00:18:37.039 --> 00:18:39.039 that data, How are you going to look into that data? 394 00:18:39.319 --> 00:18:41.680 It all costs money. So then that's where kind of 395 00:18:41.720 --> 00:18:43.799 vendors come into the mix. To be honest, if you 396 00:18:43.799 --> 00:18:46.119 wanted to build something yourself, you could, But do you 397 00:18:46.160 --> 00:18:48.920 want to be into the business of kind of building 398 00:18:48.960 --> 00:18:52.680 systems that manage your own systems or do you want 399 00:18:52.680 --> 00:18:54.759 to be in the business of building applications that move 400 00:18:54.799 --> 00:18:55.559 your business forward. 401 00:18:55.599 --> 00:18:57.319