1 00:00:08,080 --> 00:00:11,080 Speaker 1: Hello everyone, and welcome back to another episode of Adventures 2 00:00:11,080 --> 00:00:14,320 in DevOps. And I'm joined today again by my co 3 00:00:14,400 --> 00:00:16,760 host Amy Knight. How you doing, Amy? 4 00:00:16,879 --> 00:00:18,039 Speaker 2: Pretty good? Walking in? 5 00:00:19,839 --> 00:00:22,239 Speaker 1: She likes walking, that's uh. 6 00:00:22,440 --> 00:00:23,559 Speaker 2: I gotta get my steps in. 7 00:00:25,879 --> 00:00:28,280 Speaker 1: I don't have a good strategy for that yet. Yeah, 8 00:00:28,039 --> 00:00:31,839 so today I have to be completely honest. The topic 9 00:00:32,000 --> 00:00:34,759 is in and around finnops. But I don't really know 10 00:00:34,799 --> 00:00:37,679 what finnopps is. But if it's anything like DevOps, I'm 11 00:00:37,679 --> 00:00:40,280 guessing it means firing all your business analysts and hiring 12 00:00:40,320 --> 00:00:43,200 some additional software engineers to do a job they don't understand, 13 00:00:43,520 --> 00:00:46,200 and then, after your years of failing, reappropriate the name 14 00:00:46,240 --> 00:00:48,880 to mean something else and claim success. Well, Amy, you're 15 00:00:49,039 --> 00:00:51,119 already smiling, so maybe you know something I don't know. 16 00:00:52,039 --> 00:00:54,640 Speaker 3: I don't actually laughing because it's very entertaining. 17 00:00:54,759 --> 00:00:55,679 Speaker 2: And I also. 18 00:00:57,039 --> 00:01:00,119 Speaker 3: I think, like last week, I realized your title on 19 00:01:00,159 --> 00:01:02,399 LinkedIn a tech entertainer. 20 00:01:03,039 --> 00:01:04,680 Speaker 2: So that was a very entertaining intro. 21 00:01:05,280 --> 00:01:08,599 Speaker 1: You did a great job, well, thank you. So we 22 00:01:08,680 --> 00:01:11,680 pulled an expert from the field, chief strategy officer at 23 00:01:12,040 --> 00:01:18,159 cloud Bolt, Yasmin Rajabi, and I've noticed here so you 24 00:01:18,480 --> 00:01:20,400 have a history of product management, and so I'm really 25 00:01:20,400 --> 00:01:21,519 excited to have you on the show. 26 00:01:21,760 --> 00:01:23,959 Speaker 4: Welcome, thanks for having me excited to be here. 27 00:01:24,159 --> 00:01:28,480 Speaker 1: How completely off base was my thoughts about what finof says. 28 00:01:28,719 --> 00:01:30,359 Speaker 4: I mean, not so far off base. 29 00:01:31,319 --> 00:01:35,079 Speaker 5: Any time you put like an ops inside in one 30 00:01:35,079 --> 00:01:38,239 of the words, you're obviously trying to force some things together. 31 00:01:38,439 --> 00:01:39,159 Speaker 4: And it's funny. 32 00:01:39,159 --> 00:01:41,959 Speaker 5: A question I get asked often is like, okay, so 33 00:01:42,040 --> 00:01:45,640 how are finnops teams and DevOps teams coming together? And 34 00:01:45,799 --> 00:01:48,519 the OPS in pinnops is supposed to be for DevOps, 35 00:01:48,599 --> 00:01:51,359 so technically like they're supposed to already be together. They're 36 00:01:51,400 --> 00:01:54,120 not supposed to be separate teams. But there's always the 37 00:01:54,560 --> 00:01:57,159 ideal and then the reality of if you take financial 38 00:01:57,239 --> 00:02:00,840 analysts and you take engineers and you want to bring 39 00:02:00,840 --> 00:02:05,079 them together, they speak different languages, they come from different worlds, 40 00:02:05,079 --> 00:02:08,719 So there are folks that are trying to use software 41 00:02:08,759 --> 00:02:10,840 to bring that together. I'd put us in the mix 42 00:02:10,879 --> 00:02:15,199 of that, but it's still like people from completely different 43 00:02:15,199 --> 00:02:17,240 worlds and trying to get them to think and care 44 00:02:17,319 --> 00:02:18,319 about the same topic. 45 00:02:18,680 --> 00:02:21,199 Speaker 3: Okay, I was going to add too, we also need 46 00:02:21,240 --> 00:02:25,759 to add in is it greenops. So it's like getting 47 00:02:25,840 --> 00:02:29,080 environmentally savvy as well within the realm of finnops. 48 00:02:29,080 --> 00:02:31,800 Speaker 5: Okay, well, I realized that I need to reduce the waste, 49 00:02:31,840 --> 00:02:33,319 but I want to be able to measure that from 50 00:02:33,319 --> 00:02:36,639 a like my carbon footprint standpoint. And it was interesting 51 00:02:36,680 --> 00:02:38,680 I was talking to a Phinnops person that is using 52 00:02:38,759 --> 00:02:42,599 those metrics to drive people to care inside their organization, 53 00:02:42,680 --> 00:02:45,000 because the people in the organization is like, Okay, well, 54 00:02:45,039 --> 00:02:46,800 it's not my money, I'm not spending it. But when 55 00:02:46,879 --> 00:02:50,199 you actually tie it to the environmental impact, the kind 56 00:02:50,199 --> 00:02:54,479 of human nature part of people helps kind of get 57 00:02:54,520 --> 00:02:56,960 people a little bit more passionate about reducing the waste 58 00:02:57,039 --> 00:02:59,240 inside their inside their organization. 59 00:02:59,360 --> 00:03:01,479 Speaker 3: As far as the step, I did kind of lap 60 00:03:01,520 --> 00:03:02,240 it off before. 61 00:03:02,319 --> 00:03:03,680 Speaker 2: But I saw. 62 00:03:03,639 --> 00:03:05,960 Speaker 3: Something recently about like as I're putting more and more 63 00:03:06,039 --> 00:03:08,960 data centers up, like the water shortage that's happening to 64 00:03:09,000 --> 00:03:10,360 the communities around them. 65 00:03:10,319 --> 00:03:13,479 Speaker 5: Some cities are literally like their grid cannot handle any 66 00:03:13,520 --> 00:03:16,520 more data centers, data centers coming online, and like they're 67 00:03:16,639 --> 00:03:19,680 starting to buy in different area, Like it'll be interesting 68 00:03:19,680 --> 00:03:21,319 what things look like in ten years. 69 00:03:21,360 --> 00:03:23,560 Speaker 3: Like I saw like these poor families like literally like 70 00:03:23,560 --> 00:03:26,000 they can't run their appliances as they could like a 71 00:03:26,000 --> 00:03:26,680 couple of years ago. 72 00:03:26,840 --> 00:03:28,039 Speaker 2: Yeah, anyway, Okay, So I. 73 00:03:28,000 --> 00:03:30,840 Speaker 1: Think that's a really interesting topic because we've had a 74 00:03:30,879 --> 00:03:33,960 lot of our previous guests on the show, either they've 75 00:03:33,960 --> 00:03:36,879 been AI focused in the last few months or they've 76 00:03:36,919 --> 00:03:39,919 had a unique insight into data center operations. One of 77 00:03:39,960 --> 00:03:42,000 the ideas that keeps coming up is that it's actually 78 00:03:42,000 --> 00:03:46,520 not a struggle to get energy reservations for the data centers. 79 00:03:46,879 --> 00:03:49,680 But it's interesting that you bring up the water impact 80 00:03:49,680 --> 00:03:51,719 because that's a new thing that I'm not familiar with. 81 00:03:52,280 --> 00:03:55,240 Speaker 5: Where it kind of interplays with water is probably less 82 00:03:55,280 --> 00:03:58,280 of my expertise and background, just the fact that you know, 83 00:03:58,319 --> 00:04:00,159 you need the water to cool down the data center. 84 00:04:00,639 --> 00:04:02,879 The more the more power it's using, then it becomes 85 00:04:03,080 --> 00:04:04,199 it's just like a cycle. 86 00:04:04,719 --> 00:04:07,240 Speaker 4: Though I would say their. 87 00:04:06,560 --> 00:04:10,120 Speaker 5: Ability to use liquid cooling and the data centers reuses 88 00:04:10,159 --> 00:04:13,479 the water instead of an endless supply of water. The 89 00:04:13,719 --> 00:04:17,000 recycling the water that exists has at least helped make 90 00:04:17,040 --> 00:04:19,959 some of an impact. What's interesting is people are pulling 91 00:04:20,000 --> 00:04:23,839 these metrics out and trying to overlay them on their 92 00:04:23,959 --> 00:04:27,199 usage as well. So, Okay, I'm writing an application I 93 00:04:27,319 --> 00:04:30,079 deploy it. I never think about these things, but as 94 00:04:30,079 --> 00:04:32,240 I can start to pull some of those metrics out 95 00:04:32,319 --> 00:04:37,079 of either my cloud provider as they start to provide them. 96 00:04:37,199 --> 00:04:39,800 I think Microsoft started to pull in like carbon data 97 00:04:39,839 --> 00:04:41,920 and other metrics that you can pull up or if 98 00:04:41,920 --> 00:04:43,759 I'm running my own data center. Then being able to 99 00:04:43,759 --> 00:04:46,199 pull those metrics out then kind of overlays and says, hey, 100 00:04:46,199 --> 00:04:48,480 here's the impact of what you're doing, Like make sure 101 00:04:48,480 --> 00:04:52,040 you're setting these things correctly. Otherwise there's there's a lot 102 00:04:52,199 --> 00:04:53,680 bigger impact than just financial. 103 00:04:53,920 --> 00:04:56,759 Speaker 1: What's a good example of needing to pull in accounting 104 00:04:56,879 --> 00:04:59,920 or financial operations into software. 105 00:05:00,879 --> 00:05:04,560 Speaker 5: So I'll say for most of my background in this 106 00:05:04,680 --> 00:05:08,160 space is on the Kuberneti side, and when you're deploying 107 00:05:08,160 --> 00:05:11,360 a Kubernetes app, you need to set requests and limits, 108 00:05:11,399 --> 00:05:14,519 one from a reliability standpoint to make sure you actually 109 00:05:14,519 --> 00:05:17,800 have the resources, but two typically what people do is 110 00:05:17,839 --> 00:05:19,279 just like set them really high so they don't have 111 00:05:19,319 --> 00:05:21,879 to think about it, and then deploy the application. And 112 00:05:22,800 --> 00:05:25,439 it's really hard to understand the impact of that, especially 113 00:05:25,480 --> 00:05:28,079 because that's at the pod level and you pay for 114 00:05:28,160 --> 00:05:30,360 nodes and you're like, okay, well, I don't really know 115 00:05:30,399 --> 00:05:33,240 how my pods translate into my nodes. So I'm paying 116 00:05:33,319 --> 00:05:35,199 some amount of money, but how do I actually know 117 00:05:35,279 --> 00:05:38,199 the impact? And so being able to pull in both 118 00:05:38,240 --> 00:05:42,199 the billing data and then actually allocate it correctly, because 119 00:05:42,240 --> 00:05:44,600 one of the challenges in Kubernetes is, Okay, how are 120 00:05:44,600 --> 00:05:46,839 you going to allocate your costs? Do you just look 121 00:05:46,879 --> 00:05:48,959 across name spaces and say like, okay, I'm going to 122 00:05:49,399 --> 00:05:51,360 divide it by name spaces and then that's what you get. 123 00:05:52,319 --> 00:05:55,120 But what if you have a larger name space or 124 00:05:55,879 --> 00:05:59,199 you have how do you split out like cube system 125 00:05:59,319 --> 00:06:02,040 and all of anything on the control plane. That's some 126 00:06:02,079 --> 00:06:04,759 of the challenges that people have. And if you're what 127 00:06:04,800 --> 00:06:06,800 you need to be doing is pulling in that billing 128 00:06:06,879 --> 00:06:09,839 data to say here's how it's costing you, here's how 129 00:06:09,879 --> 00:06:11,199 we can split that out if we want to split 130 00:06:11,279 --> 00:06:13,079 up the pod level, and then you can be able 131 00:06:13,079 --> 00:06:16,079 to tie that back to the actual nodes that you 132 00:06:16,839 --> 00:06:17,879 pay your bill on. 133 00:06:18,160 --> 00:06:21,040 Speaker 1: So would it be accurate to say that Historically maybe 134 00:06:21,319 --> 00:06:23,920 software engineering teams haven't been held accountable for how much 135 00:06:23,959 --> 00:06:24,560 they spend in. 136 00:06:24,519 --> 00:06:27,079 Speaker 5: The cloud for sure, And often what happens is the 137 00:06:27,120 --> 00:06:31,480 developers are setting those requests, and then the platform engineering 138 00:06:31,519 --> 00:06:33,639 team are the ones that are like, Okay, well we're 139 00:06:33,680 --> 00:06:37,399 managing the cloud environment, we're managing those bills. They get 140 00:06:37,439 --> 00:06:39,560 asked to do something about it, and I don't know 141 00:06:39,600 --> 00:06:40,959 the app. What do you want me to do? If 142 00:06:41,000 --> 00:06:42,759 I touch something, I'm going to get a phone call 143 00:06:42,800 --> 00:06:45,600 from probably not a phone call, slack message from a 144 00:06:45,680 --> 00:06:48,600 developer of like, hey, what do you do to my system? 145 00:06:48,839 --> 00:06:50,920 It's down, it's having challenges. You are the last one 146 00:06:50,920 --> 00:06:51,360 to touch it. 147 00:06:51,720 --> 00:06:53,360 Speaker 1: Amy having experienced there. 148 00:06:53,360 --> 00:06:57,279 Speaker 3: Yeah, this is a very real problem. I would say, 149 00:06:57,360 --> 00:07:01,319 like the requests and the mins also, and I think 150 00:07:01,399 --> 00:07:03,959 like if a scale gets large enough, you also need 151 00:07:03,959 --> 00:07:09,199 to understand like traffic patterns and scaling appropriately for that, 152 00:07:10,439 --> 00:07:12,759 because you know, if you're an early stage company, you 153 00:07:12,800 --> 00:07:14,759 can just kind of scale at like a you know, 154 00:07:14,879 --> 00:07:17,720 relative level and stay there. If the app really grows, 155 00:07:17,720 --> 00:07:19,959 which hopefully it does, you're going to be hit with 156 00:07:20,040 --> 00:07:22,439 a like, very realistic problem. It's going to cost a 157 00:07:22,480 --> 00:07:24,839 lot more money to be running your app at you know, 158 00:07:24,959 --> 00:07:28,600 nine am versus midnight if it's a global application, to 159 00:07:28,639 --> 00:07:32,319 get into having this spread apart things and allocate resources 160 00:07:32,360 --> 00:07:35,040 appropriately versus like the distribution of the request coming in 161 00:07:35,120 --> 00:07:38,360 and things like that. So it's a very complicated problem, and. 162 00:07:38,319 --> 00:07:41,079 Speaker 1: Most of the time the people that have been sort 163 00:07:41,079 --> 00:07:43,160 of on the chopping block for figuring this out have 164 00:07:43,279 --> 00:07:45,959 not been the engineering teams who understand what the application 165 00:07:46,079 --> 00:07:47,839 is doing. So how do you get that back, Like, 166 00:07:47,879 --> 00:07:51,079 how do you align the incentives appropriately so that if 167 00:07:51,240 --> 00:07:53,519 the amount of money you're spending on your cloud provider 168 00:07:53,680 --> 00:07:57,120 or on capital allocation for on prem data center resources, 169 00:07:57,160 --> 00:07:59,439 how do you align that back with the software engineering teams. 170 00:07:59,519 --> 00:08:02,839 Speaker 5: I think a little bit of it is reporting, just 171 00:08:02,879 --> 00:08:05,600 giving them visibility. Some of our customers call it shame 172 00:08:05,680 --> 00:08:07,839 back reporting. I'm like, yeah, I get it, but maybe like, 173 00:08:07,879 --> 00:08:10,279 don't call it that. But I think on the other hand, 174 00:08:10,319 --> 00:08:13,199 it's also just giving them the tooling and the insights 175 00:08:13,360 --> 00:08:16,519 that they know, they have the confidence in the systems 176 00:08:16,560 --> 00:08:18,800 that are kind of handling this for them. I really 177 00:08:18,839 --> 00:08:20,120 don't think you're going to get far if you go 178 00:08:20,199 --> 00:08:21,879 to a developer and you're like, hey, you need to 179 00:08:21,920 --> 00:08:24,519 change your configuration settings. This is costing us a lot. 180 00:08:24,800 --> 00:08:27,839 They're almost never going to be incentivized to do anything 181 00:08:27,879 --> 00:08:31,959 about it because their goal is deploy new applications. Move 182 00:08:31,959 --> 00:08:35,360 the business forward. And so I think I would say 183 00:08:35,399 --> 00:08:38,320 it's harder to incentivize them to do the work than 184 00:08:38,360 --> 00:08:40,960 it is to give them the tooling that allows the 185 00:08:41,000 --> 00:08:43,080 work to be done for them, while they can get 186 00:08:43,080 --> 00:08:44,960 input into the tooling. So like, if you say, hey, 187 00:08:45,000 --> 00:08:47,080 I'm going to do this for you, you have no insight, 188 00:08:47,200 --> 00:08:50,639 no ability to override anything, no ability to constrain it. 189 00:08:50,679 --> 00:08:53,200 They're gonna be like, no, you can't do that. For example, 190 00:08:53,279 --> 00:08:56,919 you can take in their concerns and codify that into 191 00:08:57,000 --> 00:08:59,759 the system that's handling doing the fixes for them, then 192 00:08:59,799 --> 00:09:02,919 you're there're a lot more incentivized to kind of move 193 00:09:02,919 --> 00:09:04,960 this forward because it's less work for them and they 194 00:09:05,039 --> 00:09:06,039 get the better outcome. 195 00:09:06,240 --> 00:09:08,080 Speaker 1: So I can see like one of the core problems 196 00:09:08,080 --> 00:09:12,320 here is that maybe the team that's fundamentally responsible for 197 00:09:13,000 --> 00:09:16,000 managing the application like building it and running at the 198 00:09:16,039 --> 00:09:20,200 development team DevOps team, how how do they get the 199 00:09:20,360 --> 00:09:23,240 knowledge for what should be reasonable as far as spend goes, 200 00:09:23,559 --> 00:09:26,919 I do s disconnect, like do you have any I mean, it. 201 00:09:26,879 --> 00:09:29,200 Speaker 5: Definitely depends on the organization of like how much they're 202 00:09:29,240 --> 00:09:32,559 spending because scale's different, right, Like a very very large 203 00:09:32,559 --> 00:09:34,360 bank is going to be spending a lot more than 204 00:09:34,440 --> 00:09:37,720 a ten person startup with not always because sometimes, especially 205 00:09:37,759 --> 00:09:40,720 with AI tools, ten person startups can be spending a 206 00:09:40,720 --> 00:09:44,000 lot of money. But the metric that we usually look 207 00:09:44,000 --> 00:09:48,519 at is waste percentage, So of everything that you're doing, 208 00:09:48,559 --> 00:09:50,879 how much of it are you actually using? And it's 209 00:09:50,919 --> 00:09:53,480 usually a better metric to go to folks and say, 210 00:09:53,480 --> 00:09:56,600 do you know you're like wasting seventy percent of the 211 00:09:56,639 --> 00:09:59,759 resources that you are paying for? So we don't go 212 00:09:59,840 --> 00:10:02,080 in and say, like, you should be spending less here 213 00:10:02,120 --> 00:10:04,399 and there, because sometimes you want to be spending more. 214 00:10:04,480 --> 00:10:07,639 It's just are you efficiently using the dollars that you're spending. 215 00:10:07,879 --> 00:10:11,639 Speaker 1: Do you think that may optimize for the wrong thing though, 216 00:10:11,679 --> 00:10:15,360 that if you encourage teams to have high capacity utilization 217 00:10:15,639 --> 00:10:18,279 for their pods, nodes, or if they're using something else, 218 00:10:18,759 --> 00:10:21,159 let's say virtual machines, just like, oh the CPU is 219 00:10:21,200 --> 00:10:23,600 only at five percent, get we should use a much 220 00:10:23,600 --> 00:10:28,559 smaller machine to achieve the same thing. Optimizing at that 221 00:10:28,679 --> 00:10:32,200 level maybe optimizing the wrong part of the application, Like 222 00:10:32,240 --> 00:10:36,279 maybe the issue isn't that the utilization is too high 223 00:10:36,399 --> 00:10:39,080 or too low, but the thing that they've built is 224 00:10:39,559 --> 00:10:40,519 just wrong altogether. 225 00:10:40,799 --> 00:10:43,799 Speaker 5: Yeah, Yeah, oftentimes it can also be just the app itself, 226 00:10:43,840 --> 00:10:46,399 the way that it's architected. And even sometimes when you 227 00:10:46,440 --> 00:10:49,159 write size, then you end up with a smaller but 228 00:10:49,320 --> 00:10:51,720 higher number of pods, and all of those pods are 229 00:10:51,720 --> 00:10:54,240 going to be making network calls. So, like, it's a 230 00:10:54,240 --> 00:10:57,399 lot more complex than just looking at resources. You do 231 00:10:57,519 --> 00:10:59,320 have to look at the whole system, and so you 232 00:10:59,399 --> 00:11:02,679 have to look at the actual application itself, how it's 233 00:11:02,919 --> 00:11:05,879 consuming metrics, Maybe it needs some changes within the application. 234 00:11:06,480 --> 00:11:08,600 What can you tweak in tune there there's no like 235 00:11:08,679 --> 00:11:11,320 one size fits all, here's what's going to save all 236 00:11:11,320 --> 00:11:13,200 your problems. But it's what can you start with and 237 00:11:13,240 --> 00:11:15,639 what's the lowest hanging fruit? I mean, honestly loose hanging 238 00:11:15,679 --> 00:11:17,879 fruit for most people is just shut off systems you're 239 00:11:17,919 --> 00:11:22,240 not using. And it's surprising how often that is the 240 00:11:22,279 --> 00:11:24,399 majority of the waste in an organization. 241 00:11:25,120 --> 00:11:27,720 Speaker 1: That's that's actually surprising to hear, because I feel like 242 00:11:27,759 --> 00:11:31,759 Corey Quinn, if you're familiar with Internet personality, often jokes 243 00:11:31,759 --> 00:11:35,559 that the highest cost for you know, utilization is your 244 00:11:35,679 --> 00:11:38,480 backup buster or your you know, your database backup. You 245 00:11:38,519 --> 00:11:40,519 know you're never using that, And yeah, there's you know, 246 00:11:40,559 --> 00:11:42,399 a huge cost associated with that. I mean, there is 247 00:11:42,440 --> 00:11:45,519 something to be said like understanding your utilization patterns to 248 00:11:45,639 --> 00:11:48,080 be able to eliminate the waste in that way. Is 249 00:11:48,080 --> 00:11:49,799 that really, like, are we don't going to like fifty 250 00:11:49,799 --> 00:11:51,759 percent of the cost is just you know, right sizing 251 00:11:51,799 --> 00:11:52,440 and then you're done. 252 00:11:52,879 --> 00:11:55,840 Speaker 5: Honestly, typically like sixty seventy percent of the waste is 253 00:11:56,000 --> 00:11:59,440 just it like sounds kind of a little embarrassing because 254 00:11:59,440 --> 00:12:02,120 you're like, it's just like set your CPU requests at 255 00:12:02,159 --> 00:12:02,759 your memory request. 256 00:12:02,759 --> 00:12:03,879 Speaker 4: We're talking about two settings. 257 00:12:04,200 --> 00:12:06,120 Speaker 5: And I feel like in the VM world that was 258 00:12:06,159 --> 00:12:08,559 at least a little bit easier to say, go set it. 259 00:12:08,879 --> 00:12:11,759 But in the container world, you now have sometimes millions 260 00:12:11,759 --> 00:12:16,679 of containers, and each individual container has different resource needs, 261 00:12:16,879 --> 00:12:19,440 and so it becomes more of a scale and like 262 00:12:19,519 --> 00:12:22,840 toil problem than then it's anything else. So I do 263 00:12:22,919 --> 00:12:24,480 think a lot of the waste, or at least what 264 00:12:24,519 --> 00:12:26,440 we see is a lot of the waste is just 265 00:12:26,600 --> 00:12:30,960 in right sizing. But honestly, there's also so we track 266 00:12:31,440 --> 00:12:36,440 you mentioned storage, We track kind of unused storage and 267 00:12:37,080 --> 00:12:40,080 backups that you haven't touched in ninety days, and the 268 00:12:40,200 --> 00:12:43,279 amount of pocs that we come across and we're like, 269 00:12:43,399 --> 00:12:46,080 just just turn this on and see what happens. People 270 00:12:46,120 --> 00:12:48,039 are like, oh wow, like I got a lot of 271 00:12:48,080 --> 00:12:51,799 waste there that is just easy. Go remove it, delete it, 272 00:12:51,799 --> 00:12:55,240 turn it off. It's surprisingly more often than I would 273 00:12:55,240 --> 00:12:55,879 have expected. 274 00:12:56,000 --> 00:12:58,399 Speaker 3: Oh no, I'd say, I'll add to another problem I 275 00:12:58,440 --> 00:13:02,080 have seen. So it's also not just like I said 276 00:13:02,080 --> 00:13:02,960 it and forget. 277 00:13:02,639 --> 00:13:05,279 Speaker 2: It kind of thing. I have seen. 278 00:13:06,559 --> 00:13:09,120 Speaker 3: Issues where you have like a portion of the app 279 00:13:09,159 --> 00:13:12,279 that is maybe not touched as often as application grows, 280 00:13:12,279 --> 00:13:14,480 Like if they're only deploying it every six months, suddenly 281 00:13:14,639 --> 00:13:17,039 what you've requested like if you do a redeployment, but 282 00:13:17,320 --> 00:13:19,600 if you don't kind of babysit that you can run 283 00:13:19,600 --> 00:13:21,080 into problems too because it changes. 284 00:13:21,120 --> 00:13:22,799 Speaker 5: Yeah, if you don't have policies in place that are 285 00:13:22,840 --> 00:13:26,360 continuously looking at this like okay, great, you were able 286 00:13:26,399 --> 00:13:30,240 to either write size or remove or downscale once, but 287 00:13:30,279 --> 00:13:32,480 then your application is going to change. And you mentioned 288 00:13:32,519 --> 00:13:35,320 like apps that change every six months, some change daily, 289 00:13:35,639 --> 00:13:37,080 so how are you going to stay on top of that? 290 00:13:38,240 --> 00:13:40,759 And one of the other things you had mentioned earlier 291 00:13:40,879 --> 00:13:43,639 was just horizontal scaling of you know, you need more 292 00:13:43,679 --> 00:13:46,879 resources at nine AM. Tools like the HPA, like CATA 293 00:13:47,000 --> 00:13:50,559 are great for that because they see requests are increasing. Okay, 294 00:13:50,559 --> 00:13:52,240 I'm going to give you more replicas allow you to 295 00:13:52,240 --> 00:13:55,600 scale horizontally, and that's awesome because it'll always help you 296 00:13:55,679 --> 00:13:59,080 keep up with traffic demand. But if you're not scaled, 297 00:13:59,320 --> 00:14:03,080 if you're not at your configurations correctly, you're just duplicating 298 00:14:03,120 --> 00:14:06,519 waste across the board. And it's very good from a 299 00:14:06,559 --> 00:14:09,600 reliability standpoint, but not very good from a waste management standpoint. 300 00:14:09,639 --> 00:14:10,399 Speaker 4: So you kind of have to. 301 00:14:10,360 --> 00:14:13,559 Speaker 5: Take the vertical approach alongside the horizontal. 302 00:14:13,840 --> 00:14:16,759 Speaker 1: I had a train of thought, and I guess I 303 00:14:16,960 --> 00:14:19,399 got stuck on this a little bit. There. There is 304 00:14:19,440 --> 00:14:23,279 this aspect of I know, especially engineers in this area 305 00:14:23,399 --> 00:14:27,039 tend to want to optimize everything and and part of 306 00:14:27,039 --> 00:14:28,480 that there's a lot of products that come out. So 307 00:14:28,480 --> 00:14:30,159 I can imagine that someone wrote like some sort of 308 00:14:30,639 --> 00:14:33,600 AI based product that like you deploy into you Kubernetes 309 00:14:33,639 --> 00:14:36,200 cluster and automatically tries to write size all of the 310 00:14:36,240 --> 00:14:39,879 CPU usage and odd sizing, you know, whatever needs to 311 00:14:39,919 --> 00:14:43,240 be allocated. Like this seems like a genius idea, right. 312 00:14:43,639 --> 00:14:48,159 Speaker 5: Are you describing storm Forge or look us up ahead 313 00:14:48,159 --> 00:14:51,720 of time, so we we don't use everyone's like, oh 314 00:14:51,759 --> 00:14:52,279 I use AI. 315 00:14:52,559 --> 00:14:54,600 Speaker 4: We use machine learning, thank you, it's math. 316 00:14:54,759 --> 00:14:58,200 Speaker 5: It's like that I can't do. It's not some fancy 317 00:14:58,279 --> 00:15:02,039 AI like we and we develop it in house. We 318 00:15:02,080 --> 00:15:04,399 have PhDs that work on this stuff. It's like patent pending. 319 00:15:04,480 --> 00:15:07,440 Depends on the audience sometimes like yeah, okay AI machine learning, 320 00:15:07,759 --> 00:15:10,159 but like it's not the same. And I feel very 321 00:15:10,960 --> 00:15:15,440 passionate about just making that differentiation because I mentioned this 322 00:15:15,559 --> 00:15:17,679 is a scale problem. When you're looking at the different 323 00:15:17,679 --> 00:15:19,799 settings at the scale, then that just means it's a 324 00:15:19,799 --> 00:15:23,919 really complicated math problem. And so we use the math 325 00:15:24,039 --> 00:15:26,200 to help solve that problem. We do look at the 326 00:15:26,799 --> 00:15:30,399 trends in the data, like usage patterns, HPA, scaling patterns, 327 00:15:30,480 --> 00:15:32,720 all of that, and then we pull the metrics straight 328 00:15:32,759 --> 00:15:33,519 from Kubernets. 329 00:15:33,559 --> 00:15:35,080 Speaker 4: It goes into a Prometheus. 330 00:15:34,720 --> 00:15:38,679 Speaker 5: Database, and then our algorithms look at those scaling behaviors 331 00:15:38,720 --> 00:15:41,360 and come up with what those right configuration settings should be. 332 00:15:41,480 --> 00:15:43,600 You could say like okay, cool that anyone can do 333 00:15:43,679 --> 00:15:46,159 that and come up with the configuration settings. I think 334 00:15:46,879 --> 00:15:50,279 what makes us unique is two things. One, the horizontal 335 00:15:51,399 --> 00:15:54,360 scaling behavior we talked about before where people usually use 336 00:15:54,399 --> 00:15:58,000 the HPA KETA. Those are the most common tools when 337 00:15:58,000 --> 00:16:00,679 it comes to Kubernati scaling and have to work with 338 00:16:00,720 --> 00:16:03,840 those tools. You can't work against them because if you're 339 00:16:03,840 --> 00:16:06,879 going to go in and vertically right size a container, 340 00:16:06,919 --> 00:16:10,519 for example, and it's horizontally scaling, then the HPA is 341 00:16:10,519 --> 00:16:13,200 gonna be like, oh, well, you're using more of your 342 00:16:13,279 --> 00:16:17,159 downsize so you're moving using sorry, using more of your resources. 343 00:16:17,799 --> 00:16:20,080 I need to go give you more replicas. And then 344 00:16:20,240 --> 00:16:23,919 the VPA or any other vertical right sizing solution would 345 00:16:23,919 --> 00:16:26,360 be like, well, okay, now you need more resources. So 346 00:16:26,600 --> 00:16:29,519 you get into that turn cycle. And one of the 347 00:16:29,519 --> 00:16:32,080 things we look at is your horizontal scaling behavior in 348 00:16:32,120 --> 00:16:35,600 addition to your vertical right sizing requests, and then set 349 00:16:35,639 --> 00:16:39,039 those together so that you can continue using the tools 350 00:16:39,039 --> 00:16:41,120 that you are using that are open source, that like 351 00:16:41,759 --> 00:16:44,159 I would never want to ask anyone to rip out, 352 00:16:44,799 --> 00:16:47,399 and then also kind of reap the benefits of vertical 353 00:16:47,480 --> 00:16:49,879 right sizing and then the other pieces we've been talking 354 00:16:49,919 --> 00:16:52,960 about developers and how to incentivize them and input into 355 00:16:53,000 --> 00:16:55,840 it is important for that confidence to be able to 356 00:16:55,879 --> 00:16:57,200 actually do this automatically. 357 00:16:57,440 --> 00:16:59,039 Speaker 1: You say that and now I'm just wondering if this 358 00:16:59,080 --> 00:17:01,519 isn't a solved problem, like why is it that, say 359 00:17:01,519 --> 00:17:05,160 that JVM or Kubernetes pods or even Docker containers, Like 360 00:17:05,319 --> 00:17:07,640 why am I even serveralist solutions if I'm running on 361 00:17:07,799 --> 00:17:12,119 VERSL or aws lambda functions? Why am I specifying like 362 00:17:12,319 --> 00:17:15,960 timeouts and CPU and memory usage? Like why is it 363 00:17:16,039 --> 00:17:19,240 just not automatically determining based off of I don't know 364 00:17:19,319 --> 00:17:22,359 the last ten years and daily load, but how it 365 00:17:22,440 --> 00:17:25,680 changes over week by week, month by month where holidays 366 00:17:25,680 --> 00:17:28,119 are based off geographic regions, Like why not just take 367 00:17:28,119 --> 00:17:30,359 all this stuff and just automatically figure out what the 368 00:17:30,359 --> 00:17:33,119 scaling capacity needs to be? Like I guarantee you as 369 00:17:33,240 --> 00:17:35,319 when I was an engineer, I definitely could not have 370 00:17:35,319 --> 00:17:37,440 figured out like how many requests I was going to 371 00:17:37,519 --> 00:17:40,559 get a week from now, let alone a month from now, 372 00:17:40,759 --> 00:17:42,519 or how is that going to change based off of 373 00:17:42,519 --> 00:17:44,640 whatever marketing campaigns are being run, et cetera. 374 00:17:44,839 --> 00:17:47,279 Speaker 5: Yeah, So I mean the VPA that is open source 375 00:17:47,559 --> 00:17:50,880 comes with kubernatores allows you to do just basic like okay, 376 00:17:50,880 --> 00:17:53,000 look at my past, I think week and usage and 377 00:17:53,559 --> 00:17:56,039 come up using P ninety five with what my request 378 00:17:56,079 --> 00:18:00,000 should be and for like basic deployments that works right, 379 00:18:00,759 --> 00:18:02,680 look at what I did in the last week and 380 00:18:02,720 --> 00:18:05,920 then use that and set my request for me. I 381 00:18:05,920 --> 00:18:08,640 think some of the challenges are the VPA if it's 382 00:18:08,839 --> 00:18:11,799 scaling on CPU or memory, doesn't work well with the 383 00:18:11,920 --> 00:18:14,920 HPA what I mentioned earlier. So you're like, if you're 384 00:18:14,920 --> 00:18:17,400 going to choose between something that's horizontally scaling you or 385 00:18:17,480 --> 00:18:19,759 vertically scaling you, You're gonna go horizontal because it's more 386 00:18:19,799 --> 00:18:22,279 about reliability and how do you keep up with your 387 00:18:22,279 --> 00:18:25,160 traffic patterns versus like vpas more about how I reduce 388 00:18:25,200 --> 00:18:25,720 the waste. 389 00:18:25,799 --> 00:18:28,640 Speaker 4: And then if you if you're an organization. 390 00:18:28,240 --> 00:18:32,200 Speaker 5: That maybe has more cyclical patterns, things change over time. 391 00:18:32,240 --> 00:18:34,720 You want to be looking at like a quarter's worth, 392 00:18:34,759 --> 00:18:37,039 a year's worth of data, where you're going to store 393 00:18:37,039 --> 00:18:39,039 that data, How are you going to look into that data? 394 00:18:39,319 --> 00:18:41,680 It all costs money. So then that's where kind of 395 00:18:41,720 --> 00:18:43,799 vendors come into the mix. To be honest, if you 396 00:18:43,799 --> 00:18:46,119 wanted to build something yourself, you could, But do you 397 00:18:46,160 --> 00:18:48,920 want to be into the business of kind of building 398 00:18:48,960 --> 00:18:52,680 systems that manage your own systems or do you want 399 00:18:52,680 --> 00:18:54,759 to be in the business of building applications that move 400 00:18:54,799 --> 00:18:55,559 your business forward. 401 00:18:55,599 --> 00:18:57,319 Speaker 4: So that's kind of where where where we see it. 402 00:18:57,319 --> 00:19:00,720 Speaker 5: And sometimes I talk to companies that I've built something 403 00:19:00,720 --> 00:19:02,799 themselves and it works for them, and it's like, great, 404 00:19:02,799 --> 00:19:04,279 if it works for you, it works they could. 405 00:19:04,359 --> 00:19:06,319 Speaker 1: I'm gonna ask a really embarrassing question here, because I 406 00:19:06,359 --> 00:19:09,319 am a self proclaimed not Kubernetes expert. What is VPA 407 00:19:09,440 --> 00:19:10,119 and HP. 408 00:19:10,440 --> 00:19:14,480 Speaker 5: HPA's horizontal pod auto scaler. So HPA looks at the 409 00:19:14,799 --> 00:19:18,000 target utilization. It looks at your CPU and then you 410 00:19:18,000 --> 00:19:20,559 can set like a target utilization, say, if I'm using 411 00:19:20,599 --> 00:19:23,279 more than seventy percent of CPU, I want more pods, 412 00:19:23,319 --> 00:19:27,160 So give me more replicas copies of my my deployment 413 00:19:27,359 --> 00:19:30,920 if and it'll scale for you horizontally up as you 414 00:19:30,960 --> 00:19:32,279 need more, and it'll. 415 00:19:31,960 --> 00:19:33,359 Speaker 4: Go back down as you need less. 416 00:19:33,559 --> 00:19:36,240 Speaker 5: The VPA is a vertical pod auto scaler, and that 417 00:19:36,279 --> 00:19:39,599 looks at resources in a vertical manner of like, okay, 418 00:19:39,640 --> 00:19:41,960 you have x amount of CPU, so like you have 419 00:19:42,000 --> 00:19:44,480 maybe one hundred mili core, you need two hundred mili 420 00:19:44,480 --> 00:19:46,960 core or you need fifty mili core. Does the same 421 00:19:46,960 --> 00:19:50,000 for memory, both of them, of course, but that is 422 00:19:50,039 --> 00:19:52,519 what allows you to scale vertically. These are both tools 423 00:19:52,519 --> 00:19:57,359 that are open source, so anyone using Kubernetes can be 424 00:19:57,480 --> 00:19:59,880 using them. Should one hundred percent be using the HPA. 425 00:20:00,519 --> 00:20:03,119 There's almost never use cases where you shouldn't be using 426 00:20:03,119 --> 00:20:06,359 the HPA, though there are some, but they're tools available 427 00:20:06,400 --> 00:20:09,279 for folks and almost everyone. I think like data Dog 428 00:20:09,319 --> 00:20:13,160 did a survey now three four years ago where over 429 00:20:13,720 --> 00:20:17,559 half of their customers or prospects people that did the 430 00:20:17,559 --> 00:20:19,920 survey we're using the HPA, like one percent, we're using 431 00:20:19,960 --> 00:20:23,880 the VPA. Since then, the number's grown, but it's still 432 00:20:24,200 --> 00:20:27,440 majority HPA. Because it's very easy to use. It makes 433 00:20:27,480 --> 00:20:29,680 sense to be able to scale horse montally, and. 434 00:20:30,200 --> 00:20:33,000 Speaker 1: So one of the things that I think could be 435 00:20:33,000 --> 00:20:37,079 interesting here is you said, basically the number one thing 436 00:20:37,200 --> 00:20:41,960 that should be done is maybe time based load management, 437 00:20:42,119 --> 00:20:45,359 so reducing your capacity scaling down when you don't actually 438 00:20:45,400 --> 00:20:48,880 need these resources. I think this resonates with a lot 439 00:20:48,920 --> 00:20:52,799 of cases, especially not that long ago, and I'm sure 440 00:20:52,799 --> 00:20:54,279 there are still some companies that are doing this. 441 00:20:54,400 --> 00:20:54,720 Speaker 3: They have. 442 00:20:56,160 --> 00:21:01,240 Speaker 1: Managed action runners for GitHub or workspace runners for for 443 00:21:01,279 --> 00:21:04,599 get lab or whatever they're using for CICD, and they're 444 00:21:04,640 --> 00:21:07,279 not Those aren't being used when no one is doing 445 00:21:07,359 --> 00:21:11,480 software development. Obviously in this more remote world, that's harder 446 00:21:11,519 --> 00:21:15,279 to predict over time now, I think. But if that's 447 00:21:15,359 --> 00:21:17,240 number one, and I think that's sort of the obvious one, 448 00:21:17,279 --> 00:21:19,680 like what's number two in number three of what people 449 00:21:19,759 --> 00:21:23,640 could be doing or you frequently see as problematic or 450 00:21:23,839 --> 00:21:26,079 you know, high cost and could easily be cut out 451 00:21:26,160 --> 00:21:27,119 or maybe not too easily. 452 00:21:27,279 --> 00:21:30,559 Speaker 5: Yeah, So we talked about AI earlier, and just like 453 00:21:30,799 --> 00:21:33,279 you know, everyone's doing something AI, a lot of people 454 00:21:33,319 --> 00:21:35,359 have a bunch of data jobs, like they're using Spark, 455 00:21:36,359 --> 00:21:40,559 and so it's fairly easy to deploy jobs on Spark. 456 00:21:40,720 --> 00:21:45,440 And what I've seen is people will have a lot 457 00:21:45,480 --> 00:21:48,359 of waste in kind of similar gethub runner, like it'll 458 00:21:48,359 --> 00:21:51,200 go run, go do a bunch of stuff. How long 459 00:21:51,319 --> 00:21:53,519 does it take to go run and go do that stuff? 460 00:21:53,559 --> 00:21:56,039 How many resources does it take? Typically not an area 461 00:21:56,119 --> 00:21:58,400 you're going to go optimize because you're like, okay, well 462 00:21:58,480 --> 00:22:01,279 last maybe a couple of minutes, if even maybe it's 463 00:22:01,319 --> 00:22:03,559 under sixty seconds. But when you start to do that 464 00:22:03,599 --> 00:22:06,160 at scale, very similar to Spark jobs, that becomes a 465 00:22:06,240 --> 00:22:10,160 challenge because now you're running tens of thousands of these 466 00:22:10,279 --> 00:22:13,599 data jobs that are very compute intensive when they're running 467 00:22:13,640 --> 00:22:17,079 and then they go away, and how often are people 468 00:22:17,160 --> 00:22:19,480 pruning them making sure these are jobs that are actually 469 00:22:19,480 --> 00:22:22,319 still relevant to the application. What I've found is when 470 00:22:22,319 --> 00:22:27,279 it comes to AI related things, people are less thinking 471 00:22:27,319 --> 00:22:30,720 about how am I going to optimize this? More just okay, spence, 472 00:22:30,720 --> 00:22:33,079 spence bend, I need to just run it. Like GPUs 473 00:22:33,160 --> 00:22:36,319 come up often too, and people will ask us like, oh, 474 00:22:36,359 --> 00:22:39,519 are you optimizing GPUs? And my next question will be 475 00:22:39,720 --> 00:22:41,839 would you actually optimize your gp like do you have 476 00:22:41,839 --> 00:22:44,000 any initiatives? And they're like no, I just know it'll 477 00:22:44,039 --> 00:22:46,599 be a problem in a few years. And I'm like, okay, yes, 478 00:22:46,960 --> 00:22:47,599 but right now. 479 00:22:47,920 --> 00:22:49,279 Speaker 4: But when people get R and. 480 00:22:49,279 --> 00:22:52,440 Speaker 5: D budgets, it's just give me the fastest, most powerful 481 00:22:53,000 --> 00:22:55,960 system that I can put together. They're not really thinking about, Okay, 482 00:22:56,200 --> 00:22:57,480 is this actually cost effective? 483 00:22:57,759 --> 00:22:59,720 Speaker 1: I think it's a really interesting path to go down 484 00:22:59,720 --> 00:23:01,960 because I know, historically, at least in my career before 485 00:23:02,000 --> 00:23:07,440 I moved slightly outside of hands on software engineering, the 486 00:23:07,559 --> 00:23:10,200 joke amongst a lot of my leadership was there's no 487 00:23:10,279 --> 00:23:13,240 reason to optimize costs in any way because the amount 488 00:23:13,240 --> 00:23:15,960 that we're spending in the cloud is just completely dwarfed 489 00:23:15,960 --> 00:23:20,319 by whatever else thing we're paying for. Usually it's engineering 490 00:23:20,359 --> 00:23:24,799 salaries by like factors of magnitude. But as we get 491 00:23:24,839 --> 00:23:31,640 closer to having companies with smaller size and more hardware requirements, 492 00:23:31,720 --> 00:23:35,400 especially given the hardware requirements for running mL workloads or 493 00:23:35,559 --> 00:23:39,519 data jobs or specifically running models open source or otherwise, 494 00:23:40,720 --> 00:23:45,920 there is a huge increase in cost there per engineer. Right, 495 00:23:46,400 --> 00:23:48,240 And I don't think the mental I think the mentality 496 00:23:48,240 --> 00:23:50,519 has actually gotten worse. Right, Like it used to be 497 00:23:50,680 --> 00:23:54,480 much more conservative when like which tools we use, which 498 00:23:54,519 --> 00:23:56,559 cloud writers will use, you know, what will we write? 499 00:23:56,559 --> 00:23:58,160 And I feel like now it's gotten way more liberal, 500 00:23:58,240 --> 00:24:00,240 like no, go as fast as possible, pay as little 501 00:24:00,279 --> 00:24:03,200 attention to this. Are you still seeing companies be like okay, no, 502 00:24:03,279 --> 00:24:05,200 we actually need to think about this methodically, like we 503 00:24:05,240 --> 00:24:07,599 know it's going to cost a lot to run our 504 00:24:07,640 --> 00:24:10,240 models or is it really just a VC race? 505 00:24:11,039 --> 00:24:12,599 Speaker 4: It's definitely a race for sure. 506 00:24:12,759 --> 00:24:15,680 Speaker 5: Like every time I open up LinkedIn, I get these 507 00:24:15,720 --> 00:24:18,400 posts about like these companies that had two employees or 508 00:24:19,000 --> 00:24:21,279 ten employees and achieve this much, And I'm like, yeah, 509 00:24:21,279 --> 00:24:24,680 I wonder what they're spending on the public cloud providers. 510 00:24:25,279 --> 00:24:28,920 But we do get the question, so it's rarely are 511 00:24:28,920 --> 00:24:30,920 people like I actually want to do something about it. 512 00:24:31,240 --> 00:24:33,640 But we talked, you know, we kicked this off talking 513 00:24:33,680 --> 00:24:37,920 about finnops and DevOps and bringing that together. The finnops 514 00:24:37,960 --> 00:24:40,640 teams are the ones that are asking the question. When 515 00:24:40,640 --> 00:24:43,359 I was at fops X a month or two ago, 516 00:24:44,480 --> 00:24:47,440 I got the question about GPU optimization often of like 517 00:24:47,519 --> 00:24:50,799 how do I manage to spend here? And the thing 518 00:24:50,839 --> 00:24:53,839 I didn't want to say is like, honestly, your AIML 519 00:24:53,880 --> 00:24:55,519 team is just going to spend whatever they want, because 520 00:24:55,559 --> 00:25:00,319 is anyone giving them that guardrail of don't spend. But 521 00:25:00,640 --> 00:25:02,680 it is great that the phinops team is thinking about 522 00:25:02,680 --> 00:25:04,559 it because they know this is going to be a challenge. 523 00:25:05,279 --> 00:25:07,960 And it starts with visibility because as they are building 524 00:25:08,000 --> 00:25:10,720 the case of hey we should maybe maybe put some guardrails. 525 00:25:10,759 --> 00:25:12,079 Speaker 4: I'm not saying go go. 526 00:25:12,039 --> 00:25:13,839 Speaker 5: Tell the team they can't spend any money, but like 527 00:25:14,079 --> 00:25:17,559 some guardrails, it's okay. Getting the visibility is the first 528 00:25:17,559 --> 00:25:18,079 step to that. 529 00:25:19,920 --> 00:25:23,599 Speaker 1: Is it just ACYNC background jobs run by quote unquote 530 00:25:23,640 --> 00:25:25,240 data teams that are, you know, trying to make the 531 00:25:25,279 --> 00:25:28,160 business run or I think it's definitely. 532 00:25:27,920 --> 00:25:30,839 Speaker 5: A mix I'll even just like talk internally about cloud Bowl. 533 00:25:31,279 --> 00:25:34,440 So we talked a lot before about the machine learning 534 00:25:34,960 --> 00:25:36,920 that we do, but we also do have like we're 535 00:25:36,920 --> 00:25:39,720 working on a chatbot where you can ask questions about 536 00:25:39,720 --> 00:25:42,400 your infrastructure, and I get this question off and from 537 00:25:43,160 --> 00:25:45,039 people it's like I don't want to put them into 538 00:25:45,039 --> 00:25:46,759 the reports and have them figure it out. I just 539 00:25:46,799 --> 00:25:48,920 want them to act. Now everyone has learned how to 540 00:25:48,960 --> 00:25:51,640 interact with the chatbot that we get our main practitioner 541 00:25:51,680 --> 00:25:53,440 being like, I just want my end users to ask 542 00:25:53,480 --> 00:25:55,640 a question of what was the AWS bill for the 543 00:25:55,720 --> 00:25:58,000 last month and how to that map to what we 544 00:25:58,079 --> 00:26:01,680 had put in our budget. And so we are actively 545 00:26:02,079 --> 00:26:04,279 kind of developing based on a lot of. 546 00:26:04,279 --> 00:26:06,359 Speaker 4: The libraries that are out there and exists today. 547 00:26:06,799 --> 00:26:08,960 Speaker 5: And so it is a mix of Okay, I need 548 00:26:08,960 --> 00:26:11,759 the infrastructure to be able to deploy, and what I 549 00:26:11,839 --> 00:26:13,799 do my training on is different than when I do 550 00:26:13,880 --> 00:26:17,000 my inference on. So there is like i'd say, because 551 00:26:17,039 --> 00:26:20,160 we have some history in building the tooling, we are 552 00:26:20,359 --> 00:26:24,680 a little bit more cost conscious than other organizations would be. 553 00:26:24,759 --> 00:26:27,119 But we use that to then say, Okay, if I'm 554 00:26:27,160 --> 00:26:29,000 an end user, what is the information I would need 555 00:26:29,039 --> 00:26:29,160 to be. 556 00:26:29,160 --> 00:26:31,519 Speaker 1: Able to look at it sounds like it's not necessarily 557 00:26:31,599 --> 00:26:34,079 fin ops, which needs to happen now it's been mL 558 00:26:34,160 --> 00:26:37,720 OPS is the primary focus of cloudbal to be a 559 00:26:37,799 --> 00:26:41,559 much smarter version of the VPA and HPA. Or is 560 00:26:41,920 --> 00:26:43,440 that the set like you know, hyper focus in this 561 00:26:43,519 --> 00:26:46,519 area or is there like extended integrations. 562 00:26:46,680 --> 00:26:50,039 Speaker 5: Yeah, so cloud Bowl there's a portfolio of products, and 563 00:26:50,119 --> 00:26:54,319 the Kuberneti's focus comes from the Stormforge acquisition. That's like 564 00:26:54,440 --> 00:26:56,640 kind of my background. That's probably why I talked about 565 00:26:56,680 --> 00:26:59,960 a little bit more, and that is Kubernetes right sizing, 566 00:27:00,119 --> 00:27:02,920 so the ability to actually look at your traffic patterns, 567 00:27:03,000 --> 00:27:05,599 use machine learning to come up with what your right 568 00:27:05,640 --> 00:27:09,400 configuration settings are. And we're currently building that into the 569 00:27:09,400 --> 00:27:12,359 cloud Bolt platform, which is a is a whole sleoth 570 00:27:12,440 --> 00:27:15,960 of things. And the cloud Bolt platform pulls in billing 571 00:27:16,039 --> 00:27:19,759 data from AWS, Azure GCP and it does all that 572 00:27:19,839 --> 00:27:23,319 with the focus back. So the Finnops Foundation came up 573 00:27:23,319 --> 00:27:25,519 with the focus back to basically be like, hey, everyone 574 00:27:25,519 --> 00:27:26,720 needs to talk the same language. 575 00:27:26,759 --> 00:27:27,319 Speaker 4: Like it is. 576 00:27:28,000 --> 00:27:30,759 Speaker 5: It's hard enough to give people the right insights. Now 577 00:27:30,759 --> 00:27:33,079 they have to know what do I what is storage 578 00:27:33,079 --> 00:27:35,640 and AWS compared to Azure and when you're talking to 579 00:27:35,720 --> 00:27:38,279 Finopps folks, not all of them know all the technical terms. 580 00:27:38,279 --> 00:27:41,680 So how can we abstract that we rebuilt our data 581 00:27:41,680 --> 00:27:44,440 platform to be focused native, so we can automatically pull 582 00:27:44,480 --> 00:27:49,400 that information and give you reporting and visibility across all 583 00:27:49,480 --> 00:27:53,559 your public clouds or anything private like VMware, Openstaff, new Tannics, 584 00:27:53,599 --> 00:27:58,680 all that with that one language, one abstraction layer, and 585 00:27:58,720 --> 00:28:01,039 then you can interact with chat thought that we're still 586 00:28:01,039 --> 00:28:02,839 working on. So I'd say it's in beta, it's not 587 00:28:02,880 --> 00:28:05,480 like publicly available yet, but we are using it internally. 588 00:28:05,599 --> 00:28:09,240 Speaker 4: Is pretty cool, and interact with your data there. 589 00:28:09,319 --> 00:28:12,160 Speaker 5: You can do your cost allocation to say okay, these 590 00:28:12,160 --> 00:28:13,960 different departments are often like, we have a lot of 591 00:28:14,079 --> 00:28:17,640 MSP customers and so it's like the cloud Ponzi scheme. 592 00:28:17,920 --> 00:28:20,200 Someone's buying cloud from someone then selling it to someone 593 00:28:20,240 --> 00:28:22,920 else and they're selling it and every layer you need 594 00:28:22,960 --> 00:28:26,039 to understand what are the but the discounts that get applied, 595 00:28:26,079 --> 00:28:27,599 what are the margins? I want to add to this, 596 00:28:27,640 --> 00:28:30,400 how do I make sure that third tier customer is 597 00:28:30,440 --> 00:28:32,119 only seeing what they should be paying for, not what 598 00:28:32,240 --> 00:28:35,160 I'm actually paying for? Because I could get a little hairy. 599 00:28:35,400 --> 00:28:38,039 Speaker 3: For me ask a question like technically, how does this 600 00:28:38,079 --> 00:28:41,440 actually work? Like you said, we're looking at like traffic patterns, 601 00:28:41,440 --> 00:28:44,119 Like how is it actually intercepting and reading those traffic patterns. 602 00:28:44,480 --> 00:28:47,640 Speaker 5: Yeah, there's an agent that sits in the cluster. It 603 00:28:47,680 --> 00:28:50,480 pulls the metrics from like se advisor, pull like cube 604 00:28:50,480 --> 00:28:54,799 state metric type metrics. It's a couple pods that sits 605 00:28:54,839 --> 00:28:57,200 in the clusters, not a damisut or anything, and then 606 00:28:57,319 --> 00:29:00,799 pulls that information out and then ports it into our 607 00:29:01,359 --> 00:29:04,000 our database. And that's one source. So we also have 608 00:29:04,039 --> 00:29:06,839 an agent that pulls metrics from like VMware for example, 609 00:29:06,880 --> 00:29:11,079 looks at that utilization data and then with the focus 610 00:29:11,119 --> 00:29:16,079 spec we can pull overlay information from like your billing reports, 611 00:29:16,079 --> 00:29:18,599 and then it put that on top of like all 612 00:29:18,599 --> 00:29:19,920 the utilization metrics. 613 00:29:20,160 --> 00:29:20,440 Speaker 2: Okay. 614 00:29:20,480 --> 00:29:22,799 Speaker 3: I always get nervous when we talk about like adding agents, 615 00:29:22,960 --> 00:29:25,519 so the agents are not sitting like within like a 616 00:29:25,599 --> 00:29:27,839 current deployment or something like that, because then it's like, Okay, 617 00:29:27,839 --> 00:29:29,680 now I have like more research in quests than I 618 00:29:29,680 --> 00:29:32,880 need to account for on top of to solve this problem. 619 00:29:33,200 --> 00:29:34,079 Speaker 4: Yeah, yeah, totally. 620 00:29:34,119 --> 00:29:38,200 Speaker 5: That's the people are very sensitive to agents. The word 621 00:29:38,319 --> 00:29:41,400 I mean soot not. 622 00:29:41,480 --> 00:29:44,400 Speaker 3: Agent agents, but agents living inside your clusters. 623 00:29:44,480 --> 00:29:44,599 Speaker 2: Yea. 624 00:29:45,400 --> 00:29:49,000 Speaker 5: And what the storm Force software actually does is we 625 00:29:49,119 --> 00:29:51,440 use our own software to right size the agent so 626 00:29:51,480 --> 00:29:53,240 that we can make sure that the agent is just 627 00:29:53,359 --> 00:29:56,079 using the resources required for that cluster. And there are 628 00:29:56,079 --> 00:29:58,799 two lightweight pods at least in the kubernet space. When 629 00:29:58,799 --> 00:30:00,599 people have agent based tools, a lot of times of 630 00:30:00,680 --> 00:30:03,599 damon sets and so they take up a lot of space. 631 00:30:03,319 --> 00:30:04,319 Speaker 4: On every node. 632 00:30:04,960 --> 00:30:08,359 Speaker 5: And we've taken a deliberate approach to have a very 633 00:30:08,440 --> 00:30:11,200 lightweight agent that is just two pods that script and 634 00:30:11,240 --> 00:30:12,240 metrics and send them up. 635 00:30:12,359 --> 00:30:15,079 Speaker 1: Is it a security concern or like a capacity utilization 636 00:30:15,279 --> 00:30:17,319 issue of our resources in the cluster. 637 00:30:17,519 --> 00:30:19,559 Speaker 2: I've actually seen it as a capacity issue. 638 00:30:20,200 --> 00:30:24,240 Speaker 3: So we've I've seen people like deploy agents for various 639 00:30:24,240 --> 00:30:27,720 things and suddenly of capacity issues with because they're you know, 640 00:30:28,039 --> 00:30:30,319 everybody's fighting for the same resources. But if these are 641 00:30:30,359 --> 00:30:32,839 like you're saying, sexic deployments, I feel like that's less 642 00:30:32,839 --> 00:30:34,079 of a less of an issue. 643 00:30:34,079 --> 00:30:36,640 Speaker 1: At least you have some like previous trauma there. 644 00:30:38,440 --> 00:30:42,240 Speaker 3: Trauma it seems like, well, is. 645 00:30:42,160 --> 00:30:44,160 Speaker 1: There something that we can shame specifically? 646 00:30:48,359 --> 00:30:52,039 Speaker 3: And I actually probatuct. 647 00:30:51,240 --> 00:30:53,759 Speaker 1: Well, let's just start like this is there is there 648 00:30:53,839 --> 00:30:56,279 a frequent concern with pulling an open source you know, 649 00:30:56,359 --> 00:31:00,160 definitions into your into your cluster without really evaluating how 650 00:31:00,240 --> 00:31:03,240 much how resources they're going to consume compared to something 651 00:31:03,279 --> 00:31:07,880 that building yourself, and is that the source of most 652 00:31:07,880 --> 00:31:10,279 of the problems or you know, because these things aren't 653 00:31:10,279 --> 00:31:13,960 necessarily built for like I can I always complain about 654 00:31:13,960 --> 00:31:17,960 the security of open source not libraries, but full services 655 00:31:17,960 --> 00:31:21,359 that you deploy within your cloud environment or a cluster, 656 00:31:21,480 --> 00:31:25,039 because they're not designed for scale and reliability. They're designed 657 00:31:25,039 --> 00:31:29,240 to usually funnel you into their paid product or licensing, 658 00:31:30,480 --> 00:31:33,240 so you know, I it's like always a concern from 659 00:31:33,240 --> 00:31:35,920 that standpoint. So I can imagine an equivalent issue where 660 00:31:35,920 --> 00:31:39,720 they're not really designed to have low impact from a 661 00:31:39,759 --> 00:31:40,640 cost standpoint either. 662 00:31:40,920 --> 00:31:43,039 Speaker 3: Yeah, I mean, I'll time in I guess a little bit, 663 00:31:43,079 --> 00:31:44,720 but I'm also curious that you have in the thoughts. 664 00:31:44,759 --> 00:31:47,000 But at least just what I've seen is is more 665 00:31:47,039 --> 00:31:50,079 of it's less of a security issue but more of 666 00:31:50,079 --> 00:31:51,119 a resource issue. 667 00:31:51,200 --> 00:31:54,200 Speaker 5: I mean on our end, because so a previous version 668 00:31:54,240 --> 00:31:56,799 of our same product did actually use a lot of 669 00:31:56,799 --> 00:31:59,880 resources inside the cluster and customers would be like I'm 670 00:32:00,119 --> 00:32:02,680 using your tool to reduce my waist, but I also 671 00:32:03,160 --> 00:32:06,839 am wasting resources to use your tool. And so when 672 00:32:06,880 --> 00:32:09,000 we kind of went through a re architecture, that was 673 00:32:09,079 --> 00:32:12,079 a very important piece for us also because I mean 674 00:32:12,119 --> 00:32:15,119 it speaks to our credibility. We're supposed to be reducing resources, 675 00:32:15,200 --> 00:32:19,519 then our own bits should also be using limited resources. 676 00:32:19,720 --> 00:32:22,640 So that is why we take a very lightweight agent approach. 677 00:32:22,680 --> 00:32:25,599 And it's actually not an open source agent. We've gone 678 00:32:25,640 --> 00:32:27,440 back and forth of like should we release the code. 679 00:32:27,480 --> 00:32:29,920 It's not like it's anything secret, it's just it's always 680 00:32:30,000 --> 00:32:34,119 been a private agent. But you know, people can see 681 00:32:34,160 --> 00:32:37,960 the helm chart, they can see what resources will deploy 682 00:32:38,079 --> 00:32:41,680 with like default requests, so that obviously everything's capped. But 683 00:32:41,799 --> 00:32:45,359 then people will use the software to then write size 684 00:32:45,400 --> 00:32:47,359 it to make sure it is literally just using the 685 00:32:47,440 --> 00:32:50,519 minimal required resources and its size to the cluster too, 686 00:32:50,559 --> 00:32:53,160 so you use a small cluster, you're not spending the 687 00:32:53,200 --> 00:32:55,359 same amount of resources then if you had really like 688 00:32:55,440 --> 00:32:59,599 a large cluster smart I have sometsd from agents because 689 00:32:59,640 --> 00:33:01,799 I a lot of time at Puppet and there's a 690 00:33:01,799 --> 00:33:04,480 lot of agent wars at the time, to the point 691 00:33:04,480 --> 00:33:05,920 where I like hated the word, but. 692 00:33:06,319 --> 00:33:08,680 Speaker 1: You know, of well, you know, I guess I could 693 00:33:08,720 --> 00:33:12,440 say maybe I'm glad that you have some you know, 694 00:33:12,559 --> 00:33:15,160 PTSD from working at Puppet, because I definitely have some 695 00:33:15,279 --> 00:33:17,720 from using Puppet. But that was a long time ago, 696 00:33:18,440 --> 00:33:20,839 and it's been quite some time since I've had virtual 697 00:33:20,880 --> 00:33:24,480 machines even to run stuff on. Now I feel like most, 698 00:33:24,839 --> 00:33:28,319 well I say it most. I've read some statistic like 699 00:33:28,400 --> 00:33:32,359 still fifty percent of the world uses some PHP backed 700 00:33:33,200 --> 00:33:36,039 builders for websites, So I mean, there's still a huge 701 00:33:36,079 --> 00:33:38,240 amount of you know, people that aren't even using any 702 00:33:38,279 --> 00:33:41,799 sort of infrastructure as code solutions. And I'm over here saying, 703 00:33:41,839 --> 00:33:44,720 you know, isn't everyone pretty much using some icy and 704 00:33:44,880 --> 00:33:48,759 deploying their their stuff directly and spinning up containers. Amy's 705 00:33:48,799 --> 00:33:52,480 already shaking her head. Either mean she has a really 706 00:33:52,480 --> 00:33:54,960 great story to share or a really great story she 707 00:33:55,039 --> 00:33:55,880 doesn't want to share. 708 00:33:56,200 --> 00:33:57,440 Speaker 2: I just I. 709 00:33:59,039 --> 00:34:00,839 Speaker 3: Don't know. It's a lot of people will try to 710 00:34:00,880 --> 00:34:03,079 do the right things, and then other people are just 711 00:34:03,119 --> 00:34:09,320 gonna try to do the quickest, what seemingly work solution, 712 00:34:09,639 --> 00:34:12,840 which is just click it and go. But I see it. 713 00:34:12,880 --> 00:34:16,679 Speaker 2: I see it so often itrks me I mean. 714 00:34:16,719 --> 00:34:19,719 Speaker 1: There is a huge aspect here where it's about aligning incentives. 715 00:34:19,840 --> 00:34:23,920 So I think most people, uh most people in general 716 00:34:24,000 --> 00:34:27,280 have a logical desire to do something. They see some 717 00:34:28,079 --> 00:34:30,679 initial conditions, and then they follow that their path as 718 00:34:30,760 --> 00:34:32,679 best as they can to go up with the action 719 00:34:32,760 --> 00:34:35,280 they want to take. And part of that input is 720 00:34:35,320 --> 00:34:38,760 whatever incentives they have to go forward. And sometimes those incentives, 721 00:34:38,760 --> 00:34:41,199 you know, aren't aligned from one team member to another one, 722 00:34:41,480 --> 00:34:43,599 or from one team to another one, Like you give 723 00:34:43,800 --> 00:34:46,199 certain incentives to your develop your engineers, and then you 724 00:34:46,239 --> 00:34:49,559 give different incentives to not engineers or release folks or 725 00:34:49,599 --> 00:34:54,360 engineering folks, product management or uh we talked about having 726 00:34:56,119 --> 00:35:00,199 financial operations folks you know, have responsibility of actually reducing 727 00:35:00,280 --> 00:35:03,000 the costs, but separate from not actually making sure that 728 00:35:03,039 --> 00:35:06,519 the products that you're building are a have low footprint, right, 729 00:35:06,800 --> 00:35:10,119 and so you immediately have a different of incentive there. So, 730 00:35:10,440 --> 00:35:11,920 you know, I think there is a lot that goes 731 00:35:11,960 --> 00:35:15,199 into into that sort of poor decision making process where 732 00:35:15,239 --> 00:35:18,280 you get some optimal outcomes when your organization as a 733 00:35:18,280 --> 00:35:21,440 whole doesn't have this included. And I don't remember the 734 00:35:21,480 --> 00:35:23,320 last organization I was in or the last company I 735 00:35:23,360 --> 00:35:24,679 was talking too. I was like, yeah, you know, and 736 00:35:24,760 --> 00:35:27,679 our objectives and key results are okay, ours, we actually 737 00:35:27,719 --> 00:35:31,920 have like reduced the total cost associated with our infrastructure. 738 00:35:31,960 --> 00:35:35,320 Although I feel like everyone's gonna be like, yeah, of course, 739 00:35:35,320 --> 00:35:37,199 of course you need that. Of course you you have 740 00:35:37,239 --> 00:35:40,280 to counter your new development. But I don't think a 741 00:35:40,280 --> 00:35:41,559 lot of companies are doing that, are they. 742 00:35:41,639 --> 00:35:41,679 Speaker 3: No. 743 00:35:41,840 --> 00:35:44,119 Speaker 5: It's funny when I ask people, I'm like, oh, like, 744 00:35:44,159 --> 00:35:47,760 how important is like reducing costs, Like it's a top priority. 745 00:35:47,800 --> 00:35:49,599 I'm like, oh, okay, can you rank like your top 746 00:35:49,639 --> 00:35:51,960 priorities And it doesn't even make the top five and 747 00:35:52,320 --> 00:35:54,920 you just said reducing costs is important, Like, well, yes, 748 00:35:54,960 --> 00:35:56,880 we have the security thing we need to do with 749 00:35:56,960 --> 00:35:59,239 this new application we're trying to get out, and it's 750 00:35:59,280 --> 00:36:02,679 always compete. That's why I think like any solution is 751 00:36:02,719 --> 00:36:04,679 going to need to take minimal time and just work 752 00:36:04,719 --> 00:36:07,199 inside a system so you can be like, Okay, this 753 00:36:07,239 --> 00:36:09,239 is low hanging fruit. I can go tackle it. Even 754 00:36:09,280 --> 00:36:11,559 if it's six on my list, it's still in the 755 00:36:11,599 --> 00:36:16,000 top ten. But it's like, it is really interesting every 756 00:36:16,000 --> 00:36:17,920 time I talk to an IT leader and they're like, yeah, 757 00:36:17,960 --> 00:36:19,760 this is so important for us to costs and then 758 00:36:19,760 --> 00:36:22,079 I'm like, okay, rank your priorities. 759 00:36:22,159 --> 00:36:23,280 Speaker 4: Where does it actually sit. 760 00:36:23,519 --> 00:36:26,480 Speaker 3: You are mature enough engineer, you kind of treat this 761 00:36:26,599 --> 00:36:28,960 as like just your bread and butter is like part 762 00:36:29,000 --> 00:36:32,440 of your job, so like, yes, it helps to have 763 00:36:32,480 --> 00:36:35,840 the incentive there. In my opinion, I feel like if 764 00:36:35,880 --> 00:36:38,039 you're going into that role, like this is not something 765 00:36:38,119 --> 00:36:39,920 like management should tell you like, oh, we need to 766 00:36:39,960 --> 00:36:41,880 reduce costs, Like this is just like a given of 767 00:36:41,960 --> 00:36:45,480 like as part of the reliability metric that people measure, 768 00:36:45,599 --> 00:36:49,159 Like the cost metric could be something that you just 769 00:36:49,280 --> 00:36:51,880 go to your management team on a quarterly basis and 770 00:36:51,880 --> 00:36:52,840 be like, here's the cost. 771 00:36:52,679 --> 00:36:54,599 Speaker 2: We reduced or if here's what we did. I like 772 00:36:54,639 --> 00:36:56,800 that you shouldn't have to be told to do it. 773 00:36:56,800 --> 00:36:59,519 Speaker 1: If you're an experienced engineer or even a senior engineer, 774 00:36:59,639 --> 00:37:02,079 you may have your own internal metrics for say when 775 00:37:02,079 --> 00:37:04,519 to pull out a public package or you know open 776 00:37:04,559 --> 00:37:06,719 source solution, like oh yeah, I go to GitHub and 777 00:37:06,760 --> 00:37:09,239 I look and I see there's fifteen thousand stars. That 778 00:37:09,320 --> 00:37:11,880 means yeah, sure, I'll use it, no problem. You know 779 00:37:11,920 --> 00:37:14,199 there are ratings there that that are and I feel like, 780 00:37:14,360 --> 00:37:17,480 you know, as you ascend the sort of competency ladder, 781 00:37:17,480 --> 00:37:20,320 and you get to a staff staff plus principal engineer, 782 00:37:20,440 --> 00:37:23,119 you're not like you need your own sort of internal metrics. 783 00:37:23,159 --> 00:37:25,639 And I feel like you're increasing the scope of your 784 00:37:25,639 --> 00:37:27,639 delivery at those higher levels and there has to be 785 00:37:27,679 --> 00:37:28,960 a trade off, like how do you make make sure 786 00:37:28,960 --> 00:37:30,719 you're doing the right thing? And I totally agree with 787 00:37:30,719 --> 00:37:32,719 you Amy that part of it is like, well, how 788 00:37:32,800 --> 00:37:35,480 much am I spending to make sure that this technology 789 00:37:35,480 --> 00:37:38,440 that we released to have a business impact is costing 790 00:37:38,559 --> 00:37:41,159 us the right amount to actually have a positive ROI 791 00:37:41,199 --> 00:37:42,599 for us and not and not be negative. 792 00:37:43,039 --> 00:37:45,679 Speaker 3: I would say like, just like if you enjoy your 793 00:37:45,800 --> 00:37:49,039 job and you're looking at your runway, like this should 794 00:37:49,039 --> 00:37:51,880 be something that is consistently top of mind. And I 795 00:37:51,880 --> 00:37:55,320 mean through no fault of like more you know, newer engineers, 796 00:37:55,360 --> 00:37:57,000 this is just not something they think about. It's not 797 00:37:57,039 --> 00:37:59,559 something I definitely didn't think about my first two years. 798 00:37:59,599 --> 00:38:00,840 I was just it's like, oh my gosh, I need 799 00:38:00,880 --> 00:38:01,719 to get this speech. You're out. 800 00:38:01,920 --> 00:38:03,719 Speaker 1: I think there is definitely a cliff function here, and 801 00:38:03,960 --> 00:38:05,960 maybe either of you can correct me, but I feel 802 00:38:05,960 --> 00:38:08,480 like there's like there's a switch where early on in 803 00:38:08,480 --> 00:38:10,880 people's careers. They assume that the money that a company 804 00:38:10,920 --> 00:38:14,119 is spending is their own money, like, oh, this price 805 00:38:14,159 --> 00:38:15,760 tag is like you know, we get customers all the 806 00:38:15,800 --> 00:38:17,280 time like, oh, this is like going to be ten 807 00:38:17,320 --> 00:38:19,760 thousand a month. That's too expensive. I'm like, how, like, 808 00:38:19,800 --> 00:38:21,719 how much are you paying your engineering team to maintain 809 00:38:21,760 --> 00:38:23,960 that solution that's not even working? And how much are 810 00:38:24,000 --> 00:38:26,679 you paying your cloud provider to run that solution for you? 811 00:38:26,760 --> 00:38:28,960 I assure you it's like at least ten times how 812 00:38:29,039 --> 00:38:32,000 much you'll be paying us. And then there's a switch later. 813 00:38:32,079 --> 00:38:34,880 I feel like where they you know, they're on one 814 00:38:34,920 --> 00:38:36,960 of these two sides of extremes, right, like the money 815 00:38:36,960 --> 00:38:39,880 doesn't matter or the money matters too much, And I think, 816 00:38:39,960 --> 00:38:42,480 just like everything, I think the number one answer staff 817 00:38:42,519 --> 00:38:45,480 plus engineers love to give is you know it depends, right, 818 00:38:46,079 --> 00:38:48,920 you know, how much would we spend on our stack? Well? 819 00:38:49,599 --> 00:38:51,599 You know, well what are we running? You know, how 820 00:38:51,639 --> 00:38:54,639 important is it? What's the reliability necessary? I do want 821 00:38:54,639 --> 00:38:58,199 to ask you a question, yesman, though, is that Let's 822 00:38:58,239 --> 00:39:01,400 say I am in one of the roles, and while 823 00:39:01,480 --> 00:39:04,440 we all collectively agree that you should have the responsibility 824 00:39:04,480 --> 00:39:07,800 and accountability for either running some sort of solution that 825 00:39:07,840 --> 00:39:11,400 handles auto scaling, pulling in the appropriate sasas, or opens 826 00:39:11,440 --> 00:39:13,639 our solutions to help you. What if I'm not giving 827 00:39:13,840 --> 00:39:17,159 given that flexibility, Like what can you know on the 828 00:39:17,159 --> 00:39:20,280 ground engineers do to help make a case to say 829 00:39:20,280 --> 00:39:24,159 their leadership or their teams to have a more mature 830 00:39:24,239 --> 00:39:26,280 perspective on how to approach the problem. 831 00:39:26,679 --> 00:39:28,800 Speaker 5: The first question I usually ask people is like, what 832 00:39:28,800 --> 00:39:32,079 what do you think your cluster utilization is? And they'll guess, 833 00:39:32,360 --> 00:39:34,800 and usually they always guess higher than it actually is. 834 00:39:34,840 --> 00:39:37,440 Then they'll go look into whatever monitoring system they have 835 00:39:37,440 --> 00:39:39,039 and they're like, oh wow, it's like twenty percent. And 836 00:39:39,039 --> 00:39:41,599 I'm like, okay, what if you just doubled that? Right? Like, 837 00:39:41,880 --> 00:39:43,840 I'm not saying go to eighty percent, I'm just going 838 00:39:43,880 --> 00:39:46,599 to go from twenty to forty percent and setting that 839 00:39:46,639 --> 00:39:49,440 goal to be something that's achievable. Typically, I found that 840 00:39:49,639 --> 00:39:52,280 is something people can do on their own, without tools 841 00:39:52,559 --> 00:39:54,960 and just get an understanding of how bad of a 842 00:39:55,000 --> 00:39:57,599 problem do I have and how much waste is there? 843 00:39:57,639 --> 00:39:59,960 Because when when you go to people and you're like, hey, 844 00:40:00,280 --> 00:40:02,719 we could remove waste, then it's like the language you're 845 00:40:02,760 --> 00:40:06,119 removing something there's there's a risk in that. But when 846 00:40:06,320 --> 00:40:08,559 you say, what if you could improve your cluster utilization 847 00:40:08,639 --> 00:40:10,840 from just double it from twenty to forty percent, you 848 00:40:10,920 --> 00:40:13,880 still sixty percent headroom, Like there's still a lot of 849 00:40:13,880 --> 00:40:17,800 waste there, but you've doubled your impact. It's a pretty 850 00:40:17,840 --> 00:40:19,360 good metric that you can feel good about. 851 00:40:19,519 --> 00:40:21,159 Speaker 1: I do see this fhere of like I don't want 852 00:40:21,199 --> 00:40:23,199 to touch something I don't understand because I don't want 853 00:40:23,199 --> 00:40:25,199 to break it, Like maybe it's twenty percent right now. 854 00:40:25,239 --> 00:40:28,320 But I think the other thing that we hear frequently 855 00:40:28,360 --> 00:40:31,159 on the show is like, how as an experienced engineer 856 00:40:31,239 --> 00:40:33,639 do you get to some sort of depth? I get 857 00:40:33,679 --> 00:40:36,039 asked a lot when I'm giving toxic conferences like how 858 00:40:36,039 --> 00:40:38,559 did I become a security expert? I'm like, I have 859 00:40:38,679 --> 00:40:40,360 no idea. I guess I just spend a lot of 860 00:40:40,400 --> 00:40:42,239 time doing things. And I think this is one of 861 00:40:42,239 --> 00:40:45,679 those areas like if you see the capacity is really 862 00:40:45,760 --> 00:40:48,280 like utilization is really low, Like maybe ask the question like, 863 00:40:48,320 --> 00:40:50,079 well why is it so low? Like what should it be? 864 00:40:50,159 --> 00:40:51,719 Like you don't have to change anything, but maybe you 865 00:40:51,719 --> 00:40:53,639 should be able to answer, like, well, if I wanted 866 00:40:53,639 --> 00:40:55,639 to change it, what steps should I take how should 867 00:40:55,639 --> 00:40:56,400 I know what it should be? 868 00:40:56,480 --> 00:40:56,559 Speaker 2: Like? 869 00:40:56,599 --> 00:41:00,079 Speaker 1: What information should I pull out? I think it's really good. 870 00:41:00,480 --> 00:41:02,360 Speaker 5: I went back when I was an engineer. I learned 871 00:41:02,400 --> 00:41:05,280 everything by breaking things. Made sure I broke them in dev. 872 00:41:05,360 --> 00:41:07,800 But like, you don't learn until you try. And so 873 00:41:07,920 --> 00:41:10,800 as long as you have a safe space to try, 874 00:41:10,880 --> 00:41:12,800 and you have a cluster in dev that doesn't matter 875 00:41:12,840 --> 00:41:15,480 and it has really well utilization, you can practice different 876 00:41:15,519 --> 00:41:18,000 ways to improve that on a cluster that doesn't matter 877 00:41:18,079 --> 00:41:21,320 until you kind of hone your skills and go deploy 878 00:41:21,360 --> 00:41:22,599 it to other environments. 879 00:41:22,800 --> 00:41:25,239 Speaker 1: You can definitely say no comment here if you're not 880 00:41:25,280 --> 00:41:28,519 comfortable answering this question. But like, surely people were concerned 881 00:41:28,559 --> 00:41:31,079 about the cost of the cloud before moving there, or 882 00:41:31,119 --> 00:41:34,559 their cost of their you know, container management solution or 883 00:41:34,639 --> 00:41:36,679 you know lead of virtual machines that they were using 884 00:41:37,320 --> 00:41:40,679 vSphere on VMware, Like weren't there like solutions, Like weren't 885 00:41:40,679 --> 00:41:42,559 people already talking about and doing this before? 886 00:41:43,239 --> 00:41:45,800 Speaker 5: Yes, but there was an expectation that moving to cloud 887 00:41:45,800 --> 00:41:48,320 would be cheaper, and then there's a reality that like 888 00:41:48,519 --> 00:41:50,639 you have to put in work to actually make sure 889 00:41:50,719 --> 00:41:52,880 the cloud is cheaper, and most people didn't put in 890 00:41:52,880 --> 00:41:55,159 that work, so now they're paying for it. 891 00:41:55,239 --> 00:41:57,480 Speaker 1: Do you think that's like a huge driver for the 892 00:41:57,679 --> 00:41:59,719 let's move off of cloud because it's too expensive And 893 00:41:59,800 --> 00:42:02,320 you're like the voice of reason here, It's like, well, 894 00:42:02,360 --> 00:42:04,519 it doesn't have to be too expensive. It can be 895 00:42:04,760 --> 00:42:07,800 the right amount of expensive or even cheaper if you 896 00:42:08,400 --> 00:42:11,559 just think about what workloads you're running and what they 897 00:42:11,559 --> 00:42:14,400 are capacitily utilization, and how to scale effectively. 898 00:42:14,599 --> 00:42:17,079 Speaker 5: Yeah. I think like everyone wants this ideal world where 899 00:42:17,079 --> 00:42:19,559 they don't think about day two problems, but doesn't matter 900 00:42:19,559 --> 00:42:22,000 where you're deployed, you have to think about day two problems. 901 00:42:22,199 --> 00:42:24,599 And the reality is for some people, probably moving back 902 00:42:24,639 --> 00:42:27,199 to the data center is the right thing, and that's okay, 903 00:42:27,360 --> 00:42:29,280 Like there is this oh, you have to be on 904 00:42:29,280 --> 00:42:32,239 the cloud to be cool, which like I think people 905 00:42:32,320 --> 00:42:34,440 are starting to get that you should be on the 906 00:42:34,440 --> 00:42:38,079 cloud for the right reasons and if you are there, 907 00:42:38,239 --> 00:42:40,559 you need to be thinking about your day two challenges, 908 00:42:40,800 --> 00:42:44,159 like making sure your configurations are set correctly, like you're 909 00:42:44,199 --> 00:42:47,679 reducing wasts, you're doing the right sizing, you have policies 910 00:42:47,719 --> 00:42:51,800 to go clean up unused backups, just you know, basic things. 911 00:42:51,920 --> 00:42:53,679 Speaker 1: I just want to totally disagree with you. I mean, 912 00:42:53,719 --> 00:42:57,480 I think it's a very mature approach to say, you know, 913 00:42:57,519 --> 00:42:58,960 there are reasons that you may want to go to 914 00:42:59,000 --> 00:43:01,639 an on prem data center, but like my, my justifications 915 00:43:01,639 --> 00:43:04,119 for doing that are just drying up one after another one. 916 00:43:04,159 --> 00:43:08,119 Like I think about just the the depreciating assets for 917 00:43:08,320 --> 00:43:10,480 on prem, Like it used to be the case for 918 00:43:10,519 --> 00:43:12,519 sure that you could buy some you could buy a 919 00:43:12,559 --> 00:43:14,440 data center or even run a data center and throw 920 00:43:14,480 --> 00:43:17,559 your own server blades in and those would be usable 921 00:43:17,679 --> 00:43:21,159 for a decade or even twenty years. And now I 922 00:43:21,199 --> 00:43:25,039 feel like, especially if you're doing something cutting edge technology 923 00:43:25,039 --> 00:43:26,599 and your run and you want to do some data 924 00:43:26,639 --> 00:43:30,159 train you know, evaluation, or build some models or run 925 00:43:30,199 --> 00:43:33,519 the models and do inference, the technology to maintain a 926 00:43:33,519 --> 00:43:37,440 competitive advantage actually requires frequent hardware upgrades. I don't know. 927 00:43:37,679 --> 00:43:39,840 I actually would love to sell someone to tell me, like, no, 928 00:43:40,000 --> 00:43:42,320 actually here is this area where it makes sense to 929 00:43:42,639 --> 00:43:45,880 actually be on prem. But the more simple, the more straightforward, 930 00:43:45,880 --> 00:43:48,360 the more common what you're doing is like everyone else, 931 00:43:48,440 --> 00:43:50,559 the less of a reason there is, like it should 932 00:43:50,599 --> 00:43:53,039 it really should be cheaper being in the cloud for me, I. 933 00:43:53,039 --> 00:43:57,800 Speaker 5: Think for most startups that are also like deploying to 934 00:43:58,000 --> 00:44:00,679 cloud native applications makes sense, you still got to think 935 00:44:00,679 --> 00:44:02,920 about day two otherwise you're just going to spend like crazy. 936 00:44:03,239 --> 00:44:06,079 But like you still have retail companies that have been 937 00:44:06,119 --> 00:44:08,840 around for a while and have like oms systems that 938 00:44:09,159 --> 00:44:12,239 are not built to run on a public cloud. So 939 00:44:12,519 --> 00:44:15,559 it's you're forcing your shoehorning something that's just going to 940 00:44:15,639 --> 00:44:19,519 cost you in the long run. And because I mentioned 941 00:44:19,920 --> 00:44:23,519 our history of like being an infrastructure automation platform for 942 00:44:23,519 --> 00:44:26,880 the last fifteen years, we see a flavor of every 943 00:44:27,000 --> 00:44:29,960 type of infrastructure you can imagine, like things that are 944 00:44:29,960 --> 00:44:32,599 still running in a data center somewhere on tapes, like 945 00:44:32,800 --> 00:44:36,119 it still exists, and not everything makes sense to move 946 00:44:36,239 --> 00:44:37,920 because if you're just going to lift and shift it, 947 00:44:37,920 --> 00:44:39,159 then you will be paying more. 948 00:44:39,360 --> 00:44:41,960 Speaker 1: It's interesting because it was never for me that moving 949 00:44:41,960 --> 00:44:45,079 to the cloud was actually a cost reduction. I mean, 950 00:44:45,159 --> 00:44:47,119 I believe it, it's a reduced cost you build the 951 00:44:47,199 --> 00:44:51,280 right thing directly, but like my personal motivations were always 952 00:44:51,320 --> 00:44:55,400 like high reliability, getting cutting edge features and support the 953 00:44:55,400 --> 00:44:59,039 infrastructure platform whatever you're utilizing, and being able to you know, 954 00:44:59,159 --> 00:45:01,719 do quick have quick development cycles, you know, be more 955 00:45:01,760 --> 00:45:04,199 agile and I feel like in the agile world with 956 00:45:04,280 --> 00:45:06,880 just in time technology, there's a higher there is a 957 00:45:06,880 --> 00:45:09,199 cost associated with that, so like I do see it 958 00:45:09,239 --> 00:45:13,960 as paying for the advantage. I also like the mentality 959 00:45:14,000 --> 00:45:15,360 of like, well, it actually doesn't have to be more 960 00:45:15,400 --> 00:45:17,840 expensive if you, you know, utilize what the cloud provider 961 00:45:17,960 --> 00:45:21,079 is offering. It's interesting to see that, you know, you're 962 00:45:21,079 --> 00:45:24,480 still engage with companies that are are running off prem 963 00:45:24,519 --> 00:45:26,880 and it just will, like forever, always be cheaper. 964 00:45:27,000 --> 00:45:29,840 Speaker 5: That fifty percent metric honestly doesn't surprise me. It probably 965 00:45:29,880 --> 00:45:33,079 did before we got acquired because I've been living in 966 00:45:33,079 --> 00:45:35,800 the Kubernetes world for so long. But the more and 967 00:45:35,800 --> 00:45:38,920 more I talk to different customers and prospects that there's 968 00:45:39,039 --> 00:45:42,039 still a mix of there's a healthy mix of on 969 00:45:42,119 --> 00:45:47,119 prem and even honestly within Kubernetes, people running Kubernetes on prem, I. 970 00:45:47,079 --> 00:45:49,000 Speaker 1: Used to trust them. I mean, so I'll say where 971 00:45:49,000 --> 00:45:50,480 this is from. Like I think it was like every 972 00:45:50,559 --> 00:45:52,800 year WordPress would come out with a number of like 973 00:45:52,840 --> 00:45:54,719 the percent of the Internet that was running on WordPress, 974 00:45:54,800 --> 00:45:59,519 and like, I never liked PHP, so I wasn't particularly 975 00:45:59,559 --> 00:46:02,360 inclined to trust the numbers coming out. But what after 976 00:46:02,519 --> 00:46:05,840 every you know, local recent scandal. I even trust those 977 00:46:05,880 --> 00:46:08,800 numbers less. But whatever that number is, it's still higher 978 00:46:08,840 --> 00:46:11,079 than I would be comfortable with, you know, as a 979 00:46:11,360 --> 00:46:14,800 human society, that we're you know, overutilizing this. So there 980 00:46:14,800 --> 00:46:18,400 are tons of uh, you know, non tech zero one 981 00:46:18,519 --> 00:46:22,039 or two person companies that before the web flows and 982 00:46:22,159 --> 00:46:25,079 levables of the world bubble eyos, there there was like 983 00:46:25,159 --> 00:46:27,039 you needed to get your website up and running and 984 00:46:27,079 --> 00:46:30,920 GeoCities and whatever Yahoo had, you know, just Google sites 985 00:46:30,960 --> 00:46:33,880 you know, was not cutting it. So you did do 986 00:46:33,960 --> 00:46:35,880 this and with a number of themes and plugins like 987 00:46:35,880 --> 00:46:38,159 I do get it. But that means like they can't 988 00:46:38,239 --> 00:46:40,239 utilize any of this, and the costs for those is 989 00:46:40,320 --> 00:46:42,800 of course really high, but they have no value in 990 00:46:43,280 --> 00:46:47,440 transitioning to you know whatever. Think of like your local hairdresser, 991 00:46:47,519 --> 00:46:49,519 like they're not going online. I mean at this point though, 992 00:46:49,519 --> 00:46:51,440 I'm sure they're using some SaaS product to manage all 993 00:46:51,480 --> 00:46:56,239 their scheduling and LMS to actually do the hair dressing itself. 994 00:46:56,320 --> 00:46:56,920 Speaker 4: That'd be great. 995 00:46:57,000 --> 00:46:59,519 Speaker 5: Ask a chatbot like how should I cut my hair? 996 00:46:59,599 --> 00:47:01,480 I guess you could never thought of it. 997 00:47:01,880 --> 00:47:04,239 Speaker 1: Well, I just assume you're not too far out from 998 00:47:04,320 --> 00:47:06,519 asking the chatbot, like handing it to the scissors and 999 00:47:06,559 --> 00:47:08,719 telling it to do what it thinks is best. 1000 00:47:09,039 --> 00:47:12,360 Speaker 3: I actually saw a video on the other day of 1001 00:47:12,360 --> 00:47:14,519 one of Musk's robots doing haircuts. 1002 00:47:14,880 --> 00:47:19,000 Speaker 1: Yeah, we're not going to go into that topic otherwise 1003 00:47:19,000 --> 00:47:20,239 I think I'm gonna actually have to cut it out 1004 00:47:20,239 --> 00:47:22,639 of the podcast episode. 1005 00:47:23,119 --> 00:47:25,480 Speaker 3: So I'm just stating the facts. YEA. 1006 00:47:27,159 --> 00:47:29,239 Speaker 1: Well, you know, I think for some people the facts 1007 00:47:29,239 --> 00:47:30,239 are even up for debate. 1008 00:47:30,840 --> 00:47:32,880 Speaker 2: So, yes, it was a video. 1009 00:47:32,960 --> 00:47:34,840 Speaker 3: I'm like, did an Ai just make this video or it 1010 00:47:34,840 --> 00:47:36,079 is just like a realistic video. 1011 00:47:36,079 --> 00:47:39,079 Speaker 5: I just I don't even can never tell honestly, every 1012 00:47:39,159 --> 00:47:41,079 video I watch, I'm like, is it real? 1013 00:47:41,320 --> 00:47:41,719 Speaker 4: I don't know. 1014 00:47:42,000 --> 00:47:44,119 Speaker 1: There was there was something I'll say that was really interesting, 1015 00:47:44,199 --> 00:47:48,400 totally unrelated. Uh, there's a British comedy show called QI 1016 00:47:48,519 --> 00:47:53,159 Quite interesting and every season they would say facts and 1017 00:47:53,920 --> 00:47:56,320 they by like the thirteenth season, what they had done 1018 00:47:56,360 --> 00:47:58,639 is they actually calculated the lifetime of the facts, so 1019 00:47:58,719 --> 00:48:02,400 what they said earlier on became less correct the later 1020 00:48:02,519 --> 00:48:05,320 that season would go so like it was like the 1021 00:48:05,360 --> 00:48:08,079 previous season, like n minus one eighty six percent of 1022 00:48:08,079 --> 00:48:10,599 the facts were still correct, but like the first season, 1023 00:48:10,639 --> 00:48:12,760 by the thirteenth one, there was like something like sixty 1024 00:48:12,760 --> 00:48:14,800 percent of the facts were like just no longer accurate, 1025 00:48:15,039 --> 00:48:18,000 like science had like whatever we had decided different now, 1026 00:48:18,000 --> 00:48:19,719 And I think the canonical one is like what the 1027 00:48:19,719 --> 00:48:22,519 food pyramid? Like, what is the food pyramid supposed to be? 1028 00:48:22,920 --> 00:48:25,599 Every year there's a different change to it, and it's 1029 00:48:25,719 --> 00:48:28,840 it's never never accurate. I think I think before we 1030 00:48:28,880 --> 00:48:31,599 go too much down another tangent, maybe maybe it's a 1031 00:48:31,599 --> 00:48:35,719 good time to switch over to picks. Maybe mostly nods here. So, Amy, 1032 00:48:35,760 --> 00:48:36,840 what did you bring for us today? 1033 00:48:37,480 --> 00:48:40,199 Speaker 3: I debated on this because I don't want a lot 1034 00:48:40,280 --> 00:48:41,639 of people to go out and buy it and then 1035 00:48:41,679 --> 00:48:45,199 they have like a they're no longer able to fulfill 1036 00:48:45,239 --> 00:48:47,159 the oilers. But it's really good, so I feel like 1037 00:48:47,159 --> 00:48:50,760 I should share. I'm a big health person, hence why 1038 00:48:50,920 --> 00:48:54,440 I'm walking on the treadmill, and my pick is going 1039 00:48:54,480 --> 00:48:57,880 to be like a specific protein powder. So when I 1040 00:48:58,000 --> 00:49:01,000 was like new on my protein powder journey, I didn't 1041 00:49:01,079 --> 00:49:03,519 quite care. I was like, what does it taste like? 1042 00:49:03,719 --> 00:49:05,400 Speaker 2: And what's the price? 1043 00:49:06,039 --> 00:49:07,960 Speaker 3: So this stuff is a little of pricey, but it's 1044 00:49:08,000 --> 00:49:11,440 like super clean ingredients and it's the brand's called the clip. 1045 00:49:12,360 --> 00:49:13,119 Speaker 2: The other good thing. 1046 00:49:13,679 --> 00:49:16,880 Speaker 3: They have a protein I feel really embarrassed, like maybe 1047 00:49:16,920 --> 00:49:18,719 other people are not into this, but in case someone is, 1048 00:49:18,800 --> 00:49:21,639 they also have a protein bar now, which like, if 1049 00:49:21,639 --> 00:49:23,719 you travel for work frequently, I'm always like, if I 1050 00:49:23,719 --> 00:49:25,960 have to go to conference or something like, I need 1051 00:49:26,000 --> 00:49:30,280 something that is going to not like make me feel 1052 00:49:30,320 --> 00:49:33,360 like trash when I'm there, because sometimes the food doesn't 1053 00:49:33,400 --> 00:49:37,519 always I don't know, it's not always the best. So anyways, 1054 00:49:37,519 --> 00:49:40,199 they have a protein bar that just came out. I 1055 00:49:40,239 --> 00:49:42,440 had to order like a month in advance. That's how 1056 00:49:42,559 --> 00:49:44,039 special this prostin bar is to me. 1057 00:49:44,159 --> 00:49:45,960 Speaker 2: And I just got it. And it's still like this and. 1058 00:49:50,119 --> 00:49:53,199 Speaker 1: You know you did you did Peloton last last episode, 1059 00:49:53,280 --> 00:49:55,360 so you know, we clearly on a healthcake here. 1060 00:49:55,400 --> 00:49:57,239 Speaker 2: I guess I just want to think of something else 1061 00:49:57,320 --> 00:49:57,760 next week. 1062 00:49:57,840 --> 00:50:00,039 Speaker 1: You don't have to. You can definitely bring high. I 1063 00:50:00,079 --> 00:50:03,199 don't live a healthy lifestyle picks every single week. I think, 1064 00:50:03,880 --> 00:50:05,840 you know, especially for those of us who sit at 1065 00:50:05,840 --> 00:50:08,679 our computer, like literally sit at our computers all day long, 1066 00:50:09,239 --> 00:50:11,840 I think there's a something good to be said for you. 1067 00:50:11,960 --> 00:50:14,840 Speaker 3: Actually, here's something practical about it. It's not just healthy, 1068 00:50:14,840 --> 00:50:16,880 but a lot of people who are like trying to transition. 1069 00:50:16,920 --> 00:50:18,559 I hear like a lot of my coworkers are like, 1070 00:50:18,840 --> 00:50:20,440 I want to get healthier, and they switch to a 1071 00:50:20,440 --> 00:50:22,679 protein powder. A lot of protein powder is way based 1072 00:50:22,719 --> 00:50:25,039 and that can upset people's stomach. So this one is 1073 00:50:25,079 --> 00:50:28,280 actually made from beef protein, which you might think, Like, initially, 1074 00:50:28,320 --> 00:50:30,840 I was like, this sounds disgusting, but it actually tastes 1075 00:50:30,840 --> 00:50:31,239 really good. 1076 00:50:31,519 --> 00:50:35,519 Speaker 1: That's a that's a pretty good cell there. Okay, Osmin, 1077 00:50:35,519 --> 00:50:36,360 what'd you bring for us? 1078 00:50:36,480 --> 00:50:39,639 Speaker 5: So initially I think I had more of a fun 1079 00:50:39,639 --> 00:50:41,559 fact that I learned this week than a pick. But 1080 00:50:41,679 --> 00:50:46,280 since we're on a health pick topic, I've been having 1081 00:50:46,519 --> 00:50:50,440 bone broth recently. A friend got me into it, and 1082 00:50:51,119 --> 00:50:53,760 I've been drinking like beef bone broth, chicken bone broth. 1083 00:50:54,800 --> 00:50:58,400 There's one on Amazon that I buy by like nineteen 1084 00:50:58,480 --> 00:51:01,880 ninety snacks. It's like a orange container. I'm sure they're 1085 00:51:01,880 --> 00:51:05,719 all good, but uh, it's like actually really good instead 1086 00:51:05,760 --> 00:51:08,679 of a tea or just lunch snack. If I'm like, oh, 1087 00:51:08,679 --> 00:51:10,400 I don't have time, I'll just heat it up. And 1088 00:51:11,440 --> 00:51:13,360 I think maybe it was a thing that everybody else 1089 00:51:13,440 --> 00:51:15,119 knew about that I didn't know about. But yeah, I've 1090 00:51:15,119 --> 00:51:16,519 been really getting into that lately. 1091 00:51:17,280 --> 00:51:18,599 Speaker 2: They're also good for travel. 1092 00:51:18,920 --> 00:51:21,960 Speaker 1: I've never heard of it, and I'm I don't think 1093 00:51:22,000 --> 00:51:23,119 I'll ever have it, so. 1094 00:51:24,320 --> 00:51:25,079 Speaker 3: You should try it. 1095 00:51:25,159 --> 00:51:25,840 Speaker 4: You should just try it. 1096 00:51:25,920 --> 00:51:28,519 Speaker 3: Yeah, really, Okay, here's an idea to use it too, 1097 00:51:28,519 --> 00:51:29,840 Like if you don't want to try to drink it, 1098 00:51:29,920 --> 00:51:31,880 Like if you ever have some sort of like pasta, 1099 00:51:31,920 --> 00:51:33,079 you can cook your pa in it. 1100 00:51:33,480 --> 00:51:37,400 Speaker 1: So I absolutely go to the store or actually a 1101 00:51:37,400 --> 00:51:39,480 lot of butchers will throw away bones. You can go 1102 00:51:39,519 --> 00:51:41,679 in just ask them for the bones. And you don't 1103 00:51:41,719 --> 00:51:44,639 need to buy beef bullion or chicken bullion at the 1104 00:51:44,679 --> 00:51:48,880 store or vegetable stock. Honestly, just boil the bones and 1105 00:51:48,920 --> 00:51:51,320 some vegetables. I usually use caracter celery in a bay 1106 00:51:51,360 --> 00:51:53,960 leaf in a pot for like an hour or two, 1107 00:51:54,400 --> 00:51:56,920 and then use the liquid for whatever else you're making. 1108 00:51:57,039 --> 00:51:58,599 And it's just so much better. 1109 00:51:58,880 --> 00:51:59,960 Speaker 2: I like your idea better. 1110 00:52:00,320 --> 00:52:05,360 Speaker 5: Just try drinking it. Just try it, even if it's. 1111 00:52:04,840 --> 00:52:07,880 Speaker 1: I don't know, Like I find for me that the 1112 00:52:07,920 --> 00:52:09,679 best broths are like usually when the meat is a 1113 00:52:09,719 --> 00:52:11,840 little bit fatty. So if you use like chicken thighs 1114 00:52:11,960 --> 00:52:16,400 on set or wings like instead of breasts and uh, 1115 00:52:16,480 --> 00:52:19,079 pork or beef bones, Like I don't have that much meat, 1116 00:52:19,079 --> 00:52:22,119 but like there's a lot of flavor here. I still 1117 00:52:22,119 --> 00:52:23,920 think it's like too fatty for me, Like I don't 1118 00:52:24,000 --> 00:52:27,480 like the sensation of having like a fatty drink like 1119 00:52:27,480 --> 00:52:29,519 that that's the idea is warming up to me. But 1120 00:52:29,599 --> 00:52:31,480 it's it's a lot, it's a lot of extra effort 1121 00:52:31,519 --> 00:52:32,960 to go out and do this. But I suppose if 1122 00:52:32,960 --> 00:52:36,239 you know you, if you live close or you you 1123 00:52:36,280 --> 00:52:40,719 buy whole animals to to cut up and consume, then 1124 00:52:40,960 --> 00:52:42,840 you have probably a lot of extra bones. And if 1125 00:52:42,840 --> 00:52:45,239 you're not utilizing them to also make drinks, I guess 1126 00:52:45,239 --> 00:52:49,000 here's your here's your opportunity. Okay, So I actually wasn't 1127 00:52:49,000 --> 00:52:51,239 sure what I wanted my pick today to be, so 1128 00:52:51,440 --> 00:52:55,039 I went and I grabbed this article called lions and 1129 00:52:55,079 --> 00:53:00,679 dolphins cannot make babies, And it sounds pretty obvious on 1130 00:53:00,719 --> 00:53:04,960 the surface. I mean, obviously they're in different families in 1131 00:53:05,000 --> 00:53:11,159 the taxonomy for animals. But the article is about basically, 1132 00:53:11,880 --> 00:53:16,079 how we got to this point in technology required a 1133 00:53:16,119 --> 00:53:21,480 synonymous relationship, symbiotic between whatever the hardware we're utilizing and 1134 00:53:21,519 --> 00:53:23,199 the software that we're building. And I think this is 1135 00:53:23,239 --> 00:53:26,280 true in a lot of things. It's known as an evolution, 1136 00:53:26,360 --> 00:53:28,719 as red Queen's race. So you think of like bacteria 1137 00:53:28,760 --> 00:53:31,920 and viruses competing for resources, or even any sort of 1138 00:53:31,920 --> 00:53:36,119 parasitic organism and the host evolving. And it goes into 1139 00:53:36,119 --> 00:53:40,199 a little bit about how we probably will not have 1140 00:53:40,280 --> 00:53:43,559 any more innovation in AI unless we really take a 1141 00:53:43,639 --> 00:53:47,199 huge giant leap in some of the hardware technology available 1142 00:53:47,239 --> 00:53:51,320 to us. We can't just build better software and fix 1143 00:53:51,360 --> 00:53:52,960 some of the problems. I mean right now, I think 1144 00:53:53,119 --> 00:53:55,440 one of the limitations we have is everything that's built 1145 00:53:55,480 --> 00:53:57,159 in the last six years was based off of the 1146 00:53:58,000 --> 00:54:02,280 paper written by Google on transformer texture, and like, sure 1147 00:54:02,320 --> 00:54:05,079 there are some shortcomings, like it can't figure out patterns 1148 00:54:05,159 --> 00:54:08,480 or do math, but we'll extract some characters from you 1149 00:54:08,480 --> 00:54:10,360 can ask it like where is the math, and then 1150 00:54:10,480 --> 00:54:13,039 like rejects that and then pass it to any sort 1151 00:54:13,039 --> 00:54:14,960 of solver engine and get that. So you know, you 1152 00:54:14,960 --> 00:54:17,159 start to solve some of those problems. But fundamentally, to 1153 00:54:17,199 --> 00:54:19,280 get to the next level, we'll need something different, and 1154 00:54:19,320 --> 00:54:23,159 it's not just writing new software. I think the idea 1155 00:54:23,199 --> 00:54:26,360 here is fundamentally that we're stuck until we actually understand 1156 00:54:26,360 --> 00:54:28,760 that we may need a different kind of hardware to 1157 00:54:29,280 --> 00:54:32,320 power this. I don't know if quantum computing is sufficient 1158 00:54:32,360 --> 00:54:35,079 to running this, but really just something that's slightly different 1159 00:54:35,079 --> 00:54:36,840 from our current computational engines. 1160 00:54:36,840 --> 00:54:37,679 Speaker 4: I'd love to read that. 1161 00:54:37,639 --> 00:54:39,920 Speaker 1: It's not very long, so I think it maybe overseelling this, 1162 00:54:40,039 --> 00:54:42,920 but it'll be in the link to the in the 1163 00:54:42,920 --> 00:54:45,480 podcast episode. In the show notes, thank you so much 1164 00:54:45,519 --> 00:54:48,199 for our kind guests to show up today and tell 1165 00:54:48,280 --> 00:54:49,760 us all about it off and off, So thank you 1166 00:54:49,800 --> 00:54:53,400 for coming. It's been fantastic, And thank you to all 1167 00:54:53,440 --> 00:54:56,679 of our listeners for another great episode of adventures and 1168 00:54:56,679 --> 00:55:06,320 DevOps and I hope you're all back for the next one. 1169 00:55:00,360 --> 00:55:01,719 Speaker 3: The st