WEBVTT 1 00:00:14.279 --> 00:00:18.960 Hey, what's going on everybody? This is another episode of Adventures and DevOps 2 00:00:19.000 --> 00:00:24.199 and I'm your host for today. Will Button joining me today, I have 3 00:00:24.399 --> 00:00:32.320 Corey O'Daniel, CEO and co founder of mass Driver, but also from Let's 4 00:00:32.320 --> 00:00:36.520 See. You're a principal software architect at the Real Real Cloud Solution, architects 5 00:00:36.520 --> 00:00:41.960 at Container Heroes prior to that, staff engineer at Click Trips, and most 6 00:00:42.000 --> 00:00:47.439 importantly, taco afficionado. So two things jumped on my mind right away. 7 00:00:48.000 --> 00:00:51.439 Number one, I need to know what your definition of the perfect taco is. 8 00:00:51.479 --> 00:00:54.280 But also I want to talk about your background because it looks very similar 9 00:00:54.320 --> 00:00:58.520 to mine in terms of the amount of time you spend at each company, 10 00:00:58.560 --> 00:01:02.119 and I think that's an important thing to share with everyone, especially people who 11 00:01:02.159 --> 00:01:06.760 are just starting their careers. But we'll get to that meanwhile, Corey, 12 00:01:06.799 --> 00:01:08.879 thank you for joining me on the show today. Yeah, thanks for having 13 00:01:08.920 --> 00:01:15.439 me and opening with a taco question. Oh okay. I think my favorite 14 00:01:15.599 --> 00:01:19.319 taco, like my go to is just like a classic like Carne Sato Street 15 00:01:19.319 --> 00:01:23.959 taco like that is my happy place and I've found the place in La that's 16 00:01:25.000 --> 00:01:29.040 like one of the better ones I've had in my life. So that's that's 17 00:01:29.159 --> 00:01:32.439 very convenient. It's a place called Cacau, Mexico, Testa, and the 18 00:01:32.439 --> 00:01:34.840 thing is how you pronounced the second word in Eagle Rock. It's amazing, 19 00:01:34.920 --> 00:01:42.359 and their molat is delicious right on. I would I would be I would 20 00:01:42.400 --> 00:01:45.439 be really really sad if you told me that you were living in LA and 21 00:01:45.519 --> 00:01:49.640 couldn't find a good taco spot. I mean the thing that it's like I 22 00:01:49.680 --> 00:01:53.640 always feel I mean, of course La is to have a can taco spot, 23 00:01:53.719 --> 00:01:57.840 right, But like I've I lived in Central America, Mexico under okay, 24 00:01:59.200 --> 00:02:00.799 the place were all So I feel bad saying that. But I mean 25 00:02:00.879 --> 00:02:05.159 like there's like this like interesting thing that happens in La. There's a whole 26 00:02:05.200 --> 00:02:09.080 bunch of people here, like like California is like what thirty plus percent Hispanic, 27 00:02:09.199 --> 00:02:12.639 right, and so like there's a whole bunch of people here, and 28 00:02:12.680 --> 00:02:15.840 then all the rich people are bougie and expect the best, so like all 29 00:02:15.960 --> 00:02:19.919 food has to be fantastic, right, So like I feel like those two 30 00:02:19.919 --> 00:02:23.800 things combined for like just it's like like La is like the best food mall, 31 00:02:24.080 --> 00:02:30.639 like everything's delicious. So I say that and I always feel bad. 32 00:02:30.719 --> 00:02:32.280 So I'm like, I've had some really great tacos in Mexico City, but 33 00:02:32.400 --> 00:02:38.120 like one of the convenience of driving tenants definitely impacts it. Oh for sure, 34 00:02:38.400 --> 00:02:46.080 absolutely, yeah. Yeah. Cool. So let's talk about your your 35 00:02:46.199 --> 00:02:47.919 job history real quick, and I'm going to be real clear, I'm not 36 00:02:47.919 --> 00:02:51.800 trying to call you out on it, because, like I mentioned mine, 37 00:02:51.960 --> 00:02:55.960 it looks very similar where a lot of your your positions you were there for 38 00:02:57.360 --> 00:03:00.400 you know, a year and a half, two years, three years, 39 00:03:00.840 --> 00:03:05.960 and I, you know, like I'm older and like my parents were, 40 00:03:06.000 --> 00:03:08.120 like, you get a job and you work there for forty years and your 41 00:03:08.120 --> 00:03:13.360 golden and I've never held a job for more than three years, and I 42 00:03:13.400 --> 00:03:20.719 think that's just I think part of that is just the new norm because after 43 00:03:20.759 --> 00:03:24.439 three years, Like the way I approach it is that if after three years 44 00:03:24.479 --> 00:03:29.520 of working with you, I haven't accomplished what you hired me to do, 45 00:03:29.840 --> 00:03:32.919 then I have failed. What's your take on that. I've got a couple 46 00:03:32.919 --> 00:03:37.159 of takes on this, but a lot of hot takes, So apologies and 47 00:03:37.199 --> 00:03:42.719 advance to anybody, but uh, I think one we have a lot If 48 00:03:42.719 --> 00:03:46.199 you think about our job is software engineers. Can you just classified kind of 49 00:03:46.240 --> 00:03:50.199 all of us under the software umbrella, whether you don't know. So some 50 00:03:50.199 --> 00:03:53.680 people don't feel like software engineers because they're like DevOps engineers and themal engineers, 51 00:03:53.680 --> 00:03:55.680 but you are, like there's a different abstraction layer, different level, like 52 00:03:55.719 --> 00:03:59.599 different expertise. Like if you think about like us in the grand scheme of 53 00:03:59.599 --> 00:04:03.800 things, we're an interesting group of people where our job is being a really 54 00:04:03.840 --> 00:04:09.000 good tool for the most part, right, Like if you think about where 55 00:04:09.039 --> 00:04:12.560 software started, where it was people that were professionals and they use software to 56 00:04:12.599 --> 00:04:16.079 do their job. Right, slowly software development became its own profession, right, 57 00:04:16.160 --> 00:04:24.680 So I see what we do a lot of time is like like like 58 00:04:24.839 --> 00:04:28.680 what's the best way of phrasing it? Like we are brought in to help 59 00:04:28.720 --> 00:04:32.040 business people code five things, right, Like that's really what we're doing. 60 00:04:32.120 --> 00:04:36.040 There's people that have an idea of how to make money and they don't know 61 00:04:36.040 --> 00:04:40.040 how to do the work that we do, right, and so we're not 62 00:04:40.120 --> 00:04:45.920 always invested in the business idea necessarily that we're writing software for, right, 63 00:04:45.920 --> 00:04:48.959 And like you can see this is plenty of companies, Like I know one 64 00:04:48.959 --> 00:04:51.519 of the companies I work on very early it's not on my resumes because I 65 00:04:51.600 --> 00:04:57.480 was a consultant at the time was honest company for sending baby diapers to your 66 00:04:57.480 --> 00:05:01.160 house. There was a bunch of single men there that built the software. 67 00:05:01.319 --> 00:05:03.680 None of them were like, wow, I can't wait to get some diapers 68 00:05:03.720 --> 00:05:09.160 doing that, Like that's the first thing. Is like like you can love 69 00:05:09.240 --> 00:05:12.959 what you do but not necessarily love the thing that you're building, right, 70 00:05:13.040 --> 00:05:15.360 and that has interesting trade offs, like you might get to the point where 71 00:05:15.360 --> 00:05:20.759 it's like, okay, like all the ideation and collaboration with the business book, 72 00:05:20.759 --> 00:05:24.639 which is kind of like what I really like about software is like talking 73 00:05:24.639 --> 00:05:28.040 through the ideas and figuring out like how to make that code, not necessarily 74 00:05:28.079 --> 00:05:30.160 typing it. You to a point where it's like, Okay, we've done 75 00:05:30.160 --> 00:05:33.439 the ideation, We've done the interesting things. Like I feel like I'm really 76 00:05:33.439 --> 00:05:38.639 in a maintenance mode. I don't love the business, so it's like, 77 00:05:38.680 --> 00:05:44.439 okay, well what is here for me? Right? And a lot of 78 00:05:44.839 --> 00:05:46.519 a lot of careers we see people working ten, twelve, thirteen, fourteen 79 00:05:46.560 --> 00:05:49.839 years, they don't have that choice. I'm a lawyer. If I don't 80 00:05:49.839 --> 00:05:54.279 like being a lawyer anymore, I'm kind of shit out of lot. Right, 81 00:05:54.720 --> 00:05:57.839 and so I think that's one part of it. But we're also we're 82 00:05:57.879 --> 00:06:00.160 all hobbyists, right, Like we all it's oh, it's five o'clock, 83 00:06:00.160 --> 00:06:02.079 it's time to get off work. What am I going to do? I'm 84 00:06:02.079 --> 00:06:04.839 going to continue to sit at my computer and write some open source software. 85 00:06:04.920 --> 00:06:08.800 I'm going to go play with something, right, Like we love to play 86 00:06:08.839 --> 00:06:12.720 and learn new things. I think like those two things, especially for me, 87 00:06:12.839 --> 00:06:15.639 like I have a almost like a wonderlust, like I have to be 88 00:06:15.959 --> 00:06:18.920 doing something more, doing something interesting, engaging myself when I get to a 89 00:06:18.959 --> 00:06:21.279 place where I'm like, okay, like there's just not really anything here that 90 00:06:21.480 --> 00:06:26.600 rewards me. And guess what, the businesses aren't either. The best way 91 00:06:26.879 --> 00:06:30.879 If there's any juniors listening to this, like, sorry, guys, here's 92 00:06:30.959 --> 00:06:33.439 here's the news. The best way to get a raise is to quit your 93 00:06:33.519 --> 00:06:39.759 job and go someplace else. Absolutely a three percent raise, which means you're 94 00:06:39.759 --> 00:06:43.240 losing like eight percent of your salary over the last couple of years with inflation. 95 00:06:43.519 --> 00:06:46.560 The best way to jump twenty percent is get a new job. And 96 00:06:46.040 --> 00:06:49.839 at the end of the day, like I like writing software, but you 97 00:06:49.879 --> 00:06:54.800 know what, I really like getting paid well, right, and so it's 98 00:06:54.800 --> 00:06:57.000 like if I got to the point where it's like there's something that I can 99 00:06:57.360 --> 00:06:59.879 say I did, there's somebody at that company that I can use as a 100 00:07:00.000 --> 00:07:02.279 reference to, like Corey Rocked, and it's like, Okay, I've gotten 101 00:07:02.360 --> 00:07:06.319 like some like interesting raises and maybe I'm like in line with inflation, but 102 00:07:06.360 --> 00:07:09.959 I'm not I'm not making more money. It's like, what is the growth 103 00:07:09.959 --> 00:07:13.480 that I have here if I'm not invested in the business, if I'm not 104 00:07:13.560 --> 00:07:16.639 learning anything new about the software, and I'm not getting paid more money, 105 00:07:16.800 --> 00:07:20.759 Like I'm just degenerating in every way, right, Like, so it's time 106 00:07:20.800 --> 00:07:26.680 to go and what's really interesting? And well this doesn't offend anybody, But 107 00:07:26.720 --> 00:07:30.279 like when I see a resume and somebody's like I worked there for fifteen years, 108 00:07:30.319 --> 00:07:32.920 it's like I don't want to work with you, Like no things is 109 00:07:33.000 --> 00:07:36.519 like you you are so ingrained in like your Google you like I worked at 110 00:07:36.560 --> 00:07:39.879 Google for fifteen years. It's like, well, guess what, Like you 111 00:07:40.439 --> 00:07:45.759 didn't work in the real world, like you worked in like the perfect utopia 112 00:07:45.879 --> 00:07:49.920 of like building software around other software engineers for software engineers like that. Like 113 00:07:50.439 --> 00:07:54.360 so like seeing that person that's like they have enough tenure where you know that 114 00:07:54.360 --> 00:07:57.759 they weren't just getting fired from job to job. They had that experiences, 115 00:07:57.839 --> 00:08:01.639 different teams, different stacks, different products where they can bring a lot more 116 00:08:01.680 --> 00:08:05.839 than just being able to type. Like that's that's kind of the value I'm 117 00:08:05.879 --> 00:08:07.399 looking for. And I'm looking at a resume. I'm terrified if somebody just 118 00:08:07.439 --> 00:08:11.879 sat someplace forever, he just work on like the banner of Amazon for eight 119 00:08:11.959 --> 00:08:15.399 years, Like, what did you do? Right? So I know there's 120 00:08:15.399 --> 00:08:20.879 probably some really great time. Orry. I didn't mean yeah, I hear 121 00:08:20.920 --> 00:08:26.319 you though. That's me in a nutshell. Welcome high everyone right now. 122 00:08:26.480 --> 00:08:31.399 I had a boss that I worked for a long time ago, and I 123 00:08:31.399 --> 00:08:35.960 I interviewed for another job and he found out about it was a smaller town, 124 00:08:37.000 --> 00:08:39.759 and you know, I was terrified that he was just going to fire 125 00:08:39.799 --> 00:08:43.480 me on the spot. And he said, well, did you get the 126 00:08:43.600 --> 00:08:46.720 job? And I said, well, they're waiting, are you? Are 127 00:08:46.720 --> 00:08:50.919 you mad? He said no. He said, if if no one else 128 00:08:50.960 --> 00:08:56.240 wants to hire you, why would I want you working for me? Damn, 129 00:08:56.279 --> 00:09:00.440 that's pretty savage. The funny thing is, I like, I've seen 130 00:09:00.480 --> 00:09:05.000 this. The number of times is like a hiring manager, where like you'll 131 00:09:05.039 --> 00:09:07.559 make an offer to somebody, right they come in the interview, go al, 132 00:09:07.679 --> 00:09:09.200 I really like that person, Like I interviewed one hundred people or whatever, 133 00:09:09.279 --> 00:09:11.840 and that was the one. I'm like, that's my person, and 134 00:09:11.879 --> 00:09:16.600 they're like my company counter and I'm like, yeah, if it took you 135 00:09:16.799 --> 00:09:22.120 leaving for your company to recognize your value, then you accept that, like 136 00:09:22.879 --> 00:09:26.080 I don't, I don't know, Like I mean, you take it, 137 00:09:26.240 --> 00:09:31.440 definitely take it because it's more money, but you should immediately reshot that out 138 00:09:31.480 --> 00:09:35.240 and find something better, Like take the money while you're there, and then 139 00:09:35.399 --> 00:09:39.000 go find the person that appreciates you for who you are and wants to give 140 00:09:39.039 --> 00:09:41.320 you a good salary, not just because you're gonna walk right. Yeah. 141 00:09:41.360 --> 00:09:46.919 And the counterpoint to that is if you do stay with them, Like all 142 00:09:48.000 --> 00:09:52.360 companies go through down cycles, and whenever they hit that next downcycle, they've 143 00:09:52.399 --> 00:09:58.279 already set precedent that they're only going to do the bare minimum to keep you 144 00:09:58.399 --> 00:10:03.080 around, and so whenever they have to lay off people, they're likely to 145 00:10:03.120 --> 00:10:07.519 think, well, hey, this person has already been trying to leave anyway, 146 00:10:07.679 --> 00:10:09.840 so they need to be first on the top and block to go yeah, 147 00:10:09.960 --> 00:10:13.759 especially if like you've like if that salary is no longer convincerate with the 148 00:10:13.840 --> 00:10:18.919 value that you have. You just got a twenty percent raise because you were 149 00:10:18.919 --> 00:10:20.200 going to quit. Like now you're the biggest line item on the team, 150 00:10:20.279 --> 00:10:28.720 right, But now I do think of this analogy just hit me. I 151 00:10:28.759 --> 00:10:33.080 kind of feel like what we do are like the wagon train masters of the 152 00:10:33.159 --> 00:10:39.720 Old West, Like our job is to get the settlers and their wagons out 153 00:10:39.799 --> 00:10:43.480 west, and once we get them out west, we're done. Like you 154 00:10:43.559 --> 00:10:48.639 never hire a wagon master and say, well, we're going to use you 155 00:10:48.840 --> 00:10:54.360 to escort our wagon train for the next fifteen years. No, get us 156 00:10:54.360 --> 00:10:56.639 to this point and then go away. Yeah, it's kind of like it 157 00:10:56.720 --> 00:11:01.360 suffers, like we hump if you wanting eighteen eighty three, is that where 158 00:11:01.360 --> 00:11:05.159 this came from? Where you're just like a fore Gun trail fan. It's 159 00:11:05.240 --> 00:11:09.480 just a side quest finished eighteen eighty three is so good? Yeah, No, 160 00:11:09.600 --> 00:11:11.440 it's funny. It's like, yeah, nobody's ever like, yeah, 161 00:11:11.440 --> 00:11:15.320 we'll get out the Oregon. We'll keep the wagging guy around in case we 162 00:11:15.360 --> 00:11:18.240 want to go to like Montana or Alabama or something. Maybe we'll want to 163 00:11:18.279 --> 00:11:20.039 go back to hudos, right, and like that's what you're just like, 164 00:11:20.120 --> 00:11:22.720 Okay, well I'm here until like we have another direction we want to go, 165 00:11:22.840 --> 00:11:26.240 we to change all this stuff. Yeah, that's an interesting one them. 166 00:11:26.279 --> 00:11:28.840 I was talking to somebody the other day, a buddy of mine that's 167 00:11:28.879 --> 00:11:33.480 working on an AI code product, and I will call them out because I 168 00:11:33.519 --> 00:11:37.440 think the big disagreements on it. But like this whole notion of like we've 169 00:11:37.440 --> 00:11:39.720 had this like for like twenty five years for anybody who's newer to the industry. 170 00:11:39.759 --> 00:11:43.360 But like when Macromedia dream Weaver came out, I remember people being like, 171 00:11:43.720 --> 00:11:46.679 we'll never be like business people are gonna build all the web pages and 172 00:11:46.720 --> 00:11:52.120 it's like, ah no they aren't. But like as like as right, 173 00:11:52.159 --> 00:11:56.480 because at the end of the day, like we are tools for business, 174 00:11:56.600 --> 00:11:58.679 right, Like so this idea of like no code and AI it can help 175 00:11:58.679 --> 00:12:03.399 this is people like build software Like that is a great idea, and like 176 00:12:03.639 --> 00:12:07.440 I'm sure that that will happen eventually, But in the meantime, us being 177 00:12:07.480 --> 00:12:13.360 relegated to being code reviewers for a machine like that feels like the most miserable 178 00:12:13.360 --> 00:12:18.519 intersection of how this did go right. It's like the creativity part, and 179 00:12:18.559 --> 00:12:22.919 like thinking of how to like abstract like business logic like business needs and logic 180 00:12:22.960 --> 00:12:24.679 into code is part of the fun part. And it's like, that's just 181 00:12:24.720 --> 00:12:28.960 a computer and I'm just like, ah, I don't like the way you've 182 00:12:28.000 --> 00:12:33.279 spelled that variable, Like that's a that's a sucky life for software engineers. 183 00:12:33.399 --> 00:12:41.600 So but yeah, wagons AI, it's all the same. That could that 184 00:12:41.639 --> 00:12:50.000 could be our new startup wagon train dot a the bafest path to organ that 185 00:12:50.159 --> 00:12:56.759 ended up. But there is there is like one of the things you mentioned 186 00:12:56.840 --> 00:13:01.480 earlier, you know about the the alignment of you as an employee versus the 187 00:13:01.519 --> 00:13:07.639 company. Like the founders of the company are vested in the product. Me 188 00:13:07.840 --> 00:13:15.120 as an employee, I'm vested in providing a really good living for my family, 189 00:13:15.159 --> 00:13:16.679 putting food on a table, putting a roof over their head, giving 190 00:13:16.720 --> 00:13:20.240 them the things that make them feel like they're having a quality life. And 191 00:13:22.240 --> 00:13:28.519 so we have different goals there. But jumping around like we do also is 192 00:13:28.559 --> 00:13:33.440 a benefit to those companies because those people, like the diaper company, they're 193 00:13:33.600 --> 00:13:41.720 really passionate about how to get diapers to young families. But if I've worked 194 00:13:41.720 --> 00:13:48.519 for a bunch of different industries. I can bring perspective to them and help 195 00:13:48.559 --> 00:13:54.639 them build a product that meets that need better because I know the roadblocks that 196 00:13:54.679 --> 00:14:00.799 they aren't familiar with and can't see because they're focused on that the diapers on 197 00:14:00.799 --> 00:14:03.720 the doorstep thing. Yeah, and it's interesting because, like you know, 198 00:14:03.120 --> 00:14:07.440 in those like as a company grows, especially from startup to so you say 199 00:14:07.519 --> 00:14:11.440 series V and beyond, like the product also changes, right, So, 200 00:14:11.559 --> 00:14:16.960 like one of the things that might bring you into maybe new e commerce companies, 201 00:14:16.000 --> 00:14:18.919 the person had experienced a previously e commerce compani It's like, oh, 202 00:14:18.960 --> 00:14:24.279 we needed somebody who knows inventory systems and you know, billing and all that 203 00:14:24.320 --> 00:14:26.399 stuff. Like, let's bring that person in. But then as like this 204 00:14:26.480 --> 00:14:30.759 company evolved, let's say three years, we're doing like a lot of video 205 00:14:30.799 --> 00:14:33.240 options. It's like, why don't necessarily need like this person's deep knowledge and 206 00:14:33.840 --> 00:14:37.440 e commerce systems anymore? I know they need somebody to news how to do 207 00:14:37.919 --> 00:14:41.480 concurrent video streams to hundreds of thousands of people that might be bidding on this 208 00:14:41.600 --> 00:14:43.600 item, right, and so like our business has changed, the people that 209 00:14:43.639 --> 00:14:46.600 we need involved change, and sure like that person that's been there for three 210 00:14:46.639 --> 00:14:50.639 or four years could go learn how to do like video streaming and you know 211 00:14:50.679 --> 00:14:54.200 all that stuff. But it's like, also you could go Hoch somebody from 212 00:14:54.360 --> 00:14:58.080 you know, Prime or something like that. Right, So I think like 213 00:14:58.720 --> 00:15:03.639 that moving, I think is what makes us really good tools. Again, 214 00:15:03.720 --> 00:15:07.919 like if you sit in this narrow channel for fifteen years doing one thing for 215 00:15:07.000 --> 00:15:11.679 one product with one team, that may have people swapping in and out, 216 00:15:11.759 --> 00:15:15.240 like you were very uniform, right, and so like now when you're coming 217 00:15:15.279 --> 00:15:18.000 into another environment, I feel like it's harder to figure out like how to 218 00:15:18.799 --> 00:15:22.559 how to necessarily fit you in, right. I feel like it was kind 219 00:15:22.559 --> 00:15:24.759 of like limits where you I mean, I guess when you have the prestige 220 00:15:24.759 --> 00:15:26.120 of a big band company can kind of go wherever you want. But like 221 00:15:26.159 --> 00:15:30.559 if you were like, oh, that person was at Macy's for seventeen years, 222 00:15:30.600 --> 00:15:33.519 you know like, well we got to get them in here, right, 223 00:15:33.960 --> 00:15:35.840 But it's someways like that it was a Google for seventeen years, You're 224 00:15:35.840 --> 00:15:37.879 like, oh, how much do how much money we have to live in 225 00:15:37.879 --> 00:15:41.279 a bank. Let's give it to this person, give them the stock. 226 00:15:41.639 --> 00:15:52.039 Right. So when we were chatting over email before the podcast, you mentioned 227 00:15:52.080 --> 00:15:58.279 that DevOps is pure bullshit. So I don't want to care about this. 228 00:15:58.480 --> 00:16:02.679 Yeah, I'm gonna call you out on it. This is fine. I 229 00:16:02.720 --> 00:16:03.919 love talking about this, so, like I have to say that, this 230 00:16:04.000 --> 00:16:07.399 is one of those moments where, gosh, you can't remember who said the 231 00:16:07.480 --> 00:16:10.879 quote, like being able to hold two opposing ideas in your head at the 232 00:16:10.919 --> 00:16:15.600 same time. Who was that? Uh huh. I'm not really good at 233 00:16:15.679 --> 00:16:18.879 quotes, and this is live, so I can't google it. But ultimately 234 00:16:18.639 --> 00:16:22.039 it's like a definission of being crazy. So I was like mesuring Franklin or 235 00:16:22.080 --> 00:16:23.799 something. It's like, oh, that's a sign of intelligence. It's like 236 00:16:23.840 --> 00:16:29.440 also inside of being nuts. But it's really I think it was it. 237 00:16:30.320 --> 00:16:33.240 It was one of these much smarter and successful to me said it, so 238 00:16:33.440 --> 00:16:40.000 you got to but it it is and it isn't, And I think it's 239 00:16:40.000 --> 00:16:44.919 one of those things that's interesting interesting. It's like death ops is what it 240 00:16:44.960 --> 00:16:47.120 is based on, Like how much your zoom dinner is zoomed? Out? 241 00:16:47.200 --> 00:16:52.720 Right? You could watch somebody who's building an application. Let's say it's twenty 242 00:16:52.799 --> 00:16:57.639 thirteen and somebody's building a Ruby full stack app monolith and they deploy it on 243 00:16:57.759 --> 00:17:03.720 Heroku. They are they doing DevOps, I would argue, yeah, they 244 00:17:03.720 --> 00:17:06.240 are. They're they're operating their software happens to be running on a platform. 245 00:17:06.279 --> 00:17:10.200 It's abstracted a lot away from it, but they're operating at the abstraction that 246 00:17:10.200 --> 00:17:14.240 they've been given, right Versus Okay, I'm an engineer. I'm sitting around 247 00:17:14.279 --> 00:17:18.680 three hundred micro services. I'm configuring the Kubernators cluster and all the service meshes. 248 00:17:18.759 --> 00:17:22.200 Like am I doing DevOps? Like yeah, you are too, You're 249 00:17:22.200 --> 00:17:25.440 doing like the dev and the ops like sure, So like it can exist 250 00:17:25.440 --> 00:17:29.160 at different levels and you can create abstractions where people can still do DevOps. 251 00:17:29.160 --> 00:17:33.160 And I think that where we are today and how we like handwave a lot 252 00:17:33.160 --> 00:17:36.839 of the definition of DevOps. Like there's people that are in like it's a 253 00:17:36.880 --> 00:17:40.160 team camp. There's people like it's a collaboration camp. And there's the people 254 00:17:40.160 --> 00:17:42.000 that are like you do everything, like you build it, you run it, 255 00:17:42.039 --> 00:17:45.279 like you have to do it all. Ah, It's like okay, 256 00:17:45.319 --> 00:17:49.319 well, like we kind of like three definitions so like that right, there's 257 00:17:49.319 --> 00:17:53.200 a definite bullshit Like it gave defines, right, but like I have three 258 00:17:53.279 --> 00:18:00.519 of those like don't happen for most people, And I have proof. We 259 00:18:00.640 --> 00:18:04.240 love the Dora report and DevOps, but when you go through it, fifty 260 00:18:04.279 --> 00:18:08.640 percent of people that respond to the Door Reporter like everything's fucked here and it's 261 00:18:08.640 --> 00:18:12.200 just like numbers, like it takes us we really software once a quarter and 262 00:18:12.240 --> 00:18:15.640 it's like okay, and it's like an outage lasts up to two days, 263 00:18:15.640 --> 00:18:21.200 Like fifty percent of the Door report respondents report pure chaos. So like while 264 00:18:21.400 --> 00:18:23.559 walk like we're on Hacker News, We're like, haha, at my company, 265 00:18:23.559 --> 00:18:26.519 we do DevOps great. And it's like, okay, you're a soul 266 00:18:26.640 --> 00:18:30.559 CTO who deploys on verse cell, Like what you're doing it right, like 267 00:18:30.839 --> 00:18:33.160 your hands in fine, And then people will be like, we do devlops 268 00:18:33.160 --> 00:18:37.839 great. We have eighty thousand micro services like you also have ten thousand ops 269 00:18:37.839 --> 00:18:41.920 people who've built an internal platform that makes it where you can do a DevOps 270 00:18:41.920 --> 00:18:44.960 with the abstraction it's given to you, right, And so I feel like 271 00:18:45.839 --> 00:18:49.279 I feel like like it is plausible. It is an idea that makes sense, 272 00:18:49.319 --> 00:18:53.480 and you do want your developers to have a baseline understanding of the abstraction 273 00:18:53.599 --> 00:18:57.519 that they're given. But I think the bullshit is in like like like are 274 00:18:57.519 --> 00:19:00.359 we giving people the right abstraction, And I would argue, like, that's 275 00:19:00.359 --> 00:19:06.319 where it really gets super complicated, And the answer is also like we probably 276 00:19:06.400 --> 00:19:11.920 aren't right. So what's interesting is looking at another report. State of CD 277 00:19:12.000 --> 00:19:15.440 report is another interesting one. The most recent state of CD report found that 278 00:19:15.519 --> 00:19:19.519 only twenty seven percent of companies are using I see infrastructure's code, right, 279 00:19:21.000 --> 00:19:26.039 and so we're all DevOps engineers. We automate everything, do we? If 280 00:19:26.079 --> 00:19:29.440 only twenty seven percent of companies are using infrastructure as code and I ventured, 281 00:19:29.440 --> 00:19:32.160 I bet most of them that marked yes and that aren't using it one hundred 282 00:19:32.160 --> 00:19:36.680 percent throughout the company. How are those other seventy three percent of companies that 283 00:19:36.720 --> 00:19:41.880 are doing DevOps automating stuff? If you weren't using antibal, chef care form 284 00:19:42.319 --> 00:19:48.400 or blooming, are you using Selenium to automate clicking through the AWS dashboard? 285 00:19:48.480 --> 00:19:52.119 Like I'm confused here because we're all doing DevOps, but you're not automating any 286 00:19:52.160 --> 00:19:56.079 of your infrastructure management. That is where a whole other chunk of bullshit comes 287 00:19:56.119 --> 00:20:00.480 in infrastructure management. Like some people consider it a part of DE but it 288 00:20:00.559 --> 00:20:03.960 didn't exist really, I mean it was like CF engine, but like there 289 00:20:04.039 --> 00:20:11.359 wasn't much infrastructure management or even cloud infrastructure when like the DevOps term was pointed, 290 00:20:11.519 --> 00:20:15.359 right, And so this whole idea of cloud infrastructure, which is this 291 00:20:15.440 --> 00:20:18.759 thing that ran our software that over the past ten years has slowly become our 292 00:20:18.799 --> 00:20:22.880 software, and it's going to get worse with AI and LLLM models, Like 293 00:20:23.119 --> 00:20:26.799 a lot of our software is no longer ours. Right in two thousand and 294 00:20:26.839 --> 00:20:30.960 eight, my code was my code. In twenty twelve, it was like, 295 00:20:30.000 --> 00:20:34.039 well, all my cues and background processes are actually done by this sqssn 296 00:20:34.119 --> 00:20:37.920 S thing, Like I've got a lot less code to manage I call people 297 00:20:37.920 --> 00:20:40.880 other people's APIs and now it's like step function in land. That's like my 298 00:20:40.920 --> 00:20:44.880 code's getting smaller and smaller and this cloud service area is growing. And then 299 00:20:44.920 --> 00:20:48.400 you look at llms and it's like I don't have any code. I just 300 00:20:48.440 --> 00:20:52.079 shove hashes in somebody else's database and magic comes out of it. Right, 301 00:20:52.119 --> 00:20:56.960 Like we're getting a lot less code, a lot more consumed cloud services. 302 00:20:56.279 --> 00:21:00.319 And I don't think we ever tradition like went back and like include infrastructure management 303 00:21:00.400 --> 00:21:04.920 like in that loom, right, And then when you do, whose responsibility 304 00:21:04.960 --> 00:21:10.359 is it? Because if you say, all of aws's the responsibility of all 305 00:21:10.400 --> 00:21:14.240 the services that you may or may not use in AWS is your engineer's responsibility. 306 00:21:14.319 --> 00:21:17.559 You're out of your fucking mind. Like their responsibility is to build business 307 00:21:17.680 --> 00:21:22.680 value if they have to go read the twenty seven pages of how to monitor, 308 00:21:23.599 --> 00:21:27.559 configure secure YadA YadA, YadA, YadA, YadA dada postgress. When 309 00:21:27.559 --> 00:21:30.759 it's like locally, I just like Docker run postgress, It's like that's the 310 00:21:30.839 --> 00:21:33.319 abstraction that they should see. They should say, I need postgresses, and 311 00:21:33.319 --> 00:21:37.160 they should be able to assume that whatever's happened on the other side, I've 312 00:21:37.160 --> 00:21:41.000 got secure compliance, scalable, cost effective thing. I shouldn't have to think 313 00:21:41.039 --> 00:21:45.440 about costs, and I shouldn't have to be sidelined on it three months later 314 00:21:45.440 --> 00:21:48.960 when the cfo's pissed off or we're spending thirty grand a month on a misconfigured 315 00:21:48.000 --> 00:21:53.559 post resistance. Oh sorry, So that's why I think it's bullshit. But 316 00:21:53.599 --> 00:21:56.839 here's the catch is, like I'm also one of these people. It's like, 317 00:21:56.880 --> 00:21:59.880 hey, platform engineering, you got to do this right, which I 318 00:22:00.079 --> 00:22:03.039 will admit platform engineering is a subset of DevOps, but the catches, like 319 00:22:03.200 --> 00:22:07.440 most people don't have the budget to actually do it or the talent to actually 320 00:22:07.440 --> 00:22:11.759 do it because their DevOps team or operations team is in drowning and technical debt, 321 00:22:12.160 --> 00:22:15.680 and like trying to ticket ops for their developers and people that just have 322 00:22:15.759 --> 00:22:21.359 developers and don't have an OPS team, they're probably better served by the past 323 00:22:21.480 --> 00:22:23.160 and going directly to the cloud until they get to the point that they need 324 00:22:23.160 --> 00:22:30.519 that scale. So that's me defending the DevOps as bullshit. So it was 325 00:22:30.599 --> 00:22:34.119 like a therapy session. Yeah, I'm gonna get your insurance details after the 326 00:22:34.160 --> 00:22:41.160 podcast here that. Yeah, so I look forward to comments after the show. 327 00:22:45.200 --> 00:22:51.079 No, I agree with you. And the irony here is I actually 328 00:22:51.160 --> 00:22:56.039 have the title at my current employer of DevOps Engineering, and I agree, 329 00:22:56.079 --> 00:23:00.240 like that should not be a thing. And I think part of it just 330 00:23:00.279 --> 00:23:07.200 become to in my perspective is we've we've we're just kind of like that's where 331 00:23:07.200 --> 00:23:11.039 we've drawn the line in the sand, like, oh, we're doing DevOps, 332 00:23:11.119 --> 00:23:12.799 damn it, no matter how hard it is. And it's kind of 333 00:23:12.880 --> 00:23:18.359 like, you know, being a forward guy or a Chevy guy. You 334 00:23:18.400 --> 00:23:21.200 know, if you if you buy a Chevy, you're a Chevy guy. 335 00:23:21.240 --> 00:23:23.359 Now you you know, it's like I can't buy a Ford, or they're 336 00:23:23.400 --> 00:23:26.200 going to kick me out of the line dancing club or whatever, you know, 337 00:23:26.200 --> 00:23:33.559 and we have this failure to adapt with the way that our our industry 338 00:23:33.759 --> 00:23:37.920 has changed. And you know, because you can't, like you mentioned, 339 00:23:37.960 --> 00:23:45.519 you can't ask a software engineer to be proficient in writing code and go, 340 00:23:45.720 --> 00:23:49.160 but you also have to be an expert in Postgress database and uh, an 341 00:23:49.200 --> 00:23:56.720 expert in Docker and kubernetties and security gateways and cost management. There's just so 342 00:23:56.839 --> 00:24:03.839 much context switching there that you'll never get anything better than mediocre across the board. 343 00:24:03.319 --> 00:24:10.559 Yeah, and it's funny because, like I feel like I might be 344 00:24:10.640 --> 00:24:15.279 wrong here, I probably am. Okay, Like I feel like two things 345 00:24:15.640 --> 00:24:18.480 have kind of got us into the mess that we're in with develops, right, 346 00:24:18.480 --> 00:24:21.720 because if you think about just like an engineer, like the amount of 347 00:24:21.759 --> 00:24:26.119 stuff that we will stack as a responsibility on engineers, right, why does 348 00:24:26.160 --> 00:24:27.400 this happen? Right? So? Yeah, I'm a software developer, right, 349 00:24:27.480 --> 00:24:30.400 software? Okay, I use a database. I picked postgrats. Okay, 350 00:24:30.400 --> 00:24:34.559 well this is an Oracle nine I it's Postgress. So I'm assuming that 351 00:24:34.559 --> 00:24:38.440 you're the database administrator. Also software developer, you're going to tune your indexes 352 00:24:38.920 --> 00:24:44.359 right like a decade then fifteen years ago, like we had DBAs, but 353 00:24:44.400 --> 00:24:48.720 like you're seeing them become less and less prevalent in businesses except for like massive, 354 00:24:48.759 --> 00:24:51.240 massive businesses because like, oh, it's like that person will figure it 355 00:24:51.279 --> 00:24:56.000 out, and then so that happens. We stack that responsibility on people, 356 00:24:56.559 --> 00:25:00.720 and I think it's twofold. I think one's business people. I don't understand 357 00:25:00.839 --> 00:25:04.559 like where the lines are. And we're again we're hobbyists. So it's like 358 00:25:04.599 --> 00:25:11.279 okay, like, hey, yay, will do you know how like LLM 359 00:25:11.359 --> 00:25:14.319 building works? And you're like no, And you're like, well, you've 360 00:25:14.319 --> 00:25:15.960 got a better chance of figure it out than me over here. It's working 361 00:25:17.000 --> 00:25:19.599 on the old it's right. And so you're like, if you give a 362 00:25:19.599 --> 00:25:23.160 software engineer like time to go discover something, they will do it. Now, 363 00:25:23.160 --> 00:25:26.720 will they become a master of it? No? Will they get something 364 00:25:26.759 --> 00:25:32.359 that works enough to go into production? Yes? And that's where our product 365 00:25:32.359 --> 00:25:33.960 managers are content, like they're not like, oh, Corey, did you 366 00:25:34.039 --> 00:25:38.519 master building elements before you've lobbed this thing into a kuberinetes pluster. No? 367 00:25:40.359 --> 00:25:42.519 But it works right, And that's all matters. Like I made the button 368 00:25:42.519 --> 00:25:45.920 click faster, I sold more of the things. Like at the end of 369 00:25:45.960 --> 00:25:48.759 the day, that's what matters to the business, not necessarily that it's perfectly 370 00:25:48.799 --> 00:25:53.000 secure. Do you care about that business? Do you care that it's perfectly 371 00:25:53.079 --> 00:25:56.920 secure? Or are you going to buy an insurance plan from Equifax to cover 372 00:25:57.000 --> 00:26:02.480 people's credit when it breaches. It's much much cheaper to buy the insurance plan 373 00:26:02.559 --> 00:26:06.720 than to spend the engineering resources to make it secure. Yeah. Yeah, 374 00:26:06.960 --> 00:26:10.880 as just ask my real estate and investment company that I'm working with, because 375 00:26:10.920 --> 00:26:12.559 I prompted them on that. I'm like, how much time you spill security? 376 00:26:12.680 --> 00:26:15.440 Are things perfectly secure? Well, then we do investing. There was 377 00:26:15.480 --> 00:26:18.640 a breach and like we leaked your social Security number and all your financials. 378 00:26:18.640 --> 00:26:23.000 I was like, I knew my credit's locked already. It's fun. But 379 00:26:23.039 --> 00:26:26.559 so I think, like, you know, that's that's it's really interesting, 380 00:26:26.559 --> 00:26:32.759 Like we'll just stack stuff on people. And I think that's kind of where 381 00:26:33.400 --> 00:26:36.000 it's easy for burnout to happen, right, Like you think you're going to 382 00:26:36.039 --> 00:26:37.759 be focusing on this problem, and you are, You're trying to solve the 383 00:26:37.759 --> 00:26:41.039 problem, but all these other things are kind of tangential to it, and 384 00:26:41.079 --> 00:26:45.079 if you don't have the people that know that thing, your company is looking 385 00:26:45.119 --> 00:26:48.839 around like okay, who has either the bandwidth or who do we think is 386 00:26:48.880 --> 00:26:51.160 like the best engineer that can go figure it out? And like now they 387 00:26:51.200 --> 00:26:53.920 have another technology and other responsibility that kind of lands on their plate. If 388 00:26:53.960 --> 00:26:56.839 you're a small company, you're starting to grow and you're like, okay, 389 00:26:56.960 --> 00:26:59.480 like I'm doing a lot of this cloud stuff, like we should hire our 390 00:26:59.480 --> 00:27:03.359 first DevOps engineer, and so now it's eight people that don't have like this 391 00:27:03.440 --> 00:27:06.759 deep experience in the cloud. They are like, we should hire somebody that 392 00:27:06.799 --> 00:27:11.359 does. Then you start that job, you're the death first DevOps engineering. 393 00:27:11.400 --> 00:27:12.279 You're like, okay, I had a lot of work for like three months 394 00:27:12.319 --> 00:27:15.799 and I was stressful, But now I'm sitting around like five hours of work 395 00:27:15.839 --> 00:27:19.200 to do a month. Right. Then that's where it gets weird, because 396 00:27:19.359 --> 00:27:22.720 like you can do two things at that point time. You can either sit 397 00:27:22.759 --> 00:27:26.079 back and be like I have thirty five hours of drinking coffee, or you 398 00:27:26.119 --> 00:27:29.880 can try to do like real DevOps how do I make this easier for those 399 00:27:29.920 --> 00:27:32.640 developers? How do I make things more efficient? How do I make things 400 00:27:32.680 --> 00:27:36.279 more resilient. How can I start practicing some SRI principles since I have this 401 00:27:36.359 --> 00:27:40.000 bandwidth to help make sure their software is staying up. And that's where I 402 00:27:40.000 --> 00:27:44.799 think I feel like a lot of DevOps people, particularly if you look at 403 00:27:44.839 --> 00:27:47.599 our slash DevOps, so that's a group of people not saying it, but 404 00:27:47.720 --> 00:27:49.680 like the amount of times people that are like I have nothing to do, 405 00:27:49.920 --> 00:27:53.400 Like I've been working here, like I work like ten hours a week and 406 00:27:53.440 --> 00:27:57.079 like everything's like kind of messy still, but like nobody's given me, like 407 00:27:57.599 --> 00:28:00.519 the DNS project to work on. And it's like, well, you're supposed 408 00:28:00.519 --> 00:28:03.920 to be talking to your you're supposed to be doing that collaboration part, talking 409 00:28:03.960 --> 00:28:07.359 to your engineers for how you can make their life better, Like that is 410 00:28:07.440 --> 00:28:11.039 your job, and like that devlops role is like continue to make the software 411 00:28:11.039 --> 00:28:15.640 better, the business better, and the engineering dev experience better. And I 412 00:28:15.680 --> 00:28:19.000 think that like once you start going down that path, like that is a 413 00:28:19.000 --> 00:28:22.960 good place for DevOps to be. But where we're just tossing stuff over a 414 00:28:23.000 --> 00:28:26.920 fence to a person and we're like, oh, well, the devlops team 415 00:28:26.920 --> 00:28:32.359 they configure the cloud things for me, Like that that's that's that's just two 416 00:28:32.440 --> 00:28:38.200 thousand and five. I'll just pay for sure. Yeah, I take the 417 00:28:38.240 --> 00:28:44.880 approach that as a as my role in develops. The software engineers are my 418 00:28:45.000 --> 00:28:48.640 customers. So my job is to make it to like provide the tools that 419 00:28:48.680 --> 00:28:59.359 they can consume that let them do their job more efficiently and putting in putting 420 00:28:59.359 --> 00:29:03.400 in guardrails so that they can do what they need to do. But then 421 00:29:03.440 --> 00:29:07.640 I have guardrails in place so that they don't accidentally do something that they may 422 00:29:07.680 --> 00:29:12.000 not want to do, like you know, expose root access to the public 423 00:29:12.000 --> 00:29:17.319 Internet or something like that. Yeah, I think those guardrails are important. 424 00:29:17.319 --> 00:29:22.240 Like that's actually a term that we use pretty pretty frequently and like our sales 425 00:29:22.279 --> 00:29:27.640 calls and even in our content, Like I feel like again like our software 426 00:29:27.680 --> 00:29:32.440 is becoming more and more cloud services that we're consuming and less of our code 427 00:29:32.559 --> 00:29:36.119 a lot of used to B calls, and I feel like when you're using 428 00:29:36.160 --> 00:29:40.559 the cloud, like there's so many services like how do you do how do 429 00:29:40.599 --> 00:29:44.799 you how do you do event driven systems on AWS? That's an SQUS, 430 00:29:45.000 --> 00:29:49.240 is it event bridge? Is it POFKA, Like like I've had three or 431 00:29:49.279 --> 00:29:52.839 four options just in AWS, right, and then let's say I pick it. 432 00:29:53.559 --> 00:29:56.799 Okay, let's say I pick my solution. Looking through some of the 433 00:29:56.920 --> 00:30:02.960 AWS like models when you like post your thing to make a new thing, 434 00:30:03.079 --> 00:30:07.000 Like, some of them are insane. Like one of the most crazy services 435 00:30:07.000 --> 00:30:11.960 at AWS is that's three. That's three, like one hundred and twenty configuration 436 00:30:11.079 --> 00:30:15.680 options, right, And so there's a lot of stuff that is getting exposed 437 00:30:15.720 --> 00:30:18.799 to your engineers. So if you're like, oh, I did DevOps because 438 00:30:18.839 --> 00:30:22.920 like I installed terra form and I get of action and now they can write 439 00:30:22.960 --> 00:30:26.559 whatever, Like you didn't you just you just gave everyone a massive footgun. 440 00:30:26.599 --> 00:30:30.240 You didn't make their life better. Sure they can do whatever they want, 441 00:30:30.319 --> 00:30:33.079 but like should they be able to do whatever they want? Right? And 442 00:30:33.079 --> 00:30:37.359 I think that what we need to be doing as an industry DevOps, platform, 443 00:30:37.359 --> 00:30:44.400 engineering operations, whatever your company calls it, is like enabling those developers 444 00:30:44.440 --> 00:30:47.119 to do what they need to do, but also shrinking the surface area of 445 00:30:47.119 --> 00:30:49.880 the cloud. Right. So, like if you have an established operation scheme 446 00:30:49.920 --> 00:30:55.119 that understands all the cloud services, being able to have like a catalog of 447 00:30:55.200 --> 00:30:59.000 like here's what we use for a venture and systems, it's COFA. If 448 00:30:59.039 --> 00:31:02.000 you don't want to use COFT, that is your choice, but it's now 449 00:31:02.000 --> 00:31:06.160 your operational burden. Like make an RFCs to why we should also support e 450 00:31:06.200 --> 00:31:08.599 vent bridge and like we will take on owning a vent bridge. But like 451 00:31:08.599 --> 00:31:11.200 if you're just gonna loan range it, like go for it, right. 452 00:31:11.440 --> 00:31:15.920 And also even when you pick a service, like you need to service, 453 00:31:15.960 --> 00:31:18.640 you need to shrink the surface area of that service, right, Like the 454 00:31:18.720 --> 00:31:23.799 cloud's goal is not to be easy. AWS isn't We're going to be easy. 455 00:31:23.880 --> 00:31:27.759 If they're easy, they would only have lights fail. Their job. 456 00:31:27.880 --> 00:31:32.440 Their job is to be capable. Their job is that you can cut this 457 00:31:32.559 --> 00:31:34.440 pizza in any shape. You want to make your party happy. You want 458 00:31:34.480 --> 00:31:37.960 squares customers, you want we cut wedges, You want to do a weird 459 00:31:37.960 --> 00:31:41.039 little zigzag, you want all edge, I don't know whatever you want to 460 00:31:41.079 --> 00:31:44.599 do, Like we should be able to do it with AWS. And here's 461 00:31:44.640 --> 00:31:48.319 all these things, right, But the reality is is like a lot of 462 00:31:48.319 --> 00:31:52.559 those configuration settings don't matter for your use case high Aurora, like you can 463 00:31:52.559 --> 00:31:56.319 configure Aurora. There's so many ways that I'm gonna go provision postcress and I'm 464 00:31:56.359 --> 00:32:00.720 gonna turn on this thing. That is this field that I can turn on. 465 00:32:00.759 --> 00:32:04.319 It's like that's only for my sequel auror it's like, oh, well, 466 00:32:04.359 --> 00:32:07.480 then why was it exposed to me? I only use Postgress. I'm 467 00:32:07.480 --> 00:32:08.759 not going to randomly switch from postgress to my sqel one day, but I 468 00:32:08.839 --> 00:32:12.799 might randomly change a field name right, or like a field value right, 469 00:32:13.480 --> 00:32:16.720 So it shrinking that service area is important? Like why why? So? 470 00:32:16.799 --> 00:32:21.119 I would love somebody in the comments tell me why I'm wrong. Why is 471 00:32:21.160 --> 00:32:28.759 turning encryption off for anything an option anymore? Yeah? I agree with you, 472 00:32:28.920 --> 00:32:32.920 and I think there's like a certain amount of responsibility. I actually just 473 00:32:34.000 --> 00:32:37.880 had this conversation the other day. I think there's a certain amount of responsibility 474 00:32:37.960 --> 00:32:43.480 from the cloud providers to do a better job here. Like if if I'm 475 00:32:44.519 --> 00:32:51.359 as a cloud provider, they advocate infrastructure's code, least privileged security on all 476 00:32:51.440 --> 00:32:54.480 those kinds of things. So why in their documentation does it say, oh, 477 00:32:54.519 --> 00:33:00.359 if you want to push to the artifact repository, you need artifact repository 478 00:33:00.400 --> 00:33:06.039 dot admin privileges, Like do I really need admin privileges or was there a 479 00:33:06.240 --> 00:33:10.240 better way you could have phrased that to help me adhere to the principles that 480 00:33:10.319 --> 00:33:15.440 your marketing team is advocating. Yeah, and again like I don't, I 481 00:33:15.440 --> 00:33:17.160 don't. Yeah, it's like I don't. I don't think if their goal 482 00:33:17.240 --> 00:33:21.319 is to be easy, it's just like it's it's just to have those capabilities 483 00:33:21.319 --> 00:33:25.200 and right like I AM is like a whole other just when you can see 484 00:33:25.240 --> 00:33:28.400 it, like it's funny, like I feel like some of people like what 485 00:33:28.440 --> 00:33:30.359 are you telling me I AM. It's like I AM is cloud blocking. 486 00:33:30.559 --> 00:33:35.599 Like everything we're using Dynamo CloudLock and it's like, no, you can use 487 00:33:35.640 --> 00:33:37.519 Scala. It's like S three. We're locked in. It's like I just 488 00:33:37.559 --> 00:33:42.599 throw Minio in front of GCS or whatever. Like plenty of ways to get 489 00:33:42.640 --> 00:33:45.240 around cloud locking except for I AM. Like if you invest heavily, if 490 00:33:45.240 --> 00:33:50.839 you go beyond the asterisk, Oh my god. That is that is like 491 00:33:51.160 --> 00:33:54.200 everybody's im is just like completely different. You can you can tell how like 492 00:33:54.359 --> 00:33:59.920 hard it is, like there's I AM engineering roles now in large companies. 493 00:34:00.039 --> 00:34:05.200 It's just like, oh, like I can't like it's important. Oh for 494 00:34:05.200 --> 00:34:10.199 sure, I can't imagine. I personally can't imagine what that job. And 495 00:34:12.239 --> 00:34:14.199 hear you saying with like pushing a container. It's like, oh, you 496 00:34:14.239 --> 00:34:16.559 know, I need right. It's like getting to those places where you're like, 497 00:34:17.679 --> 00:34:21.440 my sole role is to figure I am for this place, but the 498 00:34:21.440 --> 00:34:24.119 cloud doesn't give me the ability to make maybe maybe a particular service as secure 499 00:34:24.159 --> 00:34:36.440 as I want, right because of the granularity Google. Yeah, I like 500 00:34:36.559 --> 00:34:37.920 software. It might not sound like it. I actually love it, but 501 00:34:37.960 --> 00:34:42.079 I just yeah, sorry, No, I think that's part of it. 502 00:34:42.119 --> 00:34:47.880 I think that's one of the common characteristics of software engineers is there's a really 503 00:34:47.960 --> 00:34:52.159 really tight balance of which we like more. Do we like writing a code 504 00:34:52.199 --> 00:34:55.559 more, or do we like bitching about writing the code more? And and 505 00:34:55.599 --> 00:35:00.599 it's it's a tight race bitching about it. I coulde. I still do 506 00:35:00.639 --> 00:35:06.079 a lot of development, probably more than I should, but I switched between 507 00:35:06.119 --> 00:35:09.679 Go Go Lang and a lickxer that's where our app's built in pretty frequently. 508 00:35:09.920 --> 00:35:14.199 And you know, whatever language I'm in, I'm just like, oh, 509 00:35:14.199 --> 00:35:22.119 I hate this language. I wish it's if I'm going to rewrite this in 510 00:35:23.079 --> 00:35:27.920 language acts, then you get it done and you're like, that was stupid. 511 00:35:28.239 --> 00:35:30.079 Yeah. I had one of those yesterday. Ask him about four hours 512 00:35:30.079 --> 00:35:32.800 working on something, and I was just like, I finished, I was 513 00:35:32.840 --> 00:35:37.440 like, why did I? Why did I? I'm had a purpose when 514 00:35:37.440 --> 00:35:39.960 I started, it got way out of hand, like I still do it, 515 00:35:39.960 --> 00:35:43.039 and like I over engineered the ship out of it, and I was 516 00:35:43.079 --> 00:35:45.559 like, I don't. I don't know why I spent four hours on that, 517 00:35:45.599 --> 00:35:47.760 But I gotta go do some real work now, right, Yeah, 518 00:35:47.920 --> 00:35:52.239 yeah, I'm gonna do something that I can actually claim because I can't show 519 00:35:52.320 --> 00:35:54.320 tomorrow and say that's all I did. If you feel that way, people, 520 00:35:54.360 --> 00:35:57.920 it's hard to shake. This is this is This is me years into 521 00:35:57.920 --> 00:36:00.719 being a CEO, and I will still go over engine or something if I've 522 00:36:00.719 --> 00:36:05.320 got a spare of time. Oh for sure. Yeah. I actually for 523 00:36:05.480 --> 00:36:07.679 my side projects, I have a group of guys that I hang out with, 524 00:36:08.159 --> 00:36:14.800 and we all know that our sole job as part of that group is 525 00:36:14.840 --> 00:36:19.199 to call each other out on that specific thing, like are you chasing a 526 00:36:19.239 --> 00:36:24.199 shiny object? No? Well okay maybe yes? All right, fine, 527 00:36:24.239 --> 00:36:29.039 I'll go back to what I'm supposed to be doing. I'm gonna start the 528 00:36:29.079 --> 00:36:31.360 antithesis of that group. I'm gonna start like a Rube Goldberg engineering group. 529 00:36:32.599 --> 00:36:37.719 I think you've made that more complicated and like throwing some more technology into It's 530 00:36:37.760 --> 00:36:39.599 like, you know what I could, I could I'm gonna spend a couple 531 00:36:39.639 --> 00:36:47.760 of hours this week and doing that an LLM that generates Bash scripts. Oh 532 00:36:47.800 --> 00:36:53.239 man, that would be just go. I'm afraid of I'm like the next 533 00:36:53.239 --> 00:36:57.000 five years. I'm like, oh, I'm in like like a I think 534 00:36:57.000 --> 00:37:00.239 I'm in the trophic disillusionment around AI. Like people keep asking us, we 535 00:37:00.480 --> 00:37:06.639 did our fundraise this year. We raised eight million dollars and we did it 536 00:37:06.679 --> 00:37:07.960 without putting I mean, it was a hard year to raise. We did 537 00:37:08.000 --> 00:37:13.760 it without putting the words machine learning, data science or AI in our deck. 538 00:37:14.480 --> 00:37:20.000 And we've stayed away from it in like our materials just because like I 539 00:37:20.039 --> 00:37:24.599 don't I don't want to build our product off of like some like trend that 540 00:37:24.760 --> 00:37:30.039 maybe not go where we hope it goes. And seeing some of the other 541 00:37:30.320 --> 00:37:34.320 people in the OPS space like ah AI ops and You're like, Okay, 542 00:37:35.000 --> 00:37:37.440 I'm going to give this a spin, and I'm like, I would like 543 00:37:37.960 --> 00:37:43.800 a Hippa and sock two compliant Ubernetti's cluster. And I will not mention the 544 00:37:43.880 --> 00:37:45.320 name of the other cloud provider, but I shit you not. The code 545 00:37:45.320 --> 00:37:52.679 that it generated was some type scripty codeink wink. It was all the code 546 00:37:52.679 --> 00:37:55.880 to like initialize an KAS cluster, and then the comments I kid you not, 547 00:37:57.119 --> 00:37:59.440 I kid you not. The only code was like some import statements. 548 00:37:59.440 --> 00:38:04.519 It was like stree equals new EKS cluster, and then in the comments it 549 00:38:04.599 --> 00:38:08.320 was like configuration for stock too and nip A compliance here. And I was 550 00:38:08.400 --> 00:38:15.639 like, yeah, the intelligence is pretty fricking artificial if you ask me, 551 00:38:16.840 --> 00:38:20.440 but like I don't know, like the thing that's really freaking about it. 552 00:38:20.480 --> 00:38:22.840 I kind of got it, couldn't even imagine it for bachelors be terrifying, 553 00:38:22.840 --> 00:38:24.400 But like this is this is again a part of our problem, like we 554 00:38:24.440 --> 00:38:30.320 don't have enough opps experience in most of our engineering team, and so the 555 00:38:30.400 --> 00:38:34.159 idea kind of like business people looking at engineers and not being able to tell 556 00:38:34.239 --> 00:38:37.559 like where the lines are the same thing happens when you go and ask something 557 00:38:37.599 --> 00:38:39.360 to write software for something you're not familiar with, if you aren't familiar with 558 00:38:39.639 --> 00:38:44.159 running Gubernetes, or maybe it's even just configuring some serveralist stuff, you know, 559 00:38:44.199 --> 00:38:45.960 step functions or whatever, and you're like, oh, I need some 560 00:38:45.079 --> 00:38:49.199 terrorform. It'll do all the step function stuff, make it for my computer. 561 00:38:49.559 --> 00:38:53.079 It's like okay, like cool, you might run that and it might 562 00:38:53.079 --> 00:38:57.760 apply it may or may not do what you intended it to do. But 563 00:38:57.840 --> 00:39:00.559 like now you have a bag of code in front of you, you don't 564 00:39:00.639 --> 00:39:04.440 understand it's doing something. You don't know if it's doing it right. And 565 00:39:04.559 --> 00:39:09.679 like chat GPT was confidently Hey I did it right. I think like you 566 00:39:09.679 --> 00:39:13.800 have no idea, so like are you going to go Google every single one 567 00:39:13.800 --> 00:39:16.599 of those lines of code that isn't the resource block to figure out like, 568 00:39:16.639 --> 00:39:19.880 oh, is this is this what I need to set for this? Like 569 00:39:19.920 --> 00:39:23.360 no, you're like just like the product manager looking like that works right right? 570 00:39:23.440 --> 00:39:29.480 Yeah, I got step pumption. I'm moving on. It's like that's 571 00:39:29.559 --> 00:39:30.840 that's freaking. It's like I would rather have you just gone on, Like 572 00:39:30.920 --> 00:39:35.679 I would just please go click it in the AWS console, like turn on 573 00:39:35.719 --> 00:39:38.559 a macro reporter, install the Chrome extension that led like records your mouse clicks. 574 00:39:38.800 --> 00:39:42.719 Go do that and just use the Chrome extension as like your I C 575 00:39:42.880 --> 00:39:45.360 tool at that point in time. I'd rather do that than run something through 576 00:39:45.519 --> 00:39:51.199 AI and hope it right. Like that's great. I've been I've been struggling 577 00:39:51.199 --> 00:39:53.559 with this for a while is for people who do click ops, how do 578 00:39:53.639 --> 00:40:00.280 you how do you document and track that? But I never thought of just 579 00:40:00.400 --> 00:40:06.639 using the click recorder and Chrome. That's brilliant. The problem solved. Yeah, 580 00:40:07.519 --> 00:40:09.920 yet we've got these We've got these video files. Just watch them and 581 00:40:09.960 --> 00:40:13.280 you'll see how I was built. Well, if I'm a lot of the 582 00:40:13.320 --> 00:40:15.519 I'm not sure I'm a lot to plug myself. But you're looking for something 583 00:40:15.559 --> 00:40:20.639 that feels like clickops through engineers, but you want all that operational maturity and 584 00:40:20.719 --> 00:40:23.480 security. No, I do want to talk about that, which is one 585 00:40:23.480 --> 00:40:27.079 of the reasons why I was looking forward to having you on the show here, 586 00:40:27.119 --> 00:40:34.199 because I like I'm at the point where I advocate strongly for infrastructure's code 587 00:40:34.800 --> 00:40:42.519 because six weeks from now, whenever something is different, I want to be 588 00:40:42.599 --> 00:40:46.000 able to look at a get history or poor request history and see what has 589 00:40:46.119 --> 00:40:52.000 changed, or run terraform and see that whatever's running out on that provider doesn't 590 00:40:52.000 --> 00:40:55.280 match what terraform says it should be. And that's been one of the big 591 00:40:55.360 --> 00:41:02.639 challenges I've had with the platform engineering tools, that some of them are just 592 00:41:04.280 --> 00:41:07.519 clickops on top of clickops, you know, and it's like if I just 593 00:41:07.559 --> 00:41:10.639 want to click ops, I would just use AWS. So how do you 594 00:41:10.679 --> 00:41:15.719 approach that? Yeah, so I think that. Okay, So I'm gonna 595 00:41:15.719 --> 00:41:17.360 tell you a couple of things about me. I'm a software engineer, so 596 00:41:17.400 --> 00:41:22.079 I understand like I understand vendors, and I have to struggle as like a 597 00:41:22.119 --> 00:41:27.880 CEO and a founder and also a developer where like I don't want to be 598 00:41:27.960 --> 00:41:30.079 the annoying vendor. That's why I was like, I don't know want the 599 00:41:30.079 --> 00:41:31.800 plug like I want to talk about like I want to sell this thing, 600 00:41:31.800 --> 00:41:36.079 but at the same time, like I don't want to annoy people. But 601 00:41:36.280 --> 00:41:42.320 uh but yeah, So what's interesting is so the second thing about me, 602 00:41:42.320 --> 00:41:44.480 I'm going to say something and I'm going to be I'm not going to be 603 00:41:44.480 --> 00:41:50.079 the founder that says this and is disingenuous. I think we're going about enabling 604 00:41:50.119 --> 00:41:53.320 platform engineering much differently than a lot of the other tools out there. Right, 605 00:41:53.400 --> 00:41:58.639 So what's really freaking me out about this is I think platform engineering is 606 00:41:58.679 --> 00:42:01.480 important. Actually, I make a note of like the clickops, how we 607 00:42:01.519 --> 00:42:04.639 do it different. I'm gonna come back to that in a second. Do 608 00:42:04.639 --> 00:42:06.360 I haven't have time to set way? He said, we can go on 609 00:42:06.400 --> 00:42:07.960 forever, right, I can, Yeah, there's there's there's no time on 610 00:42:08.039 --> 00:42:12.840 it. So here here's why this is important. I spend a lot of 611 00:42:12.880 --> 00:42:16.559 time like reading surveys. I like it. It's one of my weird things. 612 00:42:16.639 --> 00:42:19.440 And I feel like a lot of times, like when you get a 613 00:42:19.440 --> 00:42:22.400 survey at the door report, there's like a couple of like pull quotes and 614 00:42:22.440 --> 00:42:23.519 some numbers, and you're like, oh, we're all doing much better. 615 00:42:23.639 --> 00:42:25.639 But then when you like get to page like thirty stuff and you're like, 616 00:42:25.679 --> 00:42:32.079 holy crap, everyone's gonna die. Looking at the stack overflow survey for the 617 00:42:32.119 --> 00:42:37.599 past four or five years is really interesting. If you look at cloud operations 618 00:42:37.599 --> 00:42:44.559 experience, every every administrative role, operations role in the stack Overflow survey has 619 00:42:44.599 --> 00:42:47.800 steadily declined in the amount of people doing it over the past five years. 620 00:42:47.880 --> 00:42:52.480 DBAs, SRES, DevOps, operations, all of them are going down. 621 00:42:52.639 --> 00:42:57.119 Now are we retiring beast? Maybe we have a lot of grave yards that 622 00:42:57.159 --> 00:43:00.840 could be the reason. But also we're just making engineers past. We're making 623 00:43:00.840 --> 00:43:04.400 engineers faster than we've ever made engineers ever. Remember seventy years ago, there 624 00:43:04.480 --> 00:43:07.400 was like one software developer. Now there's more software developers on the planet than 625 00:43:07.440 --> 00:43:12.280 there are people in Australia, Like you met an entire, huge ass country 626 00:43:12.280 --> 00:43:15.119 full of kangaroos and software developers. Like that's something we could do, would 627 00:43:15.119 --> 00:43:19.639 be really weird, but we shouldn't do it. But like right, so, 628 00:43:19.800 --> 00:43:22.519 like where are these people coming from. They're coming from boot camps. 629 00:43:22.559 --> 00:43:24.840 Like we're still on a lot of people coming out of universities. We're printing 630 00:43:24.880 --> 00:43:28.639 people out of boot camps, and that's not a bad thing. Some of 631 00:43:28.639 --> 00:43:30.960 my favorite engineers I've hired are out of boot camps. There's a personality type 632 00:43:30.960 --> 00:43:34.679 that I look for when I see somebody coming from a boot camp, like 633 00:43:34.679 --> 00:43:39.360 a fresh grad no history, that's eighteen. That's a person that was really 634 00:43:39.440 --> 00:43:44.599 those probably doing software, got very excited, went straight to a boot camp 635 00:43:44.599 --> 00:43:45.239 because like I want to get in the job. I don't want to sit 636 00:43:45.239 --> 00:43:47.920 four years learning about algorithms to build a database, like I want to build 637 00:43:47.920 --> 00:43:52.159 a website. That person, right, the person that worked at a power 638 00:43:52.199 --> 00:43:57.119 plant for fifteen years and they're like forty three, that decides I'm going to 639 00:43:57.159 --> 00:44:00.119 give up on this career that I've been doing all this time to go in 640 00:44:00.239 --> 00:44:02.920 software, Like that person has grit. I love that person. I think 641 00:44:02.920 --> 00:44:07.920 that boot camps are an important part of our society. Why how do you 642 00:44:07.960 --> 00:44:14.199 attige boot camps to like civilization? Everything's moving to the cloud. Everybody is 643 00:44:14.199 --> 00:44:16.639 a software company. We're becoming more and more dependent on this stuff. We 644 00:44:16.679 --> 00:44:22.840 need more software developments. But what's happening is boot camps don't teach the cloud. 645 00:44:23.559 --> 00:44:27.800 Boot camps don't teach DevOps. They teach you to build react code. 646 00:44:27.800 --> 00:44:30.000 They teach you to build rails. They teach you to add business value as 647 00:44:30.079 --> 00:44:35.280 quickly as possible, which most businesses don't see. DevOps is business value. 648 00:44:35.360 --> 00:44:37.679 Right. So it's like you come out of this boot camp for twelve weeks, 649 00:44:37.840 --> 00:44:40.079 you got your twenty six thousand dollars tuition with like eight percent or whatever, 650 00:44:40.320 --> 00:44:43.840 and you're like, I'm ready to write some reacts. I'm ready to 651 00:44:44.079 --> 00:44:46.920 rails, generate an active record model, like let's go. And so those 652 00:44:46.920 --> 00:44:51.199 people come in and they don't have any DevOps experience. The don't have any 653 00:44:51.239 --> 00:44:53.039 operations experience, right, And I think that's where we're seeing this, like 654 00:44:53.039 --> 00:44:58.239 like this steep drop is like some people are retiring because like we do have 655 00:44:58.599 --> 00:45:00.920 a lot of gray beards, especially on might be in the data centers and 656 00:45:00.960 --> 00:45:05.599 more on the administrative like Linux to step inside. Those people are going away 657 00:45:06.000 --> 00:45:08.320 for sure. We're also just printing a bunch of people that are very much 658 00:45:09.400 --> 00:45:14.719 closer to the top end of the stack, right. And the thing that 659 00:45:14.719 --> 00:45:16.880 sucks with OPS is we don't teach it in universities either, because if you 660 00:45:16.920 --> 00:45:21.400 talk talk Kubernetes. Actually, if you if you rewind four years and you 661 00:45:21.440 --> 00:45:24.199 taught people how to deploy in the cloud, what would you be teaching them. 662 00:45:24.320 --> 00:45:28.199 Probably not Kubernetes four years ago. Today it'd be Kubernetes, but four 663 00:45:28.239 --> 00:45:34.559 years ago maybe the two instance. Yeah, maybe, Like nothing is really 664 00:45:34.559 --> 00:45:37.880 I'm gonna say something bold here. Nothing's really changed about our software in the 665 00:45:37.920 --> 00:45:44.039 past seventy years. LOOPS variables ifs sure, you've got different syntactical sugar on 666 00:45:44.119 --> 00:45:46.280 it. But in the last twenty years, everything about the way that we 667 00:45:46.440 --> 00:45:52.239 package and ship software to the cloud has changed four times. We had data 668 00:45:52.239 --> 00:45:55.360 centers, then we had vms, then we had everybody started going the servi 669 00:45:55.440 --> 00:45:59.599 list, and then containers came around, and then like big Container orchestration, 670 00:45:59.760 --> 00:46:04.920 then servilest containers, now llms, like we've changed the actually llens might be 671 00:46:04.960 --> 00:46:07.760 the first thing that's happened in like the way we write software that's fundamentally changed. 672 00:46:08.000 --> 00:46:10.320 Besides just like if look, you're not because they're all just jam and 673 00:46:10.320 --> 00:46:15.960 hashes into somebody else's database. Now sorry, So so that is sorry. 674 00:46:16.000 --> 00:46:20.079 So that was me explaining like why we're doing things a little different, Like 675 00:46:20.119 --> 00:46:24.639 the the amount of operational knowledge on this planet is going down in relation to 676 00:46:24.639 --> 00:46:28.880 the amount of software developers we have, and those software developers are only going 677 00:46:28.920 --> 00:46:32.039 to learn OPS on the job. That's a problem, right, Yeah, 678 00:46:32.239 --> 00:46:35.559 are they going to learn ops from? Are they? Are all of your 679 00:46:35.599 --> 00:46:38.400 DevOps engineers going to turn into You're going to make a fifty room campus so 680 00:46:38.440 --> 00:46:42.639 you can teach all your software developers like how to do a DevOps You have 681 00:46:42.679 --> 00:46:45.280 brown bags every single day. Like you're not going to transfer that knowledge from 682 00:46:45.280 --> 00:46:50.280 your DevOps team to your engineers in an efficient way, right, So you 683 00:46:50.360 --> 00:46:52.239 need to build the substraction. You need to lower the surface area so they 684 00:46:52.280 --> 00:46:58.079 can do as much as they have access to without having to like jump over 685 00:46:58.119 --> 00:47:00.320 that wall. Right as soon as you reach around that wall, you're like, 686 00:47:00.360 --> 00:47:04.159 hey, I was able to deploy everything myself, but like it's down 687 00:47:05.400 --> 00:47:07.280 Okay, we they gotta be able to troubleshoot themselves, right if they have 688 00:47:07.320 --> 00:47:09.519 to come around to you and be like, hey, I deployed this thing 689 00:47:09.519 --> 00:47:13.440 that build a container. It went up and that is just fucked Like if 690 00:47:13.480 --> 00:47:17.719 they come around to you like that, like they need to have that knowledge 691 00:47:17.760 --> 00:47:21.920 themselves, like we need to be building those abstractions they can solve these problems 692 00:47:21.920 --> 00:47:27.480 themselves. And also at the same time, I think as a platform and 693 00:47:27.760 --> 00:47:30.760 we do this, and we hope that our users do this for their users 694 00:47:30.960 --> 00:47:34.800 do this, but like as platforms and people this knowledge, we need to 695 00:47:34.800 --> 00:47:38.239 figure out how to get it as efficiently as we can into developers because we 696 00:47:38.239 --> 00:47:42.760 do need more people understanding how this stuff works. And I love when one 697 00:47:42.800 --> 00:47:44.960 of our customers even is like, hey, how does this work? Like 698 00:47:45.119 --> 00:47:46.400 under the hoot, It's like, yeah, you should you should be asking 699 00:47:46.440 --> 00:47:50.079 me that, like you should want to know that, but like if you 700 00:47:50.119 --> 00:47:52.599 don't, you should be able to do your job effect And so what I've 701 00:47:52.920 --> 00:47:57.440 so this is a long lead up to me saying I think we do things 702 00:47:57.440 --> 00:48:00.599 a bit differently, which every startup founder sets but the thing that made it 703 00:48:00.599 --> 00:48:06.159 different for us is everybody's talking about platform engineering, and the first thing they 704 00:48:06.239 --> 00:48:09.599 jump to is who there's like Kubernetes. Oh, guess what, I love 705 00:48:09.679 --> 00:48:14.800 Kubernetes. I develop on Kubernetes locally. Like that's how much of a family 706 00:48:15.840 --> 00:48:19.199 I would be, Like, if you want to use my product, you 707 00:48:19.280 --> 00:48:22.840 have to use Kubernetes. I just made a choice for you, right, 708 00:48:22.880 --> 00:48:27.079 Like you might be perfectly fine running on Lambda, right, but we want 709 00:48:27.079 --> 00:48:30.400 to use that forever. So I think that, and I think one of 710 00:48:30.440 --> 00:48:34.199 the things that's a little scary about this idea of platform engineering is like we're 711 00:48:34.280 --> 00:48:37.679 leaning too heavily towards Kubernetes and not accepting the fact that there are other things. 712 00:48:37.719 --> 00:48:40.719 I mean, not that anybody's going to run out and run masive sphere, 713 00:48:40.760 --> 00:48:44.880 but like there is nomad, there are other run times, there are 714 00:48:44.880 --> 00:48:50.360 things that are outside of Kubernetes. Right. It's very plausible that your software 715 00:48:50.400 --> 00:48:55.360 teams in your company operate their software differently. And so the idea that you're 716 00:48:55.360 --> 00:48:59.880 going to come in and be like we have one platform and everybody, nine 717 00:49:00.079 --> 00:49:01.960 percent of you are going to fit your code into this one platform. It's 718 00:49:02.000 --> 00:49:07.000 like, Okay, that sucks for the jam stacked people. That's suck inc. 719 00:49:07.920 --> 00:49:08.960 So what are we doing? Are we gonna run Are we gonna run 720 00:49:09.159 --> 00:49:13.679 llms on versell? Are you gonna run jam stacked on open Ai? Like? 721 00:49:13.679 --> 00:49:15.280 How are we gonna fit this all on the one platform? The reality 722 00:49:15.400 --> 00:49:20.159 is like much like organizations are different, like your teams are different, and 723 00:49:20.199 --> 00:49:24.360 so you can't have a platform that does everything. Those engineers need to be 724 00:49:24.360 --> 00:49:30.199 able to have the agency to get what they need from the cloud without constantly 725 00:49:30.239 --> 00:49:32.440 bothering another person or reading a white paper, and the business needs to know 726 00:49:32.840 --> 00:49:39.199 that that software is secure and compliant. That's hard to do when most companies 727 00:49:39.239 --> 00:49:44.639 are abysmally failing at DevOps. Fifty one percent of you based on the door 728 00:49:44.719 --> 00:49:46.719 or report, So if that's not you, one of your friends is lying. 729 00:49:50.559 --> 00:49:52.519 And also like a lot of these companies, like you can see a 730 00:49:52.559 --> 00:49:57.960 company with like fifteen engineers and like nobody really has ops experience, right, 731 00:49:58.000 --> 00:50:00.400 So it's like we just don't have enough of this, and so our approach 732 00:50:00.880 --> 00:50:07.800 has been to focus on operations engineers and DevOps engineers that are either coming to 733 00:50:07.840 --> 00:50:13.360 this realization that they cannot scale themselves or their company is kind of mandated, 734 00:50:13.400 --> 00:50:15.280 like hey, Gartner said, eighty percent. Companies are gonna have a platform 735 00:50:15.320 --> 00:50:20.559 team by talking about the wrong side platform team by twenty twenty six, and 736 00:50:20.599 --> 00:50:23.599 it's going to make DevOps more efficient. So DevOps Guide, you're a platform 737 00:50:23.639 --> 00:50:30.199 engineer now and does that mean right? And so our approach is we should 738 00:50:30.199 --> 00:50:35.119 be able to work with So like our product's number one user is operations, 739 00:50:35.159 --> 00:50:37.239 they don't use our site, they use the tools that they know. So 740 00:50:37.360 --> 00:50:43.840 mass driver today works with Terraform, open TOFU and helm and we're currently working 741 00:50:43.880 --> 00:50:49.039 on CloudFormation and Gloomy support. And so the idea is you as an operations 742 00:50:49.039 --> 00:50:52.599 team can decide what cloud services you want to support and then you limit that 743 00:50:52.679 --> 00:50:57.519 API of those cloud services using Terraform, Helm, open Too, Food, 744 00:50:57.519 --> 00:51:00.719 et cetera. And so you just publish okay we or Kafka for events, 745 00:51:00.840 --> 00:51:05.800 we support postgresses or transactional database. We run on Kubernetes. We also support 746 00:51:05.840 --> 00:51:09.880 land n API gateway for the services team, and you essentially write Terraform modules 747 00:51:09.880 --> 00:51:15.400 with your whether it's OPA or bridge crew, whatever inside of it. You 748 00:51:15.480 --> 00:51:21.480 create these modules that are your businesses, security compliance and practices spotified into the 749 00:51:21.599 --> 00:51:24.559 naming conventions, et cetera. You publish them into mass driver instead of like 750 00:51:24.599 --> 00:51:29.320 the Terraform registry or whatever. And now when your developers come in, what 751 00:51:29.400 --> 00:51:31.519 they see to state instead of seeing all the services of a WS, they 752 00:51:31.559 --> 00:51:37.159 see the services that my operations team supports. Now what's really cool is they 753 00:51:37.199 --> 00:51:39.719 just draw infrastructure. So like, okay, like we run on Kubernetes, 754 00:51:39.760 --> 00:51:43.760 Like, throw my Kubernetes cluster up there. I have a whole thought, 755 00:51:43.840 --> 00:51:45.239 I have a whole I have a whole rant. I can go on about 756 00:51:45.239 --> 00:51:49.199 how many Kubernetes plusters you should have. It should be more than one. 757 00:51:49.679 --> 00:51:57.320 Cattle cattle versus pets. People, you got all these all these applications or 758 00:51:57.360 --> 00:52:00.599 cattle cluster is certainly a pet. Is that you're afraid to change it. 759 00:52:04.159 --> 00:52:08.039 But we named it and everything we actually had, sorry i'm gonna say something, 760 00:52:08.440 --> 00:52:13.519 and a little side tangent. We had a cople of company flip trips 761 00:52:13.559 --> 00:52:15.639 that we were at. We had essentially two Kubernetes clusters of van production. 762 00:52:15.719 --> 00:52:19.440 And what we would do is every time that we were doing a Kubernettes upgrade, 763 00:52:19.559 --> 00:52:22.840 we would do it too essentially our standby, and then we'd roll applications, 764 00:52:22.920 --> 00:52:24.599 cut traffic to it. And so like that's how we kind of determined 765 00:52:24.599 --> 00:52:28.039 that, like okay, upgrading at one point two seven or whatever, it 766 00:52:28.159 --> 00:52:31.840 worked. And both of those clusters were pets ish we created them with cub 767 00:52:31.880 --> 00:52:37.599 admin or cub no cops, but they both had names and it was because 768 00:52:37.679 --> 00:52:44.320 Siri could never get Kubernetes right, so we had cuban Eddie and Kubernetti. 769 00:52:46.559 --> 00:52:51.000 But anyways, still like our our approaches, all right, I'm a I'm 770 00:52:51.039 --> 00:52:53.639 a brain map to follow me, but so approaches like make sure the operations 771 00:52:53.639 --> 00:52:57.559 people don't have to learn a new tool chain, right, like use the 772 00:52:57.559 --> 00:53:00.880 stuff that you already know, Like there's not enough of you. I can't 773 00:53:00.239 --> 00:53:05.599 as a business, have you spending more time learning a bunch of new stuff 774 00:53:06.199 --> 00:53:09.000 when engineers are already depending on you for stuff, demanding stuff from you. 775 00:53:09.000 --> 00:53:12.119 Your team is already stressed out, and now we're saying, hey, we 776 00:53:12.159 --> 00:53:15.760 want to make we want to scale you to make it more efficient for them 777 00:53:15.840 --> 00:53:17.119 by adding more work to your plate on top of the other stuff that's going 778 00:53:17.159 --> 00:53:20.639 to happen. It's like okay, well, or was if we're also kind 779 00:53:20.639 --> 00:53:22.679 of like helping with outages and monitoring, et cetera, like Are we just 780 00:53:22.679 --> 00:53:25.760 not doing that anymore? Why we go build a platform, right? Are 781 00:53:27.159 --> 00:53:30.519 we all going to turn into React engineers so we can build backstage plug ins? 782 00:53:30.519 --> 00:53:32.559 And it's like no, Like so our approach has always done an operations 783 00:53:32.599 --> 00:53:36.639 engineer, Like we wanted to build something that worked with the tools that your 784 00:53:36.639 --> 00:53:38.760 team knows. You publish it in there. Your engineers just draw the infrastructure, 785 00:53:38.760 --> 00:53:43.880 and so as they're drawing, what happens is they drag let's say they 786 00:53:43.920 --> 00:53:46.760 drag an API gateway out or a lambda. There's these additions that you can 787 00:53:46.760 --> 00:53:52.719 make to your infrastructure's code we call we internally we call them artifacts. But 788 00:53:52.000 --> 00:53:57.840 it effectively lets you create a diagram of your terraform. So you can say, 789 00:53:57.880 --> 00:54:02.599 like my application run time like terrifle module or healm module depends on Kubernetics 790 00:54:04.480 --> 00:54:07.039 and so now like when you drag that into the campus, that will automatically 791 00:54:07.039 --> 00:54:09.159 connect like the nearescubernets cluster. You can move it around like your engineers don't 792 00:54:09.159 --> 00:54:10.679 have to think about it. So it's like, Okay, we're building this 793 00:54:10.719 --> 00:54:15.079 service for the first time we wrote it. I'm going to use the helm 794 00:54:15.199 --> 00:54:20.199 chart here for running going applications. I drag it on. I set my 795 00:54:21.039 --> 00:54:24.360 Docker repository and it just connects to kubernets cluster, I hit deployed and I 796 00:54:24.360 --> 00:54:28.960 get back to work, right, And so they have this limited view of 797 00:54:28.960 --> 00:54:32.039 the cloud. It's the services that your team supports. But then also your 798 00:54:32.119 --> 00:54:37.199 terraform variables are also when they come into mass Driver, are become heavy documentation. 799 00:54:37.320 --> 00:54:43.000 So it's not just like oar name like. You can actually mass driver 800 00:54:43.000 --> 00:54:45.800 supports like super rich and like cross validation, so you can say, like 801 00:54:46.079 --> 00:54:50.679 if the name is this, then this other field has to have this specific 802 00:54:50.719 --> 00:54:52.960 validation on it as well. So you can start to codify a lot of 803 00:54:53.360 --> 00:54:57.039 constraints into the system. So you can say, okay, we have an 804 00:54:57.079 --> 00:55:04.199 Aurora postgrest bundle and it can run. If anybody's a familiar with Aurora, 805 00:55:04.320 --> 00:55:08.920 you too. There's an instance type called serverless. It's very weird, but 806 00:55:09.719 --> 00:55:14.800 so technically you can switch your Aurora back and forth between servilists and server full. 807 00:55:14.840 --> 00:55:16.559 Like you experience a downtime while it's happening, but you can switch it 808 00:55:16.599 --> 00:55:21.000 back and forth. Right, But the configuration changes wildly depending on if the 809 00:55:21.159 --> 00:55:24.440 instance type it is serverless, not if it's a servilists Aurora the instance type 810 00:55:24.440 --> 00:55:30.119 of servilist, right, and so like internally we have an Aurora Postgres bundle 811 00:55:30.159 --> 00:55:32.039 that we use and we use servilists for all of our not like our preview 812 00:55:32.079 --> 00:55:36.320 environments, and then we use to reful for our production environment. And so 813 00:55:36.679 --> 00:55:39.719 that configuration like it opens on the side, panels your terraform variables and then 814 00:55:39.800 --> 00:55:44.519 as you change values, the configuration changes it, it'll actually expose and hide 815 00:55:44.519 --> 00:55:46.559 fields based on this. So now's the operation scheme. You can say, 816 00:55:46.559 --> 00:55:52.679 you know what, I know that this my sql bin log stuff of Aurora 817 00:55:52.000 --> 00:55:55.559 doesn't apply to Postcrest whatsoever. So those fields that there's no those variables are 818 00:55:55.559 --> 00:56:00.000 even in here, you can and worry about thinking about configuring your resource. 819 00:56:00.159 --> 00:56:04.039 And you can also say it when they you have this men in max scale 820 00:56:04.079 --> 00:56:07.519 for your instances, when somebody selects a servilist instance, those fields are hidden 821 00:56:07.599 --> 00:56:12.400 so they don't accidentally say I want ten serviless and instead they show it shows 822 00:56:12.440 --> 00:56:16.719 you acus right, and so you can start to codify a lot of convention 823 00:56:16.840 --> 00:56:21.599 there for them. And so now your engineers they don't have to learn a 824 00:56:21.639 --> 00:56:24.000 new tool leaver, they don't have to go copy and paste a bunch of 825 00:56:24.039 --> 00:56:29.239 Terraform modules and do a directory, set up some GitHub actions. Literally, 826 00:56:29.320 --> 00:56:30.800 the only thing to do is they put our CLI in there and they says, 827 00:56:30.840 --> 00:56:35.000 Okay, I want to deploy my production database or I wanted to pull 828 00:56:35.000 --> 00:56:38.000 in my production application, and it's pretty much just like the name of the 829 00:56:38.039 --> 00:56:42.679 resource and mass driver and it deploys. You can put adjacent file in there 830 00:56:42.679 --> 00:56:45.559 that it'll get applied to either HELM or Terraform or flu Me to configure it 831 00:56:45.599 --> 00:56:50.639 based on the constraints that you've defined as an operations engineer. And now what's 832 00:56:50.639 --> 00:56:55.199 happened is you're able to scale this operations team. You can do platform engineering 833 00:56:55.840 --> 00:57:00.880 effectively. Mass Driver we're a platform orchestrator if you're familiar with the seven hundred 834 00:57:00.960 --> 00:57:04.679 terms that people have invented in this space, and each of your projects and 835 00:57:04.679 --> 00:57:07.679 mass rivers effectively its own internal developer platform. But the cool thing is it's 836 00:57:07.679 --> 00:57:13.639 not developers having to operations team doesn't have to go and talk to developers and 837 00:57:13.679 --> 00:57:16.719 make an MVP and make a giant form to survey them on what they like 838 00:57:16.800 --> 00:57:20.360 and don't like, and collaboration back and forth. You say, this is 839 00:57:20.400 --> 00:57:22.960 what we support as a business. This is all that we know. This 840 00:57:22.039 --> 00:57:25.519 is what we're using today, and if you need a new service, we 841 00:57:25.599 --> 00:57:29.280 get hub issue and we'll build the module for it. It'll be a mass 842 00:57:29.280 --> 00:57:30.519 reever. You can drag and drop it. Right Like, at the end 843 00:57:30.559 --> 00:57:35.800 of the day, all of your modules and mass rivers get up repositories where 844 00:57:35.800 --> 00:57:37.960 your code is like that's how we kind of do the collaboration. And then 845 00:57:38.719 --> 00:57:43.760 your ops team didn't have to learn anything new. Your engineering team didn't have 846 00:57:43.800 --> 00:57:46.280 to go learn terraform or polloomy or gett actions or anything like that. They 847 00:57:46.320 --> 00:57:50.360 can focus on writing code. They just drag something onto the canvas when they 848 00:57:50.400 --> 00:57:52.840 need it. You as the ops person, get an audit log, get 849 00:57:52.840 --> 00:57:58.599 deployment histories, you can roll back. We have automated costs and metrics built 850 00:57:58.599 --> 00:58:00.360 into it, right, so trying to give that full agency to the developer. 851 00:58:00.360 --> 00:58:02.800 As soon as you can figure something a mass driver, a day later, 852 00:58:02.920 --> 00:58:06.920 costs show up. So if you can figure this postpress citizence, you're 853 00:58:06.920 --> 00:58:09.639 like, holy shit, it's three hundred dollars like the next day right Like, 854 00:58:09.679 --> 00:58:14.920 okay, I'm gonna change my configuration here because I can also see utilizations 855 00:58:14.960 --> 00:58:19.920 three percent, so I don't have a mad engineering manager mad CFO three months 856 00:58:20.000 --> 00:58:22.920 later when they realize that cloud goes high, I, as a software developer, 857 00:58:22.159 --> 00:58:24.920 have this agency to control my costs. Do the same tool that I 858 00:58:25.000 --> 00:58:29.280 provision it in the same place that I can see my metrics. If we 859 00:58:29.320 --> 00:58:34.320 turn your dash your canvae, you're essentially your your network diagram is also your 860 00:58:34.360 --> 00:58:36.480 dashboard. So you can say, okay, I want all these metrics from 861 00:58:36.480 --> 00:58:39.800 cloud watch to show up alongside this instance and set some alarms for them. 862 00:58:40.239 --> 00:58:45.320 So now when you get an alarm goes off at two am, because that's 863 00:58:45.320 --> 00:58:49.440 when grimlins use the internet, right, I think, So it's like two 864 00:58:49.519 --> 00:58:51.840 thirteen, it's like, what in the hell is going on? Who is 865 00:58:51.880 --> 00:58:54.400 a Crown job schedule right now? So now, like at two am, 866 00:58:54.440 --> 00:58:58.760 when you get an alarm in an email, it's like you would like Peter 867 00:58:58.840 --> 00:59:00.599 do right. You think about where we are today, It's like, oh 868 00:59:00.599 --> 00:59:04.360 pat patre alert and you're like, oh fuck, I hate that sounds two 869 00:59:04.360 --> 00:59:07.400 thirteen in the morning. Click on it, uh Ari zero alarm and data 870 00:59:07.440 --> 00:59:10.039 Dog. You go to data Dog and it feels like you just walked into 871 00:59:10.159 --> 00:59:15.639 Vegas at three am, just like crash all over the ground. You're like, 872 00:59:15.679 --> 00:59:17.719 Okay, some something's broke. I can see the things going up this 873 00:59:17.840 --> 00:59:22.440 way. It's red. That's not good. But like, what what's broken? 874 00:59:22.480 --> 00:59:27.280 Okay, it looks like I'm out of database connections on my application? 875 00:59:27.679 --> 00:59:31.280 Well, shit, like our traffic's no different than normally is I din't the 876 00:59:31.320 --> 00:59:36.199 application hasn't been deployed in three weeks because it's pretty stable app. We're not 877 00:59:36.239 --> 00:59:42.199 like actively working on Like what's changed? Like if the traffics the same, 878 00:59:42.639 --> 00:59:46.000 the application hasn't changed in weeks, Why am I out of database connections? 879 00:59:46.000 --> 00:59:50.559 That's super weird. Well, the reason you're out of it is because Tony 880 00:59:51.000 --> 00:59:54.800 on the ops team, he got in trouble because the post residence was too 881 00:59:54.880 --> 00:59:58.800 expensive. CFO was mad. They're like, we got to control our costs. 882 00:59:58.960 --> 01:00:00.880 He scaled down the post for instance, and it can't handle as many 883 01:00:00.920 --> 01:00:05.960 connections now. So now this change that happened that's not in your purview as 884 01:00:05.960 --> 01:00:09.760 a developer, and a different git repo has caused an outage on your application, 885 01:00:10.400 --> 01:00:13.519 and you're sitting there trying to figure out like, why why can't my 886 01:00:13.519 --> 01:00:15.519 application pack the postgress? If you don't have access to that, get repo. 887 01:00:15.599 --> 01:00:20.280 You have no idea if you don't have a dashboard for that Postcriss sentstence, 888 01:00:20.280 --> 01:00:22.360 you have no idea. If you haven't seen the cost to change that 889 01:00:22.360 --> 01:00:23.679 post, for instance, you have no idea. Like we don't have agency 890 01:00:23.719 --> 01:00:28.039 as developers through chaos. Right, So now you're sitting there for hours trying 891 01:00:28.039 --> 01:00:31.159 to figure out what it is. You hop in the slack and you're like 892 01:00:31.320 --> 01:00:35.559 at channel like, has anybody changed anything about the data it's And Tony's like, 893 01:00:35.639 --> 01:00:37.000 yeah, yeah, wait, there's some costs of things. It was 894 01:00:37.000 --> 01:00:39.679 sitting at like ten percent CPU all the time, so we just scaled it 895 01:00:39.719 --> 01:00:44.039 down to some T three micros or whatever. And they're like, oh, 896 01:00:44.039 --> 01:00:45.840 there's your problem. Right, And so with mass drivers with the story's a 897 01:00:45.840 --> 01:00:50.519 little different. Is you get your alert, you click on the link, 898 01:00:51.119 --> 01:00:54.519 we take you to your infrastructure diagram. We highlight the thing that is broken, 899 01:00:55.000 --> 01:00:59.119 and from there you can see the metrics and you can see the change 900 01:00:59.159 --> 01:01:01.079 history of everything. It's the dependency of your applications, is your cluster. 901 01:01:01.519 --> 01:01:06.360 You see your databases, you see your cues, and I can see right 902 01:01:06.400 --> 01:01:09.400 there. I'm like, oh, Chris made a change to the database yesterday. 903 01:01:09.559 --> 01:01:14.280 What did he change, you click on the database you visually diff will 904 01:01:14.320 --> 01:01:17.039 show you the difference to actually keeping track of the changes in your terraform Holme 905 01:01:17.039 --> 01:01:21.599 et cetera. It's like, since yesterday, this is what's changed the instance 906 01:01:21.639 --> 01:01:24.559 type. And you're like, there's your problem, and now I can fix 907 01:01:24.599 --> 01:01:28.199 it. From right there, I can go back to bed and in the 908 01:01:28.199 --> 01:01:30.719 morning, I'll call the CFO and be like, that instance is going to 909 01:01:30.719 --> 01:01:35.679 be expensive again, right to my app to deal with that, maybe if 910 01:01:35.679 --> 01:01:37.440 thro a PG bouncement, then the mix, like the ability to like get 911 01:01:37.480 --> 01:01:42.360 through and resolve an issue like this was never like our intention to the platform. 912 01:01:42.360 --> 01:01:45.239 Our attention was like letting people get infrastructure securely and safely. But what's 913 01:01:45.280 --> 01:01:50.320 happened is like it's just much easier to understand what's going on when you're looking 914 01:01:50.320 --> 01:01:52.039 at a diagram, right, Like we all want diagrams. We draw it 915 01:01:52.079 --> 01:01:55.760 on a whiteboard, we go to our desks and we do some crazy shit 916 01:01:55.840 --> 01:01:59.800 and terraform. And then somebody comes along six months later and they're like, 917 01:02:00.159 --> 01:02:06.159 to diagram this, They terraform, visualize it and get that horrendous resource bag, 918 01:02:06.599 --> 01:02:08.079 and then they try to back that out to like a lucid chart or 919 01:02:08.119 --> 01:02:12.559 like cloud Craft, you're like, hey, we've diagrammed it and nobody updates 920 01:02:12.559 --> 01:02:15.119 it for six months, and then six months later somebody someone breaks and you're 921 01:02:15.159 --> 01:02:16.639 like, this isn't what our system looks like at all, Like we want 922 01:02:16.679 --> 01:02:21.239 it to do. Anybody who says we don't want it to be visual, 923 01:02:21.280 --> 01:02:24.039 do you've ever seen touch a whiteboard or cloud craft or lucid charts, You're 924 01:02:24.039 --> 01:02:28.519 a liar, Like, we want it to be visual, but like there's 925 01:02:28.519 --> 01:02:30.760 not a way of doing it. So I think that's what's fundamentally different about 926 01:02:30.800 --> 01:02:32.199 us, Like we don't want to just throw a portal on top of Kubernetes 927 01:02:32.199 --> 01:02:35.760 and be like you can see all your services. Like it's like we wouldn't 928 01:02:35.840 --> 01:02:37.519 change the way that you think about managing it and change the way to think 929 01:02:37.519 --> 01:02:43.760 about troubleshooting it without making you change all of your tooling. Like developer in 930 01:02:43.760 --> 01:02:47.280 this scenario just writes code and your engineering team or your ops team says, 931 01:02:47.320 --> 01:02:52.360 hey, we support on AWS, we support ekas, and LAMB does great. 932 01:02:53.320 --> 01:02:57.639 We have two separate platforms to land the platform in a Kubernetes platform and 933 01:02:57.639 --> 01:03:00.519 mass driver. I use the one that my team needs if I'm on the 934 01:03:00.519 --> 01:03:01.119 back end team and we're like, actually, you know what, I'm making 935 01:03:01.159 --> 01:03:04.360 the service. I don't know that need a full instance. It is very 936 01:03:04.360 --> 01:03:06.599 small. I could be able to run on LAMB, but I don't have 937 01:03:06.639 --> 01:03:09.800 to go and try to get the ops team or the platform team to like 938 01:03:09.840 --> 01:03:13.400 build out this new functionality for me. I just said, we already have 939 01:03:13.480 --> 01:03:15.800 API gateway and Lambda. The front end team uses it for some jam stack 940 01:03:15.800 --> 01:03:20.960 stuff. I'm going to drag this LAMB over, push a small function up 941 01:03:20.960 --> 01:03:22.400 in there, and call it a day, right, Like, like, 942 01:03:22.519 --> 01:03:28.599 why should I have to engage with a product team and spend time trying to 943 01:03:28.599 --> 01:03:31.039 convince them of what my needs are waiting for them to build an MVP survey. 944 01:03:31.119 --> 01:03:34.840 The rest of the team's like I need to get some work done, 945 01:03:35.239 --> 01:03:40.880 right yeah. So yeah, because then your your ops team becomes the bottleneck 946 01:03:42.480 --> 01:03:45.599 and the limiting factor to your business. And that sucks because like I feel 947 01:03:45.639 --> 01:03:51.280 like we're always the bottleneck. Yeah, they're always the bottleneck. I mean 948 01:03:51.880 --> 01:03:53.719 nobody's ever like, oh my gosh, I was going to go ask the 949 01:03:53.800 --> 01:03:57.719 ops team for something and they already imagined it and did it before I got 950 01:03:58.679 --> 01:04:00.920 No matter what even in the best scenario, you're like you're waiting a bit, 951 01:04:00.079 --> 01:04:02.800 like we're always a little bit of bottle. You bring us in early 952 01:04:03.119 --> 01:04:05.480 on a project and you're like, Okay, we're working on this project. 953 01:04:05.480 --> 01:04:09.400 We're we're trying to figure out what the cloud infrastructure is. It's like you 954 01:04:09.400 --> 01:04:12.119 can do that, But I also feel like when you're very very early in 955 01:04:12.119 --> 01:04:14.679 a project talking about what it could be, like you don't know how the 956 01:04:14.719 --> 01:04:17.440 software and requirements you're going to change, right, and so like the end 957 01:04:17.480 --> 01:04:20.880 of the day, in reality, it's like any given terraform resource or collection 958 01:04:20.920 --> 01:04:24.639 of resources isn't months of work for us, right, It's like trying to 959 01:04:24.639 --> 01:04:28.239 figure out what you need, what abstraction to expose to you, and then 960 01:04:28.239 --> 01:04:30.960 I cobble up some terraform, put in some practices, right my OPA rules, 961 01:04:30.960 --> 01:04:33.719 and like off we go, Like I don't need three months to do 962 01:04:33.760 --> 01:04:36.679 that for a couple of services you're going to use, So like when is 963 01:04:36.719 --> 01:04:40.599 the right time to bring me in? Probably like some percent of the way 964 01:04:40.599 --> 01:04:43.679 through the project, Like once you have something like viable running, maybe you're 965 01:04:43.719 --> 01:04:47.639 using local stack to like figure it out locally, or you're using mass but 966 01:04:49.000 --> 01:04:53.440 as you can, but so like like when do you bring us in, 967 01:04:53.480 --> 01:04:56.039 Like we're always a little bit of a bottleneck, and the reality is is 968 01:04:56.079 --> 01:04:59.920 like maybe Google or not because there's ten thousand OPS engineers, but like the 969 01:05:00.079 --> 01:05:02.800 average company where you're out ratio, I think we say like one to seven 970 01:05:02.920 --> 01:05:08.400 is like the good ratio of OPS to engineers. There's not one to seven 971 01:05:08.440 --> 01:05:12.280 on the planet. Yeah, but like if you look at the if you 972 01:05:12.280 --> 01:05:15.440 look at the stack Overflow survey and you look at the numbers, they're like 973 01:05:15.360 --> 01:05:19.519 it's it's not there. Like if we have twenty seven million software developers, 974 01:05:19.599 --> 01:05:24.039 we do not have two points. Sorry I'm not good enough, but we 975 01:05:24.039 --> 01:05:28.760 don't have seven million operations engineers, right, Like, there's not that many 976 01:05:28.840 --> 01:05:30.960 roles. There is that many roles and they're all sitting open, right Like 977 01:05:31.039 --> 01:05:33.400 we have one of the longest roles that sit open. We also have one 978 01:05:33.400 --> 01:05:35.840 of the shortest tenures. Why is that? Why does it take forever to 979 01:05:35.880 --> 01:05:38.840 place with us? And we quit all the time. We're also one of 980 01:05:38.880 --> 01:05:42.039 the highest paid people on the team. It's because we bounce those jobs constantly, 981 01:05:42.039 --> 01:05:45.559 get those raiss Like we are always going to be a bottleneck until we 982 01:05:45.639 --> 01:05:50.480 start doing something that helps us scale, and even in the platform engineering scenario, 983 01:05:50.599 --> 01:05:54.760 like we still are a bottleneck when somebody starts to move outside of the 984 01:05:54.760 --> 01:05:57.760 platform, right, right, we get that good balance of like the platform, 985 01:05:57.760 --> 01:06:00.599 like I'm never a bottleneck to you because the platform starves what you need 986 01:06:00.719 --> 01:06:03.760 and then when you kind of go outside of like the norm, you go 987 01:06:03.800 --> 01:06:09.320 off that golden path. Now we might be a bottle that now we're talking 988 01:06:09.320 --> 01:06:11.320 about being a product. Okay, great, Like what do you need, 989 01:06:11.400 --> 01:06:14.199 Like, how can we do this for you better? Okay, great? 990 01:06:14.199 --> 01:06:18.719 We need to support ECS because there's some ECS task integration with redshift or something. 991 01:06:18.760 --> 01:06:20.719 I don't know. I don't think that actually tips Like let's say there's 992 01:06:20.719 --> 01:06:24.559 like a very specific use case like okay, cool, let's put together a 993 01:06:24.639 --> 01:06:28.800 real proof of concept terraform module that has ECS. I'll push it up to 994 01:06:28.880 --> 01:06:30.320 you. You can throw it onto your canvas, connect it to the redshift 995 01:06:30.320 --> 01:06:34.559 module we already have, and like, let's let's iterate on this and get 996 01:06:34.559 --> 01:06:38.840 it good enough so that we can have like market like production quality for any 997 01:06:38.840 --> 01:06:44.159 other team that might need to use in ECS, right, And I think 998 01:06:44.159 --> 01:06:48.320 that gets into the social aspect of DevOps too, Like if you can socialize 999 01:06:48.400 --> 01:06:51.519 to the teams that you support, these are the products, like, these 1000 01:06:51.519 --> 01:06:58.760 are the platforms that we've built for you to use, and you can like 1001 01:06:58.920 --> 01:07:03.639 define the rules ving engagement. Then they know when they're venturing outside of those 1002 01:07:03.719 --> 01:07:06.639 rules engagement of engagement, and then they need to come back and talk to 1003 01:07:06.679 --> 01:07:14.840 you about that. But without doing that, the engineering teams see DevOps as 1004 01:07:15.039 --> 01:07:18.880 everything being one off. They're like, oh, I'm going to deploy this 1005 01:07:18.960 --> 01:07:24.239 thing. I gotta go talk to the DevOps guys, and it's such a 1006 01:07:24.239 --> 01:07:27.800 painful process that they wait till the last minute because they just don't want to 1007 01:07:27.840 --> 01:07:32.920 deal with our bullshit. And so I think, just like creating that catalog 1008 01:07:33.039 --> 01:07:36.440 for lack of a better term, like the here's the things you can choose 1009 01:07:36.440 --> 01:07:41.639 from the catalog, and we do custom orders. We just need a little 1010 01:07:41.719 --> 01:07:45.119 lead time. But if you can like get get them to that point where 1011 01:07:45.119 --> 01:07:49.199 they feel like they understand what you do, then you can socialize that conversation. 1012 01:07:50.119 --> 01:07:54.559 And I feel like those conversations are always hard, right because like I 1013 01:07:54.559 --> 01:07:59.880 feel like a lot of people again like maybe not everybody that's listening, like 1014 01:08:00.760 --> 01:08:03.679 has this experience, but like when you go to an ops person, whether 1015 01:08:03.679 --> 01:08:05.840 it's a ticket or you walk over to their desk and like, you need 1016 01:08:05.880 --> 01:08:11.920 something that's like out of the ordinary, right, Like let's let's let's imagine 1017 01:08:11.920 --> 01:08:14.840 that you had this magic tool where you can actually find all your terraform modules 1018 01:08:14.840 --> 01:08:16.239 that your company supports, right, and you go to that, you go 1019 01:08:16.279 --> 01:08:20.119 to that repo and you're like, ah, like Mango at list isn't there. 1020 01:08:20.239 --> 01:08:25.760 We need Mongo outists, We need a document store. Maybe they don't 1021 01:08:25.760 --> 01:08:29.279 have maybe they don't have postcrests and they don't realize that you just use postcrusts. 1022 01:08:29.279 --> 01:08:33.039 Sorry, I need Mongo at lists but it's not here. Now, 1023 01:08:33.079 --> 01:08:35.720 I got to go talk to the ops team. Ticket got to figure it 1024 01:08:35.720 --> 01:08:40.119 out myself. Maybe if that's our if that's our culture, and that's fine. 1025 01:08:40.279 --> 01:08:42.199 But you go to the ops person you say, hey, like we 1026 01:08:42.239 --> 01:08:45.279 need Mongo out lists with a new project, and it's like we do you 1027 01:08:45.319 --> 01:08:47.239 need Mongo at lists because they send you some marketing materials? Or do you 1028 01:08:47.279 --> 01:08:50.039 need mongo because we can throw Mango and a container for you? Like what 1029 01:08:50.039 --> 01:08:54.479 do you like? Why do you need at lists? And also like important 1030 01:08:54.520 --> 01:08:59.520 question like why not why not document dB? Yeah? In aws? Right, 1031 01:08:59.600 --> 01:09:02.840 like and the catches is like it's really funny. Like people anyone out 1032 01:09:02.840 --> 01:09:04.680 there is use document dB. I know, I know, I know, 1033 01:09:04.800 --> 01:09:09.960 chill out. I'm trying to make an example here. I'm I'm not a 1034 01:09:09.960 --> 01:09:14.159 document dB fanboy, but like reality is like there's a like document dB like 1035 01:09:14.239 --> 01:09:18.119 can have a pretty similar experience to Mongo Like there are places that like obviously 1036 01:09:18.159 --> 01:09:21.640 Mango's going to move faster in feature development and like the Mango space, yes, 1037 01:09:21.720 --> 01:09:25.760 but if you're not using you're just using some basic document store, like 1038 01:09:25.800 --> 01:09:30.479 you might not need everything that Mango has. And if it is, I 1039 01:09:30.479 --> 01:09:32.399 think Mango even publishes like a last time I checked, it was like either 1040 01:09:32.479 --> 01:09:36.760 like a list or tool that could show like what document dv didn't support. 1041 01:09:38.520 --> 01:09:41.239 Right, They're like, hey, like use us because this doesn't do this 1042 01:09:41.279 --> 01:09:44.279 one thing. It's like you don't need that operation now and you can't see 1043 01:09:44.319 --> 01:09:48.680 the chance you need that operation or configuration option in the future. Does it 1044 01:09:48.760 --> 01:09:55.640 benefit us to spend the money to have an enterprise account or whatever Mango atlas 1045 01:09:55.720 --> 01:09:58.760 where we'll bind to our vpc. We want to go through the effort of 1046 01:09:58.880 --> 01:10:01.840 peering our VPC with Mongo aut lists, Like do we want to go over 1047 01:10:01.840 --> 01:10:04.520 the public internet to hit them. Like there's a lot of questions in the 1048 01:10:04.520 --> 01:10:08.520 mind, and so like the gut reaction from the person is like why do 1049 01:10:08.560 --> 01:10:12.439 you need why do you need Mongo Outlets, which like it's very hard to 1050 01:10:12.560 --> 01:10:16.159 ask that question without sounding like you're criticizing the person that's made that decision, 1051 01:10:16.239 --> 01:10:20.119 right, And it's like, I'm not saying you're wrong, I'm just trying 1052 01:10:20.119 --> 01:10:23.439 to understand, Like I don't know if you know all the options that are 1053 01:10:23.439 --> 01:10:25.920 available, and I'm trying to figure out, like what is the right service 1054 01:10:25.960 --> 01:10:29.239 for us? Right, so, like can we can we try document dB 1055 01:10:29.399 --> 01:10:30.960 so it doesn't create a ton of work for us because it's if you need 1056 01:10:30.960 --> 01:10:34.760 document dB, I can throw together that terriform module. I'll skip lunch if 1057 01:10:34.760 --> 01:10:39.399 you do this favor. But if you need Mongo Atlas, when we have 1058 01:10:39.439 --> 01:10:43.760 to go sign to enterprise account, talk to procurement and do some VPC bindings, 1059 01:10:44.079 --> 01:10:46.640 maybe we have maybe we've made the dumb choice of having you know, 1060 01:10:46.760 --> 01:10:50.079 that that side or something that that makes it impossible for us to buy into 1061 01:10:50.119 --> 01:10:54.119 other ords it to appear to other people's networks, right, you don't know, 1062 01:10:54.279 --> 01:10:58.720 Like you don't know like how well or chaotic our VPC is, right, 1063 01:10:58.800 --> 01:11:00.279 So like there's there's other things that come to mind when we start talking 1064 01:11:00.319 --> 01:11:04.880 about using these. They says services that sit outside of AWS, and so 1065 01:11:04.960 --> 01:11:11.159 I feel like, like that that's just like a problem that exists like in 1066 01:11:11.239 --> 01:11:14.960 our in our collaboration, and I feel like that's another thing that like we 1067 01:11:15.039 --> 01:11:17.760 just need to be able to simplify, right, like being able to say, 1068 01:11:17.880 --> 01:11:20.760 like, these are the things that we support, you can quickly find 1069 01:11:20.800 --> 01:11:26.840 them, right, and be able to get new services into people's hands quickly 1070 01:11:27.119 --> 01:11:29.840 even to experiment. But like how do we get the time for that? 1071 01:11:30.119 --> 01:11:32.640 Right? Like I've had cases where people have asked or something like I could 1072 01:11:32.640 --> 01:11:38.760 throw together a document dB. Like this is a real scenario. I'm parroting 1073 01:11:38.760 --> 01:11:42.199 from a previous job. It's like I could have put a module together for 1074 01:11:42.239 --> 01:11:45.720 them in maybe an hour or two for them to try out document dB to 1075 01:11:45.720 --> 01:11:50.159 see if it's good enough. But I had too much work to do, 1076 01:11:50.680 --> 01:11:56.159 Like I didn't have enough time to do a little experiment to see if I 1077 01:11:56.159 --> 01:11:59.199 could save myself a bunch of time, a bunch of time, right, 1078 01:11:59.239 --> 01:12:00.079 And so it's just like okky, like they need this thing, and it 1079 01:12:00.119 --> 01:12:03.199 always seems like they needed now right. It's like, oh, like why 1080 01:12:03.199 --> 01:12:04.880 didn't this come up. It's like, well, we didn't know we need 1081 01:12:04.880 --> 01:12:09.399 document databases when we started the project. You were at that meeting. I 1082 01:12:09.520 --> 01:12:12.399 was at that meeting. I wasn't at subsequent ones because like there wasn't anything 1083 01:12:12.439 --> 01:12:15.760 that was needed for me or using postgrass got to the point where like somebody's 1084 01:12:15.760 --> 01:12:18.199 like, oh, we actually need Mango for this, and someone brew installed 1085 01:12:18.239 --> 01:12:21.680 Mango or they added it to their doctor composed file and they didn't think, 1086 01:12:21.760 --> 01:12:25.439 oh shit, somebody else over there is going to have to do work right, 1087 01:12:25.479 --> 01:12:28.119 So now I'm a bottleneck again. It's like I'm not a bottleneck. 1088 01:12:28.119 --> 01:12:30.800 It's that whole like, you know, lack of preparation on your part. 1089 01:12:30.840 --> 01:12:33.800 It's not an emergency on my part. Like an asshole for saying that, 1090 01:12:33.960 --> 01:12:36.880 right and sure, so like that's the thing that just kind of sucks, 1091 01:12:36.920 --> 01:12:45.279 like that collaboration is really really hard to do well for most organizations. Can 1092 01:12:45.319 --> 01:12:48.199 we jump to that thing that we were talking about earlier before the call started? 1093 01:12:48.239 --> 01:12:50.000 I want to share this story. Can we talk about do we have 1094 01:12:50.039 --> 01:12:54.159 time to talk about some IT stuff some some IT help est data stuff. 1095 01:12:54.560 --> 01:12:58.439 Yeah for sure, because we both we both started our careers in the help 1096 01:12:58.520 --> 01:13:01.840 desk. So help desk corse stories. Put take one, go for it 1097 01:13:02.239 --> 01:13:05.399 the one I wanted to share with you earlier. So my career. I 1098 01:13:05.439 --> 01:13:10.479 was originally a physics and an architecture major, but I was working at a 1099 01:13:10.520 --> 01:13:14.520 hospital and I was working in data entry. I'd been doing software development. 1100 01:13:14.640 --> 01:13:16.439 Was like fifteen or sixteen. I wrote a little program that did my job. 1101 01:13:16.840 --> 01:13:19.760 I did my data entry, so I just like read from Excel docs 1102 01:13:19.800 --> 01:13:27.600 and like input them into this like terminal system used for explanation of benefits for 1103 01:13:27.720 --> 01:13:32.920 the hospital I worked at. And when I automated my job, I wasn't 1104 01:13:32.920 --> 01:13:35.520 a software engineer, so I didn't sign like a PIA forum. I didn't. 1105 01:13:35.560 --> 01:13:39.000 They did not have rights to my software. I wrote it at home 1106 01:13:39.119 --> 01:13:41.840 like it was mine and say buy it from me, which is exciting. 1107 01:13:42.439 --> 01:13:46.079 So how I got into help desk was they offered me a thousand dollars for 1108 01:13:46.199 --> 01:13:49.439 my little program and a job on the help desk team. And I was 1109 01:13:49.479 --> 01:13:54.119 like, what, dude, pay me nine dollars an hour and give me 1110 01:13:54.159 --> 01:13:57.920 one thousand bucks. I'm eighteen. I'm like I'm rich. I literally I 1111 01:13:57.920 --> 01:14:00.239 took that thousand dollars, Like did you not I put a down payment on 1112 01:14:00.239 --> 01:14:04.640 a wave Runner like I was imagine I didn't have a place to park it. 1113 01:14:04.720 --> 01:14:08.199 I did not think this through. I lived in a dorm. I'm 1114 01:14:08.199 --> 01:14:11.319 like, what the hell am I gonna do with this wave Runner? But 1115 01:14:11.439 --> 01:14:15.520 I was just so stoked. I was on the help desk team, and 1116 01:14:15.640 --> 01:14:20.239 wow, like help desk is like we give those people so much shit, 1117 01:14:21.199 --> 01:14:25.479 Like I think of every support person at police. It's like it's so hard 1118 01:14:25.520 --> 01:14:30.239 not to be like especially like especially calling help deskers anything technical as a software 1119 01:14:30.239 --> 01:14:33.399 engineer, like yes, I know, I understand how to log and just 1120 01:14:33.520 --> 01:14:36.760 yeah, I know, like right, like and so it's frustrating. But 1121 01:14:36.840 --> 01:14:42.239 like those those like we're frustrated with them, but like their job is really 1122 01:14:42.279 --> 01:14:45.399 hard, like they do run into a bunch of people that are just not 1123 01:14:45.439 --> 01:14:47.600 familiar with computers all day long, and so they have to assume like they're 1124 01:14:47.720 --> 01:14:53.279 meeting more people like that than people that are right. Right, we had 1125 01:14:53.840 --> 01:15:00.239 seventeen offices and my second or seventeen offices across Florida, and so like, 1126 01:15:00.279 --> 01:15:04.119 aren't my my domain was pretty big, like to drive places all the time, 1127 01:15:04.640 --> 01:15:08.039 I had a couple of like data centers, like it was the first 1128 01:15:08.079 --> 01:15:10.239 one was in the building I was in. We have second one up in 1129 01:15:10.319 --> 01:15:15.640 uh, like Zephyr Hills, and so I'm in Fort Myersporters. So anybody 1130 01:15:15.680 --> 01:15:18.119 in Florida, Fort Myers or Zephyr Hills, imagine driving this distance for help 1131 01:15:18.159 --> 01:15:23.199 desk golf. So next time out that person there talking to you like you're 1132 01:15:23.239 --> 01:15:26.560 dumb, like this is this is their life. This is a multi hour 1133 01:15:26.640 --> 01:15:30.479 drive. And effectively what has happened is like there, gosh, I can't 1134 01:15:30.479 --> 01:15:34.359 remember what it was. Something was frozen on their on their computer, and 1135 01:15:34.359 --> 01:15:38.279 every time they rebooted it was frozen, and it was just like okay, 1136 01:15:38.840 --> 01:15:41.439 that's a heart and the troubleshoot, like I can't even like remote desk into 1137 01:15:41.479 --> 01:15:45.479 it, like because they rebooted it. And this is, by the way, 1138 01:15:45.479 --> 01:15:48.439 this is two thousand one or two thousand and two, so we didn't 1139 01:15:48.439 --> 01:15:54.319 have the great tools we have today. So I drive two hours to Zephyr 1140 01:15:54.399 --> 01:15:56.880 Hills. I go into the office and like, show me this computer, 1141 01:15:57.119 --> 01:16:00.039 and sure shit, I sit down at it the screen, I grabbed the 1142 01:16:00.039 --> 01:16:05.560 mouse and go to click to log in. The mouse I hit the tab 1143 01:16:05.600 --> 01:16:09.479 button and it tabs into the username field and I'm like, it's not frozen. 1144 01:16:09.479 --> 01:16:11.680 That's interesting. So that I put in like one of the AVON passers. 1145 01:16:11.680 --> 01:16:14.199 I log in and I'm like, okay, I grab the mouse and 1146 01:16:14.239 --> 01:16:18.359 move it again. Nothing and I'm like. The first thing I do is 1147 01:16:18.399 --> 01:16:24.800 I look under there's this big tower. The mouse is unplugged. My ve 1148 01:16:25.079 --> 01:16:28.279 driven two hours for an unplugged mouse, and I have to drive two hours 1149 01:16:28.319 --> 01:16:32.960 home. And I looked down and I'm just like and so I plug it 1150 01:16:33.000 --> 01:16:35.439 back in. It works, And she was like, how did you fix 1151 01:16:35.479 --> 01:16:40.079 it? I'm like, the mouse, the mouse, and this is this 1152 01:16:40.119 --> 01:16:43.239 isn't like a USB mouse is like one of those old like DIN connector things 1153 01:16:43.239 --> 01:16:46.880 like real hard to yank this thing out. So what I was able to 1154 01:16:46.880 --> 01:16:50.319 tell from looking at it as well was it was pulled out with some force. 1155 01:16:50.399 --> 01:16:56.199 So the assumption is it got wrapped around her rank ankle and pull out 1156 01:16:56.199 --> 01:16:59.399 of the machine some of those sliding things right here, but it slid out 1157 01:16:59.399 --> 01:17:01.800 with like the keyboard right in the mouse right like there was some serious force 1158 01:17:01.880 --> 01:17:04.600 pulled this thing out. Someone must have known, like I feel like a 1159 01:17:04.640 --> 01:17:09.600 trick must have happened. But yeah, that was that was one of my 1160 01:17:09.960 --> 01:17:14.399 that's one of my favorite ones. That was a little fun. My favorite 1161 01:17:14.439 --> 01:17:17.479 one that was not fun. This is second second. I ended up kind 1162 01:17:17.520 --> 01:17:21.960 of focusing in healthcare once I changed careers. So my master's is in healthcare 1163 01:17:23.000 --> 01:17:27.159 information system, so work for another company. And we had these two offices 1164 01:17:29.039 --> 01:17:31.960 that were gosh, I don't want to give away, Well, I guess 1165 01:17:31.920 --> 01:17:35.720 I have to give it away. These two offices that we're connected to the 1166 01:17:35.920 --> 01:17:40.399 data center here and literally across the street was one of our heart surgery centers. 1167 01:17:41.000 --> 01:17:44.760 And this is two thousand and three or so, and like we're pretty 1168 01:17:44.800 --> 01:17:47.119 far along, Like we've got Wi Fi all over the place. We've got 1169 01:17:47.159 --> 01:17:51.439 a like a digital pills system for dispersing like pills in our pharmacy. Like 1170 01:17:51.479 --> 01:17:56.800 we've heavily invested in technology. We had a really great engineering budget. We 1171 01:17:56.840 --> 01:18:00.359 had badges blogging the machines, which is a nightmare for him. We got 1172 01:18:00.359 --> 01:18:04.920 to find so much from doctors just leaving their batch's places by like a hip 1173 01:18:04.920 --> 01:18:10.680 a shopper who came in and like the doctor left the card and they logged 1174 01:18:10.680 --> 01:18:13.520 in and just accessed a bunch of shit. It's just like kill me. 1175 01:18:14.439 --> 01:18:17.319 But my boss was, this is like in the days when you could haze 1176 01:18:17.520 --> 01:18:19.800 new engineers. I don't know if you ever do you ever get hazed? 1177 01:18:20.880 --> 01:18:27.239 Yeah time or two? Yeah, progressing from help desk into like the network 1178 01:18:27.239 --> 01:18:30.279 operations, like working on data center. And so we had these two sites 1179 01:18:30.640 --> 01:18:33.439 and it was going to cost us something insane like one hundred and twenty thousand 1180 01:18:33.520 --> 01:18:41.079 dollars or something wild to run internet like under the road to our other office. 1181 01:18:41.560 --> 01:18:44.239 And we didn't want them like using dial up or anything like that to 1182 01:18:44.319 --> 01:18:46.159 like dial in heart surgery center, Like we need this thing to be online. 1183 01:18:46.640 --> 01:18:50.159 And so we did is we installed microwave dishes on the roofs pointed at 1184 01:18:50.159 --> 01:18:55.199 each other, and we had point to point microwave internet between between these two 1185 01:18:55.520 --> 01:19:00.279 offices. And they started having outages and they were very intermittent. They're very 1186 01:19:00.399 --> 01:19:03.039 random, and they're very brief, but like brief enough that it would like 1187 01:19:03.119 --> 01:19:06.840 stop like an upload or you know, like some sort of scan upload they're 1188 01:19:06.840 --> 01:19:12.279 scanning back to like our medical record system. And so we thought it was 1189 01:19:12.319 --> 01:19:15.359 birds. We thought birds had gotten on the on the things. So my 1190 01:19:15.399 --> 01:19:17.600 boss is like, like, we we like how we can fix this. 1191 01:19:17.600 --> 01:19:19.960 I'm gonna throw some of those little bird things on there, like stop them 1192 01:19:19.960 --> 01:19:25.479 fronts. I don't know what putting those on a microwave dish actually would have 1193 01:19:25.520 --> 01:19:28.039 done. And he's like, what's confirm that birds? First? I'm like, 1194 01:19:29.520 --> 01:19:32.640 what are we to ask them? Like? He's like uh no. 1195 01:19:33.000 --> 01:19:34.399 And I was like, okay, well, what are you gonna do? 1196 01:19:34.600 --> 01:19:38.680 And he's like come here and he walks out to his truck and I should 1197 01:19:38.720 --> 01:19:40.840 you not. He's got one of those umbrellas with a stand on it, 1198 01:19:41.560 --> 01:19:45.600 a lawn chair, and then he had to walking talkies and I'm like, 1199 01:19:45.399 --> 01:19:50.239 he's like, you're gonna this is Florida the summer. He's like, you're 1200 01:19:50.239 --> 01:19:55.000 gonna sit on the roof and when they have an outage, I'm gonna call 1201 01:19:55.000 --> 01:20:00.359 you on the walkie talkie and you're gonna tell me what you see. Plus 1202 01:20:00.680 --> 01:20:02.119 So I'm sitting up there for like six hours and all of a sudden, 1203 01:20:02.680 --> 01:20:08.359 walkie talkie, hey h surgery centers down what's going on? And I'm looking 1204 01:20:08.399 --> 01:20:11.520 at I'm looking at the thing and I'm like, dude, there's no birds 1205 01:20:11.560 --> 01:20:14.119 on this thing. Like I'm looking at like there's there's no birds like I 1206 01:20:14.159 --> 01:20:15.680 don't know why. I still there's no birds, and so like out my 1207 01:20:15.720 --> 01:20:19.039 little like fox hound thing, I can still see that there's like signal going 1208 01:20:19.079 --> 01:20:25.239 through the wire. There's just out and then all of a sudden it's back 1209 01:20:25.319 --> 01:20:30.119 up, and I'm like he's back up. I'm like, I'm stumped, 1210 01:20:30.479 --> 01:20:33.479 Like I'm stump, like this was not in the net plus book. This 1211 01:20:33.640 --> 01:20:36.359 wasn't in the A plus book, Like I got it, I got an 1212 01:20:36.479 --> 01:20:41.239 MC. I see this was not on the test right like, And so 1213 01:20:41.279 --> 01:20:44.039 I'm sitting there for like a while, and maybe like two or three hours 1214 01:20:44.119 --> 01:20:46.439 later, goes out again, and I'm like, maybe it's a burn on 1215 01:20:46.479 --> 01:20:49.560 the other side, Like maybe it's a burn on the receiver. That didn't 1216 01:20:49.600 --> 01:20:51.319 even think about this. I don't know if everybody else is like, it's 1217 01:20:51.319 --> 01:20:56.279 a burn on the other receiver, you idiot. And so I look across 1218 01:20:56.319 --> 01:21:00.159 the street and I don't see the satellite, but I see as a semi 1219 01:21:00.199 --> 01:21:06.800 truck parked on a restante. Like the other building was a house that was 1220 01:21:06.800 --> 01:21:12.279 converted into a surgery center, and we were in this essentially this strip mall 1221 01:21:12.640 --> 01:21:16.239 that had been converted into our data center and like our primary care physician's office. 1222 01:21:16.279 --> 01:21:19.479 So this thing's like thirty forty feet tall, and this house is you 1223 01:21:19.479 --> 01:21:23.600 know however tall houses, and so the beams just hitting the side of this 1224 01:21:23.640 --> 01:21:31.520 truck probably packed it tons of stuff, and so with longer polls. Yeah, 1225 01:21:31.560 --> 01:21:35.039 meanwhile, the driver who had frozen pizzas in his truck shows up with 1226 01:21:35.079 --> 01:21:40.640 fully cooked pizzas at the destination of just like blast and I don't I don't 1227 01:21:40.680 --> 01:21:44.920 even remember how faster that was with was the one hundred megabit I think back 1228 01:21:44.960 --> 01:21:46.840 then for like a network. I don't know what. I don't know what 1229 01:21:46.920 --> 01:21:50.279 the satellite, a little satellite this thing did. But yeah, he's like 1230 01:21:50.319 --> 01:21:56.840 I got fourteen cooked pizzas and a brain tumor or anything else, uh helped 1231 01:21:56.880 --> 01:21:58.920 us. I don't know, Like I kind of want maybe when I retire, 1232 01:21:58.920 --> 01:22:00.640 I'll just go back and work in it for a year where it's like 1233 01:22:00.720 --> 01:22:02.880 it's probably a lot less stressful when you don't need to do it to pay 1234 01:22:02.920 --> 01:22:06.640 your bills, Like I bet it becomes pretty fun. Then it's just like, 1235 01:22:06.680 --> 01:22:11.239 okay, like what kind of weird shit do y' all have for me 1236 01:22:11.359 --> 01:22:17.000 today? Right to see how far you can push the envelope? Oh my 1237 01:22:17.039 --> 01:22:20.319 gosh, but do you have a good help desk color story. Yeah. 1238 01:22:20.399 --> 01:22:30.279 So the first help desk job I had was working for a manufacturing company that 1239 01:22:30.319 --> 01:22:35.359 had a huge sales department selling their product. And you know, this was 1240 01:22:36.279 --> 01:22:49.640 this was like nine two thousand era, and so they the salespeople were just 1241 01:22:49.800 --> 01:22:55.000 coming out of the stage of having their roll of decks where they would pick 1242 01:22:55.039 --> 01:22:58.479 up the phone and dial and then write down in a notebook what they were 1243 01:22:58.479 --> 01:23:01.479 doing. So they were trying to get them to you CRM software, and 1244 01:23:01.760 --> 01:23:05.520 at the same time they had to build computer skills, you know. But 1245 01:23:05.640 --> 01:23:09.800 I get this call from this sales guy. He's like, hey, I'm 1246 01:23:09.800 --> 01:23:14.439 out of disk base. I can't save anything on my computer. And I 1247 01:23:14.520 --> 01:23:16.640 had the ability to check their disk base and I looked and I'm like, 1248 01:23:16.720 --> 01:23:19.800 now you've got You've got plenty of disk space. He's like, look, 1249 01:23:19.800 --> 01:23:24.560 I'm telling you I cannot save anything on my computer. I don't know what 1250 01:23:24.600 --> 01:23:28.760 your little, your little magic thing down there does, but it's wrong because 1251 01:23:28.760 --> 01:23:32.319 I'm telling you I cannot save anything on my computer. Like, all right, 1252 01:23:32.680 --> 01:23:34.920 I'll come up. So I'll go up there, up on the third 1253 01:23:34.920 --> 01:23:40.239 floor. I go up there, find him in this sea of cubicles, 1254 01:23:40.560 --> 01:23:44.680 like, Okay, what's going on? And he's like look, and he 1255 01:23:44.760 --> 01:23:50.079 shows me his desktop, and on his desktop he had icons for every saved 1256 01:23:50.520 --> 01:23:55.319 document he had sent to a customer, lined up from left to right, 1257 01:23:57.239 --> 01:24:00.359 and there was not room to put another icon there. He's like, I 1258 01:24:00.399 --> 01:24:02.359 can't save it. There's no place to put it. Yeah, he was, 1259 01:24:02.479 --> 01:24:05.279 he was. He was geographically out of space, That's what he was 1260 01:24:05.279 --> 01:24:09.399 trying to say. Right, I was writing down what I thought it was, 1261 01:24:09.439 --> 01:24:14.479 and that was not it. You guess. I was guessing he had 1262 01:24:14.479 --> 01:24:15.840 the hard drive you're seeing that, but he was trying to save something to 1263 01:24:15.880 --> 01:24:24.119 a floppy diss Yeah, it was too big, but it definitely wasn't the 1264 01:24:24.159 --> 01:24:31.920 grid on Windows ninety eight. Oh my gosh. It is fun. It 1265 01:24:32.039 --> 01:24:35.039 is fun, but it is also frustrating. So be nice to your support 1266 01:24:35.039 --> 01:24:42.279 people. Yeah. Absolutely, Like with a grain of salt, understand like 1267 01:24:43.159 --> 01:24:45.319 who they might have been talking to just before they got on the phone with 1268 01:24:45.359 --> 01:24:48.319 you. It's so hard. It's hard, like too when it's just like, 1269 01:24:48.399 --> 01:24:53.600 oh man, like like making it. I always felt bad when somebody's 1270 01:24:53.640 --> 01:24:55.960 like I feel dumb, and it's like, I mean, like I don't 1271 01:24:56.000 --> 01:24:59.239 know I'd feel pretty dumb if I was doing open heart surgery, like I'd 1272 01:24:59.279 --> 01:25:04.640 fucked out, like like you shouldn't have to know. But uh, oh 1273 01:25:04.720 --> 01:25:11.680 gosh, that's funny. Cool. Well, Corey, it's been great having 1274 01:25:11.720 --> 01:25:14.680 you on the show Man. This has been a blast. I have had 1275 01:25:14.680 --> 01:25:18.039 a great time. This is super fun cool. Well, I'm happy to 1276 01:25:18.039 --> 01:25:23.640 have you back on anytime, because I'm sure we've got more stories, especially 1277 01:25:23.760 --> 01:25:28.760 healthcare stories. We could go down the healthcare path. I worked as as 1278 01:25:28.800 --> 01:25:34.680 a support operations engineer for a teleradiology company, and there's just so many stories 1279 01:25:34.720 --> 01:25:39.359 coming out of that that are just too funny. And I think it's been 1280 01:25:39.439 --> 01:25:45.479 long enough now where like any like anything I could get sued for saying has 1281 01:25:45.560 --> 01:25:50.119 past, so I could tell those stories now. Oh my gosh, I 1282 01:25:50.159 --> 01:25:56.720 have I have one that can't be told on camera. That is right mind 1283 01:25:56.800 --> 01:25:59.239 boggling. I don't know if, like if you had this experience. But 1284 01:25:59.359 --> 01:26:02.399 doctors, doctors all think they're God, or at least most of the ones 1285 01:26:02.399 --> 01:26:06.039 that I met, Yeah, when they're doctors that like own like that. 1286 01:26:06.159 --> 01:26:10.359 I work with a big practice and the doctors were all like board members and 1287 01:26:10.399 --> 01:26:14.680 shareholders of the practice, like ooh, that ego is gets big, but 1288 01:26:14.760 --> 01:26:18.159 we had one where his ego was big and his sense of humor was brutally 1289 01:26:18.239 --> 01:26:23.800 messed up. Yeah I'll hear that story once, but not streaming live anymore 1290 01:26:23.840 --> 01:26:29.880 because it is not it is beyond rated. Are right on, And that's 1291 01:26:29.920 --> 01:26:32.680 a segue for our new podcast, Things we can't say on the air, 1292 01:26:32.840 --> 01:26:40.319 So, oh my gosh, listen. No, sincerely, thank you. 1293 01:26:40.479 --> 01:26:43.039 I've had a blast. I appreciate you coming on the show, and thanks 1294 01:26:43.039 --> 01:26:45.439 to everyone who hung out for us this entire time. I hope it was 1295 01:26:45.560 --> 01:26:49.079 helpful, entertaining, and I hope to see you all next week. Yeah, 1296 01:26:49.119 --> 01:26:51.600 if it wasn't helpful, I really hope it was entertaining. That's like 1297 01:26:51.600 --> 01:26:58.239 it was over in life and life. But they're always competing, and yeah, 1298 01:26:58.279 --> 01:26:59.960 if you could just hit one of them every once in a while, 1299 01:27:00.039 --> 01:27:06.359 all that's that's winning. If I give somebody a ball, alright, alright, 1300 01:27:06.399 --> 01:27:11.479 see everyone, Hm