WEBVTT 1 00:00:05.120 --> 00:00:08.960 Okay, what's going on? Everybody? Welcome to another episode of Adventures in 2 00:00:09.240 --> 00:00:14.240 DevOps. I'm your host, Will Button and back after attending it was the 3 00:00:14.320 --> 00:00:18.920 DevOps conference in Zurich last week, right Warren? Yeah for sure. And 4 00:00:19.039 --> 00:00:23.280 you know, I actually listened to a little bit of last episode and I 5 00:00:23.320 --> 00:00:26.640 felt like you introduced me even though I wasn't there. So I feel like 6 00:00:26.679 --> 00:00:30.280 I have a free you know, I don't need to introduce myself this episode. 7 00:00:32.759 --> 00:00:37.679 I will say that I owe everyone an interesting fact. And this time 8 00:00:37.719 --> 00:00:44.359 when we were at the conference, we specifically went out and hosted an open 9 00:00:44.359 --> 00:00:50.000 space where I talked with a group of experienced develops practitioners and we were really 10 00:00:50.039 --> 00:00:53.039 trying to define what a platform even is. I think this word is thrown 11 00:00:53.079 --> 00:00:58.200 away around everywhere, and after an hour we really came to a great conclusion. 12 00:00:58.719 --> 00:01:06.879 It's something horrioral and that's I think there's an aspect here where it doesn't 13 00:01:06.920 --> 00:01:10.000 all doesn't offer value to the end user. I think is an important thing. 14 00:01:10.079 --> 00:01:15.719 So one definition we had was everything other than the product, which is 15 00:01:15.799 --> 00:01:19.959 interesting to think that the platforms we build aren't aren't value add but I, 16 00:01:21.000 --> 00:01:23.200 you know, I there's some interesting and deep dive there. But yeah, 17 00:01:23.319 --> 00:01:26.560 no, it was great. It was great. Confidence. I'm back. 18 00:01:26.640 --> 00:01:29.560 I'm happy to be back though, right, I'm cool. Happy to have 19 00:01:29.640 --> 00:01:34.560 you back. And then also joining us in the studio the co founder and 20 00:01:34.719 --> 00:01:40.239 CEO of Kurtosis, Galen Marquetti. Galen, welcome, thank you, thank 21 00:01:40.280 --> 00:01:42.280 you for having me, and hello everybody. Yeah, I'm excited to have 22 00:01:42.359 --> 00:01:45.680 you here. You know, we've talked a little bit in the past, 23 00:01:45.799 --> 00:01:51.519 and then we've also I've also been using critosis a little bit and it's very 24 00:01:51.560 --> 00:01:57.239 basic level. But tell us just give us like the high level overview of 25 00:01:57.439 --> 00:02:02.079 Kurtosis, and then we're going to dig into what led us to this point 26 00:02:02.120 --> 00:02:07.479 in life. So the highest level overview is that Krotosis makes it easy to 27 00:02:07.560 --> 00:02:15.520 get development environments and testing environments set up for back end systems. So I 28 00:02:15.680 --> 00:02:21.800 think something like doctor composed or helm charts with more parameterizability, more flexibility, 29 00:02:22.120 --> 00:02:27.879 and some things that give you greater portability guarantees. Gotcha. So when we're 30 00:02:27.879 --> 00:02:32.039 talking about dev environments, as a developer working on a product, I need 31 00:02:32.080 --> 00:02:37.520 to test the code that I'm writing, but that means I need to run 32 00:02:37.599 --> 00:02:40.919 the application. But then I also need all the supporting micro services, and 33 00:02:40.960 --> 00:02:46.479 then those are completely worthless without the data that gives me some kind of real 34 00:02:46.599 --> 00:02:52.800 world or pseudo real world scenario to test against. Is that right? That's 35 00:02:52.800 --> 00:02:54.759 about right, And I think a good way to think about the pain that 36 00:02:54.840 --> 00:03:00.639 kind of motivated the whole thing is that part about running other people's code. 37 00:03:00.719 --> 00:03:04.520 We're running your own code. Usually people have a handle on that one. 38 00:03:04.680 --> 00:03:09.120 It's when you need six or seven other micro services plus the data. Now 39 00:03:09.159 --> 00:03:13.080 you have to dig into how do I set these things up? And a 40 00:03:13.120 --> 00:03:16.879 lot of the time that I get stuck at that phase for an in order 41 00:03:17.080 --> 00:03:20.879 amount of time, and that's kind of the heart of what we're doing. 42 00:03:21.919 --> 00:03:24.960 Yeah, for sure. I can think back of different scenarios where I've addressed 43 00:03:24.960 --> 00:03:30.400 this in the past, and some of the approaches I've used are the you 44 00:03:30.400 --> 00:03:38.000 know, like the seven thousand line Bash script. There's the golden machine or 45 00:03:38.039 --> 00:03:42.439 the golden VM that no one's allowed to touch because it has everything on it. 46 00:03:42.560 --> 00:03:47.360 Right at the moment, there's trying to bribe your security team to let 47 00:03:47.439 --> 00:03:53.240 you connect to the staging environment across the firewall. So those are some of 48 00:03:53.240 --> 00:03:58.599 the approaches I've used in the past. Yeah, it's exactly right. It's 49 00:03:58.639 --> 00:04:02.039 exactly right, and it's what a lot of the times people get a good 50 00:04:02.080 --> 00:04:05.639 system set up for kind of a golden path they do all the time, 51 00:04:05.919 --> 00:04:11.360 but then there's this like long tail of things that you always do a long 52 00:04:11.439 --> 00:04:14.319 tail thing maybe once a week, but it's a different long tail thing, 53 00:04:14.479 --> 00:04:18.360 and it's that set of issues and security and you have to touch the golden 54 00:04:18.360 --> 00:04:24.199 and VM for it, it break it, and it just adds up for 55 00:04:24.240 --> 00:04:30.120 sure. Orren, what's what's your approach in solving this problem? Oh, 56 00:04:30.160 --> 00:04:32.120 well, you know, I think we only have like an hour on this. 57 00:04:34.959 --> 00:04:43.319 There's something here where if you have really high separation, low coupling between 58 00:04:43.360 --> 00:04:46.160 your individual micro services, you can just spin up the one that you've got 59 00:04:46.199 --> 00:04:51.279 on your machine. We use local stack for our infrastructure, but I think 60 00:04:51.319 --> 00:04:57.959 that's a very extreme perspective and it takes a while to really get there, 61 00:04:57.959 --> 00:05:00.160 and we focus on having that at the beginning. And if you come from 62 00:05:00.199 --> 00:05:03.959 a monolith world or one where the architecture wasn't really figured out, there's this 63 00:05:04.160 --> 00:05:10.000 sort of dead zone in the middle where your services are still unfortunately coupled in 64 00:05:10.040 --> 00:05:14.439 some way and you really can't get the value out of them without having some 65 00:05:14.560 --> 00:05:19.040 of the ancillary pieces also spun up and working at the same time. Right 66 00:05:19.079 --> 00:05:27.759 on for sure. So Galen, obviously there was a set of circumstances in 67 00:05:27.800 --> 00:05:30.800 your life that led you to say, this is a real problem and I'm 68 00:05:30.839 --> 00:05:34.319 going to I'm going to give it my best shot at starting. And it 69 00:05:34.480 --> 00:05:39.120 led to a company. What what kind of scenarios were you working in that 70 00:05:39.240 --> 00:05:43.959 made you realize this is what you want to do? Yeah, I mean 71 00:05:43.959 --> 00:05:51.920 there's a couple. The earliest instance of this innate pain it came at my 72 00:05:53.000 --> 00:05:56.240 senior year of college. My senior year of college, I was doing a 73 00:05:56.279 --> 00:06:02.600 research project on distributed databases. And this was right before Ethereum launch, which 74 00:06:02.680 --> 00:06:09.360 is a cryptocurrency, and I was doing a research project to explore different consensus 75 00:06:09.439 --> 00:06:15.079 mechanisms, so like low level protocol comms between notes. I wanted to spin 76 00:06:15.240 --> 00:06:23.319 up three nodes of this network just to experiment with my different consensus mechanism. 77 00:06:24.240 --> 00:06:29.199 Nearly impossible to do that at the time, and a lot of my project 78 00:06:29.279 --> 00:06:32.279 was oriented around simply getting the things spun up in the first place. Of 79 00:06:32.319 --> 00:06:35.959 course, the code worked on the developers' laptops that worked on the DevOps. 80 00:06:36.000 --> 00:06:41.800 Laptops didn't work on my laptop. And it wasn't about it wasn't about building 81 00:06:41.920 --> 00:06:46.439 a particular note. It was about getting the cluster of the system set up 82 00:06:46.480 --> 00:06:51.639 correctly where there's really no good abstraction that gets you that very consistently. You 83 00:06:51.680 --> 00:06:58.160 can get it in certain architectures in certain ways decently consistently. But that was 84 00:06:58.199 --> 00:07:00.639 the first instance. It was very fresh trading, like why won't this thing 85 00:07:00.800 --> 00:07:05.240 just spin up? It's the simplest application, just run it as it is. 86 00:07:08.759 --> 00:07:11.399 Yeah, that was the very first one. I saw it later in 87 00:07:11.439 --> 00:07:17.360 my career as well as working at a company that had had a internal tool 88 00:07:17.439 --> 00:07:21.360 that spun up a full micro service deck you can think maybe two hundred micro 89 00:07:21.439 --> 00:07:28.399 services, and it was used for a variety of purposes between dev demoing uat 90 00:07:28.680 --> 00:07:32.639 so like a high level just testing user workflows to see how they would work. 91 00:07:33.519 --> 00:07:38.160 That took a lot of effort to get that system spun up, and 92 00:07:38.199 --> 00:07:41.160 at the essence of it, it's the same issue. It's I just want 93 00:07:41.160 --> 00:07:46.439 to run the system and a configuration that makes sense for the work while I'm 94 00:07:46.480 --> 00:07:53.079 doing and it can be remarkably hard to do that. So this these sets 95 00:07:51.839 --> 00:07:57.519 of issues. It's kind of the feeling of like, this shouldn't be this 96 00:07:57.639 --> 00:08:01.519 way. It should be decently easy to just run software. It should be 97 00:08:01.560 --> 00:08:07.399 hard to like build software that's that's that's amazing, but it should be easy 98 00:08:07.480 --> 00:08:09.319 to run it. It's kind of the feeling of the way the world should 99 00:08:09.319 --> 00:08:16.800 be that kind of motivated the entire start into pretosis, right, And yeah, 100 00:08:16.800 --> 00:08:22.800 and I think there's I think maybe just part of that is we build 101 00:08:22.839 --> 00:08:26.839 these complex systems, but for most of us, once we build that, 102 00:08:26.920 --> 00:08:33.240 like once we build our production environment, the people who really have the skills 103 00:08:33.279 --> 00:08:37.240 to build those, they don't build a second production. The part where this 104 00:08:37.279 --> 00:08:41.440 pain has really felt is in the development teams where they're trying to you know, 105 00:08:41.519 --> 00:08:46.200 build and tear these down to test different features and add new features to 106 00:08:46.240 --> 00:08:52.159 it. Yes, and that's one of the main themes that I've learned while 107 00:08:52.320 --> 00:08:58.080 going through this journey is it's a lot of the times these development environments they're 108 00:08:58.120 --> 00:09:03.279 about getting us close for production as possible but safe and you can do the 109 00:09:03.279 --> 00:09:07.720 workfload you need to do. But the main value access is how close can 110 00:09:07.759 --> 00:09:11.080 you get to production? So, like Warren mentioned, local stack, you 111 00:09:11.120 --> 00:09:16.759 can run a tows services on your laptop. It's one approach to it, 112 00:09:16.919 --> 00:09:24.039 bring production locally or bring production to your development environment. The other approach is 113 00:09:24.080 --> 00:09:26.799 the opposite. You put your development environment into the cloud, where you then 114 00:09:28.480 --> 00:09:33.840 basically fork everything that you have in production and you have your cloud services are 115 00:09:33.919 --> 00:09:37.960 sitting in the exact same environment where your development code is another approach to it. 116 00:09:37.519 --> 00:09:43.440 But basically, everyone who's putting a lot of investment into getting these great 117 00:09:43.480 --> 00:09:46.919 development environments does one of those two things, or some combination of both, 118 00:09:48.440 --> 00:09:54.399 bringing that bringing production locally or putting your local into production in the sense of 119 00:09:54.480 --> 00:10:00.639 into your cloud environment. Right right, what's the barrier to intrig for using 120 00:10:00.679 --> 00:10:07.960 something like critosis, Yeah, right now. The biggest barrier to entry is 121 00:10:09.399 --> 00:10:15.600 defining the setup logic for your environment. So we have a definition language. 122 00:10:15.600 --> 00:10:18.799 We use Starlarc, which is the language. It's a Python dialect that Basil 123 00:10:18.919 --> 00:10:24.240 uses to define the build rules. We use it for similar similar properties that 124 00:10:24.240 --> 00:10:28.200 it has for the build system. It's you can determine that the program is 125 00:10:28.200 --> 00:10:33.120 going to halt because it's not a full computing language. But you need to 126 00:10:33.159 --> 00:10:39.279 define your set up logic in this language. And once you do it once, 127 00:10:39.679 --> 00:10:43.039 everyone else can reuse it. But what happens a lot of the time 128 00:10:43.200 --> 00:10:48.240 is who knows how to do that? You have to It's especially in an 129 00:10:48.320 --> 00:10:52.799 environment where there's a lot of different open source tooling being used to varying degrees 130 00:10:52.799 --> 00:10:58.679 of maturity. The person who has the expertise to understand how to spin it 131 00:10:58.759 --> 00:11:03.639 up in a different right situations, that person might not be maintained that project 132 00:11:03.639 --> 00:11:09.919 anymore, even so track that down or relearn it somehow, for sure. 133 00:11:11.039 --> 00:11:18.240 Yeah. And then I think there's a related piece to that is not only 134 00:11:18.360 --> 00:11:22.799 building it, but then making sure like reverse engineering, I guess, for 135 00:11:22.919 --> 00:11:26.399 a lack of a better term, to make sure that what you've built for 136 00:11:26.440 --> 00:11:31.840 your local dev tool actually mimics what's really going on out in production, right, 137 00:11:31.879 --> 00:11:35.720 because there's a lot of different ways you can make all of these different 138 00:11:35.759 --> 00:11:37.519 components talk to each other, and you've got to try to replicate that. 139 00:11:39.600 --> 00:11:45.120 Yes, yes, and that's that's a lot of work, and it's that's 140 00:11:45.200 --> 00:11:50.360 one of the reasons why a lot of people do opt to forego that whole 141 00:11:50.360 --> 00:11:54.720 process and just put their development code directly into a production like cluster in the 142 00:11:54.759 --> 00:12:00.720 cloud as as possible. Because they're like a localist, never going to be 143 00:12:01.200 --> 00:12:05.399 close enough to production. It's going to take a lot of never ending effort 144 00:12:05.480 --> 00:12:09.159 to get it close. So let's just get to it a stage in cluster 145 00:12:09.240 --> 00:12:11.279 as fast as possible, let things soak there, and then go to abroad. 146 00:12:11.799 --> 00:12:18.720 It's another approach to it. There's benefits and cons to each one, 147 00:12:20.480 --> 00:12:24.799 for sure. Yeah, what are some of the big Well, first of 148 00:12:24.799 --> 00:12:31.360 all, how long have you been building critosis? Yeah? We've been doing 149 00:12:31.399 --> 00:12:35.360 this full time since December of twenty twenty, So what is that? Okay, 150 00:12:35.600 --> 00:12:41.120 three years and change? Yeah, yeah, yeah, cool. So 151 00:12:41.159 --> 00:12:43.960 what are some of the how is how is your thought process changed in those 152 00:12:45.120 --> 00:12:48.799 three plus years for building development environments? I mean, I'm sure you've learned 153 00:12:48.879 --> 00:12:54.720 lots of lessons and tried a lot of things that didn't quite work. Absolutely, 154 00:12:54.960 --> 00:13:03.559 absolutely. The couple of things. First, we started out, we 155 00:13:03.600 --> 00:13:11.000 started out our company selling a basically binaries. So one thing is, how 156 00:13:11.039 --> 00:13:16.679 do developers and development teams want to consume developer tooling? So we started out 157 00:13:16.720 --> 00:13:20.519 saying we're going to sell a compiled binary that you get to run as a 158 00:13:20.559 --> 00:13:26.159 license fee, and their first couple of clients we operated that way and It 159 00:13:26.360 --> 00:13:31.799 was fine in the very beginning. We were solving a very important problem for 160 00:13:31.840 --> 00:13:37.000 those clients. It was acceptable at the time, but we very quickly learned 161 00:13:37.039 --> 00:13:41.240 that this is it's not the way that developers want to and for really good 162 00:13:41.320 --> 00:13:46.639 reasons, want to be interacting with software. You do have to have the 163 00:13:46.679 --> 00:13:50.120 source code available, You do have to have a free license, you have 164 00:13:50.200 --> 00:13:54.159 to have a free period of trying it where they know, because they're integrating 165 00:13:54.360 --> 00:13:58.399 at a deep level with their tooling, if they're going to try it out, 166 00:13:58.399 --> 00:14:00.759 if they're going to invest into it, I do want to know it's 167 00:14:00.759 --> 00:14:03.759 not going to get yanked out from under them, and that's totally fair. 168 00:14:03.080 --> 00:14:09.039 So we change our licensing entirely, made the reboot open, and that change 169 00:14:09.200 --> 00:14:13.799 changed everything about the rate at which we were learning, the rate of the 170 00:14:13.799 --> 00:14:16.000 exposure to different developers, people able to try out to too long gate different 171 00:14:16.039 --> 00:14:20.320 perspectives. In retrospect, it was pretty obvious. Maybe a lot of these 172 00:14:20.399 --> 00:14:24.840 learnings are pretty obvious in retrospect, but you know, that was that was 173 00:14:24.879 --> 00:14:26.720 maybe the biggest change that we had in the business, was opening a tool 174 00:14:26.799 --> 00:14:31.240 like that. No, I think you're in good company. The past few 175 00:14:31.360 --> 00:14:35.120 weeks months I feel like there's been a huge shift and focus on the podcast. 176 00:14:35.200 --> 00:14:39.320 It keeps coming out. Wills bringing up this point about selling your idea 177 00:14:39.399 --> 00:14:43.759 within the devlof space as a product management challenge, So really understanding what your 178 00:14:45.440 --> 00:14:50.679 teams need that are your consumers. It's difficult but also really interesting and can 179 00:14:50.720 --> 00:14:56.240 help you build a great a great product for them. Yeah, and it's 180 00:14:56.519 --> 00:15:01.039 developers are they're amazing users to work with, but it's a very different dynamic. 181 00:15:01.200 --> 00:15:07.759 I think then products that maybe are selling the sales folks or business focused 182 00:15:07.799 --> 00:15:13.320 folks, you're the way that products are integrated and interactive with is quite different 183 00:15:13.200 --> 00:15:18.080 and as different dynamics. But once you're solving a problem that the developers really 184 00:15:18.120 --> 00:15:24.759 really care about, you really do develop really deep relationships for users, especially 185 00:15:24.080 --> 00:15:28.519 with like the community based approach with like an open product. So it's it's 186 00:15:28.519 --> 00:15:33.279 been game changing having that kind of a relationship working for us. Yeah. 187 00:15:33.759 --> 00:15:39.240 I think that's a really cool point there, because you're talking about who your 188 00:15:39.360 --> 00:15:45.480 target market is, and so whenever you think about like production deployments, you 189 00:15:45.519 --> 00:15:50.360 know, deploying my application to what my target customers, that's a very different 190 00:15:50.440 --> 00:15:56.639 process than deploying my stack to the local development team so they can build on 191 00:15:56.679 --> 00:16:00.840 it. It's a different, different customer and they have different goals and objectives 192 00:16:00.840 --> 00:16:06.320 like the customers, the consumers of my product want high availability, up time 193 00:16:06.480 --> 00:16:12.000 and fast response, whereas the developers want more granular access and the ability to 194 00:16:12.200 --> 00:16:18.279 tweak and turn the knobs across the environment to see how they can solve their 195 00:16:18.320 --> 00:16:22.519 problems. Is that a fair statement. Absolutely? Absolutely. In fact, 196 00:16:22.519 --> 00:16:30.519 there's even three different profiles to consider. There's the developers that are the end 197 00:16:30.600 --> 00:16:36.639 users of the tool. There's the we call it maybe the platform team DevOps 198 00:16:36.639 --> 00:16:40.399 team in for a team, they're really the ones that are responsible for the 199 00:16:40.440 --> 00:16:44.320 implementation and configuration of the tool. A lot of the times they're the ones 200 00:16:44.360 --> 00:16:48.279 getting access to the GitHub repos they're doing the high level configuration administration of the 201 00:16:48.320 --> 00:16:55.200 installation, and they're concerned about the security posture and all that. And then 202 00:16:55.519 --> 00:17:00.519 ultimately you also do have like the director or head of engineering, whoever approves 203 00:17:00.559 --> 00:17:04.720 the budget for whatever you're charging for an enterprise version. So there's three. 204 00:17:06.160 --> 00:17:11.680 There's three different concerns to worry about, and you have to think about each 205 00:17:11.720 --> 00:17:18.000 one differently. It's it's like what when you think about like a minimal valuable 206 00:17:18.440 --> 00:17:22.720 product, Like how how batting of the concerns at each of the three parties 207 00:17:22.039 --> 00:17:26.000 you really have to do to like get something off the ground. Uh yeah, 208 00:17:26.160 --> 00:17:30.119 so it's a multi dimensional problem. No one envies a startup pounder, 209 00:17:30.119 --> 00:17:33.519 I mean they do. They think you get to build your your dream, 210 00:17:33.559 --> 00:17:37.559 but then you deal with these quite challenging situations where your buyers are not your 211 00:17:37.759 --> 00:17:41.319 users that you are you know, really focused on. I have a question 212 00:17:41.400 --> 00:17:45.839 on that. Actually, I feel like and I know I'm gonna put myself 213 00:17:45.880 --> 00:17:49.799 in some bucket somewhere. I assume like everyone's using open TOFO today. I 214 00:17:49.839 --> 00:17:52.440 mean, I know that's not the case, but you know, maybe there's 215 00:17:52.480 --> 00:17:56.680 there's a huge audience there. Especially if we talk about people that are in 216 00:17:56.960 --> 00:18:02.359 the quote unquote develop space. If they are familiar with this terminology, they 217 00:18:02.400 --> 00:18:07.680 probably are using some sort of infrastructure as code solution. We use extensively CloudFormation 218 00:18:07.759 --> 00:18:12.960 ourselves. What is the how does that interact with krotosis here? Like where 219 00:18:12.960 --> 00:18:19.279 the touch points? Like if we've already defined our infrastructure in unfortunately HCl or 220 00:18:19.359 --> 00:18:25.880 worth in a programming language that doesn't really well match to declarative nature, what 221 00:18:25.920 --> 00:18:30.680 do we do to spin up a dev environment absolutely, so right now, 222 00:18:30.759 --> 00:18:37.400 krtosis operates after you apply your open tofu or your terraform scripts to spin up 223 00:18:37.680 --> 00:18:42.400 the cloud services that you would need for your development environment or your testing environment. 224 00:18:42.640 --> 00:18:48.599 You can think of it really as a abstraction layer over Kubernetes and Docker. 225 00:18:49.359 --> 00:18:55.519 So if you have can use your terraform scripts open tofu to spin up 226 00:18:55.519 --> 00:19:00.440 your Kubernetes clusters and provision everything required there and then deploy onto the clusters with 227 00:19:00.559 --> 00:19:15.759 grotosis, I see gotcha, And so then that so like that, that's 228 00:19:15.799 --> 00:19:22.839 like a way to leverage some of the tools that you're using to deploy into 229 00:19:22.960 --> 00:19:26.599 production for building that so you're not duplicating a lot of effort, right so 230 00:19:26.599 --> 00:19:32.000 you can still leverage the learnings from terra farm or open to tofu or any 231 00:19:32.039 --> 00:19:37.559 other production grade tools that you're deploying with exactly exactly. And then the cloud 232 00:19:37.680 --> 00:19:41.480 provisioning side of things is an area that we are now exploring, and it's 233 00:19:41.559 --> 00:19:45.880 not an area that's covered in the current like open tool and like the direct 234 00:19:45.920 --> 00:19:53.359 aspaciates of the open tool where it's hey, can you also handle maybe four 235 00:19:53.440 --> 00:19:56.640 K my production environment, and its like as look as you could fork an 236 00:19:56.720 --> 00:20:02.759 environment, can you handle clone everything necessary there? Right now? That's a 237 00:20:02.759 --> 00:20:07.559 lot of what's happened in people's information and terraform scripts is something like that we 238 00:20:07.680 --> 00:20:11.119 haven't That's an area that we're exploring right now learning about. You know, 239 00:20:11.759 --> 00:20:18.079 what's important when you're when you're looking at designing those those scripts. Yeah, 240 00:20:18.079 --> 00:20:22.400 for sure when it comes to documenting, like because that's a I think that's 241 00:20:22.400 --> 00:20:30.240 a would be an important artifact of this process is we've discovered how our production 242 00:20:30.319 --> 00:20:34.680 infrastructure is working, reverse engineered it. How do you approach documenting that so 243 00:20:34.720 --> 00:20:41.599 that when the next development team comes along, maybe the learning curve is a 244 00:20:41.599 --> 00:20:48.400 little bit uh, not so sharp for them. Yeah, right now, 245 00:20:48.440 --> 00:20:52.759 the document I guess there's two ways to think about it. One is one 246 00:20:52.839 --> 00:20:57.799 is how much can you write? How much can you have your users write 247 00:20:57.880 --> 00:21:03.880 code that is some self documenting, so like, can the environment definition be 248 00:21:03.079 --> 00:21:10.200 the documentation and as much as possible. I like having code that is extraordinarily 249 00:21:10.200 --> 00:21:15.359 readable and it seems self documenting, and the way that our tool is set 250 00:21:15.440 --> 00:21:18.599 up, you tend to get that in the code. It tends to be 251 00:21:18.720 --> 00:21:21.640 very composable, it tends to be very readable. It tends to be very 252 00:21:22.000 --> 00:21:27.359 intuitive, So that's one function of it. Another function of it is the 253 00:21:27.400 --> 00:21:33.920 functionality that we offer, so we have high level environment parameterizability. What that 254 00:21:33.960 --> 00:21:41.079 means is when you write your environment definition, you can expose very easily how 255 00:21:41.480 --> 00:21:45.599 to modify it in a sensible way. So you start it and you say, 256 00:21:45.599 --> 00:21:48.720 well, maybe I want to modify it this way, this way, 257 00:21:48.759 --> 00:21:52.359 this way. We don't need documentation for it because when you want it with 258 00:21:52.480 --> 00:21:55.519 that configuration, you're going to get it out of the box. So you're 259 00:21:55.519 --> 00:22:00.359 trying to reduce the need for documentation as much as possible with the design of 260 00:22:00.400 --> 00:22:03.000 the definition language. But you can only dose it so much so that after 261 00:22:03.039 --> 00:22:07.440 that it's it's traditional documentation. It's you got to rely on your users to 262 00:22:07.440 --> 00:22:12.200 write the readings, to write the docs for those specific systems. That it 263 00:22:12.240 --> 00:22:17.279 reminds me of a quote that I heard a long long time ago, back 264 00:22:17.319 --> 00:22:21.279 when everyone used to have wikis for documentation, and I worked with this. 265 00:22:21.319 --> 00:22:26.039 One guy said, yeah, the wiki is where the documentation goes to die. 266 00:22:27.480 --> 00:22:32.599 But I think like one of the cool things I think that's happened is 267 00:22:32.599 --> 00:22:36.279 if you've ever used Jupiter notebooks. I love the way that they let you 268 00:22:37.160 --> 00:22:40.680 mix, you know, different different syntax in there, so you can have 269 00:22:40.720 --> 00:22:42.680 like a markdown block and then a code block, and so you can kind 270 00:22:42.680 --> 00:22:51.839 of integrate the documentation alongside the code. And for I think for iterative or 271 00:22:51.880 --> 00:22:55.960 interactive type environments like that, it works really well because it then lets you 272 00:22:56.039 --> 00:23:00.079 keep everything in one place, because when you try to update multiple places, 273 00:23:00.880 --> 00:23:06.920 it just never happens in reality. Yeah, and I love Jupiter Notebooks the 274 00:23:07.079 --> 00:23:11.039 experience, Like, it's one of my favorite That access in the world is 275 00:23:11.759 --> 00:23:15.720 like spinning up and hacking something together in a Jupiter notebook. And the dream 276 00:23:15.839 --> 00:23:18.519 is to feel that way about all of our development work. I'm not sure 277 00:23:18.359 --> 00:23:23.680 realistic, but I love that experience. I mean, that's an interesting thread. 278 00:23:23.799 --> 00:23:29.680 What would what would the world have to look like that regular development could 279 00:23:29.880 --> 00:23:33.759 follow this sort of path here? Is it a matter of what we're working 280 00:23:33.799 --> 00:23:40.920 on or can we already sort of start doing that today? Yeah? I 281 00:23:40.960 --> 00:23:44.200 mean what's great about it? Like, one thing that's great is how fast 282 00:23:44.319 --> 00:23:48.400 you see your changes work. It's like extraordinarily fast. You write it, 283 00:23:48.519 --> 00:23:52.519 run run and run, run, run, and you get the feed back 284 00:23:52.599 --> 00:23:56.319 right away. That's the build deploy like how the build deploy loop. It's 285 00:23:56.359 --> 00:24:00.160 like, how fast does it take for you to see on real stuff off 286 00:24:00.160 --> 00:24:03.400 what your things look like? Uh, usually in Jupiter notebooks you're using. 287 00:24:03.839 --> 00:24:06.960 I mean, like a lot a lot of the fun use cases for it 288 00:24:07.000 --> 00:24:11.599 are running on relatively small data sets so it fits in memory. So you 289 00:24:11.799 --> 00:24:17.759 do that fast feed toploy build a play loop, and that's just tricky to 290 00:24:17.960 --> 00:24:21.720 get. Like you can get it with like hot reloading into containers that are 291 00:24:21.759 --> 00:24:23.839 running in your in your environment. You can get decently close to it, 292 00:24:23.880 --> 00:24:27.319 but you're still stuck at the level of like what is this What is the 293 00:24:27.359 --> 00:24:33.599 smallest build increment that you can that you can like define for your background system. 294 00:24:33.880 --> 00:24:37.480 I know a lot of people spend a lot of effort getting their their 295 00:24:37.519 --> 00:24:42.640 basal build rules so well defined that they're going to like minimize the build a 296 00:24:42.640 --> 00:24:45.000 play loop as much as possible, But that takes a lot of effort. 297 00:24:45.039 --> 00:24:49.559 You really have to look into it to get that experience. I mean, 298 00:24:49.559 --> 00:24:53.519 I'm not sure that there's a really good way to get that now, No 299 00:24:53.599 --> 00:24:57.160 for sure, for sure not but I think you know, tools, tools 300 00:24:57.160 --> 00:25:00.599 like yours you know definitely help reduce the loop here, which is like the 301 00:25:00.640 --> 00:25:08.119 most important aspect because no matter how much time you spend, really every second 302 00:25:08.200 --> 00:25:14.079 you cut off your loop allows that feedback to happen faster, which improves the 303 00:25:14.200 --> 00:25:21.279 time. I think it's a hackneyed response now that realistically people waiting for software 304 00:25:21.319 --> 00:25:23.960 development, is there what makes them unhappy? Right? If you have to 305 00:25:25.039 --> 00:25:27.039 wait for the response? Will said, You know, he's all about the 306 00:25:27.039 --> 00:25:32.160 instant gratifications. Nothing better than you know, hitting F five and having your 307 00:25:32.200 --> 00:25:37.480 program instantly work or tell you what you did wrong, which is you know, 308 00:25:37.519 --> 00:25:42.119 I don't think it's just me. I'm convinced that we are a TikTok 309 00:25:42.240 --> 00:25:45.279 generation. You know, whether you're a TikTok user or not, we've all 310 00:25:45.319 --> 00:25:51.039 gotten addicted to short form content. You know, if you're going to do 311 00:25:51.079 --> 00:25:56.559 anything that requires an attention span of over sixty seconds, you really have to 312 00:25:56.759 --> 00:26:04.039 engineer for that. And I we've been digging a lot into this this little 313 00:26:04.920 --> 00:26:10.880 that little feedback loop there and one thing that did the dream world. There's 314 00:26:10.960 --> 00:26:14.519 kind of two really important things. One is getting that build deplay loop super 315 00:26:14.640 --> 00:26:17.480 super fast, so you're coding the change, accoding to the change. The 316 00:26:17.519 --> 00:26:22.319 other one is when you see the change, how close is the change that 317 00:26:22.359 --> 00:26:25.839 you're seeing to what would happen in production? And I can you get both 318 00:26:25.880 --> 00:26:27.880 of those things maximized, so like you're getting a really really fast build to 319 00:26:27.880 --> 00:26:34.160 play loop, but also the deployment is safely basically a production when you're doing 320 00:26:34.160 --> 00:26:40.559 it. That's kind of the dream the dream world. It's it would take 321 00:26:40.559 --> 00:26:42.279 a lot to get there, I think all the way, but you kind 322 00:26:42.279 --> 00:26:47.599 of like, roughly, what's happening. I think with the more and more 323 00:26:47.599 --> 00:26:52.839 sophisticated like dev infra along the software development bicycle, roughly what's happening is you're 324 00:26:52.839 --> 00:26:56.799 getting like more and more optimized on both of those axis to the point that 325 00:26:56.880 --> 00:27:00.720 eventually it kind of feels like you're doing that like the limit of what's possible. 326 00:27:02.880 --> 00:27:07.000 I think there's a sort of a conundrum here where the providers that are 327 00:27:07.039 --> 00:27:11.359 offering production infrastructure or platforms, the amount of money that you're paying into them 328 00:27:11.440 --> 00:27:17.240 is so large or so vast for the environment you're running for production that it 329 00:27:17.319 --> 00:27:21.720 never makes sense. It's not something that is ever justified for them to make 330 00:27:22.119 --> 00:27:26.839 it faster to run or get the loop closed because they don't care. It's 331 00:27:27.000 --> 00:27:32.759 you know one like how much is your test staging, fake feral environment cost 332 00:27:33.319 --> 00:27:37.920 in actual dollars, euros and pesos whatever compared to production It's like what one 333 00:27:37.960 --> 00:27:42.920 percent less? Even so like Amazon, you know, Google, GCP platforms 334 00:27:42.920 --> 00:27:45.200 of the world, they're not going to make anything there, and so I 335 00:27:45.640 --> 00:27:52.079 feel like there it would be on them to really change that because that's sort 336 00:27:52.119 --> 00:27:55.799 of like the canonical interface that you have to abstract or make the same as 337 00:27:55.880 --> 00:28:02.359 much as possible. Yes, absolutely, and basically everyone's working against that limitation 338 00:28:02.839 --> 00:28:06.319 as far as I can tell. It is like that's the that's the blocker, 339 00:28:07.000 --> 00:28:11.839 the only way to like you're you're basically just trying to design more and 340 00:28:11.880 --> 00:28:17.680 more like innovative solutions to get around exactly that obstacle. Yeah, because let's 341 00:28:17.720 --> 00:28:22.880 be just like completely transparent here. The gold standard is them on the production 342 00:28:23.000 --> 00:28:27.880 server. Yes, you said that. I feel like that's where I started 343 00:28:27.920 --> 00:28:33.359 my career actually right for sure, Like I was working at a company where 344 00:28:33.519 --> 00:28:38.920 you did software development by sshing into another service somewhere and there was an environment 345 00:28:40.039 --> 00:28:45.079 running there which then you logged into and you literally changed production objects by opening 346 00:28:45.119 --> 00:28:48.559 a text editor there. We didn't it wasn't VIM that would have been slightly 347 00:28:48.599 --> 00:28:52.400 better than what was being used, uh and uh, and then changing them 348 00:28:52.440 --> 00:28:57.880 and then you had to record the changes you made in an overgrown internal system 349 00:28:59.160 --> 00:29:03.079 that would somehow get those changes applied to actual production when there was a release. 350 00:29:03.519 --> 00:29:06.559 So, you know, I really do feel like what was old will 351 00:29:06.599 --> 00:29:10.480 be new again, and I honestly don't know if I want that world. 352 00:29:12.880 --> 00:29:18.519 My introduction to programming was doing something very similar to that, and the guy 353 00:29:18.519 --> 00:29:19.319 who showed me how to do it, he's like, okay, when you 354 00:29:19.359 --> 00:29:23.400 get here, you want to run the LS command and you could see all 355 00:29:23.440 --> 00:29:26.759 of the files there. But then it was like, you know, for 356 00:29:26.920 --> 00:29:33.000 any given file, you know, like just say like Maine dot c, 357 00:29:33.279 --> 00:29:36.200 there was Maine dot c dot one, Maine dot C dot two, Maine 358 00:29:36.200 --> 00:29:37.960 dot c dot three. Is like, so you want to look and see 359 00:29:37.000 --> 00:29:42.200 what the biggest number here is and copy Maine dot c to the next number 360 00:29:42.279 --> 00:29:45.480 so that if you screw anything up, you've got a backup for it. 361 00:29:45.839 --> 00:29:51.160 And that was our version control system. Oh no, I'm just like, 362 00:29:51.359 --> 00:29:53.519 I think people figured it out date based, right, You don't. None 363 00:29:53.559 --> 00:29:56.319 of these are the fancy tools. All you need to do is append the 364 00:29:56.359 --> 00:30:00.880 timestamp of the last time you edited the file there and then you'd be good. 365 00:30:03.319 --> 00:30:10.440 Yeah, I mean it's the VIM on the production server. At least 366 00:30:10.440 --> 00:30:15.960 for me, this is like clearly the best debt X Like I love it. 367 00:30:15.400 --> 00:30:19.440 I love it. It's just dangerous, it's super dangerous, and like 368 00:30:19.519 --> 00:30:23.480 staying up all night to think there of an outage is miserable. So that 369 00:30:23.599 --> 00:30:30.839 experience is not great. But the feeling of the VIM directly to production is 370 00:30:30.839 --> 00:30:36.039 is is the ideal, So like to what degree can you mitigate risk to 371 00:30:36.119 --> 00:30:38.200 get as close to that as possible? I mean, you said it right, 372 00:30:38.240 --> 00:30:41.799 you know, things like kurtosis, you know, shortening the gap there 373 00:30:41.839 --> 00:30:44.720 on time to make a change so that rather than VIM on the production server, 374 00:30:44.759 --> 00:30:47.559 it's VIM on your machine or you know, using one of those better 375 00:30:47.599 --> 00:30:52.240 tools that's out there, and then making that change and just immediately that gets 376 00:30:52.279 --> 00:30:56.240 to a clone of production that is almost exactly this, I mean exactly the 377 00:30:56.279 --> 00:31:00.319 same, and you can see that change running as if you had actually applied 378 00:31:00.319 --> 00:31:06.160 it and then get all the benefits without having actually the danger of making a 379 00:31:06.160 --> 00:31:10.160 production change. Yes, yes, absolutely, that's that's that's the that's theer's 380 00:31:10.200 --> 00:31:15.480 the dream Yes, So what is the learning curve for someone who's just they've 381 00:31:15.519 --> 00:31:21.160 they've done the seven thousand line bash script and they're looking for something different and 382 00:31:21.480 --> 00:31:25.200 wanting to check out critosis. What's the learning curve or the amount of time 383 00:31:25.200 --> 00:31:30.519 it takes to be productive with this so as framed very low. And the 384 00:31:30.599 --> 00:31:37.799 reason for that is because that person did the seven thousand best script part. 385 00:31:37.920 --> 00:31:40.279 They figured out how it all works together, they figured out who needs to 386 00:31:40.279 --> 00:31:44.480 be scripted, they figured out all of this hard stuff. That learning curve 387 00:31:44.559 --> 00:31:48.559 is extraordinarily low. I mean, in Krytosis, you're using a Python like 388 00:31:48.640 --> 00:31:53.759 dialect. Most people are familiar with Python. The set of I guess functions 389 00:31:53.759 --> 00:31:59.519 and tools you have are very are very basic. Half a day to just 390 00:31:59.599 --> 00:32:01.759 rewrite it and something that's uh, that's cleaner, and then it's going to 391 00:32:01.799 --> 00:32:05.319 take you a lot further. It's probably about what you're going to get. 392 00:32:06.279 --> 00:32:09.920 Now. The part that's you know, frustrating for us and we're trying to 393 00:32:09.920 --> 00:32:13.839 figure out how to make this better is what if you haven't written the seven 394 00:32:13.839 --> 00:32:15.680 thousand mind mass script, then what do you do? You say, Hey, 395 00:32:15.680 --> 00:32:19.240 someone else wrote the seven thousand mind mass script. I don't understand it 396 00:32:19.319 --> 00:32:21.680 yet, but I want it to be. I want to get all these 397 00:32:21.680 --> 00:32:24.119 benefits that I can get from protosis. Now we have a really big now 398 00:32:24.119 --> 00:32:27.839 we have a really big verded curve, but the learning curve is mostly inside 399 00:32:27.839 --> 00:32:37.440 of the side. Interesting for sure. And then can you give us like 400 00:32:37.559 --> 00:32:45.559 some some success stories of like different problems that customers have solved. I always 401 00:32:45.599 --> 00:32:46.960 think that's kind of cool because then it, you know, it gives us 402 00:32:47.000 --> 00:32:51.160 something that we can like resonate with and be like, oh, yeah, 403 00:32:51.440 --> 00:32:52.920 I remember when I had to do that. Like, what are some cool 404 00:32:53.160 --> 00:32:57.480 things that you've seen people do with krotosis? Well, I feel like your 405 00:32:57.920 --> 00:33:01.400 whole work experience history here, you know, just recount those things and it's 406 00:33:01.519 --> 00:33:12.200 like, yeah, yeah, my career could be a like a Forrest Gump 407 00:33:12.319 --> 00:33:24.119 story in tech. Yeah. I Mean the biggest and like the most proud 408 00:33:25.119 --> 00:33:30.920 use case that we have for cryptosis is what the Ethereum Foundation is doing with 409 00:33:30.000 --> 00:33:35.920 cryptosis. That's probably the thing that we're most it's the most incredible thing. 410 00:33:37.000 --> 00:33:42.240 So for folks who aren't like deep into the blockchain ecosystem, the Ethereum main 411 00:33:42.400 --> 00:33:47.319 net, so it's like the production deployment of Ethereum is now extraordinarily complicated. 412 00:33:47.519 --> 00:33:52.720 There is like twenty different open source components that have to get splat up together. 413 00:33:52.400 --> 00:33:57.400 There's no centralized DEVLOPS team for this, so it's all different people running 414 00:33:57.440 --> 00:34:04.400 their own different service with moose communication between them. What krotosis, what they're 415 00:34:04.480 --> 00:34:07.199 using Katosis do. The Ethereum foundation is to package all of that together in 416 00:34:07.880 --> 00:34:14.639 a one line run with high level configurability, so you can say, hey, 417 00:34:14.639 --> 00:34:19.079 I want to run it with this topology like these clients and these clients, 418 00:34:19.480 --> 00:34:24.280 and it works remarkably well. It works remarkably well as it's it's a 419 00:34:24.360 --> 00:34:29.159 large degree of complexity. The Beast. I don't know what it would look 420 00:34:29.239 --> 00:34:31.960 like if we like unfolded it into the Beast script scenario that you described, 421 00:34:34.280 --> 00:34:37.840 it would be very remarkable. But now this thing it runs, if it 422 00:34:37.920 --> 00:34:43.000 runs the UH, you know, you can one click replicated Thethereum main that 423 00:34:43.199 --> 00:34:47.320 on your laptop or inside of one of your cloud boxes or Recubernators cluster at 424 00:34:47.360 --> 00:34:51.280 scale. UH, so you can kind of hit the dev all the way 425 00:34:51.360 --> 00:34:58.440 to late stage testing performance testing. UH. There was a team that built 426 00:34:57.679 --> 00:35:04.519 h They built it's basically like a sad path testing on top of this tool, 427 00:35:04.800 --> 00:35:08.079 so like a path network perturbin in lots of different ways, as you 428 00:35:08.119 --> 00:35:13.079 call it, chaos testing on top of this, and it takes you all 429 00:35:13.119 --> 00:35:16.960 the way almost to deploy the proud if you're going to make like a big 430 00:35:17.079 --> 00:35:22.880 change in the deep layers of the network. So that's that's the most That's 431 00:35:22.920 --> 00:35:25.960 the thing I'm the most proud of when it comes to like what are tools 432 00:35:25.960 --> 00:35:32.840 been able to do. There's a large degree of DevOps complexity encapsulated in that 433 00:35:32.920 --> 00:35:37.000 project, and then there's a bunch of people using it to do some really 434 00:35:37.039 --> 00:35:38.960 cool things on top of it. I feel like that makes a lot of 435 00:35:38.960 --> 00:35:43.719 sense there as well, Like when you have a lot of different testing scenarios 436 00:35:43.800 --> 00:35:49.280 that need to be captured and the configurability for them is also determined sort of 437 00:35:49.360 --> 00:35:53.079 by the constraints of the test, then you have a huge dimensionality of the 438 00:35:53.199 --> 00:35:58.320 data and configuration that you need to specify to actually execute it. There. 439 00:35:58.400 --> 00:36:00.440 Whereas I feel like, you know, some other environments that may not be 440 00:36:00.519 --> 00:36:02.719 total like always the case, but in this one, like, yeah, 441 00:36:02.760 --> 00:36:07.039 for sure, I can totally imagine, especially given the fact that it's there 442 00:36:07.119 --> 00:36:13.800 is no you know, official production environment where you know cloud provider where THEOEUM 443 00:36:13.840 --> 00:36:16.920 is running on exactly exactly, And it happens in cases where you're doing a 444 00:36:16.920 --> 00:36:22.800 lot of different testing things. And it also happens when you're I guess you 445 00:36:22.840 --> 00:36:27.480 could call it like a feature branch environment. It's it's these situations where you're 446 00:36:27.880 --> 00:36:32.519 you're like pre merge and you want to spin up a representation environment that's specific 447 00:36:32.559 --> 00:36:37.079 to you're not really deving on it because it's not inside it. You're like 448 00:36:37.440 --> 00:36:39.719 build and run loop as you're in your IDE but it's kind of close to 449 00:36:39.800 --> 00:36:44.119 like your death because they're not ready to put it into CI yet. That 450 00:36:44.280 --> 00:36:50.400 situation as well as one where inside of an organization or an open source community 451 00:36:50.639 --> 00:36:55.719 you might have hundreds of those a week of slightly different configurations, so it's 452 00:36:55.719 --> 00:36:59.079 either easy to configure or it's not. But if it's not, like this 453 00:36:59.239 --> 00:37:02.000 is a a great way to do it. This gives me the sort of 454 00:37:02.559 --> 00:37:08.199 PTSD flashbacks to the previous environment that I was working in where there was like 455 00:37:08.719 --> 00:37:14.760 different production environments that we had because they were for different customers or different internal 456 00:37:14.760 --> 00:37:19.719 business units, and then we had copies of them that were test environments that 457 00:37:20.000 --> 00:37:24.079 we're supposed to match the production one because realistically, given the ridiculous amount of 458 00:37:24.079 --> 00:37:28.760 configuration, you almost needed to run your tests in every single sort of clone 459 00:37:28.800 --> 00:37:31.159 of production however it could be possibly configured, and then of course it was 460 00:37:31.199 --> 00:37:36.480 a problem where like the actual switches from production didn't match where you're in your 461 00:37:36.519 --> 00:37:38.800 testing environment, and so you had not only did you have to run your 462 00:37:38.800 --> 00:37:43.519 test in all of these fake versions of production, but also make some changes. 463 00:37:44.039 --> 00:37:49.079 That for sure was quite the nightmare. And there's a mode of thought 464 00:37:49.159 --> 00:37:52.840 that I'm honestly a fan of this movement. There's a mode of thought where 465 00:37:52.880 --> 00:37:58.719 you just say, can we forego all of that and just make shipping the 466 00:37:58.760 --> 00:38:01.079 production as safe as possible and do all of it like test in production. 467 00:38:01.840 --> 00:38:06.199 That motive thought is like sometimes possible, I think. I think that's like 468 00:38:06.360 --> 00:38:10.880 Honeycomb's big big model. They're an observability tool like test and production. We're 469 00:38:10.880 --> 00:38:15.840 just going to give you really good monitoring tooling so the moment something goes wrong, 470 00:38:15.920 --> 00:38:20.079 you can catch it right away. For an impacts your entire customer base 471 00:38:20.199 --> 00:38:25.119 rollback like some some some products I guess can do this. Otherwise a tiny 472 00:38:25.159 --> 00:38:30.119 little bug is a disastrous like medical tooling. I don't know, Like I 473 00:38:30.119 --> 00:38:32.960 mean, I want to do it for sure, sign me up. I 474 00:38:34.039 --> 00:38:37.719 mean we're in the at authors with the five nine SLA where for sure we 475 00:38:37.760 --> 00:38:43.119 can't let stuff go through. I think the biggest challenge is depending on what 476 00:38:43.119 --> 00:38:45.039 your business is actually like, what the business logic looks like. The fear 477 00:38:45.079 --> 00:38:50.239 I've always had with these sort of automatic rollbacks in a lot of cases where 478 00:38:50.280 --> 00:38:53.079 what happens if the problem is compounded by changing the database version? And I 479 00:38:53.079 --> 00:38:58.280 always keep getting back to the Night Capital four hundred and forty million dollars loss 480 00:38:58.320 --> 00:39:01.880 in forty five minutes, kind of like the ridiculous thing is they had the 481 00:39:01.920 --> 00:39:06.639 wrong version of the software only running on one machine, and they made it 482 00:39:06.719 --> 00:39:10.679 worse by restarting the cluster. Like if they hadn't restarted anything, it would 483 00:39:10.679 --> 00:39:14.719 have been fine. And they assumed that the problem was with the rollout, 484 00:39:14.760 --> 00:39:19.079 so they rolled back everything and then also compounded the issue. And it's those 485 00:39:19.119 --> 00:39:22.360 things that really scare me, and I like, I would so totally before 486 00:39:22.800 --> 00:39:27.320 if the automatic rollback could calculate whether or not the you know, the database 487 00:39:27.360 --> 00:39:30.280 is that fault and also include that, And I feel like I'm not going 488 00:39:30.360 --> 00:39:34.320 to see that in my lifetime. I don't know. To me, it 489 00:39:34.360 --> 00:39:37.760 just sounds like we're making excuses to keep from giving people VIM access to the 490 00:39:37.760 --> 00:39:42.960 production server. I mean, you know you're I think in control of that 491 00:39:43.039 --> 00:39:45.880 actually had poligon right well, like you could just go and turn that on 492 00:39:45.000 --> 00:39:49.280 and let that happen and it could be like it could be it could be 493 00:39:49.320 --> 00:39:53.239 the real case study. You could do that right shop, whether it was 494 00:39:53.280 --> 00:39:55.639 a success. You know, published that blog post. You know that you're 495 00:39:55.679 --> 00:40:00.280 doing this, why it's so great, and you know you could get on 496 00:40:00.320 --> 00:40:04.800 board with it, and then next week's episode will be reviewing my resume because 497 00:40:04.840 --> 00:40:12.440 I'm looking for a job. I'll never forget that article that was published by 498 00:40:12.559 --> 00:40:15.719 get lab about how they you know, accidentally deleted their production database. Uh 499 00:40:15.800 --> 00:40:20.039 it was like a decade ago. Now. I was like, oops, 500 00:40:20.039 --> 00:40:23.079 Like I ran this command and I thought I was on the replication uh server, 501 00:40:23.119 --> 00:40:32.000 and I was actually in production. I've done that to do. Yeah, 502 00:40:32.280 --> 00:40:37.840 it was in AWS, and I thought that I was on the on 503 00:40:37.880 --> 00:40:45.360 the staging environment and uh just dropped an entire database cluster. Oh that that's 504 00:40:45.360 --> 00:40:47.840 not that bad, right, I mean, you just restore that from from 505 00:40:47.880 --> 00:40:52.159 the one that you you know, got snapshotted from an hour ago. Like 506 00:40:52.159 --> 00:40:54.559 it's much worse when you, like, you know, write that update statement 507 00:40:54.639 --> 00:40:59.280 in sequel and forget the select that comes after it, right, you know, 508 00:40:59.280 --> 00:41:01.760 people say you should write that first, like write the from statement first 509 00:41:01.800 --> 00:41:06.480 and then change your select to an update. Yeah, it's just it's ridiculous 510 00:41:06.519 --> 00:41:09.000 that those mistakes can be made. But I think at every company I've been 511 00:41:09.039 --> 00:41:12.440 at, there's been a special name for that, like, oh, don't 512 00:41:12.480 --> 00:41:15.280 eighty four the database, because like that's just happened to be what the update 513 00:41:15.480 --> 00:41:21.320 was, where someone didn't configure the restriction correctly and now there's some column with 514 00:41:21.400 --> 00:41:24.000 this wrong value in it and it's so noticeable too, right, Like it's 515 00:41:24.000 --> 00:41:27.920 not like, oh, database is gone, let's fix that. It's oh, 516 00:41:27.960 --> 00:41:31.039 when did someone change the name of all of our customers to eighty four? 517 00:41:34.039 --> 00:41:36.760 I mean that's kind of like a badge of honor though, like if 518 00:41:36.760 --> 00:41:39.960 you were if you were the person who implemented eighty four, like, would 519 00:41:39.960 --> 00:41:46.360 you put that on your LinkedIn? So yeah, I'm that guy. I 520 00:41:46.400 --> 00:41:52.280 think that was my biggest fear when I started my engineering career was doing one 521 00:41:52.320 --> 00:41:59.119 of these things like the terrified of touching anything that could look and feel something 522 00:41:59.199 --> 00:42:02.719 like deleting a duction database. And I think it is about your honor. 523 00:42:02.719 --> 00:42:07.000 It's the moment when you realize, like, once you do something like this, 524 00:42:07.440 --> 00:42:12.199 hopefully to a smaller scale, you kind of learn like this is a 525 00:42:12.239 --> 00:42:16.280 process issue, like this should have this should have never I never should have 526 00:42:15.480 --> 00:42:19.400 been able to do it. Shouldn't have been a risk that I would do 527 00:42:19.519 --> 00:42:22.760 this. I don't. But when you start out, you feel you feel 528 00:42:22.880 --> 00:42:28.760 I'm gonna get fired if I need one big wrong. Yeah, and it's 529 00:42:28.760 --> 00:42:30.519 one of those things. I don't think you can communicate that. I think 530 00:42:30.559 --> 00:42:37.239 the only way to learn that is to to earn that badge. This is 531 00:42:37.280 --> 00:42:39.480 a failure everyone has to go through in your life. You know, suffer 532 00:42:39.519 --> 00:42:44.960 that trauma. You'll live with the PTSD from that moment on about how the 533 00:42:45.000 --> 00:42:47.079 more the larger system, because you know you've gained more experience at this time, 534 00:42:47.119 --> 00:42:51.760 you're in a more important position. Your impact to the company is obviously 535 00:42:51.840 --> 00:42:54.320 going to be higher. You'll never make that same mistake again. Yeah, 536 00:42:54.400 --> 00:42:59.920 it's like riding a bike, Like you don't really understand the pros and con 537 00:43:00.119 --> 00:43:02.639 of riding a bike until you face plant into the side of your house. 538 00:43:02.840 --> 00:43:07.719 You're like, oh, gotcha, breaks, it was a mailbox for me. 539 00:43:14.360 --> 00:43:21.800 Cool. So let's talk about like we talk about mimicking the production environment, 540 00:43:21.800 --> 00:43:27.920 and production environments are always changing with you know, updated code and updates 541 00:43:27.960 --> 00:43:32.440 to the database and sometimes breaking changes, especially like since we've been talking about 542 00:43:32.480 --> 00:43:37.840 databases, like breaking schema changes with cartosis. How do you make sure that 543 00:43:37.880 --> 00:43:45.760 those production changes get integrated back into the local dev tool environments. Yeah, 544 00:43:45.280 --> 00:43:50.800 right now, that's something that's something the end user has to configure. We're 545 00:43:50.840 --> 00:43:54.519 looking into ways to get this more to provide like a product doze, way 546 00:43:54.679 --> 00:44:00.880 to automatically get this. But right now, when it comes to reduction change, 547 00:44:00.920 --> 00:44:04.360 my dev environment has to update to it. You got to. You 548 00:44:04.440 --> 00:44:08.920 gotta have a process in place to check out updating the configurations. Ideally, 549 00:44:08.960 --> 00:44:14.320 what you do is you any time that you make a production change, it 550 00:44:14.360 --> 00:44:19.960 goes through your krotosis set up first. That's the ideal scenario, but it's 551 00:44:20.000 --> 00:44:23.159 not always going to happen that way, of course. Yeah. No, 552 00:44:23.280 --> 00:44:29.039 I mean I imagine that you're making the database changes in your source code anyway 553 00:44:29.079 --> 00:44:32.960 to run a migration, if it's schemed or schema less you're making the code 554 00:44:34.039 --> 00:44:38.199 changes that are associated, and you would just you can just refresh your dev 555 00:44:38.280 --> 00:44:42.360 environment, like you know, blow it away and start fresh, because the 556 00:44:42.679 --> 00:44:45.119 migration should just not even need to be run potentially and it would just be 557 00:44:45.159 --> 00:44:50.360 the same as production. Well at least that would be my hope. Yes, 558 00:44:50.760 --> 00:44:58.880 yes, it's it gets goofy when people do things like like hot fixes, 559 00:44:58.880 --> 00:45:05.480 where like other people hot fixed stuff and you never looked at it. 560 00:45:05.480 --> 00:45:09.840 It can get goofy. But happy past scenario, it's not that hard to 561 00:45:09.840 --> 00:45:13.639 get it back up to. You know, these were these experiences are important. 562 00:45:13.639 --> 00:45:16.760 I remember having to do a production fix and at the time that meant 563 00:45:16.760 --> 00:45:22.920 that a different team created a SPN feature branch for us. It was a 564 00:45:22.960 --> 00:45:28.039 hot patch of production, and I ended up with this problem where there was 565 00:45:28.159 --> 00:45:31.639 code that should have worked in production, but it didn't play nicely with what 566 00:45:31.760 --> 00:45:34.639 was there, and you know, I compared it to what was actually in 567 00:45:34.920 --> 00:45:37.639 and Devin, it didn't match. And what I had found out is that 568 00:45:37.679 --> 00:45:39.719 there could be multiple production fixes that had to be done at the same time, 569 00:45:40.039 --> 00:45:44.519 and merging in SVN didn't really work that well. And what happened is 570 00:45:44.519 --> 00:45:47.559 is you would get merge conflicts there, and this team that was not the 571 00:45:47.599 --> 00:45:52.480 development team or knew how the feature should actually work, was going and actually 572 00:45:52.880 --> 00:45:58.480 doing the merge conflict resolution themselves and pushing that out. And that was a 573 00:45:58.559 --> 00:46:05.400 very horrifying learning experience for me. I think I'd rather drop the production database. 574 00:46:07.119 --> 00:46:09.320 I mean, you don't even have the code to look at to know 575 00:46:09.360 --> 00:46:12.960 what the issue was, because like I had to go find these other branches 576 00:46:12.960 --> 00:46:15.519 that weren't merged to production and like merge them into my code and see like 577 00:46:15.639 --> 00:46:20.000 it still wasn't right, Like where is this? Where is this? Wrong? 578 00:46:20.079 --> 00:46:22.599 Code just didn't exist anywhere. And then of course you have like the 579 00:46:22.639 --> 00:46:27.760 incident response team coming to your desk because in these days, of course you 580 00:46:27.800 --> 00:46:30.519 had to be in the office and like asking you, why are we losing 581 00:46:30.559 --> 00:46:35.840 one hundred thousand dollars every minute because of this problem? I'm like, I 582 00:46:35.880 --> 00:46:45.280 have literally no idea. Yeah, that's always cool too, That's just another 583 00:46:45.320 --> 00:46:51.840 fun experience of being able to quantify the cost of downtime, like, oh 584 00:46:51.880 --> 00:47:00.480 wow, this is serious. I feel bad enough to work like twenty four 585 00:47:00.480 --> 00:47:02.199 hours a day, seven days a week, and you'll never be able to 586 00:47:02.199 --> 00:47:07.719 wait from your computer in case something happens there. Right, that's the solution 587 00:47:08.239 --> 00:47:12.840 four seven with VIM on the production server. Yeah, it's just going to 588 00:47:12.960 --> 00:47:15.599 keep you know, if you keep bringing that up like I'm gonna, I'm 589 00:47:15.599 --> 00:47:17.239 going to write down this in my in my notebook, you know, see 590 00:47:17.239 --> 00:47:22.159 when Will got this product ready to go, where we'll let you you know 591 00:47:22.400 --> 00:47:24.559 it uses maybe s s H under the hood or your ID you know, 592 00:47:24.760 --> 00:47:29.840 just you called VIM and production you can even you know it even it lends 593 00:47:29.880 --> 00:47:36.280 itself a name VIP, right, and we'll see where that goes. Right, 594 00:47:36.599 --> 00:47:38.320 I'm just going to keep campaigning until they let me give a talk at 595 00:47:38.360 --> 00:47:43.480 like platform con about how to do this. Well, you know, how 596 00:47:43.519 --> 00:47:46.840 to get accepted to those conferences. What you do is you first just say 597 00:47:46.880 --> 00:47:51.679 you've done it, and and then if they get accepted, then you'll have 598 00:47:51.679 --> 00:47:54.599 to come with the whole product and actually try it out and I then present 599 00:47:54.679 --> 00:48:05.000 it were full of useful tips today. I mean I really I really like 600 00:48:05.039 --> 00:48:09.719 this idea of getting a tool in the mix here for helping build the the 601 00:48:09.719 --> 00:48:16.480 the environment for real. Yeah, and if you like to help illustrate the 602 00:48:17.119 --> 00:48:21.039 scale of this, you know, with your success with the Ethereum foundation, 603 00:48:21.239 --> 00:48:28.440 that's a monster project with completely open source, you know, and so I'm 604 00:48:28.440 --> 00:48:30.440 assuming that like if you want to if like, if I wanted to contribute 605 00:48:30.480 --> 00:48:39.639 to the Ethereum project, I could laugh and Warren, I think I'm going 606 00:48:39.679 --> 00:48:43.440 to ask for access to, you know, update this tool, and I'd 607 00:48:43.440 --> 00:48:45.039 be like, well, you know, will you know, making changes to 608 00:48:45.119 --> 00:48:52.599 it? I'm not. I'm not sure I know. But if I wanted 609 00:48:52.599 --> 00:48:57.920 to to contribute to Ethereum, like I could use crytosis to build my local 610 00:48:58.000 --> 00:49:01.400 div environment which helps me write the code, test it, and then create 611 00:49:02.320 --> 00:49:07.840 a poll or open a poll request that's going to be more likely to meet 612 00:49:07.960 --> 00:49:13.880 their contribution guidelines. Is that a fair statement? It's I mean, it 613 00:49:13.920 --> 00:49:17.000 probably is the easiest way to do that, Yeah, yeah, versus the 614 00:49:17.079 --> 00:49:23.719 alternative prior to because I worked with Ethereum before they started at adopting cretosis, 615 00:49:23.800 --> 00:49:29.719 like prior to that, you know, it was like twenty or thirty steps 616 00:49:29.760 --> 00:49:32.239 to get this thing running, you know, where you were downloading from all 617 00:49:32.280 --> 00:49:36.440 these different repositories, and then you were compiling it, and then you had 618 00:49:36.440 --> 00:49:40.599 to figure out in the config files which parts you had to change to make 619 00:49:40.679 --> 00:49:44.519 these different things that you just compiled, but you weren't really sure what they 620 00:49:44.559 --> 00:49:49.679 were talk to each other and so yeah, this like having this type of 621 00:49:49.719 --> 00:49:52.800 solution makes it easier, and that could be that could be another benefit for 622 00:49:52.840 --> 00:49:58.079 you, especially if you're an open source project, making it easier for people 623 00:49:58.119 --> 00:50:05.239 to contribute to your product. There's actually a good rule of thumb for when 624 00:50:05.360 --> 00:50:10.079 an open source project or community would would benefit from from using for dosis, 625 00:50:10.079 --> 00:50:14.400 and it's when you go to the docks and it says how to run locally 626 00:50:14.719 --> 00:50:17.719 if it's like the longer that thing is, the better you're going to get. 627 00:50:20.400 --> 00:50:22.480 It's kind of the general rule of thumb is how long is that thing? 628 00:50:23.400 --> 00:50:25.880 So yeah, for the ethereum, it was, it was huge. 629 00:50:27.239 --> 00:50:30.039 There's other general rule of thumb is that that that that's pretty much how it 630 00:50:30.039 --> 00:50:34.639 works, uh and and it is help work for contributing. It's most of 631 00:50:34.679 --> 00:50:40.199 the people, most of our implementations are open source entirely, so it's you 632 00:50:40.199 --> 00:50:45.559 know, it's it's people contributing back to those uh, people that use them, 633 00:50:45.719 --> 00:50:49.199 putting things back, getting more options available. And that really really helps 634 00:50:49.239 --> 00:50:53.599 because our team doesn't have time to handle all of the different scenarios. UH. 635 00:50:53.960 --> 00:50:59.239 When we start building examples and everything. So having people contribute back is 636 00:50:59.519 --> 00:51:04.199 I think the the Ethereum package, it's called a package, the definition of 637 00:51:04.239 --> 00:51:07.000 how to spin up an environment, the Ethereum package. It really is a 638 00:51:07.039 --> 00:51:10.840 remarkable work of open source contributions coming back. They're very, very engaged users. 639 00:51:12.679 --> 00:51:15.559 I think you touched on this a little bit earlier on about first run 640 00:51:15.599 --> 00:51:17.000 the licensing model and then and then you change off to that. Like if 641 00:51:17.079 --> 00:51:22.000 someone's going to get started today with this, is it I install a CLI 642 00:51:22.159 --> 00:51:24.800 and write a configuration file. Is there a hosted version? Like on these 643 00:51:24.880 --> 00:51:30.400 packages available and you'll deploy them automatically to my cloud environment. What does this 644 00:51:30.400 --> 00:51:37.119 look like? There's actually two there's two versions right now. The standard way 645 00:51:37.199 --> 00:51:40.400 is download the Clyde the a homebrew or app get or whatever is the preferred 646 00:51:40.719 --> 00:51:45.840 installation thing. Most people who start, they're starting because they see a package 647 00:51:45.880 --> 00:51:50.079 they want to run. So so most people they're just running a package when 648 00:51:50.079 --> 00:51:53.320 they start to see how it spins up, and then they start modifying that 649 00:51:53.360 --> 00:51:55.639 package if they want to modify it, or they're like, hey, this 650 00:51:55.719 --> 00:51:59.760 is cool, I'm going to brite my own my own system, and it's 651 00:51:59.800 --> 00:52:01.840 all all happens on your own laptop and you're own hardware, and you just 652 00:52:01.920 --> 00:52:07.199 you just run with it and go. And then we do have a cloud 653 00:52:07.679 --> 00:52:10.280 right now that we host where Hey, let's say you don't want to down 654 00:52:10.679 --> 00:52:14.440 the CLI and you don't want to be using your own compute to be running 655 00:52:14.760 --> 00:52:17.400 the environment, you can just button click. It's like a no code interface 656 00:52:17.400 --> 00:52:24.239 to run any existing package. There is also no code interface for modifying the 657 00:52:24.280 --> 00:52:28.679 package to a certain degree. That's evolving, so you can get more and 658 00:52:28.719 --> 00:52:32.679 more power over modifying it. And then once you're excited about it, you 659 00:52:32.679 --> 00:52:36.079 can write your own package and then you can deplay that there as well. 660 00:52:37.880 --> 00:52:43.519 And then the packages are those the ones that end up showing up in the 661 00:52:43.639 --> 00:52:47.800 Kurtosis catalog. Yes, yeah, because the Curtosis catalog is actually really cool. 662 00:52:47.840 --> 00:52:52.480 Specifically, you know, if you going back to the Ethereum example, 663 00:52:52.639 --> 00:52:57.639 you know, you can just grab that from the catalog and it runs for 664 00:52:57.679 --> 00:53:00.480 you, right, and then there's there's actually quite a few out there in 665 00:53:00.519 --> 00:53:04.440 your catalog. One that I happened to know of is the Polygon CDK as 666 00:53:04.480 --> 00:53:07.280 of last week, just went up in the catalog. So if you want 667 00:53:07.280 --> 00:53:10.159 to run that, you know, just grab it from the catalog and there 668 00:53:10.199 --> 00:53:14.199 you go. Interesting. Yeah, they're right there. And and what the 669 00:53:14.400 --> 00:53:16.880 I mean the way the catalog works right now is it scans public GitHub. 670 00:53:16.960 --> 00:53:22.679 So if it's an open source package, it's going to get detected and populated 671 00:53:22.719 --> 00:53:25.480 into the catalog and then you can just run it. Because it's an open 672 00:53:25.559 --> 00:53:30.880 source, it's an open source tool. So the Polygon CDK, for example, 673 00:53:31.320 --> 00:53:35.480 the Polygon team put a lot of work into to get into the package 674 00:53:35.559 --> 00:53:38.559 very in a very nice format, and now it's there and you can just 675 00:53:38.599 --> 00:53:45.159 button click run it. And one of the most exciting things about having made 676 00:53:45.159 --> 00:53:49.760 this step of opening everything is that we're seeing stuff happen without us knowing it 677 00:53:49.880 --> 00:53:52.079 or tracking it too closely, so like, oh, look like look at 678 00:53:52.119 --> 00:53:54.920 what the Polygon team has done. This is amazing And that's very exciting. 679 00:53:57.280 --> 00:54:00.920 Yeah, for sure. And I think that's because y'all are completely open source 680 00:54:01.039 --> 00:54:07.599 as well. Right, So the tool is actually the BSL license, okay, 681 00:54:07.000 --> 00:54:15.280 source available. You can do everything except for recreate our right on it, 682 00:54:15.639 --> 00:54:22.280 which seems like a fair constrained exactly. Yeah, no, but I 683 00:54:22.320 --> 00:54:29.719 think that's really cool having the catalog there, especially with the open source communities, 684 00:54:30.760 --> 00:54:35.039 and it seems that Kurtosis is really resonating there to make it easy for 685 00:54:35.119 --> 00:54:40.280 people to just run and then contribute and then improve your product and in a 686 00:54:40.320 --> 00:54:44.800 consistent format, you know, because I think that's one of the biggest challenges 687 00:54:44.800 --> 00:54:52.119 of running an open source project is verifying or vetting out the contributions that you 688 00:54:52.199 --> 00:54:58.239 get to make sure that they have done in some cases just tribal knowledge. 689 00:54:59.079 --> 00:55:02.360 Yes, yes, yes. And it's one thing we were noticing, especially 690 00:55:02.360 --> 00:55:07.400 as we work with more and more people who have cloud dependencies, is the 691 00:55:07.440 --> 00:55:14.280 more that your system is is composed of open source components, the more this 692 00:55:14.360 --> 00:55:19.880 problem is important to you because you're you're dealing with a launch of different software 693 00:55:19.880 --> 00:55:23.400 components built by different organizations, where where you can run that entire software component 694 00:55:23.440 --> 00:55:27.840 yourself. That's the whole point of the open source So you don't have the 695 00:55:28.000 --> 00:55:32.039 issue of you know, cloud provider run a production environment has zero incentive to 696 00:55:32.039 --> 00:55:36.079 like make you able to like modify it or move it. In the open 697 00:55:36.079 --> 00:55:38.239 source communities, not the case, you actually can do it entirely. So 698 00:55:38.639 --> 00:55:45.320 the more open source your tool is in terms of like the percentage of services 699 00:55:45.320 --> 00:55:49.199 that are involved in your environment which are open source. The more the more 700 00:55:49.239 --> 00:55:54.199 you get this benefit of like how do we verify that the contribution is going 701 00:55:54.280 --> 00:55:57.599 to work and makes sense? And you do kind of have to run it 702 00:55:57.639 --> 00:56:00.519 all to see it. Yeah, I would even take it further, honestly 703 00:56:00.679 --> 00:56:07.360 that most open source solutions out there that aren't using a strategy or have a 704 00:56:07.400 --> 00:56:12.480 strategy for easy for contributors to come in and run the software would benefit a 705 00:56:12.480 --> 00:56:15.199 lot from a strategy like this, because realistically, if you go out there 706 00:56:15.239 --> 00:56:20.199 and find an open solution, no matter what it is, even one used 707 00:56:20.199 --> 00:56:23.599 by thousands of organizations, it is quite the challenge to get that up and 708 00:56:23.679 --> 00:56:28.280 running. Even if it's a darker container, even if it's you know, 709 00:56:28.320 --> 00:56:30.360 in a registry like it's still there's a lot that goes on with it. 710 00:56:30.400 --> 00:56:37.760 I remember it was it was even recently that you didn't need a zookeeper anymore 711 00:56:37.800 --> 00:56:42.199 to run Kafka, Like it was always this sort of mess that if you 712 00:56:42.239 --> 00:56:45.719 ever wanted to run Kafka yourself, you needed three different services running on your 713 00:56:45.960 --> 00:56:51.840 machine with multiple nodes and clusters at the same time in order to even test 714 00:56:51.880 --> 00:56:55.000 this out. And I'm like, what, yes, yes, it's a 715 00:56:55.039 --> 00:56:59.440 big issue, and it's and it's for the most part, in my opinion, 716 00:56:59.480 --> 00:57:04.039 it's a bit of a silent issue because if you're contributing back to an 717 00:57:04.039 --> 00:57:07.159 open source project, and a lot of the time you're doing it out of 718 00:57:07.880 --> 00:57:10.559 almost like a hobby mentality. It's like, I want to help this project, 719 00:57:10.599 --> 00:57:14.920 I'm interested in it. If there's a really big friction bar for me 720 00:57:15.000 --> 00:57:17.039 to do that, I'm not getting paid to contribute back to it. That's 721 00:57:17.079 --> 00:57:22.360 the whole idea. So I'm not going to put an inorder the amount of 722 00:57:22.400 --> 00:57:25.639 effort into knocking down the door to say, like make my testing environment better, 723 00:57:25.760 --> 00:57:29.639 like make my environment better, I'm just going to disappear. So like 724 00:57:29.679 --> 00:57:35.159 the amount of open source contributions that you're missing because because people just got frustrated 725 00:57:35.159 --> 00:57:37.519 in the first five minutes and say, you know what, like someone else 726 00:57:37.599 --> 00:57:42.559 is going to handle this contribution at some point hopefully is I can just imagine 727 00:57:42.559 --> 00:57:45.440 the velocity that you're losing there. I mean, I hope that's really the 728 00:57:45.760 --> 00:57:50.920 case, that that's even where they're falling off. I find that in Unfortunately, 729 00:57:50.960 --> 00:57:52.400 the state of the open source today is like you find an issue out 730 00:57:52.440 --> 00:57:55.719 there, or you want to even make a change and you may open an 731 00:57:55.719 --> 00:58:00.000 issue or open that poor request and just never hear back. And with stuff 732 00:58:00.000 --> 00:58:05.119 like this, you know that the team has invested in what that experience is 733 00:58:05.239 --> 00:58:07.320 on building stuff up. So it could be a badge of honor really to 734 00:58:07.360 --> 00:58:13.199 say this open source project is using kurtosis or another tool that makes this experience 735 00:58:13.239 --> 00:58:15.719 better because you know that you thought about it, that they care about getting 736 00:58:15.760 --> 00:58:24.679 those contributions. Then, so, is there is there a thought pattern behind 737 00:58:24.679 --> 00:58:29.519 you, Like when we talk about production servers with the whole DevOps movement, 738 00:58:30.519 --> 00:58:36.719 we kind of came up with this philosophy of servers are cattle not petch, 739 00:58:36.800 --> 00:58:39.599 you know, referring to the fact that they should be short lived and you 740 00:58:39.639 --> 00:58:45.159 shouldn't have emotional attachment tool them. You shouldn't be like hand curating them. 741 00:58:45.440 --> 00:58:52.559 But I feel like dev environments maybe didn't fall under that umbrella. Is there 742 00:58:52.800 --> 00:58:59.559 like a thought process or something that you've noticed through critosis where you kind of 743 00:58:59.760 --> 00:59:05.400 see people shifting away from the long lived development environments because it's a little easier 744 00:59:05.599 --> 00:59:10.000 and standardized to bring them up. Now, it's I don't know if I 745 00:59:10.039 --> 00:59:14.480 would say that it's a shift away from it. It's in some ways it 746 00:59:14.559 --> 00:59:19.880 is for certain use cases, like so how could I describe this? What 747 00:59:19.920 --> 00:59:24.320 one thing crtosis does is if you run your integration tests in CI and cider 748 00:59:24.360 --> 00:59:29.440 crtosis, it's quite easy to rerun that locally in your laptop. So if 749 00:59:29.480 --> 00:59:35.320 you want to debug or failing integration and the end test, rather than recreating 750 00:59:35.320 --> 00:59:37.760 it in a remote cluster that's like a shared cluster, you would just rerun 751 00:59:37.800 --> 00:59:45.079 it on your laptop. And in that case there's a shift towards ephemeral environments, 752 00:59:45.320 --> 00:59:49.280 like you're just for depth, like you're just running vision. But there's 753 00:59:49.320 --> 00:59:52.840 also people who are like, I'm going to use kryptosis to set up a 754 00:59:52.880 --> 00:59:59.119 persistent development environment because the other deaths of my team it's just significantly easier for 755 00:59:59.159 --> 01:00:01.960 them to ping an endpoint that's sitting there stable and to get into that. 756 01:00:02.119 --> 01:00:07.239 Then even to like download a tool and then re run some commands every single 757 01:00:07.280 --> 01:00:10.800 time, and and then you get into the problem of like data management. 758 01:00:10.920 --> 01:00:17.199 So if you've invested in your dev environment, having a particular data set behind 759 01:00:17.239 --> 01:00:22.199 it that's useful for that dev project, and then investing the extra effort into 760 01:00:22.320 --> 01:00:25.960 cloning and moving that data for an ephemeral use, it often doesn't make sense 761 01:00:27.000 --> 01:00:31.400 you just use the same thing. So I think it really is It really 762 01:00:31.440 --> 01:00:40.119 is dependent on the use case. But there definitely is a meaningful difference between 763 01:00:40.280 --> 01:00:45.039 how you treat your dev servers and your broad servers. So like the cattle, 764 01:00:45.119 --> 01:00:47.800 the cattle not pets thing. I think it's totally fine for your dev 765 01:00:47.840 --> 01:00:52.519 server to be a pet. Like it's there's there's there's times when it makes 766 01:00:52.519 --> 01:00:54.519 a lot of sense. The moment you want to see that it looks like 767 01:00:54.559 --> 01:00:58.159 braud, you do kind of want to turn it into a cattle just to 768 01:00:58.199 --> 01:01:00.079 see how it works. But there's a lot of telling reasons to treat it 769 01:01:00.119 --> 01:01:07.760 like a pet, especially if you are debugging an issue or trying to figure 770 01:01:07.800 --> 01:01:10.920 out the way things kind of work. You want to have that state sitting 771 01:01:10.960 --> 01:01:15.719 around, and you want to really preserve that because it's going to take you 772 01:01:15.760 --> 01:01:19.639 some time thinking and creative time to work with it. And if it's a 773 01:01:19.719 --> 01:01:23.239 cattle and you lose it halfway through your work, it's miserable. You got 774 01:01:23.239 --> 01:01:29.000 to restart, right And I think I heard you say that there is a 775 01:01:29.679 --> 01:01:34.599 there's a use case here for using something like critosis to create a persistent dev 776 01:01:34.719 --> 01:01:39.119 environment that provides a common set of services for your dev team to access. 777 01:01:39.159 --> 01:01:44.840 So not everyone has to run that locally, but there's one that's centrally available 778 01:01:44.880 --> 01:01:50.920 for them that maybe has a specific configuration or has a something unique to it 779 01:01:50.960 --> 01:01:55.559 that everyone needs to be thinking about or addressing in their local development efforts. 780 01:01:55.599 --> 01:02:00.079 So you just provide that as a shared service for everyone. Yes, and 781 01:02:00.400 --> 01:02:06.679 that's useful for a lot of development workflows. But then there's another set of 782 01:02:06.679 --> 01:02:10.519 development workflows where you're going to say, I actually do want to isolate for 783 01:02:10.960 --> 01:02:16.719 an ephemeral purpose a particular set of micro services or a data set, and 784 01:02:16.760 --> 01:02:22.400 I want that to be my sandbox that nobody else is touching. And that 785 01:02:22.480 --> 01:02:23.480 happens a lot as well, because it's like, hey, what if three 786 01:02:23.519 --> 01:02:27.760 different teams want to use the dev cluster and they're all hitting each other, 787 01:02:27.920 --> 01:02:30.639 and then you have to go on Slack and say, hey, wait a 788 01:02:30.639 --> 01:02:31.440 minute, like, let me use it for the next two hours, you 789 01:02:31.519 --> 01:02:36.679 guys, when you blow away their work, and it's miserable, But that's 790 01:02:36.760 --> 01:02:40.199 only sometimes. So the best world is you have the choice. When you 791 01:02:40.280 --> 01:02:43.920 need to get the isolation, you can get it quite easily. When you 792 01:02:44.679 --> 01:02:47.400 don't need it, you can use other people's work. And that's the direction 793 01:02:47.440 --> 01:02:54.599 that we're trying to go towards, is how do we provide we like high 794 01:02:54.679 --> 01:02:58.519 high level. You want it to feel like you're in Harry Potter and you 795 01:02:58.519 --> 01:03:00.519 have a wand and you're just like that, you have the figure need Like 796 01:03:00.559 --> 01:03:05.239 that's the highest level user experience that you want to get to. It's like 797 01:03:05.239 --> 01:03:07.760 cool, Like, how do you design the want such that you can do 798 01:03:07.880 --> 01:03:10.960 that? And I think you just want to have the configurability to be able 799 01:03:12.000 --> 01:03:15.000 to pick which in this scenario, I want the shared duftbuster or I want 800 01:03:15.039 --> 01:03:21.960 the isolated if I'm one right on, So, continuing on that thought, 801 01:03:22.039 --> 01:03:24.800 what does the future of krotosis look like? What are the in addition to 802 01:03:24.840 --> 01:03:29.920 that, what are some of the things that you would like to see getting 803 01:03:30.000 --> 01:03:36.920 implemented? Absolutely, so I said before, it's right now. Crotosis operates 804 01:03:37.000 --> 01:03:42.679 to you give your terraform scripts or your open TOFU and it deploys your provisions, 805 01:03:42.719 --> 01:03:46.239 your hardware so to speak, and maybe your virtualization later like the communities 806 01:03:46.320 --> 01:03:52.679 Buster or Ductor Engine, and then Crotosis deploys on top of that. We 807 01:03:52.719 --> 01:03:55.079 want crtosis to move in the direction where you no longer have to do that 808 01:03:55.119 --> 01:03:59.440 first part. You no longer have to think about and dig into like how 809 01:03:59.440 --> 01:04:02.480 do I how do I split my my cloud assets? Over how do I 810 01:04:02.519 --> 01:04:06.280 provision my hardware? We want to get better and better and better at providing 811 01:04:08.679 --> 01:04:13.039 tools to not have to worry about that, so that that's where we're going. 812 01:04:13.920 --> 01:04:18.519 You know, the more that the folks in the blockchain space they use 813 01:04:18.599 --> 01:04:23.159 a lot of open source tools where it's just all can be containerized and you 814 01:04:23.199 --> 01:04:26.280 get everything in the container. The more that you're working with folks who have 815 01:04:26.360 --> 01:04:30.800 alliances on like a to bus services or manage authentication services and this kind of 816 01:04:30.800 --> 01:04:34.440 thing, the more that that abstraction is not so useful anymore because those are 817 01:04:34.440 --> 01:04:38.480 the issues, the issues of the cloud stuff. So the more that we 818 01:04:38.519 --> 01:04:41.599 can build an instruction there that captures that stuff in a way that's useful, 819 01:04:42.000 --> 01:04:44.519 that's where we want to go, and that's where we're doing all this research, 820 01:04:44.559 --> 01:04:47.440 and that's where that's where this thought experiment of like, well, to 821 01:04:47.559 --> 01:04:54.880 some degree you're limited by the cloud provider allowing you to easily get like everything 822 01:04:55.039 --> 01:04:59.440 worked over to some degree, have those limitations. So then, given that's 823 01:04:59.440 --> 01:05:01.039 the world we live in, how do we get as close as possible? 824 01:05:01.079 --> 01:05:03.679 And that's kind of the design work that we're looking at right now. Well, 825 01:05:03.719 --> 01:05:08.840 I think there's totally a gap here in the IAIC world where it doesn't 826 01:05:08.840 --> 01:05:13.679 feel like any of the offerings are super compelling. There's I mean, I 827 01:05:13.719 --> 01:05:20.440 think anyone that assigns themselves a DevOps title has a unique perspective on complaining about 828 01:05:20.480 --> 01:05:25.320 literally everything that exists. So you know, of course the tools that we 829 01:05:25.519 --> 01:05:29.000 have, you know, are not as good as they should be. And 830 01:05:29.119 --> 01:05:31.639 I think there's always a space for someone to say I wrote a better terror 831 01:05:31.639 --> 01:05:34.559 form or whatever, because I feel like that's sort of the abstraction layer that 832 01:05:35.159 --> 01:05:41.440 exists over the cloud providers. And I'm not saying that anyone should go and 833 01:05:41.480 --> 01:05:44.719 create a new is C tool. I'm sure someone just got that idea right 834 01:05:44.840 --> 01:05:47.239 when you can do that, you absolutely can do that and wrap the SDKs 835 01:05:47.280 --> 01:05:51.000 out there. But I do I do feel like that there's always there is 836 01:05:51.000 --> 01:05:56.800 definitely a room here to grow in that direction. Yeah, and then that's 837 01:05:58.159 --> 01:06:00.760 it's kind of a maybe a fork in the road. But the two flip 838 01:06:00.840 --> 01:06:04.840 philosophies is like create that layer so that it's very easy to do those forks. 839 01:06:05.079 --> 01:06:10.519 The other one is, I guess it's you call it like the honeycomb 840 01:06:10.679 --> 01:06:14.440 or the launch Darkly philosophy, where it's like, can you build tools that 841 01:06:14.559 --> 01:06:18.239 make production just safer, like feature flags, so now just don't worry about 842 01:06:18.239 --> 01:06:23.440 testing or like like so much. Just get into production under feature flag and 843 01:06:23.480 --> 01:06:26.480 then and then just turn it on for a certain set of customers and then 844 01:06:26.480 --> 01:06:30.599 you'll figure out how things work or like deserveability thing. Instead of focusing so 845 01:06:30.679 --> 01:06:33.960 much on like fourteen year staging and testing environment, you just go into production. 846 01:06:34.039 --> 01:06:38.000 We're going to monitor everything so closely that we're going to minimize any kind 847 01:06:38.000 --> 01:06:43.519 of damage. And there's I think there's gaps on both sides, and there 848 01:06:43.840 --> 01:06:47.599 the different engineering leaders and platform leaders have different opinions about about which way they 849 01:06:47.639 --> 01:06:54.360 want to go, so it's kind of a lots of different picks on. 850 01:06:54.800 --> 01:06:58.679 I mean, I definitely have an opinion on that. Realistically, it would 851 01:06:58.719 --> 01:07:00.400 be nice if we could get to that world. I don't think the cloud 852 01:07:00.400 --> 01:07:04.880 providers offer us the sort of control over the environment that would allow us to 853 01:07:04.960 --> 01:07:10.280 perform like one out of every million requests. Yeah, go and changes. 854 01:07:10.440 --> 01:07:15.639 I just feel like we're much better off pulling production down to another environment to 855 01:07:15.960 --> 01:07:20.400 validate that the code that we have is exactly executing as we want, and 856 01:07:20.440 --> 01:07:25.400 we can do it, you know, creating synthetic requests from our production environment 857 01:07:25.400 --> 01:07:30.599 and replicating that against the ephemeral, dynamically created dev one. This is a 858 01:07:30.679 --> 01:07:36.559 perfect example of what we're seeing is Warren who is like, let's get things 859 01:07:36.760 --> 01:07:41.519 synthetically mocked, Let's get everything running locally, Let's get it as close as 860 01:07:41.519 --> 01:07:44.400 possible, make sure things work, and then we have Will who wants to 861 01:07:44.440 --> 01:07:54.440 film Upbraud team VIM right here. I think valid approaches about valid approaches. 862 01:07:54.480 --> 01:07:58.159 People have different appetites what they want to do. No, I mean, 863 01:07:58.199 --> 01:08:01.519 there's still definitely things where I'm a huge proponent of pushing stuff to testing in 864 01:08:01.639 --> 01:08:05.760 production, and we do that under certain circumstances where we make sure that we 865 01:08:05.920 --> 01:08:10.119 wrapped. Like parallel execution is a huge thing for us. When we're changing 866 01:08:10.360 --> 01:08:14.079 some of the business logic of how something works, we'll usually execute two different 867 01:08:14.079 --> 01:08:18.840 paths actually simultaneously in production and validate whether or not the outcome is exactly as 868 01:08:18.840 --> 01:08:24.760 it should be, and before turning off the legacy one and when you do 869 01:08:24.840 --> 01:08:30.000 that, When you do that, does that output still go back to like 870 01:08:30.079 --> 01:08:35.159 your fork traffic or do you know it's in process? Like it'll basically the 871 01:08:35.159 --> 01:08:40.359 equivalent of having two threads. You know, you call this method and it 872 01:08:40.399 --> 01:08:43.159 does two things at the same time, and then we throw away the new 873 01:08:43.239 --> 01:08:45.880 result until we like we validate it, and then we throw it away and 874 01:08:45.960 --> 01:08:49.239 if we get any invalidations, then we you know, that's an alert that 875 01:08:49.279 --> 01:08:53.279 someone goes and looks at and comes back and says, Okay, you know 876 01:08:53.279 --> 01:08:56.680 what happened here. And if we don't get any alerts after a period of 877 01:08:56.720 --> 01:09:00.279 time, then that sort of level of testing has been completed and we can 878 01:09:00.359 --> 01:09:03.119 just delete the old code and own and actually return the new stuff. I 879 01:09:03.199 --> 01:09:06.199 love that setup. I think that's I think that's great. I think I 880 01:09:06.199 --> 01:09:12.199 think that's like decently. I mean, it's it's a far cry from like 881 01:09:12.279 --> 01:09:15.279 VIM and prade, but like it's almost a step towards that. Right. 882 01:09:15.359 --> 01:09:18.680 It's like like you're at least able to get something into production more safely, 883 01:09:18.920 --> 01:09:25.000 even though it's I imagine there's constraints on like you wouldn't be able to do 884 01:09:25.039 --> 01:09:27.920 that if there was like very secure data that was going into the response that 885 01:09:27.960 --> 01:09:29.880 you're like, you shouldn't see. You wouldn't be able to nidate it that 886 01:09:29.960 --> 01:09:31.239 well, well, I mean we we sort of had that because we're an 887 01:09:31.279 --> 01:09:36.680 identity provider federated and aggregation and no one sees the data. It's just you 888 01:09:36.680 --> 01:09:41.399 know, hey, there's something here that doesn't match, and you know, 889 01:09:41.439 --> 01:09:45.119 you can sort of revalidate that and realistically the actual who the user is doesn't 890 01:09:45.119 --> 01:09:47.760 really matter in those circumstances. You can just sort of see what it looks 891 01:09:47.840 --> 01:09:53.920 like. If for some reason there is some concern. We encrypt the logs 892 01:09:53.960 --> 01:09:57.920 that get written even before they leave the production environment. We hash the data 893 01:09:57.920 --> 01:10:01.439 that we'll get seen automatically by any of our engineers before it leaves. So 894 01:10:01.960 --> 01:10:06.520 I mean, very rarely it's the case that that actually, like what someone's 895 01:10:06.560 --> 01:10:12.039 last name was was relevant for us. That's a very cool setup. I 896 01:10:12.159 --> 01:10:16.880 like that. So that's some discipline there, though, Like that's very that 897 01:10:16.920 --> 01:10:21.159 sounds to me like that sounds like a very disciplined approach because you're being very 898 01:10:23.000 --> 01:10:30.000 deliberate about what you're testing, writing that second path for it, and then 899 01:10:30.039 --> 01:10:36.439 you're having to specifically define and measure what the success criteria for that is. 900 01:10:36.840 --> 01:10:42.399 So that's a charitable way of looking at it. You know, maybe that's 901 01:10:42.399 --> 01:10:46.239 my na You don't do that at all. I mean there was nobly a 902 01:10:46.279 --> 01:10:48.319 time in my life where I'm like, you know, you're releasing that code 903 01:10:48.359 --> 01:10:53.479 to production. Why is there an arbitrary try catch block you know around that? 904 01:10:54.279 --> 01:10:58.520 But there is another aspect here when if you're if you care about reliability 905 01:10:58.720 --> 01:11:01.439 and that your stuff don't really go down and production and the code you're changing, 906 01:11:01.600 --> 01:11:05.600 like there are too many unknowns there. Realistically, you didn't test literally 907 01:11:05.720 --> 01:11:11.720 every case, and sometimes it's not so critical that you have tested everything, 908 01:11:11.800 --> 01:11:15.560 but you know, validating something happens in production, like oh, we don't 909 01:11:15.560 --> 01:11:16.920 know if a customer is using this. Well you can at least create a 910 01:11:16.960 --> 01:11:20.560 log message that says, uh, hey, customer is using this thing, 911 01:11:20.640 --> 01:11:25.439 don't turn it off yet and run that for a bunch of weeks so that 912 01:11:25.479 --> 01:11:30.800 you have the data to validate. Uh yeah, I mean you laugh, 913 01:11:30.960 --> 01:11:34.399 but you're not if you don't take that first step, Like this is absolutely 914 01:11:34.439 --> 01:11:40.600 the advice that someone could be utilizing to improve their decision making apility, not 915 01:11:40.680 --> 01:11:44.600 like some spreadsheet with a list of features that are on or off per customer, 916 01:11:44.840 --> 01:11:46.840 but really, like what is happening in your production environment, and this 917 01:11:46.920 --> 01:11:49.560 is just the next level after that, which is release the code, have 918 01:11:49.640 --> 01:11:54.800 multiple versions running at the same time, measure performance, reliability, you know, 919 01:11:55.000 --> 01:11:59.760 consistency, whatever that is. This is just how we've happened to be 920 01:11:59.800 --> 01:12:02.680 doing and I feel like it works really well for us. Unfortunately, so 921 01:12:03.039 --> 01:12:05.520 you know, we're not going to migrate to something else. I don't know. 922 01:12:05.600 --> 01:12:10.000 I haven't heard anyone else doing something that you know really changes this. 923 01:12:10.199 --> 01:12:14.920 I mean there's blue green deployments right where you're siphoning traffic off from your new 924 01:12:15.399 --> 01:12:18.640 version. It sometimes it doesn't really make sense a lot when you have a 925 01:12:18.640 --> 01:12:24.439 lot of features going to production over time, where the features especially aren't interacting 926 01:12:24.439 --> 01:12:27.760 with each other. Having them weigh in a queue to get through this and 927 01:12:27.760 --> 01:12:30.479 then deciding to roll back or not is a huge expensive process. As you 928 01:12:30.560 --> 01:12:32.920 said, you know, feature flags can work here, although that's again a 929 01:12:32.960 --> 01:12:38.000 switch. So really parallel execution has been the only thing I can think of 930 01:12:38.079 --> 01:12:42.199 that really made sense for us early on. And how do you think about 931 01:12:42.279 --> 01:12:46.800 how that compares to like a canary environment. I mean, this is sort 932 01:12:46.840 --> 01:12:49.720 of the same thing, like when you if you have millions of requests that 933 01:12:49.760 --> 01:12:55.039 are that are coming into your service all the time, then some number of 934 01:12:55.039 --> 01:12:59.399 people would be affected by any change, no matter how small. And if 935 01:12:59.399 --> 01:13:01.039 you have the budget, which is you know, one of the key terms 936 01:13:01.039 --> 01:13:04.560 that we talk a lot about in observability, then you know, go for 937 01:13:04.640 --> 01:13:06.800 it, right, you know you don't need to take some extra steps. 938 01:13:08.079 --> 01:13:12.399 But if it's just a matter of you know, changing a bunch of parameters 939 01:13:12.439 --> 01:13:15.359 somewhere and you want to see how it validates it, especially when you have 940 01:13:15.399 --> 01:13:16.920 all the business logic on your side already, right, you know you're not 941 01:13:17.000 --> 01:13:20.960 calling a third party service, you're not doing some other things like if you 942 01:13:20.960 --> 01:13:25.800 can really test it all in your service in production at runtime, why not 943 01:13:26.399 --> 01:13:31.960 do this to validate? Oh was that hard to set up? I mean 944 01:13:32.000 --> 01:13:34.319 you really, like, you know, think of you're changing a method, 945 01:13:34.359 --> 01:13:38.159 you know, any any engineer out there. You're changing the internals of some 946 01:13:38.239 --> 01:13:42.800 method. All you need to do is duplicate that method down to a second 947 01:13:42.800 --> 01:13:45.920 block, call it V two and have the caller execute both of those things, 948 01:13:45.079 --> 01:13:48.079 wrap the other one in a trycatch block, and then validate the outputs 949 01:13:48.079 --> 01:13:51.239 in your code and if it doesn't work, you know, log a message 950 01:13:51.399 --> 01:13:55.880 and that's it's not It's honestly not a lot there. You can create some 951 01:13:55.920 --> 01:14:00.520 structure and around that with a framework where you can wrap you know, parallel 952 01:14:00.520 --> 01:14:04.399 execution and then pass it into methods and have it automatically do it. It's 953 01:14:04.439 --> 01:14:08.880 not a lot of extra overhead if you feel like you're doing this very very 954 01:14:08.920 --> 01:14:12.199 often. I think this happens, like one out of ten features that gets 955 01:14:12.199 --> 01:14:15.840 implemented goes in like this. It also gives us some security around it, 956 01:14:15.960 --> 01:14:20.079 so I wouldn't say it's complicated. I think a lot of engineers know that 957 01:14:20.760 --> 01:14:25.199 this is a great a way that you can test and this helps I think 958 01:14:25.279 --> 01:14:28.680 junior engineers, you know, they navigated this right, you know, throw 959 01:14:28.720 --> 01:14:31.479 a try catch around there. I'm scared about this affecting production. I think 960 01:14:31.640 --> 01:14:35.720 bringing that fear back in it, you know, helps us. You know, 961 01:14:35.760 --> 01:14:40.600 thinking that our code that we write is invincible is the problem here? 962 01:14:41.039 --> 01:14:45.399 Yes? Yes, So can you just share your screen for us? Warren, 963 01:14:47.199 --> 01:14:51.920 I'm kidding it's really boring right now. I mean, when I'm doing 964 01:14:51.920 --> 01:15:00.000 the podcast, I literally have a notes page and my audio equipment uli up 965 01:15:00.079 --> 01:15:02.199 in running and that's it. There's nothing else on my desktop right now. 966 01:15:03.760 --> 01:15:10.039 We are. We're completely different. I never I never closed windows, and 967 01:15:10.159 --> 01:15:16.279 never reboot. That's exactly what somebody who wants to be proud would say. 968 01:15:16.279 --> 01:15:20.359 Well, I saw a meme yesterday and it cracked me up, and it 969 01:15:20.439 --> 01:15:26.119 said, damn, I just accidentally closed that browser tab that I've been meaning 970 01:15:26.119 --> 01:15:30.199 to read for two years. It was very careful about my usage there that 971 01:15:30.239 --> 01:15:33.720 control shift T brings that up or the previous window that I've closed, you 972 01:15:33.760 --> 01:15:36.359 start all my tab Yeah, for sure. That. Like I was using 973 01:15:36.600 --> 01:15:42.399 x marks on Firefox back in the day before they bought into it, because 974 01:15:42.439 --> 01:15:45.880 it's just so critical for me that I can actually get that. The worst 975 01:15:45.880 --> 01:15:49.880 one I have is emails in my inbox because when you click archive accidentally or 976 01:15:49.960 --> 01:15:55.159 delete like at that email was months old. Good luck finding what that was. 977 01:15:58.359 --> 01:16:03.279 We have a security polow see that automatically deletes our emails after a specified 978 01:16:03.279 --> 01:16:15.680 period of time. So email email management is automated for me. Cool. 979 01:16:15.720 --> 01:16:25.399 So what well on critosis, have you found an area where it's not a 980 01:16:25.399 --> 01:16:32.239 good fit. Yes, definitely crotosis with where we are right now, if 981 01:16:32.439 --> 01:16:40.720 you are able to very adequately express your system with a doctor composed file or 982 01:16:40.760 --> 01:16:45.760 a HELM chart with a relatively small helm chart in the sense of like your 983 01:16:45.800 --> 01:16:47.319 helptot rebo doesn't have some dozen lines in it, it's got like, you 984 01:16:47.319 --> 01:16:53.560 know, the same number of configuration settings. If you have those two things, 985 01:16:53.640 --> 01:16:56.119 you're you're doing great. Krotosis is not going to be great for you. 986 01:16:56.439 --> 01:17:00.600 It's the situation where those things have to be augmented by the seven thousand 987 01:17:00.600 --> 01:17:04.000 of my mast script, the bunch of different documentations on how to configure these 988 01:17:04.039 --> 01:17:09.039 things. If your doctor post file has or your home charts have a bunch 989 01:17:09.079 --> 01:17:13.640 of lines commented out that say if you want this behavior, please uncomment these 990 01:17:13.680 --> 01:17:16.399 lines, and it's all over the place. That's where that's where you should 991 01:17:16.399 --> 01:17:19.359 think a look at prtosis uh, and that's where we're going to provide a 992 01:17:19.359 --> 01:17:30.520 lot of value at the moment. Right on excellent. So you mentioned doctor 993 01:17:30.560 --> 01:17:36.560 compose and HELM So I think I can safely infer there that you're a big 994 01:17:36.600 --> 01:17:44.359 favor of containerization. Yes. Is that specific or is that a result of 995 01:17:45.239 --> 01:17:47.760 like this is what you're you learned from your customers or from the beginning, 996 01:17:47.760 --> 01:17:54.359 where you just like containers all the way I mean, I would love to 997 01:17:54.399 --> 01:17:58.399 say it was entirely deliberate. Uh, and from like the very beginning. 998 01:17:59.079 --> 01:18:01.920 Definitely in the beginning it was like, hey, this is the standard that 999 01:18:01.920 --> 01:18:05.760 people are using to ship these things. I think it's a great abstraction. 1000 01:18:05.880 --> 01:18:11.840 I think containers. I think it's wonderful to embrace the world of containers. 1001 01:18:11.880 --> 01:18:16.039 I think right now, container right now, When people think about containers, 1002 01:18:16.039 --> 01:18:21.159 they often also think about the Docker run time, the Docker build process, 1003 01:18:21.600 --> 01:18:26.000 and so like the of a container, the thing about not only like the 1004 01:18:26.039 --> 01:18:29.199 idea of the container, but also like what does it look and feel like 1005 01:18:29.239 --> 01:18:32.279 when you build everything inside of the Docker file, which is like separate from 1006 01:18:32.359 --> 01:18:36.399 having a container. So like what I love is the container abstraction. I 1007 01:18:36.439 --> 01:18:41.560 don't so much love. It's very easy to bloat the Docker file. It's 1008 01:18:41.600 --> 01:18:45.000 very easy to blow the container so that it's like significantly bigger than it really 1009 01:18:45.079 --> 01:18:47.600 needs to be. There's a lot of stuff to improve there. But when 1010 01:18:47.600 --> 01:18:51.000 it comes to like how you wrap things up, the ideal world is like 1011 01:18:51.159 --> 01:18:59.239 containers as lightweight as possible, wonderful. The only issue is it is pretty 1012 01:19:00.039 --> 01:19:01.800 It is pretty annoying to get in and out of containers and deal with the 1013 01:19:02.600 --> 01:19:06.159 like is this a port that is exposed inside of the container, or is 1014 01:19:06.199 --> 01:19:09.800 it on my laptop or is it like like which port? Is it? 1015 01:19:09.840 --> 01:19:13.840 At what level of networking? It's pretty annoying, and it can be. 1016 01:19:13.960 --> 01:19:18.560 It can be also difficult to map between filesystems and different containers to your own 1017 01:19:18.600 --> 01:19:21.920 file system. When you have mounts mapped or not, that stuff can be 1018 01:19:21.960 --> 01:19:26.359 tricky. A lot of the abstraction that we provide in our like run time, 1019 01:19:27.439 --> 01:19:30.359 tries to make that stuff better when it comes to interacting with the def 1020 01:19:30.479 --> 01:19:34.800 environment. But I think really the the reaction folks have against containers when it 1021 01:19:34.840 --> 01:19:39.279 comes to environments, a lot of the time it comes from them being perceived 1022 01:19:39.319 --> 01:19:43.560 as heavyweight, which is often true. If you're running a doctor it often 1023 01:19:43.640 --> 01:19:45.880 is true. It's like there's a lot of there's a lot of stuff going 1024 01:19:45.920 --> 01:19:48.119 on there. But a lot of the time it doesn't have to be heavyweight. 1025 01:19:48.159 --> 01:19:54.119 It's just that the way we build containers now tends to lean towards a 1026 01:19:54.159 --> 01:19:58.399 more heavyweight approach. No, I think you nailed it there. Realistically, 1027 01:19:58.439 --> 01:20:01.720 it's like this, there's no hobb load automation. When you make a code 1028 01:20:01.800 --> 01:20:06.239 change, how do you get the necessary like auto compiling and run like you 1029 01:20:06.359 --> 01:20:11.399 just want to sort of debug it as soon as possible. I think it's 1030 01:20:11.439 --> 01:20:14.840 a small like that's suset of things. And then there's this other set which 1031 01:20:14.880 --> 01:20:17.399 you for sure mentioned like what if you're changing something at the container level right 1032 01:20:17.399 --> 01:20:21.520 like to You're definitely going to make some changes and you're going to need to 1033 01:20:21.560 --> 01:20:25.279 test that in some way. I don't think I know anyone that wrote a 1034 01:20:25.279 --> 01:20:29.119 Docker file and it was right the first time or the second time or the 1035 01:20:29.159 --> 01:20:32.399 third time. I did learn about this thing that you can like now make 1036 01:20:32.399 --> 01:20:35.960 a builder out of it, and that can really help reduce the size of 1037 01:20:36.000 --> 01:20:41.720 your containers by not storing ninety percent of what's necessary there. Like, I 1038 01:20:41.760 --> 01:20:45.560 still feel like that's really esoteric, and most people are not aware that the 1039 01:20:45.560 --> 01:20:49.600 overcon container initiative supports that sort of reduction. Are you talking about like using 1040 01:20:50.560 --> 01:20:55.439 like a multi stage Docker build where you run your build process and then store 1041 01:20:55.479 --> 01:20:58.680 that as an artifact and then launch a new Docker in it. It was 1042 01:20:58.760 --> 01:21:03.119 actually in the door one thing, right for sure? In multi stage No, 1043 01:21:03.159 --> 01:21:08.840 there's actually a in the darker file you can specify multiple images and one 1044 01:21:08.840 --> 01:21:13.079 of the images you can say is this is my builder, and then have 1045 01:21:13.199 --> 01:21:15.960 the tools in there, and then another image lower down that says, this 1046 01:21:15.039 --> 01:21:19.439 is the actual source image I want, because there's just so many run time 1047 01:21:19.560 --> 01:21:25.520 versus execution time problems that come up, where in the previous world we were 1048 01:21:25.520 --> 01:21:29.279 in a state where they're the same thing and it's really expensive, but now 1049 01:21:29.359 --> 01:21:32.640 we're much better off. Like you can get rust containers down from gigabytes to 1050 01:21:33.279 --> 01:21:39.119 really megabytes because the builder is big and expensive, but the actual built image 1051 01:21:39.199 --> 01:21:43.399 is just pretty much a wrap binary. Yeah, that's the crazy thing, 1052 01:21:44.800 --> 01:21:48.880 especially in the area of like build optimization. There's so much low hanging fruit 1053 01:21:49.399 --> 01:21:55.600 everywhere, but it it's all different low hanging fruit, and you have to, 1054 01:21:55.680 --> 01:21:59.039 like you have to do some degree of looking into it to figure out 1055 01:21:59.079 --> 01:22:00.800 that, oh, you can grab the thing and put it in here, 1056 01:22:00.840 --> 01:22:02.680 and all of a sudden, I forgive your buys the megabytes. But it's 1057 01:22:02.800 --> 01:22:06.319 so common to hear that, oh, I brought my build time down from 1058 01:22:06.399 --> 01:22:10.279 ninety seconds to nine seconds, or I brought it down from five minutes to 1059 01:22:10.319 --> 01:22:13.960 one minute, and it's these are massive improvements and you build all the time, 1060 01:22:14.319 --> 01:22:18.239 but they're right there in the city there, but there's no it's not 1061 01:22:18.359 --> 01:22:23.319 often connected, right, Yeah. I think a lot of For a lot 1062 01:22:23.399 --> 01:22:29.880 of people, there's just they've got they've done the build process and it took 1063 01:22:29.920 --> 01:22:32.319 a long time, and they just like, oh, I guess that's the 1064 01:22:32.359 --> 01:22:36.640 way it is, not realizing that there are a lot of knobs you can 1065 01:22:36.680 --> 01:22:43.239 tweak to change that. Yeah, And I think I mean, depending on 1066 01:22:43.279 --> 01:22:45.560 where you are on the build process. If you're if you're on your laptop 1067 01:22:45.600 --> 01:22:46.840 in your building and you want it to be like super fast, and if 1068 01:22:46.880 --> 01:22:49.880 it's two minutes, it's going to be a bit annoying. But if you're 1069 01:22:50.359 --> 01:22:53.760 if you're merging your pr and you're waiting for it to go to a staging 1070 01:22:53.840 --> 01:22:57.800 environment, like if it's if it's five minutes, you're probably okay with it. 1071 01:22:57.840 --> 01:23:00.319 You're probably like, Okay, this is fine, and try to get 1072 01:23:00.319 --> 01:23:02.399 it from five to two minutes. It's probably like not that big of a 1073 01:23:02.439 --> 01:23:06.479 deal for you because you don't do it that often. So then like when 1074 01:23:06.479 --> 01:23:11.880 would you ever want to optimize that? But across all the debts and all 1075 01:23:11.920 --> 01:23:14.479 the mergers, that's it adds up. It's a lot of time, but 1076 01:23:14.640 --> 01:23:21.600 you could have gone faster for sure, well, sweet, is there anything 1077 01:23:21.600 --> 01:23:27.279 else we should talk about? I feel like we kind of just ran the 1078 01:23:27.319 --> 01:23:34.720 gauntlet here we did. So if folks are interested in learning more about Kurtosis, 1079 01:23:36.720 --> 01:23:44.039 what where do y'all hang out at online discord? Well, first of 1080 01:23:44.079 --> 01:23:45.640 all, I recommend going to our GitHub. The GitHub is the place for 1081 01:23:45.680 --> 01:23:48.920 you enter. So like GitHub, it's got our code, it's got our 1082 01:23:48.960 --> 01:23:51.600 docks there, it's got the reading me. So if you want to learn 1083 01:23:51.640 --> 01:23:56.119 my business, go right to the getthub. And then if you want to 1084 01:23:56.199 --> 01:24:00.199 chat with us, we're always chatting with the community, So join our discord 1085 01:24:00.560 --> 01:24:03.279 and we'll be there. You can DM us, you can chat with us 1086 01:24:03.279 --> 01:24:08.359 in the general channel. It's fun right on cool. Yeah, and I've 1087 01:24:08.359 --> 01:24:12.199 been chatting with y'all a little bit. Y'all are you really are super active 1088 01:24:12.239 --> 01:24:18.039 and super responsive online, which is nice and it's refreshing to see. It's 1089 01:24:18.079 --> 01:24:23.319 the fun part. The fun part is chatting with the users. See, 1090 01:24:23.359 --> 01:24:30.359 that's just so anti tech right there, Like, like I would love tech 1091 01:24:30.399 --> 01:24:33.199 if it weren't for the damn users. I feel like there's the difference of 1092 01:24:33.279 --> 01:24:39.319 you know, the outward appearance versus what you're thinking when they're actually saying things. 1093 01:24:40.640 --> 01:24:43.319 And it's also the users that you're talking with too. You know, 1094 01:24:43.319 --> 01:24:48.720 when you're talking with with people who are doing the same thing, you know, 1095 01:24:49.159 --> 01:24:53.600 whether they're software engineers or whatever. I think that's a different experience than 1096 01:24:54.479 --> 01:25:00.399 trying to talk mod through getting her dial up modem connected and clearing out her 1097 01:25:00.399 --> 01:25:02.279 Internet Explorer cash or whatever. Yeah, I mean, I think that's definitely 1098 01:25:02.279 --> 01:25:08.800 difference between support right for something and real engagement with your customers. Yeah, 1099 01:25:08.800 --> 01:25:12.039 for sure. You know, I usually come away from them like, oh, 1100 01:25:12.199 --> 01:25:15.159 no, we should have that feature. I can't believe we didn't think 1101 01:25:15.159 --> 01:25:19.159 of it. I mean, one of the fun things about having a developer 1102 01:25:19.239 --> 01:25:26.239 tool is it's buy engineers for engineers. There's a lot of empathetic connection across 1103 01:25:26.279 --> 01:25:31.319 the the product user barrier, where maybe if we weren't the exact same profile, 1104 01:25:31.600 --> 01:25:36.600 you'd have to you have to bridge the gap somehow. Well. You 1105 01:25:36.640 --> 01:25:41.279 know, I've heard many times that that's what makes us successful startup is whenever 1106 01:25:41.319 --> 01:25:46.439 you're solving a problem that you personally have. I totally agree with that. 1107 01:25:47.119 --> 01:25:50.640 Yeah. Cool, Well, Gayalen, thanks for being on the show man. 1108 01:25:50.680 --> 01:25:57.279 This has been a really cool talk. Thanks for having me. Cool 1109 01:25:57.319 --> 01:26:00.000 Warren thanks for joining me, welcome back, Glad to have you back, 1110 01:26:00.079 --> 01:26:03.039 kind of show, glad to be here. And for all of the listeners, 1111 01:26:03.279 --> 01:26:06.560 thank you for listening. Really appreciate it. Hope you found it useful. 1112 01:26:06.840 --> 01:26:11.840 Oh right, yeah, yeah, picks, all right, you brought 1113 01:26:11.880 --> 01:26:14.840 it up. Kick us off oring. I just you know, I know 1114 01:26:14.920 --> 01:26:17.880 some listeners were going to be mad if you know they didn't hear on the 1115 01:26:17.920 --> 01:26:28.319 report. Yeah, So mine this week is Turn the Ship Around by David 1116 01:26:28.399 --> 01:26:32.399 Marque, which is a book that, irrelevant of your experience and technology or 1117 01:26:32.760 --> 01:26:40.159 why you're focused on either core engineering or somewhere else, can really help take 1118 01:26:40.279 --> 01:26:44.479 your understanding of how to work effectively to the next level. It's really about 1119 01:26:45.159 --> 01:26:48.399 leadership in every way, starting from the bottom all the way up, and 1120 01:26:48.439 --> 01:26:51.439 how to think about it. And I think it's really it's really interesting to 1121 01:26:51.479 --> 01:26:55.520 read, and if you feel like you can't commit to doing that, you 1122 01:26:55.560 --> 01:27:00.840 can find ten minutes to watch the shortened video online. It's something of greatness. 1123 01:27:00.920 --> 01:27:03.439 I can never win the name of the title. It's a YouTube video. 1124 01:27:03.479 --> 01:27:08.640 You can find it, which is it's a hilarious and it's really good 1125 01:27:08.640 --> 01:27:15.239 content right on. Awesome Galen putting on the spot. What'd you bring for 1126 01:27:15.279 --> 01:27:18.720 a pick my pig? It can be it can be a music Oh for 1127 01:27:18.760 --> 01:27:25.239 sure. Yeah, so I have been very into jelly Roll lately. I 1128 01:27:25.239 --> 01:27:29.000 don't know if you guys know jelly roll country singer. Oh yeah, yeah, 1129 01:27:29.039 --> 01:27:33.760 yeah, he's he's uh the country post Malone. Yes, yes, 1130 01:27:34.680 --> 01:27:38.079 that's hilarious. I've never heard it before, but as soon as you said 1131 01:27:38.119 --> 01:27:42.119 it, I knew that you were thinking of the right person. Yes, 1132 01:27:42.239 --> 01:27:46.000 Country post Malone. Jelly Roll is amazing. I've just been It's rare that 1133 01:27:46.039 --> 01:27:49.720 I find an artist. And then I just listened to a dozen of his 1134 01:27:49.880 --> 01:27:54.119 songs on a regular basis. He hit me with, hit me with all 1135 01:27:54.199 --> 01:27:57.000 of them. So if you haven't checked out jelly Roll, if you're into 1136 01:27:57.039 --> 01:28:00.079 country, if you're into hip hop, it's a really nice combination of both. 1137 01:28:00.359 --> 01:28:08.920 So for sure, Yeah, I last year I stumbled across hardy and 1138 01:28:08.960 --> 01:28:15.840 which hardy is like country meets heavy metal or or hard rock. And then 1139 01:28:15.840 --> 01:28:17.880 at some point he did a collaboration with jelly Roll, which took me down 1140 01:28:17.880 --> 01:28:21.399 that path. Yeah, so I'll agree with you on that one. That's 1141 01:28:21.399 --> 01:28:28.000 pretty cool and it's you know, it's definitely not what you would expect from 1142 01:28:28.199 --> 01:28:33.680 country music. It's it's quite a bit different. I'm not sure Hank william 1143 01:28:33.720 --> 01:28:41.920 Senior Senior would approve of it, but here we are cool. So I 1144 01:28:42.000 --> 01:28:48.039 brought two picks this week. The first one is going to totally destroy my 1145 01:28:48.920 --> 01:28:56.199 nerd credibility because I have never read the book Dune or seen the movie Dune. 1146 01:28:56.720 --> 01:29:02.119 So I am starting to read the book Dune, and dude, maybe 1147 01:29:02.239 --> 01:29:06.319 one of the best books I've ever read. I mean, the things that 1148 01:29:06.319 --> 01:29:12.319 that Herbert does in the book is just unmatched by other stories realistically, like 1149 01:29:12.359 --> 01:29:15.520 that you can tell us by the number of new concepts and things that were 1150 01:29:15.560 --> 01:29:19.640 just made up. It's not like a world that just easily imaginable, right, 1151 01:29:19.680 --> 01:29:23.399 And I'm looking forward to it and I've I've heard from multiple people. 1152 01:29:23.840 --> 01:29:26.279 You know, it's a common saying, oh, the book's better than the 1153 01:29:26.279 --> 01:29:28.880 movie, but I've heard from from multiple people like, no, No, 1154 01:29:29.520 --> 01:29:32.960 the book is better than the movie on many, many levels. So I'm 1155 01:29:32.960 --> 01:29:36.760 looking forward, looking forward to getting into that. The other pick I'm going 1156 01:29:36.840 --> 01:29:44.279 to bring up is I kind of just lost I kind of fell out of 1157 01:29:44.319 --> 01:29:47.159 love with Chrome as my browser, and I wanted to see what else was 1158 01:29:47.199 --> 01:29:51.439 out there, Chrome, It's not you, it's me. I just felt 1159 01:29:51.479 --> 01:30:00.560 like to see other browsers across the ARC Browser, and I think I completely 1160 01:30:00.640 --> 01:30:14.239 locked up here there you I did here may be back. I got completely 1161 01:30:14.359 --> 01:30:20.680 kicked out for talking trash about Chrome, didn't I. I'm going to go 1162 01:30:20.720 --> 01:30:25.079 through with it anyway. The ARC Browser has been really cool. I've been 1163 01:30:25.159 --> 01:30:29.039 using it for about a week now, and there's so many cool, subtle 1164 01:30:29.039 --> 01:30:33.760 little features that I didn't realize belonged in a browser. Like my favorite one 1165 01:30:34.520 --> 01:30:41.039 is when you're in a Google meet, you know, and you navigate off 1166 01:30:41.079 --> 01:30:45.800 to another tab, and then somebody asked you a question. You can never 1167 01:30:45.920 --> 01:30:48.680 find which tab is the one for meat, right, So then you got 1168 01:30:48.680 --> 01:30:51.479 this awkward silence and you gotta admit that you were screwing off and not paying 1169 01:30:51.479 --> 01:30:56.199 attention to the meeting. Well, with the ARC Browser, when you leave 1170 01:30:56.279 --> 01:31:00.000 the tab, it just puts a little thumbnail in the corner of your screw 1171 01:31:00.560 --> 01:31:04.239 with that video meet still going. So when you never have to go back 1172 01:31:04.239 --> 01:31:06.439 to it, it's just down there in the corner, you click into it 1173 01:31:06.479 --> 01:31:09.640 and it pops you right back up. And I just thought that was super 1174 01:31:09.640 --> 01:31:15.560 cool. So that's my second pick for the week, and with that, 1175 01:31:16.399 --> 01:31:19.640 I think we're done here. Galen. Thanks again, Ben, this has 1176 01:31:19.680 --> 01:31:23.760 been pretty cool. I've enjoyed it. I hope the listeners enjoyed it. 1177 01:31:23.800 --> 01:31:29.359 Thank you for listening. We appreciate your support, and yeah, I hope 1178 01:31:29.439 --> 01:31:30.239 see you all next week.