WEBVTT 1 00:00:01.080 --> 00:00:05.679 How'd you like to listen to dot NetRocks with no ads? Easy? Become 2 00:00:05.679 --> 00:00:09.839 a patron for just five dollars a month. You get access to a private 3 00:00:10.000 --> 00:00:14.400 RSS feed where all the shows have no ads. Twenty dollars a month will 4 00:00:14.439 --> 00:00:18.839 get you that and a special dot net Rocks patron mug. Sign up now 5 00:00:18.839 --> 00:00:50.560 at Patreon dot dot NetRocks dot com. Hi, it's all right, Holy 6 00:00:50.960 --> 00:00:55.560 crap, this is a great room. It's a great lot of echo. 7 00:00:55.880 --> 00:01:02.719 Yes, certainly filled this space. He determined level of absolutely the blast radius. 8 00:01:03.240 --> 00:01:07.400 Right, we are back in Portugal. I love I love Porto, 9 00:01:07.519 --> 00:01:14.519 I love Portugal. I was in the middle of town and I came across 10 00:01:14.560 --> 00:01:23.000 a restaurant and the sign said churiscaria, which means grill. Right they infante. 11 00:01:23.959 --> 00:01:27.599 I didn't know you guys ate babies over here. So no, no, 12 00:01:27.599 --> 00:01:32.280 no, not funny, too long, too long, too soon? 13 00:01:33.799 --> 00:01:38.519 Infante. I guess Infante is the name of the area. Yeah, but 14 00:01:38.599 --> 00:01:42.799 it looked like we eat our babies. Well done, nice on a skewer, 15 00:01:42.840 --> 00:01:44.560 I think it was. W. C. Field said it was all 16 00:01:44.560 --> 00:01:51.439 about the sauce. Yes, I love children exactly. You got it. 17 00:01:53.280 --> 00:01:55.439 Oh, we're gonna have some fun. Today. We are going to have 18 00:01:55.519 --> 00:01:59.000 some fun. But first we have this little thing called better no Framework. 19 00:01:59.239 --> 00:02:12.000 Roll that crazy music. Well I've been waiting for this one because it happened 20 00:02:12.000 --> 00:02:15.800 a while ago. But if you don't know, I have a consultancy called 21 00:02:15.879 --> 00:02:22.639 at the next and we are the shepherds of an open source project called Polly. 22 00:02:22.879 --> 00:02:25.919 Anybody who's Polly? How about uh? How about a clap of hands? 23 00:02:27.360 --> 00:02:31.240 Use poly lot of Well, Reverend Billy just walked in the room. 24 00:02:31.280 --> 00:02:36.520 Well, Polly just did a make just came out with a major update, 25 00:02:37.080 --> 00:02:39.520 version eight. And I, believe it or not, it started with the 26 00:02:39.560 --> 00:02:45.400 dot net team because the dot Net team was basically looking at the source code 27 00:02:45.400 --> 00:02:49.639 and said, hey, we think we can improve the performance and the resource 28 00:02:49.759 --> 00:02:54.360 usage of Polly, but it's going to require some new interfaces and you know, 29 00:02:54.879 --> 00:03:00.240 almost a complete rewrite. And so the rest of us said, yeah, 30 00:03:00.280 --> 00:03:04.400 I'm sorry. The dot Net team call you, yeah, say they 31 00:03:04.439 --> 00:03:07.199 want to make your project. Yeah, I mean what are you saying? 32 00:03:07.319 --> 00:03:14.800 Nah? So it took a lot of meetings and a lot of understanding, 33 00:03:14.800 --> 00:03:19.400 and basically what they were able to do is without you having to change any 34 00:03:19.439 --> 00:03:23.560 of your code that uses Polly, you will get the benefits of the new 35 00:03:23.599 --> 00:03:29.520 performance and resource allocation that's under the hood. But if you want to use 36 00:03:29.560 --> 00:03:34.439 the new models and the paradigms and the interfaces, you you can do that. 37 00:03:34.680 --> 00:03:38.960 So Greenfield, you can go forward with a new style. But just 38 00:03:39.000 --> 00:03:45.639 if you use Polly in place, I'm not sure if it's completely compatible or 39 00:03:45.680 --> 00:03:49.759 you have to change some class to some other class. But it's pretty much 40 00:03:50.000 --> 00:03:54.080 a simple a simple fix. And I have been emailing with Joel hewle In 41 00:03:54.080 --> 00:04:00.879 to schedule Polly show for Yes, we definitely will get cool. So that's 42 00:04:00.919 --> 00:04:02.879 what that's what I got. Awesome. Who's talking to us? Richard Grabby 43 00:04:02.879 --> 00:04:06.960 comment off of show eighteen sixty one we did with Jeremy Miller back in the 44 00:04:06.960 --> 00:04:11.840 summer of twenty three, talk about minimal architecture because Jeremy likes to cause trouble. 45 00:04:11.879 --> 00:04:15.720 Goodness nose And this comment comes from Trevor who says, I love this 46 00:04:15.759 --> 00:04:19.879 discussion, enjoyed the comments on microservices Worth's monoliths, which is actually a reference 47 00:04:19.920 --> 00:04:25.120 to an earlier show we didn't reporter. We've been following this trend of people 48 00:04:25.439 --> 00:04:30.040 sort of pushing back on microservice. Yes, I got pushed heavily into microservices 49 00:04:30.040 --> 00:04:33.639 approaches with a product that we'd built and re architected into microservices, and it 50 00:04:33.720 --> 00:04:40.079 was the worst mistake ever. Things just became more complex, it was harder 51 00:04:40.120 --> 00:04:43.279 to maintain, it added a bunch of latency and security issues in the complexity 52 00:04:43.399 --> 00:04:46.879 was just not worth it. And so I came up with a new acronym 53 00:04:46.480 --> 00:04:55.600 for appropriately sized service or as nice I can relate. This is a good 54 00:04:55.639 --> 00:04:58.800 one. I one hundred percent believe in services, separation of concerns, and 55 00:04:58.879 --> 00:05:02.399 clean architectures, but the approach must be appropriate to the complexity, solution and 56 00:05:02.480 --> 00:05:06.079 the size of the team. It makes no sense to have one hundred separate 57 00:05:06.079 --> 00:05:10.199 services for a team of ten people. But then it also makes no sense 58 00:05:10.199 --> 00:05:12.639 to have a massive single deployment with a code base in the team of a 59 00:05:12.639 --> 00:05:15.399 two hundred and fifty people. The services need to work with the cognitive load 60 00:05:15.639 --> 00:05:20.600 and be appropriate to the organization and team structures. And I was loving the 61 00:05:20.600 --> 00:05:26.319 discussion on all of this except for that one point. Stop making the CTO 62 00:05:26.399 --> 00:05:32.279 out to be the bad guy. Love from Trevor CTO. That's great, 63 00:05:32.399 --> 00:05:36.879 Yep, that's fair. I definitely think we need a whole show on ass 64 00:05:38.000 --> 00:05:40.920 I think so it seems up. Yah, So, Trevor, thank you 65 00:05:40.959 --> 00:05:43.319 so much for your comment, and a copy of music by its own its 66 00:05:43.360 --> 00:05:45.240 way to you. And if you'd like a copy of music co buy, 67 00:05:45.240 --> 00:05:46.680 I write a comment on the website at dot at Rocks dot com or on 68 00:05:46.680 --> 00:05:49.439 the Facebook, so you published every show there, and if you comment there 69 00:05:49.439 --> 00:05:51.639 and everyday in the show, it's like your copy of music Cobe. And 70 00:05:51.759 --> 00:05:56.399 you can also follow us on Twitter if you want to. But the real 71 00:05:56.759 --> 00:06:00.439 cool kids are over. I'm massedon, I'm at Carl Frank tech dot Social 72 00:06:00.519 --> 00:06:04.480 and Ambridge Campbell at maps. Send us a two we'll get around reading it 73 00:06:05.920 --> 00:06:11.920 all, publish it and with that let us introduce Charity Majors to the show. 74 00:06:12.079 --> 00:06:16.319 Charity is an OPS engineer and CTO at Honeycomb dot Io. Before that, 75 00:06:16.480 --> 00:06:23.240 she worked at Parse, Facebook and Linden Lab on operations and developer tools 76 00:06:23.279 --> 00:06:27.160 and always seemed to wind up running the databases. That's because it's where all 77 00:06:27.160 --> 00:06:30.360 the problems were. Yeah, we stand next to the database. You're going 78 00:06:30.399 --> 00:06:35.240 to be running it. Also co author of O'Reilly's Database Reliability Engineering and the 79 00:06:35.319 --> 00:06:41.879 newly released Observability Engineering. Charity Loves free speech, free software, and single 80 00:06:41.920 --> 00:06:50.439 malt Scotch blamer. Do you ever have round the bloud charity Majors? Okay, 81 00:06:50.759 --> 00:06:53.839 I guess we would have start at the beginning, right at the beginning. 82 00:06:53.879 --> 00:06:58.199 What the heck is observability engineering? That's a great question. It's like 83 00:06:58.199 --> 00:07:03.519 the engineering of Windows, yeah, kind of. I mean observability comes from 84 00:07:03.560 --> 00:07:09.319 control theory, right, and it's like, how well can you understand what's 85 00:07:09.360 --> 00:07:17.040 going on inside your systems just by observing outputs? And yeah, exactly. 86 00:07:17.759 --> 00:07:23.959 And you know, I for years was like really religious about trying to define 87 00:07:23.959 --> 00:07:26.560 it in a very specific way, and I should have won, but I 88 00:07:26.639 --> 00:07:33.160 lost. So so I mean it's come to it's come to just kind of 89 00:07:33.160 --> 00:07:40.399 be a generic sitting, which is what it is. But when we were 90 00:07:40.399 --> 00:07:43.079 trying to figure out how to talk about, what I think of is just 91 00:07:43.160 --> 00:07:47.079 kind of the next generation of telemetry. It's kind of distinguished from the last 92 00:07:47.120 --> 00:07:50.639 generation of peletry, obviously, which was very much focused around the metric, 93 00:07:50.959 --> 00:07:56.079 right, which is just a number. It's tags depended, doesn't handle high 94 00:07:56.079 --> 00:08:01.079 cardinality, doesn't handle dimensionality, doesn't handle it's super fast. Is that powerful? 95 00:08:01.160 --> 00:08:05.839 Now you drop some OLAP terms into their cardinality flexibility, Like it's funny 96 00:08:05.879 --> 00:08:11.319 for a database person to drop all lap, but you're talking about just any 97 00:08:11.360 --> 00:08:15.839 way that you can really observe the state, the internal state, not necessarily 98 00:08:15.879 --> 00:08:18.319 what it's doing on the outside. It's about observing the internal state and being 99 00:08:18.399 --> 00:08:24.279 able to explore it right, not having to decide in advance, here's the 100 00:08:24.319 --> 00:08:26.920 data I'm going to collect, because here's the questions I'm going to need to 101 00:08:26.959 --> 00:08:30.560 answer. Here's my dashboard. You know, it's about being able to go 102 00:08:30.959 --> 00:08:35.759 to combine your questions to ask because, like anything that you're trying to understand 103 00:08:35.840 --> 00:08:37.759 these days is going to be a very complicated answer to cart. It's like, 104 00:08:39.320 --> 00:08:43.440 okay, these errors are spiking, but only for users that are running 105 00:08:43.440 --> 00:08:46.440 this version of Android, who's a particular firmware in this region, with this 106 00:08:46.559 --> 00:08:50.879 language pack. With each of those are the high cardinality to mention it. 107 00:08:50.960 --> 00:08:54.240 And if you don't capture the data in a way that preserves all that context, 108 00:08:54.720 --> 00:08:58.639 you can't ask me questions. Do you have some examples of how observe 109 00:09:00.039 --> 00:09:05.799 observability has improved a project in particular? Sure, I mean I think of 110 00:09:05.840 --> 00:09:11.480 it as it's really it's kind of where development meets operations, right, Like, 111 00:09:11.600 --> 00:09:16.279 I feel like big picture. You know, in the beginning, there 112 00:09:16.320 --> 00:09:18.519 were engineers who wrote code and they owned it in production, right, right, 113 00:09:18.720 --> 00:09:22.639 And then everything got super complicated and we're like, ah, there's too 114 00:09:22.720 --> 00:09:24.320 much. So some of us are going to write code and some of us 115 00:09:24.360 --> 00:09:28.600 are going to understand it. And that was not about it. That was 116 00:09:28.639 --> 00:09:31.799 not a great idea, and so like we're kind of like reunifying the streams 117 00:09:31.840 --> 00:09:35.759 now. I think every engineer should be writing their code and owning it in 118 00:09:35.799 --> 00:09:43.000 production. Everyone who's especialist operations should be also like opening the door and looking 119 00:09:43.080 --> 00:09:46.559 under the hood and understanding the code. Right. There's specialization is great, 120 00:09:46.080 --> 00:09:50.039 but ultimately, you know, our systems have gotten so complex that you have 121 00:09:50.080 --> 00:09:52.559 to write it and understand. I feel like you got to dig into that 122 00:09:52.799 --> 00:09:58.679 own it in production because it's not like they're also going to be sisumits exactly 123 00:09:58.080 --> 00:10:03.759 as they are responsible for the You're responsible for your systems, right, you 124 00:10:03.759 --> 00:10:07.080 wrote it, you own it, You unleashed this support upon the world. 125 00:10:07.039 --> 00:10:09.759 I mean, I feel like there are these feedback loops in the heart of 126 00:10:09.840 --> 00:10:13.320 engineering. Some of them are like code review, right. Some of them 127 00:10:13.960 --> 00:10:18.639 are like deploys. But like, if you don't hook up the feedback loop, 128 00:10:18.840 --> 00:10:22.559 if you aren't being exposed to the consequence of what you're doing, then 129 00:10:22.600 --> 00:10:24.840 like you're not you don't actually know if your code is good or not. 130 00:10:24.120 --> 00:10:28.919 Well, I think there's a great point there as a developer then that if 131 00:10:28.919 --> 00:10:31.399 my telemetry just tells me how many times my code was hit, that doesn't 132 00:10:31.399 --> 00:10:35.639 necessarily give me anything to do. And this is this is where I feel 133 00:10:35.679 --> 00:10:39.960 like operations folks have had a harder time embracing serviability in some ways than software 134 00:10:39.960 --> 00:10:43.759 engineers have because with up people, it's like we learned how to debug, 135 00:10:43.799 --> 00:10:48.600 but it looked like this, I've got a dashboard. Something's wrong, So 136 00:10:48.600 --> 00:10:52.919 I'm gonna start paging through dashboards and looking for similar spikes, just like pattern 137 00:10:52.919 --> 00:10:54.879 maatting with my eyeballs, Like right, oh, it looks like it's redus, 138 00:10:54.919 --> 00:10:58.120 you know, and you get it. It's great because you're like you 139 00:10:58.159 --> 00:11:01.879 get this hero journey where you just jump to the and you understand what's going 140 00:11:01.879 --> 00:11:03.840 on because you're in this shit all day every day. Nobody else does, 141 00:11:05.120 --> 00:11:07.120 like whoa, how did you do? That? Was reset? The redit 142 00:11:07.159 --> 00:11:11.799 service? Problems went away? Right? But like that's not debugging. Countermatching 143 00:11:11.840 --> 00:11:16.159 with your eyeballs is not debugging. Debugging looks like you take the step, 144 00:11:16.440 --> 00:11:20.519 you ask a question, you look at the answer. Based on the answer, 145 00:11:20.559 --> 00:11:22.720 you take another step. It's like following a trail of bread crupts. 146 00:11:22.759 --> 00:11:26.639 You don't know what the answer looks like until you get there. Can we 147 00:11:26.679 --> 00:11:31.600 talk about some of the new modern observability tools that we might think about using 148 00:11:31.639 --> 00:11:35.919 to replace the tools that we're currently using. Yeah, I mean, I 149 00:11:35.919 --> 00:11:41.000 think big picture, it has to be based It can't just be based on 150 00:11:41.039 --> 00:11:46.159 the metric because remember you've discarded all that you're looking output exactly. It has 151 00:11:46.200 --> 00:11:50.519 to be based on ar truly wide structured TETA blocks, which now look like 152 00:11:50.600 --> 00:11:56.200 scams, right, Those are just like wide events structured which you can trace 153 00:11:56.240 --> 00:11:58.240 because there's been a number that's appended to it. That's what you need in 154 00:11:58.279 --> 00:12:01.960 order to understand your telemetry and production. Because I can imagine at a peak 155 00:12:03.080 --> 00:12:05.320 load, like we think about a metric that shows, you know, this 156 00:12:05.360 --> 00:12:09.519 is when we're posting the most number of transactions. You're now really interested in 157 00:12:09.519 --> 00:12:13.960 the state of yes, we're we queuing out yes, what's happening? Like 158 00:12:13.120 --> 00:12:16.799 metrics are great, but they're they're limited, right, they're a snapshot. 159 00:12:18.080 --> 00:12:20.559 What you want to be is like, you know, okay, when this 160 00:12:20.679 --> 00:12:24.519 happened, what else happened? Right? What else is connected to it? 161 00:12:24.679 --> 00:12:26.279 You know? And like the old generation of tool are ones where you find 162 00:12:26.799 --> 00:12:31.039 you're capturing this data another time. For every single tool you're like, okay, 163 00:12:31.039 --> 00:12:35.559 here's my dashboards and the metrics, here's my logs, here's my traces. 164 00:12:35.639 --> 00:12:37.000 So every time you're like I've got a spike, I want to find 165 00:12:37.080 --> 00:12:41.720 the logs, there's nothing that connects them. You're just eyeballing timestamps and hoping 166 00:12:41.720 --> 00:12:46.360 that they happen to match up. And like, if you're finding the logs, 167 00:12:46.360 --> 00:12:48.080 we want to jump to a trace. Like that's not actually good enough. 168 00:12:48.320 --> 00:12:54.200 You can derive all of those data formats from these arbitrary white from spans. 169 00:12:54.240 --> 00:12:56.039 You can't go in the other direction. When you say spans, what 170 00:12:56.159 --> 00:13:01.840 exactly you're talking about? A span is a one hop of the trace, 171 00:13:01.120 --> 00:13:05.840 okay, across all of them. So should we be gathering spans? All 172 00:13:05.879 --> 00:13:11.679 of the gathering telemetry one event per request, per service, all of the 173 00:13:11.799 --> 00:13:16.399 data should be aggregated into that one arm chreary wide production, so you have 174 00:13:16.519 --> 00:13:22.120 all that context, Like a really mature instrumented service will have like two hundred 175 00:13:22.200 --> 00:13:26.879 three hundred dimensions per per hop and that's that's magic because you're passing along all 176 00:13:26.919 --> 00:13:31.559 of the parameters, you're passing along all of the I E. S, 177 00:13:31.759 --> 00:13:35.919 you're passing along all of that context, which lets you after the fact come 178 00:13:35.000 --> 00:13:39.879 back and say, oh, this thing and this service that happened was connected 179 00:13:39.919 --> 00:13:41.679 to that thing and that service that that happened. Now, this is not 180 00:13:41.679 --> 00:13:46.840 necessarily a per transaction level, like you're not just chasing a transaction. What's 181 00:13:46.960 --> 00:13:56.759 this It basically it's one span from a time it's well typically, well this 182 00:13:56.000 --> 00:14:03.240 complicated. There are lots of ways that you can define as span, but 183 00:14:03.399 --> 00:14:05.240 typically I like to think it about if you want to have a span around 184 00:14:05.240 --> 00:14:09.399 something that's interesting. So like if it's anytime that you're crossing the network, 185 00:14:09.559 --> 00:14:13.840 you want to span. Anytime you're taking a database request, you want to 186 00:14:13.879 --> 00:14:18.399 span because that's historically where problems happen. It's wherever you're crossing the right. 187 00:14:18.440 --> 00:14:20.960 So when you start it at a user interface interaction and go from there and 188 00:14:22.000 --> 00:14:26.879 then you know, likewise we're back end services that go on time like you 189 00:14:26.919 --> 00:14:30.879 can have you can have spans and tracing in models too, and it's super 190 00:14:31.120 --> 00:14:37.039 super useful there as well. But it becomes indispensable once you have service. 191 00:14:39.000 --> 00:14:41.200 Yeah, because if you think about it as a monolith, at least you 192 00:14:41.320 --> 00:14:46.399 have all that context and it persists throughout the request. When you jump across 193 00:14:46.440 --> 00:14:50.120 the network from service to service, you're deciding what state's going to come with. 194 00:14:50.200 --> 00:14:54.679 And so how do you do all this without bringing the server CPU to 195 00:14:54.759 --> 00:14:58.799 its knees? Do you do this? Typically? The way we do it 196 00:14:58.799 --> 00:15:01.639 now is it attached to back around threads and that kind of stuff, and 197 00:15:01.720 --> 00:15:03.720 you can lose you know, if those threats hang, you can lose data. 198 00:15:05.120 --> 00:15:07.120 There are lots of ways to do it. Obviously, I think that 199 00:15:07.200 --> 00:15:11.960 my service does it best. Your Honeycomb is, well, you've got to 200 00:15:11.960 --> 00:15:16.279 tell us about Honeycomb. Well, sure, you know I'm not I'm not 201 00:15:16.320 --> 00:15:20.200 really great at pitching, but I will say that, like, you know, 202 00:15:20.960 --> 00:15:24.480 the idea of how observability should happen is how we built our service, 203 00:15:24.679 --> 00:15:30.039 you know, down to like the data store, like because like, well 204 00:15:30.039 --> 00:15:31.919 back how many have you any if you ever built apps on pars? The 205 00:15:33.000 --> 00:15:37.919 mobile back end is the service. No, wow, I loved so much. 206 00:15:39.879 --> 00:15:45.039 It was it was like Firebase but better and earlier. Any Facebook people 207 00:15:45.080 --> 00:15:48.960 here, Okay, cool, I will have I will have a brudg against 208 00:15:48.960 --> 00:15:54.120 Spark Soccer for forever for what he did to Pars. Okay, they shut 209 00:15:54.120 --> 00:15:58.960 it down. It's like we got acquired. Anytime you want to get acquire 210 00:16:00.120 --> 00:16:04.679 or make sure that you have an executive level sponsor who believes you were fond. 211 00:16:04.960 --> 00:16:11.320 Right. Not so we got shut down even though we were still growing 212 00:16:11.320 --> 00:16:15.639 like aang busters anyway. Cars had had one hundred million apps by the time 213 00:16:15.799 --> 00:16:19.840 I left, and we had built our service originally on Ruby on Rails, 214 00:16:21.000 --> 00:16:23.720 which was not a terrible decision because most startups fail. It's usually not because 215 00:16:23.759 --> 00:16:27.879 it's and Rails had the strength of you can move fast, you can move 216 00:16:29.720 --> 00:16:33.960 It's just a we believed everything. Yeah, ok, say whatever you want. 217 00:16:33.039 --> 00:16:40.960 Doesn't that Ruby on rails. The downside is it's got it doesn't have 218 00:16:41.000 --> 00:16:44.519 threads right, fixed pool of workers, right. And so that was fine. 219 00:16:44.639 --> 00:16:47.399 We had one hundred thousand apps, but we got bigger and bigger, 220 00:16:47.399 --> 00:16:51.320 and instead of having one database and back end, we now had thirty forty 221 00:16:51.399 --> 00:16:55.159 fifty. And when you've got that many something slow at any given time, 222 00:16:55.519 --> 00:17:00.120 which means that the sixth pool is going it's filling up constantly with threads that 223 00:17:00.159 --> 00:17:03.599 are waiting on that one back end service got oh and like, as a 224 00:17:03.640 --> 00:17:10.559 reliability engineer, this was personally humiliating. You're going down every day just like 225 00:17:10.839 --> 00:17:15.400 a hit the top ten and iTunes down, goes parts again and again. 226 00:17:17.000 --> 00:17:21.920 And I tried everything to try and figure this out. And what finally helped 227 00:17:21.960 --> 00:17:25.319 us was number one, we did a rewrite to go length. We actually 228 00:17:25.559 --> 00:17:30.599 considered uh using dot net and it got out voted. And I learned later 229 00:17:30.720 --> 00:17:34.160 that the blog post that I wrote about why it got outvoted, they had 230 00:17:34.200 --> 00:17:38.039 a lot of people with Microsoft very angry and changed a lot of their decisions, 231 00:17:38.079 --> 00:17:42.480 which is great. Yeah, inspiring anger is really like, but I 232 00:17:42.480 --> 00:17:45.759 mean, I can I made a career. I have a tough time of 233 00:17:45.119 --> 00:17:48.000 disagree with you on picking go line too. When you think about a back 234 00:17:48.079 --> 00:17:52.920 end service at velocity like that, language is very well suited for that. 235 00:17:53.119 --> 00:17:56.240 It was great, but it was half of the half of the answer because 236 00:17:56.279 --> 00:18:00.640 we also had to understand what was going on just right to code understand it 237 00:18:00.880 --> 00:18:06.519 observab This is where observability came into play because Facebook also had this service called 238 00:18:06.519 --> 00:18:11.160 Scuba, which was, don't get me wrong, but ugly aggressively hostile to 239 00:18:11.279 --> 00:18:15.039 users, but it did one thing really well, which is people that you 240 00:18:15.079 --> 00:18:21.839 slice and dice in near real time on dimensions of high cardinality with wide events. 241 00:18:21.920 --> 00:18:25.759 Right high cardinality for those who don't know, it's the number of unique 242 00:18:25.759 --> 00:18:27.440 idemans in the set. So if you've got a collection of one hundred million 243 00:18:27.519 --> 00:18:33.680 users, any unique idea like social security number for the US folks would be 244 00:18:33.079 --> 00:18:37.880 the highest possible cardinality. Something like species equals human would be the lowest because 245 00:18:38.079 --> 00:18:42.079 only one right first thing lasting high cardinality, but there's some dupes, so 246 00:18:42.119 --> 00:18:47.480 it's not as high as the security number. So everything around metrics is oriented 247 00:18:47.519 --> 00:18:52.920 around low cardinality dimensions, but everything you want to use for debugging requires high 248 00:18:52.920 --> 00:18:59.039 cardinaliti ice some people over you run into the influence before. So Scuba, 249 00:18:59.160 --> 00:19:02.519 let us slice it some of these high cardinality mentions and instead of having to 250 00:19:02.599 --> 00:19:06.680 like, you know, obscur and be like either I either I read it 251 00:19:06.680 --> 00:19:10.839 to dashboard for it or it's going to be hours to like just dive through 252 00:19:10.880 --> 00:19:14.759 the logs and figure it out and everything. It's like, instead it would 253 00:19:14.799 --> 00:19:18.119 be like, okay, we're getting a spike and eras, let's break down 254 00:19:18.160 --> 00:19:21.839 by app one in ten million appies. Break down by that. Okay, 255 00:19:21.960 --> 00:19:26.319 now break down by her rights. Don't break down by by normalized database query. 256 00:19:26.359 --> 00:19:36.559 Now are you making cube gestures or gestures? Call them after Colum. 257 00:19:36.599 --> 00:19:40.200 It was just like, step by step it would take me to It's like 258 00:19:40.240 --> 00:19:45.119 it isn't even engineering anymore. It's like support. Right, These problems went 259 00:19:45.119 --> 00:19:48.640 from being like intractable, like it would be I'm doing mean, from like 260 00:19:48.680 --> 00:19:51.200 it would take us a day to figure out and then it would never happen 261 00:19:51.240 --> 00:19:56.039 again, to just being like, you know, thirty seconds, like every 262 00:19:56.119 --> 00:20:00.279 single time. And that was what like when I was leading Facebook. You 263 00:20:00.319 --> 00:20:02.960 know, I've never been one of those kids who's like I don't want to 264 00:20:03.000 --> 00:20:07.759 start a company because I kind of hate those people. But when I was 265 00:20:07.799 --> 00:20:11.960 thinking about having to live without this tooling, I was like, I can't. 266 00:20:11.039 --> 00:20:14.880 I actually can't conceive of it, Like it's becomes so coore to how 267 00:20:14.920 --> 00:20:18.680 I how I perceive the world as an engineer, Like I just can't imagine 268 00:20:18.680 --> 00:20:22.599 going back. And so that's why I made talking on So when how did 269 00:20:22.640 --> 00:20:29.000 the rest of the people in the organization react to this new culture of observed 270 00:20:29.000 --> 00:20:36.240 ability and spam? There's a learning curve, right, We've all spent our 271 00:20:36.279 --> 00:20:40.160 careers fitting our brain into asking questions in the metrics and dactions type of way. 272 00:20:40.680 --> 00:20:45.440 But like you know how every job I've ever had, the person who's 273 00:20:45.440 --> 00:20:48.640 best to be bugging is always a person who's been in the moms. Always. 274 00:20:48.920 --> 00:20:53.559 That's no longer true when we have different tools because instead of relying so 275 00:20:53.680 --> 00:20:57.720 much on what's in your head to reason about system, it's right in front 276 00:20:57.759 --> 00:21:02.119 of you and you're just asking questions and it's and it's more like the more 277 00:21:02.240 --> 00:21:06.400 curious you are, the more debugging you do, the better you get. 278 00:21:06.519 --> 00:21:08.160 You don't have to you don't have to have the whole system in your head. 279 00:21:08.640 --> 00:21:12.400 You can find the answer more quickly that way, and it's kind of 280 00:21:12.440 --> 00:21:17.359 a beautiful thing. Yeah, process of discovery too, right, find exceptions. 281 00:21:17.440 --> 00:21:21.200 That's why, like observability is not just about yes, you're gonna have 282 00:21:21.200 --> 00:21:22.400 to have a Columner store. Yes, you're gonna have to have all these 283 00:21:22.400 --> 00:21:26.599 things in the back end and make it fast. Because the other thing about 284 00:21:26.680 --> 00:21:29.200 logging tools is like if you want to ask something interesting, it's like you 285 00:21:29.319 --> 00:21:32.000 enter the tool, you know, the and then you're like, Okay, 286 00:21:32.000 --> 00:21:34.079 I'm gonna take thirty minutes and go out for coffee because it's gonna like it 287 00:21:34.119 --> 00:21:37.519 has to be fast, it has to be interactive. It has to be 288 00:21:37.599 --> 00:21:40.759 like under a seconds because you're like, you're taking steps and you have to 289 00:21:40.799 --> 00:21:44.680 stay in the zone, right you're yeah, exactly. It has to be 290 00:21:44.720 --> 00:21:47.759 explorable, it has to be interactive, and it has to let you, 291 00:21:48.039 --> 00:21:51.920 I think most importantly, draw on the on the brains of the people around 292 00:21:51.960 --> 00:21:55.920 you. So something we built into Honeycomb is is history. You know, 293 00:21:55.920 --> 00:22:00.119 how you're debugging and it's like, oh, I've lost the thread, so 294 00:22:02.160 --> 00:22:03.440 you can just go that, you scroll back up that's where I knew I 295 00:22:03.440 --> 00:22:06.880 had it right, and you branch out and you try something else. But 296 00:22:06.920 --> 00:22:11.400 then also you have access to the history of everyone on your team. So 297 00:22:11.599 --> 00:22:15.400 if it's like last Thanksgiving, we had this terrible my squel outage, you 298 00:22:15.440 --> 00:22:18.440 know, and everything was uh and Ben and Emily were on call, say, 299 00:22:18.759 --> 00:22:22.160 and then I'm on call in March and I'm like, this feels a 300 00:22:22.200 --> 00:22:26.759 lot like what was happening last last November. I'm gonna go and look at 301 00:22:26.759 --> 00:22:30.000 like, well, what were Ben and Emily doing and what did they say? 302 00:22:30.039 --> 00:22:36.559 Help them find out what trace are So journaling, yeah, actions systems. 303 00:22:36.799 --> 00:22:40.920 History doesn't repeat that. It rhinds right so much. So much of 304 00:22:40.960 --> 00:22:45.599 the wisdom of your like these are socio technical systems. It's not just production. 305 00:22:47.000 --> 00:22:49.440 Like an example I often is this, You've got the New York Times 306 00:22:49.440 --> 00:22:53.880 on the Washington Post. They're like both big newspapers, right, but if 307 00:22:53.920 --> 00:22:59.400 you took their teams and swapped them, you couldn't actually do that because so 308 00:22:59.480 --> 00:23:03.599 much of this this different system lives in the heads of the people us write 309 00:23:03.640 --> 00:23:07.119 it. So like being able to draw on that wisdom and use it, 310 00:23:07.200 --> 00:23:11.559 like it makes you a better engineer. Like all of the ship that I 311 00:23:11.640 --> 00:23:15.519 learned about being an engineer was looking up over their shoulder of amazing engineers that 312 00:23:15.559 --> 00:23:18.039 I got. But it sounds like the journaling approach you're talking about allows us 313 00:23:18.079 --> 00:23:23.759 to look over the best exactly and and and get to know you don't have 314 00:23:23.799 --> 00:23:29.839 to remember how they did it because it's recorded there, and so you know 315 00:23:30.039 --> 00:23:33.880 you can even learn their approach and how they attacked that I feel like, 316 00:23:33.039 --> 00:23:37.160 you know, especially now nowadays, when we're doing so much distributed working, 317 00:23:37.279 --> 00:23:40.759 like remote working, I worry a lot about how are we going to bring 318 00:23:40.839 --> 00:23:44.720 up the next generation of engineers, you know, and I feel I hope 319 00:23:44.720 --> 00:23:47.759 that we're all starting to think about making this more of our tooling. Just 320 00:23:48.000 --> 00:23:51.039 how can we learn from you know, it's kind of embarrassing, but like 321 00:23:51.240 --> 00:23:53.799 when I was in college, I learned so much from just going around and 322 00:23:53.960 --> 00:24:02.200 reading the Bash histories of all the people I knew, from either trying the 323 00:24:02.200 --> 00:24:07.799 commands. You know, it's fucking fascinating. Right, Oh, that's how 324 00:24:07.839 --> 00:24:11.519 you have to learn, said not right, what does this do? I 325 00:24:11.519 --> 00:24:12.720 think we need a lot more of that in our tools. Yeah, and 326 00:24:14.119 --> 00:24:18.200 I worry that we're making it even harder to make that jump from junior to 327 00:24:18.440 --> 00:24:22.480 an intermediate. I mean, we've always had a problem with intermediates anyway, 328 00:24:22.839 --> 00:24:27.000 but a lot of the automation tools that are taking a lot of are eliminating 329 00:24:27.039 --> 00:24:33.039 the beginner stuff. Yeah, like this whole like the generative AI stuff, 330 00:24:33.759 --> 00:24:37.279 Like it's great for senior engineers. You're now so much more productive, you 331 00:24:37.319 --> 00:24:40.839 can put so much faster. But like the way that you get to that 332 00:24:40.880 --> 00:24:48.079 point is it's scarts. How how are we going to force ourselves? I 333 00:24:48.119 --> 00:24:52.039 mean, I believe that the solutions will emerge. I hope it looks pretty 334 00:24:52.039 --> 00:24:56.440 bad. And I also think that younger generation will find them too, because 335 00:24:56.480 --> 00:24:59.640 they are not you know, we've done this show where we're talking about is 336 00:24:59.640 --> 00:25:03.839 all this scar tissue actually holding us back? Right? That we have some 337 00:25:03.880 --> 00:25:07.039 of it? Yeah, some of it's value, I think. You know, 338 00:25:07.079 --> 00:25:10.599 you have to internalize the damage and say, like, what does this 339 00:25:10.680 --> 00:25:12.839 really look like? In generally speaking, And when someone says I will never 340 00:25:12.960 --> 00:25:17.160 use X product or X technique, it's like you have not eternalized your scar 341 00:25:17.160 --> 00:25:22.000 as well. I love that every team has, I think, but also 342 00:25:22.039 --> 00:25:26.640 that can they learn to speak to These are the concerns I have when you 343 00:25:26.680 --> 00:25:30.480 think about the broader approaches to things that might have created that problem back in 344 00:25:30.519 --> 00:25:33.799 the past. I read this book called The Trauma of Everyday Life, which 345 00:25:33.839 --> 00:25:38.160 is written by this guy who's as psychiatrist and a Zen Buddhist, and he's 346 00:25:38.200 --> 00:25:45.559 talking about how trauma isn't necessarily something to be avoided because it's literally what shakes 347 00:25:45.599 --> 00:25:49.559 you think about a bond sire that's just a normal tree, but it was 348 00:25:49.599 --> 00:25:53.519 it was put in this very specific where its roots couldn't grow right, and 349 00:25:53.559 --> 00:25:57.720 so it's not it's like not RECTI people like trauma is great, but it's 350 00:25:57.759 --> 00:26:02.759 also like there's scar. She was just going to be different. And again, 351 00:26:02.759 --> 00:26:03.519 how you react to and how you work to it, you can make 352 00:26:03.559 --> 00:26:08.640 beautiful things. So what are some of the other pitfalls that people will encounter 353 00:26:08.839 --> 00:26:15.640 when sort of moving to this observability. The big one is the cognitive just 354 00:26:15.920 --> 00:26:19.480 the model that we have in our brain. I feel like our industry has 355 00:26:22.240 --> 00:26:23.920 avoided this for a long time. I feel like there's a bit of a 356 00:26:23.960 --> 00:26:29.039 reckoning. You know, Open telemetry. By the way, I've got to 357 00:26:29.039 --> 00:26:33.119 put in a quick plug for open here. It's amazing. I know all 358 00:26:33.160 --> 00:26:37.240 of us hate redoing our code, but like the promise of open telemetry is 359 00:26:37.599 --> 00:26:42.839 you reinstrument your code once and then vendors have to compete for your dollars based 360 00:26:42.880 --> 00:26:48.720 on being awesome instead of having you locked in. It is. It's it's 361 00:26:48.759 --> 00:26:55.559 the number two project after Kubernetes in the what you call it thank you CF 362 00:26:55.960 --> 00:26:59.519 It's super active. A lot of contributors. I was pretty skeptical about this, 363 00:26:59.640 --> 00:27:03.119 but it's it's it's the way I wish we had had this ten years 364 00:27:03.119 --> 00:27:04.880 ago. I think we'd all parties. You know, we could have we 365 00:27:04.920 --> 00:27:10.119 could have just chose to political problem, not detectives totally. We're there now, 366 00:27:10.400 --> 00:27:12.160 but here we are. These are the tools that we have. Open 367 00:27:12.160 --> 00:27:18.039 telemetry is worth putting on your roadmap for the next year or two because there's 368 00:27:18.119 --> 00:27:22.680 also this this reckoning that's happening with costs right now. Most of these vendors 369 00:27:22.759 --> 00:27:27.480 are billing just like ungodly amounts of dollars that do not correspond to the value 370 00:27:27.480 --> 00:27:30.359 that you get out of them because they can't because they got you're locked. 371 00:27:30.680 --> 00:27:33.720 Yeah, yeah, And so I feel like we need to take her powers 372 00:27:33.720 --> 00:27:37.880 well. And it's a great pitch for a feature that's not necessarily a new 373 00:27:37.920 --> 00:27:41.119 features to say, hey, I can reduce our costs by moving us off 374 00:27:41.279 --> 00:27:45.160 this tool and onto open telemetry. You know, I'm compliance some of these 375 00:27:45.200 --> 00:27:49.319 sidebar rants here, but like I feel like learning to treat like one artifact 376 00:27:49.400 --> 00:27:52.839 of the zero interest rate like period, was it engineers forgot how to talk 377 00:27:52.880 --> 00:27:56.839 about our work in terms of dollars, you know, because like dollars are 378 00:27:56.920 --> 00:28:00.720 the universal denominator. Maybe something the Euros. I don't know, but like 379 00:28:00.880 --> 00:28:06.519 money is the universal denominator, and if we can't learn to talk about the 380 00:28:06.640 --> 00:28:11.400 value of the shit that we provide to people in finance people, I feel 381 00:28:11.400 --> 00:28:15.279 like many many vps of engineering and CTOs have this phenomena where they feel like 382 00:28:15.319 --> 00:28:18.160 the junior partner at the table. They aren't really invited to all the critical 383 00:28:18.200 --> 00:28:21.960 meetings and stuff. And I believe that that's because we haven't learned to talk 384 00:28:22.000 --> 00:28:26.640 about the value that we bring and cost in the same language as every other 385 00:28:26.680 --> 00:28:29.359 team. Because if we did, we generate a lot of value. We 386 00:28:29.599 --> 00:28:34.200 generate all company, We have all the power we should need to have. 387 00:28:34.599 --> 00:28:37.920 We got to do is get a hand on it. And Charity when they 388 00:28:37.960 --> 00:28:45.839 were up for one moment for this very important message and we're back. It's 389 00:28:45.880 --> 00:28:48.039 not at Rocks. I'm Richard Campbell. Let's Carl Franklin. Hey to our 390 00:28:48.039 --> 00:28:56.920 friend Charity Majors about observability engineering and watching the sausage being made. And so 391 00:28:56.079 --> 00:29:00.279 I want to follow up on you. You brought up generative to a and 392 00:29:00.359 --> 00:29:04.960 things for programmers. It's great for senior programmers who can be more productive with 393 00:29:06.039 --> 00:29:08.799 stuff they might have forgotten how to write or don't really care to figure out 394 00:29:08.880 --> 00:29:14.720 and just let chat GPT do it for you. But what do you think 395 00:29:14.880 --> 00:29:18.839 the future of observability is, especially in lieu of AI and where it's going. 396 00:29:18.920 --> 00:29:23.839 Do you think that we'll have AI bots sort of watching our telemetry and 397 00:29:23.920 --> 00:29:30.480 giving us English prompts, you know, sending us text messages. Vendors are 398 00:29:30.519 --> 00:29:33.799 going to sell CTOs and pps like tens of billions of dollars worth of bullshit 399 00:29:33.880 --> 00:29:37.200 that says that they can do that. Yeah, something that blew my mind 400 00:29:37.200 --> 00:29:41.599 when I became to see So are you saying we don't need this? We 401 00:29:41.160 --> 00:29:45.319 have everything that we need right there in front of something that blew my mind 402 00:29:45.319 --> 00:29:51.279 when I keep CTO to be wild internalized. But the most executives have more 403 00:29:51.319 --> 00:29:56.720 trust and confidence in their vendor relationships than their employees, because employees coming out 404 00:29:56.039 --> 00:30:02.759 the vendors left forever as long as you keep paying them. In my mind, 405 00:30:02.200 --> 00:30:06.759 but what they're selling when they come in. This is why my dander 406 00:30:06.839 --> 00:30:10.440 got raised so much by the whole AIO saying, because they are all just 407 00:30:10.640 --> 00:30:12.640 like you don't need to understand your systems. Pay us all this money, 408 00:30:12.799 --> 00:30:17.440 we'll understand it for you, and but like the false positives are ridiculous and 409 00:30:17.480 --> 00:30:19.920 off the charts, all of the data is junk. You would be better 410 00:30:21.000 --> 00:30:23.680 off just like turning off all that data, Like it's just so many problems 411 00:30:23.720 --> 00:30:29.440 with it. I believe that we should be looking at computers that do what 412 00:30:29.519 --> 00:30:33.440 computers do best, and people to do it people do best, and computers 413 00:30:33.480 --> 00:30:37.319 crunch numbers, people attack meaning to things. Sure, like your graphs are 414 00:30:37.359 --> 00:30:42.039 spiking all day long, most of them you don't care about, because our 415 00:30:42.039 --> 00:30:45.599 computers are now resilient to a whole lot of failure. Sure, it really 416 00:30:45.680 --> 00:30:52.160 takes a person coming along and going matters. That matters often because it mattered 417 00:30:52.200 --> 00:30:56.000 to another person, and you're the person who they're connecting those dots. And 418 00:30:56.039 --> 00:31:00.480 once you've decided it matters, you need to understand why. And I think 419 00:31:00.480 --> 00:31:03.200 there are all kinds of ways for computers to help us do that. We 420 00:31:03.279 --> 00:31:07.200 do this really cool thing called bubble up the honeycomb, where any graph that 421 00:31:07.319 --> 00:31:11.160 any heat map that you've constructed, you draw a little bubble around something you're 422 00:31:11.160 --> 00:31:14.720 like, I care about this, and then we compute the baseline for all 423 00:31:14.759 --> 00:31:18.920 the hundreds of dimensions and the dimensions that are inside the thing you care about, 424 00:31:18.079 --> 00:31:21.640 and then we dip them and sort them, so it's like, Okay, 425 00:31:21.680 --> 00:31:23.759 this thing you care about, here are the five to ten ways that 426 00:31:23.880 --> 00:31:27.200 is different from everything that you don't care about. Computers are great at that, 427 00:31:27.720 --> 00:31:30.160 but they can't tell you what to care about, and they shouldn't try 428 00:31:30.160 --> 00:31:33.880 it, because it's a fucking mess. Maybe ten years from now I will 429 00:31:33.920 --> 00:31:37.039 be eating my words, but for the foreseeable future, I really think that 430 00:31:37.079 --> 00:31:44.160 we're all best served if we focus on helping people understand what has meaning and 431 00:31:44.240 --> 00:31:45.599 letting computers take care of their rest. Yeah. I mean I can see 432 00:31:45.640 --> 00:31:49.240 the machine tools helping to point us too unusually. Sure, Yeah, but 433 00:31:49.440 --> 00:31:53.359 you still have to interpret them. Yeah. Still you want them to create 434 00:31:53.400 --> 00:31:57.440 that graph for you. You want them to intelligently sample often, you want 435 00:31:57.480 --> 00:32:00.799 them to do you know, but you don't don't want them in the business 436 00:32:00.799 --> 00:32:04.160 of telling you what that No, they don't know, you don't know, 437 00:32:04.359 --> 00:32:08.279 and more and more stately, like they're not even qualified to make that s 438 00:32:08.319 --> 00:32:13.119 that's been in any way. That being said, like I can tell you 439 00:32:13.160 --> 00:32:15.119 we're talking a lot of old lap terms here, like a lot of data 440 00:32:15.200 --> 00:32:21.640 analytic terms around all this and machine learning models evolved from a lot of that 441 00:32:21.759 --> 00:32:24.680 technology. So you can see a shape of this shape of history. You 442 00:32:24.799 --> 00:32:29.799 can't see a shape, but I don't believe that it is. It is 443 00:32:29.920 --> 00:32:35.000 one that So here's the thing. At the bottom line, we are forget 444 00:32:35.000 --> 00:32:38.160 technology. We are held legally accountable for your engineers. We are legally and 445 00:32:38.240 --> 00:32:43.400 ethically and morally accountable for the codes we put out into the world. Right, 446 00:32:43.880 --> 00:32:46.279 we can't point an algorithm it comes to that, even if it's a 447 00:32:46.359 --> 00:32:50.519 machine learning I think, I don't know if you've read it yula lately, 448 00:32:51.359 --> 00:32:53.160 A boy, oh boy, they work really hard to make sure we're not 449 00:32:53.279 --> 00:33:01.160 legally accountable for any I believe in the near infinite possibility employers. That's that's 450 00:33:01.200 --> 00:33:07.039 true. I want to make sure that I understand. I also, I 451 00:33:07.079 --> 00:33:09.559 mean, I like that we're also going to moral and ethical aspect because I 452 00:33:09.559 --> 00:33:14.559 think we need. I think that legal aspects holding us back, that we 453 00:33:14.599 --> 00:33:17.920 can't own the value of what we makeout, that we can't own the value 454 00:33:17.960 --> 00:33:23.559 of a make as long as we're obligating our responsibilities. And then really, 455 00:33:23.599 --> 00:33:28.480 you know, the yula was invented to allow us to not hold liability for 456 00:33:28.519 --> 00:33:34.039 the impact of our software, and so we're kind of in a trap right 457 00:33:34.079 --> 00:33:37.960 as an industry. If we were responsible for the damage we did, we 458 00:33:38.000 --> 00:33:43.480 would we would our employers would insist on higher standards because they're getting caught up 459 00:33:43.480 --> 00:33:47.759 in that as well. But because we've avoided the responsibility so thoroughly, I 460 00:33:47.799 --> 00:33:52.400 see what you're saying. That being said like this is now we get into 461 00:33:52.400 --> 00:33:54.880 a pretty deep philosophical side of this thing, like let's face it, good 462 00:33:54.920 --> 00:33:58.640 telemeter. In the end, we're trying to understand why is the software behavior 463 00:33:58.640 --> 00:34:01.759 and its behavior? Why are our customers unhappy? I mean, those are 464 00:34:01.759 --> 00:34:06.519 the things that actually matter. I think. The more often that we as 465 00:34:06.599 --> 00:34:09.639 technologists speak in the term of the customers, I think, why are our 466 00:34:09.639 --> 00:34:13.719 customers onhappy? You know? And this is something I've been really grappling with 467 00:34:13.840 --> 00:34:15.039 lately. I don't know if I'm alone in this or not, but I 468 00:34:15.079 --> 00:34:22.719 have like an almost knee jerk, almost disgusted or like reaction towards like customer 469 00:34:22.920 --> 00:34:27.840 and value and things. And I've been trying to because we've been battered with 470 00:34:27.920 --> 00:34:31.440 it, because we've been battered beaten up those words. Yeah, I don't 471 00:34:31.480 --> 00:34:36.320 know, just the business aspect, like I think there's some vestors of me. 472 00:34:36.400 --> 00:34:38.440 There's still like ew, we're better than that. And I hate myself 473 00:34:38.440 --> 00:34:42.679 as I'm saying that. You know, and you're also open with the dollars 474 00:34:42.719 --> 00:34:45.639 matter they do, they're kind to come from the customers. Oh, you 475 00:34:45.639 --> 00:34:47.880 should have seen me ten years ago, because this is a chill version. 476 00:34:47.960 --> 00:34:52.440 I get that. Okay, No, but you're you're absolutely right. We 477 00:34:52.519 --> 00:34:54.239 do this for the customer. We do this for our users. That's the 478 00:34:54.320 --> 00:34:58.840 reason we exist, and we have a responsibility to them. Sure, And 479 00:34:58.880 --> 00:35:00.840 I don't think I'll ever still comfortable saying, well, the machine told me 480 00:35:00.920 --> 00:35:05.239 it was fine. That's a cop out every time. Because the machine didn't 481 00:35:05.280 --> 00:35:08.079 tell you anything, you interpreted it and chose to vocal. You know, 482 00:35:08.159 --> 00:35:14.559 in the end, everything we've talked about program it's fine, getting very philosophical, 483 00:35:14.679 --> 00:35:17.159 but also none of this is described an action we should take. All 484 00:35:17.159 --> 00:35:21.440 we're doing is observe what's going on. We still have to decide on the 485 00:35:21.480 --> 00:35:24.079 action. How would you change the code? Given you've seen this in dilematry 486 00:35:24.199 --> 00:35:28.000 and you know what else? Like I feel like this looks back really nicely 487 00:35:28.039 --> 00:35:31.679 into just like what is what is the meaningful life? Right? Like because 488 00:35:31.760 --> 00:35:36.639 like that book that what's his face wrote about about work and what makes us 489 00:35:36.679 --> 00:35:40.800 happy is it's not like having twenty hours a day whatever, but it's like 490 00:35:42.199 --> 00:35:46.880 autonomy, mastery and meaning purpose. Yeah, Daniel pink, Dangel pink, 491 00:35:46.960 --> 00:35:51.960 thank you, and like the meaning the purpose that comes into play for us 492 00:35:52.000 --> 00:35:53.920 when it impacts other people. Well, and you hit on the key thing, 493 00:35:53.960 --> 00:35:58.800 which is when we crack this, not like every time you chase a 494 00:35:58.840 --> 00:36:01.480 problem downline that and it turns into a code change, you can make that 495 00:36:01.639 --> 00:36:07.079 then in later testing shows that problems occur. Boy, that's a good day. 496 00:36:07.280 --> 00:36:13.000 Like you talk about purpose, there is nothing better than figuring that complicated 497 00:36:13.000 --> 00:36:16.880 problem out and then literally, like you, you live in a very hypothesis 498 00:36:16.960 --> 00:36:21.119 based world. It's like, well, I've seen this telemetry, I've seen 499 00:36:21.119 --> 00:36:24.119 this output. I believe it's this code problem. Now I'm going to make 500 00:36:24.119 --> 00:36:28.559 a modification. I'm going to put it into the stream and I'm going to 501 00:36:28.639 --> 00:36:31.280 go back and test again. And if I don't see it, then I 502 00:36:31.400 --> 00:36:35.679 can, you know, hypothesize really because I might be wrong. We may 503 00:36:35.719 --> 00:36:39.079 not have reconded recreated conditions perfectly. That we're on it, that we're pushing 504 00:36:39.079 --> 00:36:43.320 the right thing, and nobody knows just how deep that went. No, 505 00:36:43.880 --> 00:36:46.360 I also wonder, you know how many times have you been fighting a problem 506 00:36:46.440 --> 00:36:51.159 like that and you chart changing code? Just see if you can change behavior 507 00:36:51.679 --> 00:36:55.440 at all? Like, am I even assistant? They have emergent properties, 508 00:36:55.440 --> 00:37:00.199 They're no longer like I feel like part of moving from like the old version 509 00:37:00.199 --> 00:37:05.880 of the new is except that TDD is not enough interesting like the tests tell 510 00:37:05.960 --> 00:37:10.039 you will this logically execute, but that reality ends at the border of your 511 00:37:10.079 --> 00:37:14.320 laptop. Yes, and the universe is weirder. The weird intera is so 512 00:37:14.440 --> 00:37:16.920 much weirder than that. I feel like our jobs are not done. It's 513 00:37:16.920 --> 00:37:22.719 like until we've instrumented that code, deployed it and watched it in production and 514 00:37:22.800 --> 00:37:25.400 asked ourselves, is it doing what I expected to do? And if anything 515 00:37:25.400 --> 00:37:30.119 else look weird? I know that on the show before I was you know, 516 00:37:30.199 --> 00:37:31.719 I've did a lot of load testing. It's like I have never invented 517 00:37:31.719 --> 00:37:37.159 the load tests as weird as customers on Saturday actually comes even come close. 518 00:37:37.480 --> 00:37:44.480 So customers are evil do things you can't. They really opened six windows and 519 00:37:44.519 --> 00:37:47.400 hit refresh all at the same time. Did he really really? Okay? 520 00:37:49.360 --> 00:37:52.679 May I see some practical advice on behalf of the listeners. So let's say 521 00:37:52.679 --> 00:38:00.559 you're listening, you're you've been surfing, you went to uncombed dot I and 522 00:38:00.639 --> 00:38:02.400 you checked it out, and you're thinking this might be good. How do 523 00:38:02.440 --> 00:38:06.000 you go back to here? How do these people in the audience go back 524 00:38:06.039 --> 00:38:10.320 to their teams and introduce this concept without getting flogged? You know? How 525 00:38:10.519 --> 00:38:14.880 how do you approach that? I mean, that's a great question. My 526 00:38:14.960 --> 00:38:20.199 approach is always to look for something that's really painful, like, you know, 527 00:38:20.599 --> 00:38:23.199 things that are going down, you don't understand, problem, problem, 528 00:38:23.239 --> 00:38:29.159 you can't crack and especially this the siloed approach to telemetry that we're doing anyhow, 529 00:38:29.239 --> 00:38:30.840 things that are waking people up in the middle. Then I you know, 530 00:38:31.000 --> 00:38:34.679 we've seen this a lot where you know, people have tried to bring 531 00:38:34.760 --> 00:38:37.440 it in whatever, but then there's an intractable problem and they put money come 532 00:38:37.480 --> 00:38:42.559 on it, and it's just like like we've even had multiple times we've had 533 00:38:42.559 --> 00:38:47.360 our sales engineers doing demos on people's production systems and you're about to have an 534 00:38:47.360 --> 00:38:51.719 outage here because this thing's happened, and they're like what in the like ten 535 00:38:51.719 --> 00:38:55.440 minutes later they get paiged because it is that Like I know, I'm a 536 00:38:55.440 --> 00:38:59.400 founder, believe nothing I say, But is that much easier when you have 537 00:38:59.480 --> 00:39:01.440 the right tool, when you have the right visibility, just to be able 538 00:39:01.440 --> 00:39:05.079 to see what's going on? Yeah, looking at something like that, or 539 00:39:05.079 --> 00:39:09.280 somewhat counterintuitively the other side, another place we've seen a lot of success is 540 00:39:09.280 --> 00:39:14.559 people insumenting their CiCe pipelines, right, because if you insument your CCE pipeline 541 00:39:14.559 --> 00:39:17.199 as a trace, you've can see where all that time is going. Yeah. 542 00:39:17.480 --> 00:39:22.320 Yeah, that's kind of another approach to this, the model of what 543 00:39:22.480 --> 00:39:27.119 is the hard work here? What's actually hurting us? The struggle is only 544 00:39:27.119 --> 00:39:30.760 getting in the front door. We have like zero turn if the company didn't 545 00:39:30.760 --> 00:39:34.480 go out of business to keep buying us. But it's difficult to get in 546 00:39:34.519 --> 00:39:37.280 the front door. But once we get inside, like no, but I 547 00:39:37.280 --> 00:39:39.679 think you've made the most compelling argument, and that's going to be tough for 548 00:39:39.719 --> 00:39:43.400 anyone in the room wo's thinking about this. It's like you have to go 549 00:39:43.440 --> 00:39:45.960 pick the largest dragon in the room and say, I think I could take 550 00:39:46.000 --> 00:39:50.760 that one on if I had this lance. If I can get this lance 551 00:39:50.800 --> 00:39:52.360 and gay it go, I'll go for the big guy. Yeah, and 552 00:39:52.400 --> 00:39:55.239 that's the kind of bet you need to make. But you know the underlying 553 00:39:55.280 --> 00:39:58.519 part of this, because a lot of the software is already set up for 554 00:39:58.519 --> 00:40:00.960 the right plum tree, but it's the customs suff we're building it is not 555 00:40:00.960 --> 00:40:05.559 is how you provide give visibility into that yep, orienting it around. You 556 00:40:05.559 --> 00:40:07.199 know a lot of people also come most when the start their open plumbatry journey 557 00:40:07.199 --> 00:40:12.440 because we have many of the world's best experts in Hotel, so we could 558 00:40:12.440 --> 00:40:16.280 actually help consult. What do I need to push onto the open telemet pelementary 559 00:40:16.320 --> 00:40:20.320 stack. That's going to help me, that's going to let these tools understand. 560 00:40:20.320 --> 00:40:23.920 What do any of you in the room have a question for charity? 561 00:40:24.119 --> 00:40:28.280 Raise your hand, It's all right right here. Phil Hack has a question, 562 00:40:28.719 --> 00:40:31.280 So why don't you repeat the question? The question is that's a lot 563 00:40:31.320 --> 00:40:35.880 of data and how does that cost? How does the coast get out? 564 00:40:36.119 --> 00:40:39.119 Is the costack get out? Again? This is why So on Twitter I 565 00:40:39.159 --> 00:40:42.320 was joking the other week and it kind of got out of control and I 566 00:40:42.320 --> 00:40:45.039 could never write a database. No, really, never write a database. 567 00:40:45.320 --> 00:40:49.239 And I thought it was a very fun self owned because we wrote a database. 568 00:40:49.480 --> 00:40:52.639 People didn't understand that. So yeah, it's a fund of data. 569 00:40:52.719 --> 00:40:57.920 Like we've got like seven hundred customers and we run the combined production modes of 570 00:40:57.960 --> 00:41:00.000 all of them. It's like two billion events persons or something like that, 571 00:41:01.719 --> 00:41:07.480 and we give everyone sixty days of storage basically for free. And the way 572 00:41:07.480 --> 00:41:15.000 that we do this we so it's a culundar store. So indexes are roboten 573 00:41:15.119 --> 00:41:17.760 for observability because indexes are way of picking. I want this to run fast 574 00:41:17.760 --> 00:41:21.000 and nothing else to run fast. You want to be able to query on 575 00:41:21.079 --> 00:41:24.360 any of these dimensions. So it's a calundar store. And you're right, 576 00:41:24.480 --> 00:41:28.480 like two years in we ran into this. We're never going to be profitable 577 00:41:28.480 --> 00:41:31.840 because there's so all of these SSDs, all this ram. And that's when 578 00:41:32.119 --> 00:41:37.760 one of my I brilliant engineer I've been working he was my first manager. 579 00:41:37.039 --> 00:41:40.639 Name is Ian. He's he's nowhere on the internet and he's amazing. Uh. 580 00:41:40.800 --> 00:41:45.760 He started looking into the cost models and did some tests and so now 581 00:41:45.840 --> 00:41:51.280 actually we data comes in hits, the API gets dropped into Kaffa and then 582 00:41:51.320 --> 00:41:54.840 gets read off onto you know a pair of notes, which are as you 583 00:41:54.840 --> 00:41:59.840 would think like lots of CPO lots of RAM, but then after like thirty 584 00:42:00.000 --> 00:42:06.320 six minutes it gets tailed out to S three. The queery planner actually runs 585 00:42:06.360 --> 00:42:12.400 the LANDA jobs. Uh so the query planner comes in forks out spans and 586 00:42:12.400 --> 00:42:15.719 and like we thought it was going to be so much shorter like doing processing, 587 00:42:15.800 --> 00:42:20.880 you know from all these S threeboutives, it wasn't. It was different 588 00:42:20.920 --> 00:42:24.480 performance characteristics, but most careers still return with under a second and S three 589 00:42:24.559 --> 00:42:29.320 is the cheap so that's what most of the data is, and the lambda 590 00:42:29.400 --> 00:42:31.239 jobs are pretty expensive. That's a big line end up in on our bills. 591 00:42:31.239 --> 00:42:35.920 So we've done we've actually done some really great talks and written some great 592 00:42:35.920 --> 00:42:40.320 pieces about how we use Honeycomb to optimize our LANDA jobs so that the planner, 593 00:42:40.960 --> 00:42:45.360 yeah, it's all. Yeah. Our Honeycomb block, by the way, 594 00:42:45.480 --> 00:42:49.639 is dope. Like we we don't do a lot of selling there. 595 00:42:49.679 --> 00:42:52.519 We just talk about a lot of engineering and it's pretty great. There was 596 00:42:52.559 --> 00:42:58.320 another question back here. First somebody back there had there end up. Okay, 597 00:42:58.320 --> 00:43:00.920 it wasn't you, but go ahead, son, repeat the question. 598 00:43:00.960 --> 00:43:07.159 I'm sorry. So the question I think was something there's something about SLOs and 599 00:43:07.559 --> 00:43:12.920 metrics and it's too expensive to store all the choices for all events, okay, 600 00:43:14.280 --> 00:43:17.079 And there's a few different answers for this. I would probably want to 601 00:43:17.079 --> 00:43:21.880 ask you some more questions. It's feel free to find me afterwards. But 602 00:43:21.960 --> 00:43:24.400 you're absolutely right. It can be absolutely cost prohibitive to store the trace for 603 00:43:24.480 --> 00:43:28.039 every if you have a lot of traffic, because if you think about it, 604 00:43:28.360 --> 00:43:34.400 you might find yourself storing five to thirty times as much prelemetry data as 605 00:43:34.559 --> 00:43:38.960 production traffic. Obviously that's not tenable, right. The first solution that we 606 00:43:39.199 --> 00:43:45.840 usually steer people towards is intelligence sampling, which does not mean just like dumb 607 00:43:45.920 --> 00:43:47.400 dumb sampling, we're like one out of the routen you drop them. It 608 00:43:47.440 --> 00:43:52.000 means like we have a thing called refinerate, where there's a different between head 609 00:43:52.039 --> 00:43:54.960 sampling and tail sampling, meaning sampling before you know what it is coming and 610 00:43:55.079 --> 00:43:59.079 after you know what it's coming. So some of these things you sample after 611 00:43:59.119 --> 00:44:01.159 you know what's coming. Be sure and grab all of the slow events, 612 00:44:01.639 --> 00:44:05.599 right. Some of it is head sampling where you're just like, okay, 613 00:44:06.960 --> 00:44:10.119 for example, requests there are health checks, there are two hundreds. This 614 00:44:10.239 --> 00:44:13.800 is junk. I don't need to store all of these there's gonna be like 615 00:44:13.800 --> 00:44:16.800 a quarter of your traffic sometimes, so like sample them heavily, two hundred 616 00:44:16.840 --> 00:44:22.639 okays to the main page, sample the medium, keep every request that's in 617 00:44:22.840 --> 00:44:25.519 error, or every request that is to slash payments or to billing or you 618 00:44:25.559 --> 00:44:30.880 know, like there's a lot of that is trash that you can like to 619 00:44:30.159 --> 00:44:35.519 discard if you kind of go in there with the fine teeth. Come the 620 00:44:35.559 --> 00:44:44.480 part that was about about SLOs. We derive SLOs from events, and it's 621 00:44:44.519 --> 00:44:50.800 actually really important that they're not derived from metrics. We're actually the only product 622 00:44:50.800 --> 00:44:54.400 out there that does SLOs the way they're supposed to be done for the Google 623 00:44:54.480 --> 00:44:59.360 sor rebook, because other companies don't actually capture their data in a way that 624 00:44:59.440 --> 00:45:04.159 lets them do that. It's actually pretty dope. Like you have your SLOs, 625 00:45:04.199 --> 00:45:07.000 it tells you how how quickly you're burning down the budget, and then 626 00:45:07.320 --> 00:45:12.559 it tells you what what is different about the requests that are erring that are 627 00:45:12.559 --> 00:45:16.880 burning down the budget the other blah blah blah. We also have the metrics, 628 00:45:17.239 --> 00:45:22.599 but I'm not sure if you're talking about our metrics product or the events. 629 00:45:22.800 --> 00:45:25.039 Like the number one answer to the events be too expensive is you use 630 00:45:25.920 --> 00:45:30.039 smart sampling? And the number one answer to the SLOs is you want those 631 00:45:30.280 --> 00:45:34.239 badly. You can absolutely do sampling. So one of the one of the 632 00:45:34.280 --> 00:45:37.239 things in every event that ge is sent to us, there is a sample 633 00:45:37.320 --> 00:45:40.840 rate embedded in it, so everyone will say like one slash five and that 634 00:45:40.960 --> 00:45:45.920 means compute this to be five like this, so the numbers all all work 635 00:45:45.960 --> 00:45:50.880 out to look like they weren't sampled. We had questions. You got to 636 00:45:50.880 --> 00:45:52.960 move on question right in the front, hay on can you repeat that with 637 00:45:53.000 --> 00:45:58.000 the micro The question was, obviously, you can go wrong with logging because 638 00:45:58.079 --> 00:46:00.800 you can get way too many log events. You can go wrong with metrics 639 00:46:00.840 --> 00:46:04.760 because you could you can have famously like the thirty thousand dollars metric that had 640 00:46:04.840 --> 00:46:08.079 high cardinality in it and oops your budget. Answer the question is, I 641 00:46:08.079 --> 00:46:14.760 think, how can you go wrong with observability as distinct from those you know? 642 00:46:14.840 --> 00:46:16.800 And I want to say, even if you aren't doing traces, if 643 00:46:17.920 --> 00:46:22.280 the most important thing to take away when it comes to telemetry is the magic 644 00:46:22.360 --> 00:46:28.480 of the one wide structured event per request per service. I actually found out 645 00:46:28.639 --> 00:46:31.159 years into this that this is how Amazon is done there telemetry all along, 646 00:46:31.480 --> 00:46:36.920 they had like a flat file at the root domain of every node where they 647 00:46:37.000 --> 00:46:40.880 keep one of these like wide It's it's magic. It makes everything because a 648 00:46:40.880 --> 00:46:45.840 lot of the logs that you are encountering, or because like when a request 649 00:46:45.880 --> 00:46:51.159 is executing through through a service, it's just like, oh, all these 650 00:46:51.159 --> 00:46:54.480 strings, right, But if you just like collapse them into one wide event 651 00:46:54.760 --> 00:47:00.199 with all of those keys and values, then you have that context, right 652 00:47:00.199 --> 00:47:04.719 you can put its magic. So the number one thing that I think people 653 00:47:04.800 --> 00:47:07.960 get wrong with observability is not understanding that that's the heart of everything. It 654 00:47:07.960 --> 00:47:12.480 isn't much tool you're using. It isn't whether you're tracing or not. It's 655 00:47:12.559 --> 00:47:16.199 that it's that that is the number one thing that everyone should be caring about. 656 00:47:16.719 --> 00:47:22.559 The number two thing I think comes out when dealing with spans slightly ordered 657 00:47:22.679 --> 00:47:25.960 higher order problem, and that's because I feel like as an industry we have 658 00:47:27.079 --> 00:47:31.400 the really we aren't really we don't really have a set of like good conventions. 659 00:47:31.440 --> 00:47:34.760 You were asking me, like, when should do you have this span? 660 00:47:35.679 --> 00:47:37.880 Man? Like, I hope five years from Everybody's like, well, 661 00:47:37.880 --> 00:47:40.079 obviously you should have this span blah blah blah. But we aren't there yet, 662 00:47:40.239 --> 00:47:45.800 right, and so it's really easy to either generate too many spans and 663 00:47:45.840 --> 00:47:47.599 they get lost in the noise kind of like with logs, or too few 664 00:47:47.639 --> 00:47:52.199 spans and then not have the detail that you need when you needed. The 665 00:47:52.280 --> 00:47:57.639 question is where to start with open planetry, And there are only two good 666 00:47:57.679 --> 00:48:02.920 answers. One is my favorite, uh, with the biggest pain and if 667 00:48:02.960 --> 00:48:07.719 you have to really like you're like, if you have a really resistant culture, 668 00:48:08.079 --> 00:48:12.719 then start with the least pain. But I actually think that the best 669 00:48:12.719 --> 00:48:15.239 way to roll anything that has to do with cymmetry out is is it kind 670 00:48:15.280 --> 00:48:20.360 of think of your attention like a headlamp, and if you're on call for 671 00:48:20.360 --> 00:48:24.480 something that's breaking, have an instruments first mentality, like you've instrument to figure 672 00:48:24.480 --> 00:48:29.400 out what's wrong, not if you've around with your instrument, have to tell 673 00:48:29.400 --> 00:48:31.280 you the answer, and then it's there for the next time you get paid 674 00:48:31.280 --> 00:48:36.000 again, instrument to find the problem, and it's there. And as your 675 00:48:36.199 --> 00:48:38.639 head lamp kind of moves around the stack, you know, within a couple 676 00:48:38.679 --> 00:48:42.840 of months most of the stuff that it really matters will be instrumented and then 677 00:48:42.840 --> 00:48:45.519 you can put it on the backlog to do the rest and finish up and 678 00:48:45.559 --> 00:48:47.719 get rid of your ovenders. All right, Well, I think that's it, 679 00:48:47.800 --> 00:48:53.559 so let's give charity majors a big round of law. I will see 680 00:48:53.599 --> 00:49:21.480 you next time. On time dot net Rocks is brought to you by Franklin's 681 00:49:21.519 --> 00:49:25.480 Net and produced by Pop Studios, a full service audio, video and post 682 00:49:25.480 --> 00:49:30.239 production facility located physically in New London, Connecticut, and of course in the 683 00:49:30.280 --> 00:49:37.159 cloud online at pwop dot com. Visit our website at d O T N 684 00:49:37.199 --> 00:49:40.760 E t R O c k S dot com for RSS feeds, downloads, 685 00:49:40.920 --> 00:49:45.159 mobile apps, comments, and access to the full archives going back to show 686 00:49:45.239 --> 00:49:50.519 number one, recorded in September two thousand and two. And make sure you 687 00:49:50.599 --> 00:49:53.159 check out our sponsors. They keep us in business. Now, go write 688 00:49:53.159 --> 00:50:01.559 some code, See you next time you got jacks. See a summer time 689 00:50:01.760 --> 00:50:07.960 on that means home. Then my texes in my credit b