WEBVTT 1 00:00:01.080 --> 00:00:04.799 How'd you like to listen to dot net rocks with no ads? Easy? 2 00:00:05.360 --> 00:00:09.839 Become a patron for just five dollars a month you get access to a private 3 00:00:10.000 --> 00:00:14.439 RSS feed where all the shows have no ads. Twenty dollars a month will 4 00:00:14.439 --> 00:00:18.839 get you that and a special dot net Rocks patron mug. Sign up now 5 00:00:18.839 --> 00:00:35.960 at Patreon dot dot NetRocks dot com. Hey guess what, it's dot net 6 00:00:36.079 --> 00:00:42.200 rocks. I'm Carl Franklin at Amgard Campbell and we're representing four time zones today. 7 00:00:42.520 --> 00:00:47.399 Nice. We have two guests. Everybody in their own time zone. 8 00:00:48.000 --> 00:00:50.759 We're in their little corner of the world and their little corner of the world. 9 00:00:50.880 --> 00:00:54.799 However, electrons do not observe time zones, so we can all be 10 00:00:54.960 --> 00:00:59.119 together on this call. I think they observe it a little bit, maybe, 11 00:00:59.159 --> 00:01:02.000 you know, maybe a little bit. You know, Apple has some 12 00:01:02.119 --> 00:01:08.480 software. There's an old reference for you story. That was a great story 13 00:01:08.799 --> 00:01:15.560 call. Apple's got some software coming up, all right, speed light, 14 00:01:15.719 --> 00:01:19.640 hard to beat. Yeah, it's kind of the law Buffalo funny. How 15 00:01:19.640 --> 00:01:23.159 that works? Well, Uh, let's get right into it with better no 16 00:01:23.239 --> 00:01:34.799 framework awesome? Alright, Well, I found this series on Microsoft Learned. 17 00:01:36.200 --> 00:01:40.680 Oh that is really cool. It's and you know what's great about this is 18 00:01:40.719 --> 00:01:47.400 that it's a it's fundamental stuff, leveraging branches with GitHub and Visual Studio and 19 00:01:47.560 --> 00:01:51.280 intermediate series. It doesn't talk over your head and it doesn't talk down to 20 00:01:51.319 --> 00:01:55.239 you. But you know, there are those guys on the team that are 21 00:01:55.280 --> 00:01:59.519 afraid of Gethub and afraid of get you know, I used to be one 22 00:01:59.519 --> 00:02:01.799 of them. Sure, it's so heavily integrated into studio now, like, 23 00:02:01.840 --> 00:02:06.280 yeah, may just make your life pretty easy. Yeah, so this, 24 00:02:06.719 --> 00:02:10.680 you know, this is required learning. And so even if you know this 25 00:02:10.719 --> 00:02:14.479 stuff and you think you know it, it's good to brush up on it, 26 00:02:14.599 --> 00:02:17.680 especially if you're in charge of, you know, teaching other people on 27 00:02:17.719 --> 00:02:23.120 your team all about it. But anyway, I put a link to it. 28 00:02:23.120 --> 00:02:25.919 It's since this is episode eighteen eighty seven. You can go to eighteen 29 00:02:25.960 --> 00:02:30.919 eighty seven dot pop, dot m E and then I'll get you there. 30 00:02:31.400 --> 00:02:38.280 It's had what twenty five hundred views since January fifteenth. Yeah, it's just 31 00:02:38.439 --> 00:02:42.400 it was just put up a little while ago. Yeah, it's very good. 32 00:02:42.439 --> 00:02:45.120 It's cool and a lot of thumbs up and I watched it and it's 33 00:02:45.159 --> 00:02:47.360 good. No learning love it. Who's talking to us today, Richard, 34 00:02:47.719 --> 00:02:52.000 I grabbed a calm on top of show eighteen sixty nine, which is from 35 00:02:52.080 --> 00:02:55.560 last fall October twenty twenty three when we were at n DC Porto an on 36 00:02:55.719 --> 00:03:01.840 stage interview with one Charity Majors from company called the honey Comb that we put 37 00:03:01.800 --> 00:03:06.960 a terrible pressure on Martin or anything, but she was awesome. I don't 38 00:03:06.960 --> 00:03:09.879 know if Martin is gonna have and and see we're gonna have as many F 39 00:03:09.960 --> 00:03:14.520 bombs however, No, but that's not about it either. And just to 40 00:03:14.599 --> 00:03:19.000 reinforce the point, and g Houseworth had this comment which he said, I 41 00:03:19.080 --> 00:03:22.759 never missed an episode of dot Net Ross because they absolutely rock, and this 42 00:03:22.800 --> 00:03:27.520 episode with Charity Majors was none, if not the best. Yeah, no 43 00:03:27.560 --> 00:03:30.360 pressure, the insight and energy was awesome, and we did have the distinct 44 00:03:30.360 --> 00:03:35.319 advantage of being on stage with her. Two you're at NDCA Porto at the 45 00:03:35.360 --> 00:03:38.719 time. But you know, we're talking about observability more and more, and 46 00:03:38.759 --> 00:03:42.639 I'm really looking forward to getting into this one because we've got a couple of 47 00:03:42.680 --> 00:03:45.199 different viewpoints coming at it. But if you want to get started on this 48 00:03:45.280 --> 00:03:47.919 subject, that show we did with Charity no fool and it was great. 49 00:03:49.080 --> 00:03:52.319 It was great. G thanks so much for your comment and a copy of 50 00:03:52.360 --> 00:03:53.280 music Code by. It's on its way to unit. If you'd like a 51 00:03:53.319 --> 00:03:57.080 copy of music Code buy. I write a comment on the website at dot 52 00:03:57.080 --> 00:04:00.080 at Rocks dot com or on the Facebook to publish every show there. And 53 00:04:00.080 --> 00:04:01.080 if you comment there and I reading on the show, was that your copy 54 00:04:01.120 --> 00:04:04.199 of music go by? And uh, you can follow us on Twitter if 55 00:04:04.199 --> 00:04:08.039 you want. But the cool kids are at masadon and you know, I 56 00:04:08.120 --> 00:04:12.759 put all of my social media links at Carl Franklin dot com cool so we 57 00:04:12.800 --> 00:04:15.520 don't have to just rattle them off. I got a bunch of them up 58 00:04:15.560 --> 00:04:20.720 there. I think I've got a Richard dot Campbell dot me, you know, 59 00:04:20.759 --> 00:04:24.480 one of the about me pages that has to catch that stuff up there 60 00:04:24.480 --> 00:04:27.399 too. If I actually looked at it recently, I don't know. I 61 00:04:27.399 --> 00:04:30.199 think I did it when we when we were doing the mass account thing. 62 00:04:30.600 --> 00:04:33.720 Yeah, but yeah, you know it is. But you're at rich Campbell 63 00:04:33.759 --> 00:04:39.360 on Twitter and what's your masdon and pretty and rich Campbell at masadon dot social 64 00:04:39.519 --> 00:04:43.519 you know, yeah, yeah, oh no, it's our Campbell dot me. 65 00:04:43.720 --> 00:04:46.680 There you go. R Campbell dot me. All right, got them 66 00:04:46.680 --> 00:04:51.120 all. Well I'm going to just start talking about that. Carl Franklin dot 67 00:04:51.120 --> 00:04:54.120 com. Just go there. You want to do that now, Okay? 68 00:04:54.279 --> 00:04:58.839 Like yeah, okay, shorter, Yeah it is shorter, and uh you 69 00:04:58.879 --> 00:05:01.360 know. Then then everybody has a lot more to read that they didn't want 70 00:05:01.399 --> 00:05:10.800 to so too long, don't read? Yeah, go away. So let's 71 00:05:10.800 --> 00:05:15.160 bring on our guests, Steve Gordon and Martin Thwaits. Steve is a plural 72 00:05:15.199 --> 00:05:20.079 site author, Microsoft MVP and engineer at Elastic maintaining their dot net libraries. 73 00:05:20.120 --> 00:05:25.680 Steve enjoys sharing his knowledge by presenting talks at user groups and conferences, writing 74 00:05:25.720 --> 00:05:30.759 on his blog at Stevejgordon dot co dot uk, and creating videos. You 75 00:05:30.759 --> 00:05:33.720 can find him on most social media platforms under the username at Steve J. 76 00:05:33.879 --> 00:05:42.600 Gordon. Martin Thwaits is an observability evangelist, dot net developer, Microsoft MVP. 77 00:05:43.120 --> 00:05:47.040 He works for Honeycomb dot Io, an observability vendor. As a developer 78 00:05:47.120 --> 00:05:53.879 advocate, he travels the world spreading the good word about open telemetry and observability 79 00:05:53.920 --> 00:05:59.079 engineering. He'll talk to anyone about observability, even the likes of us Jamos 80 00:06:01.600 --> 00:06:08.560 that's not very nice. So welcome guys. Hey, thanks having us, 81 00:06:08.879 --> 00:06:13.959 Yeah, thanks for having me. So this culminated at in the fish Powl 82 00:06:14.000 --> 00:06:15.560 at NDC London, right, Yeah, we were all hanging out, you 83 00:06:15.600 --> 00:06:23.639 know. I was taking it easy because a co host wasn't there, so 84 00:06:23.639 --> 00:06:29.600 so at some point somebody just declared it Richard's office. There was a few 85 00:06:29.600 --> 00:06:31.920 other foot I think Miss Manners came in and recorded some shows like there's a 86 00:06:31.920 --> 00:06:34.240 few folks came and went and used it. I did some run as as 87 00:06:34.240 --> 00:06:36.720 well. I was there. Yeah, you could. Mostly we sat around. 88 00:06:36.879 --> 00:06:41.279 I do believe a couple of bottles of whiskey may have just disappeared into 89 00:06:41.279 --> 00:06:46.120 that room and never we're seen again, as they do, as they do. 90 00:06:46.560 --> 00:06:48.439 Well, Steve was in the UK and it's afternoon there or evening, 91 00:06:48.560 --> 00:06:51.839 right, so you've got your whiskey there, Steve. That's right. Yeah, 92 00:06:53.000 --> 00:06:57.319 I'm I'm declaring it post six pm. Fine time to drink, right, 93 00:06:57.600 --> 00:07:02.560 I am going tonight to Texas Day Brazil with Robert Ramsey. Nice Richard, 94 00:07:02.800 --> 00:07:08.199 Yeah, and we will I'm sure have plenty of meat and whiskey. 95 00:07:09.160 --> 00:07:13.120 So anyway, what do you guys think of following up with on Charity's Talk 96 00:07:13.160 --> 00:07:20.079 here. Are you scared or are you up to the challenge? Martin said 97 00:07:20.079 --> 00:07:23.879 that very well, don't you get it? Very well? Answered that question. 98 00:07:24.600 --> 00:07:28.879 I mean, F bombs is kind of our company policy. No kidding, 99 00:07:28.920 --> 00:07:30.959 you know, I'm not saying he's part of the interview process, but 100 00:07:31.000 --> 00:07:34.439 I'm not not saying that either. That is so cool. I mean, 101 00:07:34.480 --> 00:07:36.439 you know, let's think about it for a second. I mean, who 102 00:07:36.439 --> 00:07:40.399 are you going to offend? Right? You go to any movie, even 103 00:07:40.439 --> 00:07:44.600 if it's PG, and there's F bombs everywhere, right everywhere. But I 104 00:07:44.600 --> 00:07:47.519 mean we do get some email from folks on set when they show up. 105 00:07:47.519 --> 00:07:51.600 We do try to bleep them too, Yes we do. But I appreciate 106 00:07:51.639 --> 00:07:57.680 the sentiment either way. It's like there's so much of observer your ability that 107 00:07:57.720 --> 00:08:01.439 falls into the what the heck is going on in this app or in this 108 00:08:01.519 --> 00:08:03.839 case, what the f is going on? And it's you know, trying 109 00:08:03.879 --> 00:08:09.279 to be Canadian here. Yeah, we have a we have a you've heard 110 00:08:09.279 --> 00:08:13.920 of MTTR the Meantime to Recovery, and so we have a thing called mtt 111 00:08:13.959 --> 00:08:18.720 WTF, which when you're debugging systems, it's how long does it take you 112 00:08:18.800 --> 00:08:24.040 to go what there was that No, that thing doesn't call that thing, 113 00:08:24.120 --> 00:08:28.399 that doesn't happen. That's not right at all. So what I haven't used 114 00:08:28.439 --> 00:08:37.840 this Honeycomb, But what I remember distinctly about that conversation was it's architecturally different 115 00:08:37.960 --> 00:08:43.080 from a lot of other, you know, well known telemetry kind of things 116 00:08:43.080 --> 00:08:48.159 that you hook on or add on side cars and to get any kind of 117 00:08:48.240 --> 00:08:52.840 insight and observability into what's going on in your application places where you just can't 118 00:08:52.879 --> 00:08:56.000 put a break point. Yeah, I mean, I think this is the 119 00:08:56.080 --> 00:09:01.559 whole tracing debate versus blogs and APM type stuff, where tracing is the thing 120 00:09:01.600 --> 00:09:05.399 that's different. It's the evolution, if you like, of that idea of 121 00:09:05.519 --> 00:09:09.080 just log everything, throw it in a big bucket, try and run some 122 00:09:09.480 --> 00:09:13.279 log stash things on the back of it, or fluent stuff to build it 123 00:09:13.320 --> 00:09:16.679 all and try and get a picture. It's really the tracing stuff that is 124 00:09:16.720 --> 00:09:22.279 the sort of next evolution that isn't is not new, and that's the thing 125 00:09:22.279 --> 00:09:24.679 that really gets me. I was going to say, we've had tracing for 126 00:09:24.720 --> 00:09:28.240 a long time, but it's new about honeycomb tracing. So well, Honeycomber 127 00:09:28.320 --> 00:09:33.519 is just a back end, we just take spans and allow you to query 128 00:09:33.519 --> 00:09:37.919 them in weird and wonderful ways. In rapid fire, you know the idea 129 00:09:37.960 --> 00:09:43.360 of an outage going on and you're waiting five minutes for telemetry data to get 130 00:09:43.399 --> 00:09:48.039 through for you to ask a question, to say, did I fix it? 131 00:09:48.080 --> 00:09:52.080 Is it now fixed? That idea of just rapid fire, ask questions, 132 00:09:52.120 --> 00:09:54.000 Ask questions, because that's really what you want to do. You're on 133 00:09:54.080 --> 00:09:58.840 that conversation with your production system that's like, you know what's happening? Who 134 00:09:58.919 --> 00:10:03.440 hurt you? Yeah? And that's really what we're trying to show me where 135 00:10:03.480 --> 00:10:07.600 the bad man? Yeah, it couldn't help, but notice your pain. 136 00:10:09.519 --> 00:10:13.840 But that's what we try to do is really try to allow people to just 137 00:10:13.919 --> 00:10:20.360 ask weird and wonderful questions. The example we users is, you know what 138 00:10:20.519 --> 00:10:26.639 happens when the French user is currently in Norway using the French language pack in 139 00:10:26.679 --> 00:10:31.679 the Norway region on iOS fourteen point one, and that's the era that's happening. 140 00:10:31.679 --> 00:10:35.919 It's only on those users, because that's that's where we are with systems 141 00:10:35.960 --> 00:10:39.480 now. There's no longer this system that you use inside of your organization that 142 00:10:39.559 --> 00:10:45.200 only your users are only your organizational users. Everything's global. You know, 143 00:10:45.279 --> 00:10:48.799 people are working from home, working from anywhere. So even if you've got 144 00:10:48.840 --> 00:10:52.200 just an internal system, these people are from everywhere now, so you know, 145 00:10:52.240 --> 00:10:54.840 we're starting to get into a world where things have changed. It's not 146 00:10:56.480 --> 00:11:00.519 about I've got six month release cycle. I can wait months. I can 147 00:11:00.600 --> 00:11:05.759 do a QA cycle. And my my whole thing at the moment is if 148 00:11:05.759 --> 00:11:09.039 we don't change the tools, and I'm not talking about the vendor back ends. 149 00:11:09.080 --> 00:11:13.480 I'm talking about like open telemetry and tracing and profiles that are coming up 150 00:11:13.480 --> 00:11:16.879 and various other signals that are coming up, we don't give people better tools, 151 00:11:16.240 --> 00:11:20.120 but we're saying you need to do things better than you did before. 152 00:11:20.720 --> 00:11:24.799 Like it's not it's not okay, yeah, yeah, the the yeah, 153 00:11:24.919 --> 00:11:30.399 the reliability strategy of be more careful next time. Yeah, don't do that. 154 00:11:31.240 --> 00:11:35.000 Yeah, don't touch that. It's yeah. The real point here is 155 00:11:35.320 --> 00:11:39.200 the tools should lead us to the path of success. They should resist. 156 00:11:39.279 --> 00:11:43.480 We resist as are making mistakes, and so we get to still be human 157 00:11:43.639 --> 00:11:48.919 and have reliable software. What a concept. Yeah, And I mean that 158 00:11:48.039 --> 00:11:52.519 was I think from my experience that was one of the problems with just sort 159 00:11:52.519 --> 00:11:56.320 of relying on logging as your observability. Logging in a few metrics was was 160 00:11:56.399 --> 00:12:01.679 us doing observability. But logging can be quite good if you spend a heck 161 00:12:01.720 --> 00:12:05.879 of a lot of time and I use heck there heck of a lot of 162 00:12:05.919 --> 00:12:09.480 time getting the logging in that you need. It tends to be quite retrospective. 163 00:12:09.919 --> 00:12:13.799 You figure out there's a problem in a particular code path, you put 164 00:12:13.799 --> 00:12:16.240 a new log in, you'd ship to production, You wait for someone else 165 00:12:16.240 --> 00:12:18.240 to hit the error, and it logs something else that gives you another clue 166 00:12:18.240 --> 00:12:22.440 of where you want to log next, right, And so it's it's very 167 00:12:22.440 --> 00:12:26.440 retrospective and looking back. Whereas with traces, if you design them well and 168 00:12:26.480 --> 00:12:31.960 you put tracing around your key points in your application code, you get the 169 00:12:31.000 --> 00:12:35.720 full, you know, waterfall effect of what's happened, and you've added hopefully 170 00:12:35.759 --> 00:12:39.480 decent attributes on there so that you can, as Martin said, go back 171 00:12:39.480 --> 00:12:46.519 and answer any weird and wonderful questions and combinations of events with particular attributes on 172 00:12:46.559 --> 00:12:48.159 them. Awesome, And Steve, I got to ask you, like I 173 00:12:48.240 --> 00:12:52.039 normally resist two guest shows because you get up with a lot of voices and 174 00:12:52.120 --> 00:12:56.360 I but we're talking of observability, and Martin works for the company. Arguably 175 00:12:56.399 --> 00:13:01.279 that makes one of the most popular observeability tools out there. Does Alaska you 176 00:13:01.639 --> 00:13:05.639 does Elastic use Honeycomb or is it just you know, you're trying to understand 177 00:13:05.679 --> 00:13:09.639 what your own product does. What's the story there? So we provide an 178 00:13:09.679 --> 00:13:13.799 application performance monitoring solution as well, so we compete in a nice friendly way. 179 00:13:15.000 --> 00:13:16.360 So we have a different back end. Our back ends built on top 180 00:13:16.399 --> 00:13:22.879 of Elastic Search, which is kind of Elastic's main product, and our APM 181 00:13:22.960 --> 00:13:26.399 server uses that as the data store, so you can ship your telemetry data 182 00:13:28.360 --> 00:13:31.080 to us. So you can use our APM agent to do that, or 183 00:13:31.240 --> 00:13:35.919 today you can also use any open telemetry capable agent as well. So the 184 00:13:37.559 --> 00:13:41.519 sort of built in what we call the Vanilla Open Telemetry SDK can be used 185 00:13:41.559 --> 00:13:48.159 to ship over the open telemetry protocol directly into Elastic APM server and then we'll 186 00:13:48.440 --> 00:13:52.279 stare that for you and then give you the again a different form of you 187 00:13:52.799 --> 00:13:56.679 to search over those traces and those logs. I'm just staggered at the idea 188 00:13:56.679 --> 00:14:01.200 that we're talking about Elastic for something other than search. What I know, 189 00:14:01.639 --> 00:14:03.080 I thought they made other products, but when don't we ever discussed that? 190 00:14:03.200 --> 00:14:07.600 So okay, so this is the elastic has their eight PM product, which 191 00:14:07.639 --> 00:14:11.519 is another aspect of them. But you're both guys are in observability in a 192 00:14:11.559 --> 00:14:15.440 big way. That's right. Yeah, I think that the really I mean, 193 00:14:15.600 --> 00:14:16.919 one of the reasons why I thought it was really good that we're both 194 00:14:16.960 --> 00:14:22.080 here is because different back ends provide you with different query and capabilities. Right. 195 00:14:22.120 --> 00:14:26.799 You know, the idea that you just buy one tool off the shelf 196 00:14:26.840 --> 00:14:30.679 and everything just works is just not going to work in today's world. You 197 00:14:30.720 --> 00:14:37.559 really need crazy you know, you really need to think about what's going to 198 00:14:37.600 --> 00:14:39.440 work for your organization. You know, what are you trying to achieve with 199 00:14:39.480 --> 00:14:43.120 observability, and then choose the back ends that work. And that's where open 200 00:14:43.159 --> 00:14:48.639 celemetry is really awesome because you can send it to two providers. You can 201 00:14:48.679 --> 00:14:52.759 send that same exact data to two separate viders and say, well, this 202 00:14:52.799 --> 00:14:56.600 one's really good at metrics. It's really good at showing me some dashboards about 203 00:14:56.960 --> 00:15:01.759 pre aggregated data. Awesome, that's a really good tool for that thing. 204 00:15:01.759 --> 00:15:07.639 It does infrastructure metrics. There's one that does say costing and takes your metrics 205 00:15:07.679 --> 00:15:11.679 and can project costs based on previous metrics. Those are really cool tools. 206 00:15:11.799 --> 00:15:16.200 There's other ones that focus on, say Servilus, there's other ones that focus 207 00:15:16.279 --> 00:15:18.799 on you know, everything's just they've got their own niches. You know, 208 00:15:20.080 --> 00:15:24.480 Elastic has their own niche. We have our own niche. So open celemetry 209 00:15:24.519 --> 00:15:28.240 just makes it easy as your needs change, you can change provider. There's 210 00:15:28.279 --> 00:15:33.320 no I'm going to change provider. We'll install the new agent on fifteen hundred 211 00:15:33.399 --> 00:15:37.320 machines. You know, well, I'm nobody's going to do that. Or 212 00:15:37.440 --> 00:15:41.799 now it's I'm going to move vender. I'll change three lines of config right 213 00:15:43.120 --> 00:15:46.600 now. Speaking of that, I mean you bring up pain right. I 214 00:15:46.639 --> 00:15:52.840 mean, let's say I've got an app in production and I've got you know, 215 00:15:52.600 --> 00:15:56.240 thousands and maybe one hundreds of thousands of lines of code. How does 216 00:15:56.279 --> 00:16:03.240 one just retrofit this kind of tracing into an app like that? Obviously there 217 00:16:03.279 --> 00:16:07.919 isn't any kind of wizard where you can just snap your fingers and it's instant 218 00:16:08.039 --> 00:16:15.799 insights, right, ten lines of code, ten ten lines of code in 219 00:16:15.879 --> 00:16:19.960 not for every object in every page. And so I mean the thing is 220 00:16:21.000 --> 00:16:22.960 with open celemetry, we've got three care signals, we've got tracing, we've 221 00:16:23.000 --> 00:16:29.200 got metrics, and we've got locks. Tracing is predominantly back end. It's 222 00:16:29.240 --> 00:16:33.240 focused on this idea of we've got a context. So a trace is a 223 00:16:33.279 --> 00:16:36.600 context, a thing that's been done. So it really works best in the 224 00:16:36.639 --> 00:16:41.080 back end because you've got an inbuilt context of a web request. It starts 225 00:16:41.120 --> 00:16:44.679 and it finishes and things happen. It doesn't work well currently in the front 226 00:16:44.759 --> 00:16:48.200 end, so it doesn't work well in the client side. What people call 227 00:16:48.559 --> 00:16:55.759 rum or really use monishing, so it doesn't work well in that way. 228 00:16:55.960 --> 00:17:00.399 Open celemetry is trying to build a signal for that. So in the backkend 229 00:17:00.799 --> 00:17:03.000 it's actually really easy. You just add say ten lines of code to your 230 00:17:03.000 --> 00:17:07.759 startup and say I would like to see a span for every HDTP request that 231 00:17:07.799 --> 00:17:11.240 I get into my site. I would like to see a span for every 232 00:17:11.519 --> 00:17:14.359 HDDP request I make out of my site. And when you say span, 233 00:17:15.160 --> 00:17:18.720 there's several definitions of that word, but in this context, what does it 234 00:17:18.799 --> 00:17:22.920 mean. So we're on a dot net podcast, so let's use dot net 235 00:17:22.000 --> 00:17:27.480 terms. You can create an activity, so activity and this is where we 236 00:17:27.480 --> 00:17:34.119 say we've had this for ages because activity has existed since I think dot Net 237 00:17:34.119 --> 00:17:38.359 called two point one I think was when they first started bringing in activity. 238 00:17:40.279 --> 00:17:45.759 It's been there for a while, and activity kind of represents a sort of 239 00:17:45.839 --> 00:17:49.359 unit of work, if you like, and an actor you create an activity, 240 00:17:49.480 --> 00:17:53.440 you create one for your HDTP inbound request, one for your outbound request, 241 00:17:53.519 --> 00:17:56.640 one for your when you hit a database. And these are all in 242 00:17:56.720 --> 00:18:03.119 built into the libraries already. Espnut core has It's equal client has it, 243 00:18:03.599 --> 00:18:08.160 Myscore client has it, MPGSQL Entity Framework, you name it. All of 244 00:18:08.200 --> 00:18:14.160 those have been moved up. Even the Azure client SDKs have now been changed 245 00:18:14.160 --> 00:18:18.440 over to using activity sources and activities to create these spands. So a span 246 00:18:18.599 --> 00:18:22.319 being this, I've got a start time, and I've got an end time, 247 00:18:22.359 --> 00:18:26.839 and I've got some attributes, and it allows you to build these really 248 00:18:26.960 --> 00:18:30.160 rich waterfalls that allow you to see how things transition through your system. But 249 00:18:30.240 --> 00:18:33.480 you can do that with about ten lines of code in the startup of your 250 00:18:33.480 --> 00:18:38.359 application. And with a spire, they're actually promoting this idea of these shared 251 00:18:38.400 --> 00:18:42.599 projects where all of that setup happens once, so it's not even ten lines 252 00:18:42.640 --> 00:18:47.960 of code per application, it's ten lines of code in a common startup, 253 00:18:48.880 --> 00:18:53.079 So you can get essentially all this automatic data just by adding ten lines of 254 00:18:53.079 --> 00:18:56.000 code and then putting in a little bit config to say where you're going to 255 00:18:56.000 --> 00:18:59.519 send it. You're going to put in the elastic APM end points, you're 256 00:18:59.519 --> 00:19:02.319 going to put in the honeycombman pints, you're going to put it into any 257 00:19:02.319 --> 00:19:03.799 of the end points, and if you want to go further, you can 258 00:19:03.920 --> 00:19:07.799 you can do zero lines and code as well. So there's also this kind 259 00:19:07.799 --> 00:19:12.400 of concept of auto instrumentation where by flicking a few environment variables to configure the 260 00:19:12.519 --> 00:19:18.200 profiling hooks that the dot net run time has built into it, if you 261 00:19:18.240 --> 00:19:22.440 can point it at this auto instrumenting profil and that can go in and wire 262 00:19:22.559 --> 00:19:26.359 up whether it be the open telemetary STK or the elastic APM agent wire out 263 00:19:26.519 --> 00:19:30.759 for you in your application with no code changes at all. Just deploy the 264 00:19:30.839 --> 00:19:34.240 environment variables and restart the app, which can be a really good way to 265 00:19:34.240 --> 00:19:40.119 get started with. That's amazing actually, Yeah, with basic instrumentation of what's 266 00:19:40.160 --> 00:19:44.440 happening. Guys, you're talking about the easy part collecting too much, Like 267 00:19:44.559 --> 00:19:49.000 that's really easy. Yeah, you know, I'm a guy who who spent 268 00:19:49.079 --> 00:19:52.160 time in the sequel world where we used to do full traces of every transaction 269 00:19:52.279 --> 00:19:56.279 going through a SQL server because it was really great at filling up hard drives 270 00:19:56.279 --> 00:20:00.519 a a bit. It's the real question is after you gather all this stuff, 271 00:20:00.799 --> 00:20:03.720 and I emphasis on stuff, I can think of several other words that 272 00:20:03.720 --> 00:20:10.960 would apply begin with s, Yeah, how do you mash this in the 273 00:20:11.039 --> 00:20:14.680 submission? And you know to the line I would use when working with the 274 00:20:14.720 --> 00:20:19.400 team was create actionable items, things we can do to make the system better. 275 00:20:21.279 --> 00:20:23.960 That is the rub really, you know, and that I think is 276 00:20:25.039 --> 00:20:32.359 what has been tracing's downfall for probably the last sort of five years really because 277 00:20:32.720 --> 00:20:37.160 people see tracing as a I have a trace ID and now I can see 278 00:20:37.160 --> 00:20:41.279 a trace waterfall for that thing, but I need something else in order to 279 00:20:41.319 --> 00:20:44.720 tell me what trace ID to use. And what that means is you end 280 00:20:44.799 --> 00:20:48.559 up with say ninety nine point nine nine nine recurring percent of your data not 281 00:20:48.640 --> 00:20:53.480 being used and just sitting on hard drives and costing you money. And that 282 00:20:53.519 --> 00:20:56.079 I think has been one of the big downfalls that people have said, well, 283 00:20:56.240 --> 00:21:00.720 tracing is not really useful because I need to know what trace it was 284 00:21:00.759 --> 00:21:03.519 so I'm just going to use logs because I have to keep them anyway, 285 00:21:04.680 --> 00:21:08.359 And I think that's been one of the big downfalls. But I think there's 286 00:21:08.480 --> 00:21:14.559 there's a movement now around tracing, around things like sampling and doing various different 287 00:21:14.599 --> 00:21:18.079 mechanisms for how we reduce down the amount of data that we store and only 288 00:21:18.079 --> 00:21:22.200 store the interesting stuff. Yeah, that's assessing what's interesting is not a trivial 289 00:21:22.240 --> 00:21:26.880 problem either, right, Yeah, I mean interesting is subjective at best, 290 00:21:27.799 --> 00:21:32.240 you know, you know, but the general rule is, you know, 291 00:21:32.359 --> 00:21:36.079 you keep every one of your slow traces and keep everyone that has an error 292 00:21:36.119 --> 00:21:38.119 in it, because there was at least going to be the interesting ones. 293 00:21:40.440 --> 00:21:44.200 But you do that, and then you can start to think about how do 294 00:21:44.279 --> 00:21:48.240 I query my trace data, not how do I query my logs and my 295 00:21:48.359 --> 00:21:51.839 metrics, and then get back to tracing. How do I start with that 296 00:21:51.920 --> 00:21:56.039 tracing data, because it's really rich. And I think that's where we're seeing 297 00:21:56.079 --> 00:21:59.920 this big change now with open telemetry, because the back ends, all the 298 00:22:00.039 --> 00:22:03.440 back ends now are starting to look at this trace analysis type stuff, whether 299 00:22:03.480 --> 00:22:06.720 it's ours or Elastic or a lot of the open source ones that are normally 300 00:22:06.720 --> 00:22:11.359 based on ClickHouse as a data store. They're starting to do this idea of 301 00:22:11.440 --> 00:22:14.759 well, you know, how do I get and ask questions of my trace 302 00:22:14.839 --> 00:22:17.960 data? Because it's really rich, it's really wide, it's got loads of 303 00:22:18.000 --> 00:22:22.440 attributes, all that interesting stuff, and that really is the rub. Yeah, 304 00:22:22.559 --> 00:22:26.240 so always a question of what's actually interesting, because you're always afraid of 305 00:22:26.240 --> 00:22:30.039 what you shave off because that might have been the interesting data. But yeah, 306 00:22:30.039 --> 00:22:33.720 we've had so many people who come to us with, yeah, I 307 00:22:33.759 --> 00:22:37.000 don't really like tracing because you know, every time I wanted to ask a 308 00:22:37.079 --> 00:22:41.640 question, the customer got an error. It was sampled out because or discarded 309 00:22:41.680 --> 00:22:47.079 because somebody took one percent of random sample data. And the era happens, 310 00:22:47.119 --> 00:22:49.119 you know, once in a million times, so they never get the trace 311 00:22:49.119 --> 00:22:53.039 that they want. And that's where we're doing a lot of evolution right now 312 00:22:53.079 --> 00:22:56.400 in the open telemetric collector and some of the stuff that even me and Steve 313 00:22:56.400 --> 00:23:02.279 were talking about at NBC London about how we roll up stands and write tell 314 00:23:02.359 --> 00:23:06.079 something processes that are more bespoke and you know, this is the sort of 315 00:23:06.079 --> 00:23:10.759 conversations we're having now because it's vendor agnostic and so you've got so many more 316 00:23:11.240 --> 00:23:12.839 people. You know, me and Steve, we don't work for the same 317 00:23:12.880 --> 00:23:18.759 company, but we're working towards a common goal of reducing the amount of data, 318 00:23:18.799 --> 00:23:21.920 making it more actionable, and making it easier for people to implement. 319 00:23:22.319 --> 00:23:26.200 So you've got, you know, hundreds of developers all working on that same 320 00:23:26.240 --> 00:23:30.039 code base. And that's the major advantage of open telemetry as a goal, 321 00:23:30.079 --> 00:23:33.559 isn't it for the industry that rather than every vendor focusing on their way of 322 00:23:33.599 --> 00:23:37.640 doing something and having their own little pieces that they can use in sales and 323 00:23:37.680 --> 00:23:42.559 marketing, it's actually what's the right solution across the industry for how this should 324 00:23:42.559 --> 00:23:47.519 be done in the best way for the organized relations that need this information to 325 00:23:47.559 --> 00:23:51.839 be able to store this stuff in a sensible, cost effective kind of way 326 00:23:51.920 --> 00:23:56.440 to do this level of sort of advanced sampling, sort of intelligent sampling, 327 00:23:56.440 --> 00:24:00.640 if you will, on the on the tail of things coming through. And 328 00:24:00.680 --> 00:24:03.079 I think we're starting to see that shift with vendors now kind of all starting 329 00:24:03.079 --> 00:24:07.920 to adopt finally, I think open telemetry as that solution. Yeah, it's 330 00:24:07.960 --> 00:24:11.519 always a question of mentioning this data together into like do you make sense of 331 00:24:11.519 --> 00:24:15.160 it? They and what's that you see mostly it's the same stuff over and 332 00:24:15.200 --> 00:24:18.359 over again. It is what you already know. I'm tapping on the book 333 00:24:18.359 --> 00:24:22.440 I've never freaking finished and still hope to one day. With the history of 334 00:24:22.480 --> 00:24:30.640 dot net, where the seql team had a tool where anytime a unique execution 335 00:24:30.839 --> 00:24:33.200 path happened inside its equal server, it would stop. It would hit a 336 00:24:33.200 --> 00:24:37.839 breakpoint, so as long as it was executing the normal path or normal paths 337 00:24:38.000 --> 00:24:41.279 over time, it would continue running. And so it was one It was 338 00:24:41.359 --> 00:24:45.599 like a tool designed to find the one in one hundred million case. So 339 00:24:45.640 --> 00:24:49.720 you're iterating rapidly and it's like initially it's breaking on each unique it's breaking on 340 00:24:49.759 --> 00:24:52.720 each unique path, but after you say, okay, that's a fine path, 341 00:24:52.839 --> 00:24:56.559 keep going. Then eventually it's like running for days before it's finally like 342 00:24:56.759 --> 00:25:00.240 think how about this? You're like, now, well the heck did you 343 00:25:00.279 --> 00:25:04.880 get there? And the reasons in the story of the context of dot net 344 00:25:04.920 --> 00:25:10.160 is because that's what made dot net two point zero. Making dot Net run 345 00:25:10.200 --> 00:25:15.559 inside a sequel server, made them test dot net in away never been tested 346 00:25:15.599 --> 00:25:19.119 before, and they found rare and unusual bugs because they were running into context 347 00:25:19.160 --> 00:25:23.759 of sequel server. Here end of the history lesson. I love the idea 348 00:25:23.880 --> 00:25:29.359 of like the system just going hold on a minute, can you just get 349 00:25:29.359 --> 00:25:37.279 your supervise it this road before boys, let's have a little chat. And 350 00:25:37.720 --> 00:25:41.039 speaking of that, my friends, let's take a brief break for this very 351 00:25:41.079 --> 00:25:48.519 important message, and we're back. It's dot net rocks. I'm Richard Campbell, 352 00:25:48.680 --> 00:25:52.200 it's Martin Waite's and Steve Gordon and my buddy Carl Hey and talking a 353 00:25:52.240 --> 00:25:56.559 little bit about more than one way to get observability here and the fact that 354 00:25:56.799 --> 00:25:59.759 you know, now you're really creating trouble for me. It's not like I 355 00:25:59.799 --> 00:26:02.880 was get too much telemetry out of my Honeycomb stack, but now I'm gonna 356 00:26:02.920 --> 00:26:06.599 go get elastic eight PM as well, feed the same data to it twice 357 00:26:06.880 --> 00:26:10.799 because I just own too many drives, Like let's try and kill them all. 358 00:26:11.319 --> 00:26:17.160 Like doing analysis between those data sets could be interesting too. I mean, 359 00:26:17.160 --> 00:26:19.319 we do that law, you know, the Bakoff type stuff. You 360 00:26:19.359 --> 00:26:22.319 know, once you've got an open celemetry collector in place, you just sort 361 00:26:22.319 --> 00:26:30.519 of send it to five vendors, see which ones do the best fit out 362 00:26:30.519 --> 00:26:33.559 what you want to hear. You know, in accounting they call it reconciliation, 363 00:26:33.079 --> 00:26:36.599 right, Like you got to add up the numbers more than one way. 364 00:26:37.759 --> 00:26:41.599 I wonder we're not there yet in our industry. Like if I really 365 00:26:41.960 --> 00:26:48.799 wanted to be a pain in the butt about a performance contract, the fact 366 00:26:48.839 --> 00:26:52.559 that I could ask her to the telemetry to be sent to two different sources 367 00:26:52.599 --> 00:26:56.039 and we'll reconcile the performance metrics against it from two different places and see if 368 00:26:56.079 --> 00:26:59.680 they actually agree. When they don't, wonder why, Like, yeah, 369 00:26:59.759 --> 00:27:04.000 you really press against that contract. Yeah, I haven't been in there be 370 00:27:04.119 --> 00:27:12.559 CTO in too long. You've given me new ideas on behalf of myself, 371 00:27:12.599 --> 00:27:22.480 and I am sorry to the industry. Yeah, I mean it's all Yeah. 372 00:27:22.480 --> 00:27:25.839 The other side of this is like nowurkicking, we're doing multiple analysis on 373 00:27:25.880 --> 00:27:29.920 the same sets of data. Hopefully obviously different tools, like you said, 374 00:27:30.000 --> 00:27:33.440 have better insights in different areas. But yeah, I'd love to have a 375 00:27:33.480 --> 00:27:37.400 standard set of comparisons. Says oh yeah, no, we reconcile the same 376 00:27:37.440 --> 00:27:41.599 way we agree on stuff, so you get more confidence. There's also different 377 00:27:41.680 --> 00:27:45.640 users as well. You know, we've got the you know, just in 378 00:27:45.680 --> 00:27:49.319 the sort of reliability space, you've got developers who are using this information You've 379 00:27:49.359 --> 00:27:55.240 got srs that are using this information to optimize the infrastructure. You've got platform 380 00:27:55.279 --> 00:27:57.839 engineers that are using it for the same things to build tools. You know, 381 00:27:57.880 --> 00:28:02.359 that's just three users just in engineering. You know, you start to 382 00:28:02.359 --> 00:28:04.960 think that what else could we use this information for. I know of a 383 00:28:06.000 --> 00:28:08.960 company that's running at the moment the idea of taking metrics to you to do 384 00:28:10.039 --> 00:28:14.119 cost projections. Right, So take the metrics of your kuminettes poor to how 385 00:28:14.200 --> 00:28:17.240 much you might need to scale in the future. Take that metrics data. 386 00:28:17.640 --> 00:28:21.480 Don't allow people to quer it and do dashboards, but take it and do 387 00:28:21.559 --> 00:28:26.200 projections. You know, this is now into the thinops world. Well, 388 00:28:26.200 --> 00:28:30.279 this is my mid life, right, Like all too often we'd roll out 389 00:28:30.319 --> 00:28:33.559 a new feature and it would tip the system over a week later because we 390 00:28:33.559 --> 00:28:37.519 weren't looking at the additional resource impacts. And as we you know, as 391 00:28:37.519 --> 00:28:42.079 e commerce site started getting really big, we were dark launching features and running 392 00:28:42.119 --> 00:28:45.240 them on one server just to see what the load differences were. So I 393 00:28:45.319 --> 00:28:48.119 knew how much hardware to order, which just speaks to how old I am, 394 00:28:48.200 --> 00:28:51.400 right, because back then it was like, see, we love that 395 00:28:51.440 --> 00:28:53.839 feature, but we literally need thirty more servers to roll it out. Yeah, 396 00:28:53.880 --> 00:28:56.559 and then you end up with one server running on a previous version and 397 00:28:56.559 --> 00:29:03.799 then you bankrupt the company, right yeah, well yeah, we don't scare 398 00:29:03.839 --> 00:29:08.240 me Martin like dinged. But I have used that line in the boardroom where 399 00:29:08.240 --> 00:29:11.759 it's like, I think we're about a week away from bankruptcy unless we turn 400 00:29:11.839 --> 00:29:14.799 this thing off, because we're gonna we're gonna crush the whole thing, Like 401 00:29:14.839 --> 00:29:19.200 we have to think about it. Yeah, it's uh, it is it 402 00:29:19.200 --> 00:29:25.640 it? And it's that whole part about the telemetry part was the it's very 403 00:29:25.680 --> 00:29:29.000 hard to estimate the impact of a feature. Not only you don't really understand 404 00:29:29.000 --> 00:29:32.240 the resource consumption of a feature anyway, but you certainly don't know what the 405 00:29:32.400 --> 00:29:36.200 user utilization is going to be, like you're just making this stuff up, 406 00:29:36.359 --> 00:29:40.759 right, So it is only telemetry that really tells us how much is it 407 00:29:40.799 --> 00:29:44.240 being used? What's the actual impact on the systems? At least now we're 408 00:29:44.279 --> 00:29:47.640 living in a cloud where you can just flip a knob and the CFO find 409 00:29:47.640 --> 00:29:51.160 out about it in about thirty days, right, Like, you not your 410 00:29:51.240 --> 00:29:52.920 problem. You kept the system up. That's my metric. I'm up, 411 00:29:53.039 --> 00:29:59.119 Like I'm good. Well, what is this about a bill? And this 412 00:29:59.200 --> 00:30:02.160 is the the other flip side of that is what we call the nines don't 413 00:30:02.160 --> 00:30:04.359 matter if uses unhappy, it's like, yeah, ninety nine time ninet nine 414 00:30:04.400 --> 00:30:11.160 percent O time awesome, twenty second response times though, that's a good yeah, 415 00:30:11.200 --> 00:30:14.279 but yeah, this is the game we're playing, right is looking through 416 00:30:14.279 --> 00:30:18.759 all of these things. And really we came into this conversation talking about what's 417 00:30:18.799 --> 00:30:22.960 the app up to, like we're not even you know here we're talking about 418 00:30:22.000 --> 00:30:26.799 portant business cases, but mostly it's just trying to diagnose weird behavior, like 419 00:30:27.119 --> 00:30:30.599 I don't know what's going on, and I have a customer that can totally 420 00:30:30.680 --> 00:30:37.319 use this Right now, after upgrading to blazer dot Net eight from six, 421 00:30:38.559 --> 00:30:45.400 all sorts of weird things are happening the and we're still using the same services 422 00:30:45.400 --> 00:30:49.279 and stuff, just the newer versions. So it's probably just some versioning thing, 423 00:30:49.440 --> 00:30:59.359 but weird unhandled exceptions that are evading the error handlers and closing the whole 424 00:30:59.400 --> 00:31:03.519 thing and not giving us an opportunity to find out where they are. Like 425 00:31:03.920 --> 00:31:07.680 this this is going to be a game changer for them. So what I 426 00:31:07.680 --> 00:31:11.079 should mention at that point is it doesn't work well in Blazer. I'm really 427 00:31:11.119 --> 00:31:21.440 sorry Stack or is there something inherent to Blazer. So it's more to do 428 00:31:21.519 --> 00:31:25.039 with the front end side. So if you're using Blazer web assembly, you 429 00:31:25.079 --> 00:31:30.720 have a single thread and telemetry should inherently be done in the background. So 430 00:31:30.680 --> 00:31:37.759 the Blazer server has some problems with the gRPC connections and various other bits that 431 00:31:38.519 --> 00:31:42.960 we're working on it. I'm currently talking to somebody who runs one of the 432 00:31:44.000 --> 00:31:49.519 Blazer University things who's trying all this stuff out and trying to to work out 433 00:31:49.519 --> 00:31:53.359 what what that should look like for Blazer. But there's a lot of things 434 00:31:53.359 --> 00:31:57.640 that are just they're new, they're not they're not in the tracing paradigm right 435 00:31:57.680 --> 00:32:02.799 now. Durable functions is another one that has some really big interesting nuances around 436 00:32:04.119 --> 00:32:07.599 how do we propagate a context, So how do we say that this function 437 00:32:07.720 --> 00:32:12.279 execution was related to this other function execution, and how do we get that 438 00:32:12.359 --> 00:32:15.319 data between the two because it's not as easy as just past some HTTP headers. 439 00:32:16.720 --> 00:32:21.480 So you know, we've got as your service bus has similar nuances. 440 00:32:22.640 --> 00:32:27.599 But we're seeing we're seeing right now a lot of open source libraries that are 441 00:32:27.599 --> 00:32:31.319 going I need to add tracing because people want this, They want that internal 442 00:32:31.359 --> 00:32:37.319 information. They want to know how the cash server is working. They want 443 00:32:37.319 --> 00:32:40.240 to know the messaging systems and how long it takes me to post a message 444 00:32:40.240 --> 00:32:44.160 to my messaging system, not just how long it's going to take to receive 445 00:32:44.200 --> 00:32:47.400 and process it afterwards. So they're adding it. Rabbit MQ recently in the 446 00:32:47.400 --> 00:32:52.039 new version is going to have tracing as a first class citizen in there, 447 00:32:52.079 --> 00:32:54.720 and then it's going to have metrics as well. Fusion Cash was another one 448 00:32:55.160 --> 00:32:59.440 that I worked on with some people, and we've just seen loads more people 449 00:32:59.480 --> 00:33:02.279 going, yeah, tracing, I need the tracing now. So it's just 450 00:33:02.359 --> 00:33:06.079 it's really good to see, right. Yeah, we're at that kind of 451 00:33:06.079 --> 00:33:08.599 tipping point of adoption, which is yeah, very encouraging. I think as 452 00:33:08.680 --> 00:33:13.240 people get benefit out of the box, when all of these libraries they're using 453 00:33:13.400 --> 00:33:16.680 and they're already getting useful traces that tell them a lot of stuff even without 454 00:33:16.680 --> 00:33:21.559 them instrumenting their own code, that's the benefit of everyone. I mean, 455 00:33:21.599 --> 00:33:25.799 we recently instrumented the client library for the language client for Elastic Search, so 456 00:33:25.839 --> 00:33:30.319 that's a separate package you pull in if you want to do you know, 457 00:33:30.480 --> 00:33:34.359 dot net code to talk to elastic search. So we've added some instrumentation down 458 00:33:34.519 --> 00:33:37.119 in the transport layer so you can kind of see what's going on there. 459 00:33:37.599 --> 00:33:39.720 Added our own useful attribute, so out of the box, if you're using 460 00:33:39.720 --> 00:33:46.200 that library and you're using an open telemetry capable you know, collection process, 461 00:33:46.279 --> 00:33:50.960 then you immediately get that information and you can you can diagnose what's going on 462 00:33:51.000 --> 00:33:53.559 at that level. Yeah, and Aspire, you know, you had David 463 00:33:53.599 --> 00:33:58.680 on talking about Aspire. I think that is one of going to be the 464 00:33:58.720 --> 00:34:01.599 real big game changer in a option because one of the big blockers is people 465 00:34:01.680 --> 00:34:05.279 like logs, Well, I can see them in my console when I'm running, 466 00:34:05.599 --> 00:34:07.599 why would I use traces because I can't see them in my console? 467 00:34:08.320 --> 00:34:13.079 But now with Aspire, it's like you've got a nice, convenient little dashboard 468 00:34:13.079 --> 00:34:15.880 that's just running there that takes your metrics and your logs and your traces. 469 00:34:15.440 --> 00:34:19.039 And I'm seeing loads of people use that. They look at the logs thing 470 00:34:19.320 --> 00:34:21.519 and then look at the traces thing and say, well, the traces are 471 00:34:21.519 --> 00:34:24.559 way richer than the logs, So why would I use the logs? And 472 00:34:24.559 --> 00:34:29.679 I'm like, I've been saying this for five years. Thank you you're right. 473 00:34:29.760 --> 00:34:31.079 You know. The one thing I came away with while I came up 474 00:34:31.119 --> 00:34:35.480 with many things from David's show, but it was this, we're trying to 475 00:34:35.559 --> 00:34:39.079 make all the right defaults for you when you're building a cloud native app, 476 00:34:39.119 --> 00:34:43.559 and one of them is you use open telemetry, like, you use that 477 00:34:43.679 --> 00:34:47.519 level of instrumentation. So you know, it's exactly that you have to fight 478 00:34:47.599 --> 00:34:52.360 against it now to not have all that stuff in place. Yeah, it's 479 00:34:52.400 --> 00:34:53.880 the pit of success in it. You know, Yeah, it's right. 480 00:34:53.920 --> 00:34:57.480 But you know, it's an interesting point because there's also folks that are like, 481 00:34:57.519 --> 00:35:00.000 I don't know why we need this, and it's like, yeah, 482 00:35:00.360 --> 00:35:05.400 you know, because it's too many choices. You know, if you're perfect, 483 00:35:05.679 --> 00:35:07.920 if you can get everything right every time, you know, if you 484 00:35:08.039 --> 00:35:10.840 never need to be more careful because you're always careful, then fine, you 485 00:35:10.880 --> 00:35:15.280 don't need it. But I'm just not that careful. It's like unit tests. 486 00:35:15.920 --> 00:35:19.920 Spend several days trying to analyze logs in your head, and you know, 487 00:35:19.920 --> 00:35:23.199 correlate everything across any even a good logging system, and you'll you'll soon 488 00:35:23.239 --> 00:35:28.519 realize that actually a trace might have saved you a few sort of head against 489 00:35:28.559 --> 00:35:30.679 all moments. Yeah, just like unit tests. I don't need unit tests 490 00:35:30.719 --> 00:35:34.119 unit tests with people who don't trust that they can write code right for the 491 00:35:34.199 --> 00:35:37.360 first time. Yeah, that's true. I know I can't write code fight 492 00:35:37.440 --> 00:35:43.280 right the first time. A certain quote from a certain Billy Harleist comes to 493 00:35:43.280 --> 00:35:47.719 mind. Yeah, you might be addicted to code. You might be. 494 00:35:47.920 --> 00:35:53.519 Yeah, and you must suck as a code if you need two people writing 495 00:35:53.599 --> 00:35:58.840 ten lines of code for every line of code you right. Yeah, I 496 00:35:58.880 --> 00:36:01.239 mean I even speaking to an extreme case, but it's like, look, 497 00:36:01.320 --> 00:36:05.519 people make mistakes, they get tired, they work too late, they don't 498 00:36:05.599 --> 00:36:08.920 quite understand the problem. Like, all these things are real, and you 499 00:36:08.960 --> 00:36:14.199 look at all the work we're doing lately around shortening the internal cycle, the 500 00:36:14.239 --> 00:36:16.639 impacts of GitHub co pilot, they're all speaking of the same thing, more 501 00:36:17.280 --> 00:36:22.519 correct code and an aspire to me looks like this tool that says, okay, 502 00:36:22.559 --> 00:36:27.239 so you're going to be cloud native and you haven't done it for thirty 503 00:36:27.320 --> 00:36:30.440 years for some strange reason. So now we're going to give you a tool 504 00:36:30.480 --> 00:36:34.559 set, you know, a framework that leads you to that path. It's 505 00:36:34.559 --> 00:36:37.519 a distributed debugge. Yes, you know full the cloud that's you know, 506 00:36:38.360 --> 00:36:44.639 I joked in some of my talks over the years about you can't attach a 507 00:36:44.719 --> 00:36:49.079 debugger to production. You know, nobody nobody attaches their visual studio instance to 508 00:36:49.079 --> 00:36:55.440 production. Well sorry, nobody does it more than once. But you know 509 00:36:55.480 --> 00:37:00.199 nobody does that because well, for a start, it blocks all the threads, 510 00:36:59.719 --> 00:37:02.320 and that's you know, probably a bad thing. But also, now 511 00:37:02.360 --> 00:37:07.360 we've got you have to attach a debugger to fifteen different services because you've got 512 00:37:07.400 --> 00:37:10.639 five different services running three replicas and you don't know which one is going to 513 00:37:10.719 --> 00:37:15.639 hit. And you know, we're in this world now where we need that 514 00:37:15.679 --> 00:37:17.719 debugging experience. Nothing's changed. You know, we still need to debug our 515 00:37:17.800 --> 00:37:22.519 systems, but we need to debug them in an environment where they're replicated and 516 00:37:22.599 --> 00:37:27.800 they're distributed. Some of them are using the event driven systems, and you're 517 00:37:27.840 --> 00:37:31.320 like, I can't, I can't correlate these logs across an hour's worth of 518 00:37:31.360 --> 00:37:36.920 time window. And yeah, it's just it's hard. And that's where I 519 00:37:36.960 --> 00:37:40.679 think people are getting to now that they're starting to add tools and tools and 520 00:37:40.679 --> 00:37:45.519 tools on top of the old logging systems that they had to try and meet 521 00:37:45.599 --> 00:37:50.480 the demands of these distributed systems. And then they realize that actually know there's 522 00:37:50.519 --> 00:37:54.599 something that's more native that they can use instead, and it requires a paradigm 523 00:37:54.599 --> 00:37:59.960 shift. And I think that's where as an industry we're bad at acknowledge it. 524 00:38:00.119 --> 00:38:06.039 The telemetry and logging aren't the same thing. They because they're not weirdly 525 00:38:06.159 --> 00:38:08.480 enough. But you know, I've been saying for a while now that if 526 00:38:08.519 --> 00:38:13.360 you can answer all the questions that you need from the logs that you've got, 527 00:38:13.800 --> 00:38:16.679 you don't need traces. Yes, stay still fine. Yeah, but 528 00:38:16.880 --> 00:38:22.360 the question is can you and I don't think you can or you're not asking 529 00:38:22.400 --> 00:38:34.360 good questions you Yeah, I mean that's that's the number of rises above none. 530 00:38:35.519 --> 00:38:37.280 That's an old scalability problem where I found that. I found an app 531 00:38:37.280 --> 00:38:39.400 whor it's like it's a long one person was used as it was fine, 532 00:38:39.400 --> 00:38:44.599 but as soon as two did not, it's so good. That was a 533 00:38:44.679 --> 00:38:49.280 long time ago, My goodness. M I mean, I've hit that problem 534 00:38:49.360 --> 00:38:52.960 so many times when people use singletons instead of transients, and you know, 535 00:38:52.039 --> 00:38:57.679 they still stay in a singleton, and especially in multi tenant systems, and 536 00:38:57.760 --> 00:39:00.159 it's like, yeah, when we've got one tenant, it's fine, but 537 00:39:00.199 --> 00:39:02.000 as soon as we get two tenants. Everybody's got everybody else's data, and 538 00:39:02.000 --> 00:39:07.280 that's a bad thing, I think. Holy man, Yeah, easy mistakes 539 00:39:07.280 --> 00:39:12.079 to make too, right, Like, you realize we don't think about that 540 00:39:12.119 --> 00:39:15.679 anymore, that the framework is just doing that for us now most of the 541 00:39:15.719 --> 00:39:19.039 time, as long as you follow the rules. Like, we're definitely working 542 00:39:19.079 --> 00:39:22.639 on a different tier of problems now as we get you We're used to distributing 543 00:39:22.679 --> 00:39:25.880 across multiple machines and now it's you know, in different clouds, with different 544 00:39:25.920 --> 00:39:31.000 platforms. We definitely build way more complicated software. It's no question that we 545 00:39:31.079 --> 00:39:38.280 need to change the way you measure them. Uh. The debugging story is 546 00:39:38.280 --> 00:39:42.559 separate though, right, Like you just sort of casually mentioned the debugging on 547 00:39:42.599 --> 00:39:45.880 top of it. But telemetry is not necessarily for debugging, is it? 548 00:39:46.320 --> 00:39:51.639 So it can be for telemetry isn't a debugging tool, but you can use 549 00:39:51.679 --> 00:39:54.159 it for debugging. You know. One of the big things we talk about 550 00:39:54.159 --> 00:40:00.159 at the moment is the difference between three pillars of observability and just telemetry signals. 551 00:40:00.519 --> 00:40:04.840 So the three telemetry signals being logs, petrics, and traces, they 552 00:40:04.880 --> 00:40:07.280 allow you to do lots of things, and they allow you to build dashboards. 553 00:40:07.599 --> 00:40:14.280 They allow you to do monitoring build alerts, don't. You don't need 554 00:40:14.320 --> 00:40:15.719 one or one or more of these. You can use one, you can 555 00:40:15.840 --> 00:40:21.119 use three. You don't need all of the signals. But when you're starting 556 00:40:21.159 --> 00:40:23.719 to do the debugging side, you need to work out which signals you need 557 00:40:23.760 --> 00:40:28.360 to do it, and you can correlate them. Sometimes one of the key 558 00:40:28.440 --> 00:40:32.079 ones is the user request is taking too long and it turns out it's actually 559 00:40:32.079 --> 00:40:37.199 just one pod and that one pods has its CPU spinning out, and you 560 00:40:37.239 --> 00:40:43.400 look at infrastructure metrics for those things. So debugging or do you call it 561 00:40:43.440 --> 00:40:49.559 debugging working out what the problem is in production the root cause analysis. Yeah, 562 00:40:49.960 --> 00:40:53.639 Debugging to me speaks to I am fixing code often because I'm looking in 563 00:40:53.719 --> 00:40:59.400 an executing environment for what's going on within that code. I could see having 564 00:40:59.400 --> 00:41:02.039 the telemetry sitting beside me because we're seeing a behavior in production, we're not 565 00:41:02.119 --> 00:41:06.519 quite sure what it is, and so now I set up a debugging environment 566 00:41:06.559 --> 00:41:09.119 to try and capture that. And I suppose so I've been playing with this 567 00:41:09.199 --> 00:41:15.800 narrative, and let's try this, which is I think tracing is kind of 568 00:41:15.800 --> 00:41:22.280 like conditional breakpoints that you can use with a time machine. Now, when 569 00:41:22.320 --> 00:41:27.840 I discovered conditional great breakpoints in dotnet, it was like a revelation. When 570 00:41:27.840 --> 00:41:30.039 you're running a SPA app on the front end and it's going through this line 571 00:41:30.039 --> 00:41:32.280 of code, and you want to know when it hits this line of code. 572 00:41:32.519 --> 00:41:35.960 But when it comes from this end point is when I want to be 573 00:41:36.000 --> 00:41:37.320 able to hit it, because it hits sixteen times and I don't want to 574 00:41:37.320 --> 00:41:40.119 have to hit play every time, and I've got multiple threads going on. 575 00:41:42.039 --> 00:41:45.360 If you think about that, but being able to do it in production and 576 00:41:45.400 --> 00:41:52.119 then say what happened yesterday when this person hit this end point for this particular 577 00:41:52.159 --> 00:41:59.039 method or this particular database call that I think is that this revelation of you, 578 00:41:59.079 --> 00:42:02.880 now you are doing deep You're you're using that telemetry from production to do 579 00:42:04.000 --> 00:42:08.119 debugging, yeah, to construct an executing environment to recreate the problem in production. 580 00:42:08.760 --> 00:42:12.199 Like it gives you the cues to say, we need to be in 581 00:42:12.239 --> 00:42:15.599 this state to be able to see this bug. Yeah, to be able 582 00:42:15.639 --> 00:42:19.960 to see those constraints. Because you still can't set a break point in production. 583 00:42:20.880 --> 00:42:27.239 Well you can, you just won't have a job after it. Story 584 00:42:27.280 --> 00:42:30.360 Steve, I stepped down you there yeah, no, I was just yeah, 585 00:42:30.599 --> 00:42:32.039 I'm totally on board with you know, not this point there. And 586 00:42:32.039 --> 00:42:37.280 I think it's it's the ability to apply constraints to the data that you have 587 00:42:37.440 --> 00:42:40.920 that would be very difficult with just pure logs, where you want to set 588 00:42:40.960 --> 00:42:45.400 up this series of criteria around I want to look at this coming from this 589 00:42:45.559 --> 00:42:50.400 in this region, you know, deployed on this particular piece of infrastructure, 590 00:42:50.440 --> 00:42:54.679 and then see a waterfall visually of Okay, what did that flow through in 591 00:42:54.719 --> 00:43:00.760 the code? Where are those you know, lines on this waterfall wider taking 592 00:43:00.800 --> 00:43:06.119 longer? And then you narrow down, Okay, well that's this block of 593 00:43:06.159 --> 00:43:09.000 code or this library. That's where I need to go and look to try 594 00:43:09.039 --> 00:43:13.519 and find out why that's taking so long that piece of the puzzle with these 595 00:43:13.559 --> 00:43:20.840 given sets of criteria applied to it. And that's something I think only only 596 00:43:20.920 --> 00:43:23.159 well defined tracing can can really give you. You could, you could sort 597 00:43:23.159 --> 00:43:27.320 of get there with logs, but you you would take a lot longer to 598 00:43:27.400 --> 00:43:30.280 find there. I think, yeah, there's always a question of do I 599 00:43:30.360 --> 00:43:36.840 understand with any given log set, it's like these six logs, where is 600 00:43:36.920 --> 00:43:39.679 the trends this particular transaction across these logs, like we've always had a tough 601 00:43:39.719 --> 00:43:44.079 time joining that stuff up and tell them she does a better jar with that, 602 00:43:44.639 --> 00:43:47.039 yeah, and then relating that back to infrastructure because you know, we 603 00:43:47.119 --> 00:43:51.599 don't run our code in isolation. Infrastructure is a thing that we use. 604 00:43:52.719 --> 00:43:55.800 So yes, it ran slower. But was it running slow because that Kubernetti's 605 00:43:55.840 --> 00:44:00.599 node was overloaded? Was it running slow because we just switched slots in service? 606 00:44:00.280 --> 00:44:06.679 You know though that infrastructure stuff is just as important. Yeah, it's 607 00:44:06.920 --> 00:44:08.119 you know again, I've spent a long time on the ops side of this 608 00:44:08.159 --> 00:44:12.480 because why is this slow? And it's like it's your hardware, No, 609 00:44:12.599 --> 00:44:15.920 it's your crappy software. Like you go back and forth on that a fair 610 00:44:15.039 --> 00:44:22.079 bit. I have had a situation where we had a hundred based tea hub 611 00:44:22.320 --> 00:44:25.920 in the in the network we'd forgotten about and literally were throttling all the network 612 00:44:25.960 --> 00:44:30.840 and traffic. So you know, it looked like the database was slow, 613 00:44:30.840 --> 00:44:32.480 and it's like, literally it took a long time for the data to come 614 00:44:32.480 --> 00:44:37.920 and go from that because this network mistake was there. Like, those things 615 00:44:37.960 --> 00:44:40.960 happen, and the trick is to not be mean about it. Like everybody 616 00:44:42.000 --> 00:44:45.039 sit on the same side of the table in the boardroom with the projector on 617 00:44:45.519 --> 00:44:49.559 going. What is this data telling us? And that's correlation? Yeah, 618 00:44:49.599 --> 00:44:52.360 you know, that's the idea of you know, the ops people are looking 619 00:44:52.360 --> 00:44:55.119 at their infrastructure dashboards and you know there's a there's a thing here, and 620 00:44:55.159 --> 00:45:00.079 then the developers are looking at their log aggregation dashboard that show them some stuff 621 00:45:00.079 --> 00:45:05.079 and they've got some graphs that they can It's like, whoa these two graphs 622 00:45:05.079 --> 00:45:08.320 they go up at the same time. Ah, maybe there is something here. 623 00:45:08.480 --> 00:45:12.280 Is there a knob here that we can wiggle, like, you know, 624 00:45:13.719 --> 00:45:16.039 just kick it out? Can we change that number for anything? Like 625 00:45:16.679 --> 00:45:20.599 when we do this, does it make a difference. So at least you 626 00:45:20.599 --> 00:45:23.679 have a sense that, yeah, there's a there's a constraint here and it's 627 00:45:23.760 --> 00:45:28.639 related to this value, and if we modify it, we can improve it 628 00:45:28.639 --> 00:45:31.039 in some way. Convincing that we have to buy a new hardware is challenging. 629 00:45:31.800 --> 00:45:36.960 And again Cloud took a lot of this away because it costs you nothing 630 00:45:37.039 --> 00:45:40.800 to turn that up for a test and then turn it back down again like 631 00:45:40.960 --> 00:45:45.920 a buck. It's just you solve a lot of problems just by doing those 632 00:45:45.000 --> 00:45:50.880 tests. Yeah, I just run a workshop on Kubernettes for some attendees, 633 00:45:50.920 --> 00:45:54.719 and I span up a Cubernettes cluster for every one of the attendees for two 634 00:45:54.800 --> 00:46:01.360 days. And it costs me I think four pounds per attendee to run a 635 00:46:01.480 --> 00:46:07.199 three note Kubernettes cluster in a KS for two days. I mean, can 636 00:46:07.239 --> 00:46:09.519 you imagine if you were having to do that on premise and ship in all 637 00:46:09.559 --> 00:46:15.800 that hardware three three hundreds just for each intending and a rack whirring in the 638 00:46:15.800 --> 00:46:19.199 background, but with hal blinky lights, So that'd be a thing. Yeah, 639 00:46:19.280 --> 00:46:21.280 yeah, we'd have blinky lights. But you know you can get a 640 00:46:21.440 --> 00:46:22.719 get a bunch of r G B L e eds, put them in a 641 00:46:22.760 --> 00:46:27.960 box just called the computer right anytime the wire goes back into the cloud. 642 00:46:30.239 --> 00:46:31.320 Might do that for the next workshop. It's like, yeah, you're all 643 00:46:31.360 --> 00:46:36.679 running those blinking lights in the corner. That one it's off. That's Carl 644 00:46:39.440 --> 00:46:46.639 Carl oh Man. What if we what have we not talked about here? 645 00:46:46.719 --> 00:46:51.800 Because there's a lot to doing a good job here. What are the learning 646 00:46:51.920 --> 00:46:54.159 ingredients? Like how do people really get You know, it's quick to install, 647 00:46:54.280 --> 00:46:59.159 but do you ten ten lines of code? You're up and running but 648 00:46:59.360 --> 00:47:04.760 there is some things you need to know. I think start with aspire. 649 00:47:05.039 --> 00:47:07.880 You know, start with a local dashboard. Start by turning some things on 650 00:47:07.960 --> 00:47:13.679 locally in your local environment, development environment, and see what some of these 651 00:47:13.719 --> 00:47:19.800 traces look like and see whether they're useful before you start thinking about where I'm 652 00:47:19.800 --> 00:47:22.039 going to put them, what observability back? In? What scale do I 653 00:47:22.079 --> 00:47:24.360 need? Am I going to use open source and host it myself? Am 654 00:47:24.400 --> 00:47:27.639 I going to go with a SaaS vender? Am I going to go one 655 00:47:27.679 --> 00:47:31.800 in built into my cloud? Use the spire? That's that's what the real 656 00:47:31.920 --> 00:47:36.639 cool thing about it's available as a container. Now that you can just run 657 00:47:36.719 --> 00:47:39.920 up locally. You don't need to use the DCP and all the other bits 658 00:47:39.960 --> 00:47:44.840 that go with it. You can just run it in a container and just 659 00:47:44.960 --> 00:47:49.079 experiment with it. What data do you see? Is it going to help 660 00:47:49.159 --> 00:47:52.880 you? You know, the value of the value to work ratio should be 661 00:47:52.920 --> 00:47:57.960 really small anyway, in like ten minds of code. Great, but start 662 00:47:57.960 --> 00:48:00.440 with something like that locally because you don't need to choose a vendor. I 663 00:48:00.840 --> 00:48:05.360 equate this to you. Remember when they brought an eye logger, And the 664 00:48:05.400 --> 00:48:08.559 great thing about eyelogger is it saved six months on every project because the first 665 00:48:08.559 --> 00:48:15.639 six months of any project was deciding between n log and love pnette and the 666 00:48:15.840 --> 00:48:20.519 arguments. The arguments that people had. I mean, that was the true 667 00:48:20.519 --> 00:48:23.599 flame Wars, you know. But they brought an eye logger, and the 668 00:48:23.639 --> 00:48:29.320 problem was that you needed to make that decision really far up from because if 669 00:48:29.320 --> 00:48:31.639 you didn't, then you'd have to retrofit it across every all of your code 670 00:48:31.840 --> 00:48:37.800 develop everybody this one. They brought an eye logger and they were like, 671 00:48:37.840 --> 00:48:39.559 great, yeah, everybody use ilogger. It'll just send it through eyelogger and 672 00:48:39.599 --> 00:48:43.599 then you'll choose your vendor. And I think open celemetry is the same thing 673 00:48:44.519 --> 00:48:47.039 is we've got a gimme. Now. It should be in everything, whether 674 00:48:47.039 --> 00:48:50.639 you're to wire it up or not. It costs you nothing to have it 675 00:48:50.679 --> 00:48:52.039 there, and when you do wire it out, mental help. Yeah, 676 00:48:52.079 --> 00:48:54.559 and that's the place to start. Well, And I like this tipping point 677 00:48:54.599 --> 00:48:59.159 angle that Steve brought up, which is that and more and more vendors see 678 00:48:59.199 --> 00:49:02.400 it that way. So the information you're getting out of open Columbatry just keeps 679 00:49:02.400 --> 00:49:07.719 getting better. Yeah, this whole thing works best as more and more vendors 680 00:49:07.719 --> 00:49:09.280 get on board. So that's the library authors, it's the you know, 681 00:49:09.360 --> 00:49:15.400 the various different language run times, it's the open Telemetry team. It's people 682 00:49:15.480 --> 00:49:20.599 within your organization, engineers within your organization buying into the fact that, okay, 683 00:49:20.639 --> 00:49:22.360 we've got this. I mean in dot Net it's super easy because we 684 00:49:22.400 --> 00:49:27.119 do have activity built in. It's it's there. If you're using dot net 685 00:49:27.159 --> 00:49:30.239 modern dot net, you've got the libraries, you've got the APIs. You 686 00:49:30.280 --> 00:49:34.760 need just to turn this stuff on and start instrumenting code. But as all 687 00:49:34.800 --> 00:49:38.159 of these people shift towards that general direction, I think that's where why open 688 00:49:38.159 --> 00:49:43.679 telemetry now is really starting to look like a good solution for everyone. What's 689 00:49:43.719 --> 00:49:47.320 the app Insights story? Should we just rip it out? And so Application 690 00:49:47.400 --> 00:49:52.239 Insights is currently redeveloping all of their SDKs to use activity source and activities. 691 00:49:52.239 --> 00:49:57.639 So it's not wow, I'm getting those now saying you're going to have to 692 00:49:57.719 --> 00:50:01.000 change this soon. So it shows that you Microsoft's listening to the open telemetry 693 00:50:01.000 --> 00:50:06.039 story. I mean, they had the experimental flags in the as your SDK 694 00:50:06.440 --> 00:50:07.599 last year, I think, and I think those are being removed now. 695 00:50:07.679 --> 00:50:12.920 So all the AZ your SDK is anything that you use from Cosmos to service 696 00:50:13.000 --> 00:50:16.639 bos if Angrid. All of those things will be instrumented with activities which means 697 00:50:16.679 --> 00:50:22.679 they're compatible with open celametry to send it and as your monitor. They've got 698 00:50:22.719 --> 00:50:27.960 their own exporter. Now they don't accept OTLP yet. Whether they will, 699 00:50:27.960 --> 00:50:30.840 I don't know. I don't work on that team, but they do have 700 00:50:30.880 --> 00:50:37.079 a distro so that you can do add as your monitor exporter and then you 701 00:50:37.119 --> 00:50:39.599 can just use open telemetry and decide later away're going to send it or just 702 00:50:39.639 --> 00:50:43.800 add the one line of code that says add to a your monitor and it 703 00:50:43.840 --> 00:50:47.079 will appear in app insites. Everybody's doing it. And to go back to 704 00:50:47.119 --> 00:50:52.880 this Blazer thing, do you recall hearing any kind of timeline as to when 705 00:50:53.880 --> 00:51:01.480 things will be more copestic with tracing and everything else that's going on. So 706 00:51:01.639 --> 00:51:07.599 I had a long conversation with the other Steve NDC London about exactly that, 707 00:51:07.639 --> 00:51:19.639 the more famous Steve. So they're looking at it, they are aware of 708 00:51:19.679 --> 00:51:22.440 the issues that are coming up, and they are you know, wanting to 709 00:51:22.480 --> 00:51:27.480 solve them. So I would imagine that yes, they will be solved soon 710 00:51:27.760 --> 00:51:30.639 and you can do it. It's just it's not as easy as ten lines 711 00:51:30.639 --> 00:51:32.039 of code. You know, you've got to go through a few more hoops, 712 00:51:32.159 --> 00:51:36.559 especially with the whole render mode thing. If you're using dynamic render modes, 713 00:51:36.639 --> 00:51:40.639 You've got some other problems that have to get solved, first state problems 714 00:51:40.639 --> 00:51:46.400 and things. Something I'm currently working on. But I'm hopeful. I'm hopeful 715 00:51:46.440 --> 00:51:51.559 for a future where this is the default. You know, this is well, 716 00:51:51.599 --> 00:51:54.159 I just add those ten lines of code, or even just add one 717 00:51:54.159 --> 00:51:58.679 line of code when we've worked on the SDKs a little bit more, and 718 00:51:58.800 --> 00:52:02.239 just add the one thing that's says send my stuff, and you know it's 719 00:52:02.239 --> 00:52:09.719 a what is it services? Don't let me debug my things? You know 720 00:52:09.800 --> 00:52:15.239 that that kind of simplicity just becomes the new cargo cult if you like, 721 00:52:15.639 --> 00:52:19.840 I don't just cargo cult thing. I well, I do cargo cult thing, 722 00:52:19.840 --> 00:52:22.280 but it's the new thing I cago cult. Well, guys, great 723 00:52:22.320 --> 00:52:27.880 conversation, Thanks so much for coming on absolutely and what's next. Let's send 724 00:52:27.920 --> 00:52:34.480 your inbox Marten. So I'm doing a load of conferences. I'm doing some 725 00:52:34.519 --> 00:52:38.440 meetups around the UK. I'll be at Cubcon, the Kubernettes Cloud Native Conference 726 00:52:38.920 --> 00:52:45.079 in match so but yeah, I am flying around the world spreading the good 727 00:52:45.079 --> 00:52:49.880 word about this sort of stuff. How about you you? Yeah, conferences 728 00:52:50.000 --> 00:52:52.880 coming up, so I'll be in Sweet Sweden next week for Sweet Tug, 729 00:52:52.960 --> 00:52:55.400 which will be in the past by the time this this one goes out. 730 00:52:57.519 --> 00:53:01.679 And then yeah, day to day we're going to be focused on where Elastic 731 00:53:01.719 --> 00:53:06.280 and the engineers can contribute to open telemetry. Really, I'm really starting to 732 00:53:06.320 --> 00:53:09.800 poke around in the open telemetry sd K repays and think about how we can 733 00:53:10.280 --> 00:53:14.320 make that experience good for good for everyone. Say, yeah, I think 734 00:53:14.320 --> 00:53:16.599 that will keep you busy for a while yet. Well, this is all 735 00:53:16.639 --> 00:53:22.159 great news. Thanks again, and I'll catch up with you at some conference 736 00:53:22.199 --> 00:53:27.559 somewhere. I'm sure, sure, all right, We'll see you next time 737 00:53:27.679 --> 00:53:51.719 on dot net rocks. Dot net Rocks is brought to you by Franklin's Net 738 00:53:51.800 --> 00:53:55.679 and produced by Pop Studios, a full service audio, video and post production 739 00:53:55.800 --> 00:54:00.440 facility located physically in New London, Connecticut, and of course in the cloud 740 00:54:01.039 --> 00:54:07.119 online at pwop dot com. Visit our website at d O T N E 741 00:54:07.199 --> 00:54:12.039 t R O c k S dot com for RSS feeds, downloads, mobile 742 00:54:12.079 --> 00:54:15.960 apps, comments, and access to the full archives. Going back to show 743 00:54:15.039 --> 00:54:20.599 number one recorded in September two thousand and two. And make sure you check 744 00:54:20.639 --> 00:54:23.679 out our sponsors. They keep us in business. Now go write some code. 745 00:54:24.000 --> 00:54:35.280 See you next time. You got javans and