1 00:00:01,080 --> 00:00:04,799 How'd you like to listen to dot net rocks with no ads? Easy? 2 00:00:05,360 --> 00:00:09,839 Become a patron for just five dollars a month you get access to a private 3 00:00:10,000 --> 00:00:14,439 RSS feed where all the shows have no ads. Twenty dollars a month will 4 00:00:14,439 --> 00:00:18,839 get you that and a special dot net Rocks patron mug. Sign up now 5 00:00:18,839 --> 00:00:35,960 at Patreon dot dot NetRocks dot com. Hey guess what, it's dot net 6 00:00:36,079 --> 00:00:42,200 rocks. I'm Carl Franklin at Amgard Campbell and we're representing four time zones today. 7 00:00:42,520 --> 00:00:47,399 Nice. We have two guests. Everybody in their own time zone. 8 00:00:48,000 --> 00:00:50,759 We're in their little corner of the world and their little corner of the world. 9 00:00:50,880 --> 00:00:54,799 However, electrons do not observe time zones, so we can all be 10 00:00:54,960 --> 00:00:59,119 together on this call. I think they observe it a little bit, maybe, 11 00:00:59,159 --> 00:01:02,000 you know, maybe a little bit. You know, Apple has some 12 00:01:02,119 --> 00:01:08,480 software. There's an old reference for you story. That was a great story 13 00:01:08,799 --> 00:01:15,560 call. Apple's got some software coming up, all right, speed light, 14 00:01:15,719 --> 00:01:19,640 hard to beat. Yeah, it's kind of the law Buffalo funny. How 15 00:01:19,640 --> 00:01:23,159 that works? Well, Uh, let's get right into it with better no 16 00:01:23,239 --> 00:01:34,799 framework awesome? Alright, Well, I found this series on Microsoft Learned. 17 00:01:36,200 --> 00:01:40,680 Oh that is really cool. It's and you know what's great about this is 18 00:01:40,719 --> 00:01:47,400 that it's a it's fundamental stuff, leveraging branches with GitHub and Visual Studio and 19 00:01:47,560 --> 00:01:51,280 intermediate series. It doesn't talk over your head and it doesn't talk down to 20 00:01:51,319 --> 00:01:55,239 you. But you know, there are those guys on the team that are 21 00:01:55,280 --> 00:01:59,519 afraid of Gethub and afraid of get you know, I used to be one 22 00:01:59,519 --> 00:02:01,799 of them. Sure, it's so heavily integrated into studio now, like, 23 00:02:01,840 --> 00:02:06,280 yeah, may just make your life pretty easy. Yeah, so this, 24 00:02:06,719 --> 00:02:10,680 you know, this is required learning. And so even if you know this 25 00:02:10,719 --> 00:02:14,479 stuff and you think you know it, it's good to brush up on it, 26 00:02:14,599 --> 00:02:17,680 especially if you're in charge of, you know, teaching other people on 27 00:02:17,719 --> 00:02:23,120 your team all about it. But anyway, I put a link to it. 28 00:02:23,120 --> 00:02:25,919 It's since this is episode eighteen eighty seven. You can go to eighteen 29 00:02:25,960 --> 00:02:30,919 eighty seven dot pop, dot m E and then I'll get you there. 30 00:02:31,400 --> 00:02:38,280 It's had what twenty five hundred views since January fifteenth. Yeah, it's just 31 00:02:38,439 --> 00:02:42,400 it was just put up a little while ago. Yeah, it's very good. 32 00:02:42,439 --> 00:02:45,120 It's cool and a lot of thumbs up and I watched it and it's 33 00:02:45,159 --> 00:02:47,360 good. No learning love it. Who's talking to us today, Richard, 34 00:02:47,719 --> 00:02:52,000 I grabbed a calm on top of show eighteen sixty nine, which is from 35 00:02:52,080 --> 00:02:55,560 last fall October twenty twenty three when we were at n DC Porto an on 36 00:02:55,719 --> 00:03:01,840 stage interview with one Charity Majors from company called the honey Comb that we put 37 00:03:01,800 --> 00:03:06,960 a terrible pressure on Martin or anything, but she was awesome. I don't 38 00:03:06,960 --> 00:03:09,879 know if Martin is gonna have and and see we're gonna have as many F 39 00:03:09,960 --> 00:03:14,520 bombs however, No, but that's not about it either. And just to 40 00:03:14,599 --> 00:03:19,000 reinforce the point, and g Houseworth had this comment which he said, I 41 00:03:19,080 --> 00:03:22,759 never missed an episode of dot Net Ross because they absolutely rock, and this 42 00:03:22,800 --> 00:03:27,520 episode with Charity Majors was none, if not the best. Yeah, no 43 00:03:27,560 --> 00:03:30,360 pressure, the insight and energy was awesome, and we did have the distinct 44 00:03:30,360 --> 00:03:35,319 advantage of being on stage with her. Two you're at NDCA Porto at the 45 00:03:35,360 --> 00:03:38,719 time. But you know, we're talking about observability more and more, and 46 00:03:38,759 --> 00:03:42,639 I'm really looking forward to getting into this one because we've got a couple of 47 00:03:42,680 --> 00:03:45,199 different viewpoints coming at it. But if you want to get started on this 48 00:03:45,280 --> 00:03:47,919 subject, that show we did with Charity no fool and it was great. 49 00:03:49,080 --> 00:03:52,319 It was great. G thanks so much for your comment and a copy of 50 00:03:52,360 --> 00:03:53,280 music Code by. It's on its way to unit. If you'd like a 51 00:03:53,319 --> 00:03:57,080 copy of music Code buy. I write a comment on the website at dot 52 00:03:57,080 --> 00:04:00,080 at Rocks dot com or on the Facebook to publish every show there. And 53 00:04:00,080 --> 00:04:01,080 if you comment there and I reading on the show, was that your copy 54 00:04:01,120 --> 00:04:04,199 of music go by? And uh, you can follow us on Twitter if 55 00:04:04,199 --> 00:04:08,039 you want. But the cool kids are at masadon and you know, I 56 00:04:08,120 --> 00:04:12,759 put all of my social media links at Carl Franklin dot com cool so we 57 00:04:12,800 --> 00:04:15,520 don't have to just rattle them off. I got a bunch of them up 58 00:04:15,560 --> 00:04:20,720 there. I think I've got a Richard dot Campbell dot me, you know, 59 00:04:20,759 --> 00:04:24,480 one of the about me pages that has to catch that stuff up there 60 00:04:24,480 --> 00:04:27,399 too. If I actually looked at it recently, I don't know. I 61 00:04:27,399 --> 00:04:30,199 think I did it when we when we were doing the mass account thing. 62 00:04:30,600 --> 00:04:33,720 Yeah, but yeah, you know it is. But you're at rich Campbell 63 00:04:33,759 --> 00:04:39,360 on Twitter and what's your masdon and pretty and rich Campbell at masadon dot social 64 00:04:39,519 --> 00:04:43,519 you know, yeah, yeah, oh no, it's our Campbell dot me. 65 00:04:43,720 --> 00:04:46,680 There you go. R Campbell dot me. All right, got them 66 00:04:46,680 --> 00:04:51,120 all. Well I'm going to just start talking about that. Carl Franklin dot 67 00:04:51,120 --> 00:04:54,120 com. Just go there. You want to do that now, Okay? 68 00:04:54,279 --> 00:04:58,839 Like yeah, okay, shorter, Yeah it is shorter, and uh you 69 00:04:58,879 --> 00:05:01,360 know. Then then everybody has a lot more to read that they didn't want 70 00:05:01,399 --> 00:05:10,800 to so too long, don't read? Yeah, go away. So let's 71 00:05:10,800 --> 00:05:15,160 bring on our guests, Steve Gordon and Martin Thwaits. Steve is a plural 72 00:05:15,199 --> 00:05:20,079 site author, Microsoft MVP and engineer at Elastic maintaining their dot net libraries. 73 00:05:20,120 --> 00:05:25,680 Steve enjoys sharing his knowledge by presenting talks at user groups and conferences, writing 74 00:05:25,720 --> 00:05:30,759 on his blog at Stevejgordon dot co dot uk, and creating videos. You 75 00:05:30,759 --> 00:05:33,720 can find him on most social media platforms under the username at Steve J. 76 00:05:33,879 --> 00:05:42,600 Gordon. Martin Thwaits is an observability evangelist, dot net developer, Microsoft MVP. 77 00:05:43,120 --> 00:05:47,040 He works for Honeycomb dot Io, an observability vendor. As a developer 78 00:05:47,120 --> 00:05:53,879 advocate, he travels the world spreading the good word about open telemetry and observability 79 00:05:53,920 --> 00:05:59,079 engineering. He'll talk to anyone about observability, even the likes of us Jamos 80 00:06:01,600 --> 00:06:08,560 that's not very nice. So welcome guys. Hey, thanks having us, 81 00:06:08,879 --> 00:06:13,959 Yeah, thanks for having me. So this culminated at in the fish Powl 82 00:06:14,000 --> 00:06:15,560 at NDC London, right, Yeah, we were all hanging out, you 83 00:06:15,600 --> 00:06:23,639 know. I was taking it easy because a co host wasn't there, so 84 00:06:23,639 --> 00:06:29,600 so at some point somebody just declared it Richard's office. There was a few 85 00:06:29,600 --> 00:06:31,920 other foot I think Miss Manners came in and recorded some shows like there's a 86 00:06:31,920 --> 00:06:34,240 few folks came and went and used it. I did some run as as 87 00:06:34,240 --> 00:06:36,720 well. I was there. Yeah, you could. Mostly we sat around. 88 00:06:36,879 --> 00:06:41,279 I do believe a couple of bottles of whiskey may have just disappeared into 89 00:06:41,279 --> 00:06:46,120 that room and never we're seen again, as they do, as they do. 90 00:06:46,560 --> 00:06:48,439 Well, Steve was in the UK and it's afternoon there or evening, 91 00:06:48,560 --> 00:06:51,839 right, so you've got your whiskey there, Steve. That's right. Yeah, 92 00:06:53,000 --> 00:06:57,319 I'm I'm declaring it post six pm. Fine time to drink, right, 93 00:06:57,600 --> 00:07:02,560 I am going tonight to Texas Day Brazil with Robert Ramsey. Nice Richard, 94 00:07:02,800 --> 00:07:08,199 Yeah, and we will I'm sure have plenty of meat and whiskey. 95 00:07:09,160 --> 00:07:13,120 So anyway, what do you guys think of following up with on Charity's Talk 96 00:07:13,160 --> 00:07:20,079 here. Are you scared or are you up to the challenge? Martin said 97 00:07:20,079 --> 00:07:23,879 that very well, don't you get it? Very well? Answered that question. 98 00:07:24,600 --> 00:07:28,879 I mean, F bombs is kind of our company policy. No kidding, 99 00:07:28,920 --> 00:07:30,959 you know, I'm not saying he's part of the interview process, but 100 00:07:31,000 --> 00:07:34,439 I'm not not saying that either. That is so cool. I mean, 101 00:07:34,480 --> 00:07:36,439 you know, let's think about it for a second. I mean, who 102 00:07:36,439 --> 00:07:40,399 are you going to offend? Right? You go to any movie, even 103 00:07:40,439 --> 00:07:44,600 if it's PG, and there's F bombs everywhere, right everywhere. But I 104 00:07:44,600 --> 00:07:47,519 mean we do get some email from folks on set when they show up. 105 00:07:47,519 --> 00:07:51,600 We do try to bleep them too, Yes we do. But I appreciate 106 00:07:51,639 --> 00:07:57,680 the sentiment either way. It's like there's so much of observer your ability that 107 00:07:57,720 --> 00:08:01,439 falls into the what the heck is going on in this app or in this 108 00:08:01,519 --> 00:08:03,839 case, what the f is going on? And it's you know, trying 109 00:08:03,879 --> 00:08:09,279 to be Canadian here. Yeah, we have a we have a you've heard 110 00:08:09,279 --> 00:08:13,920 of MTTR the Meantime to Recovery, and so we have a thing called mtt 111 00:08:13,959 --> 00:08:18,720 WTF, which when you're debugging systems, it's how long does it take you 112 00:08:18,800 --> 00:08:24,040 to go what there was that No, that thing doesn't call that thing, 113 00:08:24,120 --> 00:08:28,399 that doesn't happen. That's not right at all. So what I haven't used 114 00:08:28,439 --> 00:08:37,840 this Honeycomb, But what I remember distinctly about that conversation was it's architecturally different 115 00:08:37,960 --> 00:08:43,080 from a lot of other, you know, well known telemetry kind of things 116 00:08:43,080 --> 00:08:48,159 that you hook on or add on side cars and to get any kind of 117 00:08:48,240 --> 00:08:52,840 insight and observability into what's going on in your application places where you just can't 118 00:08:52,879 --> 00:08:56,000 put a break point. Yeah, I mean, I think this is the 119 00:08:56,080 --> 00:09:01,559 whole tracing debate versus blogs and APM type stuff, where tracing is the thing 120 00:09:01,600 --> 00:09:05,399 that's different. It's the evolution, if you like, of that idea of 121 00:09:05,519 --> 00:09:09,080 just log everything, throw it in a big bucket, try and run some 122 00:09:09,480 --> 00:09:13,279 log stash things on the back of it, or fluent stuff to build it 123 00:09:13,320 --> 00:09:16,679 all and try and get a picture. It's really the tracing stuff that is 124 00:09:16,720 --> 00:09:22,279 the sort of next evolution that isn't is not new, and that's the thing 125 00:09:22,279 --> 00:09:24,679 that really gets me. I was going to say, we've had tracing for 126 00:09:24,720 --> 00:09:28,240 a long time, but it's new about honeycomb tracing. So well, Honeycomber 127 00:09:28,320 --> 00:09:33,519 is just a back end, we just take spans and allow you to query 128 00:09:33,519 --> 00:09:37,919 them in weird and wonderful ways. In rapid fire, you know the idea 129 00:09:37,960 --> 00:09:43,360 of an outage going on and you're waiting five minutes for telemetry data to get 130 00:09:43,399 --> 00:09:48,039 through for you to ask a question, to say, did I fix it? 131 00:09:48,080 --> 00:09:52,080 Is it now fixed? That idea of just rapid fire, ask questions, 132 00:09:52,120 --> 00:09:54,000 Ask questions, because that's really what you want to do. You're on 133 00:09:54,080 --> 00:09:58,840 that conversation with your production system that's like, you know what's happening? Who 134 00:09:58,919 --> 00:10:03,440 hurt you? Yeah? And that's really what we're trying to show me where 135 00:10:03,480 --> 00:10:07,600 the bad man? Yeah, it couldn't help, but notice your pain. 136 00:10:09,519 --> 00:10:13,840 But that's what we try to do is really try to allow people to just 137 00:10:13,919 --> 00:10:20,360 ask weird and wonderful questions. The example we users is, you know what 138 00:10:20,519 --> 00:10:26,639 happens when the French user is currently in Norway using the French language pack in 139 00:10:26,679 --> 00:10:31,679 the Norway region on iOS fourteen point one, and that's the era that's happening. 140 00:10:31,679 --> 00:10:35,919 It's only on those users, because that's that's where we are with systems 141 00:10:35,960 --> 00:10:39,480 now. There's no longer this system that you use inside of your organization that 142 00:10:39,559 --> 00:10:45,200 only your users are only your organizational users. Everything's global. You know, 143 00:10:45,279 --> 00:10:48,799 people are working from home, working from anywhere. So even if you've got 144 00:10:48,840 --> 00:10:52,200 just an internal system, these people are from everywhere now, so you know, 145 00:10:52,240 --> 00:10:54,840 we're starting to get into a world where things have changed. It's not 146 00:10:56,480 --> 00:11:00,519 about I've got six month release cycle. I can wait months. I can 147 00:11:00,600 --> 00:11:05,759 do a QA cycle. And my my whole thing at the moment is if 148 00:11:05,759 --> 00:11:09,039 we don't change the tools, and I'm not talking about the vendor back ends. 149 00:11:09,080 --> 00:11:13,480 I'm talking about like open telemetry and tracing and profiles that are coming up 150 00:11:13,480 --> 00:11:16,879 and various other signals that are coming up, we don't give people better tools, 151 00:11:16,240 --> 00:11:20,120 but we're saying you need to do things better than you did before. 152 00:11:20,720 --> 00:11:24,799 Like it's not it's not okay, yeah, yeah, the the yeah, 153 00:11:24,919 --> 00:11:30,399 the reliability strategy of be more careful next time. Yeah, don't do that. 154 00:11:31,240 --> 00:11:35,000 Yeah, don't touch that. It's yeah. The real point here is 155 00:11:35,320 --> 00:11:39,200 the tools should lead us to the path of success. They should resist. 156 00:11:39,279 --> 00:11:43,480 We resist as are making mistakes, and so we get to still be human 157 00:11:43,639 --> 00:11:48,919 and have reliable software. What a concept. Yeah, And I mean that 158 00:11:48,039 --> 00:11:52,519 was I think from my experience that was one of the problems with just sort 159 00:11:52,519 --> 00:11:56,320 of relying on logging as your observability. Logging in a few metrics was was 160 00:11:56,399 --> 00:12:01,679 us doing observability. But logging can be quite good if you spend a heck 161 00:12:01,720 --> 00:12:05,879 of a lot of time and I use heck there heck of a lot of 162 00:12:05,919 --> 00:12:09,480 time getting the logging in that you need. It tends to be quite retrospective. 163 00:12:09,919 --> 00:12:13,799 You figure out there's a problem in a particular code path, you put 164 00:12:13,799 --> 00:12:16,240 a new log in, you'd ship to production, You wait for someone else 165 00:12:16,240 --> 00:12:18,240 to hit the error, and it logs something else that gives you another clue 166 00:12:18,240 --> 00:12:22,440 of where you want to log next, right, And so it's it's very 167 00:12:22,440 --> 00:12:26,440 retrospective and looking back. Whereas with traces, if you design them well and 168 00:12:26,480 --> 00:12:31,960 you put tracing around your key points in your application code, you get the 169 00:12:31,000 --> 00:12:35,720 full, you know, waterfall effect of what's happened, and you've added hopefully 170 00:12:35,759 --> 00:12:39,480 decent attributes on there so that you can, as Martin said, go back 171 00:12:39,480 --> 00:12:46,519 and answer any weird and wonderful questions and combinations of events with particular attributes on 172 00:12:46,559 --> 00:12:48,159 them. Awesome, And Steve, I got to ask you, like I 173 00:12:48,240 --> 00:12:52,039 normally resist two guest shows because you get up with a lot of voices and 174 00:12:52,120 --> 00:12:56,360 I but we're talking of observability, and Martin works for the company. Arguably 175 00:12:56,399 --> 00:13:01,279 that makes one of the most popular observeability tools out there. Does Alaska you 176 00:13:01,639 --> 00:13:05,639 does Elastic use Honeycomb or is it just you know, you're trying to understand 177 00:13:05,679 --> 00:13:09,639 what your own product does. What's the story there? So we provide an 178 00:13:09,679 --> 00:13:13,799 application performance monitoring solution as well, so we compete in a nice friendly way. 179 00:13:15,000 --> 00:13:16,360 So we have a different back end. Our back ends built on top 180 00:13:16,399 --> 00:13:22,879 of Elastic Search, which is kind of Elastic's main product, and our APM 181 00:13:22,960 --> 00:13:26,399 server uses that as the data store, so you can ship your telemetry data 182 00:13:28,360 --> 00:13:31,080 to us. So you can use our APM agent to do that, or 183 00:13:31,240 --> 00:13:35,919 today you can also use any open telemetry capable agent as well. So the 184 00:13:37,559 --> 00:13:41,519 sort of built in what we call the Vanilla Open Telemetry SDK can be used 185 00:13:41,559 --> 00:13:48,159 to ship over the open telemetry protocol directly into Elastic APM server and then we'll 186 00:13:48,440 --> 00:13:52,279 stare that for you and then give you the again a different form of you 187 00:13:52,799 --> 00:13:56,679 to search over those traces and those logs. I'm just staggered at the idea 188 00:13:56,679 --> 00:14:01,200 that we're talking about Elastic for something other than search. What I know, 189 00:14:01,639 --> 00:14:03,080 I thought they made other products, but when don't we ever discussed that? 190 00:14:03,200 --> 00:14:07,600 So okay, so this is the elastic has their eight PM product, which 191 00:14:07,639 --> 00:14:11,519 is another aspect of them. But you're both guys are in observability in a 192 00:14:11,559 --> 00:14:15,440 big way. That's right. Yeah, I think that the really I mean, 193 00:14:15,600 --> 00:14:16,919 one of the reasons why I thought it was really good that we're both 194 00:14:16,960 --> 00:14:22,080 here is because different back ends provide you with different query and capabilities. Right. 195 00:14:22,120 --> 00:14:26,799 You know, the idea that you just buy one tool off the shelf 196 00:14:26,840 --> 00:14:30,679 and everything just works is just not going to work in today's world. You 197 00:14:30,720 --> 00:14:37,559 really need crazy you know, you really need to think about what's going to 198 00:14:37,600 --> 00:14:39,440 work for your organization. You know, what are you trying to achieve with 199 00:14:39,480 --> 00:14:43,120 observability, and then choose the back ends that work. And that's where open 200 00:14:43,159 --> 00:14:48,639 celemetry is really awesome because you can send it to two providers. You can 201 00:14:48,679 --> 00:14:52,759 send that same exact data to two separate viders and say, well, this 202 00:14:52,799 --> 00:14:56,600 one's really good at metrics. It's really good at showing me some dashboards about 203 00:14:56,960 --> 00:15:01,759 pre aggregated data. Awesome, that's a really good tool for that thing. 204 00:15:01,759 --> 00:15:07,639 It does infrastructure metrics. There's one that does say costing and takes your metrics 205 00:15:07,679 --> 00:15:11,679 and can project costs based on previous metrics. Those are really cool tools. 206 00:15:11,799 --> 00:15:16,200 There's other ones that focus on, say Servilus, there's other ones that focus 207 00:15:16,279 --> 00:15:18,799 on you know, everything's just they've got their own niches. You know, 208 00:15:20,080 --> 00:15:24,480 Elastic has their own niche. We have our own niche. So open celemetry 209 00:15:24,519 --> 00:15:28,240 just makes it easy as your needs change, you can change provider. There's 210 00:15:28,279 --> 00:15:33,320 no I'm going to change provider. We'll install the new agent on fifteen hundred 211 00:15:33,399 --> 00:15:37,320 machines. You know, well, I'm nobody's going to do that. Or 212 00:15:37,440 --> 00:15:41,799 now it's I'm going to move vender. I'll change three lines of config right 213 00:15:43,120 --> 00:15:46,600 now. Speaking of that, I mean you bring up pain right. I 214 00:15:46,639 --> 00:15:52,840 mean, let's say I've got an app in production and I've got you know, 215 00:15:52,600 --> 00:15:56,240 thousands and maybe one hundreds of thousands of lines of code. How does 216 00:15:56,279 --> 00:16:03,240 one just retrofit this kind of tracing into an app like that? Obviously there 217 00:16:03,279 --> 00:16:07,919 isn't any kind of wizard where you can just snap your fingers and it's instant 218 00:16:08,039 --> 00:16:15,799 insights, right, ten lines of code, ten ten lines of code in 219 00:16:15,879 --> 00:16:19,960 not for every object in every page. And so I mean the thing is 220 00:16:21,000 --> 00:16:22,960 with open celemetry, we've got three care signals, we've got tracing, we've 221 00:16:23,000 --> 00:16:29,200 got metrics, and we've got locks. Tracing is predominantly back end. It's 222 00:16:29,240 --> 00:16:33,240 focused on this idea of we've got a context. So a trace is a 223 00:16:33,279 --> 00:16:36,600 context, a thing that's been done. So it really works best in the 224 00:16:36,639 --> 00:16:41,080 back end because you've got an inbuilt context of a web request. It starts 225 00:16:41,120 --> 00:16:44,679 and it finishes and things happen. It doesn't work well currently in the front 226 00:16:44,759 --> 00:16:48,200 end, so it doesn't work well in the client side. What people call 227 00:16:48,559 --> 00:16:55,759 rum or really use monishing, so it doesn't work well in that way. 228 00:16:55,960 --> 00:17:00,399 Open celemetry is trying to build a signal for that. So in the backkend 229 00:17:00,799 --> 00:17:03,000 it's actually really easy. You just add say ten lines of code to your 230 00:17:03,000 --> 00:17:07,759 startup and say I would like to see a span for every HDTP request that 231 00:17:07,799 --> 00:17:11,240 I get into my site. I would like to see a span for every 232 00:17:11,519 --> 00:17:14,359 HDDP request I make out of my site. And when you say span, 233 00:17:15,160 --> 00:17:18,720 there's several definitions of that word, but in this context, what does it 234 00:17:18,799 --> 00:17:22,920 mean. So we're on a dot net podcast, so let's use dot net 235 00:17:22,000 --> 00:17:27,480 terms. You can create an activity, so activity and this is where we 236 00:17:27,480 --> 00:17:34,119 say we've had this for ages because activity has existed since I think dot Net 237 00:17:34,119 --> 00:17:38,359 called two point one I think was when they first started bringing in activity. 238 00:17:40,279 --> 00:17:45,759 It's been there for a while, and activity kind of represents a sort of 239 00:17:45,839 --> 00:17:49,359 unit of work, if you like, and an actor you create an activity, 240 00:17:49,480 --> 00:17:53,440 you create one for your HDTP inbound request, one for your outbound request, 241 00:17:53,519 --> 00:17:56,640 one for your when you hit a database. And these are all in 242 00:17:56,720 --> 00:18:03,119 built into the libraries already. Espnut core has It's equal client has it, 243 00:18:03,599 --> 00:18:08,160 Myscore client has it, MPGSQL Entity Framework, you name it. All of 244 00:18:08,200 --> 00:18:14,160 those have been moved up. Even the Azure client SDKs have now been changed 245 00:18:14,160 --> 00:18:18,440 over to using activity sources and activities to create these spands. So a span 246 00:18:18,599 --> 00:18:22,319 being this, I've got a start time, and I've got an end time, 247 00:18:22,359 --> 00:18:26,839 and I've got some attributes, and it allows you to build these really 248 00:18:26,960 --> 00:18:30,160 rich waterfalls that allow you to see how things transition through your system. But 249 00:18:30,240 --> 00:18:33,480 you can do that with about ten lines of code in the startup of your 250 00:18:33,480 --> 00:18:38,359 application. And with a spire, they're actually promoting this idea of these shared 251 00:18:38,400 --> 00:18:42,599 projects where all of that setup happens once, so it's not even ten lines 252 00:18:42,640 --> 00:18:47,960 of code per application, it's ten lines of code in a common startup, 253 00:18:48,880 --> 00:18:53,079 So you can get essentially all this automatic data just by adding ten lines of 254 00:18:53,079 --> 00:18:56,000 code and then putting in a little bit config to say where you're going to 255 00:18:56,000 --> 00:18:59,519 send it. You're going to put in the elastic APM end points, you're 256 00:18:59,519 --> 00:19:02,319 going to put in the honeycombman pints, you're going to put it into any 257 00:19:02,319 --> 00:19:03,799 of the end points, and if you want to go further, you can 258 00:19:03,920 --> 00:19:07,799 you can do zero lines and code as well. So there's also this kind 259 00:19:07,799 --> 00:19:12,400 of concept of auto instrumentation where by flicking a few environment variables to configure the 260 00:19:12,519 --> 00:19:18,200 profiling hooks that the dot net run time has built into it, if you 261 00:19:18,240 --> 00:19:22,440 can point it at this auto instrumenting profil and that can go in and wire 262 00:19:22,559 --> 00:19:26,359 up whether it be the open telemetary STK or the elastic APM agent wire out 263 00:19:26,519 --> 00:19:30,759 for you in your application with no code changes at all. Just deploy the 264 00:19:30,839 --> 00:19:34,240 environment variables and restart the app, which can be a really good way to 265 00:19:34,240 --> 00:19:40,119 get started with. That's amazing actually, Yeah, with basic instrumentation of what's 266 00:19:40,160 --> 00:19:44,440 happening. Guys, you're talking about the easy part collecting too much, Like 267 00:19:44,559 --> 00:19:49,000 that's really easy. Yeah, you know, I'm a guy who who spent 268 00:19:49,079 --> 00:19:52,160 time in the sequel world where we used to do full traces of every transaction 269 00:19:52,279 --> 00:19:56,279 going through a SQL server because it was really great at filling up hard drives 270 00:19:56,279 --> 00:20:00,519 a a bit. It's the real question is after you gather all this stuff, 271 00:20:00,799 --> 00:20:03,720 and I emphasis on stuff, I can think of several other words that 272 00:20:03,720 --> 00:20:10,960 would apply begin with s, Yeah, how do you mash this in the 273 00:20:11,039 --> 00:20:14,680 submission? And you know to the line I would use when working with the 274 00:20:14,720 --> 00:20:19,400 team was create actionable items, things we can do to make the system better. 275 00:20:21,279 --> 00:20:23,960 That is the rub really, you know, and that I think is 276 00:20:25,039 --> 00:20:32,359 what has been tracing's downfall for probably the last sort of five years really because 277 00:20:32,720 --> 00:20:37,160 people see tracing as a I have a trace ID and now I can see 278 00:20:37,160 --> 00:20:41,279 a trace waterfall for that thing, but I need something else in order to 279 00:20:41,319 --> 00:20:44,720 tell me what trace ID to use. And what that means is you end 280 00:20:44,799 --> 00:20:48,559 up with say ninety nine point nine nine nine recurring percent of your data not 281 00:20:48,640 --> 00:20:53,480 being used and just sitting on hard drives and costing you money. And that 282 00:20:53,519 --> 00:20:56,079 I think has been one of the big downfalls that people have said, well, 283 00:20:56,240 --> 00:21:00,720 tracing is not really useful because I need to know what trace it was 284 00:21:00,759 --> 00:21:03,519 so I'm just going to use logs because I have to keep them anyway, 285 00:21:04,680 --> 00:21:08,359 And I think that's been one of the big downfalls. But I think there's 286 00:21:08,480 --> 00:21:14,559 there's a movement now around tracing, around things like sampling and doing various different 287 00:21:14,599 --> 00:21:18,079 mechanisms for how we reduce down the amount of data that we store and only 288 00:21:18,079 --> 00:21:22,200 store the interesting stuff. Yeah, that's assessing what's interesting is not a trivial 289 00:21:22,240 --> 00:21:26,880 problem either, right, Yeah, I mean interesting is subjective at best, 290 00:21:27,799 --> 00:21:32,240 you know, you know, but the general rule is, you know, 291 00:21:32,359 --> 00:21:36,079 you keep every one of your slow traces and keep everyone that has an error 292 00:21:36,119 --> 00:21:38,119 in it, because there was at least going to be the interesting ones. 293 00:21:40,440 --> 00:21:44,200 But you do that, and then you can start to think about how do 294 00:21:44,279 --> 00:21:48,240 I query my trace data, not how do I query my logs and my 295 00:21:48,359 --> 00:21:51,839 metrics, and then get back to tracing. How do I start with that 296 00:21:51,920 --> 00:21:56,039 tracing data, because it's really rich. And I think that's where we're seeing 297 00:21:56,079 --> 00:21:59,920 this big change now with open telemetry, because the back ends, all the 298 00:22:00,039 --> 00:22:03,440 back ends now are starting to look at this trace analysis type stuff, whether 299 00:22:03,480 --> 00:22:06,720 it's ours or Elastic or a lot of the open source ones that are normally 300 00:22:06,720 --> 00:22:11,359 based on ClickHouse as a data store. They're starting to do this idea of 301 00:22:11,440 --> 00:22:14,759 well, you know, how do I get and ask questions of my trace 302 00:22:14,839 --> 00:22:17,960 data? Because it's really rich, it's really wide, it's got loads of 303 00:22:18,000 --> 00:22:22,440 attributes, all that interesting stuff, and that really is the rub. Yeah, 304 00:22:22,559 --> 00:22:26,240 so always a question of what's actually interesting, because you're always afraid of 305 00:22:26,240 --> 00:22:30,039 what you shave off because that might have been the interesting data. But yeah, 306 00:22:30,039 --> 00:22:33,720 we've had so many people who come to us with, yeah, I 307 00:22:33,759 --> 00:22:37,000 don't really like tracing because you know, every time I wanted to ask a 308 00:22:37,079 --> 00:22:41,640 question, the customer got an error. It was sampled out because or discarded 309 00:22:41,680 --> 00:22:47,079 because somebody took one percent of random sample data. And the era happens, 310 00:22:47,119 --> 00:22:49,119 you know, once in a million times, so they never get the trace 311 00:22:49,119 --> 00:22:53,039 that they want. And that's where we're doing a lot of evolution right now 312 00:22:53,079 --> 00:22:56,400 in the open telemetric collector and some of the stuff that even me and Steve 313 00:22:56,400 --> 00:23:02,279 were talking about at NBC London about how we roll up stands and write tell 314 00:23:02,359 --> 00:23:06,079 something processes that are more bespoke and you know, this is the sort of 315 00:23:06,079 --> 00:23:10,759 conversations we're having now because it's vendor agnostic and so you've got so many more 316 00:23:11,240 --> 00:23:12,839 people. You know, me and Steve, we don't work for the same 317 00:23:12,880 --> 00:23:18,759 company, but we're working towards a common goal of reducing the amount of data, 318 00:23:18,799 --> 00:23:21,920 making it more actionable, and making it easier for people to implement. 319 00:23:22,319 --> 00:23:26,200 So you've got, you know, hundreds of developers all working on that same 320 00:23:26,240 --> 00:23:30,039 code base. And that's the major advantage of open telemetry as a goal, 321 00:23:30,079 --> 00:23:33,559 isn't it for the industry that rather than every vendor focusing on their way of 322 00:23:33,599 --> 00:23:37,640 doing something and having their own little pieces that they can use in sales and 323 00:23:37,680 --> 00:23:42,559 marketing, it's actually what's the right solution across the industry for how this should 324 00:23:42,559 --> 00:23:47,519 be done in the best way for the organized relations that need this information to 325 00:23:47,559 --> 00:23:51,839 be able to store this stuff in a sensible, cost effective kind of way 326 00:23:51,920 --> 00:23:56,440 to do this level of sort of advanced sampling, sort of intelligent sampling, 327 00:23:56,440 --> 00:24:00,640 if you will, on the on the tail of things coming through. And 328 00:24:00,680 --> 00:24:03,079 I think we're starting to see that shift with vendors now kind of all starting 329 00:24:03,079 --> 00:24:07,920 to adopt finally, I think open telemetry as that solution. Yeah, it's 330 00:24:07,960 --> 00:24:11,519 always a question of mentioning this data together into like do you make sense of 331 00:24:11,519 --> 00:24:15,160 it? They and what's that you see mostly it's the same stuff over and 332 00:24:15,200 --> 00:24:18,359 over again. It is what you already know. I'm tapping on the book 333 00:24:18,359 --> 00:24:22,440 I've never freaking finished and still hope to one day. With the history of 334 00:24:22,480 --> 00:24:30,640 dot net, where the seql team had a tool where anytime a unique execution 335 00:24:30,839 --> 00:24:33,200 path happened inside its equal server, it would stop. It would hit a 336 00:24:33,200 --> 00:24:37,839 breakpoint, so as long as it was executing the normal path or normal paths 337 00:24:38,000 --> 00:24:41,279 over time, it would continue running. And so it was one It was 338 00:24:41,359 --> 00:24:45,599 like a tool designed to find the one in one hundred million case. So 339 00:24:45,640 --> 00:24:49,720 you're iterating rapidly and it's like initially it's breaking on each unique it's breaking on 340 00:24:49,759 --> 00:24:52,720 each unique path, but after you say, okay, that's a fine path, 341 00:24:52,839 --> 00:24:56,559 keep going. Then eventually it's like running for days before it's finally like 342 00:24:56,759 --> 00:25:00,240 think how about this? You're like, now, well the heck did you 343 00:25:00,279 --> 00:25:04,880 get there? And the reasons in the story of the context of dot net 344 00:25:04,920 --> 00:25:10,160 is because that's what made dot net two point zero. Making dot Net run 345 00:25:10,200 --> 00:25:15,559 inside a sequel server, made them test dot net in away never been tested 346 00:25:15,599 --> 00:25:19,119 before, and they found rare and unusual bugs because they were running into context 347 00:25:19,160 --> 00:25:23,759 of sequel server. Here end of the history lesson. I love the idea 348 00:25:23,880 --> 00:25:29,359 of like the system just going hold on a minute, can you just get 349 00:25:29,359 --> 00:25:37,279 your supervise it this road before boys, let's have a little chat. And 350 00:25:37,720 --> 00:25:41,039 speaking of that, my friends, let's take a brief break for this very 351 00:25:41,079 --> 00:25:48,519 important message, and we're back. It's dot net rocks. I'm Richard Campbell, 352 00:25:48,680 --> 00:25:52,200 it's Martin Waite's and Steve Gordon and my buddy Carl Hey and talking a 353 00:25:52,240 --> 00:25:56,559 little bit about more than one way to get observability here and the fact that 354 00:25:56,799 --> 00:25:59,759 you know, now you're really creating trouble for me. It's not like I 355 00:25:59,799 --> 00:26:02,880 was get too much telemetry out of my Honeycomb stack, but now I'm gonna 356 00:26:02,920 --> 00:26:06,599 go get elastic eight PM as well, feed the same data to it twice 357 00:26:06,880 --> 00:26:10,799 because I just own too many drives, Like let's try and kill them all. 358 00:26:11,319 --> 00:26:17,160 Like doing analysis between those data sets could be interesting too. I mean, 359 00:26:17,160 --> 00:26:19,319 we do that law, you know, the Bakoff type stuff. You 360 00:26:19,359 --> 00:26:22,319 know, once you've got an open celemetry collector in place, you just sort 361 00:26:22,319 --> 00:26:30,519 of send it to five vendors, see which ones do the best fit out 362 00:26:30,519 --> 00:26:33,559 what you want to hear. You know, in accounting they call it reconciliation, 363 00:26:33,079 --> 00:26:36,599 right, Like you got to add up the numbers more than one way. 364 00:26:37,759 --> 00:26:41,599 I wonder we're not there yet in our industry. Like if I really 365 00:26:41,960 --> 00:26:48,799 wanted to be a pain in the butt about a performance contract, the fact 366 00:26:48,839 --> 00:26:52,559 that I could ask her to the telemetry to be sent to two different sources 367 00:26:52,599 --> 00:26:56,039 and we'll reconcile the performance metrics against it from two different places and see if 368 00:26:56,079 --> 00:26:59,680 they actually agree. When they don't, wonder why, Like, yeah, 369 00:26:59,759 --> 00:27:04,000 you really press against that contract. Yeah, I haven't been in there be 370 00:27:04,119 --> 00:27:12,559 CTO in too long. You've given me new ideas on behalf of myself, 371 00:27:12,599 --> 00:27:22,480 and I am sorry to the industry. Yeah, I mean it's all Yeah. 372 00:27:22,480 --> 00:27:25,839 The other side of this is like nowurkicking, we're doing multiple analysis on 373 00:27:25,880 --> 00:27:29,920 the same sets of data. Hopefully obviously different tools, like you said, 374 00:27:30,000 --> 00:27:33,440 have better insights in different areas. But yeah, I'd love to have a 375 00:27:33,480 --> 00:27:37,400 standard set of comparisons. Says oh yeah, no, we reconcile the same 376 00:27:37,440 --> 00:27:41,599 way we agree on stuff, so you get more confidence. There's also different 377 00:27:41,680 --> 00:27:45,640 users as well. You know, we've got the you know, just in 378 00:27:45,680 --> 00:27:49,319 the sort of reliability space, you've got developers who are using this information You've 379 00:27:49,359 --> 00:27:55,240 got srs that are using this information to optimize the infrastructure. You've got platform 380 00:27:55,279 --> 00:27:57,839 engineers that are using it for the same things to build tools. You know, 381 00:27:57,880 --> 00:28:02,359 that's just three users just in engineering. You know, you start to 382 00:28:02,359 --> 00:28:04,960 think that what else could we use this information for. I know of a 383 00:28:06,000 --> 00:28:08,960 company that's running at the moment the idea of taking metrics to you to do 384 00:28:10,039 --> 00:28:14,119 cost projections. Right, So take the metrics of your kuminettes poor to how 385 00:28:14,200 --> 00:28:17,240 much you might need to scale in the future. Take that metrics data. 386 00:28:17,640 --> 00:28:21,480 Don't allow people to quer it and do dashboards, but take it and do 387 00:28:21,559 --> 00:28:26,200 projections. You know, this is now into the thinops world. Well, 388 00:28:26,200 --> 00:28:30,279 this is my mid life, right, Like all too often we'd roll out 389 00:28:30,319 --> 00:28:33,559 a new feature and it would tip the system over a week later because we 390 00:28:33,559 --> 00:28:37,519 weren't looking at the additional resource impacts. And as we you know, as 391 00:28:37,519 --> 00:28:42,079 e commerce site started getting really big, we were dark launching features and running 392 00:28:42,119 --> 00:28:45,240 them on one server just to see what the load differences were. So I 393 00:28:45,319 --> 00:28:48,119 knew how much hardware to order, which just speaks to how old I am, 394 00:28:48,200 --> 00:28:51,400 right, because back then it was like, see, we love that 395 00:28:51,440 --> 00:28:53,839 feature, but we literally need thirty more servers to roll it out. Yeah, 396 00:28:53,880 --> 00:28:56,559 and then you end up with one server running on a previous version and 397 00:28:56,559 --> 00:29:03,799 then you bankrupt the company, right yeah, well yeah, we don't scare 398 00:29:03,839 --> 00:29:08,240 me Martin like dinged. But I have used that line in the boardroom where 399 00:29:08,240 --> 00:29:11,759 it's like, I think we're about a week away from bankruptcy unless we turn 400 00:29:11,839 --> 00:29:14,799 this thing off, because we're gonna we're gonna crush the whole thing, Like 401 00:29:14,839 --> 00:29:19,200 we have to think about it. Yeah, it's uh, it is it 402 00:29:19,200 --> 00:29:25,640 it? And it's that whole part about the telemetry part was the it's very 403 00:29:25,680 --> 00:29:29,000 hard to estimate the impact of a feature. Not only you don't really understand 404 00:29:29,000 --> 00:29:32,240 the resource consumption of a feature anyway, but you certainly don't know what the 405 00:29:32,400 --> 00:29:36,200 user utilization is going to be, like you're just making this stuff up, 406 00:29:36,359 --> 00:29:40,759 right, So it is only telemetry that really tells us how much is it 407 00:29:40,799 --> 00:29:44,240 being used? What's the actual impact on the systems? At least now we're 408 00:29:44,279 --> 00:29:47,640 living in a cloud where you can just flip a knob and the CFO find 409 00:29:47,640 --> 00:29:51,160 out about it in about thirty days, right, Like, you not your 410 00:29:51,240 --> 00:29:52,920 problem. You kept the system up. That's my metric. I'm up, 411 00:29:53,039 --> 00:29:59,119 Like I'm good. Well, what is this about a bill? And this 412 00:29:59,200 --> 00:30:02,160 is the the other flip side of that is what we call the nines don't 413 00:30:02,160 --> 00:30:04,359 matter if uses unhappy, it's like, yeah, ninety nine time ninet nine 414 00:30:04,400 --> 00:30:11,160 percent O time awesome, twenty second response times though, that's a good yeah, 415 00:30:11,200 --> 00:30:14,279 but yeah, this is the game we're playing, right is looking through 416 00:30:14,279 --> 00:30:18,759 all of these things. And really we came into this conversation talking about what's 417 00:30:18,799 --> 00:30:22,960 the app up to, like we're not even you know here we're talking about 418 00:30:22,000 --> 00:30:26,799 portant business cases, but mostly it's just trying to diagnose weird behavior, like 419 00:30:27,119 --> 00:30:30,599 I don't know what's going on, and I have a customer that can totally 420 00:30:30,680 --> 00:30:37,319 use this Right now, after upgrading to blazer dot Net eight from six, 421 00:30:38,559 --> 00:30:45,400 all sorts of weird things are happening the and we're still using the same services 422 00:30:45,400 --> 00:30:49,279 and stuff, just the newer versions. So it's probably just some versioning thing, 423 00:30:49,440 --> 00:30:59,359 but weird unhandled exceptions that are evading the error handlers and closing the whole 424 00:30:59,400 --> 00:31:03,519 thing and not giving us an opportunity to find out where they are. Like 425 00:31:03,920 --> 00:31:07,680 this this is going to be a game changer for them. So what I 426 00:31:07,680 --> 00:31:11,079 should mention at that point is it doesn't work well in Blazer. I'm really 427 00:31:11,119 --> 00:31:21,440 sorry Stack or is there something inherent to Blazer. So it's more to do 428 00:31:21,519 --> 00:31:25,039 with the front end side. So if you're using Blazer web assembly, you 429 00:31:25,079 --> 00:31:30,720 have a single thread and telemetry should inherently be done in the background. So 430 00:31:30,680 --> 00:31:37,759 the Blazer server has some problems with the gRPC connections and various other bits that 431 00:31:38,519 --> 00:31:42,960 we're working on it. I'm currently talking to somebody who runs one of the 432 00:31:44,000 --> 00:31:49,519 Blazer University things who's trying all this stuff out and trying to to work out 433 00:31:49,519 --> 00:31:53,359 what what that should look like for Blazer. But there's a lot of things 434 00:31:53,359 --> 00:31:57,640 that are just they're new, they're not they're not in the tracing paradigm right 435 00:31:57,680 --> 00:32:02,799 now. Durable functions is another one that has some really big interesting nuances around 436 00:32:04,119 --> 00:32:07,599 how do we propagate a context, So how do we say that this function 437 00:32:07,720 --> 00:32:12,279 execution was related to this other function execution, and how do we get that 438 00:32:12,359 --> 00:32:15,319 data between the two because it's not as easy as just past some HTTP headers. 439 00:32:16,720 --> 00:32:21,480 So you know, we've got as your service bus has similar nuances. 440 00:32:22,640 --> 00:32:27,599 But we're seeing we're seeing right now a lot of open source libraries that are 441 00:32:27,599 --> 00:32:31,319 going I need to add tracing because people want this, They want that internal 442 00:32:31,359 --> 00:32:37,319 information. They want to know how the cash server is working. They want 443 00:32:37,319 --> 00:32:40,240 to know the messaging systems and how long it takes me to post a message 444 00:32:40,240 --> 00:32:44,160 to my messaging system, not just how long it's going to take to receive 445 00:32:44,200 --> 00:32:47,400 and process it afterwards. So they're adding it. Rabbit MQ recently in the 446 00:32:47,400 --> 00:32:52,039 new version is going to have tracing as a first class citizen in there, 447 00:32:52,079 --> 00:32:54,720 and then it's going to have metrics as well. Fusion Cash was another one 448 00:32:55,160 --> 00:32:59,440 that I worked on with some people, and we've just seen loads more people 449 00:32:59,480 --> 00:33:02,279 going, yeah, tracing, I need the tracing now. So it's just 450 00:33:02,359 --> 00:33:06,079 it's really good to see, right. Yeah, we're at that kind of 451 00:33:06,079 --> 00:33:08,599 tipping point of adoption, which is yeah, very encouraging. I think as 452 00:33:08,680 --> 00:33:13,240 people get benefit out of the box, when all of these libraries they're using 453 00:33:13,400 --> 00:33:16,680 and they're already getting useful traces that tell them a lot of stuff even without 454 00:33:16,680 --> 00:33:21,559 them instrumenting their own code, that's the benefit of everyone. I mean, 455 00:33:21,599 --> 00:33:25,799 we recently instrumented the client library for the language client for Elastic Search, so 456 00:33:25,839 --> 00:33:30,319 that's a separate package you pull in if you want to do you know, 457 00:33:30,480 --> 00:33:34,359 dot net code to talk to elastic search. So we've added some instrumentation down 458 00:33:34,519 --> 00:33:37,119 in the transport layer so you can kind of see what's going on there. 459 00:33:37,599 --> 00:33:39,720 Added our own useful attribute, so out of the box, if you're using 460 00:33:39,720 --> 00:33:46,200 that library and you're using an open telemetry capable you know, collection process, 461 00:33:46,279 --> 00:33:50,960 then you immediately get that information and you can you can diagnose what's going on 462 00:33:51,000 --> 00:33:53,559 at that level. Yeah, and Aspire, you know, you had David 463 00:33:53,599 --> 00:33:58,680 on talking about Aspire. I think that is one of going to be the 464 00:33:58,720 --> 00:34:01,599 real big game changer in a option because one of the big blockers is people 465 00:34:01,680 --> 00:34:05,279 like logs, Well, I can see them in my console when I'm running, 466 00:34:05,599 --> 00:34:07,599 why would I use traces because I can't see them in my console? 467 00:34:08,320 --> 00:34:13,079 But now with Aspire, it's like you've got a nice, convenient little dashboard 468 00:34:13,079 --> 00:34:15,880 that's just running there that takes your metrics and your logs and your traces. 469 00:34:15,440 --> 00:34:19,039 And I'm seeing loads of people use that. They look at the logs thing 470 00:34:19,320 --> 00:34:21,519 and then look at the traces thing and say, well, the traces are 471 00:34:21,519 --> 00:34:24,559 way richer than the logs, So why would I use the logs? And 472 00:34:24,559 --> 00:34:29,679 I'm like, I've been saying this for five years. Thank you you're right. 473 00:34:29,760 --> 00:34:31,079 You know. The one thing I came away with while I came up 474 00:34:31,119 --> 00:34:35,480 with many things from David's show, but it was this, we're trying to 475 00:34:35,559 --> 00:34:39,079 make all the right defaults for you when you're building a cloud native app, 476 00:34:39,119 --> 00:34:43,559 and one of them is you use open telemetry, like, you use that 477 00:34:43,679 --> 00:34:47,519 level of instrumentation. So you know, it's exactly that you have to fight 478 00:34:47,599 --> 00:34:52,360 against it now to not have all that stuff in place. Yeah, it's 479 00:34:52,400 --> 00:34:53,880 the pit of success in it. You know, Yeah, it's right. 480 00:34:53,920 --> 00:34:57,480 But you know, it's an interesting point because there's also folks that are like, 481 00:34:57,519 --> 00:35:00,000 I don't know why we need this, and it's like, yeah, 482 00:35:00,360 --> 00:35:05,400 you know, because it's too many choices. You know, if you're perfect, 483 00:35:05,679 --> 00:35:07,920 if you can get everything right every time, you know, if you 484 00:35:08,039 --> 00:35:10,840 never need to be more careful because you're always careful, then fine, you 485 00:35:10,880 --> 00:35:15,280 don't need it. But I'm just not that careful. It's like unit tests. 486 00:35:15,920 --> 00:35:19,920 Spend several days trying to analyze logs in your head, and you know, 487 00:35:19,920 --> 00:35:23,199 correlate everything across any even a good logging system, and you'll you'll soon 488 00:35:23,239 --> 00:35:28,519 realize that actually a trace might have saved you a few sort of head against 489 00:35:28,559 --> 00:35:30,679 all moments. Yeah, just like unit tests. I don't need unit tests 490 00:35:30,719 --> 00:35:34,119 unit tests with people who don't trust that they can write code right for the 491 00:35:34,199 --> 00:35:37,360 first time. Yeah, that's true. I know I can't write code fight 492 00:35:37,440 --> 00:35:43,280 right the first time. A certain quote from a certain Billy Harleist comes to 493 00:35:43,280 --> 00:35:47,719 mind. Yeah, you might be addicted to code. You might be. 494 00:35:47,920 --> 00:35:53,519 Yeah, and you must suck as a code if you need two people writing 495 00:35:53,599 --> 00:35:58,840 ten lines of code for every line of code you right. Yeah, I 496 00:35:58,880 --> 00:36:01,239 mean I even speaking to an extreme case, but it's like, look, 497 00:36:01,320 --> 00:36:05,519 people make mistakes, they get tired, they work too late, they don't 498 00:36:05,599 --> 00:36:08,920 quite understand the problem. Like, all these things are real, and you 499 00:36:08,960 --> 00:36:14,199 look at all the work we're doing lately around shortening the internal cycle, the 500 00:36:14,239 --> 00:36:16,639 impacts of GitHub co pilot, they're all speaking of the same thing, more 501 00:36:17,280 --> 00:36:22,519 correct code and an aspire to me looks like this tool that says, okay, 502 00:36:22,559 --> 00:36:27,239 so you're going to be cloud native and you haven't done it for thirty 503 00:36:27,320 --> 00:36:30,440 years for some strange reason. So now we're going to give you a tool 504 00:36:30,480 --> 00:36:34,559 set, you know, a framework that leads you to that path. It's 505 00:36:34,559 --> 00:36:37,519 a distributed debugge. Yes, you know full the cloud that's you know, 506 00:36:38,360 --> 00:36:44,639 I joked in some of my talks over the years about you can't attach a 507 00:36:44,719 --> 00:36:49,079 debugger to production. You know, nobody nobody attaches their visual studio instance to 508 00:36:49,079 --> 00:36:55,440 production. Well sorry, nobody does it more than once. But you know 509 00:36:55,480 --> 00:37:00,199 nobody does that because well, for a start, it blocks all the threads, 510 00:36:59,719 --> 00:37:02,320 and that's you know, probably a bad thing. But also, now 511 00:37:02,360 --> 00:37:07,360 we've got you have to attach a debugger to fifteen different services because you've got 512 00:37:07,400 --> 00:37:10,639 five different services running three replicas and you don't know which one is going to 513 00:37:10,719 --> 00:37:15,639 hit. And you know, we're in this world now where we need that 514 00:37:15,679 --> 00:37:17,719 debugging experience. Nothing's changed. You know, we still need to debug our 515 00:37:17,800 --> 00:37:22,519 systems, but we need to debug them in an environment where they're replicated and 516 00:37:22,599 --> 00:37:27,800 they're distributed. Some of them are using the event driven systems, and you're 517 00:37:27,840 --> 00:37:31,320 like, I can't, I can't correlate these logs across an hour's worth of 518 00:37:31,360 --> 00:37:36,920 time window. And yeah, it's just it's hard. And that's where I 519 00:37:36,960 --> 00:37:40,679 think people are getting to now that they're starting to add tools and tools and 520 00:37:40,679 --> 00:37:45,519 tools on top of the old logging systems that they had to try and meet 521 00:37:45,599 --> 00:37:50,480 the demands of these distributed systems. And then they realize that actually know there's 522 00:37:50,519 --> 00:37:54,599 something that's more native that they can use instead, and it requires a paradigm 523 00:37:54,599 --> 00:37:59,960 shift. And I think that's where as an industry we're bad at acknowledge it. 524 00:38:00,119 --> 00:38:06,039 The telemetry and logging aren't the same thing. They because they're not weirdly 525 00:38:06,159 --> 00:38:08,480 enough. But you know, I've been saying for a while now that if 526 00:38:08,519 --> 00:38:13,360 you can answer all the questions that you need from the logs that you've got, 527 00:38:13,800 --> 00:38:16,679 you don't need traces. Yes, stay still fine. Yeah, but 528 00:38:16,880 --> 00:38:22,360 the question is can you and I don't think you can or you're not asking 529 00:38:22,400 --> 00:38:34,360 good questions you Yeah, I mean that's that's the number of rises above none. 530 00:38:35,519 --> 00:38:37,280 That's an old scalability problem where I found that. I found an app 531 00:38:37,280 --> 00:38:39,400 whor it's like it's a long one person was used as it was fine, 532 00:38:39,400 --> 00:38:44,599 but as soon as two did not, it's so good. That was a 533 00:38:44,679 --> 00:38:49,280 long time ago, My goodness. M I mean, I've hit that problem 534 00:38:49,360 --> 00:38:52,960 so many times when people use singletons instead of transients, and you know, 535 00:38:52,039 --> 00:38:57,679 they still stay in a singleton, and especially in multi tenant systems, and 536 00:38:57,760 --> 00:39:00,159 it's like, yeah, when we've got one tenant, it's fine, but 537 00:39:00,199 --> 00:39:02,000 as soon as we get two tenants. Everybody's got everybody else's data, and 538 00:39:02,000 --> 00:39:07,280 that's a bad thing, I think. Holy man, Yeah, easy mistakes 539 00:39:07,280 --> 00:39:12,079 to make too, right, Like, you realize we don't think about that 540 00:39:12,119 --> 00:39:15,679 anymore, that the framework is just doing that for us now most of the 541 00:39:15,719 --> 00:39:19,039 time, as long as you follow the rules. Like, we're definitely working 542 00:39:19,079 --> 00:39:22,639 on a different tier of problems now as we get you We're used to distributing 543 00:39:22,679 --> 00:39:25,880 across multiple machines and now it's you know, in different clouds, with different 544 00:39:25,920 --> 00:39:31,000 platforms. We definitely build way more complicated software. It's no question that we 545 00:39:31,079 --> 00:39:38,280 need to change the way you measure them. Uh. The debugging story is 546 00:39:38,280 --> 00:39:42,559 separate though, right, Like you just sort of casually mentioned the debugging on 547 00:39:42,599 --> 00:39:45,880 top of it. But telemetry is not necessarily for debugging, is it? 548 00:39:46,320 --> 00:39:51,639 So it can be for telemetry isn't a debugging tool, but you can use 549 00:39:51,679 --> 00:39:54,159 it for debugging. You know. One of the big things we talk about 550 00:39:54,159 --> 00:40:00,159 at the moment is the difference between three pillars of observability and just telemetry signals. 551 00:40:00,519 --> 00:40:04,840 So the three telemetry signals being logs, petrics, and traces, they 552 00:40:04,880 --> 00:40:07,280 allow you to do lots of things, and they allow you to build dashboards. 553 00:40:07,599 --> 00:40:14,280 They allow you to do monitoring build alerts, don't. You don't need 554 00:40:14,320 --> 00:40:15,719 one or one or more of these. You can use one, you can 555 00:40:15,840 --> 00:40:21,119 use three. You don't need all of the signals. But when you're starting 556 00:40:21,159 --> 00:40:23,719 to do the debugging side, you need to work out which signals you need 557 00:40:23,760 --> 00:40:28,360 to do it, and you can correlate them. Sometimes one of the key 558 00:40:28,440 --> 00:40:32,079 ones is the user request is taking too long and it turns out it's actually 559 00:40:32,079 --> 00:40:37,199 just one pod and that one pods has its CPU spinning out, and you 560 00:40:37,239 --> 00:40:43,400 look at infrastructure metrics for those things. So debugging or do you call it 561 00:40:43,440 --> 00:40:49,559 debugging working out what the problem is in production the root cause analysis. Yeah, 562 00:40:49,960 --> 00:40:53,639 Debugging to me speaks to I am fixing code often because I'm looking in 563 00:40:53,719 --> 00:40:59,400 an executing environment for what's going on within that code. I could see having 564 00:40:59,400 --> 00:41:02,039 the telemetry sitting beside me because we're seeing a behavior in production, we're not 565 00:41:02,119 --> 00:41:06,519 quite sure what it is, and so now I set up a debugging environment 566 00:41:06,559 --> 00:41:09,119 to try and capture that. And I suppose so I've been playing with this 567 00:41:09,199 --> 00:41:15,800 narrative, and let's try this, which is I think tracing is kind of 568 00:41:15,800 --> 00:41:22,280 like conditional breakpoints that you can use with a time machine. Now, when 569 00:41:22,320 --> 00:41:27,840 I discovered conditional great breakpoints in dotnet, it was like a revelation. When 570 00:41:27,840 --> 00:41:30,039 you're running a SPA app on the front end and it's going through this line 571 00:41:30,039 --> 00:41:32,280 of code, and you want to know when it hits this line of code. 572 00:41:32,519 --> 00:41:35,960 But when it comes from this end point is when I want to be 573 00:41:36,000 --> 00:41:37,320 able to hit it, because it hits sixteen times and I don't want to 574 00:41:37,320 --> 00:41:40,119 have to hit play every time, and I've got multiple threads going on. 575 00:41:42,039 --> 00:41:45,360 If you think about that, but being able to do it in production and 576 00:41:45,400 --> 00:41:52,119 then say what happened yesterday when this person hit this end point for this particular 577 00:41:52,159 --> 00:41:59,039 method or this particular database call that I think is that this revelation of you, 578 00:41:59,079 --> 00:42:02,880 now you are doing deep You're you're using that telemetry from production to do 579 00:42:04,000 --> 00:42:08,119 debugging, yeah, to construct an executing environment to recreate the problem in production. 580 00:42:08,760 --> 00:42:12,199 Like it gives you the cues to say, we need to be in 581 00:42:12,239 --> 00:42:15,599 this state to be able to see this bug. Yeah, to be able 582 00:42:15,639 --> 00:42:19,960 to see those constraints. Because you still can't set a break point in production. 583 00:42:20,880 --> 00:42:27,239 Well you can, you just won't have a job after it. Story 584 00:42:27,280 --> 00:42:30,360 Steve, I stepped down you there yeah, no, I was just yeah, 585 00:42:30,599 --> 00:42:32,039 I'm totally on board with you know, not this point there. And 586 00:42:32,039 --> 00:42:37,280 I think it's it's the ability to apply constraints to the data that you have 587 00:42:37,440 --> 00:42:40,920 that would be very difficult with just pure logs, where you want to set 588 00:42:40,960 --> 00:42:45,400 up this series of criteria around I want to look at this coming from this 589 00:42:45,559 --> 00:42:50,400 in this region, you know, deployed on this particular piece of infrastructure, 590 00:42:50,440 --> 00:42:54,679 and then see a waterfall visually of Okay, what did that flow through in 591 00:42:54,719 --> 00:43:00,760 the code? Where are those you know, lines on this waterfall wider taking 592 00:43:00,800 --> 00:43:06,119 longer? And then you narrow down, Okay, well that's this block of 593 00:43:06,159 --> 00:43:09,000 code or this library. That's where I need to go and look to try 594 00:43:09,039 --> 00:43:13,519 and find out why that's taking so long that piece of the puzzle with these 595 00:43:13,559 --> 00:43:20,840 given sets of criteria applied to it. And that's something I think only only 596 00:43:20,920 --> 00:43:23,159 well defined tracing can can really give you. You could, you could sort 597 00:43:23,159 --> 00:43:27,320 of get there with logs, but you you would take a lot longer to 598 00:43:27,400 --> 00:43:30,280 find there. I think, yeah, there's always a question of do I 599 00:43:30,360 --> 00:43:36,840 understand with any given log set, it's like these six logs, where is 600 00:43:36,920 --> 00:43:39,679 the trends this particular transaction across these logs, like we've always had a tough 601 00:43:39,719 --> 00:43:44,079 time joining that stuff up and tell them she does a better jar with that, 602 00:43:44,639 --> 00:43:47,039 yeah, and then relating that back to infrastructure because you know, we 603 00:43:47,119 --> 00:43:51,599 don't run our code in isolation. Infrastructure is a thing that we use. 604 00:43:52,719 --> 00:43:55,800 So yes, it ran slower. But was it running slow because that Kubernetti's 605 00:43:55,840 --> 00:44:00,599 node was overloaded? Was it running slow because we just switched slots in service? 606 00:44:00,280 --> 00:44:06,679 You know though that infrastructure stuff is just as important. Yeah, it's 607 00:44:06,920 --> 00:44:08,119 you know again, I've spent a long time on the ops side of this 608 00:44:08,159 --> 00:44:12,480 because why is this slow? And it's like it's your hardware, No, 609 00:44:12,599 --> 00:44:15,920 it's your crappy software. Like you go back and forth on that a fair 610 00:44:15,039 --> 00:44:22,079 bit. I have had a situation where we had a hundred based tea hub 611 00:44:22,320 --> 00:44:25,920 in the in the network we'd forgotten about and literally were throttling all the network 612 00:44:25,960 --> 00:44:30,840 and traffic. So you know, it looked like the database was slow, 613 00:44:30,840 --> 00:44:32,480 and it's like, literally it took a long time for the data to come 614 00:44:32,480 --> 00:44:37,920 and go from that because this network mistake was there. Like, those things 615 00:44:37,960 --> 00:44:40,960 happen, and the trick is to not be mean about it. Like everybody 616 00:44:42,000 --> 00:44:45,039 sit on the same side of the table in the boardroom with the projector on 617 00:44:45,519 --> 00:44:49,559 going. What is this data telling us? And that's correlation? Yeah, 618 00:44:49,599 --> 00:44:52,360 you know, that's the idea of you know, the ops people are looking 619 00:44:52,360 --> 00:44:55,119 at their infrastructure dashboards and you know there's a there's a thing here, and 620 00:44:55,159 --> 00:45:00,079 then the developers are looking at their log aggregation dashboard that show them some stuff 621 00:45:00,079 --> 00:45:05,079 and they've got some graphs that they can It's like, whoa these two graphs 622 00:45:05,079 --> 00:45:08,320 they go up at the same time. Ah, maybe there is something here. 623 00:45:08,480 --> 00:45:12,280 Is there a knob here that we can wiggle, like, you know, 624 00:45:13,719 --> 00:45:16,039 just kick it out? Can we change that number for anything? Like 625 00:45:16,679 --> 00:45:20,599 when we do this, does it make a difference. So at least you 626 00:45:20,599 --> 00:45:23,679 have a sense that, yeah, there's a there's a constraint here and it's 627 00:45:23,760 --> 00:45:28,639 related to this value, and if we modify it, we can improve it 628 00:45:28,639 --> 00:45:31,039 in some way. Convincing that we have to buy a new hardware is challenging. 629 00:45:31,800 --> 00:45:36,960 And again Cloud took a lot of this away because it costs you nothing 630 00:45:37,039 --> 00:45:40,800 to turn that up for a test and then turn it back down again like 631 00:45:40,960 --> 00:45:45,920 a buck. It's just you solve a lot of problems just by doing those 632 00:45:45,000 --> 00:45:50,880 tests. Yeah, I just run a workshop on Kubernettes for some attendees, 633 00:45:50,920 --> 00:45:54,719 and I span up a Cubernettes cluster for every one of the attendees for two 634 00:45:54,800 --> 00:46:01,360 days. And it costs me I think four pounds per attendee to run a 635 00:46:01,480 --> 00:46:07,199 three note Kubernettes cluster in a KS for two days. I mean, can 636 00:46:07,239 --> 00:46:09,519 you imagine if you were having to do that on premise and ship in all 637 00:46:09,559 --> 00:46:15,800 that hardware three three hundreds just for each intending and a rack whirring in the 638 00:46:15,800 --> 00:46:19,199 background, but with hal blinky lights, So that'd be a thing. Yeah, 639 00:46:19,280 --> 00:46:21,280 yeah, we'd have blinky lights. But you know you can get a 640 00:46:21,440 --> 00:46:22,719 get a bunch of r G B L e eds, put them in a 641 00:46:22,760 --> 00:46:27,960 box just called the computer right anytime the wire goes back into the cloud. 642 00:46:30,239 --> 00:46:31,320 Might do that for the next workshop. It's like, yeah, you're all 643 00:46:31,360 --> 00:46:36,679 running those blinking lights in the corner. That one it's off. That's Carl 644 00:46:39,440 --> 00:46:46,639 Carl oh Man. What if we what have we not talked about here? 645 00:46:46,719 --> 00:46:51,800 Because there's a lot to doing a good job here. What are the learning 646 00:46:51,920 --> 00:46:54,159 ingredients? Like how do people really get You know, it's quick to install, 647 00:46:54,280 --> 00:46:59,159 but do you ten ten lines of code? You're up and running but 648 00:46:59,360 --> 00:47:04,760 there is some things you need to know. I think start with aspire. 649 00:47:05,039 --> 00:47:07,880 You know, start with a local dashboard. Start by turning some things on 650 00:47:07,960 --> 00:47:13,679 locally in your local environment, development environment, and see what some of these 651 00:47:13,719 --> 00:47:19,800 traces look like and see whether they're useful before you start thinking about where I'm 652 00:47:19,800 --> 00:47:22,039 going to put them, what observability back? In? What scale do I 653 00:47:22,079 --> 00:47:24,360 need? Am I going to use open source and host it myself? Am 654 00:47:24,400 --> 00:47:27,639 I going to go with a SaaS vender? Am I going to go one 655 00:47:27,679 --> 00:47:31,800 in built into my cloud? Use the spire? That's that's what the real 656 00:47:31,920 --> 00:47:36,639 cool thing about it's available as a container. Now that you can just run 657 00:47:36,719 --> 00:47:39,920 up locally. You don't need to use the DCP and all the other bits 658 00:47:39,960 --> 00:47:44,840 that go with it. You can just run it in a container and just 659 00:47:44,960 --> 00:47:49,079 experiment with it. What data do you see? Is it going to help 660 00:47:49,159 --> 00:47:52,880 you? You know, the value of the value to work ratio should be 661 00:47:52,920 --> 00:47:57,960 really small anyway, in like ten minds of code. Great, but start 662 00:47:57,960 --> 00:48:00,440 with something like that locally because you don't need to choose a vendor. I 663 00:48:00,840 --> 00:48:05,360 equate this to you. Remember when they brought an eye logger, And the 664 00:48:05,400 --> 00:48:08,559 great thing about eyelogger is it saved six months on every project because the first 665 00:48:08,559 --> 00:48:15,639 six months of any project was deciding between n log and love pnette and the 666 00:48:15,840 --> 00:48:20,519 arguments. The arguments that people had. I mean, that was the true 667 00:48:20,519 --> 00:48:23,599 flame Wars, you know. But they brought an eye logger, and the 668 00:48:23,639 --> 00:48:29,320 problem was that you needed to make that decision really far up from because if 669 00:48:29,320 --> 00:48:31,639 you didn't, then you'd have to retrofit it across every all of your code 670 00:48:31,840 --> 00:48:37,800 develop everybody this one. They brought an eye logger and they were like, 671 00:48:37,840 --> 00:48:39,559 great, yeah, everybody use ilogger. It'll just send it through eyelogger and 672 00:48:39,599 --> 00:48:43,599 then you'll choose your vendor. And I think open celemetry is the same thing 673 00:48:44,519 --> 00:48:47,039 is we've got a gimme. Now. It should be in everything, whether 674 00:48:47,039 --> 00:48:50,639 you're to wire it up or not. It costs you nothing to have it 675 00:48:50,679 --> 00:48:52,039 there, and when you do wire it out, mental help. Yeah, 676 00:48:52,079 --> 00:48:54,559 and that's the place to start. Well, And I like this tipping point 677 00:48:54,599 --> 00:48:59,159 angle that Steve brought up, which is that and more and more vendors see 678 00:48:59,199 --> 00:49:02,400 it that way. So the information you're getting out of open Columbatry just keeps 679 00:49:02,400 --> 00:49:07,719 getting better. Yeah, this whole thing works best as more and more vendors 680 00:49:07,719 --> 00:49:09,280 get on board. So that's the library authors, it's the you know, 681 00:49:09,360 --> 00:49:15,400 the various different language run times, it's the open Telemetry team. It's people 682 00:49:15,480 --> 00:49:20,599 within your organization, engineers within your organization buying into the fact that, okay, 683 00:49:20,639 --> 00:49:22,360 we've got this. I mean in dot Net it's super easy because we 684 00:49:22,400 --> 00:49:27,119 do have activity built in. It's it's there. If you're using dot net 685 00:49:27,159 --> 00:49:30,239 modern dot net, you've got the libraries, you've got the APIs. You 686 00:49:30,280 --> 00:49:34,760 need just to turn this stuff on and start instrumenting code. But as all 687 00:49:34,800 --> 00:49:38,159 of these people shift towards that general direction, I think that's where why open 688 00:49:38,159 --> 00:49:43,679 telemetry now is really starting to look like a good solution for everyone. What's 689 00:49:43,719 --> 00:49:47,320 the app Insights story? Should we just rip it out? And so Application 690 00:49:47,400 --> 00:49:52,239 Insights is currently redeveloping all of their SDKs to use activity source and activities. 691 00:49:52,239 --> 00:49:57,639 So it's not wow, I'm getting those now saying you're going to have to 692 00:49:57,719 --> 00:50:01,000 change this soon. So it shows that you Microsoft's listening to the open telemetry 693 00:50:01,000 --> 00:50:06,039 story. I mean, they had the experimental flags in the as your SDK 694 00:50:06,440 --> 00:50:07,599 last year, I think, and I think those are being removed now. 695 00:50:07,679 --> 00:50:12,920 So all the AZ your SDK is anything that you use from Cosmos to service 696 00:50:13,000 --> 00:50:16,639 bos if Angrid. All of those things will be instrumented with activities which means 697 00:50:16,679 --> 00:50:22,679 they're compatible with open celametry to send it and as your monitor. They've got 698 00:50:22,719 --> 00:50:27,960 their own exporter. Now they don't accept OTLP yet. Whether they will, 699 00:50:27,960 --> 00:50:30,840 I don't know. I don't work on that team, but they do have 700 00:50:30,880 --> 00:50:37,079 a distro so that you can do add as your monitor exporter and then you 701 00:50:37,119 --> 00:50:39,599 can just use open telemetry and decide later away're going to send it or just 702 00:50:39,639 --> 00:50:43,800 add the one line of code that says add to a your monitor and it 703 00:50:43,840 --> 00:50:47,079 will appear in app insites. Everybody's doing it. And to go back to 704 00:50:47,119 --> 00:50:52,880 this Blazer thing, do you recall hearing any kind of timeline as to when 705 00:50:53,880 --> 00:51:01,480 things will be more copestic with tracing and everything else that's going on. So 706 00:51:01,639 --> 00:51:07,599 I had a long conversation with the other Steve NDC London about exactly that, 707 00:51:07,639 --> 00:51:19,639 the more famous Steve. So they're looking at it, they are aware of 708 00:51:19,679 --> 00:51:22,440 the issues that are coming up, and they are you know, wanting to 709 00:51:22,480 --> 00:51:27,480 solve them. So I would imagine that yes, they will be solved soon 710 00:51:27,760 --> 00:51:30,639 and you can do it. It's just it's not as easy as ten lines 711 00:51:30,639 --> 00:51:32,039 of code. You know, you've got to go through a few more hoops, 712 00:51:32,159 --> 00:51:36,559 especially with the whole render mode thing. If you're using dynamic render modes, 713 00:51:36,639 --> 00:51:40,639 You've got some other problems that have to get solved, first state problems 714 00:51:40,639 --> 00:51:46,400 and things. Something I'm currently working on. But I'm hopeful. I'm hopeful 715 00:51:46,440 --> 00:51:51,559 for a future where this is the default. You know, this is well, 716 00:51:51,599 --> 00:51:54,159 I just add those ten lines of code, or even just add one 717 00:51:54,159 --> 00:51:58,679 line of code when we've worked on the SDKs a little bit more, and 718 00:51:58,800 --> 00:52:02,239 just add the one thing that's says send my stuff, and you know it's 719 00:52:02,239 --> 00:52:09,719 a what is it services? Don't let me debug my things? You know 720 00:52:09,800 --> 00:52:15,239 that that kind of simplicity just becomes the new cargo cult if you like, 721 00:52:15,639 --> 00:52:19,840 I don't just cargo cult thing. I well, I do cargo cult thing, 722 00:52:19,840 --> 00:52:22,280 but it's the new thing I cago cult. Well, guys, great 723 00:52:22,320 --> 00:52:27,880 conversation, Thanks so much for coming on absolutely and what's next. Let's send 724 00:52:27,920 --> 00:52:34,480 your inbox Marten. So I'm doing a load of conferences. I'm doing some 725 00:52:34,519 --> 00:52:38,440 meetups around the UK. I'll be at Cubcon, the Kubernettes Cloud Native Conference 726 00:52:38,920 --> 00:52:45,079 in match so but yeah, I am flying around the world spreading the good 727 00:52:45,079 --> 00:52:49,880 word about this sort of stuff. How about you you? Yeah, conferences 728 00:52:50,000 --> 00:52:52,880 coming up, so I'll be in Sweet Sweden next week for Sweet Tug, 729 00:52:52,960 --> 00:52:55,400 which will be in the past by the time this this one goes out. 730 00:52:57,519 --> 00:53:01,679 And then yeah, day to day we're going to be focused on where Elastic 731 00:53:01,719 --> 00:53:06,280 and the engineers can contribute to open telemetry. Really, I'm really starting to 732 00:53:06,320 --> 00:53:09,800 poke around in the open telemetry sd K repays and think about how we can 733 00:53:10,280 --> 00:53:14,320 make that experience good for good for everyone. Say, yeah, I think 734 00:53:14,320 --> 00:53:16,599 that will keep you busy for a while yet. Well, this is all 735 00:53:16,639 --> 00:53:22,159 great news. Thanks again, and I'll catch up with you at some conference 736 00:53:22,199 --> 00:53:27,559 somewhere. I'm sure, sure, all right, We'll see you next time 737 00:53:27,679 --> 00:53:51,719 on dot net rocks. Dot net Rocks is brought to you by Franklin's Net 738 00:53:51,800 --> 00:53:55,679 and produced by Pop Studios, a full service audio, video and post production 739 00:53:55,800 --> 00:54:00,440 facility located physically in New London, Connecticut, and of course in the cloud 740 00:54:01,039 --> 00:54:07,119 online at pwop dot com. Visit our website at d O T N E 741 00:54:07,199 --> 00:54:12,039 t R O c k S dot com for RSS feeds, downloads, mobile 742 00:54:12,079 --> 00:54:15,960 apps, comments, and access to the full archives. Going back to show 743 00:54:15,039 --> 00:54:20,599 number one recorded in September two thousand and two. And make sure you check 744 00:54:20,639 --> 00:54:23,679 out our sponsors. They keep us in business. Now go write some code. 745 00:54:24,000 --> 00:54:35,280 See you next time. You got javans and