WEBVTT 1 00:00:01.000 --> 00:00:04.759 How'd you like to listen to dot net Rocks with no ads? Easy? 2 00:00:05.320 --> 00:00:09.400 Become a patron for just five dollars a month. You get access to a 3 00:00:09.480 --> 00:00:14.199 private RSS feed where all the shows have no ads. Twenty dollars a month, 4 00:00:14.240 --> 00:00:18.399 we'll get you that and a special dot net Rocks patron mug. Sign 5 00:00:18.480 --> 00:00:24.079 up now at Patreon dot dot net rocks dot com. Hey there, this 6 00:00:24.120 --> 00:00:28.440 is Jeff Fritz, the Purple Blazer guy from Microsoft, letting you in on 7 00:00:28.480 --> 00:00:32.640 a little secret about my friend Carl Franklin. You know, the guy who 8 00:00:32.679 --> 00:00:37.399 started dot net Rocks, the first podcast about dot net in two thousand and 9 00:00:37.399 --> 00:00:43.000 two, The guy who's been teaching Blazer on YouTube since twenty twenty. Yeah, 10 00:00:43.039 --> 00:00:47.600 that Carl Franklin. Well, Carl's joined up with the folks from Code 11 00:00:47.600 --> 00:00:52.399 in a castle to teach a week long hands on Blazer class at Are you 12 00:00:52.399 --> 00:00:58.439 ready to get this? At a castle slash villa in Tuscany. It's sort 13 00:00:58.439 --> 00:01:03.920 of a luxury vacation. It's Blazer learning built in. Carl's calling it the 14 00:01:03.000 --> 00:01:10.000 Blazer Master Class. You'll learn Blazer from the ground up, finishing the week 15 00:01:10.040 --> 00:01:15.680 with the ability to build and deploy Blazer applications. Since the training happens for 16 00:01:15.719 --> 00:01:19.920 only four hours in the morning over six days, you can bring your significant 17 00:01:19.959 --> 00:01:26.040 other your partner with you and you should right This part of Italy is absolutely 18 00:01:26.079 --> 00:01:30.560 beautiful. There's so much to see and do and in. Larry and Marco 19 00:01:30.680 --> 00:01:34.799 from code In to Castle are organizing daily activities both at the castle and in 20 00:01:34.840 --> 00:01:38.959 the area. The castle is in the Marema, a less touristed region of 21 00:01:40.079 --> 00:01:45.879 Tuscany, offering both classic Tuscan hill country as well as easy access to the 22 00:01:45.879 --> 00:01:52.239 Etruscan Riviera. With sublime local food, wine and olive oil around every corner. 23 00:01:52.519 --> 00:01:56.400 Breakfast is included every day. There will be two communal dinners at the 24 00:01:56.439 --> 00:02:00.680 castle book ending the experience, and most other meals and all activities are included. 25 00:02:01.480 --> 00:02:07.319 And did I mention you'll learn Blazer in person from Carl Franklin Listen. 26 00:02:07.719 --> 00:02:12.199 Space is limited and for very good reason. This is quality training in a 27 00:02:12.240 --> 00:02:19.759 beautiful setting. Go to code Inacastle dot com slash Blazer twenty twenty three that's 28 00:02:19.879 --> 00:02:25.159 bla z O R two zero two three to take advantage of this amazing opportunity 29 00:02:25.360 --> 00:02:31.240 to join Carl in Tuscany for an unforgettable week of la dolce Vita while advancing 30 00:02:31.240 --> 00:02:38.879 your programming skills in this important new technology. After building software for a while, 31 00:02:38.960 --> 00:02:43.560 you know it's only a matter of time before you see an HTTP timeout 32 00:02:43.639 --> 00:02:46.800 or a database deadlock. In software, it's not a case of if things 33 00:02:46.879 --> 00:02:52.199 fail, but a case of when one mishap like this and valuable data is 34 00:02:52.240 --> 00:02:57.400 lost forever. And these failures occur all the time, but it doesn't have 35 00:02:57.439 --> 00:03:00.560 to be this way. Introducing n service bus, the ultimate tool to build 36 00:03:00.639 --> 00:03:07.759 robust and reliable systems that can handle failures gracefully, maintain high availability, and 37 00:03:07.919 --> 00:03:12.479 scale to meet growing demand. For more than fifteen years, end service bus 38 00:03:12.479 --> 00:03:16.360 has been trusted to run mission critical systems that must not go down or lose 39 00:03:16.479 --> 00:03:22.199 any data ever, and now you can try it for yourself. End service 40 00:03:22.240 --> 00:03:25.680 Bus integrates seamlessly with your dot net applications and could be hosted on premises or 41 00:03:25.719 --> 00:03:30.919 in the cloud. Say goodbye to loss data and system failures and say hello 42 00:03:30.960 --> 00:03:35.879 to a better, more reliable way of building distributed systems. Try end service 43 00:03:35.879 --> 00:03:40.439 bus today by heading over to go dot particular dot net slash dot net rocks 44 00:03:40.800 --> 00:04:00.879 and start building better systems with asynchronous messaging using end service bus Hey, Antwerp. 45 00:04:00.240 --> 00:04:11.319 It's dot Rock Holy crap, there must be fifty thousand people here. 46 00:04:11.479 --> 00:04:13.719 Yeah, who knew we were in a stadium? I know, right, 47 00:04:13.759 --> 00:04:15.040 we are in a state. We're actually in a movie theater, which is 48 00:04:15.079 --> 00:04:18.240 cool. It is cool. The last time we did a dot Net Rocks 49 00:04:18.240 --> 00:04:23.480 in a movie theater I think was in Sofia, Bulgaria. Oh yeah, 50 00:04:23.519 --> 00:04:27.160 do you remember that? Yeah, a few years that was dev reached every 51 00:04:27.199 --> 00:04:30.319 And the funny thing was the Bulgarians thought it would be funny if Richard and 52 00:04:30.360 --> 00:04:35.639 I announced the names of the winners, the Bulgarian winners of the swag at 53 00:04:35.680 --> 00:04:41.480 the end of the show, and that was funny for them, for them, 54 00:04:43.399 --> 00:04:50.560 Bulgarian names need to buy a bow Okay, it lacks where snudge and 55 00:04:50.720 --> 00:04:55.120 they were just laughing their butts off. That's you. Oh you were there? 56 00:04:55.160 --> 00:04:59.480 Okay, all right, well, uh it's we're glad to be here 57 00:05:00.079 --> 00:05:02.959 obviously. Yeah, last day of the show. Fun to do a live 58 00:05:03.000 --> 00:05:06.800 show. Yeah, it's a live show. We have some good stuff coming 59 00:05:06.879 --> 00:05:11.199 up here. Layla Bougria is here and we'll be talking to her in a 60 00:05:11.240 --> 00:05:14.319 minute. But first we have to do this little thing called better know a 61 00:05:14.360 --> 00:05:24.759 framework roll the crazy music. All right, buddy, what do you got? 62 00:05:25.319 --> 00:05:28.959 Well, I saw this come across Twitter. It's a tweet and while 63 00:05:28.959 --> 00:05:32.519 I'm linking to it, Boston Dynamics. You know who those crazy people are, 64 00:05:32.399 --> 00:05:35.839 the guys with the robots. They make the robots that dance, they 65 00:05:35.879 --> 00:05:40.600 do backflips. Now, yeah, a little parkour with robots and stuff. 66 00:05:40.639 --> 00:05:44.160 Yeah, they used to have a little uncanny heathered robots, robots that were 67 00:05:44.160 --> 00:05:47.279 tethered, and then they got gas powered robots. Now they're battery driven and 68 00:05:47.399 --> 00:05:51.160 they put out videos every once in a while of things that look like animals, 69 00:05:51.199 --> 00:05:57.720 like cougars and dogs. Well, anyway, they've put chat GPT into 70 00:05:57.720 --> 00:06:01.000 a robot and now you can talk to it and it will talk to you 71 00:06:01.079 --> 00:06:05.120 back. And so there's a tweet about that. It was from April, 72 00:06:05.240 --> 00:06:10.720 but I thought it was so cool and a little bit scary. But they're 73 00:06:10.759 --> 00:06:14.720 asking a questions like, you know, are you function It's like data, 74 00:06:14.800 --> 00:06:16.600 are you functioning within normal parameters? You know? And it would say, 75 00:06:16.720 --> 00:06:20.160 yes, you know, I have this blah blah blah. What my levels 76 00:06:20.199 --> 00:06:25.199 are? You know these kinds of things, my battery level? How many 77 00:06:25.560 --> 00:06:30.360 what did it say? How many interactions in your last mission, and it 78 00:06:30.399 --> 00:06:33.920 will tell you about its last mission and where it went. Just as they 79 00:06:33.959 --> 00:06:38.040 used the word mission, I don't even know if it was mission, but 80 00:06:38.120 --> 00:06:42.519 that an extermination mission. It could have been just wondering. It's pretty scary, 81 00:06:42.560 --> 00:06:45.399 but it's cool. That's awesome. So that's what I got it. 82 00:06:45.600 --> 00:06:48.480 It's a tweet with a video. Okay, yeah, Boston Dynamis videos are 83 00:06:48.480 --> 00:06:51.959 always amazing, fun. Good one who's talking to us? Richard grabbed a 84 00:06:51.959 --> 00:06:57.399 comment off a show seventeen fifty three. That's the one we did with Mika 85 00:06:57.480 --> 00:07:02.759 about Visual Studio twenty twenty two productivity back in August last year, and Mark 86 00:07:02.759 --> 00:07:05.240 Wansel had this great comment. Mark has a lot of great common spells, 87 00:07:05.240 --> 00:07:10.079 and he says one of the things that Mika noted a few times was telemetry. 88 00:07:11.040 --> 00:07:14.000 I think that this would be an interesting show topic. There's a popular 89 00:07:14.079 --> 00:07:17.759 GitHub project called open telemetry that might be a good starting space. There seems 90 00:07:17.759 --> 00:07:20.759 to be an art of telemetry. What to collect, how much, performance 91 00:07:20.800 --> 00:07:26.759 considerations and primacy considerations. Check out open telemetry dot io. What do you 92 00:07:26.759 --> 00:07:30.639 think that that's not a good idea? Not want to do that? Noah, 93 00:07:30.920 --> 00:07:32.240 we won't do that show. Yeah, sorry, Mark, I wish 94 00:07:32.240 --> 00:07:34.680 we could help you, but we can't help you. Yeah, but I 95 00:07:34.720 --> 00:07:38.600 will send you a copy of music Cobi And if you'd like copy of music 96 00:07:38.759 --> 00:07:41.480 by I read a comment on the website at dot net rocks dot com or 97 00:07:41.480 --> 00:07:43.959 on the facebooks. We publish every show there, and if you comment there 98 00:07:43.959 --> 00:07:45.639 and ever reading on the show, we'll say, do you a copy of 99 00:07:45.720 --> 00:07:48.160 music by? And you should follow us on Twitter. But the real fun 100 00:07:48.279 --> 00:07:54.120 happens on Mastodon. I'm Masodon, I'm at Carl Franklin at tech Hubs Social, 101 00:07:54.240 --> 00:07:58.120 and I'm Rich Campbell at masodonda Social. Send us a two let us 102 00:07:58.160 --> 00:08:03.759 know you're listening, and that brings us to our show on open telemetry. 103 00:08:03.839 --> 00:08:07.839 What yeah, how sorry, that's the topic. Lila Bougrie is here. 104 00:08:07.920 --> 00:08:11.800 She's a software engineer with over fifteen years of experience in the dot net space 105 00:08:11.240 --> 00:08:16.959 and currently works in particular software where they build nd service bus. Maybe use 106 00:08:16.000 --> 00:08:18.720 it maybe forard a bit, certainly don't know the show a few times. 107 00:08:18.759 --> 00:08:24.120 Yeah, She's a Microsoft MVP and frequent speaker conferences and interspare time. She 108 00:08:24.160 --> 00:08:28.720 loves to knit and crochet. Welcome, thank you. How about Lila, 109 00:08:28.879 --> 00:08:35.200 huh, we were lying when we said we weren't going to do a show 110 00:08:35.240 --> 00:08:39.320 about open telemetry. That's kind of why we read that. Yeah, yeah, 111 00:08:39.320 --> 00:08:41.759 I figured if I've got a comment literally somebody asked her for to open 112 00:08:41.799 --> 00:08:43.879 telemetry, now is the time to read it. So, Lila, what's 113 00:08:43.879 --> 00:08:52.440 the elevator pitch for open telemetry? Well, the elevator pitch. So we've 114 00:08:52.679 --> 00:08:58.320 we've had multiple telemetry signals in software for years, right, the oldest one 115 00:08:58.320 --> 00:09:03.000 being logs. I think we all know logs. We've also mostly used metrics. 116 00:09:03.480 --> 00:09:07.399 Maybe I guess distributed tracing is a newest signal, but I would say 117 00:09:07.440 --> 00:09:13.960 that the elevator pitch for open telemetry is really correlating them all together. And 118 00:09:13.080 --> 00:09:18.440 that's what makes it really interesting for me at least, because you know, 119 00:09:18.480 --> 00:09:22.279 each signal has its own value. And we've also been logging for years, 120 00:09:22.320 --> 00:09:24.279 and even though we can see, like you know, tracing might be the 121 00:09:24.360 --> 00:09:28.440 better option today, but then we still have all of those logs we don't 122 00:09:28.480 --> 00:09:33.639 want to go rewrite and higher applications though. We do have log providers so 123 00:09:33.759 --> 00:09:37.080 you can plug in however you want to do your logging, but that's not 124 00:09:37.159 --> 00:09:41.480 really that's taking the same source and putting it in different places, isn't it. 125 00:09:41.519 --> 00:09:46.480 You're talking about ingesting different sources into one place exactly, Yeah, and 126 00:09:46.519 --> 00:09:52.399 then being able to connect it. So basically, imagine, you know, 127 00:09:52.480 --> 00:09:58.480 over the rainbow that you get lowered and there's a metric that looks out of 128 00:09:58.519 --> 00:10:01.240 whack, and it's like, okay, something is up. What is up? 129 00:10:01.519 --> 00:10:05.200 I don't know, just the metric is out of whack. So you 130 00:10:05.279 --> 00:10:09.080 have to go figure out how to do that, and that's usually a challenge 131 00:10:09.159 --> 00:10:13.559 because you then have to go figure out, Okay, how do I connect 132 00:10:13.559 --> 00:10:16.240 that to those other signals that I have. But what if you could look 133 00:10:16.240 --> 00:10:20.320 at the metric and say, okay, I can see that it's correlated to 134 00:10:20.759 --> 00:10:24.000 these traces, and then go look at that, and then the traces would 135 00:10:24.000 --> 00:10:28.799 also be connected to the logs and you would be able to basically, yeah, 136 00:10:28.840 --> 00:10:33.399 paint that entire picture of what's going on, and you wouldn't be losing 137 00:10:33.440 --> 00:10:37.919 all of that time doing that thing. Manually sounds like a Wikipedia rabbit hole, 138 00:10:37.080 --> 00:10:41.639 right. One thing leads you to another to another. So I'm trying 139 00:10:41.639 --> 00:10:46.039 to distinguish between all these different things. I mean, logging means a particular 140 00:10:46.039 --> 00:10:54.679 product like sequel server spitting out logs about how it's functioning, possibly versus metrics 141 00:10:54.720 --> 00:10:58.240 being more things like the state of a server, like the hard drives pinned 142 00:10:58.279 --> 00:11:03.759 or running low memory. And then when I think about traces, I think 143 00:11:03.799 --> 00:11:09.360 about there's tools that specifically are about following a button click from the client to 144 00:11:09.559 --> 00:11:15.759 the server to the database and back again. Right. Yeah, I like 145 00:11:15.799 --> 00:11:18.679 to think of that like following a business transaction, right, right, like 146 00:11:18.759 --> 00:11:22.720 the workflow. Right. So it's true, Yeah, that's definitely true. 147 00:11:22.720 --> 00:11:26.120 But at the same time, we've also been logging in our applications a lot, 148 00:11:26.759 --> 00:11:31.720 and that's still then connects together. This is as a developer writing code 149 00:11:31.759 --> 00:11:37.039 to push messages onto a log right, So that's usually I'm a developer, 150 00:11:37.120 --> 00:11:43.000 right, So I'm always looking at things from the application perspective, and how 151 00:11:43.000 --> 00:11:46.639 do I make this application observable? And there's multiple signals, and you would 152 00:11:46.679 --> 00:11:52.960 choose each individual signal based on whatever your scenario is. It could even be 153 00:11:52.080 --> 00:11:58.600 that for a specific scenario you might come to the conclusion that using multiple signals 154 00:11:58.679 --> 00:12:03.320 might be useful. So for example, let's say that a failure occurs, 155 00:12:03.559 --> 00:12:07.200 right, you want to keep track of that and have a trace that reflects 156 00:12:07.240 --> 00:12:11.480 that failure. But you might also want to have that reflected emetrics so you 157 00:12:11.519 --> 00:12:16.240 can do the alert de alerting and all of that. Yeah, So that 158 00:12:16.480 --> 00:12:18.200 meaning to trip it up in through the system means to say there was a 159 00:12:18.240 --> 00:12:22.679 failure that occurred as well as what shows up in the law. Right now, 160 00:12:22.960 --> 00:12:28.559 there's a lot of third party products out there that do telemetry. Or 161 00:12:28.799 --> 00:12:33.720 does open Telemetry allow you to use those as sources and then pull them together, 162 00:12:33.840 --> 00:12:37.919 or does open Telemetry have its own things that you can plug in or 163 00:12:37.960 --> 00:12:41.799 both? It's both, and that's a good part because I think if we 164 00:12:41.840 --> 00:12:46.720 look at our applications and we want to make them observable, there's a lot 165 00:12:46.759 --> 00:12:50.799 that we can do. But I think with the Open Telemetry project, and 166 00:12:50.919 --> 00:12:56.200 also like all of the effort that the entire community has basically put into this, 167 00:12:56.559 --> 00:13:00.120 they've made it easy for us, right because you could basically say what 168 00:13:00.240 --> 00:13:03.960 second are using, Oh, I'm using a speed on at core and I'm 169 00:13:03.039 --> 00:13:07.039 using the Azure is thek whatever it is, right, and you could just 170 00:13:07.799 --> 00:13:13.399 use those instrumentation libraries that are available from those frameworks. It could be built 171 00:13:13.440 --> 00:13:18.200 into the framework or could be a dedicated package and you can turn them on 172 00:13:18.000 --> 00:13:22.120 and just by doing that, you're already collecting a bunch of information, and 173 00:13:22.159 --> 00:13:26.159 specifically in the distributed system, it's usually going to give you insight into that 174 00:13:26.519 --> 00:13:33.200 interservice communication where we have the blind gaps. So it already like gives you 175 00:13:33.399 --> 00:13:37.679 a lot of information. So the cool thing is that you can then intercept 176 00:13:37.080 --> 00:13:43.039 basically those traces that are being generated by those libraries, and in that that 177 00:13:43.159 --> 00:13:46.440 I would be calling activity. It occurred, right, and you could add 178 00:13:46.960 --> 00:13:50.960 you could create your own activities, which is, by by the way, 179 00:13:50.000 --> 00:13:56.759 the sort of same thing as a span. But yeah, they basically what 180 00:13:56.840 --> 00:14:01.159 they did is the activity API already existed, and instead of creating a completely 181 00:14:01.200 --> 00:14:05.120 new API to match the naming of the open telum Try specification, they just 182 00:14:05.279 --> 00:14:09.799 implemented the specification inside the activity API. So that's why the name is a 183 00:14:09.840 --> 00:14:13.200 little bit different. But basically, what you could do is take the current 184 00:14:13.200 --> 00:14:16.120 activity, which could be omitted by an instrumentation library, and say, I 185 00:14:16.159 --> 00:14:22.639 want to add some information to this that is specific to my application, to 186 00:14:22.679 --> 00:14:26.840 the workflow I'm running in so that I can get even more insight. What's 187 00:14:26.879 --> 00:14:31.840 the voodoo that allows us to go between the different tiers and an app and 188 00:14:31.879 --> 00:14:35.840 say these are all part of the same transaction. Right, that's basically a 189 00:14:35.919 --> 00:14:41.759 propagation mechanism really, because if you think if a trace, it's basically a 190 00:14:41.799 --> 00:14:46.120 bunch of spans that are connected to each other. Now, what happens is 191 00:14:46.200 --> 00:14:50.639 at the beginning of the trace, we basically get a trace ID assigned and 192 00:14:50.720 --> 00:14:56.759 that's going to be carried across all of the spens and then each span has 193 00:14:56.799 --> 00:15:01.720 a unique ID. Now, in order for that information to propagate across multiple 194 00:15:01.759 --> 00:15:07.440 services, we need a propagation mechanism. And there's multiple protocols that are basically 195 00:15:07.480 --> 00:15:09.519 supported by the Open Telemetry Project, of most of well known. One is 196 00:15:09.559 --> 00:15:15.320 a W three C trace context for HTP headers always feel like I need a 197 00:15:15.320 --> 00:15:22.200 breath, yeah, And then there's another one for gRPC. So it depends 198 00:15:22.200 --> 00:15:26.200 on what you're doing there. So I have, for example, and as 199 00:15:26.240 --> 00:15:31.759 your web app, right, I have Application Insights turned on, I've got 200 00:15:31.799 --> 00:15:35.480 all the switches lit up, and do I need open telemetry at that point? 201 00:15:35.559 --> 00:15:39.480 What's it kind of give me over what Azure Insights already has. So 202 00:15:39.519 --> 00:15:46.679 the way that I look at that is what Application Insights provides you is also 203 00:15:46.799 --> 00:15:52.639 sort of known as black box instrumentation, so it's basically independent of your specific 204 00:15:52.639 --> 00:15:58.639 application code. Yeah, but obviously the things that we are doing in our 205 00:15:58.679 --> 00:16:03.799 code is usually the interesting bits, and sometimes we need a little bit more 206 00:16:03.840 --> 00:16:07.279 insight to understand, you know, what pieces of code are we executing there, 207 00:16:07.320 --> 00:16:11.639 what are we doing, what is like the cause of latency or whatever 208 00:16:11.639 --> 00:16:15.200 it is, what pieces slow? Application Insights doesn't provide that well. I 209 00:16:15.240 --> 00:16:22.720 think it's it's quite different to to compare because at least to my to my 210 00:16:22.840 --> 00:16:26.600 understanding, it's more of an overall view that you get, and you can 211 00:16:26.639 --> 00:16:33.519 still use the Application Insights as AK directly and still emit like application specific telemetry. 212 00:16:33.600 --> 00:16:36.960 But but then you're tied, right, You're tied to the vendor, 213 00:16:37.360 --> 00:16:41.039 right, and the thing is right right exactly, and if you don't want 214 00:16:41.120 --> 00:16:45.799 to change to another vendor, you're have that vanderlock. It's the point, 215 00:16:45.039 --> 00:16:49.320 right, Yeah, So if you use open telemetry, then you you could 216 00:16:49.360 --> 00:16:53.480 just wire up a different exporter. I mean that would also app Insights has 217 00:16:53.559 --> 00:16:57.399 good features for dot net specific absolutely, Yeah, if you've got some other 218 00:16:57.440 --> 00:17:00.799 code written in other things that aren't ornet related, yeah, happid sence is 219 00:17:00.840 --> 00:17:03.160 only going to do so much for you and certainly not going to work if 220 00:17:03.160 --> 00:17:07.119 you're in a container on AWS, is it. Yeah? So, I 221 00:17:07.119 --> 00:17:11.960 mean it certainly opens the door to working with more platforms, even more places 222 00:17:12.039 --> 00:17:17.400 and open hey, hey, there's a concept for you. Yeah, and 223 00:17:17.400 --> 00:17:22.359 it being available cross platform and cross front time as well. Like there's implementations 224 00:17:22.440 --> 00:17:27.599 for instrumentation libraries in many languages for many framework So that's really cool, especially 225 00:17:27.640 --> 00:17:30.839 if you think of like multi stack applications. Right. Sure, if I've 226 00:17:30.839 --> 00:17:34.559 got a group of Python developers going to build a data importer for me, 227 00:17:34.640 --> 00:17:37.559 the fact that I can instrument it the same way as everything I've gotten built 228 00:17:37.599 --> 00:17:41.319 and dot net, that's pretty compelling, right, and bring everything under one 229 00:17:41.359 --> 00:17:45.960 roof and measured the same way. I mean, that's the real problem is 230 00:17:45.960 --> 00:17:51.559 that often we're chasing problems that transition between different systems, and because they're measured 231 00:17:51.559 --> 00:17:55.519 differently, it's very hard to associate restuff together. I'm just thinking in terms 232 00:17:55.519 --> 00:17:57.279 of how much code I need to write as a developer to take advantage of 233 00:17:57.279 --> 00:18:00.839 all this and how much of it comes of the box. Well, that's 234 00:18:00.839 --> 00:18:06.559 what I That's why I mentioned the instrumentation libraries, right, and even if 235 00:18:06.599 --> 00:18:11.920 you know things like event counters and stuff like that, even that has dedicated 236 00:18:11.119 --> 00:18:15.920 libraries available already, so you can basically just turn them on and like I 237 00:18:15.960 --> 00:18:22.079 said, you could plug into that and add to that information information. So 238 00:18:22.160 --> 00:18:25.920 usually what I say is, look at what it already gives you. Turn 239 00:18:25.960 --> 00:18:30.039 on the instrumentation library. Yeah exactly, don't reinvent the library. So look 240 00:18:30.039 --> 00:18:33.240 at what's out there. Turn it on and see what it emits, and 241 00:18:33.640 --> 00:18:37.559 take a look at what type of insight that already gives you, and it 242 00:18:37.559 --> 00:18:40.599 probably is picked up by open telebras you're just fine. Yeah, yeah, 243 00:18:40.640 --> 00:18:45.720 Well, for example, let's say that you have aspe core instrumentation enabled, 244 00:18:45.839 --> 00:18:48.559 right, so you're going to see the request, but you don't get a 245 00:18:48.559 --> 00:18:52.359 lot of insight into the request. But I mean it has hooks that you 246 00:18:52.400 --> 00:18:59.799 could then plug in additional information and expose whatever is you know, interesting to 247 00:18:59.799 --> 00:19:03.039 you in that scenario, so that you could understand like the sort of business 248 00:19:03.119 --> 00:19:07.200 context of what's going on. And that's what makes it really powerful. I 249 00:19:07.240 --> 00:19:11.240 spend enough time on the firefighting side of being assisted min where there's stuff's being 250 00:19:11.240 --> 00:19:15.920 spewed out in the logs that we're looking at. We just don't know what 251 00:19:15.960 --> 00:19:18.519 it means, right, right, like anything there? Yeah, it's clear. 252 00:19:18.559 --> 00:19:21.759 It's like your own little Internet. Everything you need to know is there, 253 00:19:22.119 --> 00:19:25.720 you just can't find it. Yeah, So you know, how do 254 00:19:25.720 --> 00:19:30.480 you add the additional information that helps someone see? This is where we're having 255 00:19:30.640 --> 00:19:34.759 right. That's where defenders really come in, right, because they are then 256 00:19:34.839 --> 00:19:40.559 going to offer capabilities that allow you to querry, to analyze that information, 257 00:19:41.000 --> 00:19:45.000 and to basically get to actionable insights, because that's the whole point of telemetry. 258 00:19:45.079 --> 00:19:47.400 Right. That's a bunch of data, But then what do I do 259 00:19:47.480 --> 00:19:52.440 with it? Right? So you want to basically have tools that help you 260 00:19:52.279 --> 00:19:56.279 get pointers on how do I fix this problem or how do I improve this 261 00:19:56.400 --> 00:20:03.240 latency issue that I'm seeing, or maybe even see like which feature gets used 262 00:20:03.240 --> 00:20:06.759 a lot? And things like that. She is almost more of a profiling 263 00:20:06.920 --> 00:20:08.559 thing, right, Like what are the functions are we called the most often? 264 00:20:08.640 --> 00:20:11.640 And we know perfectly well why the system is slow. It's the database 265 00:20:11.759 --> 00:20:17.240 is fault. We just blame the DBA. Then we're done. Life is 266 00:20:17.240 --> 00:20:22.759 good, all right. You know Another one that I always think of is 267 00:20:22.880 --> 00:20:30.480 because I'm you know, I'm like an observability enthusiast, I would say, 268 00:20:30.759 --> 00:20:36.720 right. And the reason why I'm so enthusiastic about it is because, well, 269 00:20:36.720 --> 00:20:41.119 my sort of core focus has been message based systems for two years are 270 00:20:41.240 --> 00:20:47.720 to trust. Shoot, yeah, it's just clim messages split across Q and 271 00:20:47.759 --> 00:20:49.680 then you get out of order messages and it's like, what's going on? 272 00:20:49.960 --> 00:20:53.839 Why am I seeing this fail? And especially in a message based system, 273 00:20:55.480 --> 00:20:59.680 usually the problem is happening like for the up stream right, right, So 274 00:20:59.759 --> 00:21:03.279 how do you get to that? Right? So? And how do you 275 00:21:03.319 --> 00:21:07.880 connect at even outside of the messages that are being sent, because it has 276 00:21:07.880 --> 00:21:10.519 to maybe connect back to, like you said, a click somewhere on a 277 00:21:10.640 --> 00:21:15.880 user interface. So being able to have that full visibility across all of the 278 00:21:15.000 --> 00:21:18.799 subsystems is really really powerful. Yeah, and again I'm still worrying this a 279 00:21:18.839 --> 00:21:22.359 lot of codforma, right, But you're telling me that when when you're using 280 00:21:22.359 --> 00:21:26.200 a library that has protocol understanding, it's going to insert a lot of that 281 00:21:26.200 --> 00:21:30.920 information automatically for us. So it's selfol together. Yes, Yeah, And 282 00:21:30.000 --> 00:21:36.559 because of the sort of nature of how distributed tracing works, that information is 283 00:21:36.559 --> 00:21:41.799 going to be connected together through that same trace idea that's basically being propagated does 284 00:21:41.839 --> 00:21:45.359 have substantial overhead. Is there is there any reason to only turn it on 285 00:21:45.400 --> 00:21:47.920 when you have a problem or can you leave it on all the time? 286 00:21:48.519 --> 00:21:53.039 Okay, that's that that's going to be a long question. As you can 287 00:21:53.160 --> 00:22:00.480 you can go for the It dependspends definitely. So yeah, it definitely depends. 288 00:22:00.640 --> 00:22:04.839 But so usually what I tend to say is make sure that what you're 289 00:22:04.920 --> 00:22:11.920 collecting is useful, right start there, because if you just turn I don't 290 00:22:11.920 --> 00:22:15.640 know, every instrumentational library on the planet on, we log all the things 291 00:22:15.720 --> 00:22:19.039 and capture all of the traces and all of that, you're going to have 292 00:22:19.079 --> 00:22:23.200 to sift through that all of that information to be able to understand like what's 293 00:22:23.240 --> 00:22:27.039 going on. Right, So it's also not a thing of oh, look 294 00:22:27.079 --> 00:22:30.759 at all of these instrumentation libraries and then turning everything on, because you're going 295 00:22:30.799 --> 00:22:34.960 to be incredibly overwhelmed, to the point that as a developer you might feel 296 00:22:36.000 --> 00:22:38.200 like, okay, this is not useful. Yeah, let's just turn it 297 00:22:38.240 --> 00:22:41.640 all back off. I mean, I've also had the problem where I've said, 298 00:22:41.680 --> 00:22:44.319 Okay, I'm not going to measure this thing, and then I never 299 00:22:44.400 --> 00:22:47.279 get data for that thing, Like it turns out I'm looking for the wrong 300 00:22:47.359 --> 00:22:51.720 in the wrong place. Like that's not a number that moves. So some 301 00:22:51.759 --> 00:22:56.039 of these telemetry products that are out there have ways that they can work on 302 00:22:56.079 --> 00:23:03.160 a background thread or they can attach as a sidecar, you know. So 303 00:23:03.839 --> 00:23:07.960 do you have those kinds of things where you can sort of stay out of 304 00:23:07.960 --> 00:23:11.160 the way so if there is something that takes up some more time, it 305 00:23:11.160 --> 00:23:15.079 can happen on a background thread. Yeah. So that's where the open telemetry 306 00:23:15.119 --> 00:23:18.799 project is also really interesting because if let's say that you look at the basic 307 00:23:18.880 --> 00:23:22.920 samples that are out there for dot net right, what you're going to see 308 00:23:22.960 --> 00:23:26.000 there is that you can basically, in a service enable open telemetry at an 309 00:23:26.000 --> 00:23:30.200 exporter, which means that you're collecting let's say, for the sake of the 310 00:23:30.240 --> 00:23:36.519 example traces and sending them directly to an observability back end. Basically what that 311 00:23:36.519 --> 00:23:41.279 could be as your application Insights or Jager or Honeygo, whatever it is. 312 00:23:41.079 --> 00:23:47.519 So, but the thing is is that there's many problems to that. First 313 00:23:47.559 --> 00:23:49.880 of all, like you said, there is overhead for that service because it 314 00:23:49.920 --> 00:23:55.400 has to collect all of that information. There might even be some processing behind 315 00:23:55.440 --> 00:24:00.759 the scenes happening as well. We had a service bus, well, well 316 00:24:00.880 --> 00:24:03.319 that's we'll get to that. It will get to that, yeah, and 317 00:24:03.359 --> 00:24:07.680 then you have to export it. But then imagine that the observability back end 318 00:24:07.799 --> 00:24:12.400 is not available for a few seconds because it's you know, it's the network. 319 00:24:12.599 --> 00:24:21.559 It's the network. Yeah right, so well no, that's usually that's 320 00:24:21.680 --> 00:24:26.960 handled by the libraries themselves, but it is adding that pressure to the services 321 00:24:27.400 --> 00:24:32.359 telemetry, right, yeah, you don't want to have that disconnected information and 322 00:24:32.400 --> 00:24:34.599 then looking at half of the story, right, But there are ways to 323 00:24:34.640 --> 00:24:38.079 solve this, and that's where the open telemetry collector comes in. And then 324 00:24:38.079 --> 00:24:42.200 you have multiple deployment options on how to run that. The first one is, 325 00:24:42.240 --> 00:24:45.039 like you said, a sidecar, So basically you're going to have a 326 00:24:45.079 --> 00:24:51.759 sidecar for each service that you're instrumenting, and then immediately you're offloading all of 327 00:24:51.799 --> 00:24:55.960 the telemetry. Well, you're collecting it and sending it through to the sidecar. 328 00:24:56.079 --> 00:24:59.799 But there are any processing that needs to be done, like redacting information 329 00:25:00.039 --> 00:25:04.839 because remember sensitive information, you don't want that you're in all of that telemetry 330 00:25:04.839 --> 00:25:10.079 you're collecting. So that's just down the road. We can be fined very 331 00:25:10.079 --> 00:25:14.880 easily. I call that digital white out. So then you have all of 332 00:25:14.880 --> 00:25:18.599 that processing and then you could export it then to the observability back end and 333 00:25:18.640 --> 00:25:22.839 you'd be able to handle all those communication issues and all of that in the 334 00:25:22.920 --> 00:25:27.599 side car and your service is not affected. Now that's one option, but 335 00:25:27.759 --> 00:25:33.599 you could also set up the open telemetry collector as as at the gateway, 336 00:25:33.640 --> 00:25:37.599 so it's a central components and all of the services can basically send their telemetry 337 00:25:37.640 --> 00:25:42.640 information to that central component, which then will take care of processing all of 338 00:25:42.680 --> 00:25:48.519 that information to the to the back Yeah, and it could batch that information 339 00:25:48.720 --> 00:25:52.319 and it's a pretty powerful thing, and you could even let go crazy or 340 00:25:52.400 --> 00:25:56.319 if you need it right, but you could have a sort of hybrid model 341 00:25:56.359 --> 00:26:00.480 in which you have a sidecar per service which then sending their reformation to the 342 00:26:00.480 --> 00:26:06.119 central collectors. Is there anything special in the storage on that and there's just 343 00:26:06.160 --> 00:26:10.799 blobs or text files or are they actually using a database on that? Well? 344 00:26:10.839 --> 00:26:15.599 I think it depends on the signal, sure, because usually metrics go 345 00:26:15.799 --> 00:26:21.960 to time serious databases and then logs. Honestly, I don't know. It's 346 00:26:22.160 --> 00:26:26.000 a good question, depends but it depends. Yeah. But well, actually 347 00:26:26.039 --> 00:26:32.519 about those time series databases. That's a sort of interesting topic on its own 348 00:26:32.559 --> 00:26:36.920 because at the beginning I was talking about TELME tree correlation, right, and 349 00:26:37.079 --> 00:26:42.079 basically adding the trace ID to the metric, just called them exemplar in open 350 00:26:42.079 --> 00:26:48.319 CELM tree naming, so that you would be able to connect that together, 351 00:26:48.400 --> 00:26:49.799 so you'd see a spike in the metric and see, oh, that's caused 352 00:26:49.839 --> 00:26:55.440 by those traces. Right now, the thing is that you have to be 353 00:26:55.480 --> 00:27:00.920 aware of what's known as cardinality explosion. So I'll try to when a bomb 354 00:27:02.000 --> 00:27:06.200 goes off in the Vatican. Dude, but did I say that. I'm 355 00:27:06.240 --> 00:27:12.839 sorry, those are cardinal explosions, not the same cardinality explosion. Cardinality explosion. 356 00:27:14.480 --> 00:27:17.720 I'll try to explain that, but usually I do this visually, so 357 00:27:17.839 --> 00:27:21.720 okay, I'll give me a bit to try. But think of the exemplars 358 00:27:21.759 --> 00:27:25.440 basically a label, right that you're adding to a metric, and that's going 359 00:27:25.480 --> 00:27:30.359 to give you some insights on the context in which that metric is being collected. 360 00:27:30.799 --> 00:27:34.559 So let's say that I have a metric called failure rate, because I 361 00:27:34.599 --> 00:27:40.640 want insight into that and to have a little bit more background information. I 362 00:27:40.680 --> 00:27:44.240 want to know which environment that that happened, bill, development, tests, 363 00:27:44.480 --> 00:27:48.240 production, whatever it is. And I also want to know which hp status 364 00:27:48.279 --> 00:27:52.440 coode came back. Now that HSP status coode in a production environment is going 365 00:27:52.480 --> 00:27:57.200 to have like for the sake of the example, thirty possible values. And 366 00:27:57.319 --> 00:28:03.839 we have three different environments. So that's three possible values for that environment label. 367 00:28:03.799 --> 00:28:12.480 Now the cardinality is basically all of the possible combinations of those values of 368 00:28:12.720 --> 00:28:18.759 every label that you add that is a cardinality. So it's a multiplier exactly. 369 00:28:18.799 --> 00:28:23.839 So we're fine with the environment and then having the HTP statoscope and then 370 00:28:23.839 --> 00:28:29.160 I add customer ID. Oh boy, so why let you do that? 371 00:28:29.200 --> 00:28:32.400 We're like one hundred combinations, and then you threw in twenty thousand customers a 372 00:28:32.519 --> 00:28:36.319 ruined everything more. Yeah, how do thousand customers a million customers? Somebody 373 00:28:36.319 --> 00:28:41.960 would ever do vouch? And that's how we then basically get cardinality explosion because 374 00:28:41.960 --> 00:28:45.160 what happens in the time series database is that every time you have a sort 375 00:28:45.160 --> 00:28:51.079 of unique combination and your series is created, so your cost goes up right 376 00:28:51.200 --> 00:28:55.599 and it becomes really hard to quire that information. So it's also important to 377 00:28:55.960 --> 00:28:59.960 one field, which just one field. If you just don't do that, 378 00:29:00.400 --> 00:29:04.319 there won't be this problem. Yeah, well it seems like a good idea 379 00:29:04.319 --> 00:29:10.759 when you do it right until cardinalities exactly. So it's also important to be 380 00:29:10.839 --> 00:29:14.480 aware of, you know, what observability back end are you using and how 381 00:29:14.519 --> 00:29:18.160 does that work, because there are some tools out there that do support high 382 00:29:18.160 --> 00:29:21.960 cardinality. So it's just something that you have to be aware of. Yeah, 383 00:29:22.279 --> 00:29:25.200 and so you can tolerate if you really come to the resolution you have 384 00:29:25.240 --> 00:29:29.359 to do that one way or the other. I mean, custerrity doesn't seem 385 00:29:29.400 --> 00:29:33.920 that crazy because it is useful if you've got a customer on a phone to 386 00:29:33.000 --> 00:29:37.799 say, hey, I could pull all the transactions, all of those streams 387 00:29:37.799 --> 00:29:41.279 for all of that customer and sort of look at where they were having problems. 388 00:29:41.240 --> 00:29:47.440 Yep. That's like full production debugability, right, yeah, without doubt. 389 00:29:48.400 --> 00:29:49.759 And with that, I've got to interrupt for one moment for this very 390 00:29:49.839 --> 00:29:57.799 important message too, and we're back. It's dotting at Rocks. I'm Richard 391 00:29:57.799 --> 00:30:02.079 Campbell, that's Carl Franklin. Hey, Hey, talking to our friend Leila 392 00:30:02.200 --> 00:30:04.880 about open telemetry. Hey hey, and we've kind of gotten to that place 393 00:30:06.000 --> 00:30:11.400 now, right, Like we we how do we visualize this because you're getting 394 00:30:11.400 --> 00:30:15.240 you're probably it a lot of information like what is are the tooling that comes 395 00:30:15.240 --> 00:30:18.720 with it? Or I have to write my own what are the dashboards? 396 00:30:18.799 --> 00:30:23.160 Well, that's hopefully where you choose a vendor, right, and then you 397 00:30:23.200 --> 00:30:27.359 know, with the abilities of open telemetry, by standardizing all of that information, 398 00:30:27.400 --> 00:30:30.880 I usually just say, like, try a bunch out. You see 399 00:30:30.880 --> 00:30:37.000 what are the requirements for choices? You imagine you can just look these up 400 00:30:37.000 --> 00:30:41.119 to standard graph controls and things like that from your various vendors. If you 401 00:30:41.160 --> 00:30:45.920 want a dashboard, yeah, yeah, yeah, there's just so many options, 402 00:30:45.960 --> 00:30:52.519 and I feel like each of them has their own strength. But yeah, 403 00:30:52.559 --> 00:30:55.599 for example, if you're if you're running in the Azure stack then and 404 00:30:55.680 --> 00:30:59.359 you're already using application insights, it's a thing you know, can it makes 405 00:30:59.359 --> 00:31:03.880 sense against and you're using that as well. Yeah, But then for example, 406 00:31:03.359 --> 00:31:07.519 I've played around with Honeycomb, and I think that they're especially like the 407 00:31:07.680 --> 00:31:12.200 collaboration that they've built into the tool is really cool as well as AWS product. 408 00:31:14.400 --> 00:31:17.200 No, no, it's its own company, Honeycomb. Okay, big 409 00:31:17.200 --> 00:31:21.240 big taste and a big big bike. Nice. That's the breakfast Cereal. 410 00:31:21.319 --> 00:31:23.279 But okay, I was a kid of the you know what I'm thinking. 411 00:31:23.319 --> 00:31:27.880 I'm thinking of that there is a honey Honey something product in AWSS, no 412 00:31:27.960 --> 00:31:32.319 code product, yeah, okay, different one, but yeah, Honeycomb is 413 00:31:32.359 --> 00:31:37.960 an instrumentation library. Yeah. So, and they're very invested in open telemetry 414 00:31:37.000 --> 00:31:41.839 as well, and I think it's one of those tools that supports high cardinality. 415 00:31:41.880 --> 00:31:44.480 They talk about it a lot as well, right, yeah, but 416 00:31:44.640 --> 00:31:48.000 yeah, it would sit out to me. There's really the sort of collaborative 417 00:31:48.359 --> 00:31:52.279 feature features that they had, because usually if you're looking at a huge problem, 418 00:31:52.599 --> 00:31:55.000 you're not doing that by yourselves, right, yeah, you know, 419 00:31:55.079 --> 00:32:00.160 and if it's like if you if you have a twenty four seventeen and one 420 00:32:00.240 --> 00:32:01.920 and stair shift, you want to be able to hand over where you left 421 00:32:01.960 --> 00:32:07.240 off things like that. Yeah, I'm wondering about So who is this Typically 422 00:32:07.279 --> 00:32:08.920 the systems that are going to get these packages in the first place, that 423 00:32:08.920 --> 00:32:14.000 that's where the air resh is first show up, where the problems appear, 424 00:32:14.079 --> 00:32:16.160 and then they might be passing it to development saying hey, we're looking at 425 00:32:16.160 --> 00:32:22.519 this and we think it's it's this kind of problem. Well that's where I'm 426 00:32:22.559 --> 00:32:29.640 also expecting some evolution because yes, usually now it would be a different team 427 00:32:29.680 --> 00:32:32.880 when they would be looking at something that looks funny, yeah right, and 428 00:32:32.920 --> 00:32:38.119 then get some understanding, hopefully some actionable insights, right, and then being 429 00:32:38.160 --> 00:32:44.200 able to bring that back to the development team. But it's it's really interesting 430 00:32:44.240 --> 00:32:49.440 to me in the sense that we're the developers, We are the ones going 431 00:32:49.559 --> 00:32:55.839 to be writing that application specific telemetry. So it's it's also really important to 432 00:32:55.880 --> 00:33:02.640 get like organization wide alignment as well on the type of telemetry that you're going 433 00:33:02.640 --> 00:33:07.039 to be collecting. So this seems like there's an infinite number of decisions to 434 00:33:07.079 --> 00:33:10.920 make here, right, I mean you just by using open telemetry, that's 435 00:33:10.960 --> 00:33:15.599 just one step of many. Yeah, what are we going to be looking 436 00:33:15.640 --> 00:33:17.759 at how what's the granularity of it, How are we going to query it, 437 00:33:19.039 --> 00:33:22.000 how are we going to look at it visually? Like, these are 438 00:33:22.039 --> 00:33:24.559 all things that aren't just in the box. You have to think them through. 439 00:33:24.839 --> 00:33:30.359 Yeah. Well, usually what I try to give a world try to 440 00:33:30.400 --> 00:33:35.920 advise as well, is that to sort of documents and guidelines like what are 441 00:33:35.960 --> 00:33:38.759 you looking for with your telemetry? What are the problems that you're trying to 442 00:33:38.799 --> 00:33:45.240 solve and like come up with a specific set of questions that you could as 443 00:33:45.240 --> 00:33:51.680 a developer when you write a feature, ask yourself so that you could add 444 00:33:51.720 --> 00:33:53.920 the telemetry that is going to answer those questions. Right, I mean, 445 00:33:53.960 --> 00:33:57.759 what if you don't know what you don't know? What if you don't know 446 00:33:57.799 --> 00:34:00.480 what you want to look for as a guidance? Are right to say? 447 00:34:00.400 --> 00:34:05.079 Yeah? Some of this sounds like business related decisions, like we sell widgets, 448 00:34:05.119 --> 00:34:07.880 and I want to know when a sale fails because of technology rather than 449 00:34:07.920 --> 00:34:12.280 the customer didn't want to buy it, right, yeah, definitely, or 450 00:34:12.360 --> 00:34:16.280 simple things like this page took too long to render or whatever. Yeah. 451 00:34:15.639 --> 00:34:21.079 Yeah. And then from the sort of failure perspective, I usually try to 452 00:34:21.599 --> 00:34:24.599 look at the code and think to myself, if something were failing here, 453 00:34:24.800 --> 00:34:29.159 what I would be What would I be looking at if I were debugging this, 454 00:34:29.360 --> 00:34:32.920 like, what what state would be interesting to me? What variables would 455 00:34:32.960 --> 00:34:37.679 I be looking at? Have I captured the input enough? Have I captured 456 00:34:37.719 --> 00:34:42.760 what's going out to be able to understand and be able to then you know, 457 00:34:42.840 --> 00:34:45.320 go back and try to understand what happened from the outside. Right, 458 00:34:45.440 --> 00:34:49.280 there's an easy solution all of this you just put all your code in a 459 00:34:49.320 --> 00:34:53.760 try with an empty catch. No problem is that on error resumed next a 460 00:34:53.920 --> 00:34:59.159 sort of yeah, it's more like slash day to turn off all the debuggs. 461 00:34:59.239 --> 00:35:01.039 I don't want to know. Just keep getting don't tell me about these 462 00:35:01.079 --> 00:35:06.159 guys. I mean a lot of this we've talked about very proactively, like 463 00:35:06.199 --> 00:35:09.559 we're going to detect the errors before the customer does or before the customer complaint. 464 00:35:09.639 --> 00:35:14.000 Yeah. I think there's another dynamic where the customer is complaining and we're 465 00:35:14.039 --> 00:35:19.159 getting a ticket that's like what error. I think it'd be very challenging to 466 00:35:19.159 --> 00:35:22.559 say, you've got this ticket is about this customer, it was roughly at 467 00:35:22.559 --> 00:35:24.280 this time, and now you want to go dig through the logs to say, 468 00:35:24.320 --> 00:35:28.679 can we see what this person's complained about. To do is to have 469 00:35:28.719 --> 00:35:31.679 the customer on the phone and enable this sort of thing, but just for 470 00:35:31.719 --> 00:35:37.480 them, just for their customer ID and say and then just watch it as 471 00:35:37.519 --> 00:35:39.679 they're going through the process where it fails and now you've got something. But 472 00:35:40.440 --> 00:35:45.639 is that impossible? Well? I think so, but it sort of depends 473 00:35:45.679 --> 00:35:49.920 on what type of sampling strategies that you're applying. And also, like you 474 00:35:50.000 --> 00:35:52.840 know the type of observability that you're collecting, because let's say that if you 475 00:35:53.159 --> 00:35:59.800 want full insight, so basically any error that occurs to any user, right, 476 00:36:00.360 --> 00:36:02.880 be able to say, oh, you know that was since we're an 477 00:36:02.880 --> 00:36:08.119 antwer that was Shalts and it was three pm on a Friday, and basically 478 00:36:08.119 --> 00:36:13.480 be able to find that request and look at what were they doing, because 479 00:36:13.480 --> 00:36:16.840 the thing is that users when they open tickets or whatever it is, the 480 00:36:16.840 --> 00:36:21.920 thing is that they weren't paying attention to doing their usual thing, right. 481 00:36:22.000 --> 00:36:24.840 They didn't set down See, let's cause an error, and I'm not thinking 482 00:36:24.880 --> 00:36:29.679 about every step that I did so I could be able to explain it to 483 00:36:29.719 --> 00:36:32.320 you later. I usually get it doesn't work. What can you be more 484 00:36:32.360 --> 00:36:38.760 explicit? It doesn't work? Right, So then being able to connect that 485 00:36:38.840 --> 00:36:43.559 information back and say, oh, yeah, that was shuts right, and 486 00:36:43.599 --> 00:36:46.679 then find that, you know, Tracey's locks metric whatever was connected to that 487 00:36:46.800 --> 00:36:52.639 information. It's like being able to debug in production. Really yeah, And 488 00:36:52.719 --> 00:36:55.719 then we think you just described like turning up a lot of data there too, 489 00:36:57.280 --> 00:37:00.480 Like that's also you know, we were also warning not to do that 490 00:37:00.519 --> 00:37:05.480 because you're getting buried in minutia yea, yep. That's where the sampling strategies 491 00:37:05.519 --> 00:37:10.000 commits. Sure, and there's different ways to go about that. So if 492 00:37:10.079 --> 00:37:15.880 you think about traces specifically, you basically can choose between head and tail sampling 493 00:37:16.119 --> 00:37:22.599 and head sampling and dare You're basically going to decide whether to sample the trace 494 00:37:22.679 --> 00:37:25.639 at the beginning. So let's say when the business transaction starts, right, 495 00:37:25.679 --> 00:37:29.760 they're going to immediately make the decision of I'm keeping this or I just don't 496 00:37:29.800 --> 00:37:34.840 care about it. Right. Usually that's the most unbiased type of sampling as 497 00:37:34.840 --> 00:37:40.039 well quick order. Right. The thing is what if something fails, right, 498 00:37:40.559 --> 00:37:45.239 you don't know that upfront, What if this request was super slow. 499 00:37:45.960 --> 00:37:50.159 It's not something that you can know up front, so you could be losing 500 00:37:50.639 --> 00:37:53.880 a lot of insightful information. Sure, and that's where you get, you 501 00:37:53.920 --> 00:38:00.519 know, the tail based approach where you're basically going to collect everything so the 502 00:38:00.719 --> 00:38:05.199 entire trace across all of the services that it goes through, and at the 503 00:38:05.320 --> 00:38:08.239 end make the decision of is this an interesting trace? Do I want to 504 00:38:08.320 --> 00:38:12.400 keep it? So? For example, does it carry some specific attributes that 505 00:38:12.440 --> 00:38:15.559 I care about? Or was it slow? Yeah? And banks, the 506 00:38:15.639 --> 00:38:19.880 question can you turn these things on and off in production without restarting. Right. 507 00:38:19.880 --> 00:38:23.320 So that's where it becomes important again how you deployed this, right, 508 00:38:23.360 --> 00:38:28.639 because if you have a sort of direct export and it's really tricky, but 509 00:38:29.119 --> 00:38:31.159 if if you had it deployed as a side card, and it could be 510 00:38:31.239 --> 00:38:36.440 a thing of changing the configuration of the side card, like tailbase says, 511 00:38:36.519 --> 00:38:39.440 I'm going to assess the finish transactions, right, there's nothing special about this, 512 00:38:39.840 --> 00:38:43.800 throw it out right. Oh, this one had an unusual value, 513 00:38:43.800 --> 00:38:45.679 it took too long, it generated this air. So forth, I'm going 514 00:38:45.679 --> 00:38:50.440 to keep this one. And so that way you're sort of sculling as you 515 00:38:50.519 --> 00:38:52.800 complete exactly. That's pretty cool. Yeah, there's a cost that comes with 516 00:38:52.840 --> 00:38:57.599 that, sure, because basically you're collecting everything, so you've got those overhead 517 00:38:57.679 --> 00:39:01.639 on the workload. Although hopefully synchronous to some degree. Actually that asynchronicsy brings 518 00:39:01.639 --> 00:39:05.000 some dishing point that you wanted to have the customer on the phone while you 519 00:39:05.000 --> 00:39:07.000 work them through it, like that one scenario. How much latency do we 520 00:39:07.079 --> 00:39:10.639 have when we're doing all that processing separately, Like how soon can we see 521 00:39:10.719 --> 00:39:15.159 data from when it tries actionized? You happen, right, that's again another 522 00:39:15.239 --> 00:39:20.239 concern that you have to basically be focused though, right, like what is 523 00:39:21.559 --> 00:39:25.199 what type of latency are you willing to accept? And then it becomes a 524 00:39:25.239 --> 00:39:29.400 thing of being very mindful of what type of processing that you're doing there, 525 00:39:29.440 --> 00:39:32.119 because obviously all of that telemetry is going to have to go through that pipeline 526 00:39:32.519 --> 00:39:37.400 of processors, so you have to be very mindful of that. And then 527 00:39:37.960 --> 00:39:39.719 you have to get it out as soon as you can so that you can 528 00:39:39.760 --> 00:39:44.519 see it as quickly as you can and you can create as quickly as you 529 00:39:44.559 --> 00:39:49.599 can. But we're still talking seconds, aren't we really? Yeah, I 530 00:39:49.599 --> 00:39:54.320 hope even less milliseconds. That's been my usual experiences with asynchronous and telemetry. 531 00:39:54.480 --> 00:40:00.000 It's only slightly behind it, but it's not holding the transaction while it finished, 532 00:40:00.239 --> 00:40:02.440 you know that. That to me, the big sin here is don't 533 00:40:02.519 --> 00:40:07.960 delay the customer. Get they get the working transaction done. All telemetry can 534 00:40:07.000 --> 00:40:12.320 happen later, even though that later is a few milliseconds, right, That's 535 00:40:12.360 --> 00:40:15.679 why you offload. That's an our components again, Yeah, safer, safer 536 00:40:15.719 --> 00:40:19.840 to work that way anyway, which you don't want as a transaction field because 537 00:40:19.880 --> 00:40:23.440 you were measuring it exactly exactly that's dumb, that's no no. Yeah, 538 00:40:23.480 --> 00:40:30.039 so yeah, definitely quantum. I just got it, just like that, 539 00:40:30.079 --> 00:40:32.159 boom, it all makes it just makes sense, even though it still don't 540 00:40:32.199 --> 00:40:36.400 understand quantum. I'll leave that to you and sip, okay, we'll work 541 00:40:36.440 --> 00:40:39.320 We'll keep working on that problem. Yeah, that's definitely something to keep in 542 00:40:39.360 --> 00:40:44.880 mind is that, you know, instrumentation is at the end of the day, 543 00:40:44.960 --> 00:40:47.559 nice to have. It's not a mission critical component, so have to 544 00:40:47.599 --> 00:40:52.719 be very mindful of of how you do that. So one of the biggest 545 00:40:52.719 --> 00:40:58.000 mistakes that customers are people using the open til I'm train make when they first 546 00:40:58.000 --> 00:41:00.519 started. The biggest mistake, well, I turned everything on. Of course 547 00:41:00.519 --> 00:41:05.119 you did. I do that too. I like all the knobs, turn 548 00:41:05.199 --> 00:41:10.039 on all the knobs and then yeah, another thing was just figuring out what 549 00:41:10.079 --> 00:41:16.320 do I really mean? Right, So it's really understanding what your requirement is, 550 00:41:16.840 --> 00:41:22.119 because if you're saying I want to be able to debug any user request 551 00:41:22.159 --> 00:41:28.599 that happens in the system and have full visibility into that as opposed to for 552 00:41:28.639 --> 00:41:35.480 example, I walked into a project it's massive, there's zero documentation, and 553 00:41:35.519 --> 00:41:37.800 all of the people that worked on it, they're gone, right, and 554 00:41:37.920 --> 00:41:42.679 there's no instrumentation. Yeah, at that point, what you would like to 555 00:41:42.719 --> 00:41:46.599 have observability wise, it's just insight into how does this thing work? Yeah? 556 00:41:46.679 --> 00:41:51.440 Just follow a transaction, right, you know, have one end to 557 00:41:51.559 --> 00:41:54.239 end trace once and you'll have made a lot of progress. Yes, Now, 558 00:41:54.280 --> 00:41:58.679 the amount of telemetry that you need for those two things, it's completely 559 00:41:58.719 --> 00:42:02.559 the opposite, right, it's like one percent versus one hundred. So it's 560 00:42:02.599 --> 00:42:07.199 definitely still for me as well a learning experience. I mean, this is 561 00:42:07.239 --> 00:42:12.960 still pretty new and we're basically just adjusting to see still how the project is 562 00:42:12.960 --> 00:42:16.519 evolving. Yeah, because I mean many parts of the specification are stable by 563 00:42:16.559 --> 00:42:21.280 now, but a lot of things are still evolving. Yeah. Another thing 564 00:42:21.360 --> 00:42:25.400 that keeps coming up is sort of the three pillars of observability being you know, 565 00:42:25.440 --> 00:42:29.400 traces, metrics, and logs, And I wonder, but what if 566 00:42:29.400 --> 00:42:32.400 a first one comes along that might just happen. I can't think of a 567 00:42:32.440 --> 00:42:37.599 fourth one, right, neither find us pretty well. Yeah, when when 568 00:42:37.679 --> 00:42:40.239 the only thing we had was logs, could we think of metrics and traces? 569 00:42:40.280 --> 00:42:44.239 I guess that's true. Yeah, we started dreaming about them. It's 570 00:42:44.280 --> 00:42:46.400 like I'm trying to I've had that experience of I have the log from this 571 00:42:46.480 --> 00:42:50.920 machine, the log from this machine, now try and line those entries up. 572 00:42:51.119 --> 00:42:54.639 Would quantum computing introduce a fourth pillar? I think it would introduce to 573 00:42:54.639 --> 00:42:59.679 the sixteenth pillars, or two to the two hundred and thirty seconds parallel there. 574 00:43:00.239 --> 00:43:02.320 I also like this idea of knowing it failed before the customer does have 575 00:43:02.440 --> 00:43:06.400 failed. You know, you get what I'm hoping to, this tail loog 576 00:43:06.519 --> 00:43:09.559 thing of I see these numbers as insufficient. In some way, it kicks 577 00:43:09.880 --> 00:43:13.559 up into a system where someone can look at it and perhaps even call a 578 00:43:13.559 --> 00:43:16.960 customer and say, hey, we noticed, can we help you with You 579 00:43:17.000 --> 00:43:21.199 know, it's not just you're waiting for the for the people to complain or 580 00:43:21.239 --> 00:43:23.559 if it has to be on fire. I have never gotten a message or 581 00:43:23.559 --> 00:43:28.320 an email like that from a company that I and if I if that happened 582 00:43:28.320 --> 00:43:31.599 to me, I would be like really impressed. So I'm on a website, 583 00:43:31.639 --> 00:43:35.880 for example, and it screws up, and then I immediately get an 584 00:43:35.880 --> 00:43:37.880 email that says, hey, we noticed you had this problem, didn't work. 585 00:43:37.960 --> 00:43:43.400 Here's a solution. M Wow, that would be amazed. I ever 586 00:43:43.440 --> 00:43:45.599 see that is like a credit card when I'm traveling, yeah where every so 587 00:43:45.639 --> 00:43:50.119 often I use the card and a minute or so later the phone rings and 588 00:43:50.119 --> 00:43:53.079 it's hey, are you in Belgium? Right, yep, yep, I'm 589 00:43:53.079 --> 00:43:57.039 in Belgium. Okay, that's good. Then thanks, And then you're like, 590 00:43:57.239 --> 00:44:00.800 that's pretty cool. It turns out that was Bob from New Jersey. 591 00:44:00.039 --> 00:44:06.440 Just just want to know that happens to me all the time. I asked 592 00:44:06.480 --> 00:44:09.639 you about how service bus or you know, service buses and service bus plays 593 00:44:09.639 --> 00:44:14.000 into this. Obviously, this is what you do for your job. How 594 00:44:14.039 --> 00:44:21.679 does using a messaging system figure into open telemetry. Well, like I said 595 00:44:21.679 --> 00:44:25.719 earlier, right, um, building a message based system is is really nice. 596 00:44:25.880 --> 00:44:29.760 Well, you know, if you take into account the entire problem space. 597 00:44:30.280 --> 00:44:32.480 But one of the things you can go around around, basically is how 598 00:44:32.480 --> 00:44:37.400 hard it becomes the troubleshoot things in that type of system. Now, with 599 00:44:37.559 --> 00:44:42.840 the platform that we're building at particular, we don't only have in service bus 600 00:44:42.840 --> 00:44:45.440 as a middleware framework, but we also have a bunch of tools. Now 601 00:44:45.880 --> 00:44:52.960 service Insite specifically is basically already that sort of black box instrumentation. It's like 602 00:44:52.000 --> 00:44:57.159 you don't need to do anything, you just need to configure that you want 603 00:44:58.239 --> 00:45:04.199 basically those platform tools to be enabled, and it already gives you insight into 604 00:45:04.360 --> 00:45:08.199 all of the messages that are being sent around in the system, and you 605 00:45:08.199 --> 00:45:14.760 can see that in a production environment and understand which exact flow and where that 606 00:45:14.840 --> 00:45:20.639 come from and which message led to which message. So we've really been doing 607 00:45:20.679 --> 00:45:22.679 observability for years. Yeah, that's the thing. Well, because of the 608 00:45:22.719 --> 00:45:28.280 asyncrency and out of orderness, like you can't count on timestamps, you really 609 00:45:28.320 --> 00:45:32.039 need to have some kind of attribute flag to be able to know the related 610 00:45:32.119 --> 00:45:35.920 Yeah, those are things that we capture in the message headers, so we 611 00:45:35.960 --> 00:45:40.639 can basically know which message LEDs to which other messages, So that that's really 612 00:45:40.679 --> 00:45:45.159 cool. But the thing that we could never solve, and that's the sort 613 00:45:45.159 --> 00:45:51.239 of gap that opens telemetry closes, is having that visibility system wise, so 614 00:45:51.360 --> 00:45:55.800 connecting back to you know, a fronted or even a database, a web 615 00:45:55.800 --> 00:46:00.599 server that restarted in the middle of something like those kinds of things like, 616 00:46:00.599 --> 00:46:02.559 that's where you want the metrics to show. Hey, this machine got into 617 00:46:02.599 --> 00:46:07.840 crisis and the supervisor killed the process and it recovery and continued, but it 618 00:46:08.199 --> 00:46:12.760 kicked off all this weirdness right, like I can. You're looking at it, 619 00:46:13.039 --> 00:46:15.639 You're looking at the trades and going, what the heck happened here? 620 00:46:15.840 --> 00:46:17.280 Right? Right? Is the is the program broken? It's just a bug? 621 00:46:17.320 --> 00:46:20.519 And it's like, no, this is what recovery looks. Yeah, 622 00:46:20.599 --> 00:46:23.599 yeah, exactly is one of these machines. Some of that is also built 623 00:46:23.639 --> 00:46:29.880 into the platform because we have insight into what are the failure how many failures 624 00:46:29.880 --> 00:46:35.199 are happening, So we also have messaging specific metrics already in there, and 625 00:46:35.199 --> 00:46:40.039 and yeah, now we're working to basically make sure that it also connects and 626 00:46:40.199 --> 00:46:46.280 feeds into the open telemetry signals so that if people are using it, that 627 00:46:46.400 --> 00:46:51.000 they get all of that information in there as well. Wow, that's pretty 628 00:46:51.239 --> 00:46:54.440 pretty substantially cool, especially when you start throwing cloud in here where it's entirely 629 00:46:54.480 --> 00:46:59.800 possible the cloud vendor might move you and it might have impact on your software, 630 00:47:00.159 --> 00:47:02.800 like stuff you literally don't have control over, Like hopefully your telemetry can 631 00:47:02.840 --> 00:47:06.400 surface that in a way where you're like, oh, this wasn't us. 632 00:47:07.440 --> 00:47:12.159 It's the vendor change something for whatever reason, and we should absorb it. 633 00:47:12.440 --> 00:47:15.440 But we don't know how yet because we've never had this happen before and right 634 00:47:15.519 --> 00:47:16.639 now we have to look at it and saying what would we do differently? 635 00:47:17.639 --> 00:47:22.880 Yeah, yeah, because we have sort of recoverability built in, so retries 636 00:47:22.920 --> 00:47:25.800 and all of that happens out of the box. You don't even have to 637 00:47:25.800 --> 00:47:30.360 configure them. But yeah, then it's like, okay, why does it 638 00:47:30.360 --> 00:47:36.280 take ten times for this message to be processed? Every time a message of 639 00:47:36.320 --> 00:47:38.719 that type comes in? Right, So it's you don't want to hide that 640 00:47:38.800 --> 00:47:43.920 away. It could be that maybe there's a database suffering underneath, right, 641 00:47:43.960 --> 00:47:45.559 it retries too quickly and it takes that long for that thing to get up 642 00:47:45.559 --> 00:47:49.920 to speed or yeah, yeah, along one of the side effects. Yeah, 643 00:47:50.039 --> 00:47:53.440 that's also why we have delayed retries. So we'll basically also have this 644 00:47:53.559 --> 00:47:59.239 sort of back off mechanism. We'll retry immediately. But if we see exactly 645 00:47:59.400 --> 00:48:01.760 that's too yep, yeah, it's the same. It's the same concept. 646 00:48:01.840 --> 00:48:07.639 Yeah. Yeah. So what's next for you? What are you doing next? 647 00:48:07.840 --> 00:48:12.440 What's in your inbox? What's in my inbox? Well, uh, 648 00:48:12.480 --> 00:48:15.960 we're so a techarama today. Then I'm taking a few days with the family. 649 00:48:16.119 --> 00:48:20.360 It's a long weekend, is it? It's a long Weekend's holiday Tomorrow? 650 00:48:20.400 --> 00:48:25.199 And then they're basically bridging to the weekend and then on Sunday, I'm 651 00:48:25.280 --> 00:48:30.400 leaving to Oslo for NDC. Very great. I will not be joining you 652 00:48:30.440 --> 00:48:34.159 there this year, so I heard. Yeah, not happy to hear that. 653 00:48:34.400 --> 00:48:37.639 Sorry, just schedules. Yeah. We love Oslo, we love NDC. 654 00:48:37.800 --> 00:48:39.880 We're there all the time. Usually, I'm gonna have to miss it 655 00:48:39.920 --> 00:48:43.639 this year. Yeah, I'm really excited to go. It's my first time 656 00:48:43.679 --> 00:48:46.639 at Oslo's top sure to get into. Yeah, you know, and I 657 00:48:46.760 --> 00:48:51.280 always loved the way they did the show on the floor in the but I 658 00:48:51.400 --> 00:48:57.760 kind of tell you this techarama's pretty close to the Yeah, and I have 659 00:48:57.840 --> 00:49:01.559 it homies. Yeah, yeah, all right, it's really cool. Everybody 660 00:49:01.559 --> 00:49:09.679 give it up for Lila Bilbria and we'll see you next time on dot net 661 00:49:09.800 --> 00:49:36.719 rocks. Dot net Rocks is brought to you by Franklin's Net and produced by 662 00:49:36.800 --> 00:49:42.760 Pop Studios, a full service audio, video and post production facility located physically 663 00:49:42.800 --> 00:49:46.719 in New London, Connecticut, and of course in the cloud online at pwop 664 00:49:47.000 --> 00:49:52.599 dot com. Visit our website at dt n et r ocks dot com for 665 00:49:52.800 --> 00:49:58.599 RSS feeds, downloads, mobile apps, comments, and access to the full 666 00:49:58.719 --> 00:50:02.280 archives going back to sh number one, recorded in September two thousand and two. 667 00:50:02.800 --> 00:50:06.960 And make sure you check out our sponsors. They keep us in business. 668 00:50:07.440 --> 00:50:10.159 Now go write some code. See you next time. You got a 669 00:50:10.239 --> 00:50:12.400 dead middle band