WEBVTT 1 00:00:01.080 --> 00:00:04.799 How'd you like to listen to dot net Rocks with no ads? Easy? 2 00:00:05.360 --> 00:00:09.480 Become a patron for just five dollars a month. You get access to a 3 00:00:09.480 --> 00:00:14.240 private RSS feed where all the shows have no ads. Twenty dollars a month, 4 00:00:14.279 --> 00:00:18.440 we'll get you that and a special dot net Rocks patron mug. Sign 5 00:00:18.519 --> 00:00:35.880 up now at Patreon dot dot NetRocks dot com. Welcome back to dot net 6 00:00:35.960 --> 00:00:39.439 Rocks. This is Carl Franklin, this is Richard Gamble and this is the 7 00:00:40.439 --> 00:00:43.320 coming out on the twenty first. So this is the last dot net Rocks 8 00:00:43.359 --> 00:00:47.439 show published before Christmas, but we've got a few more to record. Just 9 00:00:47.479 --> 00:00:51.479 Geekouts. How you been, man? You know I've been. I've been 10 00:00:51.520 --> 00:00:54.359 working on the scripture the Geekouts turns out. You know, it's a lot 11 00:00:54.399 --> 00:00:59.200 of stuff. It's been a busy year, yeah, sure has. How 12 00:00:59.240 --> 00:01:03.039 about you? What are you well? Last night I did my first recording 13 00:01:03.320 --> 00:01:07.799 of one of my songs in the studio, in the new studio. Oh, 14 00:01:07.840 --> 00:01:11.000 a new track, A new well. It's an older track that nobody's 15 00:01:11.040 --> 00:01:15.879 heard yet. But the first recording I did was all kind of discombobulated. 16 00:01:17.319 --> 00:01:22.799 It's slower, it's at one hundred bpm and and there, and with acoustics 17 00:01:22.799 --> 00:01:26.560 strumming and picking and stuff, and therefore it's very easy to get off time. 18 00:01:26.040 --> 00:01:30.000 So I want to tell you about the cool stuff that I did that 19 00:01:30.079 --> 00:01:34.239 I did this with. I recorded the acoustic guitar in the bass first, 20 00:01:34.359 --> 00:01:40.879 and then I used a studio one's bend tool to quantize the audio. What 21 00:01:41.200 --> 00:01:46.920 quantize the audio? It basically worked right out of the box, and I 22 00:01:47.040 --> 00:01:49.879 had tried it before, but it hadn't really worked well. Maybe the default 23 00:01:49.920 --> 00:01:56.439 settings are good now. But what it does is it finds the transience according 24 00:01:56.480 --> 00:02:01.719 to your quantize value eighth notes or whatever, and then it essentially moves the 25 00:02:02.000 --> 00:02:09.439 transience and either stretches or compresses the audio between the transience and basically turns it 26 00:02:09.479 --> 00:02:15.199 into quantized like it's right on the beat. Okay, so it's just like 27 00:02:15.360 --> 00:02:20.960 making the beat perfect. Yeah, And there's no it does not sound artifacted 28 00:02:21.159 --> 00:02:24.000 at all interesting to me anyway. So then I did the whole the same 29 00:02:24.039 --> 00:02:29.240 thing with the drums. For the drums, I used my iPad at the 30 00:02:29.319 --> 00:02:31.400 drum set because it's all the way across the room right right, and the 31 00:02:31.439 --> 00:02:37.840 iPad I installed this thing called studio one remote, and so I set markers 32 00:02:37.879 --> 00:02:42.520 in the three places where I wanted to record drums and I could just go 33 00:02:42.680 --> 00:02:46.759 back undo record. So I did all the recording from the drum set with 34 00:02:46.960 --> 00:02:53.439 my iPad on a music stand and I did two sections, one with brushes 35 00:02:53.479 --> 00:02:57.840 and one without. And that it took a while to record, you know, 36 00:02:57.879 --> 00:03:00.800 but it was great because I could just take another takenother take till it 37 00:03:00.840 --> 00:03:05.719 was right, and then I quantized the drums. Those drummers are never on 38 00:03:05.800 --> 00:03:09.199 beat anyway. I was pretty close, but I wasn't perfect, and I 39 00:03:09.319 --> 00:03:14.159 really wanted this to be tight interesting and it really is. So I'm very 40 00:03:14.159 --> 00:03:19.560 excited. Everything worked and yeah, what can I say? Very happy the 41 00:03:19.639 --> 00:03:25.759 new studio. Yeah, fully up, rational nice. Hey let's get started 42 00:03:25.759 --> 00:03:36.560 with better no framework? All right, man, what do you got? 43 00:03:36.599 --> 00:03:39.159 Well? I found this post on board panda dot com. Have you ever 44 00:03:39.199 --> 00:03:44.800 seen that crazy site? Yeah? Yeah, So this is thirty of the 45 00:03:44.840 --> 00:03:53.039 worst Christmas gifts people ever received, as shared in this online group. One 46 00:03:53.080 --> 00:04:02.400 of my favorites is a dish towel. I was eight, It was eight 47 00:04:02.479 --> 00:04:06.439 years old, and my parents gave me a dish towel. Nice and it 48 00:04:06.479 --> 00:04:12.240 wasn't even a new dish towel. I learn it was just like they went 49 00:04:12.280 --> 00:04:15.560 to the kitchen, found a dish towel, wrapped it up in paper, 50 00:04:15.599 --> 00:04:21.480 and then get to this eight year old kid. H it. Yeah, 51 00:04:23.000 --> 00:04:29.319 Dad, Christmas presents so hours and hours of fun and yucks and silliness. 52 00:04:29.399 --> 00:04:31.519 Silliness, Yeah, which we need once in a while, right, especially 53 00:04:31.600 --> 00:04:34.160 around the end of the year. It's like we've been working hard all year. 54 00:04:34.959 --> 00:04:39.240 Now let's take some time. Awesome. Anyway, That's what I got. 55 00:04:39.399 --> 00:04:42.879 Who's talking to us? Richard grabbed a comment of of Show eighteen seventy 56 00:04:42.879 --> 00:04:46.720 three, the one we just published a little while ago with Leah Melanino from 57 00:04:46.800 --> 00:04:50.439 NDC and Portal. We were talking about sustainable development. You know, I'm 58 00:04:50.560 --> 00:04:54.160 just thinking about the energy can say. We've got a great comment here from 59 00:04:54.199 --> 00:04:56.279 Jackie. You said, Yeah, Hi, Carlin Richard. I want to 60 00:04:56.279 --> 00:05:00.360 express my thanks for tackling this important subject. This conversation resonated with me a 61 00:05:00.360 --> 00:05:03.040 lot, as I'm a software engineer working with Azure Cloud technologies. In our 62 00:05:03.079 --> 00:05:08.000 related note, I'd like to share my recent experience with an iPhone. I've 63 00:05:08.079 --> 00:05:11.759 always been an Android user, primarily do the perception that iPhones are overpriced to 64 00:05:11.839 --> 00:05:15.560 non standard devices. I think Apple pretty much dictates the standard. Just look 65 00:05:15.560 --> 00:05:18.959 at out what happened they did to RSS. Often seen as more suitable for 66 00:05:19.040 --> 00:05:24.040 users seeking opinionated ux like my mother, or a symbol of social status like 67 00:05:24.079 --> 00:05:29.439 my brother Jackie, like your diss in the fan, right it's Christmas. 68 00:05:30.680 --> 00:05:32.839 However, when my Google pixel broke and I had to let it in for 69 00:05:32.920 --> 00:05:36.959 service, I was left with no choice but to use a backup iPhone fives, 70 00:05:38.000 --> 00:05:41.000 a decade old model offered by my brother. See he may be a 71 00:05:41.040 --> 00:05:44.240 social status seeker, but at least they'll give you a spare iPhone. It's 72 00:05:44.279 --> 00:05:46.680 not a bad guy. He's not a bad guy. Yeah, you had 73 00:05:46.720 --> 00:05:50.600 to reassess this relationship, man. And to my surprise, the iPhone five 74 00:05:50.800 --> 00:05:56.720 S was still receiving security updates and all my essential mobile apps function flawlessly on 75 00:05:56.800 --> 00:06:00.319 it. This is something I couldn't even say about some five year old low 76 00:06:00.399 --> 00:06:04.040 end Android devices, not that any iPhone is a low end device. The 77 00:06:04.079 --> 00:06:10.279 only drawback was battery life. Yeah, battery is not going to be great. 78 00:06:10.759 --> 00:06:14.480 This experience made me realize that even though iPhones may not be considered entirely 79 00:06:14.519 --> 00:06:17.879 sustainable, due to a lack of adherence to certain standards like USBC or Place 80 00:06:17.879 --> 00:06:23.399 Little batteries. Their longevity is great right up until you drop them. Therefore, 81 00:06:23.439 --> 00:06:27.120 I've decided to offer an iPhone model with a USB C port. Thank 82 00:06:27.160 --> 00:06:32.720 you EU who demanded that Apple start using USB c's. This all allow me 83 00:06:32.720 --> 00:06:35.600 to utilize most of my Android gadgets and extended lifespan on the phone. And 84 00:06:35.600 --> 00:06:40.399 thanks again for your dedication to the dot net Rocks podcast. It has been 85 00:06:40.480 --> 00:06:44.639 a valuable source of knowledge and inspiration, especially during my career transition to dot 86 00:06:44.680 --> 00:06:47.399 net c sharp looking forward to future episode, and I can add to that 87 00:06:49.040 --> 00:06:57.240 the iPhones are inherently more secure than Android phones just because Apple is such a 88 00:06:57.279 --> 00:07:00.959 closed system. That's one of the benefits actually of having It's sort of security 89 00:07:00.000 --> 00:07:02.839 by obscurity. Nobody wants to try and act them. Really, I don't 90 00:07:02.839 --> 00:07:08.959 know about obscurity, but security by by iron fist control over everything, everything 91 00:07:08.959 --> 00:07:13.759 that goes on that phone, right, So yeah, I like I That's 92 00:07:13.959 --> 00:07:15.560 that's one of the reasons why I have an iPhone. And the guys at 93 00:07:15.600 --> 00:07:19.040 Security this week Patrick Hanes and Twain Laflat say the same thing. Yeah, 94 00:07:19.040 --> 00:07:23.560 I get the iPhone. The phone is more secure the Android phone. I 95 00:07:23.560 --> 00:07:27.240 mean under the hood androids Linux. Everybody knows Linux is terrible, especially Daniel. 96 00:07:27.240 --> 00:07:30.199 He knows that that. I'm just gonna poke at the Linux guy the 97 00:07:30.319 --> 00:07:34.360 whole day. So we're gonna do but we're gonna poke at each other at 98 00:07:34.439 --> 00:07:40.079 Christmas time. Come on, hey, Jackie, thank you so much for 99 00:07:40.120 --> 00:07:42.519 your comment. Glad you really liked the show. And a copy of music 100 00:07:42.519 --> 00:07:44.680 co Buy is on its way to a un If you'd like a copy of 101 00:07:44.759 --> 00:07:47.120 music Cobe, I write a comment on the website at dotnet rocks dot com 102 00:07:47.279 --> 00:07:49.480 or on the facebooks. We publish every show there, and if you comment 103 00:07:49.519 --> 00:07:51.959 there and read on the show, we'll send to your copy of music code 104 00:07:51.959 --> 00:07:55.279 by and you can follow us on Twitter if you like. But the real 105 00:07:55.319 --> 00:08:00.600 fun happens. I'm mastadon, I'm at Carl Franklin at tech hub Social, 106 00:08:00.680 --> 00:08:03.439 and I'm Rich Campbell at masodon do Social send us a two You might get 107 00:08:03.480 --> 00:08:07.720 a mug if we read it on the show. Think so pretty sure you 108 00:08:07.759 --> 00:08:09.959 won't? Yeah, pretty sure? You get a copy of music opy. 109 00:08:09.199 --> 00:08:11.560 Kind of how that worked? Did I say mug? He did? All 110 00:08:11.680 --> 00:08:15.399 right? Let me say that again, Brandon, all Right, senas, 111 00:08:15.439 --> 00:08:16.480 although that was kind of funny, we might want to leave it in. 112 00:08:16.519 --> 00:08:22.120 It's kind of funny. I'm pretty sure you won't. Let's leave it in. 113 00:08:22.199 --> 00:08:26.360 It's Christmas time, what the hell? All right, let me bring 114 00:08:26.399 --> 00:08:33.399 on our guest, Daniel Marbach as a distinguished Microsoft MVP and software maestro at 115 00:08:33.440 --> 00:08:37.159 Particular Software, Daniel knows a thing or two about code. By day, 116 00:08:37.399 --> 00:08:43.840 he's a devoted dot net crusader espousing the virtues of message based systems. By 117 00:08:43.919 --> 00:08:48.360 night, he's racing against his own mischievous router hack, committing a bevy of 118 00:08:48.440 --> 00:09:00.720 performance improvements before the clock strikes midnight and he turns into a pumpkin. Yes, 119 00:09:00.879 --> 00:09:05.120 exactly that. What's a router hack? What are you going to do? 120 00:09:05.240 --> 00:09:11.360 Low wrt? No, it's just it's a very simple sort of trick 121 00:09:11.399 --> 00:09:16.600 because I've been contributing to open source and various things, and I'm just I 122 00:09:16.759 --> 00:09:20.480 just like to spend some time with code because I feel like it's sharp and 123 00:09:20.519 --> 00:09:24.759 sort of my understanding of just actor i'm working with. And I had appeared 124 00:09:24.799 --> 00:09:28.639 in my life where I just couldn't stop right because it's like and then it 125 00:09:28.759 --> 00:09:31.039 was at first it was like one am, two am, three am in 126 00:09:31.080 --> 00:09:37.919 the morning. And then luckily I always basically my only one rule was that 127 00:09:37.960 --> 00:09:41.200 I will not extend my alarm clock to a later point in the day, 128 00:09:41.679 --> 00:09:43.960 so otherwise my days would have shifted. But at some point I was like, 129 00:09:45.000 --> 00:09:46.559 Okay, that's it. I need to change something in my life. 130 00:09:46.919 --> 00:09:52.480 So basically I'm switching off the internet around midnights at my whole Oh and that's 131 00:09:52.519 --> 00:09:58.480 when you have to hack your router so exactly, because that usually when you're 132 00:09:58.559 --> 00:10:01.879 when you're like in the middle of thing and is I should Google bing this 133 00:10:01.080 --> 00:10:05.080 or whatever you're using, right, you're like, ah, the internet doesn't 134 00:10:05.080 --> 00:10:07.639 work anymore. Ah, it has time until tomorrow and then I switch off 135 00:10:07.679 --> 00:10:11.240 and just go to bed. So yeah, I never worked that way for 136 00:10:11.360 --> 00:10:13.080 me. I had to turn the bugger back. I put the same rule 137 00:10:13.120 --> 00:10:18.240 in place because I had teenage daughters. And now you hear the audible groans 138 00:10:18.279 --> 00:10:20.600 at midnight, right, it's like, oh, yeah, everybody's in bed, 139 00:10:20.879 --> 00:10:24.039 sure they are. But me, it's like, I guess got to 140 00:10:24.120 --> 00:10:26.960 you know, finishing up to the go to push the code fails. Yeah, 141 00:10:28.080 --> 00:10:31.120 you're like, uh, it's me. I got to fix the rider. 142 00:10:31.360 --> 00:10:37.559 So your performance wonk, are you? Well? Actually, I work 143 00:10:37.639 --> 00:10:43.279 for the company called Particular Software. I guess you had Dhan on the police 144 00:10:43.960 --> 00:10:48.399 before, right, so you know we had a few particular rights and a 145 00:10:48.440 --> 00:10:52.559 few others and yeah, so I basically my day to day shop is I'm 146 00:10:52.879 --> 00:10:58.720 building a robust and reliable frameworks and libraries for people that sort of want to 147 00:10:58.759 --> 00:11:03.639 sort of build it distributed systems primarily based on messaging stuff like ASH, service 148 00:11:03.679 --> 00:11:09.840 bus, SQS, SNS, storage queues in the old days, God forbids 149 00:11:09.960 --> 00:11:13.519 MSMQ. Right, but it's still it's still out there thriving. Surprise, 150 00:11:13.559 --> 00:11:20.600 surprise, It's quite heavily used there in the industry. Yeah, and it's 151 00:11:20.600 --> 00:11:24.279 one of the great things is feature complete, right, it it's just there 152 00:11:24.360 --> 00:11:28.919 if you're still running on Windows and other on Linux like I do. And 153 00:11:35.639 --> 00:11:39.320 one of the things that we do there is we want to make sure that 154 00:11:39.840 --> 00:11:43.559 the customers that they're using in service bus can focus on their just writing their 155 00:11:43.600 --> 00:11:48.600 business code, don't need to write any plumbing code, and that stuff should 156 00:11:48.679 --> 00:11:56.000 run as efficiently as possible. Right, So I guess performance throughput was always 157 00:11:56.039 --> 00:11:58.879 sort of right from the center sort of in my day to day job. 158 00:12:00.039 --> 00:12:05.240 But they also care a lot about it because I believe especially it came us 159 00:12:05.320 --> 00:12:07.879 out in the comment right that you read out Richard. There's like, if 160 00:12:07.879 --> 00:12:11.279 you're targeting the cloud, or if you're shifting the cloud, or if you're 161 00:12:11.320 --> 00:12:18.039 already are in the cloud, sort of you are basically built by the amount 162 00:12:18.039 --> 00:12:22.399 of resources that you're using in the cloud. Yeah, so there's a direct 163 00:12:22.720 --> 00:12:28.320 revenue relationship to that consumption, which it leaves giuse of some kind of incentive. 164 00:12:28.960 --> 00:12:31.080 Yeah. Correct, You put down your credit card and then you get 165 00:12:31.120 --> 00:12:37.120 surprised at the end of the month. Especially I'm surprising yourself as one thing. 166 00:12:37.240 --> 00:12:41.399 Surprising the CFO is another. That's a very loud noise from a large 167 00:12:41.440 --> 00:12:46.720 office. Yeah, that's the question. I mean, there's so many ways 168 00:12:46.720 --> 00:12:50.799 to tweak performance. One is just by using the latest netstack correct, and 169 00:12:50.840 --> 00:12:56.840 then you know, keeping your new get packages updated. But on top of 170 00:12:56.879 --> 00:13:00.000 that, you know, are what are kind of knobs are you post? 171 00:13:00.200 --> 00:13:03.399 Are you pulling software knobs, hardware knobs? All of the above. So 172 00:13:03.480 --> 00:13:07.600 let me cook before I answer your question. Let me quickly go to what 173 00:13:07.639 --> 00:13:11.879 you said about updating the dot net the dot net version. That's actually really 174 00:13:11.960 --> 00:13:18.600 interesting because Microsoft has this block series where they essentially talk about their teams migrating 175 00:13:18.639 --> 00:13:24.120 to for example, from dot Net framework to newer dot Net versions or from 176 00:13:24.159 --> 00:13:26.759 dot net six net eight, And one of the cool things they blocked there 177 00:13:26.840 --> 00:13:35.279 is they the Microsoft Teams Infrastructure team, they basically migrated from dot Net framework 178 00:13:35.440 --> 00:13:41.720 to dot net six and just by basically migrating to that LTS version of dot 179 00:13:41.759 --> 00:13:46.399 Net, they were actually able to sort of buy almost twenty four percent reduce 180 00:13:46.480 --> 00:13:50.440 their monthly cost expenditure in asure, which is pretty amazing if you think that's 181 00:13:50.519 --> 00:13:54.759 amazing about that, right, And that's definitely one way to do it. 182 00:13:54.799 --> 00:13:58.000 So I always encourage people to sort of if they can stay up to date 183 00:13:58.120 --> 00:14:03.559 with the with the latest dot Net versions. Definitely, Yeah, I'm a 184 00:14:03.559 --> 00:14:07.240 big fan of that series on the dot net blog, just because you know, 185 00:14:07.440 --> 00:14:11.720 you talk about like the teams guys migrating to a new version of dot 186 00:14:11.799 --> 00:14:15.840 net and the benefits they got from it, and also the things they struggled 187 00:14:15.879 --> 00:14:18.159 with on that, like what problems they had. But to me, more 188 00:14:18.159 --> 00:14:22.159 than anything, it's like, hey, you know, if these guys got 189 00:14:22.200 --> 00:14:26.200 this kind of benefit and we're able to move that big an app you're going 190 00:14:26.279 --> 00:14:30.759 to be okay, like you can do it. Yeah, absolutely. But 191 00:14:31.200 --> 00:14:33.559 to come back to your question, Carl, I think one of the things 192 00:14:33.559 --> 00:14:37.080 that I try to apply to sort of in my thinking is I want to 193 00:14:37.120 --> 00:14:43.480 make explicit trade off as I'm going with things, and so that means I 194 00:14:43.519 --> 00:14:46.759 want to be aware of sort of is this code going to be executed on 195 00:14:46.799 --> 00:14:50.360 the hot path or at scale, right, so, and how many times 196 00:14:50.399 --> 00:14:54.320 a second is that going to be or is it just something that runs on 197 00:14:54.360 --> 00:14:58.879 a sort of a background chop once a day or twice a day or because 198 00:14:58.960 --> 00:15:03.840 then usually if it's just executed once or twice a day, it doesn't really 199 00:15:03.879 --> 00:15:09.240 matter that much whether it's super fast or not. But then when it's executed 200 00:15:09.279 --> 00:15:15.600 on the hot path potentially hundreds and thousand times per second, then it's usually 201 00:15:15.600 --> 00:15:20.919 good to sort of become more of performance aware, but performance aware. Unfortunately, 202 00:15:22.039 --> 00:15:24.480 this is also something that gets thrown around quite a lot sort of in 203 00:15:24.519 --> 00:15:30.519 the industry, and then everyone assumes everyone knows what performance awareness means, right. 204 00:15:30.919 --> 00:15:33.879 But one of the things that I struggled with was where should I even 205 00:15:33.919 --> 00:15:39.960 get started to become performance aware, Because apparently if you go down sort of 206 00:15:39.000 --> 00:15:46.440 the literature of looking at benchmarking and performance optimizations. You can actually go from 207 00:15:46.639 --> 00:15:52.080 just doing little things up to setting up your entire CICD pipeline with dedicated hardware 208 00:15:52.720 --> 00:15:58.840 doing regression testing. Right, But usually we don't start there. Usually we 209 00:16:00.120 --> 00:16:03.799 or somewhere else. And that's one of the things that I'm trying to apply. 210 00:16:03.879 --> 00:16:06.960 So usually I ask myself a bunch of questions when I look at the 211 00:16:07.000 --> 00:16:10.440 code, right, So, for example, I I go and look for, 212 00:16:11.120 --> 00:16:15.039 well, what could be the CP and memory characteristics? What could that 213 00:16:15.080 --> 00:16:18.759 be for the specific line that I'm looking at that I know is on the 214 00:16:18.799 --> 00:16:22.759 hot path? And then I usually start thinking about, so, are there 215 00:16:22.759 --> 00:16:25.960 any sort of low hanging fruits that I can sort of apply to this, 216 00:16:26.240 --> 00:16:30.639 maybe do some more efficient string splitting options and stuff like that that I know 217 00:16:30.799 --> 00:16:36.559 from reading the performance blog posts, and I can apply there and that is 218 00:16:36.600 --> 00:16:40.480 and I'm presumed that's Stephen Taub's post, Yes, of course, yeah, 219 00:16:40.600 --> 00:16:45.080 that the book the Book of the Taub right y, yeah, exactly. 220 00:16:45.759 --> 00:16:48.840 And then most of the time, right, so there are a few tricks 221 00:16:48.879 --> 00:16:52.960 you can apply. So, for example, if you're allocating a byttery, 222 00:16:52.279 --> 00:16:56.799 right, and you know that you you actually just need it for every iteration. 223 00:16:57.039 --> 00:17:00.679 What you can do is you can sort of move that away from the 224 00:17:00.679 --> 00:17:04.640 hot path allocated once and clear it and then you're no longer having allocations that 225 00:17:04.680 --> 00:17:10.839 you're executing like on the hot path, and then the garbage collection doesn't have 226 00:17:10.920 --> 00:17:14.759 to clean up a lot of things, and there things get get get better 227 00:17:14.759 --> 00:17:18.720 as well. But then what how do you measure that, Daniel, Like, 228 00:17:19.000 --> 00:17:22.440 how do you know it got? Like, that's a tricky one that 229 00:17:22.720 --> 00:17:26.720 the impact of garbage collection, Like, how would I measure that I reduce 230 00:17:26.759 --> 00:17:30.240 the amount of garbage collecting? Yes, that's that's a that's a very good 231 00:17:30.319 --> 00:17:37.559 question. So usually what I do is before I so I call this performance 232 00:17:37.599 --> 00:17:41.119 loop. So, for lack of a better term, I call the performance 233 00:17:41.160 --> 00:17:45.839 loop. So what I usually start with is when I have ip I hypothesis 234 00:17:45.880 --> 00:17:52.799 about a piece of code that needs to that it creates uh garbage. Then 235 00:17:52.880 --> 00:17:57.400 I what I do is I write a test harness, and essentially what that 236 00:17:57.519 --> 00:18:02.559 harness sort of does. It sort of takes whatever I'm looking at, takes 237 00:18:02.559 --> 00:18:07.519 it into specific context of my suspicions, and then it executes it in that 238 00:18:07.559 --> 00:18:14.200 specific scenarios. And then I attached profilers to that piece of code, And 239 00:18:14.720 --> 00:18:19.079 what I usually do is I take at least a memory snapshot and also create 240 00:18:19.480 --> 00:18:25.599 a CPO snapshot. And of course, if you have an io bond system, 241 00:18:26.079 --> 00:18:29.920 like you have a database in place, or you have HDP called stuff 242 00:18:29.960 --> 00:18:33.000 like that, you also want to look at your you want to do io 243 00:18:33.119 --> 00:18:37.640 based profiling as an example, right, because usually when you look at your 244 00:18:37.640 --> 00:18:41.920 iosystem like the database, that's you can basically achieve orders of magnitude of performance 245 00:18:41.960 --> 00:18:48.640 improvement by tweaking your sequel querits right before you will start thinking about memory allocations 246 00:18:48.680 --> 00:18:52.119 and stuff like that. But assuming you have sort of removed that part, 247 00:18:52.640 --> 00:18:59.720 then essentially those sort of profiler snapshots give you an indication of where could you 248 00:18:59.759 --> 00:19:04.240 voteocus on And here comes the next problem when you attach your profiler is that 249 00:19:04.720 --> 00:19:11.359 you might see lots and lots of allocations from different subsystems and component on that 250 00:19:11.440 --> 00:19:17.519 specific cultree and where should you even start? So I usually try to sort 251 00:19:17.519 --> 00:19:23.799 of apply a combination of I call it the one percent improvement philosophy vverasus deliberate 252 00:19:23.960 --> 00:19:32.119 contextual based optimizations that I want that I want to do so because for example, 253 00:19:32.200 --> 00:19:37.000 I believe that if you do enough little performance optimizations over time, that's 254 00:19:37.000 --> 00:19:42.559 the one percent improvement sort of philosophy, then eventually they will end up making 255 00:19:42.599 --> 00:19:47.359 a big impact. And we can see that Microsoft applies it as well to 256 00:19:47.400 --> 00:19:49.559 do dot net run time. Right, They're doing lots and lots and lots 257 00:19:49.559 --> 00:19:55.279 and lots of small changes all over the place, and the compounding effect of 258 00:19:55.359 --> 00:19:59.920 these changes they essentially are massive when you look at them in sort of the 259 00:20:00.039 --> 00:20:03.720 greater scheme of things, right, And what profiling tools are you using here? 260 00:20:03.839 --> 00:20:07.599 Is this just like the built in profiler. Okay, you made a 261 00:20:07.680 --> 00:20:15.440 joke about me. You're running on Linux, right, so so well, 262 00:20:15.519 --> 00:20:21.160 so I'm a big fan of the sort of chet Brains tools I've been using 263 00:20:21.400 --> 00:20:26.799 for a year alone. So I'm using usually dot trace and dot memory sort 264 00:20:26.799 --> 00:20:33.640 of those two tools, and Writer also has some built in analysis, so 265 00:20:33.680 --> 00:20:37.720 for example, they do when you execute your tests or you execute your solution. 266 00:20:37.839 --> 00:20:41.039 They also do some dynamic program analysis where they show you sort of the 267 00:20:41.079 --> 00:20:45.920 allocations that your stuff had or the the CP that it wasted. So I 268 00:20:47.000 --> 00:20:51.400 tried to use those tools, but primarily dot trace and dot memory to get 269 00:20:51.599 --> 00:20:56.240 to get an overview of what is actually going going on, right, And 270 00:20:56.759 --> 00:21:00.599 I mean, yeah, there are built in profilers, but if you're willing 271 00:21:00.640 --> 00:21:04.079 to pay for one, there are better ones. Yeah, I mean, 272 00:21:04.400 --> 00:21:07.160 to be fair, a visual Studio is great, right, It's like visual 273 00:21:07.200 --> 00:21:11.880 Studio has depending on the license, I'm not entirely familiar with the licensing terms 274 00:21:11.920 --> 00:21:17.400 there, but it has great tooling built in. Or if you are sort 275 00:21:17.400 --> 00:21:22.240 of very advanced and mostly on Windows, you can also use perfew, right. 276 00:21:22.319 --> 00:21:26.960 So perfew is a very powerful tool that you can use, although I 277 00:21:27.000 --> 00:21:33.079 struggle with it a bit, I must say every time it's like using WINDPG. 278 00:21:33.440 --> 00:21:36.720 Every time I use these tools, I have to sort of get some 279 00:21:36.839 --> 00:21:41.200 cheat sheets onto my machine in order to remember the complex commands. And that's 280 00:21:41.559 --> 00:21:45.200 one of the benefits those too. There are tools you need to learn, 281 00:21:45.440 --> 00:21:49.519 and I mean I suspect you use them more than most people. And if 282 00:21:49.519 --> 00:21:56.359 you can't keep them in your head, then nobody can. I mean, 283 00:21:56.400 --> 00:21:59.519 I've done a lot of performance tuning over the years, and people already surprised. 284 00:21:59.519 --> 00:22:00.680 It's like why are are you reading the docs? Like don't you know 285 00:22:00.759 --> 00:22:04.839 this? It's like, listen, there's a lot of knobs on these things, 286 00:22:04.920 --> 00:22:07.720 and if you don't go through the steps you can waste a lot of 287 00:22:07.759 --> 00:22:11.799 time, yes, and actually wasting a lot of time. That's actually a 288 00:22:11.920 --> 00:22:15.359 very good, good sort of comment that you made there, because I think 289 00:22:15.440 --> 00:22:22.759 even if even if you are more familiar with performance optimizations and benchmarking, it's 290 00:22:22.799 --> 00:22:26.000 like and profiling at the end of the day, that's not my day job. 291 00:22:27.319 --> 00:22:33.680 My day job is building robust and reliable messaging frameworks and middlewares and the 292 00:22:33.759 --> 00:22:37.599 platform at particular, and not doing performance optimizations all day long. That's not 293 00:22:37.680 --> 00:22:41.480 my job description, right, And I guess that many people that are also 294 00:22:41.519 --> 00:22:45.680 listening to that podcast have the same thing, right, They would like to 295 00:22:45.759 --> 00:22:52.480 dive into profiling and benchmarking performance optimizations, but they only have a limited budget 296 00:22:52.920 --> 00:22:56.480 in order to spend on those types of things. And that's why I always 297 00:22:56.519 --> 00:23:03.559 recommend start with a test harness, reproduce the scenario, attach your profiler, 298 00:23:03.599 --> 00:23:07.799 and then use your domain knowledge of the things that you're working to basically sift 299 00:23:07.880 --> 00:23:12.079 through the noise of allocations and CPU and on the call stack and then figure 300 00:23:12.079 --> 00:23:17.920 out Okay, probably here is where we can make the biggest impact on sort 301 00:23:17.960 --> 00:23:23.000 of reducing the numbers of CPU cycle spent or reducing the numbers of garbage allocations 302 00:23:23.039 --> 00:23:27.599 that are happening there. But sometimes, like I said before, it's also 303 00:23:29.240 --> 00:23:33.079 where you think you have the most knowledge in and then applying sort of the 304 00:23:33.400 --> 00:23:37.319 one percent improvement over time in order to sort of make things better and better 305 00:23:37.359 --> 00:23:45.160 and not trying to gold plate so everything out of an existing code path. 306 00:23:45.400 --> 00:23:51.000 But I guess we now have sort of touched a little bit on the how 307 00:23:51.000 --> 00:23:53.039 would I even know where to get started? Right? We talked about profiling, 308 00:23:53.079 --> 00:23:57.759 We talked about doing CPO memory at least right always get sort of two 309 00:23:57.839 --> 00:24:03.039 views on the code base. But the next thing is then I mean improvements. 310 00:24:03.559 --> 00:24:07.920 Of course, that goes more into the territory of knowing your stack, 311 00:24:07.400 --> 00:24:11.640 knowing your language, knowing the libraries that you work with. Like Carl said, 312 00:24:11.759 --> 00:24:15.720 also looking for has the library and you release that we can pull in 313 00:24:15.920 --> 00:24:19.920 or maybe reach out to the maintainers and say, hey, by the way, 314 00:24:21.079 --> 00:24:26.119 we did the profiling snapshot and we found out that this library allocates that 315 00:24:26.240 --> 00:24:30.720 much amount of memory. And guess what when you reach out with a profiler 316 00:24:30.799 --> 00:24:37.000 snapshot to third party tooling providers, they are like super happy because you're then 317 00:24:37.039 --> 00:24:41.759 in the one percent sort of customers and then it's like, hey, now 318 00:24:41.759 --> 00:24:45.559 we have data from the customer that we can see what's actually going on. 319 00:24:47.680 --> 00:24:52.240 You've now described a workload in a meaningful way to them correct correct, right, 320 00:24:52.839 --> 00:24:59.119 And so for example, I myself have done that as well. At 321 00:24:59.119 --> 00:25:03.839 some point I st over sort of memory inefficiencies in the ash service bus s 322 00:25:03.920 --> 00:25:10.000 d K, and then I saw I had a hunch. I wrote the 323 00:25:10.039 --> 00:25:15.359 test harness attached profiler was able to show that when you access the bodies of 324 00:25:15.400 --> 00:25:21.240 a service bus message that allocates unnecessary memories every time you essentially access the body. 325 00:25:21.799 --> 00:25:25.680 I was able to sort of show the profiler snapshots show the memories. 326 00:25:25.680 --> 00:25:29.480 And then I, even because I was lucky, I already knew the library 327 00:25:29.480 --> 00:25:33.279 a little bit, I guess, and I was able to contribute a fix 328 00:25:33.799 --> 00:25:40.279 for that memory allocation problem to the Azure service bus. Yeah that's a good 329 00:25:40.279 --> 00:25:49.960 way to get an email from Clements. Yeah, yeah, I've I've I've 330 00:25:51.039 --> 00:25:55.799 had comments from from Clements originally on on some of my pull requests as well. 331 00:25:57.359 --> 00:26:00.400 And then and there in lies the point, like you get into it. 332 00:26:00.400 --> 00:26:03.000 And of course the kicker, of course is at its open source, 333 00:26:03.039 --> 00:26:07.160 so you could just contribute a fix yes, yes, yeah, but I 334 00:26:07.160 --> 00:26:11.400 guess, I mean, nobody expects you to write but at least having a 335 00:26:11.440 --> 00:26:15.039 memory profile or a CPU profile, and like you said, Richard, showing 336 00:26:15.079 --> 00:26:18.279 what's going on in production, is it. Yeah, I just think it's 337 00:26:18.279 --> 00:26:22.960 always challenging to get down to the brass tax like that. To me, 338 00:26:23.039 --> 00:26:27.319 most of my profiling experience has been trying to optimize an e commerce site where 339 00:26:27.359 --> 00:26:30.079 it's like we're just you know, we're running. We're now looking at a 340 00:26:30.119 --> 00:26:34.680 buy of more servers because the site's so busy, like an optimization can mean 341 00:26:34.720 --> 00:26:41.920 a lot of money and the prob I was a nance guy at the time, 342 00:26:41.599 --> 00:26:45.000 and that was the tool that showed me. I mean, this may 343 00:26:45.039 --> 00:26:48.000 not have been it was always that balance between this is a very complicated method 344 00:26:48.160 --> 00:26:52.240 and so it's consuming a lot of resources and it's a very simple method, 345 00:26:52.359 --> 00:26:56.400 but it's called hundreds of thousands of times, and so the fact that the 346 00:26:56.440 --> 00:27:03.960 tool would help sort out that weight of often called, and so were thought 347 00:27:03.160 --> 00:27:10.279 minute optimizations will make big differences versus rarely called but complex enough that you will 348 00:27:10.279 --> 00:27:14.599 get some return on that, you know, didn't I'd never worried about optimizing 349 00:27:14.640 --> 00:27:18.559 in mind calls because it just didn't call that often. But all of that 350 00:27:18.640 --> 00:27:26.200 mainstream shopping cart recommendation engine, you know, custom render pieces, ad pieces 351 00:27:26.240 --> 00:27:29.680 like those, were all the things where it's like these get called a lot 352 00:27:29.960 --> 00:27:33.920 even though they don't look that big, and just playing with string compilation like 353 00:27:34.039 --> 00:27:40.279 those kinds of things made a huge difference in the end. But the challenge 354 00:27:40.279 --> 00:27:41.240 I think for a lot of folks is they just want to get into the 355 00:27:41.240 --> 00:27:47.319 code and this idea of you snap the harness on first and get a baseline 356 00:27:47.359 --> 00:27:49.839 set of profiles in place, and then as you said, the magic word, 357 00:27:49.880 --> 00:27:53.200 the hypothesis says, if we do an optimization here, it'll make a 358 00:27:53.240 --> 00:28:00.359 difference. Now you go tinker, then run the benchmarks again and it's they 359 00:28:00.400 --> 00:28:04.640 did we make a difference? And King and if you didn't revert. Yes, 360 00:28:06.200 --> 00:28:10.920 And because there's no performance code i've ever written them was easier to read 361 00:28:10.920 --> 00:28:14.920 than the original. Yes, ever, absolutely true, And I think what 362 00:28:15.319 --> 00:28:18.319 you said is super crucial. That is sort of the performance look that I 363 00:28:19.359 --> 00:28:25.519 that I apply. Is so when you have the harness and then usually that 364 00:28:25.559 --> 00:28:27.880 reproduces this scenario right, and then like you said, you do the improvements 365 00:28:27.920 --> 00:28:33.559 that might be several iterations of ideas that you tinker around with, and then 366 00:28:33.920 --> 00:28:38.319 you might execute several benchmarks to sort of look at those sort of optimizations that 367 00:28:38.359 --> 00:28:44.599 you're doing. Maybe it's even several micro benchmarks sort of measuring sort of little 368 00:28:44.640 --> 00:28:48.720 improvements in that call stack that you came up with during that tinkering phase. 369 00:28:48.240 --> 00:28:52.599 And then at the end what you do is you bring it back into that 370 00:28:52.680 --> 00:28:56.000 harness, right, and then you look at sort of the end to end 371 00:28:56.039 --> 00:29:02.200 profile again where you look at again the CPU and memory profile at least actually 372 00:29:02.519 --> 00:29:04.599 to actually see the before and after right, and then you see on your 373 00:29:04.640 --> 00:29:11.039 graphs, oh, we spend six hundred and fifty megabytes of memory on that 374 00:29:11.359 --> 00:29:15.640 specific scenario before. Now we're at six hundred. Now you know that you 375 00:29:15.759 --> 00:29:22.000 actually have gained something, and you also have the numbers that from the benchmarks 376 00:29:22.000 --> 00:29:29.279 that you run against each individual part of the callstack that you try to optimize. 377 00:29:30.200 --> 00:29:33.119 I have one question to you, Richard. You said you use the 378 00:29:33.240 --> 00:29:37.000 NDS profiler. Do you also happen to sort of because I had a period 379 00:29:37.680 --> 00:29:41.759 where I used several tools because I had sort of I know it when I 380 00:29:41.799 --> 00:29:45.720 see it type of investigations, So I used several tools that have had different 381 00:29:45.759 --> 00:29:51.599 sort of dashboards and overviews that sometimes gave me sort of a slightly different view 382 00:29:52.240 --> 00:29:56.039 based on the preferences of the tooling against the test harness, and then I 383 00:29:56.079 --> 00:30:00.160 was like, oh, there it is. Yeah, I mean different problems 384 00:30:00.160 --> 00:30:04.359 in different spaces. And granted a lot of my experiences are from a while 385 00:30:04.440 --> 00:30:08.960 ago where there wasn't as many tools as there are today, but yeah, 386 00:30:10.079 --> 00:30:15.720 definitely there's a difference between tweaking a piece of code that you know sits in 387 00:30:15.759 --> 00:30:19.119 a call stack for a web page and understanding a sort of end to end 388 00:30:19.200 --> 00:30:23.799 run where it's like, oh, the real problem here is that there's a 389 00:30:23.839 --> 00:30:29.759 repeated call to a database enough that it's doing a reauthenticate in the middle, 390 00:30:30.039 --> 00:30:33.359 or it's forcing a recompile of a stored procedure. By the way, on 391 00:30:33.440 --> 00:30:37.599 the day you find one of those from a method call and you got all 392 00:30:37.599 --> 00:30:40.720 the way down to but we call it this many times and so forces is 393 00:30:40.799 --> 00:30:45.240 recompiled and that creates this overhead like that's a very good day because those are 394 00:30:45.359 --> 00:30:49.400 hard to find, like just a tough place get too, but you know 395 00:30:49.559 --> 00:30:56.640 your point's well taken. There are Each tool provides its own viewed to that 396 00:30:56.880 --> 00:31:00.319 and we ended up I think it was a dying trace where we were only 397 00:31:00.359 --> 00:31:06.480 able to see all this is a multiple database interaction problem before we really saw 398 00:31:06.559 --> 00:31:11.400 the behavior correctly. And guys, I want to pause for just a few 399 00:31:11.440 --> 00:31:18.279 moments for these very important messages, and we're back. It's dot net rocks. 400 00:31:18.279 --> 00:31:22.640 I'm Carl Franklin, that's Richard Campbell, hey, and that's Daniel Marbach. 401 00:31:22.799 --> 00:31:30.240 We're talking about performance, squeezing performance out of our applications. And Daniel, 402 00:31:30.319 --> 00:31:34.039 right before the break, you were going to make a point about memory 403 00:31:34.079 --> 00:31:40.319 allocations. Memory allocations. Yeah, So what I wanted to say is I 404 00:31:40.880 --> 00:31:44.480 feel like I need to sort of clarify one thing because I talked a lot 405 00:31:44.519 --> 00:31:48.880 about sort of memory allocations. I also sort of highlighted a little bit the 406 00:31:48.279 --> 00:31:52.720 CPU stuff, right, But people might get sort of the message that all 407 00:31:52.759 --> 00:31:59.319 that matters is memory allocations, and I definitely don't want to say that way 408 00:31:59.599 --> 00:32:02.440 because I just feel like for me, I've always sort of started looking at 409 00:32:02.440 --> 00:32:07.720 memory allocations because I've seen that these are the areas in the applications that I 410 00:32:07.839 --> 00:32:10.839 worked with and the systems that I worked with where I can make the sort 411 00:32:10.880 --> 00:32:16.400 of the biggest impact to reduce the GC overhead without sort of going into sort 412 00:32:16.400 --> 00:32:22.079 of the algorithmic complexity and stuff like that that sometimes comes with tweaking algorithms where 413 00:32:22.240 --> 00:32:28.440 CPU cycles are spent. And I remember, I don't know the exact quote, 414 00:32:28.440 --> 00:32:31.920 but David Fowler once sort of tweeted or is it still called tweet, 415 00:32:31.960 --> 00:32:38.880 I don't know, but he shouted into the interwebs that essentially, apparently memory 416 00:32:38.920 --> 00:32:43.960 stream two array and other sort of two array are still the biggest source of 417 00:32:44.039 --> 00:32:49.559 memory allocations in dot net systems out there, which kind of shows how important 418 00:32:50.000 --> 00:32:55.559 sort of thinking about memory allocations actually and certainly in this case of scale that 419 00:32:55.680 --> 00:32:59.039 you know, when we're dealing with lots of federations. Again, I come 420 00:32:59.079 --> 00:33:01.880 from the e commerce space. The other thing I ran into was we typically 421 00:33:01.920 --> 00:33:07.279 had to build our load tests to run for longer because you needed to get 422 00:33:07.359 --> 00:33:16.279 into multi generational memory to actually understand behavior in production. That lighting up a 423 00:33:16.319 --> 00:33:22.240 load test that ran for ten minutes and wrapped up didn't give you the same 424 00:33:22.400 --> 00:33:28.759 results as what was actually happening with your server, which was two days in. 425 00:33:29.160 --> 00:33:34.039 Because the way that memory gets allocated over multiple generations became a huge part 426 00:33:34.039 --> 00:33:37.240 of the problem not that we had to wait two days, but often we 427 00:33:37.279 --> 00:33:40.920 had to go a couple of hours before you actually get that Gen two, 428 00:33:42.000 --> 00:33:46.720 Gen three reshuffling a memory enough to say this is fully fragmented and restacked memory 429 00:33:46.720 --> 00:33:52.920 a few times, and that's when those orphan long duration objects created problems for 430 00:33:52.039 --> 00:33:55.000 us. It was like, there's this chunk of memory in the middle of 431 00:33:55.039 --> 00:33:59.559 the pool and it's been there for two hours, and what the hell is 432 00:33:59.559 --> 00:34:05.359 that? Screwing up every GZ But you only found that from these longer runs, 433 00:34:05.559 --> 00:34:09.760 Daniel, is there anywhere that replacing a task with a value task will 434 00:34:09.800 --> 00:34:14.800 be a problem. I mean, that's a common way to reduce gen zero 435 00:34:15.000 --> 00:34:21.000 allocations is to use value tasks. Value task is definitely an interesting, sort 436 00:34:21.039 --> 00:34:27.639 of a newer ish addition to dot net. I know that this is a 437 00:34:27.639 --> 00:34:30.719 little bit of a controversial topic because some people are like, yeah, you 438 00:34:30.719 --> 00:34:35.599 should be using value task everywhere. I'm more sort of in the camp of 439 00:34:35.920 --> 00:34:39.599 using it where it was designed for, which is for io bound paths where 440 00:34:39.639 --> 00:34:45.360 you essentially have the maturity of the calls or sort of getting a cached value 441 00:34:46.679 --> 00:34:52.719 and then only like I don't know, out of ten calls maybe one or 442 00:34:52.760 --> 00:34:57.079 two are actually doing the actually operation. That's where I feel like value task 443 00:34:57.199 --> 00:35:00.880 is sort of a new sort of approach to sort of make sure that you 444 00:35:01.199 --> 00:35:07.800 don't have that many allocations anymore. But to be honest, I think in 445 00:35:07.960 --> 00:35:12.559 most systems, when you have to sort of do that type of optimizations, 446 00:35:12.639 --> 00:35:17.920 then you're already super far because I bet many sort of applications systems that they're 447 00:35:19.000 --> 00:35:22.599 running on top of maybe ASPO, net Core or some others, they have 448 00:35:22.719 --> 00:35:30.400 other problems like memory stream two array, unnecessary bitary allocations, stringifying stuff that 449 00:35:30.480 --> 00:35:35.079 doesn't need to be stringified, all sorts of that stuff. Instead of really 450 00:35:35.480 --> 00:35:39.639 thinking about switching from task to value task, how about switching from web API 451 00:35:39.800 --> 00:35:52.400 to gRPC. It's a good one. Yeah, I mean even for example, 452 00:35:52.440 --> 00:35:58.440 switching from newtons of chasing to system text chasing right or using source generated 453 00:35:59.079 --> 00:36:01.719 I'm sorry, which ones faster? Are you going to touch that? No? 454 00:36:01.920 --> 00:36:06.440 Let's not go there, let's not well, yeah, I mean you 455 00:36:06.480 --> 00:36:10.559 open the door, you might as well walk through it. I would much 456 00:36:10.679 --> 00:36:15.719 rather to talk about a little Swiss cheese or something like that than talking about 457 00:36:15.760 --> 00:36:20.199 that. Well, I mean, we don't want the listeners to get the 458 00:36:20.480 --> 00:36:24.599 impression that just by switching from Newton's Soft to system text Chason, you're going 459 00:36:24.679 --> 00:36:30.760 to find a performance improvement. Did you really mean that? So we actually 460 00:36:31.239 --> 00:36:39.159 have seen when we switched from Newton's Jason across the board quite hefty improvements, 461 00:36:39.800 --> 00:36:47.679 especially when combined the system text Chasin's US combined with source generated approaches, where 462 00:36:47.760 --> 00:36:52.119 you also sort of are more AOT friendly. As an example, you have 463 00:36:52.199 --> 00:36:59.480 far faster startup times on ash of Functions or aw ISLAMDA. So there is 464 00:36:59.519 --> 00:37:04.519 definitely merits to to to that. But I don't wanna I don't want to 465 00:37:04.559 --> 00:37:07.599 say Newton's have Chason is bad. I think it has its place. Well, 466 00:37:07.679 --> 00:37:12.559 the same guy's working on system texts Chason. I mean it's James Newton 467 00:37:12.639 --> 00:37:15.400 King, yeah exactly, and g RPC web right right exactly. Yeah, 468 00:37:16.280 --> 00:37:20.800 being part of that new version of dot net and integrated with that team. 469 00:37:20.920 --> 00:37:23.079 Like there's a lot of performance people there. You just have the resources. 470 00:37:23.239 --> 00:37:28.079 Yeah, Like if the eye of sore On that is Stephen Tabb is paying 471 00:37:28.119 --> 00:37:31.360 a chance in your code, your coach is going to be faster. Yeah. 472 00:37:31.440 --> 00:37:35.880 Yeah, that guy's amazing. But again, I mean, if if 473 00:37:35.920 --> 00:37:38.559 you if you have your harness and you attach it and do some profiling, 474 00:37:38.599 --> 00:37:45.360 and you find out that the civilization subsystem is actually the problem that makes everything 475 00:37:45.440 --> 00:37:49.400 so much slower, then that change makes sense. But I guess there are 476 00:37:49.440 --> 00:37:53.599 also other areas of improvement that you can sort of leverage. But I would 477 00:37:53.679 --> 00:37:57.239 like to, if you don't mind, I would like to switch a little 478 00:37:57.239 --> 00:38:00.960 bit into sort of the benchmarking stuff if we still have some sure for you. 479 00:38:02.400 --> 00:38:08.280 Because Richard and I we talked about this as well in recently in Warsaw 480 00:38:08.280 --> 00:38:13.920 and in Importal about benchmarking stuff, and I think that was a point where 481 00:38:13.920 --> 00:38:15.960 I said, all right, we need to make a show about this. 482 00:38:17.760 --> 00:38:22.920 Yeah. So one of the things that I found really interesting is when the 483 00:38:22.960 --> 00:38:29.880 first time I sort of got into contact with benchmarking was I read lots of 484 00:38:29.920 --> 00:38:34.400 blog posts about benchmark net and sort of I was looking at sort of these 485 00:38:34.880 --> 00:38:37.920 benchmarks out there's like ads, it's easy. It's like a unit test, 486 00:38:38.039 --> 00:38:42.719 right, It's like I've written plenty of unit tests with x units, god 487 00:38:42.719 --> 00:38:45.920 forbid MS test or whatever. Right, So it's like I know this, 488 00:38:45.079 --> 00:38:51.239 it's it's not going to be difficult. But I was quite surprised that essentially 489 00:38:52.679 --> 00:38:57.599 you required to have a different understanding in order to have a good benchmark. 490 00:38:57.639 --> 00:39:00.840 And one is a unit test has two days, right, it's either red 491 00:39:01.000 --> 00:39:07.039 or it's green, so fast or failed. Right. But a benchmark is 492 00:39:07.079 --> 00:39:10.880 something really different because what a benchmark does is essentially especially when you use a 493 00:39:10.920 --> 00:39:15.320 benchmark on net by the way, excellent tool, shout out to all the 494 00:39:15.360 --> 00:39:22.480 people that have been involved there, it's it's like you are executing a given 495 00:39:22.559 --> 00:39:28.599 scenario under hundreds and thousands of iterations, right, And so what it means 496 00:39:28.679 --> 00:39:32.719 is there is no pass or failed. You basically get sort of standard deviations, 497 00:39:32.760 --> 00:39:37.239 you get g C, you measure the GC involvement and stuff like that. 498 00:39:38.039 --> 00:39:44.199 So that's what the benchmark is, right. But it doesn't just start 499 00:39:44.239 --> 00:39:47.960 with understanding what the benchmark is. It also goes to what how should I 500 00:39:49.000 --> 00:39:53.800 even put my code under benchmark harness? Right? And that turned out to 501 00:39:53.840 --> 00:39:58.760 me the really tricky part because I was only reading about this. Oh, 502 00:39:58.800 --> 00:40:04.480 here is a before and after comparison between string concatenation, a string builder and 503 00:40:04.519 --> 00:40:07.960 the value string builder, which is the fastest, right, super easyst scenario. 504 00:40:08.400 --> 00:40:15.159 But when you're actually taking your production system's code that you had under a 505 00:40:15.199 --> 00:40:17.679 test harness, and you sort of filtered in and say, okay, there 506 00:40:17.719 --> 00:40:22.280 is a bunch of things that we need to improve, and then you want 507 00:40:22.320 --> 00:40:25.320 to measure it before and after. How do you take this code which is 508 00:40:25.360 --> 00:40:30.559 probably not just a public method on a static class that you can take and 509 00:40:30.599 --> 00:40:35.320 then put into a benchmark. How do you actually take all the sort of 510 00:40:35.360 --> 00:40:38.360 make sure that you can measure what's going on, remove all the side effects 511 00:40:38.360 --> 00:40:43.199 that you don't want to have in your code, so that you have reliable 512 00:40:43.400 --> 00:40:46.119 sort of benchmarking results, and that you can compare the before and enough to 513 00:40:46.199 --> 00:40:52.840 And that was the thing that I struggled tremendously with it, and I found 514 00:40:52.880 --> 00:40:57.320 my way sort of to do it, especially sort of when I was still 515 00:40:57.360 --> 00:41:01.840 sort of growing on the before becoming performance aware. And that was essentially I 516 00:41:01.880 --> 00:41:06.400 went with a very simple, simple approach. I essentially took sort of the 517 00:41:06.480 --> 00:41:10.320 components that were sort of on that code path. I usually copy pasted the 518 00:41:10.400 --> 00:41:16.719 code into a sort of a dedicated source repository, stripped away all the unnecessary 519 00:41:16.760 --> 00:41:21.280 stuff. For example, when you when I had an IOC container in place, 520 00:41:21.719 --> 00:41:24.159 I removed it and I added anew of the thing that I wanted to 521 00:41:24.239 --> 00:41:28.679 new up. Or if I had IO bond stuff and I didn't want to 522 00:41:28.719 --> 00:41:31.480 measure the IE bound stuff, I removed it with a task complete the task 523 00:41:31.880 --> 00:41:37.320 stuff like that, right, and then I had sort of a dedicated code 524 00:41:37.320 --> 00:41:42.079 base that was in a specific state, in a controllable state where I knew 525 00:41:42.360 --> 00:41:45.320 all the noise that is that I don't want to sort of measure is gone, 526 00:41:45.400 --> 00:41:52.000 and now I can focus on that specific sort of benchmarking scenario that I'm 527 00:41:52.039 --> 00:41:57.280 looking at. But it is the unit testyle test. Effectively, it was 528 00:41:57.400 --> 00:42:00.599 like just bench this piece so that I know I've gotten results from that. 529 00:42:01.280 --> 00:42:07.280 I'd also say, as a corollary to this, that you do get decreases 530 00:42:07.280 --> 00:42:13.079 and performance with later versions because you're doing more and so often it's like, 531 00:42:13.239 --> 00:42:16.599 you know, the whole conversation with the PM, because I've gotten a situation 532 00:42:16.639 --> 00:42:22.039 where we literally had benchmarks as part of the CICD pipeline. Now they're coming 533 00:42:22.039 --> 00:42:24.599 back and saying, hey, the new version is slower than the old version. 534 00:42:24.599 --> 00:42:28.800 It's like the old version didn't do all these things you asked for, 535 00:42:29.039 --> 00:42:34.239 Like this is the overhead for the feature you asked and that was getting into 536 00:42:34.360 --> 00:42:37.159 SLA rules and things where it's like the customer expects us to deliver this in 537 00:42:37.480 --> 00:42:40.400 x many second fractions of a second, it's like, well, we're getting 538 00:42:40.440 --> 00:42:45.559 closer to the limit because the customer asked to do more update the SLA. 539 00:42:46.159 --> 00:42:51.599 It's interesting that you mentioned that because I think one of the sort of benefits 540 00:42:51.639 --> 00:42:54.679 of the approach that just described is that you can easily get start with not 541 00:42:54.760 --> 00:43:00.280 even thinking about what is how can we actually capture regression or where should we 542 00:43:00.400 --> 00:43:06.760 execute those tests? What is a reliable cic the environment in order to have 543 00:43:06.960 --> 00:43:14.880 sort of measurable and statistically relevant results from this environment, right, But what 544 00:43:14.920 --> 00:43:19.639 you're essentially talking about is sort of more regression testing as well. And there's 545 00:43:19.679 --> 00:43:23.599 actually there's actually a lot of great guidance a little bit hidden in the dot 546 00:43:23.639 --> 00:43:30.360 net Performance Repository. I think it was also driven by Adam Sidnik and some 547 00:43:30.519 --> 00:43:36.519 other people from his team. I actually talked to him at a little bit 548 00:43:36.639 --> 00:43:39.599 over the course of I was preparing for a talk about this specific topic. 549 00:43:39.639 --> 00:43:44.320 I talked to him a lot about it, and they have a tool that 550 00:43:44.480 --> 00:43:46.920 essentially allows you to when you use the benchmark dot net, what you can 551 00:43:46.960 --> 00:43:52.599 do is you can actually execute benchmark dot net against the specific version of a 552 00:43:52.639 --> 00:43:57.000 benchmark and the specific version of the code, you can store the artifacts, 553 00:43:57.599 --> 00:44:01.440 and then you can execute it again against the changed version, store the artifacts, 554 00:44:01.440 --> 00:44:05.840 and then you can use the compared to that they have to sort of 555 00:44:06.840 --> 00:44:09.920 sort of create a diff between the before and after version, and then you 556 00:44:09.960 --> 00:44:16.519 can define a threshold that sort of determines when it was an unacceptable sort of 557 00:44:17.119 --> 00:44:23.159 regression, and then for example, you can fail your CICD environment but kind 558 00:44:23.239 --> 00:44:28.880 of performance because of performance, right, but CICD. It's also interesting because 559 00:44:29.119 --> 00:44:35.280 andre Akinschin wrote an excellent blog post about this topic because he did as part 560 00:44:35.320 --> 00:44:37.360 of sort of investigations, he looked, for example, at git ub action 561 00:44:37.519 --> 00:44:46.000 runners and the result there is that you essentially cannot use git ub action runners 562 00:44:46.639 --> 00:44:54.159 to actually do regression testing for performance because they're just so unreliable. So basically 563 00:44:54.360 --> 00:45:05.800 you basically have up to three times different sort of three times different execute memory 564 00:45:05.960 --> 00:45:12.519 and CPO difference between builds. Right, So it's it's insane. So wow, 565 00:45:12.559 --> 00:45:16.760 that's crazy. Yeah, yeah, it's it's quite fascinating. Right. 566 00:45:16.800 --> 00:45:20.760 So that means you just means if you're going to benchmark issue this way, 567 00:45:20.800 --> 00:45:22.920 you have to control your CICD pipeline record, right, and you have to 568 00:45:22.960 --> 00:45:30.039 have dedicated hardware that you sort of put somewhere or rent basically bare metal hardware 569 00:45:30.079 --> 00:45:34.440 where you and have run their infrastructure that sort of allows you to sort of 570 00:45:34.920 --> 00:45:38.039 put those tests to that reliable hardware. And you can't use sure. I 571 00:45:38.039 --> 00:45:43.760 mean you're saying the clouds not an option here, Well it is definitely it 572 00:45:43.880 --> 00:45:46.280 is an option, right, Yeah, it's just you're gonna you can't use 573 00:45:46.320 --> 00:45:50.079 the built in infrastructure for this. You have to set it. You have 574 00:45:50.079 --> 00:45:55.280 to write your own YAML YAMO you have done this. Like one of the 575 00:45:55.320 --> 00:45:59.119 one of the issues we were dealing with is we were up to I don't 576 00:45:59.119 --> 00:46:02.000 know, three or four thousand different web tests we wanted to do, and 577 00:46:02.079 --> 00:46:07.360 they took a day and we needed under fifteen minutes, and so we would 578 00:46:07.440 --> 00:46:12.400 light up twenty instances of the site in the cloud and parallelize all the tests. 579 00:46:12.800 --> 00:46:15.559 My goal was always by the time you got back to your desk from 580 00:46:15.639 --> 00:46:20.280 coffee, the test results were back and you had failed. Yeah, right, 581 00:46:20.360 --> 00:46:22.800 like that that was because instead it's still in your head. Like what 582 00:46:22.880 --> 00:46:30.719 we realized was that every minute that goes by after they've pushed the codes leaking 583 00:46:30.760 --> 00:46:35.320 from their mind. And if it's and if it's a day, it could 584 00:46:35.360 --> 00:46:37.199 be anybody, like they're going to have to start over picking it back up 585 00:46:37.239 --> 00:46:40.239 again. But if we could get it to them in under an hour, 586 00:46:40.360 --> 00:46:45.320 like fifteen minutes was the magic number, they knew exactly what Oh, I 587 00:46:45.320 --> 00:46:46.519 know what that error is, and off they'd go again, like it just 588 00:46:46.639 --> 00:46:52.800 saves so much remediation. Yeah, so I jacked up the test bill because 589 00:46:52.800 --> 00:46:55.719 it saved the dev bill. Amazing. Yeah. So, and again I 590 00:46:55.719 --> 00:47:00.840 think it's it's all. When we talk about ESHA DevOps run or get up 591 00:47:00.840 --> 00:47:05.760 action runs, usually there is a shared sort of infrastructure that you have there. 592 00:47:05.840 --> 00:47:08.239 And yeah, so you just it's not good for benchmarket. You don't 593 00:47:08.320 --> 00:47:13.480 know what performance you're going to get, no repeatable results, which oddly are 594 00:47:13.639 --> 00:47:17.440 necessary. But if you're talking past fail, that's fine. You don't care 595 00:47:17.440 --> 00:47:21.800 if it ran twice as long, half as long. It's just pass fail. 596 00:47:21.840 --> 00:47:24.119 For functionality, that's just fine. Correctly, no big yeah. But 597 00:47:24.159 --> 00:47:29.920 again my message here is I want to hammer this home. I think becoming 598 00:47:30.000 --> 00:47:34.320 performance aware, I think it's a journey and you don't need to basically end 599 00:47:34.400 --> 00:47:37.599 up where we just oh, well, you will end up eventually there where 600 00:47:37.639 --> 00:47:42.360 we just talked about with having potentially your own hardware if you need to do 601 00:47:42.400 --> 00:47:46.599 regressive testing, but already having your harness in place, understanding the profilers so 602 00:47:47.079 --> 00:47:52.559 that you can sum in where you should actually make those sort of performance improvements. 603 00:47:52.880 --> 00:47:55.199 Then using a tool like benchmark dot net, which saves you a lot 604 00:47:55.239 --> 00:48:00.199 of headache because it's sort of sort of is already designed to mitigate the most 605 00:48:00.320 --> 00:48:04.360 all the stuff you'd end up writing for yourself any direct So just use it 606 00:48:04.400 --> 00:48:08.119 and a bunch of smart people are working on correct and it mitigates, and 607 00:48:08.159 --> 00:48:12.599 then you can start there, maybe copy paste your code at the beginning, 608 00:48:12.719 --> 00:48:15.199 isolate the things that you want to do, get started there, and then 609 00:48:15.320 --> 00:48:20.599 at the later point in time, where when your company is already sort of 610 00:48:20.719 --> 00:48:25.519 more performance aware, you can start slowly introducing sort of more sort of mature 611 00:48:25.800 --> 00:48:30.880 ways of actually doing performance testing and regression testing all the way along. Yeah, 612 00:48:30.920 --> 00:48:34.960 but you're you know, what you're applying there is like you're pretty far 613 00:48:35.000 --> 00:48:37.880 down the path at that point too. Yes, to me. The big 614 00:48:37.920 --> 00:48:44.760 thing here is when does performance creep into the requirements? Because a lot of 615 00:48:44.800 --> 00:48:46.960 folks you know, early days of projects, it's just not even on the 616 00:48:47.039 --> 00:48:52.760 radar, right, but you know, to actually file performance problems as a 617 00:48:52.800 --> 00:48:55.880 bug to get them on the sprint, right, to be part of the 618 00:48:55.960 --> 00:49:00.079 conversation at all, Like that's already that's arguably the starting point of any of 619 00:49:00.119 --> 00:49:06.360 that path is that it's bubbled up to the point where business cares about it. 620 00:49:07.039 --> 00:49:08.239 You know, the line I used to do when I did these talks 621 00:49:08.440 --> 00:49:12.800 was performance is like air. You only care about it when you don't have 622 00:49:12.880 --> 00:49:20.239 it. Yeah. I'm a big believer in having non functional requirements in the 623 00:49:20.280 --> 00:49:24.719 design or and the architecture sort of built in and having explicit discussions about non 624 00:49:24.760 --> 00:49:30.039 functional requirements. It's also prioritize them with your business stakeholders in order to make 625 00:49:30.079 --> 00:49:34.440 the right trade offs. Well, and you know the sneaky part about that 626 00:49:34.559 --> 00:49:37.719 is let let them tell you it isn't important, so later when they decide 627 00:49:37.719 --> 00:49:42.880 it is important. But you said, you know, because again, performance 628 00:49:42.920 --> 00:49:45.159 is one of those things where nobody cares about it until they do. Yes, 629 00:49:45.280 --> 00:49:47.159 yeah, right, if I suck the air out of the room, 630 00:49:47.159 --> 00:49:52.960 you're suddenly really interested in air. Absolutely, And it's the same thing. 631 00:49:52.000 --> 00:49:54.840 It's like you never thought about. You know, you can debate all day 632 00:49:54.920 --> 00:49:59.079 a render time of two seconds versus four seconds. I'm sorry, I'm so 633 00:49:59.159 --> 00:50:02.280 web centric on this stuff, and that's not a big deal. Everybody knows 634 00:50:02.320 --> 00:50:07.760 that thirty seconds is bad, right, and so that's sort of these kinds 635 00:50:07.800 --> 00:50:13.440 of thresholds, and it's hard to talk about that in out of context. 636 00:50:13.760 --> 00:50:17.320 You kind of have to make a slow page for everybody to start getting hey, 637 00:50:17.639 --> 00:50:22.920 slow page bad, set requirements for minimum performance and figure out where they 638 00:50:22.920 --> 00:50:25.719 are. Any good news is, of course there's lots of written documentation on 639 00:50:25.760 --> 00:50:29.679 its like you don't have to invest And one of the things that I also 640 00:50:29.760 --> 00:50:35.400 really appreciate by doing these types of investigations is and small improvements is you learn 641 00:50:35.440 --> 00:50:38.960 a ton about the cold path in question, and that gives you a lot 642 00:50:38.960 --> 00:50:43.840 of insights into a potential redesign in the future, right, because so many 643 00:50:43.840 --> 00:50:46.000 people are just throwing out there, just rewrite this, right, but they 644 00:50:46.000 --> 00:50:50.239 have make it, make it faster. You're not going to make it better 645 00:50:50.360 --> 00:50:52.320 if you don't know what they are like this, like, yes, I 646 00:50:52.360 --> 00:50:57.639 also find that most folks who spend time in the tuning part understand the x 647 00:50:57.679 --> 00:51:00.159 you should have, like the behavior of software in a deeper level than a 648 00:51:00.199 --> 00:51:04.000 lot of folks that wrote it in the first place, because often you're just 649 00:51:04.039 --> 00:51:07.039 trying to get to the deliverable. Does the future do the future requirements that 650 00:51:07.079 --> 00:51:10.599 were there? I know we only have a few minutes left, but at 651 00:51:10.599 --> 00:51:17.480 one point in your investigation, do you consider re architecture, which is obviously 652 00:51:17.920 --> 00:51:25.159 the most expensive and risky thing way to improve performance. But you know, 653 00:51:25.280 --> 00:51:30.559 I mean, as you're going through a project and looking or a tool or 654 00:51:30.559 --> 00:51:35.599 something and looking at every little thing that you can squeeze out, and something 655 00:51:35.679 --> 00:51:39.039 jumps out at you, Oh, well, you know this should be refactored 656 00:51:39.159 --> 00:51:45.960 or maybe even completely re architected. How often does that happen? It's a 657 00:51:45.960 --> 00:51:50.000 difficult question to answer generically, but I can give you a complete example. 658 00:51:50.360 --> 00:51:53.760 So I wrote a lot. It contributes a lot to the ASHER service bust 659 00:51:53.840 --> 00:51:59.639 on that st K, and essentially I think it was down sort of twenty 660 00:51:59.719 --> 00:52:04.960 pool requests on the sort of the path where the sort of the ASH service 661 00:52:04.960 --> 00:52:07.719 bus sort of takes you get the byte the race from ASH service bus and 662 00:52:07.760 --> 00:52:12.320 handed over to ASH of functions or to your code that is running. Where 663 00:52:12.599 --> 00:52:17.119 I did lots of lots of tiny improvements until I actually understood sort of how 664 00:52:17.159 --> 00:52:23.320 the body management of the byte payloads actually really really work. And then only 665 00:52:23.400 --> 00:52:29.079 then I came up with a better idea how to sort of manage that that 666 00:52:29.159 --> 00:52:32.960 body work from different aspects, and that then led to sort of even more 667 00:52:34.079 --> 00:52:39.119 orders of magnitude of improvement how the body is sort of managed and less allocations, 668 00:52:39.119 --> 00:52:44.599 more efficient in CPU cycles. But it was like I think, Okay, 669 00:52:44.639 --> 00:52:47.239 I contributed in my free time, whatever that means these days when you 670 00:52:47.239 --> 00:52:51.880 were constantly online, right, But I contributed in my free time, I 671 00:52:51.880 --> 00:52:57.679 guess, over a year to this code base, until together with the team, 672 00:52:57.719 --> 00:53:00.840 we realized, oh, there are there are actually things we can sort 673 00:53:00.840 --> 00:53:06.239 of really refactor and make things even faster. So I get I'm actually I 674 00:53:06.280 --> 00:53:09.880 have the tendency to go a very long time on a specific code path before 675 00:53:09.920 --> 00:53:16.159 I even reconsider re architecting or redesigning. Of course, small impress can also 676 00:53:16.199 --> 00:53:21.519 sort of sometimes mean you're not doing up something. You're making something a singleton 677 00:53:21.599 --> 00:53:25.440 that previously wasn't a singles or something like that, right, and if it's 678 00:53:25.440 --> 00:53:31.159 still not performant enough, that's when you think about re architecture, because what's 679 00:53:31.199 --> 00:53:35.559 also great is right when it's running in production, it's making money right, 680 00:53:35.719 --> 00:53:38.679 and it gives you insights about the good about what it's doing. And when 681 00:53:38.679 --> 00:53:45.079 you're architectu and redesigning for that period of time, you have no validation whether 682 00:53:45.159 --> 00:53:49.559 the stuff that you're changing towards will work. And with the small improvements you 683 00:53:49.639 --> 00:53:53.400 have constant feedback loops. And I think I feel that's super super important. 684 00:53:55.280 --> 00:53:59.880 So what's next for you? Man? What's in your inbox? Well? 685 00:54:00.280 --> 00:54:05.199 Christmas time of course, drinking way too much graft beer probably over Christmas time, 686 00:54:05.239 --> 00:54:15.280 a delicious stout or something like that. What is this? But but 687 00:54:15.400 --> 00:54:20.440 next year I will be I will be in dot Net Day Romania at eating 688 00:54:20.599 --> 00:54:25.559 US conference. I will be delivering a workshop about reliable messaging in ASHER, 689 00:54:25.960 --> 00:54:30.840 sort of deep diving into ash service, bus storage, ques, event hups 690 00:54:30.880 --> 00:54:36.440 and event grets. That's going to be really interesting. And what else I 691 00:54:36.480 --> 00:54:40.440 don't know yet from a conference perspective, but definitely sort of increase a little 692 00:54:40.440 --> 00:54:45.599 bit more of my contributions to open source stuff because I still have a few 693 00:54:46.320 --> 00:54:52.039 to contribute to a few open source libraries. Yeah, it sounds good, 694 00:54:52.079 --> 00:54:54.840 well, Daniel, thanks for spending this hour with us. It's been great, 695 00:54:55.000 --> 00:54:59.639 Thank you all right, and we'll see you next time on dot net 696 00:54:59.760 --> 00:55:24.199 rock. Dot net Rocks is brought to you by Franklin's Net and produced by 697 00:55:24.280 --> 00:55:30.199 Pop Studios, a full service audio, video and post production facility located physically 698 00:55:30.239 --> 00:55:36.199 in New London, Connecticut, and of course in the cloud online at pwop 699 00:55:36.480 --> 00:55:39.320 dot com. Visit our website at d O T N E t R O 700 00:55:39.400 --> 00:55:45.719 c k S dot com for RSS feeds, downloads, mobile apps, comments, 701 00:55:45.039 --> 00:55:50.119 and access to the full archives going back to show number one, recorded 702 00:55:50.159 --> 00:55:53.079 in September two thousand and two, and make sure you check out our sponsors. 703 00:55:53.239 --> 00:55:57.920 They keep us in business. Now go write some code, See you 704 00:55:57.960 --> 00:56:07.000 next time in the band. Sadly seen a summer time that means hard than 705 00:56:07.119 --> 00:56:09.559 my Texas in Line Revell