How'd you like to listen to dot net Rocks with no ads? Easy?
Become a patron for just five dollars a month. You get access to a
private RSS feed where all the shows have no ads. Twenty dollars a month,
we'll get you that and a special dot net Rocks patron mug. Sign
up now at Patreon dot dot NetRocks dot com. Welcome back to dot net
Rocks. This is Carl Franklin, this is Richard Gamble and this is the
coming out on the twenty first. So this is the last dot net Rocks
show published before Christmas, but we've got a few more to record. Just
Geekouts. How you been, man? You know I've been. I've been
working on the scripture the Geekouts turns out. You know, it's a lot
of stuff. It's been a busy year, yeah, sure has. How
about you? What are you well? Last night I did my first recording
of one of my songs in the studio, in the new studio. Oh,
a new track, A new well. It's an older track that nobody's
heard yet. But the first recording I did was all kind of discombobulated.
It's slower, it's at one hundred bpm and and there, and with acoustics
strumming and picking and stuff, and therefore it's very easy to get off time.
So I want to tell you about the cool stuff that I did that
I did this with. I recorded the acoustic guitar in the bass first,
and then I used a studio one's bend tool to quantize the audio. What
quantize the audio? It basically worked right out of the box, and I
had tried it before, but it hadn't really worked well. Maybe the default
settings are good now. But what it does is it finds the transience according
to your quantize value eighth notes or whatever, and then it essentially moves the
transience and either stretches or compresses the audio between the transience and basically turns it
into quantized like it's right on the beat. Okay, so it's just like
making the beat perfect. Yeah, And there's no it does not sound artifacted
at all interesting to me anyway. So then I did the whole the same
thing with the drums. For the drums, I used my iPad at the
drum set because it's all the way across the room right right, and the
iPad I installed this thing called studio one remote, and so I set markers
in the three places where I wanted to record drums and I could just go
back undo record. So I did all the recording from the drum set with
my iPad on a music stand and I did two sections, one with brushes
and one without. And that it took a while to record, you know,
but it was great because I could just take another takenother take till it
was right, and then I quantized the drums. Those drummers are never on
beat anyway. I was pretty close, but I wasn't perfect, and I
really wanted this to be tight interesting and it really is. So I'm very
excited. Everything worked and yeah, what can I say? Very happy the
new studio. Yeah, fully up, rational nice. Hey let's get started
with better no framework? All right, man, what do you got?
Well? I found this post on board panda dot com. Have you ever
seen that crazy site? Yeah? Yeah, So this is thirty of the
worst Christmas gifts people ever received, as shared in this online group. One
of my favorites is a dish towel. I was eight, It was eight
years old, and my parents gave me a dish towel. Nice and it
wasn't even a new dish towel. I learn it was just like they went
to the kitchen, found a dish towel, wrapped it up in paper,
and then get to this eight year old kid. H it. Yeah,
Dad, Christmas presents so hours and hours of fun and yucks and silliness.
Silliness, Yeah, which we need once in a while, right, especially
around the end of the year. It's like we've been working hard all year.
Now let's take some time. Awesome. Anyway, That's what I got.
Who's talking to us? Richard grabbed a comment of of Show eighteen seventy
three, the one we just published a little while ago with Leah Melanino from
NDC and Portal. We were talking about sustainable development. You know, I'm
just thinking about the energy can say. We've got a great comment here from
Jackie. You said, Yeah, Hi, Carlin Richard. I want to
express my thanks for tackling this important subject. This conversation resonated with me a
lot, as I'm a software engineer working with Azure Cloud technologies. In our
related note, I'd like to share my recent experience with an iPhone. I've
always been an Android user, primarily do the perception that iPhones are overpriced to
non standard devices. I think Apple pretty much dictates the standard. Just look
at out what happened they did to RSS. Often seen as more suitable for
users seeking opinionated ux like my mother, or a symbol of social status like
my brother Jackie, like your diss in the fan, right it's Christmas.
However, when my Google pixel broke and I had to let it in for
service, I was left with no choice but to use a backup iPhone fives,
a decade old model offered by my brother. See he may be a
social status seeker, but at least they'll give you a spare iPhone. It's
not a bad guy. He's not a bad guy. Yeah, you had
to reassess this relationship, man. And to my surprise, the iPhone five
S was still receiving security updates and all my essential mobile apps function flawlessly on
it. This is something I couldn't even say about some five year old low
end Android devices, not that any iPhone is a low end device. The
only drawback was battery life. Yeah, battery is not going to be great.
This experience made me realize that even though iPhones may not be considered entirely
sustainable, due to a lack of adherence to certain standards like USBC or Place
Little batteries. Their longevity is great right up until you drop them. Therefore,
I've decided to offer an iPhone model with a USB C port. Thank
you EU who demanded that Apple start using USB c's. This all allow me
to utilize most of my Android gadgets and extended lifespan on the phone. And
thanks again for your dedication to the dot net Rocks podcast. It has been
a valuable source of knowledge and inspiration, especially during my career transition to dot
net c sharp looking forward to future episode, and I can add to that
the iPhones are inherently more secure than Android phones just because Apple is such a
closed system. That's one of the benefits actually of having It's sort of security
by obscurity. Nobody wants to try and act them. Really, I don't
know about obscurity, but security by by iron fist control over everything, everything
that goes on that phone, right, So yeah, I like I That's
that's one of the reasons why I have an iPhone. And the guys at
Security this week Patrick Hanes and Twain Laflat say the same thing. Yeah,
I get the iPhone. The phone is more secure the Android phone. I
mean under the hood androids Linux. Everybody knows Linux is terrible, especially Daniel.
He knows that that. I'm just gonna poke at the Linux guy the
whole day. So we're gonna do but we're gonna poke at each other at
Christmas time. Come on, hey, Jackie, thank you so much for
your comment. Glad you really liked the show. And a copy of music
co Buy is on its way to a un If you'd like a copy of
music Cobe, I write a comment on the website at dotnet rocks dot com
or on the facebooks. We publish every show there, and if you comment
there and read on the show, we'll send to your copy of music code
by and you can follow us on Twitter if you like. But the real
fun happens. I'm mastadon, I'm at Carl Franklin at tech hub Social,
and I'm Rich Campbell at masodon do Social send us a two You might get
a mug if we read it on the show. Think so pretty sure you
won't? Yeah, pretty sure? You get a copy of music opy.
Kind of how that worked? Did I say mug? He did? All
right? Let me say that again, Brandon, all Right, senas,
although that was kind of funny, we might want to leave it in.
It's kind of funny. I'm pretty sure you won't. Let's leave it in.
It's Christmas time, what the hell? All right, let me bring
on our guest, Daniel Marbach as a distinguished Microsoft MVP and software maestro at
Particular Software, Daniel knows a thing or two about code. By day,
he's a devoted dot net crusader espousing the virtues of message based systems. By
night, he's racing against his own mischievous router hack, committing a bevy of
performance improvements before the clock strikes midnight and he turns into a pumpkin. Yes,
exactly that. What's a router hack? What are you going to do?
Low wrt? No, it's just it's a very simple sort of trick
because I've been contributing to open source and various things, and I'm just I
just like to spend some time with code because I feel like it's sharp and
sort of my understanding of just actor i'm working with. And I had appeared
in my life where I just couldn't stop right because it's like and then it
was at first it was like one am, two am, three am in
the morning. And then luckily I always basically my only one rule was that
I will not extend my alarm clock to a later point in the day,
so otherwise my days would have shifted. But at some point I was like,
Okay, that's it. I need to change something in my life.
So basically I'm switching off the internet around midnights at my whole Oh and that's
when you have to hack your router so exactly, because that usually when you're
when you're like in the middle of thing and is I should Google bing this
or whatever you're using, right, you're like, ah, the internet doesn't
work anymore. Ah, it has time until tomorrow and then I switch off
and just go to bed. So yeah, I never worked that way for
me. I had to turn the bugger back. I put the same rule
in place because I had teenage daughters. And now you hear the audible groans
at midnight, right, it's like, oh, yeah, everybody's in bed,
sure they are. But me, it's like, I guess got to
you know, finishing up to the go to push the code fails. Yeah,
you're like, uh, it's me. I got to fix the rider.
So your performance wonk, are you? Well? Actually, I work
for the company called Particular Software. I guess you had Dhan on the police
before, right, so you know we had a few particular rights and a
few others and yeah, so I basically my day to day shop is I'm
building a robust and reliable frameworks and libraries for people that sort of want to
sort of build it distributed systems primarily based on messaging stuff like ASH, service
bus, SQS, SNS, storage queues in the old days, God forbids
MSMQ. Right, but it's still it's still out there thriving. Surprise,
surprise, It's quite heavily used there in the industry. Yeah, and it's
one of the great things is feature complete, right, it it's just there
if you're still running on Windows and other on Linux like I do. And
one of the things that we do there is we want to make sure that
the customers that they're using in service bus can focus on their just writing their
business code, don't need to write any plumbing code, and that stuff should
run as efficiently as possible. Right, So I guess performance throughput was always
sort of right from the center sort of in my day to day job.
But they also care a lot about it because I believe especially it came us
out in the comment right that you read out Richard. There's like, if
you're targeting the cloud, or if you're shifting the cloud, or if you're
already are in the cloud, sort of you are basically built by the amount
of resources that you're using in the cloud. Yeah, so there's a direct
revenue relationship to that consumption, which it leaves giuse of some kind of incentive.
Yeah. Correct, You put down your credit card and then you get
surprised at the end of the month. Especially I'm surprising yourself as one thing.
Surprising the CFO is another. That's a very loud noise from a large
office. Yeah, that's the question. I mean, there's so many ways
to tweak performance. One is just by using the latest netstack correct, and
then you know, keeping your new get packages updated. But on top of
that, you know, are what are kind of knobs are you post?
Are you pulling software knobs, hardware knobs? All of the above. So
let me cook before I answer your question. Let me quickly go to what
you said about updating the dot net the dot net version. That's actually really
interesting because Microsoft has this block series where they essentially talk about their teams migrating
to for example, from dot Net framework to newer dot Net versions or from
dot net six net eight, And one of the cool things they blocked there
is they the Microsoft Teams Infrastructure team, they basically migrated from dot Net framework
to dot net six and just by basically migrating to that LTS version of dot
Net, they were actually able to sort of buy almost twenty four percent reduce
their monthly cost expenditure in asure, which is pretty amazing if you think that's
amazing about that, right, And that's definitely one way to do it.
So I always encourage people to sort of if they can stay up to date
with the with the latest dot Net versions. Definitely, Yeah, I'm a
big fan of that series on the dot net blog, just because you know,
you talk about like the teams guys migrating to a new version of dot
net and the benefits they got from it, and also the things they struggled
with on that, like what problems they had. But to me, more
than anything, it's like, hey, you know, if these guys got
this kind of benefit and we're able to move that big an app you're going
to be okay, like you can do it. Yeah, absolutely. But
to come back to your question, Carl, I think one of the things
that I try to apply to sort of in my thinking is I want to
make explicit trade off as I'm going with things, and so that means I
want to be aware of sort of is this code going to be executed on
the hot path or at scale, right, so, and how many times
a second is that going to be or is it just something that runs on
a sort of a background chop once a day or twice a day or because
then usually if it's just executed once or twice a day, it doesn't really
matter that much whether it's super fast or not. But then when it's executed
on the hot path potentially hundreds and thousand times per second, then it's usually
good to sort of become more of performance aware, but performance aware. Unfortunately,
this is also something that gets thrown around quite a lot sort of in
the industry, and then everyone assumes everyone knows what performance awareness means, right.
But one of the things that I struggled with was where should I even
get started to become performance aware, Because apparently if you go down sort of
the literature of looking at benchmarking and performance optimizations. You can actually go from
just doing little things up to setting up your entire CICD pipeline with dedicated hardware
doing regression testing. Right, But usually we don't start there. Usually we
or somewhere else. And that's one of the things that I'm trying to apply.
So usually I ask myself a bunch of questions when I look at the
code, right, So, for example, I I go and look for,
well, what could be the CP and memory characteristics? What could that
be for the specific line that I'm looking at that I know is on the
hot path? And then I usually start thinking about, so, are there
any sort of low hanging fruits that I can sort of apply to this,
maybe do some more efficient string splitting options and stuff like that that I know
from reading the performance blog posts, and I can apply there and that is
and I'm presumed that's Stephen Taub's post, Yes, of course, yeah,
that the book the Book of the Taub right y, yeah, exactly.
And then most of the time, right, so there are a few tricks
you can apply. So, for example, if you're allocating a byttery,
right, and you know that you you actually just need it for every iteration.
What you can do is you can sort of move that away from the
hot path allocated once and clear it and then you're no longer having allocations that
you're executing like on the hot path, and then the garbage collection doesn't have
to clean up a lot of things, and there things get get get better
as well. But then what how do you measure that, Daniel, Like,
how do you know it got? Like, that's a tricky one that
the impact of garbage collection, Like, how would I measure that I reduce
the amount of garbage collecting? Yes, that's that's a that's a very good
question. So usually what I do is before I so I call this performance
loop. So, for lack of a better term, I call the performance
loop. So what I usually start with is when I have ip I hypothesis
about a piece of code that needs to that it creates uh garbage. Then
I what I do is I write a test harness, and essentially what that
harness sort of does. It sort of takes whatever I'm looking at, takes
it into specific context of my suspicions, and then it executes it in that
specific scenarios. And then I attached profilers to that piece of code, And
what I usually do is I take at least a memory snapshot and also create
a CPO snapshot. And of course, if you have an io bond system,
like you have a database in place, or you have HDP called stuff
like that, you also want to look at your you want to do io
based profiling as an example, right, because usually when you look at your
iosystem like the database, that's you can basically achieve orders of magnitude of performance
improvement by tweaking your sequel querits right before you will start thinking about memory allocations
and stuff like that. But assuming you have sort of removed that part,
then essentially those sort of profiler snapshots give you an indication of where could you
voteocus on And here comes the next problem when you attach your profiler is that
you might see lots and lots of allocations from different subsystems and component on that
specific cultree and where should you even start? So I usually try to sort
of apply a combination of I call it the one percent improvement philosophy vverasus deliberate
contextual based optimizations that I want that I want to do so because for example,
I believe that if you do enough little performance optimizations over time, that's
the one percent improvement sort of philosophy, then eventually they will end up making
a big impact. And we can see that Microsoft applies it as well to
do dot net run time. Right, They're doing lots and lots and lots
and lots of small changes all over the place, and the compounding effect of
these changes they essentially are massive when you look at them in sort of the
greater scheme of things, right, And what profiling tools are you using here?
Is this just like the built in profiler. Okay, you made a
joke about me. You're running on Linux, right, so so well,
so I'm a big fan of the sort of chet Brains tools I've been using
for a year alone. So I'm using usually dot trace and dot memory sort
of those two tools, and Writer also has some built in analysis, so
for example, they do when you execute your tests or you execute your solution.
They also do some dynamic program analysis where they show you sort of the
allocations that your stuff had or the the CP that it wasted. So I
tried to use those tools, but primarily dot trace and dot memory to get
to get an overview of what is actually going going on, right, And
I mean, yeah, there are built in profilers, but if you're willing
to pay for one, there are better ones. Yeah, I mean,
to be fair, a visual Studio is great, right, It's like visual
Studio has depending on the license, I'm not entirely familiar with the licensing terms
there, but it has great tooling built in. Or if you are sort
of very advanced and mostly on Windows, you can also use perfew, right.
So perfew is a very powerful tool that you can use, although I
struggle with it a bit, I must say every time it's like using WINDPG.
Every time I use these tools, I have to sort of get some
cheat sheets onto my machine in order to remember the complex commands. And that's
one of the benefits those too. There are tools you need to learn,
and I mean I suspect you use them more than most people. And if
you can't keep them in your head, then nobody can. I mean,
I've done a lot of performance tuning over the years, and people already surprised.
It's like why are are you reading the docs? Like don't you know
this? It's like, listen, there's a lot of knobs on these things,
and if you don't go through the steps you can waste a lot of
time, yes, and actually wasting a lot of time. That's actually a
very good, good sort of comment that you made there, because I think
even if even if you are more familiar with performance optimizations and benchmarking, it's
like and profiling at the end of the day, that's not my day job.
My day job is building robust and reliable messaging frameworks and middlewares and the
platform at particular, and not doing performance optimizations all day long. That's not
my job description, right, And I guess that many people that are also
listening to that podcast have the same thing, right, They would like to
dive into profiling and benchmarking performance optimizations, but they only have a limited budget
in order to spend on those types of things. And that's why I always
recommend start with a test harness, reproduce the scenario, attach your profiler,
and then use your domain knowledge of the things that you're working to basically sift
through the noise of allocations and CPU and on the call stack and then figure
out Okay, probably here is where we can make the biggest impact on sort
of reducing the numbers of CPU cycle spent or reducing the numbers of garbage allocations
that are happening there. But sometimes, like I said before, it's also
where you think you have the most knowledge in and then applying sort of the
one percent improvement over time in order to sort of make things better and better
and not trying to gold plate so everything out of an existing code path.
But I guess we now have sort of touched a little bit on the how
would I even know where to get started? Right? We talked about profiling,
We talked about doing CPO memory at least right always get sort of two
views on the code base. But the next thing is then I mean improvements.
Of course, that goes more into the territory of knowing your stack,
knowing your language, knowing the libraries that you work with. Like Carl said,
also looking for has the library and you release that we can pull in
or maybe reach out to the maintainers and say, hey, by the way,
we did the profiling snapshot and we found out that this library allocates that
much amount of memory. And guess what when you reach out with a profiler
snapshot to third party tooling providers, they are like super happy because you're then
in the one percent sort of customers and then it's like, hey, now
we have data from the customer that we can see what's actually going on.
You've now described a workload in a meaningful way to them correct correct, right,
And so for example, I myself have done that as well. At
some point I st over sort of memory inefficiencies in the ash service bus s
d K, and then I saw I had a hunch. I wrote the
test harness attached profiler was able to show that when you access the bodies of
a service bus message that allocates unnecessary memories every time you essentially access the body.
I was able to sort of show the profiler snapshots show the memories.
And then I, even because I was lucky, I already knew the library
a little bit, I guess, and I was able to contribute a fix
for that memory allocation problem to the Azure service bus. Yeah that's a good
way to get an email from Clements. Yeah, yeah, I've I've I've
had comments from from Clements originally on on some of my pull requests as well.
And then and there in lies the point, like you get into it.
And of course the kicker, of course is at its open source,
so you could just contribute a fix yes, yes, yeah, but I
guess, I mean, nobody expects you to write but at least having a
memory profile or a CPU profile, and like you said, Richard, showing
what's going on in production, is it. Yeah, I just think it's
always challenging to get down to the brass tax like that. To me,
most of my profiling experience has been trying to optimize an e commerce site where
it's like we're just you know, we're running. We're now looking at a
buy of more servers because the site's so busy, like an optimization can mean
a lot of money and the prob I was a nance guy at the time,
and that was the tool that showed me. I mean, this may
not have been it was always that balance between this is a very complicated method
and so it's consuming a lot of resources and it's a very simple method,
but it's called hundreds of thousands of times, and so the fact that the
tool would help sort out that weight of often called, and so were thought
minute optimizations will make big differences versus rarely called but complex enough that you will
get some return on that, you know, didn't I'd never worried about optimizing
in mind calls because it just didn't call that often. But all of that
mainstream shopping cart recommendation engine, you know, custom render pieces, ad pieces
like those, were all the things where it's like these get called a lot
even though they don't look that big, and just playing with string compilation like
those kinds of things made a huge difference in the end. But the challenge
I think for a lot of folks is they just want to get into the
code and this idea of you snap the harness on first and get a baseline
set of profiles in place, and then as you said, the magic word,
the hypothesis says, if we do an optimization here, it'll make a
difference. Now you go tinker, then run the benchmarks again and it's they
did we make a difference? And King and if you didn't revert. Yes,
And because there's no performance code i've ever written them was easier to read
than the original. Yes, ever, absolutely true, And I think what
you said is super crucial. That is sort of the performance look that I
that I apply. Is so when you have the harness and then usually that
reproduces this scenario right, and then like you said, you do the improvements
that might be several iterations of ideas that you tinker around with, and then
you might execute several benchmarks to sort of look at those sort of optimizations that
you're doing. Maybe it's even several micro benchmarks sort of measuring sort of little
improvements in that call stack that you came up with during that tinkering phase.
And then at the end what you do is you bring it back into that
harness, right, and then you look at sort of the end to end
profile again where you look at again the CPU and memory profile at least actually
to actually see the before and after right, and then you see on your
graphs, oh, we spend six hundred and fifty megabytes of memory on that
specific scenario before. Now we're at six hundred. Now you know that you
actually have gained something, and you also have the numbers that from the benchmarks
that you run against each individual part of the callstack that you try to optimize.
I have one question to you, Richard. You said you use the
NDS profiler. Do you also happen to sort of because I had a period
where I used several tools because I had sort of I know it when I
see it type of investigations, So I used several tools that have had different
sort of dashboards and overviews that sometimes gave me sort of a slightly different view
based on the preferences of the tooling against the test harness, and then I
was like, oh, there it is. Yeah, I mean different problems
in different spaces. And granted a lot of my experiences are from a while
ago where there wasn't as many tools as there are today, but yeah,
definitely there's a difference between tweaking a piece of code that you know sits in
a call stack for a web page and understanding a sort of end to end
run where it's like, oh, the real problem here is that there's a
repeated call to a database enough that it's doing a reauthenticate in the middle,
or it's forcing a recompile of a stored procedure. By the way, on
the day you find one of those from a method call and you got all
the way down to but we call it this many times and so forces is
recompiled and that creates this overhead like that's a very good day because those are
hard to find, like just a tough place get too, but you know
your point's well taken. There are Each tool provides its own viewed to that
and we ended up I think it was a dying trace where we were only
able to see all this is a multiple database interaction problem before we really saw
the behavior correctly. And guys, I want to pause for just a few
moments for these very important messages, and we're back. It's dot net rocks.
I'm Carl Franklin, that's Richard Campbell, hey, and that's Daniel Marbach.
We're talking about performance, squeezing performance out of our applications. And Daniel,
right before the break, you were going to make a point about memory
allocations. Memory allocations. Yeah, So what I wanted to say is I
feel like I need to sort of clarify one thing because I talked a lot
about sort of memory allocations. I also sort of highlighted a little bit the
CPU stuff, right, But people might get sort of the message that all
that matters is memory allocations, and I definitely don't want to say that way
because I just feel like for me, I've always sort of started looking at
memory allocations because I've seen that these are the areas in the applications that I
worked with and the systems that I worked with where I can make the sort
of the biggest impact to reduce the GC overhead without sort of going into sort
of the algorithmic complexity and stuff like that that sometimes comes with tweaking algorithms where
CPU cycles are spent. And I remember, I don't know the exact quote,
but David Fowler once sort of tweeted or is it still called tweet,
I don't know, but he shouted into the interwebs that essentially, apparently memory
stream two array and other sort of two array are still the biggest source of
memory allocations in dot net systems out there, which kind of shows how important
sort of thinking about memory allocations actually and certainly in this case of scale that
you know, when we're dealing with lots of federations. Again, I come
from the e commerce space. The other thing I ran into was we typically
had to build our load tests to run for longer because you needed to get
into multi generational memory to actually understand behavior in production. That lighting up a
load test that ran for ten minutes and wrapped up didn't give you the same
results as what was actually happening with your server, which was two days in.
Because the way that memory gets allocated over multiple generations became a huge part
of the problem not that we had to wait two days, but often we
had to go a couple of hours before you actually get that Gen two,
Gen three reshuffling a memory enough to say this is fully fragmented and restacked memory
a few times, and that's when those orphan long duration objects created problems for
us. It was like, there's this chunk of memory in the middle of
the pool and it's been there for two hours, and what the hell is
that? Screwing up every GZ But you only found that from these longer runs,
Daniel, is there anywhere that replacing a task with a value task will
be a problem. I mean, that's a common way to reduce gen zero
allocations is to use value tasks. Value task is definitely an interesting, sort
of a newer ish addition to dot net. I know that this is a
little bit of a controversial topic because some people are like, yeah, you
should be using value task everywhere. I'm more sort of in the camp of
using it where it was designed for, which is for io bound paths where
you essentially have the maturity of the calls or sort of getting a cached value
and then only like I don't know, out of ten calls maybe one or
two are actually doing the actually operation. That's where I feel like value task
is sort of a new sort of approach to sort of make sure that you
don't have that many allocations anymore. But to be honest, I think in
most systems, when you have to sort of do that type of optimizations,
then you're already super far because I bet many sort of applications systems that they're
running on top of maybe ASPO, net Core or some others, they have
other problems like memory stream two array, unnecessary bitary allocations, stringifying stuff that
doesn't need to be stringified, all sorts of that stuff. Instead of really
thinking about switching from task to value task, how about switching from web API
to gRPC. It's a good one. Yeah, I mean even for example,
switching from newtons of chasing to system text chasing right or using source generated
I'm sorry, which ones faster? Are you going to touch that? No?
Let's not go there, let's not well, yeah, I mean you
open the door, you might as well walk through it. I would much
rather to talk about a little Swiss cheese or something like that than talking about
that. Well, I mean, we don't want the listeners to get the
impression that just by switching from Newton's Soft to system text Chason, you're going
to find a performance improvement. Did you really mean that? So we actually
have seen when we switched from Newton's Jason across the board quite hefty improvements,
especially when combined the system text Chasin's US combined with source generated approaches, where
you also sort of are more AOT friendly. As an example, you have
far faster startup times on ash of Functions or aw ISLAMDA. So there is
definitely merits to to to that. But I don't wanna I don't want to
say Newton's have Chason is bad. I think it has its place. Well,
the same guy's working on system texts Chason. I mean it's James Newton
King, yeah exactly, and g RPC web right right exactly. Yeah,
being part of that new version of dot net and integrated with that team.
Like there's a lot of performance people there. You just have the resources.
Yeah, Like if the eye of sore On that is Stephen Tabb is paying
a chance in your code, your coach is going to be faster. Yeah.
Yeah, that guy's amazing. But again, I mean, if if
you if you have your harness and you attach it and do some profiling,
and you find out that the civilization subsystem is actually the problem that makes everything
so much slower, then that change makes sense. But I guess there are
also other areas of improvement that you can sort of leverage. But I would
like to, if you don't mind, I would like to switch a little
bit into sort of the benchmarking stuff if we still have some sure for you.
Because Richard and I we talked about this as well in recently in Warsaw
and in Importal about benchmarking stuff, and I think that was a point where
I said, all right, we need to make a show about this.
Yeah. So one of the things that I found really interesting is when the
first time I sort of got into contact with benchmarking was I read lots of
blog posts about benchmark net and sort of I was looking at sort of these
benchmarks out there's like ads, it's easy. It's like a unit test,
right, It's like I've written plenty of unit tests with x units, god
forbid MS test or whatever. Right, So it's like I know this,
it's it's not going to be difficult. But I was quite surprised that essentially
you required to have a different understanding in order to have a good benchmark.
And one is a unit test has two days, right, it's either red
or it's green, so fast or failed. Right. But a benchmark is
something really different because what a benchmark does is essentially especially when you use a
benchmark on net by the way, excellent tool, shout out to all the
people that have been involved there, it's it's like you are executing a given
scenario under hundreds and thousands of iterations, right, And so what it means
is there is no pass or failed. You basically get sort of standard deviations,
you get g C, you measure the GC involvement and stuff like that.
So that's what the benchmark is, right. But it doesn't just start
with understanding what the benchmark is. It also goes to what how should I
even put my code under benchmark harness? Right? And that turned out to
me the really tricky part because I was only reading about this. Oh,
here is a before and after comparison between string concatenation, a string builder and
the value string builder, which is the fastest, right, super easyst scenario.
But when you're actually taking your production system's code that you had under a
test harness, and you sort of filtered in and say, okay, there
is a bunch of things that we need to improve, and then you want
to measure it before and after. How do you take this code which is
probably not just a public method on a static class that you can take and
then put into a benchmark. How do you actually take all the sort of
make sure that you can measure what's going on, remove all the side effects
that you don't want to have in your code, so that you have reliable
sort of benchmarking results, and that you can compare the before and enough to
And that was the thing that I struggled tremendously with it, and I found
my way sort of to do it, especially sort of when I was still
sort of growing on the before becoming performance aware. And that was essentially I
went with a very simple, simple approach. I essentially took sort of the
components that were sort of on that code path. I usually copy pasted the
code into a sort of a dedicated source repository, stripped away all the unnecessary
stuff. For example, when you when I had an IOC container in place,
I removed it and I added anew of the thing that I wanted to
new up. Or if I had IO bond stuff and I didn't want to
measure the IE bound stuff, I removed it with a task complete the task
stuff like that, right, and then I had sort of a dedicated code
base that was in a specific state, in a controllable state where I knew
all the noise that is that I don't want to sort of measure is gone,
and now I can focus on that specific sort of benchmarking scenario that I'm
looking at. But it is the unit testyle test. Effectively, it was
like just bench this piece so that I know I've gotten results from that.
I'd also say, as a corollary to this, that you do get decreases
and performance with later versions because you're doing more and so often it's like,
you know, the whole conversation with the PM, because I've gotten a situation
where we literally had benchmarks as part of the CICD pipeline. Now they're coming
back and saying, hey, the new version is slower than the old version.
It's like the old version didn't do all these things you asked for,
Like this is the overhead for the feature you asked and that was getting into
SLA rules and things where it's like the customer expects us to deliver this in
x many second fractions of a second, it's like, well, we're getting
closer to the limit because the customer asked to do more update the SLA.
It's interesting that you mentioned that because I think one of the sort of benefits
of the approach that just described is that you can easily get start with not
even thinking about what is how can we actually capture regression or where should we
execute those tests? What is a reliable cic the environment in order to have
sort of measurable and statistically relevant results from this environment, right, But what
you're essentially talking about is sort of more regression testing as well. And there's
actually there's actually a lot of great guidance a little bit hidden in the dot
net Performance Repository. I think it was also driven by Adam Sidnik and some
other people from his team. I actually talked to him at a little bit
over the course of I was preparing for a talk about this specific topic.
I talked to him a lot about it, and they have a tool that
essentially allows you to when you use the benchmark dot net, what you can
do is you can actually execute benchmark dot net against the specific version of a
benchmark and the specific version of the code, you can store the artifacts,
and then you can execute it again against the changed version, store the artifacts,
and then you can use the compared to that they have to sort of
sort of create a diff between the before and after version, and then you
can define a threshold that sort of determines when it was an unacceptable sort of
regression, and then for example, you can fail your CICD environment but kind
of performance because of performance, right, but CICD. It's also interesting because
andre Akinschin wrote an excellent blog post about this topic because he did as part
of sort of investigations, he looked, for example, at git ub action
runners and the result there is that you essentially cannot use git ub action runners
to actually do regression testing for performance because they're just so unreliable. So basically
you basically have up to three times different sort of three times different execute memory
and CPO difference between builds. Right, So it's it's insane. So wow,
that's crazy. Yeah, yeah, it's it's quite fascinating. Right.
So that means you just means if you're going to benchmark issue this way,
you have to control your CICD pipeline record, right, and you have to
have dedicated hardware that you sort of put somewhere or rent basically bare metal hardware
where you and have run their infrastructure that sort of allows you to sort of
put those tests to that reliable hardware. And you can't use sure. I
mean you're saying the clouds not an option here, Well it is definitely it
is an option, right, Yeah, it's just you're gonna you can't use
the built in infrastructure for this. You have to set it. You have
to write your own YAML YAMO you have done this. Like one of the
one of the issues we were dealing with is we were up to I don't
know, three or four thousand different web tests we wanted to do, and
they took a day and we needed under fifteen minutes, and so we would
light up twenty instances of the site in the cloud and parallelize all the tests.
My goal was always by the time you got back to your desk from
coffee, the test results were back and you had failed. Yeah, right,
like that that was because instead it's still in your head. Like what
we realized was that every minute that goes by after they've pushed the codes leaking
from their mind. And if it's and if it's a day, it could
be anybody, like they're going to have to start over picking it back up
again. But if we could get it to them in under an hour,
like fifteen minutes was the magic number, they knew exactly what Oh, I
know what that error is, and off they'd go again, like it just
saves so much remediation. Yeah, so I jacked up the test bill because
it saved the dev bill. Amazing. Yeah. So, and again I
think it's it's all. When we talk about ESHA DevOps run or get up
action runs, usually there is a shared sort of infrastructure that you have there.
And yeah, so you just it's not good for benchmarket. You don't
know what performance you're going to get, no repeatable results, which oddly are
necessary. But if you're talking past fail, that's fine. You don't care
if it ran twice as long, half as long. It's just pass fail.
For functionality, that's just fine. Correctly, no big yeah. But
again my message here is I want to hammer this home. I think becoming
performance aware, I think it's a journey and you don't need to basically end
up where we just oh, well, you will end up eventually there where
we just talked about with having potentially your own hardware if you need to do
regressive testing, but already having your harness in place, understanding the profilers so
that you can sum in where you should actually make those sort of performance improvements.
Then using a tool like benchmark dot net, which saves you a lot
of headache because it's sort of sort of is already designed to mitigate the most
all the stuff you'd end up writing for yourself any direct So just use it
and a bunch of smart people are working on correct and it mitigates, and
then you can start there, maybe copy paste your code at the beginning,
isolate the things that you want to do, get started there, and then
at the later point in time, where when your company is already sort of
more performance aware, you can start slowly introducing sort of more sort of mature
ways of actually doing performance testing and regression testing all the way along. Yeah,
but you're you know, what you're applying there is like you're pretty far
down the path at that point too. Yes, to me. The big
thing here is when does performance creep into the requirements? Because a lot of
folks you know, early days of projects, it's just not even on the
radar, right, but you know, to actually file performance problems as a
bug to get them on the sprint, right, to be part of the
conversation at all, Like that's already that's arguably the starting point of any of
that path is that it's bubbled up to the point where business cares about it.
You know, the line I used to do when I did these talks
was performance is like air. You only care about it when you don't have
it. Yeah. I'm a big believer in having non functional requirements in the
design or and the architecture sort of built in and having explicit discussions about non
functional requirements. It's also prioritize them with your business stakeholders in order to make
the right trade offs. Well, and you know the sneaky part about that
is let let them tell you it isn't important, so later when they decide
it is important. But you said, you know, because again, performance
is one of those things where nobody cares about it until they do. Yes,
yeah, right, if I suck the air out of the room,
you're suddenly really interested in air. Absolutely, And it's the same thing.
It's like you never thought about. You know, you can debate all day
a render time of two seconds versus four seconds. I'm sorry, I'm so
web centric on this stuff, and that's not a big deal. Everybody knows
that thirty seconds is bad, right, and so that's sort of these kinds
of thresholds, and it's hard to talk about that in out of context.
You kind of have to make a slow page for everybody to start getting hey,
slow page bad, set requirements for minimum performance and figure out where they
are. Any good news is, of course there's lots of written documentation on
its like you don't have to invest And one of the things that I also
really appreciate by doing these types of investigations is and small improvements is you learn
a ton about the cold path in question, and that gives you a lot
of insights into a potential redesign in the future, right, because so many
people are just throwing out there, just rewrite this, right, but they
have make it, make it faster. You're not going to make it better
if you don't know what they are like this, like, yes, I
also find that most folks who spend time in the tuning part understand the x
you should have, like the behavior of software in a deeper level than a
lot of folks that wrote it in the first place, because often you're just
trying to get to the deliverable. Does the future do the future requirements that
were there? I know we only have a few minutes left, but at
one point in your investigation, do you consider re architecture, which is obviously
the most expensive and risky thing way to improve performance. But you know,
I mean, as you're going through a project and looking or a tool or
something and looking at every little thing that you can squeeze out, and something
jumps out at you, Oh, well, you know this should be refactored
or maybe even completely re architected. How often does that happen? It's a
difficult question to answer generically, but I can give you a complete example.
So I wrote a lot. It contributes a lot to the ASHER service bust
on that st K, and essentially I think it was down sort of twenty
pool requests on the sort of the path where the sort of the ASH service
bus sort of takes you get the byte the race from ASH service bus and
handed over to ASH of functions or to your code that is running. Where
I did lots of lots of tiny improvements until I actually understood sort of how
the body management of the byte payloads actually really really work. And then only
then I came up with a better idea how to sort of manage that that
body work from different aspects, and that then led to sort of even more
orders of magnitude of improvement how the body is sort of managed and less allocations,
more efficient in CPU cycles. But it was like I think, Okay,
I contributed in my free time, whatever that means these days when you
were constantly online, right, But I contributed in my free time, I
guess, over a year to this code base, until together with the team,
we realized, oh, there are there are actually things we can sort
of really refactor and make things even faster. So I get I'm actually I
have the tendency to go a very long time on a specific code path before
I even reconsider re architecting or redesigning. Of course, small impress can also
sort of sometimes mean you're not doing up something. You're making something a singleton
that previously wasn't a singles or something like that, right, and if it's
still not performant enough, that's when you think about re architecture, because what's
also great is right when it's running in production, it's making money right,
and it gives you insights about the good about what it's doing. And when
you're architectu and redesigning for that period of time, you have no validation whether
the stuff that you're changing towards will work. And with the small improvements you
have constant feedback loops. And I think I feel that's super super important.
So what's next for you? Man? What's in your inbox? Well?
Christmas time of course, drinking way too much graft beer probably over Christmas time,
a delicious stout or something like that. What is this? But but
next year I will be I will be in dot Net Day Romania at eating
US conference. I will be delivering a workshop about reliable messaging in ASHER,
sort of deep diving into ash service, bus storage, ques, event hups
and event grets. That's going to be really interesting. And what else I
don't know yet from a conference perspective, but definitely sort of increase a little
bit more of my contributions to open source stuff because I still have a few
to contribute to a few open source libraries. Yeah, it sounds good,
well, Daniel, thanks for spending this hour with us. It's been great,
Thank you all right, and we'll see you next time on dot net
rock. Dot net Rocks is brought to you by Franklin's Net and produced by
Pop Studios, a full service audio, video and post production facility located physically
in New London, Connecticut, and of course in the cloud online at pwop
dot com. Visit our website at d O T N E t R O
c k S dot com for RSS feeds, downloads, mobile apps, comments,
and access to the full archives going back to show number one, recorded
in September two thousand and two, and make sure you check out our sponsors.
They keep us in business. Now go write some code, See you
next time in the band. Sadly seen a summer time that means hard than
my Texas in Line Revell