Hey, folks, welcome back to another episode of JavaScript Jabber. This week,
on a panel, we have Dan shapere Hi from a very hot Tel
Aviv. We also have Steve Edwards. Hello from a very hot Tel Aviv
style of Portland. Right, it's I'm like really hot here. Like,
yeah, I'm Charles Maxwood from top end devs. Yeah, it's it's been
getting warm here too. I think it's all over the place, maybe maybe
out where our guest is too. We have a special guest this week.
It's uh, I'm not even sure how to say your name, ven that's
me, Yes, called Vinnie for sure. Yeah, Vinnie. I'm pretty
used to being called Vinnie, to be honest. Yeah, my cousin Vinny
exactly. That's kind of how started whenever I started speaking to American people.
But yes, Venice is from Stockholm, Sweden, where I wish it was
warmer. It's been a very rainy summer, especially rainy and way worse than
normal Swedish summers, even though Sweden is not necessarily known for the good weather.
Yeah, I was gonna say, I'm kind of wishing for that here.
But I saw a headline a couple of days ago in the newspaper that
said that you tie is no longer in a drought, and in fact,
all of our reservoirs are over full. So that's maybe we don't need the
rain. I don't know anyway. Yeah, we brought you on to talk
about performance, and you shared an article with me before these guys got on.
I know you've also been chatting with Dan. So I'm just gonna let
you guys take the lead as far as where we go, and then I'll
chime in with my basic question since I am not a performance expert like you
guys are. Yeah, sounds good. You guys you mean those two because
I'm not the performance expert either. Ah. Yes, you know, performance
is kind of important. I don't think everybody can should or can be an
expert. But and I guess this is one of the things that we'll be
talking about. It's not something that you can ignore either. Let's put it
this way, right, Yeah, Yeah, it's uh, it's it's one
of those topics that it's like the more you know, the more you realize
things go deeper, and you can't get pretty deep, you know what.
Yeah, but one example I've heard frequently mentioned, and I've thought about it
myself is you know, I'll look at stack overflow posts and people talk about,
Okay, what's the best way to loop through an array that's more performer,
or what's the best way to you know, handle large data sets in
code and and sort of stuff like that, and people will obsess over these
little things that will increase performance, and then they've got a six meg image
file downloading on the same page everything anyway, you know. So, so
I think it's safe to say that when you're talking performance, it's sort of
got to be comprehensive and not just focused on you know, code performance I
think, or bundling of code and that kind of stuff. It also needs
to serve a purpose at the end of the day, which again is something
that I guess we will be talking about. But performance is not is it's
not an ego trip. It's not about you know, bragging rights. It's
about serving your customers, your users. So anything that you do needs to
be with that front and center. If it actually brings value to your users
and your customers or or it doesn't, and if it doesn't then it's kind
of pointless. Yeah, yeah, absolutely, And the whole micro benchmarking thing
is it was a very good example, like it's very easy to look into
like things that sound very buoyant, flamboyant, and you kind of missed the
target where the actual hurt is, which is on the user's experience. Also,
you know, in the context of JavaScript, it's lies, damn lies
and micro benchmarks. Yes, that's a good one. Yeah, because the
way that the JavaScript engines work, the modern JavaScript engines and the optimizers they
contained. I've read some quote unquote horror stories about you know, people drawing
conclusions from micro benchmarks that were effectively not just meaningless, but in fact holy
misleading. Let's put it this way, like people wrote loops, but because
it had no side effect, as it were, the optimizer kind of optimized
the loop into nothing and then what are you actually even measuring? So so
yeah, but put to put it bluntly, I wish my problems were how
fast you know, some loop in jobscript works, although occasionally it does actually
happen. And I'll finish with that. That this part I actually contributed back
to the Prometheists client for Node and I don't know if you're familiar with it.
It's a mechan it's a system or service for monitoring and alerting and stuff
like that. And the optimization was actually an optimization about how to loop and
build the response string because in that particular case, via profiling, I proved
that it actually did make significant difference. So it can, but it's usually
does anyway. Now from my titter chatter now over to venis yeah, the
I mean the benchmarks and has to serve a purpose, right, And if
you just benchmar like if you're micro focusing on things, they might not be
painting a good picture for even what you're even trying to measure. This actually
brings to a very interesting conversation that I've had a chance to have a JAS
nation in Amsterdam couples of a couple of weeks back with Ryan from Solid Jazz
and Attila. So we were talking about how can one build well structured benchmark
to test different frameworks on some aspects, right, And we started talking about
how it would be good to have something that can bring TODs web virus because
it's something we already kind of have standardized, and that's how we can natural
measure actual impact and like break down different threshold like different kind of common problems
where you can try to build correlations and build a good delta between different frameworks,
like what kind of strategy spays off the most, right, Because Solid
does it one way, he React does it another, So it cannot necessarily
measure them apples to apples in a way, And so you need to try
to establish like a good baseline of how can you benchmark those kinds of things
that are in the end just deliver the kind of similar results, but they
do it differently, so you're not really trying to benchmark in a way the
framework itself, but different strategies that they use. Yeah, it's an interesting
conversation. First of all, it's worth mentioning that Ryan is kind of a
regular guest here on the podcast. We've had in quite a number of times.
He's an incredible smart person, like you said, is the creator of
solid Solid Start, is also kind of the CEO of Signals really popularize that,
and is also one of the people most knowledgeable about frameworks in general.
I mean, like his hobby is to get his hands on, you know,
more or less every framework out there and then do all sorts of comparative
testing and analysis and whatnot. So, yes, if anybody is knowledgeable about
how to best compare performance and other aspects of frameworks, it's probably ryan.
It's also worth mentioning a tool created by the Quick people called Mitosis, which
you can actually use it to compile like sort of pseudo code like React like
pseudocode into various frameworks, and then you can kind of more easily build the
same application using different frameworks and then be able to compare it. Because,
like you were saying, if we want to get away from micro benchmarks and
we want to actually compare real applications built in various frameworks, we run right
into the problem of the overhead of building and maintaining sophisticated applications in a variety
of frameworks. And you know who wants to do that. I just do
want to mention that there's an alternative approach if you're interested in the performance of
various frameworks, and I actually gave a talk about that at several conferences,
including I think jays Nation a year before, which is to use rum data
to compare the performance of frameworks. So you're not looking like at a specific
application, you're looking across all websites built with a particular framework, and then
you're not so much saying like how fast will my application be with this framework.
You're more looking at how likely am I going to how likely is it
that I'll be able to build a fast website or web app using a particular
framework, And you might say this framework is more likely to produce faster websites
or web apps, and this one is less likely, you know, not
surprising. The one that is least likely is angular, and the ones that
are most likely are Quick, solid spelled you know these. But it's really
you really need to be careful about mixing correlation and causation. Like, for
example, people who use Quick are people who are more likely to be concerned
about web performance. So are they producing faster websites because they're using Quick or
are they producing faster websites because there are people that care about web performance and
then so they know what to do and they also happen to be using Quick
for that reason. So you need to be kind of careful but doing these
types of conclusions. That's a very good point actually, because it's one of
the things that I like you come to realize when you start collecting run data.
So I established I helped sublished out time out, time out. I
know that we've defined this in other episodes. But run data what is it.
It's real usermetrics. So you have two different sides of performance monitoring.
So you have what is called the lab and what it's called the rum side.
So the lab is just you know, CICD lighthouse and whatever you do
to make sure that you don't perform aggressions before changes actually go live to users.
And on the run side you have different providers and you can even use
like GA Google Analytics even have like collect automatic luber vitals. Yeah, it's
it's basically the question of are you testing your performance in a synthetic lab style
setting, maybe even on your own computer while you're developing and comparing that way,
or are you collecting performance data from the field, from actual, real
user sessions. So obviously you can collect data for your own particular website,
especially if you have enough traffic coming in. But the interesting thing, and
we actually had Rik Viskomi from Google here to talk about it. Google actually
collects data from all sessions on Chrome and they put it into this database called
Chrome User Experience Report or CRUs for short. Here and Steve usually makes a
joke about the crux of the matter or the crux of the issue or whatever.
Yeah, exactly. And the nice thing about what Google does is that
they actually give everybody essentially access to this data. And also they attach all
sorts of meta data to this data, like you know which framework was used
to build which website and whatnot, so you can do all sorts of slicing
and dicing, so you can look at the performance of you know, your
own website or competing websites or eCos websites in general, in particular geos,
or particularly particular types of devices, or created using particular frameworks, using particular
libraries, et cetera. It's interesting data for those of us who like to
geek out on performance. Yeah, that's a that's a really good point to
start. So for like for building, for building web websites, like,
things can go wrong in many different ways even when people use the same framework
and two point exactly done like some people when building like because also like when
you pick reacts, most people pick reactors, and one of the most like
is probably now these demos used framework out. Really, I don't keep up
with the trends, but oh yeah, for sure. Yeah, so's it's
it's just because of the sheer volume you have, right of people writing like,
there would be all kinds of quality out there. It's kind of the
game. It's a case in point effectively, just to throw the numbers out,
there are many as many websites and web apps being built in React as
all the other frameworks put together. Yeah, yeah, I would, I
would imagine so and so in that mix. There would be a lot of
so when you're trying to divide, like if you're building some sort of collection
tool based on craucks and you're trying to divide persontiles for each framework, the
React having so many more samples, right, it would kind of push the
data towards the left, or rather towards the right side, where you're going
to have more worse metrics overall, just because of how many just the sheer
volume of it. Yeah, that's one aspect. Another aspect is the fact
that a lot of React websites, like there's a long tail of React websites,
websites either built like long ago or built with particular focus in mind.
So, for example, if you're building a website in React, then you're
not using service side rendering or static side generation if you're only using client side
rendering, which means that you're building the dorm representation of the website on the
client side, then by definition, effectively you're not going to have good cowed
vitals, which is the way that Google measures loading performance. You might have
other aspects of performance that are good, like I don't know, responsiveness or
stuff like that, but in terms of loading performance, if you're just using
client side rendering of CSR, it's not going to be good. And the
reality is that a lot of reacts websites you see s are some of them
don't care. If they're not indexed, then maybe they do. They don't
care, But if you're looking at the metrics that Google is collecting, that's
what you see. Yeah, And it's interesting that the whole pivoting that we
are now having tools the service side as well, because one of the things
that like I've I've worked I've worked for Spotify and I worked for Klana before
Spotify, and in both places have set up the like monitoring tools and start
collecting round data and all this kind of stuff. And for CSR, one
of the strategies that people have is normally like cold splitting lie that's pretty much
your only or one of the few things you can do to try to improve
that first paint and then in the LCP and even that, like when I
run an experiment once trying to understand especially for CLAN when I was working at
KLANA, I was trying to build benchmarks on how can we make sure that
we have as a third party, how can you make sure to affect the
loading of the site you're integrated with the you know the least of how can
you make sure that you're not the one who is causing the harm? And
I've ran an experiment on on on doing cold splitting. And back then,
mind you, it was gosh, it was like twenty six, twenty eighteen,
I think or twenty seventeen, so it was I've like did like a
full cold splitting kind of like manualal splitting, splitting chunks everywhere and like lazy
loading some stuff, And back then it was kind of a revelation. Now
there's I think more people understand why. But like when you do that many
chunks, things can get very congested when you're loading, when you're loading websites
and doing a lot of JavaScript small chunks. At the time, it was
a lot in compression size as well, and you're also going to be hogging
you know, CPU time and back then not even using HDP two if I
remember correctly, if you though it was already available, So you know,
like that the whole critical render path was absolutely destroyed us in congestion. And
I think back then we didn't even have the priloscanner to try to like hacken
to as like priority and all this kind of stuff. Yeah, so it
was like back then it was a revelation on yeah, you can have too
much of a good thing and then like start working with prioritization of chances and
this kind of stuff. Yeah. We actually had Robin Marx recently to talk
about this whole network thing and how it often behaves in a way that's not
expected and to bring it to a more general aspect and kind of related to
the main topic that we've yet to focus on. I would say that one
of my main things regarding performance that I always say is that you've got to
measure. You've got to measure in order to decide what to focus on,
and after you make it change, you got to measure to verify that the
change you made actually improves things and doesn't even potentially degrade them. And I
can give a case in point, like you know, if your CSS is
small, a lot of times you will hear that you should inline that CSS
into your HTML to avoid the extra round trip to bring the CSS because CSS
is rendered blocking, so you want to get it down as quickly as possible.
And I've seen cases where inlining the CSS actually degraded performance. And you
know, and without going into the details of why, the fact is that
once you see that it's actually degraded, you roll it back, yes,
because you know it is measuring the whole measuring its Like I always mention,
whenever you start working with performance, you always start from the data. You
always starts. If you don't have collection story, if you don't have a
good data story, you have to start collecting data from your users. That
is right, so because because lab data doesn't really tell you the whole story,
as you've been breaking down earlier, So you need to understand how is
the actual user experience out there, and you need to understand even what kind
of data do you want to collect, which kind of brings us into the
topic we've been discussing as well on the article and the whole thing of understanding
your product and where you want to collect data and what they're collecting, and
you know, building the story around performance from that perspective first instead of just
trying to micro optimize. First, is how you're going to make sure to
deliver impactful results to users. So yeah, So, I mean I read
part of the article and yeah, and what I'm wondering, you know,
just jumping in. Then you talked in your article about lab versus rum and
things like that and how to measure it. But I mean, if I'm
if I'm really new to this, right, and you know, Dan mentioned
having measurements and knowing what your numbers are so that you can, you know,
verify that you had the effect that you wanted. How do you start
gathering that? I'm assuming that's kind of your first step, right, is
gathering that information, whether it's in a lab setting or rum setting. Yeah,
I mean, very often, if you're like engaged in a performance project
or related project, you'll be under kind of the gun to show results,
and it's kind of tempting to start optimizing things. And whenever I was engaged
in projects like that, I quickly pushed back and said, no, we've
got to get the data first, We've got to have the graphs. We've
got to have the measurements because otherwise you're just working blind. And if you
want an extra incentive when it comes to your time for your annual review or
half annual review and you want to like prove your worth to the company,
having a graph that shows like this is what I did. It's a line
that goes up down depending on what you're actually graphing is a great thing that
you know, you want to have when you're trying to get a raise or
something like that. So I literally, in various occasions pushed back against management
who were trying to get me to start optimizing before we actually had good measurements
in place. And and to your question, Shark, We've had some guests
on the show to talk about you know these days. There are really two
good, really good ways that I have to mention to get good metrics.
One is if you is basically just use the Google the Google Search console.
In the Google Search Console, they have a Cowork Vitals panel and you can
get you know, information about pages that have performance issues. It's it's not
an ideal and optimal rum solution. You know. For example, they average
the results over twenty eight over a period of a month effectively. So improvements
you will make will take time to show, and likewise, degradations will take
time until they actually manifest themselves. But you know, it's a good starting
point. Another good starting point is that they're like an issues mentioned before,
there are a lot of third party tools and services out there that are fairly
straightforward to integrate into your website. Some of them are even like partially free.
But you know, we've had people from Centry, they have a rum
solution, we had people from gun Ray, gun Akama, We've had people
from Akamai. They have the impulse that there are a lot of there a
lot of great tools out there that are fairly straightforward to integrate and that you
can start getting data from. You know, yeah, you do. You
have a lot of options like debug, beear and rum vision as a speed
curve. But there is the question real quick, we're talking about tools,
how often do you see tools because a lot of these tools are basically jumping
JavaScript into your page, right, so I know Google Analytics is a classic.
So how many times do you see the tool that you're putting into measure
performance hurting your performance because of everything that's putting in your site to measure performance
fun you ask, I have heard horror stories with ot L and UH integrated.
I'm not gonna yeah, I'm not gonna name it any games, but
I haven't I have heard interesting stories when trying to implement open telemetry, and
that's one of the examples where it can go pretty bad in one way when
you're trying to measure Yeah, I remember, now if I had a service
that you could pay for, I haven't seen it a while where whether they
would run stuff like that on the server so you know you're not dumping JavaScript
into your page. I never used that reading about it, seeing it as
an alternative. Most of the back ends have services like that too. And
sure they don't give you like the core web vitles right because they're not Yeah,
they're front end things because yeah, because their front end measurements not back
end measurements. But they give you some information about how long it takes to
Yeah, but I will say this, you know, like Venius mentioned,
you kind of need to be careful. But most of the rum providers that
actually measure coed vitos, you know they know about with vitals, they've done
work to ensure that whatever they're providing has has little or no impact on your
score exactly. Yeah, if one of them does, it will be it
will pretty quickly come out. But you know, just google it, your
friends, search what people say about it, and you know, try it
out yourself and see what happening. Yeah, and on the whole topic of
actually, on the whole topic of back end, I've wrote an article last
yeah for Perth Calendar about server timings, and I think there's there's a good
amount of things you can do to help understand and like end to ends sort
of tracing when it comes to even to back end and stuff. I mean,
we don't have it as well standardized as web vitls, but understanding because
one of the things that a performance specialist becomes is like it comes in three
parts, right, Like it's a salesman, is a data analytics person that
has to understand what kind of data you're trying to collect and understand the data
collected in order to visualize it well. And you know, a good an
engineer side as well. So the back and part. There is a whole
lot of attributions that you can build within the back inside. Unfortunately not as
well standardized as web vials but but that's actually a good opportunity right where one
can try to define good metrics within back end. But like when you're trying
to understand especially for instance, when you're now starting to have more react to
being around on the severn right with Next, the no neuros versions of Next,
the newest version of React, and things are kind of moved will be
moved a bit more to the back end and processing part time is going to
be spent more. You know that TTFB window, and it does have a
great amount of benefits that you can get out of here, but some things
can be a bit unexpected, like so you can have some of the performance
degradation as well on some types of websites out there, because not all web
sites, perf have trying to differ the same kind of experience. Does not
all parts of a virus matter equally, so do all the websites, And
so if there is if you are working on something that is very sensitive to
that CTFB window, you really want to make sure to understand from the server
timings, you know, from behind a curtain network curtain what's going on,
and you can collect that data as well. Yeah, and and by the
way, like I said. We had Robin Marks on the show, and
one of the things that came up is basically that if you really care about
TTFB, and especially if you've got a global audience, then you probably want
to be using a CDN and then the TTFB of your own server is not
irrelevant, but is you know, less less impactful I call it. Let's
put it this way, but I would actually like to pull us back a
little bit and looking at your article which I have read. Uh, one
of the things that you wrote about that really resonated with me because it really
matches my own experience is about getting buy in from management. Can you talk
about that a little bit? Yeah, absolutely. It is the salesman part
of the of the triad, So it is it is like, in a
way, you really have to understand the product you're working with. So if
you're working in a product team, one of the things that you constantly hear
and I've certainly faced it myself as well when I was working on product teams,
is you have a certain pressure to ship features constantly. So sometimes sometimes
and sometimes more than others, you will have pressure looming and trying to solve
backlog stuff, which some in most cases performance kind of falls under you.
You it's hard to get that buy in coming from management. So one way
you can start doing it is, of course, as we mentioned, COLAC
data, and with that data you collect, you want to try to understand
correlations between you know, pain points of your users and how your product is
performing. Because when you're trying to build work within performance, you really want
to understand on if you work on a certain part of application, you will
deliver the most well the most optimal results for the perform for the products to
perform. So you're not only trying to cover the engineering side, but you
will like when getting buying, you want to make sure to cover the product
side. You know that the shipping good metrics not only for EPMs and for
your performance metrics, but also understand how those who affect your product metrics.
So the business side also matters. So you have to wear kind of like
both hats and build that bridge between engineering and product. That's how you can
strengthen the buying you get from management. I totally agree with that, and
it kind of reminds me of an interest seeing experience that I've had quite a
number of years ago. And I won't be naming names but I remember talking
with this person. I was working at a certain company and I was speaking
with one of the people in product and I asked that person why their product
specifications never include any performance requirements and they answered first he said, I don't
know how to specify performance requirements. And I get that, And it's become
easier these days again thanks to Google with co wored vitals we have. They've
made the industry aware of a relatively small set of well defined metrics that you
know, you can tell that person from product Hey read this article. I
actually even gave a talk about it at the previous employer to people to kind
of teach them about how performance is measured and the impact of performance and whatnot.
So's that was his first point. And the second point, which really
made me laugh, he said, but I expect our developers to write the
code in the most performed way possible, because that's what developers do. And
I really started laughing, because you know, we work, like you said,
we're always in a rush to deliver features. We're always under the gun.
We're always facing short time frames and not enough resources, and certain aspects
like performance, can you know, go out the window in these types of
scenarios if you're pressured to deliver a feature and you know you won't be really
worrying about the performance impact if you hardly have time to finish the requirements that
are actually in the spec and and yeah, so so if you want.
And that's from my perspective why management buying is so important because very often we
hear about people saying, hey, can we do stuff like that grassroots?
Can we introduce quality like from the bottom up and stuff like that? And
the answer is no, because if you're going to be engaged in a large
scale, ongoing project that's going to require a lot of resources, time and
effort, maybe even money, if you're let's say buying rum tools and other
and other things, then management needs to be aligned with that otherwise it just
can't happen absolutely absolutely Yeah, and then becomes a fight right over what your
engineering team interests are and where your product team trying to ship and and you
know all the things that product teams have agreed on for quarters. So it
just does become like you need to get a lot more involved with the site
and understand what kind of values matter the most for the product side. And
that's why it becomes a salesman pitch because it's not just blindly trying to web
viders have done wonders for that conversation start. Like, I think most people
nowadays understand some aspects of revisers are at the very least when you mention it
that it matters for CEO, it matters for the product. So that's great,
that's really like because I still remember from the days of Klana, we're
having to build, you know, like what metrics matter the most for our
product? Was was an actual task you had to So many metrics are available
to you and so little knowledge around how they affect different parts of the product.
But when it comes to building those kind of values and translating them to
product, it becomes onto a where it's very important to understand the product that
you're working on, because like, not all products will have like as I
mentioned a bit earlier, like not all products will have the same kind of
values. Or you can't apply all metrics blindly to all products because some metrics
just don't matter as much to certain type of products. For instance, me
working at now, I've overcause one of the products that I'm currently working very
closely with is the configurator that they have and the configurators as a as a
product is just an actual product team that cares for it, and it does
have very very different business metrics. Product metrics then one would think about when
you think about conversion and all this kind of stuff. So how the product
behaves in real time is a lot more important than how it loads, because
you can do a lot of stuff to try to trick it loading, but
once it is loaded, that's the thing. Like you have the I in
P metric for all that product is like king that's the most important part IP
and cls, so you know you need to and which metrics is going to
be delivered the most impact all the types of products you're trying to ship.
So you can build you know, KPIs and slas with downstream teams and try
to have this kind of consolidated governance model or the least just understand how things
impact your application differently, and from that part within the engineering, you can
understand also how to tie up those engineering metrics with what kind of product metrics
matters the most, and you can try to build correlations from that. I
totally agree with what you just said. An interesting case that I had experience
with was when I was working WIS. There was the WIS editor that you
would use to build websites using withywig dragging and dropping things around, and you
had the actual websites that you built with the WIS editor. So for the
websites, the most important aspect was the loading performance as measured by let's say
LCP Largest Content for pain but for the editor itself, I mean, obviously
you don't want it to take half an hour to load, but most customers
didn't care so much about how long it took the editor to load. What
was more important for them was that once the editor was loaded, it was
really responsive and snapping, that when you drag things around, they would drag
smoothly and not like you know, with this jitter or trailing after the cursor
or stuff like that. So you really need to be cognizant about what it
is that is important to your users in the context of performance, and what
it is that you want to optimize, and again going back to the aspect
of getting buy in from management that it's actually worthwhile to invest the effort because
one of the things that I like to say is that performance is a journey,
not a destination. It's not like I'm going to invest a little bit
of time and effort get their performance house in order, and then we can
forget about it forever because it's done. That's not how it works. You
need to put systems in place to identify regressions because regressions will happen, and
then when you detect a regression, you're going to have to invest effort in
fixing that regression. So it's ongoing effort. And by the way, in
this context, I'm going to post a good link, so again, thank
you to the people at Google. Their web dot dev website has a section
called case studies where they where they post a lot of case studies about how
improving performance has impact variety of businesses and products. So you can actually go
in there find services that are similar to what you're building, and then be
able to bring that to management and say, here, you know, this
company X does something that's similar to what we're doing. This is how improving
performance has impacted their business, their bottom line, and this is why it's
worthwhile for us to invest this effort as well. Yeah, yeah, absolutely,
and there. You know. It's funny that back in the days where
I actually have an article from that time. I'm posting here on the chat
and the building building and then performance monitoring that was for lab That the article
that are based based there. So it's like build before the time when we
had c I c D with Lighthouse. So I was trying to integrate Lighthouse
into c CD manually and build my own C I c D server with Lighthouse.
One of the things the Lighthouse allowed I think it still allows us to
do. I haven't dug into the Lighthouse source code for quite some time.
It was to define different weights for different metrics. So when you get a
Lighthouse score from from from within your c I c D, you can just
find, for instance, you for if you your application Maths is more I
pay much as more for application, you can customize the weights of the score
you get from Lighthouse based on that. At least you could back then.
I'm not entirely sure if you still can because I haven't dug into the source
code for some time, but it's one of the things that back then top
and source so you could probably do whatever you want if you. Yeah,
the effort. Yeah, But like back then when I was doing that kind
of integration, is that defining for for instance, back then, gosh,
like one of the metrics that we used to have, I mean l CPFCP
still where we're around, so like those as a third party at Tallana was
the ones that matter the most. So I like, you can find tune
the scores based on the weights that you want, you know, matter the
most for you. So we kind of talked about it already, but still
one of the sections in your article is about metrics and KPIs and s l
as. Can you elaborate about this? Yeah, absolutely, So the so
government, if you try and establish governor if you try governance, and or
if you're trying to get a better understanding even from what you're trying to ship.
And normally we don't I mean we don't work in asolation. And as
you're working for a very very small company, normally you have teams either above
you or below you, so you are d downstream or you are working teams
downstream, so a graph service or an API or a back end. So
you you after you understood which kind of metrics master the most for you and
for your product, and you're building starting to build the correlations. One of
the natural next steps is to build KPI so key performance indicators for your application
with a lens of performance of course, and building also a service level agreements
as a as against teams that might affect your performance. So this comes with
in you know, the any KRE person would know those by HARP. But
it's one of the things that really up to strengthen your because it's pretty much
what you said, it's not a race. It's a continuous thing, and
you will have regressions, and whenever you have regressions, you first of all
want to document those. And you always want to document both regressions and any
performance improvements that you shape because you want to understand how things you know worked
over time or not. And having those esslays is a good way to communicate
downstream and sure to have more resilience on things that you're trying to ship.
And because some parts of the performance when you work in a big product,
some parts of the performance spectrum is not directly coming from you. One of
the things, for instance, that we have and I think most people out
there will know this is GTM and GTM can really you know, if you
have a marketing marketing team that's pushing a lot of tags on DTM that can
really mess you up on performance and it's not even coming from your code base
necessarily. And that's Google tech Manager, right, Google tach Manager. Yeah,
So so Google Tech Manager is a brilliant thing. It's a really really
interesting product. But anything can be misused and such as the such as the
nature with GTM in some teams and one of the marketers junk drawer. It
really is. But yeah, yeah, and one of the things that I'm
currently trying to invest in a lot of time in the governance model that I'm
trying to build this is around GTM as well. But then how can you
even it's like those the other side with the GTM side is a completely different
set of skills, So how can you make sure that you build a connection
on things that both of you can align on and understand. So that's where
the KPIs and slas come in, and you can you can really make sure
that you can have a proper well structured process around it. By the way,
we did have I think Adam Bradley from a builder io who spoke about
the tool that they build called party Town which can which can reduce the impact
of third party scripts and pixels and whatnot by moving them off of the main
thread onto a web worker. The big challenge is that it's a pretty manual
effort. You know, we spoke about needing management buy in in order to
approve effort. That's one of those things. It's not a drop in solution
that you can just put in your product and watch the improvement. You probably
need to do some manual integration work and in order to get the real benefits.
But when you do, the benefits can be substantial. In the case
of Next Insurance, where I previously worked, by putting in party town,
we were able to dramatically improve our I n P. The number of pages
that have good I n P scores went from about something like less than half
our pages to effectively all our pages thanks thanks to using party tap. Yeah,
because Yeah, Unfortunately, a lot of these third party scripts can have
significant impact on the responsiveness of the website. They run a lot of JavaScript
on the main thread unless you know you can figure it otherwise, and that
impacts how responsive the pages to user interactions. Unfortunately. Yeah, and that's
why building a good attribution model is is like absolutely fundamental. It's one of
the things that I'm obsessing as a late because once you have the metrics up
on the monitor into that's a great start. You're off to a great start.
You are starting to understand, you know, the nature of a product
and you're starting to understand how the things you do impact it. But then
again, as previously mentioned, like performance comes from a performance recradition can come
from different passive web you have the bugbear I should release a great, great
article just a few weeks ago about the impact of Chrome plugins. So if
you have like different different plugins within Google Chrome, they can also affect your
your performance, and some more than others. And there's a brittiant article there.
I'll see if I can find before these recording ends. But it's really
it's a really good example on how performance sometimes you can have some degradations coming,
you know, on your rear usermetrics, all nonetheless and coming from different
kinds of things, even out of control. So building good attributions means not
only having the metrics, but you know, web vitals have an actual attribution
build that you can do use where you can send more information about each one
of the metrics you're collecting. So if you have you know, i NP
you will show you if you will like details about the love data long animation
frame, and you can have a better intelligence on where is it coming from.
Is it coming from a third party, is it coming from or even
a comm extension. Is it coming from Google Tag Manager or something like that,
and that can give you more even more power we're talking about. Yeah,
a lot of the tools provided by run providers have kind of attribution capabilities
built in, so you know, you don't necessarily need to be an expert.
But that's actually another point that I want to mention is that it does
require some expertise. I don't expect that you know to if you want to
do some performance improvements on your website, it does require some know how.
So either you know, you find it really interesting and you invest the time
and an effort and learn it. Alternatively, you can bring in an expert,
maybe hire an expert if you're large enough. If you're not, you
can bring in a consultant, somebody like Roberts, which we mentioned before we
started recording. We actually had him as a guest on the show. Can
bring on somebody like that to do an audit for you and kind of point
you at the right directions. Yeah, I'm curious with you mentioned Chrome extensions
as an example, and you have attribution that that tells you when you gather
the information. Hey, look, you know people who have this extension are
having this impact right one way, one way or the other. What do
you do about it? I mean, can you just add it to your
CI for some of your tests to check for the performance to get you know,
once you've set those performance standards, do you put a banner up that
detects, Hey, you're running this Chrome extension and it impacts you know,
slow down your thing? I mean, yeah, what do you do with
it? Yeah? No, So first of all, yes, you can
put it as part of your test environment for sure. The question is is
the slowdown happening because just the fact that the extension is there, like gives
this extension just doing generally a bad thing, or is it something that's kind
of a bad into react, you know, some sort of reaction or or
like, you know, your page specifically doesn't work well with this extension for
some reason. Now, if it's the latter, then obviously, if it's
and it's a popular extension, then you may want to invest effort in trying
to fix it. Like, obviously you can't fix the extension, but you
can try to fix your stuff if it's the former. If it's just a
problem with the extension, basically it is what it is. You may want
to put some notice on your website. I wouldn't pop up an alert that's
is overly intrusive, but like you know, having something in your knowledge base
so at least your support people would be cognizant of the problem and if a
customer complaints, then they can ask them, you know, are you using
this extension and if so, please don't. But it's like a fact of
life, right, yeah, So on that, we have a great talk
from last year's Perth now by team Verike. He mean is the name of
talk is noise noise canceling rumor or something like that. I posted a link
here on the chat and that talks about what can you do to eliminate noise
out of your metrics? And that's like, that's why having this kind of
attribution and understanding your attribution, and you're very right like this this kind of
and this is one of the problems when you to performance as a subject is
because it is somewhat of a very steep vertical for people to get into.
But one of the things you can do is once you to start understanding these
kind of things is to understand also how to remove noise from a metrics and
understand better how your actual personiles are split up onto and understanding the distributions because
it also, like building better attributions for a product, comes from bringing back
to the whole, like tying up with your product values, right, it
comes to understanding what kind of market markets means much as the most for your
product. Right, So you don't want to really measure globally because the bigger
the metrics scope you're trying to have, the harder it is for you to
find attributions or the hard it is for you to understand them because you're trying
to deal with something that is too large of a context. And it's a
lot easier when you're trying to slice their data up. And like let's say,
for certain products, certain markets are a lot more important, and for
that product, certain pages or fear functionalities are a lot more important. So
you definitely want to slice your analytics and your data and your run metrics in
such a way where you can cut through the noise that you gather globally into
something that builds better attribution models and builds better relations out of it. So
that like removing this kind of noise from a metrics is also a really really
good way to avoid I'll give a different example for you know, it might
be that certain geos have performance issues. So for example that again talking about
Wiggs Global Company, we saw very different performance profiles for example for North America
and for South America. On the other hand, you need to be careful
with these sorts of things because for example, you might say, hey,
something really happens bad happened in Colombia for some reason, I'm seeing a significant
degradation, and then you realize, hey, I only have like eight sessions
a day, so if just one bad session can skew my my data for
that geo, So you kind of need to be careful. And again this
is something that you also talked about in the article about how you need to
be careful with blind spots, like how you know, some data can hide
or obscure some other data and cause you to miss important aspects. Yeah,
yeah, yeah, it's like I bring within the ark, I bring to
the same talk by Tim and it is like it is one of the things
that it comes again from the triad right where you that's from the data analytical
part where you need to understand the alternal the data you have, because otherwise
you might be looking too too much noise, and then it's harder to work
with that data. Right Like, then your performance work that you're trying to
carve out of the data might even be is aligned, So when you're trying
to ship performance, you might not even see the needle moving properly because you're
just being affected by by noisy run data. I will say one thing,
and I think we can summarize this particular aspect with it, is that I'm
also a member of the W three C WHERE Performance Working Group, and there's
always tension there with regard to attribution because you've got the run providers who are
pushing to get ever more accurate and detailed attribution data that they can externalize and
expose to their customers. And on the other hand, you've got the browser
manufacturers who are concerned about privacy issues. So you know, with too much
attribution that opens the door to all sorts of fingerprinting techniques and stuff like that.
So you kind of need to strike a good balance there, or put
another way, you know, yet another reason why we can't have good things.
Yeah, that's a good point. Well, there's valid concerns, so
yeah, it is a valid concern. It is very good. They're both
valid concerns. I really want to know what's causing my headaches, but yeah,
I also want to protect my users. Yeah, there's a very good
ways to anonymize that data, and most most collection tools will make sure to
do so. But that is a very if you're building your own collection tool,
that's a very valid concern to have. Yeah, and again it's also
a question for the browser manufacturers whether or not to provide certain APIs or not,
because you know, obviously, if you could ensure that only good actors
actually use those APIs, that would be wonderful, but unfortunately the world doesn't
work that. Yeah. Absolutely, I think we're nearing the end of our
show. Is there anything in particular that you wanted to mention that we haven't
mentioned so far? I mean, I can basically summarize that working with performance
in a company large or small, it starts with data, and understanding that
data is naturally the second step because a lot of times you're very eager to
get you know, get down with actual code and try and move that.
You know, sometimes it can be a bit too early, and you're if
you try to work on something that is not going to be shifting not only
the actual application performance towards the right way, but not affecting the right product
metrics. You know, it's it's very hard to get that buying again and
just gets harder to get a performance work off the ground. Of course,
if you're working with a mature engineering organization, that can be a different story,
but most people are not in that kind of stage, right, So
we're with attributions first and data and analyzing it. It might seem the most
boring part, but it's the most important one as well, So make sure
to just understand your data before trying to work and performance. Good deal,
All right, Well let's go ahead and do our picks before we do that,
though, where do people find you on the internet if they're thinking,
oh, I want to know more? Absolutely, I am mostly nowadays on
Twitter x X. Everybody knows what you mean when you say Twitter. I'm
okay, yeah, it's just I am not yet most of the times I
forgot. It's been renamed to be honestly, but I think everybody would prefer
to forget there. I am Webb Tita, and I mean if I have
a pretty easy name to look on LinkedIn, but that I spend most of
the time talking about performance on twater. That's where most people confinm web tweeter
with t w I t R. It's a bit u, it's a bit
shortened. Cool. All right, Well, let's do it and do our
picks. Steve, you want to start us off going for the high points
first, Okay, uh, before I get to the dad jokes of the
week. I do have one pick. And this has been around for a
while and I've heard other people talk about it extensively, but it's something I
finally started digging into, and that's the warp terminal. For years, I've
always just used the built in terminal app that comes on Mac. And you
know, I've gotten fairly good at at you know, switching me tabs tabs
and I've got you know, uh tab group set up and all that.
But I had heard a lot about it, so I started playing around the
warp terminal and it's really really slick, makes it very easy to find things.
There's all kinds of enhance since you can create custom workflows, which is
sort of like custom commands where you pass in parameters and you know you can
it's pretty easy to customize your prompt to show like what get branch you're on,
and how what the difference is between you and your remote in terms of
normal commit colors. There's all kinds of customizations you can do. I know
a lot of people use it term two and I've used that in the past,
but I really like the WARP terminal just in what I've played around.
You can find it at warp dot dev on the internets. I'm gonna jump
in here because warp is a recent thing that I've picked up as well,
and yeah, all the things that Steve said. It also has some AI
so you can basically tell it what you wanted to do and kind of like
an AI prompt, and it sometimes works, sometimes it doesn't work quite how
you want. Sometimes it also interprets my get commands as or my commands with
them, like you know, using the rail cli and it'll say, oh,
what you're saying is you want to do this, and I'm like yes,
because that is literally the command to do that. So it's not perfect,
but you know, sometimes it's handy when you're trying to remember the exact
combination of said AWK and GP that you want in order to get the thing.
So anyway, yeah, it also has some team stuff where you can
use it with teams and share workflows and share out pit if. I haven't
use any of that. I haven't had a chance to play any either.
But the downside that some people don't like is that you have to create an
account and they say why you need to do that, and so some people
don't like that. But anyway, it's it's really pretty cool. I really
liked it. Yep, all right, dad, Jokes of the week.
So I've never understood while people wear black when they want to be sneaky,
they should just wear leather armor because it's made of hide. Didn't even get
a smile at it, Dan, all Right, I gotta keep working here,
so it's gonna see somebody laughed. Don't laugh, it only encourages him.
Uh So, I took my kids to see Disney on Ice and it
really sucked. He was just some old dead guy in a box I had.
I actually responded to that, Yes that I said that I actually expected
a dead mouse rather than a dead true. Well, if you notice on
Twitter somebody responded with an actual cartoon of that same joke, and it shows
these people, uh looking, here is Walt Disney in a clear coffin on
I. It's really pretty morbid, but it's very funny too. And then
finally, uh, what famous military general was killed by a cannon? Napoleon
blown apart? Those are my jokes of the week. All right, well,
thank you for elevating my week, Dan, what are your picks?
First? Would like to unmute. I would like to remind us all that
Napoleon actually started as a gunnery officer and one got one of his big promotions
because you know, there was this demonstration and they were trying to mob the
parliament and he basically dispersed the crowd by firing the cannons point blank into the
crowd. Yeah, that got him to become a general. So yeah,
anyway, So about being blown apart by Bonaparte? Uh yeah, anyway,
So I don't have that much in a way of picks this week. So
one thing that I have is that Matt Pocock, who we've had added as
a guest on our show to talk about Typescript, has written a book about
Surprise, Surprise typescript, And since I consider Matt to be one of the
foremost experts on this topic, this is a really good thing, especially given
that to the online version is available for free. So I'm actually going to
share the link here right now, and we'll also obviously put it in the
show notes. And the dog is barking, so hopefully you can hear what
I'm actually saying. But here's that link to that book online. Highly recommended.
Making effective use of typescript is incredibly useful and incredibly powerful, and it's
something that's kind of expected from web developers these days, you know, I
think it can be. It's a first statement to say that almost all of
us have moved off of you know, untyped JavaScript and into typescript. So
that's that would be my first pick. My second pick is the fact that
Google has kind of integrated AI into the browsers sort of, so they're actually
experimenting with the Window dot AI. It's currently I think it's last time I
checked, it was only in the beta version of Chrome and not in the
release version, and you needed to enable a flag to get it to work.
But it's basically what they've done is that they've baked Gemini Micro whatever they
call it, into the browser itself, so that you've got U a large
language model that's a small large language model that's effectively baked into the browser itself
and runs locally so that you don't need to be concerned with paying the bills
for running it in the cloud, which turns out is how NVIDI is making
a ton of money and all the AI startups are burning through cash by paying
all their money to Nvidia. So being able to run models locally is a
very interesting technological development and it will be interesting to see what happens with that
and how various websites are able to leverage this technology once it becomes mainstream and
open to all. Yeah, I mean, considering how well the original gem
and I roll out went, I think this should go very smoothly without any
hitches or anything exactly. And also there's the fact that it's a model that's
not optimized for any particular use case. It's kind of a general model,
so there's also that aspect to factor in. I don't know what you want
to do with AI, but you know, it does open up some interesting
opportunities for user interactions that's that's put it this way in websites, Yeah,
because it has because it's a Gemini micro or whatever, it has much smaller
context on what it was trained on. But it's still very good to as
a as a transformer, as a as an LLM. If you are curious
about that stuff, I definitely suggest following Jason Mays, the webi lead from
Google, that he has a lot of great demos on what you can do
with that stuff, because not only the web ai APIs the upcoming, but
you can also load models with web assembly and web GP you they can use
your own models kind of locally. So that's pretty that's it does a lot
of good stuff can Yeah. The only downside there is that downloading running the
model locally, like I said, solves a lot of the coast issues.
But it's also these models can be even the smaller models can be fairly large,
so it can downloading. So downloading them locally that can that can be
that can you know, we were talking about performance, that could be a
performance issue. I was going to say that could hurt your performance. I
can hurt your performance, But then again, it depends on the nature of
a product, right because if your look, for instance, working with something
like Photoshop on the web, they do kind of they kind of do that
and it's not about the loading times but execution times in that case. Yeah,
which then you're actually gains significant boost, right because it's local, so
the latency is non existence. Well, and then I guess it's my picks.
Yeah you can go, then I'll go yes, So go ahead,
Okay, So for my picks, I have two things. Since we're on
the topic of AI and mL, I've being started to dive into mL training
and machine learning, building models and this kind of stuff, and I actually
chose to do it with Alexia as a language. It's a brilliant language as
just as fellow Brazilian I'm also from Brazil. It's a really nice functional language
to get started with. And one of the things is that, especially come
from JavaScript, there is not as much tooling fatigue because when you within Alexi,
things are kind of like very coherent and there are you know, all
the options are kind of built by the same people, so everything kind of
speaks not and no point intend the same language, so things kind of flow
very easily. So it's not like, forri instance, you've tried to learn
Python and then choosing an mL library to do things on top of and compatibility
issues and all this kind of stuff. So that's my technical pick, and
non technical I would I'm all finishing the last season of freet and if you
haven't watched that series, it's really good to watch. With the cases though,
if the kids is old enough, and it's a very nice, like
halted series to watch kind of tries to be like hearted in the bocalytical setting,
which is an interesting thing. Awesome. Well, I'm going to throw
in a couple of picks on my own. First of all, I just
want to point out, if you are interested in Alixir, we have an
Elixir podcast. It's called a Elixer Mix. I am not on that one,
but those guys are awesome and they cover things very very well. I've
also talked to Jose on multiple occasions from when he was in the Ruby community
and then when he started doing Elixir stuff, so it's it's a very very
cool language. And if you're whenever I hear about the Elixir, it's usually
in the context of how great it is, and to my shame, I've
yet to learn it. Maybe I should start listening to that podcast. Then,
yeah. Yeah, it's a very very not so hard language to get
into. Even though it's fully functional, it's a very very easy function to
the programming language. Still getting too very approachable. I'm gonna start out with
game pick. I always pick a board game or a card game. When
pick a card game, it's called six NIM that's n I m M.
Now. When I looked it up on board game Geek, it says that
it is the same game as Take five. It has a bunch of other
names uhrendi, which looks like is Italian category five, Take six blah blah
blah blah. Anyway, board game Geek rates it or has a weight of
one point one nine says eight and older can play it, which is probably
pretty accurate. Really, just quickly, what it is is everybody plays a
card face down. So you put it. You put out four cards.
There are four piles. Then you put out your card face down, and
then you go from lowest to highest and you put your card. You put
the card onto whatever pile it would go on, right, So it's it's
whatever number is below it and nearest to it, right. So you know,
if you put if there's a thirty four out there, and you put
out a thirty five, it's going to go on the thirty four. You
put out a forty, the forty will go on it. As long as
there's nothing between thirty five and forty out there, it'll go on the other
card, right, And it's always the highest card in the pile that you're
playing on. When you play the sixth card on a pile, you get
all five cards in the pile, and that sixth card that you play becomes
the start of the next pile. The different cards are worth different points,
And the only other I guess rule is is if you play a number that
is lower than any of the piles out there, then you just take whichever
pile you want and replace it with your card. So it's like you played
the sixth card. But in a lot of times there's a one or two
card pile that only is worth one or two points, and so you take
that pile because you're trying to get the lowest number of points. You play
till somebody gets to sixty six. Game's over there. You know how to
play it. It was a lot of fun. Numbers are numbered one to
one oh four, I think, and so yeah, you just deal out
ten cards to every player and you put the rest off to the side,
so some of the some of the numbers aren't gonna be out there, but
anyway, it was a lot of fun. I think we played it in
what ten to twenty minutes. I mean, it's real, real quick play.
But if you look up for a fast fun game six nim or take
five or whatever it's called where you live, that was super fun game.
I'm also going to pile on the AI stuff, so I've started getting into
writing AI code. I've been playing with it in both Ruby and JavaScript.
I have to say that a lot of the tools for AI are actually nicer
and Ruby. That might surprise some people, but anyway, what I'm looking
to do, and so keep an eye out for this, I bought the
domains AI for Ruby fo R Ruby and AI for JavaScript, and so if
you're uh and I'm gonna I'm put together a newsletter for both of those topics.
I'm also going to be putting on a summit, probably the week after
Labor Day here in the US, which is the first Monday of September,
so it'll probably be Friday and Saturday. I'll do the Ruby one first and
then the JavaScript right after that, and then two weeks after that, So
toward the end of September beginning of October, I'm going to do a three
month boot camp and I'm going to be teaching people how to do JAVA or
how to do AI. But the difference between my boot camp and and it'll
be a three month boot camp, so we're going to get into prompt engineering
and uh, you know, building chatbots and right were we're going to be
using the APIs and you know, we might do some model training, but
it's going to be fairly lightweight, you know, so you don't have to
know the math, you don't have to you know, have a deep understanding
of the models. This is how do you add AI to your app,
you know, your web app basically, but it could be a desktop app
or something else if you're writing it Ruby or John script. So and if
you're if you're you know, primary language is a different language. Maybe it's
Elixir. I'm I'm almost certain they have libraries that attached to the same stuff.
And so you can probably figure it out. And I may or may
not be able to point you in the right direction. But but this is
what we're gonna do. And so yeah, so uh AI for JavaScript and
AI for Ruby dot com. If you go to those websites, you'll be
able to sign up for the newsletter for free. You'll be able to sign
up for the summit for free. If you want the videos after the summit,
that that's what you're paying for on those and then the boot camp will
will cost money as well. But I want to talk to people make sure
that they're in a position actually take advantage of the boot camp because it's not
going to be a cheap thing. So anyway, that's what I'm working on
these days, and really I'm really digging it. It's it's fun, fun,
fun stuff. Another pick that I have this is a Ruby Rogues episode.
I'll put a link to it in the show note. But we recently
talked to Obi Fernandez and he has a book about building AI stuff. It's
language agnostic. I don't think we've released the Ruby Rogues episode yet. We
might have, but oh yeah, it is up here. I'll put the
here it is, I'll get the link in here. But anyway, I'm
really really loving the AI stuff. I think it's amazing. I love the
only thing that I guess JavaScript has that Ruby doesn't is TensorFlow jas right,
there's no TensorFlow Ruby, so you have to interface through JavaScript or Python.
But beyond that, yeah, I'm I'm really enjoying just what you can put
together with it, and I think a lot of the capabilities that come out
of it go well beyond just the prompt for something like chat GPT. Obie
actually explains it very well. What he's done is he's built essentially virtual AI
assistance that they take the prompt, but then they also have API capabilities into
systems that you use, like your email and stuff, and so you can
actually write a prompt where it'll go find an email for you, or respond
to certain kinds of emails for you, or things like that. And so
you know, you start getting into Okay, I'm not going to just prompt
you to write something out or give me an answer. I'm going to prompt
you to actually go do something useful. So anyway, I have pontificated on
that longer than I needed to, and I am very much enjoying what I've
got. There one other pick that I have, and this is a technology
pick. It's something that is put out by base Camp or thirty seven Signals.
It's called Turbinative, and it's essentially a way of raping your web applications
into a native app that you can deploy to the app stores on Android and
iOS. And I have been very very happy with what I've been able to
do with that without having to write a full on native app. And I
can see the pros and cons, right, I mean, if you're not
on the web, it won't work right, and you know, maybe you
can use some local storage or things like that to make it do what you
want and maybe load anyway, but the majority of it just requires you to
operate, you know, connected to the Internet. But most people are operating
on their phones connected to the internet anyway, and so you know, anyway,
for what that's worth, I'm really really enjoying that looking at ways to
get my web apps onto other systems though beyond phones, like the Firestick TV
and things like that. But Firestick TV has a way of wrapping web apps
anyway, and so anyway, that's just another area that I'm I'm diving into
and I might put together a boot camper a course on that as well.
But anyway, so turber Native is my last pick for Hey, I and
Alex say you have an X and bumblebee. So if you were trying to
get like familiar with that subject training models and running models and ex. Simbo
will because you covered cool, I just well, there we go. It
made the window go away and I couldn't click on it. All right,
good deal. Well thanks for coming. Uh I'm always hesitating the same name,
Niches. Thank you for coming. This is This has been really cool
and I love kind of just getting into Hey, you know these are the
steps and kind of levels to performance and how to read the data and then
how to make your case with the data. Yeah, my pleasure. Always
happy to talk about it. All right, Well we'll go ahead and wrap
it up here. Until next time, folks, max out