How'd you like to listen to dot NetRocks with no ads? Easy? Become
a patron for just five dollars a month. You get access to a private
RSS feed where all the shows have no ads. Twenty dollars a month will
get you that and a special dot net Rocks patron mug. Sign up now
at Patreon dot dot NetRocks dot com. Hi, it's all right, Holy
crap, this is a great room. It's a great lot of echo.
Yes, certainly filled this space. He determined level of absolutely the blast radius.
Right, we are back in Portugal. I love I love Porto,
I love Portugal. I was in the middle of town and I came across
a restaurant and the sign said churiscaria, which means grill. Right they infante.
I didn't know you guys ate babies over here. So no, no,
no, not funny, too long, too long, too soon?
Infante. I guess Infante is the name of the area. Yeah, but
it looked like we eat our babies. Well done, nice on a skewer,
I think it was. W. C. Field said it was all
about the sauce. Yes, I love children exactly. You got it.
Oh, we're gonna have some fun. Today. We are going to have
some fun. But first we have this little thing called better no Framework.
Roll that crazy music. Well I've been waiting for this one because it happened
a while ago. But if you don't know, I have a consultancy called
at the next and we are the shepherds of an open source project called Polly.
Anybody who's Polly? How about uh? How about a clap of hands?
Use poly lot of Well, Reverend Billy just walked in the room.
Well, Polly just did a make just came out with a major update,
version eight. And I, believe it or not, it started with the
dot net team because the dot Net team was basically looking at the source code
and said, hey, we think we can improve the performance and the resource
usage of Polly, but it's going to require some new interfaces and you know,
almost a complete rewrite. And so the rest of us said, yeah,
I'm sorry. The dot Net team call you, yeah, say they
want to make your project. Yeah, I mean what are you saying?
Nah? So it took a lot of meetings and a lot of understanding,
and basically what they were able to do is without you having to change any
of your code that uses Polly, you will get the benefits of the new
performance and resource allocation that's under the hood. But if you want to use
the new models and the paradigms and the interfaces, you you can do that.
So Greenfield, you can go forward with a new style. But just
if you use Polly in place, I'm not sure if it's completely compatible or
you have to change some class to some other class. But it's pretty much
a simple a simple fix. And I have been emailing with Joel hewle In
to schedule Polly show for Yes, we definitely will get cool. So that's
what that's what I got. Awesome. Who's talking to us? Richard Grabby
comment off of show eighteen sixty one we did with Jeremy Miller back in the
summer of twenty three, talk about minimal architecture because Jeremy likes to cause trouble.
Goodness nose And this comment comes from Trevor who says, I love this
discussion, enjoyed the comments on microservices Worth's monoliths, which is actually a reference
to an earlier show we didn't reporter. We've been following this trend of people
sort of pushing back on microservice. Yes, I got pushed heavily into microservices
approaches with a product that we'd built and re architected into microservices, and it
was the worst mistake ever. Things just became more complex, it was harder
to maintain, it added a bunch of latency and security issues in the complexity
was just not worth it. And so I came up with a new acronym
for appropriately sized service or as nice I can relate. This is a good
one. I one hundred percent believe in services, separation of concerns, and
clean architectures, but the approach must be appropriate to the complexity, solution and
the size of the team. It makes no sense to have one hundred separate
services for a team of ten people. But then it also makes no sense
to have a massive single deployment with a code base in the team of a
two hundred and fifty people. The services need to work with the cognitive load
and be appropriate to the organization and team structures. And I was loving the
discussion on all of this except for that one point. Stop making the CTO
out to be the bad guy. Love from Trevor CTO. That's great,
Yep, that's fair. I definitely think we need a whole show on ass
I think so it seems up. Yah, So, Trevor, thank you
so much for your comment, and a copy of music by its own its
way to you. And if you'd like a copy of music co buy,
I write a comment on the website at dot at Rocks dot com or on
the Facebook, so you published every show there, and if you comment there
and everyday in the show, it's like your copy of music Cobe. And
you can also follow us on Twitter if you want to. But the real
cool kids are over. I'm massedon, I'm at Carl Frank tech dot Social
and Ambridge Campbell at maps. Send us a two we'll get around reading it
all, publish it and with that let us introduce Charity Majors to the show.
Charity is an OPS engineer and CTO at Honeycomb dot Io. Before that,
she worked at Parse, Facebook and Linden Lab on operations and developer tools
and always seemed to wind up running the databases. That's because it's where all
the problems were. Yeah, we stand next to the database. You're going
to be running it. Also co author of O'Reilly's Database Reliability Engineering and the
newly released Observability Engineering. Charity Loves free speech, free software, and single
malt Scotch blamer. Do you ever have round the bloud charity Majors? Okay,
I guess we would have start at the beginning, right at the beginning.
What the heck is observability engineering? That's a great question. It's like
the engineering of Windows, yeah, kind of. I mean observability comes from
control theory, right, and it's like, how well can you understand what's
going on inside your systems just by observing outputs? And yeah, exactly.
And you know, I for years was like really religious about trying to define
it in a very specific way, and I should have won, but I
lost. So so I mean it's come to it's come to just kind of
be a generic sitting, which is what it is. But when we were
trying to figure out how to talk about, what I think of is just
kind of the next generation of telemetry. It's kind of distinguished from the last
generation of peletry, obviously, which was very much focused around the metric,
right, which is just a number. It's tags depended, doesn't handle high
cardinality, doesn't handle dimensionality, doesn't handle it's super fast. Is that powerful?
Now you drop some OLAP terms into their cardinality flexibility, Like it's funny
for a database person to drop all lap, but you're talking about just any
way that you can really observe the state, the internal state, not necessarily
what it's doing on the outside. It's about observing the internal state and being
able to explore it right, not having to decide in advance, here's the
data I'm going to collect, because here's the questions I'm going to need to
answer. Here's my dashboard. You know, it's about being able to go
to combine your questions to ask because, like anything that you're trying to understand
these days is going to be a very complicated answer to cart. It's like,
okay, these errors are spiking, but only for users that are running
this version of Android, who's a particular firmware in this region, with this
language pack. With each of those are the high cardinality to mention it.
And if you don't capture the data in a way that preserves all that context,
you can't ask me questions. Do you have some examples of how observe
observability has improved a project in particular? Sure, I mean I think of
it as it's really it's kind of where development meets operations, right, Like,
I feel like big picture. You know, in the beginning, there
were engineers who wrote code and they owned it in production, right, right,
And then everything got super complicated and we're like, ah, there's too
much. So some of us are going to write code and some of us
are going to understand it. And that was not about it. That was
not a great idea, and so like we're kind of like reunifying the streams
now. I think every engineer should be writing their code and owning it in
production. Everyone who's especialist operations should be also like opening the door and looking
under the hood and understanding the code. Right. There's specialization is great,
but ultimately, you know, our systems have gotten so complex that you have
to write it and understand. I feel like you got to dig into that
own it in production because it's not like they're also going to be sisumits exactly
as they are responsible for the You're responsible for your systems, right, you
wrote it, you own it, You unleashed this support upon the world.
I mean, I feel like there are these feedback loops in the heart of
engineering. Some of them are like code review, right. Some of them
are like deploys. But like, if you don't hook up the feedback loop,
if you aren't being exposed to the consequence of what you're doing, then
like you're not you don't actually know if your code is good or not.
Well, I think there's a great point there as a developer then that if
my telemetry just tells me how many times my code was hit, that doesn't
necessarily give me anything to do. And this is this is where I feel
like operations folks have had a harder time embracing serviability in some ways than software
engineers have because with up people, it's like we learned how to debug,
but it looked like this, I've got a dashboard. Something's wrong, So
I'm gonna start paging through dashboards and looking for similar spikes, just like pattern
maatting with my eyeballs, Like right, oh, it looks like it's redus,
you know, and you get it. It's great because you're like you
get this hero journey where you just jump to the and you understand what's going
on because you're in this shit all day every day. Nobody else does,
like whoa, how did you do? That? Was reset? The redit
service? Problems went away? Right? But like that's not debugging. Countermatching
with your eyeballs is not debugging. Debugging looks like you take the step,
you ask a question, you look at the answer. Based on the answer,
you take another step. It's like following a trail of bread crupts.
You don't know what the answer looks like until you get there. Can we
talk about some of the new modern observability tools that we might think about using
to replace the tools that we're currently using. Yeah, I mean, I
think big picture, it has to be based It can't just be based on
the metric because remember you've discarded all that you're looking output exactly. It has
to be based on ar truly wide structured TETA blocks, which now look like
scams, right, Those are just like wide events structured which you can trace
because there's been a number that's appended to it. That's what you need in
order to understand your telemetry and production. Because I can imagine at a peak
load, like we think about a metric that shows, you know, this
is when we're posting the most number of transactions. You're now really interested in
the state of yes, we're we queuing out yes, what's happening? Like
metrics are great, but they're they're limited, right, they're a snapshot.
What you want to be is like, you know, okay, when this
happened, what else happened? Right? What else is connected to it?
You know? And like the old generation of tool are ones where you find
you're capturing this data another time. For every single tool you're like, okay,
here's my dashboards and the metrics, here's my logs, here's my traces.
So every time you're like I've got a spike, I want to find
the logs, there's nothing that connects them. You're just eyeballing timestamps and hoping
that they happen to match up. And like, if you're finding the logs,
we want to jump to a trace. Like that's not actually good enough.
You can derive all of those data formats from these arbitrary white from spans.
You can't go in the other direction. When you say spans, what
exactly you're talking about? A span is a one hop of the trace,
okay, across all of them. So should we be gathering spans? All
of the gathering telemetry one event per request, per service, all of the
data should be aggregated into that one arm chreary wide production, so you have
all that context, Like a really mature instrumented service will have like two hundred
three hundred dimensions per per hop and that's that's magic because you're passing along all
of the parameters, you're passing along all of the I E. S,
you're passing along all of that context, which lets you after the fact come
back and say, oh, this thing and this service that happened was connected
to that thing and that service that that happened. Now, this is not
necessarily a per transaction level, like you're not just chasing a transaction. What's
this It basically it's one span from a time it's well typically, well this
complicated. There are lots of ways that you can define as span, but
typically I like to think it about if you want to have a span around
something that's interesting. So like if it's anytime that you're crossing the network,
you want to span. Anytime you're taking a database request, you want to
span because that's historically where problems happen. It's wherever you're crossing the right.
So when you start it at a user interface interaction and go from there and
then you know, likewise we're back end services that go on time like you
can have you can have spans and tracing in models too, and it's super
super useful there as well. But it becomes indispensable once you have service.
Yeah, because if you think about it as a monolith, at least you
have all that context and it persists throughout the request. When you jump across
the network from service to service, you're deciding what state's going to come with.
And so how do you do all this without bringing the server CPU to
its knees? Do you do this? Typically? The way we do it
now is it attached to back around threads and that kind of stuff, and
you can lose you know, if those threats hang, you can lose data.
There are lots of ways to do it. Obviously, I think that
my service does it best. Your Honeycomb is, well, you've got to
tell us about Honeycomb. Well, sure, you know I'm not I'm not
really great at pitching, but I will say that, like, you know,
the idea of how observability should happen is how we built our service,
you know, down to like the data store, like because like, well
back how many have you any if you ever built apps on pars? The
mobile back end is the service. No, wow, I loved so much.
It was it was like Firebase but better and earlier. Any Facebook people
here, Okay, cool, I will have I will have a brudg against
Spark Soccer for forever for what he did to Pars. Okay, they shut
it down. It's like we got acquired. Anytime you want to get acquire
or make sure that you have an executive level sponsor who believes you were fond.
Right. Not so we got shut down even though we were still growing
like aang busters anyway. Cars had had one hundred million apps by the time
I left, and we had built our service originally on Ruby on Rails,
which was not a terrible decision because most startups fail. It's usually not because
it's and Rails had the strength of you can move fast, you can move
It's just a we believed everything. Yeah, ok, say whatever you want.
Doesn't that Ruby on rails. The downside is it's got it doesn't have
threads right, fixed pool of workers, right. And so that was fine.
We had one hundred thousand apps, but we got bigger and bigger,
and instead of having one database and back end, we now had thirty forty
fifty. And when you've got that many something slow at any given time,
which means that the sixth pool is going it's filling up constantly with threads that
are waiting on that one back end service got oh and like, as a
reliability engineer, this was personally humiliating. You're going down every day just like
a hit the top ten and iTunes down, goes parts again and again.
And I tried everything to try and figure this out. And what finally helped
us was number one, we did a rewrite to go length. We actually
considered uh using dot net and it got out voted. And I learned later
that the blog post that I wrote about why it got outvoted, they had
a lot of people with Microsoft very angry and changed a lot of their decisions,
which is great. Yeah, inspiring anger is really like, but I
mean, I can I made a career. I have a tough time of
disagree with you on picking go line too. When you think about a back
end service at velocity like that, language is very well suited for that.
It was great, but it was half of the half of the answer because
we also had to understand what was going on just right to code understand it
observab This is where observability came into play because Facebook also had this service called
Scuba, which was, don't get me wrong, but ugly aggressively hostile to
users, but it did one thing really well, which is people that you
slice and dice in near real time on dimensions of high cardinality with wide events.
Right high cardinality for those who don't know, it's the number of unique
idemans in the set. So if you've got a collection of one hundred million
users, any unique idea like social security number for the US folks would be
the highest possible cardinality. Something like species equals human would be the lowest because
only one right first thing lasting high cardinality, but there's some dupes, so
it's not as high as the security number. So everything around metrics is oriented
around low cardinality dimensions, but everything you want to use for debugging requires high
cardinaliti ice some people over you run into the influence before. So Scuba,
let us slice it some of these high cardinality mentions and instead of having to
like, you know, obscur and be like either I either I read it
to dashboard for it or it's going to be hours to like just dive through
the logs and figure it out and everything. It's like, instead it would
be like, okay, we're getting a spike and eras, let's break down
by app one in ten million appies. Break down by that. Okay,
now break down by her rights. Don't break down by by normalized database query.
Now are you making cube gestures or gestures? Call them after Colum.
It was just like, step by step it would take me to It's like
it isn't even engineering anymore. It's like support. Right, These problems went
from being like intractable, like it would be I'm doing mean, from like
it would take us a day to figure out and then it would never happen
again, to just being like, you know, thirty seconds, like every
single time. And that was what like when I was leading Facebook. You
know, I've never been one of those kids who's like I don't want to
start a company because I kind of hate those people. But when I was
thinking about having to live without this tooling, I was like, I can't.
I actually can't conceive of it, Like it's becomes so coore to how
I how I perceive the world as an engineer, Like I just can't imagine
going back. And so that's why I made talking on So when how did
the rest of the people in the organization react to this new culture of observed
ability and spam? There's a learning curve, right, We've all spent our
careers fitting our brain into asking questions in the metrics and dactions type of way.
But like you know how every job I've ever had, the person who's
best to be bugging is always a person who's been in the moms. Always.
That's no longer true when we have different tools because instead of relying so
much on what's in your head to reason about system, it's right in front
of you and you're just asking questions and it's and it's more like the more
curious you are, the more debugging you do, the better you get.
You don't have to you don't have to have the whole system in your head.
You can find the answer more quickly that way, and it's kind of
a beautiful thing. Yeah, process of discovery too, right, find exceptions.
That's why, like observability is not just about yes, you're gonna have
to have a Columner store. Yes, you're gonna have to have all these
things in the back end and make it fast. Because the other thing about
logging tools is like if you want to ask something interesting, it's like you
enter the tool, you know, the and then you're like, Okay,
I'm gonna take thirty minutes and go out for coffee because it's gonna like it
has to be fast, it has to be interactive. It has to be
like under a seconds because you're like, you're taking steps and you have to
stay in the zone, right you're yeah, exactly. It has to be
explorable, it has to be interactive, and it has to let you,
I think most importantly, draw on the on the brains of the people around
you. So something we built into Honeycomb is is history. You know,
how you're debugging and it's like, oh, I've lost the thread, so
you can just go that, you scroll back up that's where I knew I
had it right, and you branch out and you try something else. But
then also you have access to the history of everyone on your team. So
if it's like last Thanksgiving, we had this terrible my squel outage, you
know, and everything was uh and Ben and Emily were on call, say,
and then I'm on call in March and I'm like, this feels a
lot like what was happening last last November. I'm gonna go and look at
like, well, what were Ben and Emily doing and what did they say?
Help them find out what trace are So journaling, yeah, actions systems.
History doesn't repeat that. It rhinds right so much. So much of
the wisdom of your like these are socio technical systems. It's not just production.
Like an example I often is this, You've got the New York Times
on the Washington Post. They're like both big newspapers, right, but if
you took their teams and swapped them, you couldn't actually do that because so
much of this this different system lives in the heads of the people us write
it. So like being able to draw on that wisdom and use it,
like it makes you a better engineer. Like all of the ship that I
learned about being an engineer was looking up over their shoulder of amazing engineers that
I got. But it sounds like the journaling approach you're talking about allows us
to look over the best exactly and and and get to know you don't have
to remember how they did it because it's recorded there, and so you know
you can even learn their approach and how they attacked that I feel like,
you know, especially now nowadays, when we're doing so much distributed working,
like remote working, I worry a lot about how are we going to bring
up the next generation of engineers, you know, and I feel I hope
that we're all starting to think about making this more of our tooling. Just
how can we learn from you know, it's kind of embarrassing, but like
when I was in college, I learned so much from just going around and
reading the Bash histories of all the people I knew, from either trying the
commands. You know, it's fucking fascinating. Right, Oh, that's how
you have to learn, said not right, what does this do? I
think we need a lot more of that in our tools. Yeah, and
I worry that we're making it even harder to make that jump from junior to
an intermediate. I mean, we've always had a problem with intermediates anyway,
but a lot of the automation tools that are taking a lot of are eliminating
the beginner stuff. Yeah, like this whole like the generative AI stuff,
Like it's great for senior engineers. You're now so much more productive, you
can put so much faster. But like the way that you get to that
point is it's scarts. How how are we going to force ourselves? I
mean, I believe that the solutions will emerge. I hope it looks pretty
bad. And I also think that younger generation will find them too, because
they are not you know, we've done this show where we're talking about is
all this scar tissue actually holding us back? Right? That we have some
of it? Yeah, some of it's value, I think. You know,
you have to internalize the damage and say, like, what does this
really look like? In generally speaking, And when someone says I will never
use X product or X technique, it's like you have not eternalized your scar
as well. I love that every team has, I think, but also
that can they learn to speak to These are the concerns I have when you
think about the broader approaches to things that might have created that problem back in
the past. I read this book called The Trauma of Everyday Life, which
is written by this guy who's as psychiatrist and a Zen Buddhist, and he's
talking about how trauma isn't necessarily something to be avoided because it's literally what shakes
you think about a bond sire that's just a normal tree, but it was
it was put in this very specific where its roots couldn't grow right, and
so it's not it's like not RECTI people like trauma is great, but it's
also like there's scar. She was just going to be different. And again,
how you react to and how you work to it, you can make
beautiful things. So what are some of the other pitfalls that people will encounter
when sort of moving to this observability. The big one is the cognitive just
the model that we have in our brain. I feel like our industry has
avoided this for a long time. I feel like there's a bit of a
reckoning. You know, Open telemetry. By the way, I've got to
put in a quick plug for open here. It's amazing. I know all
of us hate redoing our code, but like the promise of open telemetry is
you reinstrument your code once and then vendors have to compete for your dollars based
on being awesome instead of having you locked in. It is. It's it's
the number two project after Kubernetes in the what you call it thank you CF
It's super active. A lot of contributors. I was pretty skeptical about this,
but it's it's it's the way I wish we had had this ten years
ago. I think we'd all parties. You know, we could have we
could have just chose to political problem, not detectives totally. We're there now,
but here we are. These are the tools that we have. Open
telemetry is worth putting on your roadmap for the next year or two because there's
also this this reckoning that's happening with costs right now. Most of these vendors
are billing just like ungodly amounts of dollars that do not correspond to the value
that you get out of them because they can't because they got you're locked.
Yeah, yeah, And so I feel like we need to take her powers
well. And it's a great pitch for a feature that's not necessarily a new
features to say, hey, I can reduce our costs by moving us off
this tool and onto open telemetry. You know, I'm compliance some of these
sidebar rants here, but like I feel like learning to treat like one artifact
of the zero interest rate like period, was it engineers forgot how to talk
about our work in terms of dollars, you know, because like dollars are
the universal denominator. Maybe something the Euros. I don't know, but like
money is the universal denominator, and if we can't learn to talk about the
value of the shit that we provide to people in finance people, I feel
like many many vps of engineering and CTOs have this phenomena where they feel like
the junior partner at the table. They aren't really invited to all the critical
meetings and stuff. And I believe that that's because we haven't learned to talk
about the value that we bring and cost in the same language as every other
team. Because if we did, we generate a lot of value. We
generate all company, We have all the power we should need to have.
We got to do is get a hand on it. And Charity when they
were up for one moment for this very important message and we're back. It's
not at Rocks. I'm Richard Campbell. Let's Carl Franklin. Hey to our
friend Charity Majors about observability engineering and watching the sausage being made. And so
I want to follow up on you. You brought up generative to a and
things for programmers. It's great for senior programmers who can be more productive with
stuff they might have forgotten how to write or don't really care to figure out
and just let chat GPT do it for you. But what do you think
the future of observability is, especially in lieu of AI and where it's going.
Do you think that we'll have AI bots sort of watching our telemetry and
giving us English prompts, you know, sending us text messages. Vendors are
going to sell CTOs and pps like tens of billions of dollars worth of bullshit
that says that they can do that. Yeah, something that blew my mind
when I became to see So are you saying we don't need this? We
have everything that we need right there in front of something that blew my mind
when I keep CTO to be wild internalized. But the most executives have more
trust and confidence in their vendor relationships than their employees, because employees coming out
the vendors left forever as long as you keep paying them. In my mind,
but what they're selling when they come in. This is why my dander
got raised so much by the whole AIO saying, because they are all just
like you don't need to understand your systems. Pay us all this money,
we'll understand it for you, and but like the false positives are ridiculous and
off the charts, all of the data is junk. You would be better
off just like turning off all that data, Like it's just so many problems
with it. I believe that we should be looking at computers that do what
computers do best, and people to do it people do best, and computers
crunch numbers, people attack meaning to things. Sure, like your graphs are
spiking all day long, most of them you don't care about, because our
computers are now resilient to a whole lot of failure. Sure, it really
takes a person coming along and going matters. That matters often because it mattered
to another person, and you're the person who they're connecting those dots. And
once you've decided it matters, you need to understand why. And I think
there are all kinds of ways for computers to help us do that. We
do this really cool thing called bubble up the honeycomb, where any graph that
any heat map that you've constructed, you draw a little bubble around something you're
like, I care about this, and then we compute the baseline for all
the hundreds of dimensions and the dimensions that are inside the thing you care about,
and then we dip them and sort them, so it's like, Okay,
this thing you care about, here are the five to ten ways that
is different from everything that you don't care about. Computers are great at that,
but they can't tell you what to care about, and they shouldn't try
it, because it's a fucking mess. Maybe ten years from now I will
be eating my words, but for the foreseeable future, I really think that
we're all best served if we focus on helping people understand what has meaning and
letting computers take care of their rest. Yeah. I mean I can see
the machine tools helping to point us too unusually. Sure, Yeah, but
you still have to interpret them. Yeah. Still you want them to create
that graph for you. You want them to intelligently sample often, you want
them to do you know, but you don't don't want them in the business
of telling you what that No, they don't know, you don't know,
and more and more stately, like they're not even qualified to make that s
that's been in any way. That being said, like I can tell you
we're talking a lot of old lap terms here, like a lot of data
analytic terms around all this and machine learning models evolved from a lot of that
technology. So you can see a shape of this shape of history. You
can't see a shape, but I don't believe that it is. It is
one that So here's the thing. At the bottom line, we are forget
technology. We are held legally accountable for your engineers. We are legally and
ethically and morally accountable for the codes we put out into the world. Right,
we can't point an algorithm it comes to that, even if it's a
machine learning I think, I don't know if you've read it yula lately,
A boy, oh boy, they work really hard to make sure we're not
legally accountable for any I believe in the near infinite possibility employers. That's that's
true. I want to make sure that I understand. I also, I
mean, I like that we're also going to moral and ethical aspect because I
think we need. I think that legal aspects holding us back, that we
can't own the value of what we makeout, that we can't own the value
of a make as long as we're obligating our responsibilities. And then really,
you know, the yula was invented to allow us to not hold liability for
the impact of our software, and so we're kind of in a trap right
as an industry. If we were responsible for the damage we did, we
would we would our employers would insist on higher standards because they're getting caught up
in that as well. But because we've avoided the responsibility so thoroughly, I
see what you're saying. That being said like this is now we get into
a pretty deep philosophical side of this thing, like let's face it, good
telemeter. In the end, we're trying to understand why is the software behavior
and its behavior? Why are our customers unhappy? I mean, those are
the things that actually matter. I think. The more often that we as
technologists speak in the term of the customers, I think, why are our
customers onhappy? You know? And this is something I've been really grappling with
lately. I don't know if I'm alone in this or not, but I
have like an almost knee jerk, almost disgusted or like reaction towards like customer
and value and things. And I've been trying to because we've been battered with
it, because we've been battered beaten up those words. Yeah, I don't
know, just the business aspect, like I think there's some vestors of me.
There's still like ew, we're better than that. And I hate myself
as I'm saying that. You know, and you're also open with the dollars
matter they do, they're kind to come from the customers. Oh, you
should have seen me ten years ago, because this is a chill version.
I get that. Okay, No, but you're you're absolutely right. We
do this for the customer. We do this for our users. That's the
reason we exist, and we have a responsibility to them. Sure, And
I don't think I'll ever still comfortable saying, well, the machine told me
it was fine. That's a cop out every time. Because the machine didn't
tell you anything, you interpreted it and chose to vocal. You know,
in the end, everything we've talked about program it's fine, getting very philosophical,
but also none of this is described an action we should take. All
we're doing is observe what's going on. We still have to decide on the
action. How would you change the code? Given you've seen this in dilematry
and you know what else? Like I feel like this looks back really nicely
into just like what is what is the meaningful life? Right? Like because
like that book that what's his face wrote about about work and what makes us
happy is it's not like having twenty hours a day whatever, but it's like
autonomy, mastery and meaning purpose. Yeah, Daniel pink, Dangel pink,
thank you, and like the meaning the purpose that comes into play for us
when it impacts other people. Well, and you hit on the key thing,
which is when we crack this, not like every time you chase a
problem downline that and it turns into a code change, you can make that
then in later testing shows that problems occur. Boy, that's a good day.
Like you talk about purpose, there is nothing better than figuring that complicated
problem out and then literally, like you, you live in a very hypothesis
based world. It's like, well, I've seen this telemetry, I've seen
this output. I believe it's this code problem. Now I'm going to make
a modification. I'm going to put it into the stream and I'm going to
go back and test again. And if I don't see it, then I
can, you know, hypothesize really because I might be wrong. We may
not have reconded recreated conditions perfectly. That we're on it, that we're pushing
the right thing, and nobody knows just how deep that went. No,
I also wonder, you know how many times have you been fighting a problem
like that and you chart changing code? Just see if you can change behavior
at all? Like, am I even assistant? They have emergent properties,
They're no longer like I feel like part of moving from like the old version
of the new is except that TDD is not enough interesting like the tests tell
you will this logically execute, but that reality ends at the border of your
laptop. Yes, and the universe is weirder. The weird intera is so
much weirder than that. I feel like our jobs are not done. It's
like until we've instrumented that code, deployed it and watched it in production and
asked ourselves, is it doing what I expected to do? And if anything
else look weird? I know that on the show before I was you know,
I've did a lot of load testing. It's like I have never invented
the load tests as weird as customers on Saturday actually comes even come close.
So customers are evil do things you can't. They really opened six windows and
hit refresh all at the same time. Did he really really? Okay?
May I see some practical advice on behalf of the listeners. So let's say
you're listening, you're you've been surfing, you went to uncombed dot I and
you checked it out, and you're thinking this might be good. How do
you go back to here? How do these people in the audience go back
to their teams and introduce this concept without getting flogged? You know? How
how do you approach that? I mean, that's a great question. My
approach is always to look for something that's really painful, like, you know,
things that are going down, you don't understand, problem, problem,
you can't crack and especially this the siloed approach to telemetry that we're doing anyhow,
things that are waking people up in the middle. Then I you know,
we've seen this a lot where you know, people have tried to bring
it in whatever, but then there's an intractable problem and they put money come
on it, and it's just like like we've even had multiple times we've had
our sales engineers doing demos on people's production systems and you're about to have an
outage here because this thing's happened, and they're like what in the like ten
minutes later they get paiged because it is that Like I know, I'm a
founder, believe nothing I say, But is that much easier when you have
the right tool, when you have the right visibility, just to be able
to see what's going on? Yeah, looking at something like that, or
somewhat counterintuitively the other side, another place we've seen a lot of success is
people insumenting their CiCe pipelines, right, because if you insument your CCE pipeline
as a trace, you've can see where all that time is going. Yeah.
Yeah, that's kind of another approach to this, the model of what
is the hard work here? What's actually hurting us? The struggle is only
getting in the front door. We have like zero turn if the company didn't
go out of business to keep buying us. But it's difficult to get in
the front door. But once we get inside, like no, but I
think you've made the most compelling argument, and that's going to be tough for
anyone in the room wo's thinking about this. It's like you have to go
pick the largest dragon in the room and say, I think I could take
that one on if I had this lance. If I can get this lance
and gay it go, I'll go for the big guy. Yeah, and
that's the kind of bet you need to make. But you know the underlying
part of this, because a lot of the software is already set up for
the right plum tree, but it's the customs suff we're building it is not
is how you provide give visibility into that yep, orienting it around. You
know a lot of people also come most when the start their open plumbatry journey
because we have many of the world's best experts in Hotel, so we could
actually help consult. What do I need to push onto the open telemet pelementary
stack. That's going to help me, that's going to let these tools understand.
What do any of you in the room have a question for charity?
Raise your hand, It's all right right here. Phil Hack has a question,
So why don't you repeat the question? The question is that's a lot
of data and how does that cost? How does the coast get out?
Is the costack get out? Again? This is why So on Twitter I
was joking the other week and it kind of got out of control and I
could never write a database. No, really, never write a database.
And I thought it was a very fun self owned because we wrote a database.
People didn't understand that. So yeah, it's a fund of data.
Like we've got like seven hundred customers and we run the combined production modes of
all of them. It's like two billion events persons or something like that,
and we give everyone sixty days of storage basically for free. And the way
that we do this we so it's a culundar store. So indexes are roboten
for observability because indexes are way of picking. I want this to run fast
and nothing else to run fast. You want to be able to query on
any of these dimensions. So it's a calundar store. And you're right,
like two years in we ran into this. We're never going to be profitable
because there's so all of these SSDs, all this ram. And that's when
one of my I brilliant engineer I've been working he was my first manager.
Name is Ian. He's he's nowhere on the internet and he's amazing. Uh.
He started looking into the cost models and did some tests and so now
actually we data comes in hits, the API gets dropped into Kaffa and then
gets read off onto you know a pair of notes, which are as you
would think like lots of CPO lots of RAM, but then after like thirty
six minutes it gets tailed out to S three. The queery planner actually runs
the LANDA jobs. Uh so the query planner comes in forks out spans and
and like we thought it was going to be so much shorter like doing processing,
you know from all these S threeboutives, it wasn't. It was different
performance characteristics, but most careers still return with under a second and S three
is the cheap so that's what most of the data is, and the lambda
jobs are pretty expensive. That's a big line end up in on our bills.
So we've done we've actually done some really great talks and written some great
pieces about how we use Honeycomb to optimize our LANDA jobs so that the planner,
yeah, it's all. Yeah. Our Honeycomb block, by the way,
is dope. Like we we don't do a lot of selling there.
We just talk about a lot of engineering and it's pretty great. There was
another question back here. First somebody back there had there end up. Okay,
it wasn't you, but go ahead, son, repeat the question.
I'm sorry. So the question I think was something there's something about SLOs and
metrics and it's too expensive to store all the choices for all events, okay,
And there's a few different answers for this. I would probably want to
ask you some more questions. It's feel free to find me afterwards. But
you're absolutely right. It can be absolutely cost prohibitive to store the trace for
every if you have a lot of traffic, because if you think about it,
you might find yourself storing five to thirty times as much prelemetry data as
production traffic. Obviously that's not tenable, right. The first solution that we
usually steer people towards is intelligence sampling, which does not mean just like dumb
dumb sampling, we're like one out of the routen you drop them. It
means like we have a thing called refinerate, where there's a different between head
sampling and tail sampling, meaning sampling before you know what it is coming and
after you know what it's coming. So some of these things you sample after
you know what's coming. Be sure and grab all of the slow events,
right. Some of it is head sampling where you're just like, okay,
for example, requests there are health checks, there are two hundreds. This
is junk. I don't need to store all of these there's gonna be like
a quarter of your traffic sometimes, so like sample them heavily, two hundred
okays to the main page, sample the medium, keep every request that's in
error, or every request that is to slash payments or to billing or you
know, like there's a lot of that is trash that you can like to
discard if you kind of go in there with the fine teeth. Come the
part that was about about SLOs. We derive SLOs from events, and it's
actually really important that they're not derived from metrics. We're actually the only product
out there that does SLOs the way they're supposed to be done for the Google
sor rebook, because other companies don't actually capture their data in a way that
lets them do that. It's actually pretty dope. Like you have your SLOs,
it tells you how how quickly you're burning down the budget, and then
it tells you what what is different about the requests that are erring that are
burning down the budget the other blah blah blah. We also have the metrics,
but I'm not sure if you're talking about our metrics product or the events.
Like the number one answer to the events be too expensive is you use
smart sampling? And the number one answer to the SLOs is you want those
badly. You can absolutely do sampling. So one of the one of the
things in every event that ge is sent to us, there is a sample
rate embedded in it, so everyone will say like one slash five and that
means compute this to be five like this, so the numbers all all work
out to look like they weren't sampled. We had questions. You got to
move on question right in the front, hay on can you repeat that with
the micro The question was, obviously, you can go wrong with logging because
you can get way too many log events. You can go wrong with metrics
because you could you can have famously like the thirty thousand dollars metric that had
high cardinality in it and oops your budget. Answer the question is, I
think, how can you go wrong with observability as distinct from those you know?
And I want to say, even if you aren't doing traces, if
the most important thing to take away when it comes to telemetry is the magic
of the one wide structured event per request per service. I actually found out
years into this that this is how Amazon is done there telemetry all along,
they had like a flat file at the root domain of every node where they
keep one of these like wide It's it's magic. It makes everything because a
lot of the logs that you are encountering, or because like when a request
is executing through through a service, it's just like, oh, all these
strings, right, But if you just like collapse them into one wide event
with all of those keys and values, then you have that context, right
you can put its magic. So the number one thing that I think people
get wrong with observability is not understanding that that's the heart of everything. It
isn't much tool you're using. It isn't whether you're tracing or not. It's
that it's that that is the number one thing that everyone should be caring about.
The number two thing I think comes out when dealing with spans slightly ordered
higher order problem, and that's because I feel like as an industry we have
the really we aren't really we don't really have a set of like good conventions.
You were asking me, like, when should do you have this span?
Man? Like, I hope five years from Everybody's like, well,
obviously you should have this span blah blah blah. But we aren't there yet,
right, and so it's really easy to either generate too many spans and
they get lost in the noise kind of like with logs, or too few
spans and then not have the detail that you need when you needed. The
question is where to start with open planetry, And there are only two good
answers. One is my favorite, uh, with the biggest pain and if
you have to really like you're like, if you have a really resistant culture,
then start with the least pain. But I actually think that the best
way to roll anything that has to do with cymmetry out is is it kind
of think of your attention like a headlamp, and if you're on call for
something that's breaking, have an instruments first mentality, like you've instrument to figure
out what's wrong, not if you've around with your instrument, have to tell
you the answer, and then it's there for the next time you get paid
again, instrument to find the problem, and it's there. And as your
head lamp kind of moves around the stack, you know, within a couple
of months most of the stuff that it really matters will be instrumented and then
you can put it on the backlog to do the rest and finish up and
get rid of your ovenders. All right, Well, I think that's it,
so let's give charity majors a big round of law. I will see
you next time. On time dot net Rocks is brought to you by Franklin's
Net and produced by Pop Studios, a full service audio, video and post
production facility located physically in New London, Connecticut, and of course in the
cloud online at pwop dot com. Visit our website at d O T N
E t R O c k S dot com for RSS feeds, downloads,
mobile apps, comments, and access to the full archives going back to show
number one, recorded in September two thousand and two. And make sure you
check out our sponsors. They keep us in business. Now, go write
some code, See you next time you got jacks. See a summer time
on that means home. Then my texes in my credit b