Speaker 1: Hey, everybody, Welcome to another episode of the Ruby Rouges podcast.
Speaker 1: I am your host today in Valentino, Stall and we
Speaker 1: are joined by a very special guest today, John Gallagher. John,
Speaker 1: can you introduce yourself and tell everybody a little bit
Speaker 1: about yourself from why we heard you on today?
Speaker 2: Sure? Thanks for having me on. My name is John Gallaghan,
Speaker 2: and I am a senior engineer. Had a company called
Speaker 2: Bigger Pockets and we teach how to invest in real
Speaker 2: estate based in the US, and I also run my
Speaker 2: own business on the side called Joyful Programming to introduce
Speaker 2: more joy to the world of programming. And I'm on
Speaker 2: today to talk a bit about observability, which is one
Speaker 2: of my many passions. I'm a bit of a polymath.
Speaker 2: This is one of the things that is really really
Speaker 2: important to me and I'm passionate about and particularly passionate
Speaker 2: about introducing this into rails apps. So thanks for having
Speaker 2: me on.
Speaker 1: Yeah, and thank you for all the joy you're bringing
Speaker 1: to people. I hope you definitely pick the right language Ruby.
Speaker 1: If you're not familiar with this podcast, Ruby is a
Speaker 1: very joyful experience personally, so it's very cool. I love
Speaker 1: I've loved been digging into all of the observability talk
Speaker 1: that you have on joyful programming, And it's a kind
Speaker 1: of a very important topic that I feel is definitely
Speaker 1: kind of overlooked. If you're starting up, maybe you get
Speaker 1: some like you know, bug alerting or something like that
Speaker 1: in place as like a standard, but kind of anything
Speaker 1: performance monitoring wise is kind of like a oh no,
Speaker 1: like something happened, let's look into it now. I feel
Speaker 1: like it's like the typical flow of things as people
Speaker 1: start up. Do you want to just give us like
Speaker 1: a high level like what is observability and why should
Speaker 1: we care?
Speaker 2: What?
Speaker 1: You know? We could drill into the details of it
Speaker 1: after it realiant.
Speaker 2: Well, I don't actually think anybody should care about observability,
Speaker 2: and I don't care about observability as a thing because
Speaker 2: it's just a means to an end, and what's the
Speaker 2: actual goal. It doesn't matter how you get there, but
Speaker 2: the goal is being able to number one understand your
Speaker 2: rails app in production, and number two be able to
Speaker 2: ask unusual questions, not questions that you've thought of a day,
Speaker 2: two days, three weeks ago, because that's not really very
Speaker 2: useful or interesting if we knew exactly the questions to ask.
Speaker 2: In the future of our apps, everything will be easy.
Speaker 2: Just be like, how many two hundreds have we had
Speaker 2: in last week? Kind of boring question to ask, maybe
Speaker 2: a bit useful. I find the more obvious the question,
Speaker 2: the less useful it is. So observability is the practice
Speaker 2: of making a black box system more transparent. So I
Speaker 2: like to think of it. Imagine your entire rails app,
Speaker 2: all the hosting, everything to do with that app is
Speaker 2: wrapped up in an opaque black box, and somebody says,
Speaker 2: how does it work? And why is this thing going wrong?
Speaker 2: You would have no hope of understanding it if the
Speaker 2: box is completely translucent and you can see everything, which
Speaker 2: of course is completely impossible in software. But in theory,
Speaker 2: you'd have this completely translucent box and you can ask
Speaker 2: all these questions and you get instant that's like one
Speaker 2: hundred percent observability. And of course that is absolutely impossible.
Speaker 2: And so what we're trying to do with observability is
Speaker 2: understand what is going on, not just when it goes wrong,
Speaker 2: although that's the obvious use case. Is we have an incident.
Speaker 2: The most you know, the most critical point where observability
Speaker 2: comes into play is an exact scenario that I landed
Speaker 2: in two weeks into a new role I had, So
Speaker 2: it was two weeks in the site had gone down.
Speaker 2: I was in the U, I am in the UK,
Speaker 2: and my tea, the rest of my team we're in
Speaker 2: the US and there were two other engineers in my
Speaker 2: time zone, and all of us have been at the
Speaker 2: company for a total of five weeks. So we've got
Speaker 2: this app. It's down, it's on fire, and we need
Speaker 2: to put the fire out. And we just the three
Speaker 2: of us looked at each other. We're like, what should
Speaker 2: we just restart the dinos? Yeah, so we restarted the dyno.
Speaker 2: We crossed our fingers and it was pure. Look that
Speaker 2: the app came back up. That is the exact opposite
Speaker 2: of what we want. And we've now moved to a
Speaker 2: situation where we can ask our app a whole load
Speaker 2: of very unusual questions and we will get an answer
Speaker 2: to that. Why are there a peak of four h
Speaker 2: fours on iOS at three a m. Looks like that
Speaker 2: a lot of them are coming from this IP address. Okay,
Speaker 2: what's that IP address doing on the site? Okay? Interesting?
Speaker 2: How many users are using that IP address? Five? So
Speaker 2: only five people using and you keep So that's the
Speaker 2: point of observability to me, to be able to ask
Speaker 2: unusual questions that you haven't thought of already dynamically and
Speaker 2: explore the space and come to some conclusions.
Speaker 1: Yeah, I think that's a great overview. And your debugging
Speaker 1: reminded me of that. I have the lucky experience of
Speaker 1: running rails with Ruby one point eighty seven, and every
Speaker 1: once in a while you just had to like give
Speaker 1: the give the server a little kick because it started
Speaker 1: to grow in memory size and just you know, giving
Speaker 1: it a quick little flush like reset things. And you're
Speaker 1: just like, oh, I guess that's how we're gonna do
Speaker 1: it until we can get like some insight into what's happening, right,
Speaker 1: And I think that's definitely underlines the importance of observability
Speaker 1: in general, Like, uh, you know, how do you get
Speaker 1: those insights to begin with? And maybe that's a great
Speaker 1: starting point, Like where do you start like looking at it,
Speaker 1: like adding this insights? Right, Like what's the Is there
Speaker 1: a modular approach you could take or is it more
Speaker 1: of like you should look at doing everything all at
Speaker 1: once kind of thing?
Speaker 2: You should definitely not look at everything at once as
Speaker 2: I think we can all agree in software doing everything
Speaker 2: all that wants to the rest be for disaster. No
Speaker 2: matter what you're doing.
Speaker 1: There's no vendor you could just like pay money to
Speaker 1: and like you get onbility.
Speaker 2: There are vendors that tell you that you can do that,
Speaker 2: whether you actually kind of know, it's a different matter.
Speaker 2: Spoiler alert, you can't. So I just want to back
Speaker 2: up a little bit and talk about the feelings, because
Speaker 2: I think it's the feelings that is where all of
Speaker 2: this start for me, pon me. So I got into observability.
Speaker 2: And it's funny because for the first kind of year
Speaker 2: of my journey doing this, I didn't even realize I
Speaker 2: was doing observability. I'd heard about this observability thing and
Speaker 2: it was out there in the universe. Okay, maybe I
Speaker 2: should learn that. I should learn that, and I kept
Speaker 2: using this. Should I should learn this? Have loads of
Speaker 2: other stuff to do. I've got loads of other things.
Speaker 2: I don't know what it is. I know it comes
Speaker 2: from controls here, and there's a Wikipedia page. It's really
Speaker 2: complex and really confusing whatever. I've got real work to do.
Speaker 2: But what I know is that he kept coming across
Speaker 2: these bugs in bugs, Century air Break, choose your error
Speaker 2: reporting tool. They all help you to a degree, but
Speaker 2: they're not a silver bullet. And I kept coming across
Speaker 2: these defects over and the story was exactly the same.
Speaker 2: Come across the defect, I'd see the stack trace in
Speaker 2: the error reporting tool, and I would look at it
Speaker 2: and first first emotion right out of the gate, complete confusion.
Speaker 2: What is going on here? No idea? So I dig
Speaker 2: a little bit into the code and dig it a
Speaker 2: little bit into the stack trass. So it's coming from
Speaker 2: here and this thing is nil. Classic right, this thing
Speaker 2: is nil? Where is it being passed in this nil?
Speaker 2: I don't know. So now I'm like, well, I can't
Speaker 2: just say I can't fix this, so I now have
Speaker 2: to well do what exactly? I don't have any information
Speaker 2: to go off. Well, I guess we'll do that. Let's
Speaker 2: look at the next one. And this just kept happening,
Speaker 2: and I would find myself going through all the bugs
Speaker 2: in the backlog and I couldn't fix any of them.
Speaker 2: And I just wasted four hours looking at things, asking
Speaker 2: questions that I couldn't explain, looking at things I didn't understand,
Speaker 2: And for years I thought the problem was with me.
Speaker 2: I honestly thought, I'm just not smart enough. I'm not
Speaker 2: a good engineer. Blah blah blah blah blah. Bug fixing
Speaker 2: isn't really my thing. It's I'm just not really good
Speaker 2: at it. And then, after many, many years of this,
Speaker 2: I was in a company and I just got really
Speaker 2: sick of this. We just released a brand new app
Speaker 2: and it was it was a customer account app, and
Speaker 2: we're getting all these weird bug reports, people saying how logging.
Speaker 2: People kept saying, I can't reset my password. And every
Speaker 2: time we did this, we would add a little bit
Speaker 2: of kind of this ad hoc logging and then put
Speaker 2: the bug back in the backlog, and then it would
Speaker 2: come up again and come up again. And after a while,
Speaker 2: I was just like, this is just this is ridicul
Speaker 2: We're highly paid engineers. Is it not a better way?
Speaker 2: So then I started looking into we were using Cabana
Speaker 2: at the time, or rather I should say we were
Speaker 2: not using Cabano at the time. Cabana was there, we
Speaker 2: were paying for it, and I was like, I've heard
Speaker 2: this is something to do with logging, So where do
Speaker 2: we do our logging? People like Cabana, I have no
Speaker 2: idea what this even is. Let's open it up, and
Speaker 2: there was just all this trash, all this rubbish. I
Speaker 2: was like, what's this? How is this supposed to be useful?
Speaker 2: People like, oh, we don't really look at that. It's
Speaker 2: not very useful. So so how do you figure out bugs?
Speaker 2: And they're like, well, we just yes, we just figure
Speaker 2: it out. Well, yes, but we're not figuring it out.
Speaker 2: So all of this was born through frustration. And so
Speaker 2: what they did back then is what I recommend everybody
Speaker 2: does now to answer to your To answer your question,
Speaker 2: come back to the point, John, which is, take a
Speaker 2: question that you wish you knew the answer to a
Speaker 2: very specific question, not why is our app not performing
Speaker 2: as we want? Not as in like why do our
Speaker 2: you know? A very very specific question. So take your big,
Speaker 2: big question, and the time this was why are people
Speaker 2: being locked out of the app? Why can they not
Speaker 2: reset the password? They're clicking on this password link and
Speaker 2: they're saying it's expired or it goes nowhere or it
Speaker 2: doesn't work. Okay, why are those people like? Why is
Speaker 2: that happening? So that's quite a general question, and you
Speaker 2: want to break it down into some hypotheses. So that's
Speaker 2: the first thing. I have a five step process, and
Speaker 2: this is step one. I'll go through the five step
Speaker 2: process in a minute. So Step one is think of
Speaker 2: a specific question. So a specific question. This might case
Speaker 2: might be Okay, I've got one customer here. There's many,
Speaker 2: many different types of defect. So this one customer here
Speaker 2: is saying it was expired. I went to the web
Speaker 2: page and the said it had expired. Okay, when did
Speaker 2: they click on that link? What response did the app
Speaker 2: give to them? And when did the token time out? Right?
Speaker 2: So those are three questions. Now, they're not going to
Speaker 2: get us to the answer directly, but there are three questions,
Speaker 2: very specific questions that we can add instrumentation for. So
Speaker 2: I take one of those questions, when did the token
Speaker 2: time out? Great question. So in order to do that,
Speaker 2: we need to know when the token was created and
Speaker 2: what the expire of the token. Well, this is just
Speaker 2: a random example off the top of my head. So
Speaker 2: you'd be like, Okay, well we need to know the
Speaker 2: customer ID. We need to know the token. We don't
Speaker 2: actually need to know the exact token, but we need
Speaker 2: to know the customer ID. We need to know the
Speaker 2: time that the token was created and the spiry time
Speaker 2: of that token. Is it fifteen minutes, is it two hours? Whatever?
Speaker 2: So I would then look into the code. That's the
Speaker 2: next So we've done step two. Step two is define
Speaker 2: the data you want to collect, user ID token expiry,
Speaker 2: and an event saying the token has been created now
Speaker 2: for this user ID. Okay, So that's the second step.
Speaker 2: The third step is build the instrumentation to do that.
Speaker 2: So whatever you have to do, maybe it's you need
Speaker 2: to actually add structured logging to your entire app. I
Speaker 2: don't know. Maybe it's that you've got the structured logging fine,
Speaker 2: but there's nothing listening to it. Maybe maybe the tool
Speaker 2: just can't actually measure what you want it to measure,
Speaker 2: So maybe you need to invest in a new tool,
Speaker 2: whatever it is. And then you build some code to
Speaker 2: instrument just that very small piece of functionality. And then
Speaker 2: once you've done that, you wait for it to deploy
Speaker 2: and then you look at the graphs, you look at
Speaker 2: the logs, at the charts, whatever out but you've got,
Speaker 2: and what normally happens is for me, I look at
Speaker 2: the charts and I say that is not what I
Speaker 2: wanted at all, Actually I've misunderstood the problem. I've misunderstood
Speaker 2: the data I want. Now that I see it. Ah,
Speaker 2: just like you would with agility, true agility not agile,
Speaker 2: because agile mean something else now, But true agility is
Speaker 2: you do a little bit of work, You develop a feature,
Speaker 2: You show the customer. They say, not quite right, go back,
Speaker 2: adjust it closer, but still not quite right. But if
Speaker 2: you ask them to describe it exactly right from the beginning,
Speaker 2: it doesn't align with what they want at all. You
Speaker 2: need to show them, and it's only by showing them
Speaker 2: that you get feedback. And the same is true for ourselves.
Speaker 2: It's only by looking at the graphs and the logs
Speaker 2: that I realized that actually isn't what I wanted to
Speaker 2: begin with, or it is, or I'm onto something there,
Speaker 2: and so I keep. Then sort of I've used the graph,
Speaker 2: maybe it was unusable, maybe I could query the parameter.
Speaker 2: Maybe there's all sorts of things that might be happening there.
Speaker 2: So then the last stage is improved, and so from
Speaker 2: improve you can go back to the very beginning, ask
Speaker 2: a different question, or maybe you just want to a
Speaker 2: trade on the instrumentation a bit, deploy it again. Oh,
Speaker 2: that's more like it. Okay, So now we know the
Speaker 2: token expiry, what's the next question we want to ask? Well,
Speaker 2: why did like, when did the user actually hit the site?
Speaker 2: Was it after the token expiry or before? Hmm? Okay,
Speaker 2: sounds like an obvious question. But maybe maybe it's after,
Speaker 2: which would indicate the token really had expired. Oh it's before. Huh?
Speaker 2: How could it be expired when it was before? Oh?
Speaker 2: Hang on, what's the time zone of the token? Now
Speaker 2: we're getting into it, right, so you've logged the time zone.
Speaker 2: Holy how the time zone of the token is out
Speaker 2: of sync with the time zone of the user? That's
Speaker 2: what it is. Yeah?
Speaker 1: I love that. I love that analogy of identifying the
Speaker 1: use case in order to expose what to observe and
Speaker 1: where to insert. You know, all of these pieces that
Speaker 1: are missing, I identify them really right, not to just
Speaker 1: insert them, but to identify them. I think that's very important.
Speaker 1: I think in general, is like trying to identify the
Speaker 1: actual use cases in order to know what you even
Speaker 1: want to capture to begin with. Right, Like, Yeah, we
Speaker 1: get to throw a wall of logs at a source
Speaker 1: resource like Cabana, and it's not very useful. But once
Speaker 1: you start to abstract the ideas and use cases and
Speaker 1: how people are actually like using the thing that you've built,
Speaker 1: you know, you can definitely isolate what it is that
Speaker 1: you actually care about. And I think that I think
Speaker 1: you're right, like that is like kind of the whole
Speaker 1: importance of observability is identifying the use case and exposing
Speaker 1: what you actually care about as far as all of
Speaker 1: these things that are because I mean, you know, there's
Speaker 1: HTTP logs, there's like all kinds of logs and information
Speaker 1: available that's just like emitting all the time. Like how
Speaker 1: do you know and identify you know, which are really important?
Speaker 1: And I think it just depends, right, like what do
Speaker 1: you you know? What are you trying to capture? So
Speaker 1: it's a it's a great like step wise way to
Speaker 1: just like start to figure that out, right, because yeah,
Speaker 1: just depending on your role and depending on what you
Speaker 1: know your responsibilities are, that could change and that could
Speaker 1: be different, and your observability needs will change with that.
Speaker 1: So identifying that is probably most important, I think.
Speaker 2: But as with everything else, I would say, if you're
Speaker 2: really not feeling any pain, don't bother, Just don't bother.
Speaker 2: I'm not into kind of not really interested in telling
Speaker 2: people what they should be doing or could be doing.
Speaker 2: Havan goodness. Mean we hear off of that in engineering,
Speaker 2: don't we. You should really learn a language every year.
Speaker 2: You should be blair, you should be blair. You'm sick
Speaker 2: of it, absolutely sick of all these gurus telling me
Speaker 2: what to do and what I should be learning and
Speaker 2: what I and very few of them talk about, well,
Speaker 2: what's the benefit to me? And in order for me
Speaker 2: to do anything, in order for me to change as
Speaker 2: a human being in any way learn anything, I have
Speaker 2: to feel the pain of it. If you're not feeling
Speaker 2: the pain, don't bother. But if you are feeling the pain,
Speaker 2: if deploys are really glitchy, if you keep asking for me,
Speaker 2: the kicker is if I keep asking questions I don't
Speaker 2: have the answer to, that's a concern. And if they're
Speaker 2: just mine, oh, like why did I wake up ten
Speaker 2: minutes late today? Who cares? It's not important. But if
Speaker 2: the site's gone down for the fourth time this month,
Speaker 2: and every time the site goes down, we lose at
Speaker 2: least five grand, ten grand, maybe even more, and even worse,
Speaker 2: every single type in the site does go down we
Speaker 2: just kind of get it back up more by luck
Speaker 2: than good judgment, this kind of feeling of, oh, we
Speaker 2: kind of got away with it that time. That's okay.
Speaker 2: I know there was this weird thing and it's still
Speaker 2: not really figured that one out, but that's okay. We'll
Speaker 2: just put it in the back blog. It's the operational risk.
Speaker 2: You've got to decide are you comfortable with that operational
Speaker 2: risk or not? Is it big enough? And in my experience,
Speaker 2: you've kind of got to hit rock bottom with this
Speaker 2: stuff as I did. There were loads and loads of
Speaker 2: bugs that I could have investigated and added logging for
Speaker 2: and fixed. But you know, it's pushing a boulder up
Speaker 2: a hill. It's not actually worth it. And it was
Speaker 2: only when it reached my threshold of pain I was like,
Speaker 2: you know what, I have to do something about this now.
Speaker 2: This is just ridiculous. We're professional people, were being paid
Speaker 2: a lot of money and it's not working. The app
Speaker 2: that we've delivered is not working, and what's more, we
Speaker 2: don't know why. But also I do just want to add,
Speaker 2: and this may broaden out the conversation a little bit.
Speaker 2: You may want to maybe we may want to keep
Speaker 2: it narrow on rails apps. But I've realized that observability
Speaker 2: principles go way beyond how does our web back work.
Speaker 2: It applies to any black box. So as an example,
Speaker 2: a few years ago, I was working at a company
Speaker 2: and their SEO wasn't great, and they just kind of
Speaker 2: were like, oh, you know, well, we'll try and fix it.
Speaker 2: And they had several attempts to fix it. None of
Speaker 2: them really worked, and every attempt was the same. They
Speaker 2: would get some expert in the expert would give us
Speaker 2: a list of one hundred things to do. We would
Speaker 2: do eighty of the hundred, and then nothing would really improve.
Speaker 2: And then they'd be like, well, we did everything you said,
Speaker 2: and then they move on to another and rin some repeat,
Speaker 2: keep doing that, and then one day, within four weeks,
Speaker 2: twenty percent of the site traffic disappeared, and nobody could
Speaker 2: tell us why. Nobody understood why observability. Now, Google is
Speaker 2: a black box, so you know you're not going to
Speaker 2: be able to instrument Google. But there's lots of tools
Speaker 2: that allow you to peer into the inner workings of Google,
Speaker 2: Zen Rush, Screaming Frog, all these kind of tools. They are,
Speaker 2: in my opinion, actually in to some degree the observability space.
Speaker 2: They're not you know, everybody thinks of them as marketing tools, SERPs,
Speaker 2: engine and optimization tools, whatever, whatever whatever. They're allowing you
Speaker 2: to make reasoned guesses about why your searchers aren't performing
Speaker 2: the way they are, and then you can actually take
Speaker 2: action on that because now you have some data. Oh
Speaker 2: this keyword dropped from place four to place one hundred,
Speaker 2: why is that? Okay, let's try a let's try hypothesis A.
Speaker 2: Put that live and see if Google will respond to that.
Speaker 2: Oh and now up to you know, position eighty whatever
Speaker 2: it is. So the idea of observability goes way way
Speaker 2: beyond like data dog and you relic and obviously all
Speaker 2: of those people in the observability space. But I see
Speaker 2: it as a much much wider and much more applicable topic.
Speaker 1: Yeah, I hear you there, and I'm all against. I'm
Speaker 1: all also like, you know, let's not just add new
Speaker 1: relic to every app that we deploy or you know
Speaker 1: it is bugsnag even needed for every app. Like these
Speaker 1: are questions that I ask myself too, like what value
Speaker 1: are you getting from all these auxiliary services they give
Speaker 1: you the observability into like just blanket things right, like
Speaker 1: what at what point do you? Like stop like that
Speaker 1: kind of mentality and be like, I will you know,
Speaker 1: every rails app should at least be able to get
Speaker 1: insight into the logs so that you can see what
Speaker 1: the application is doing, Like, well, how long do you
Speaker 1: capture that? Like what kind of time frame do you have?
Speaker 1: Any like default standards where you're like, well, I know
Speaker 1: that I'm going to need to look at this at
Speaker 1: some point in the application cycle, like what are your defaults?
Speaker 2: Great question, I would say if you're if you're making
Speaker 2: a small app with very little traffic and it's thresholds
Speaker 2: like kind ofthing. Else, if you're making a small app
Speaker 2: with very little traffic. I have a client at the
Speaker 2: moment I'm consulting for and I've made them an app
Speaker 2: and it has maybe flipping twenty visits a day or
Speaker 2: something twenty hits a day. So I installed rollbar free
Speaker 2: version of roll bar. Anything goes wrong again, an outtion,
Speaker 2: It's fine. The further up the stack you move, the
Speaker 2: more the defaults change. For a rails app that's mission
Speaker 2: critical that I'm not even going to say mission critical,
Speaker 2: but just serving a decent number of hits a month
Speaker 2: ten twenty thousand. I don't know. I've tried a lot
Speaker 2: of observability tools and there's no one that yet that
Speaker 2: I can unreservedly recommend. They all got their pros and cons.
Speaker 2: Data dog is a good option if money is no object.
Speaker 2: I kind of don't want to get into the tooling
Speaker 2: debate because it's kind of a bit of a red herring.
Speaker 2: I think in many ways there's various costs benefit trade
Speaker 2: offs there, but in terms of the defaults, in terms
Speaker 2: of what you observe requests, has got to be up there.
Speaker 2: So every app that I I have in my care
Speaker 2: of any significant size, I would always say installed Semantic Logger.
Speaker 2: Semantic Logger is the best logger I've found out does
Speaker 2: Jason out of the box. It's quite extensible. There are
Speaker 2: many problems with it, but it's the best option that
Speaker 2: we've got. So that's number one that will log every
Speaker 2: like Rails already logs every request for you that will
Speaker 2: form it in Jason for you. There are some notable
Speaker 2: missing defaults in semantic Logger, and I'm working on a
Speaker 2: gem at the moment that will add some even more
Speaker 2: sensible defaults into it. So For example, I believe that
Speaker 2: request headers do not get logged out of the box.
Speaker 2: Certainly request body does not get logged out of the box.
Speaker 2: Request headers might be the user agent doesn't get logged
Speaker 2: out of the box. I mean, this is just pretty
Speaker 2: basic stuff, right, But so I have a setup that
Speaker 2: I use is the logs a whole load of things
Speaker 2: about the requests out of the box. I liked adding
Speaker 2: user ID out of the box. And it depends what
Speaker 2: kind of setup you have for authentication. But at the
Speaker 2: very very least, if somebody's logged in the idea of
Speaker 2: them should be logged in every single request. That is
Speaker 2: absolutely you know, absolutely basic stuff. A request I D
Speaker 2: is also a really really useful one. I have a
Speaker 2: complex relationship with logs and tracing because tracing is essentially
Speaker 2: the pinnacle of observability. I hear a lot of people
Speaker 2: say logging look like logging's a be all and end all.
Speaker 2: Logging is a great place to start, but tracing is
Speaker 2: really where it's at, And I can go into that
Speaker 2: why that is in a bit. But logging is a
Speaker 2: great default. Logging is a good place to start. Start
Speaker 2: with semantic logger. Basically every single thing that's important in
Speaker 2: any request should be logged, so that's every header. Obviously
Speaker 2: you need to be careful with sensitive data and headers
Speaker 2: like do your rails active. I can't remember what it's called,
Speaker 2: but there's the filtering module that you can add in
Speaker 2: and sometimes semantic logger doesn't give you that by defaults
Speaker 2: you need to be a bit careful. A good default
Speaker 2: as well is logging all background jobs. Background jobs are
Speaker 2: one of the most painful areas of observability that I've experienced,
Speaker 2: and we still haven't really cracked it. We have some
Speaker 2: very very basic logging out of the box, and semantic
Speaker 2: Logger I believe it logs the job class, the job ID,
Speaker 2: and a few other things, but it doesn't log the latency,
Speaker 2: which is a huge, huge missed opportunity. And it also
Speaker 2: I don't believe it logs the request I d from
Speaker 2: whence it was in queued. So when a job is
Speaker 2: in queued, it will by default semantic Logo will trigger
Speaker 2: a little entry in the logs this job is in queued,
Speaker 2: and it will tell you what request it came from.
Speaker 2: But on the other side, when it's picked up and
Speaker 2: the job is performed, that request ID is missing. So
Speaker 2: you need to kind of go into the request ID,
Speaker 2: find the qued job, find the job I D and
Speaker 2: then take that next leap. So I mean it's a
Speaker 2: bit clunky, but it's manageable. So in short, semantic logo
Speaker 2: gives you you some okay defaults out of the box,
Speaker 2: but there's some really basics that it still misses. And
Speaker 2: so background jobs requests those are the two really really
Speaker 2: big ones to start out with. But as you can imagine,
Speaker 2: there are a tunnel.
Speaker 1: Yeah, you mentioned kind of some key pieces I always
Speaker 1: think of with observability in general, which is like separating
Speaker 1: the the pieces into their own puzzle, right, Like we
Speaker 1: have logs which are kind of just like our data,
Speaker 1: and then we have individual metrics that were like snapshotting
Speaker 1: the logs for particular segments like traffic or a number
Speaker 1: of people using it, like the number of jobs that
Speaker 1: are running. And then there are traces which we could
Speaker 1: dig into next. Because I have a lot of I
Speaker 1: have a lot of love for all of the standards
Speaker 1: that are coming out of this, with open tracing and
Speaker 1: things like that. I love to dig in there. But
Speaker 1: also like alerting, like you know, how does anybody know
Speaker 1: that there's ever a problem Like yeah, I mean and
Speaker 1: I love I love like thinking about it in these
Speaker 1: separate groups and categories, because I think it also helps
Speaker 1: to think about like the overarching theme. We're just like
Speaker 1: getting insight but also like getting meaningful insight and like
Speaker 1: when you want really like the only the only reason
Speaker 1: anybody ever cares about the ability anyway is like when
Speaker 1: something goes wrong or something is problematic that causes something
Speaker 1: to go wrong, and do you want to either catch
Speaker 1: it early or you know, try and remediate and so
Speaker 1: like where do you find like, I mean, background jobs
Speaker 1: are like kind of like I feel like the first
Speaker 1: instance where people realize, like, oh, like we need to
Speaker 1: start looking at you know, what it's doing, right, Like
Speaker 1: you start throwing stuff in the background, You're like, okay, great,
Speaker 1: like it's doing the work. And then you don't maybe
Speaker 1: realize if you're on the same node, like well those
Speaker 1: you know, slow requests can block the web requests, right
Speaker 1: and then okay, well if you split those up, finally
Speaker 1: got that result. But then okay, well one problematic you
Speaker 1: know job can pack up a queue that it's on,
Speaker 1: you know, like where do you uh.
Speaker 2: To me?
Speaker 1: Like the background processing aspect is like why we have
Speaker 1: tracing to begin with, because it does like its concurrency, right,
Speaker 1: So it's like that's where everybody like ends up hitting
Speaker 1: their pitfalls as soon as you start doing things like
Speaker 1: all at once, like and thinking, oh, like we just
Speaker 1: throw it in the background and like process things as
Speaker 1: they come and as things start to scale, it causes
Speaker 1: more problems as you try and figure out timing and
Speaker 1: stuff like that, like where do you find the most
Speaker 1: important pieces of like making sure that you you know,
Speaker 1: are capturing the right segments and the right flows you
Speaker 1: know in that process.
Speaker 2: Yeah, there's so many things you touched on there. I
Speaker 2: want to come back to to answer your question. First
Speaker 2: of all, it's the five steps that I walk through. Yeah,
Speaker 2: that's the short answer is if you have a specific
Speaker 2: question that you cannot answer, what we're really talking about
Speaker 2: is the implementation details of how you answer that question.
Speaker 2: So what question you pick determines a whole load of
Speaker 2: a whole load of stuff. I can't just give you
Speaker 2: a bog standard answer because it just it depends. I
Speaker 2: hate saying that, but it does. So I think the
Speaker 2: first question is to ask the question, figure out what
Speaker 2: data is missing, and then choose the right piece to
Speaker 2: add into your logs. I feel like I've maybe not
Speaker 2: understood your question.
Speaker 1: Maybe yeah, I mean it's more of like an open,
Speaker 1: open question. I guess like when trying to think about,
Speaker 1: like one of my biggest debugging pitfalls is like trying
Speaker 1: to like reconstruct the state of what happened when something
Speaker 1: went wrong. It's like, I feel like that's like one
Speaker 1: of the most typical things. It's like, Okay, something been
Speaker 1: like well, like it's the data has changed since something
Speaker 1: had happened. Maybe the change resolved the issue, but like
Speaker 1: you know, trying to figure out what that is and
Speaker 1: going running through those questions, right, like how do you
Speaker 1: think about like reconstructing data or reconstructing the state of
Speaker 1: an issue? Like is that not the right way to
Speaker 1: go about it? Or do you try and like do
Speaker 1: something else?
Speaker 2: Fantastic question. So and this gets to the route of
Speaker 2: why the three pillars are complete nonsense? Okay, so the wait,
Speaker 2: what are the three pillars metrics, traces, and logs? Okay, nonsense,
Speaker 2: They're not three pillars. And the analogy I like to
Speaker 2: use is saying that observability is three pillars, and it's traces, logs,
Speaker 2: and metrics is a bit like saying programming is three pillars.
Speaker 2: It's to raise integers and strings. It's the same kind
Speaker 2: of deals. No, it's nothing to do with those things.
Speaker 2: Well it is because you use those every day. Yes,
Speaker 2: but you kind of missing the point. So thanks to
Speaker 2: some amazing work by people at Honeycomb and Charity Majors
Speaker 2: and reading their stuff and reading their incredible work, I've
Speaker 2: realized that the metrics tracing logs and missing the point.
Speaker 2: The point is we want to see events that happened
Speaker 2: at some point in time, and that neatly answers your
Speaker 2: question about how do you reconstruct the state of y APP?
Speaker 2: I mean, the short answer is, of course you can't.
Speaker 2: If you're not in an event driven to the system,
Speaker 2: if you're in a crud AUP, if you're storing state
Speaker 2: to the database. There is no way you can go
Speaker 2: back in time and accurately recreate it. But we can
Speaker 2: give it a reasonably good stab. And we can do
Speaker 2: this by capturing the state of each event when it was.
Speaker 2: Forget about observability tools and logging and structured logging and tracing.
Speaker 2: Just now imagine if when that incident happened, let's say
Speaker 2: my expired token would be would be maybe potentially a
Speaker 2: good example. There are several points in that timeline that
Speaker 2: we want to understand. Number one, when the token was created.
Speaker 2: Number two when the user hit the website, and maybe
Speaker 2: there's a third one when the account was created. Let's
Speaker 2: say that. So imagine if at each of those three
Speaker 2: points we had a rich event with everything related to
Speaker 2: that event in it. So when the account was created,
Speaker 2: we had the account i'd, the status of the account,
Speaker 2: whether it's pending or not, the creation date, the customer,
Speaker 2: the customer ID, blah blah blah, blah blah. And then
Speaker 2: when the user visited the site, what was the request?
Speaker 2: What was the request idea? What was user I D?
Speaker 2: What was the anonymous user I D? Et cetera, et cetera.
Speaker 2: And then when the token was created, what was the expiry?
Speaker 2: What was this? What was that? What was the user
Speaker 2: I D? Okay, So if we have those three events,
Speaker 2: and we have enough rick data gathered with each of
Speaker 2: the events, we can answer your question does that make
Speaker 2: sense so far? And there's a whole load of more
Speaker 2: blah blah blah, But does that make sense so far?
Speaker 1: I think that you're making some great points of like
Speaker 1: capturing the transactional uh like user information or users actions.
Speaker 2: Yes, and also also other events that happening in the system.
Speaker 2: So there's user did something, computer did something, computer and
Speaker 2: cuter background job, performed a job, et cetera, et cetera.
Speaker 2: So the way I think about it is everything that
Speaker 2: happens in your app, whether it's initialized by the computer
Speaker 2: and external data source use its basic events storming stuff
Speaker 2: really that creates an event, and that event, if you
Speaker 2: don't capture enough data, that is it. The data is
Speaker 2: lost forever if you're not in an event and assuming
Speaker 2: you're not doing event sourcing, and assuming you're not in
Speaker 2: an event driven system. So to the way I think
Speaker 2: about it at the most core fundamental level is whether
Speaker 2: it's druck logs, traces, metrics, whatever it is, we need
Speaker 2: a way of capturing those events. And more importantly, ideally
Speaker 2: we need to link the events together. And this is really,
Speaker 2: really really important. So if somebody create let's say somebody
Speaker 2: hits our app and it creates the token, well there's
Speaker 2: two parts that they hit the app. There was a
Speaker 2: request to our app, and then in the call stacked
Speaker 2: somewhere the token is created. Those two things are two
Speaker 2: separate events, but they're nested. We want to capture that
Speaker 2: causal relationship one cause the other one is a subset
Speaker 2: of the other. One is a parent a child.
Speaker 1: However you want to.
Speaker 2: Put it without that call link, we're lost again. We
Speaker 2: don't know what caused what. So there are some three
Speaker 2: or four ideas here. Number one events, number two contextual
Speaker 2: data with each of those events, and number three nested events.
Speaker 2: If you like causal relationships between events, and with those
Speaker 2: three things, you can debug any problem that you would like,
Speaker 2: is my claim. And so if you just keep that
Speaker 2: model in mind, let's examine traces, logs, and metrics and
Speaker 2: see where they fall short. See which one meets those criteria.
Speaker 2: So tracing gives us all three you can. So for
Speaker 2: those of who I should explain what tracing is, because
Speaker 2: I was confused about what tracing even was for absolutely years.
Speaker 2: So tracing allows you to when somebody hits your app,
Speaker 2: a trace is started. So there are two concepts in tracing.
Speaker 2: There's traces and there are spans, and then there's the
Speaker 2: data associated with spans. But let's just leave that to
Speaker 2: one side. So when somebody hits your app with a request,
Speaker 2: a trace is started, and so the trace would be like, okay,
Speaker 2: I've started here, I am you can append any data
Speaker 2: that you want to me whilst I'm open. It's like
Speaker 2: opening the cupboard door and then you keep putting stuff
Speaker 2: in the cupboard, and then once a cupboard doors closed,
Speaker 2: you can't put any more stuff in it. Very simple analogy.
Speaker 2: So we open the door, we start the trace, and
Speaker 2: so it goes down to the controller level and the
Speaker 2: controller says, oh, I'm going to glom on some data
Speaker 2: into whatever the existing traces about the method, the post body,
Speaker 2: the request blah blah blah blah blah, headers, whatever it is.
Speaker 2: I'm going to glomb that onto the current trace. And
Speaker 2: then we get down into maybe you've got a service object.
Speaker 2: I know some people hate them. I love themlah blah
Speaker 2: blah whatever. That's not the podcast about John. So you
Speaker 2: get into a service object and the service object says, oh,
Speaker 2: whatever is in the current trace, I want you to
Speaker 2: know you hit me, and you hit me with these arguments. Cool,
Speaker 2: I'm going to append that to the trace as well,
Speaker 2: and then we enqure background job. That event gets added
Speaker 2: onto the trace, and then even more excitingly, there's a
Speaker 2: setting in open cemetry where when the job is picked
Speaker 2: up and performed that the trace is kept open. And
Speaker 2: there's a whole load of debate about whether this is
Speaker 2: a good idea or not, but you can do it.
Speaker 2: You can keep the trace open until that job is started.
Speaker 2: And so the job says, ah, I've kicked off. Now
Speaker 2: it gloms a whole load must maybe you make an
Speaker 2: API request in the job. It gloms a whole load
Speaker 2: more stuff into the trace, and then it comes all
Speaker 2: the way back up the stat and you have this
Speaker 2: trace with all this nested context. And when it's saying
Speaker 2: I'm going to glom this data onto the trace, that's
Speaker 2: called a span, and a span is nested, so you
Speaker 2: can have spans nested inside spans inside span. So essentially
Speaker 2: it's this big tree structure. And you might have seen
Speaker 2: this before. It's the flame graph that you get in
Speaker 2: data Dog and new Relic and all these kind of things.
Speaker 2: And everybody looks at these things and things are really pretty,
Speaker 2: and they are indeed they are. So that is the
Speaker 2: that's the pinnacle of observability. In my head. Traces give
Speaker 2: it us all and we can say, as you can
Speaker 2: do in any of these observability tools that support tracing,
Speaker 2: you can do some really cool stuff. Show me all
Speaker 2: the requests that were a two hundred the enqueued a
Speaker 2: job where the job lasted for more than three seconds.
Speaker 2: Holy cow, now we're cooking with gas. We've got everything
Speaker 2: that we need. Show me all the spans that indicated
Speaker 2: anything to do with the background job where it was
Speaker 2: a five hundred response but the user was logged in
Speaker 2: and and so we can start to not only query
Speaker 2: the spans, but query the parents of the spans. So
Speaker 2: you've got all of these nested cause of relationship and
Speaker 2: it gets ridiculously powerful. So that's traces cool. Let's look
Speaker 2: at logs. What do logs give us? Well, it gives
Speaker 2: us events. That's all logs are. Really, it's a series
Speaker 2: of events that happened. Does it give us the ability
Speaker 2: to nest events inside one another? Nop, Sorry, your looks out.
Speaker 2: You can log causation IDs and you can link them together.
Speaker 2: And obviously you can log request IDs and filter everything
Speaker 2: by the request I do, but there's no concept in
Speaker 2: the log of this log is nested inside this other log.
Speaker 2: So that information goodbye, it's gone. Don't have it, but
Speaker 2: you have the rich data in every event. Let's look
Speaker 2: at metrics. What does metrics give you. Doesn't give you
Speaker 2: the events, doesn't give you the nesting, and it just
Speaker 2: gives you some aggregated numbers. So I don't think of
Speaker 2: them as three pillars. They're three rungs of a ladder.
Speaker 2: The very top rung is tracing awesome, The next rung
Speaker 2: down is logs. Pretty good. Metrics are useless. Now when
Speaker 2: I say metrics are useless, people get upset with me
Speaker 2: and say, oh, well, I look at metrics all the
Speaker 2: time to understand my app Yeah, okay, but if you
Speaker 2: derive metrics from higher rungs, that's totally cool, totally fine.
Speaker 2: But what's a really bad idea is to directly say
Speaker 2: I'm going to send this metric right now to my
Speaker 2: back end. And people do this all the time. People
Speaker 2: think this is a good idea. It's okay, I mean,
Speaker 2: it's better than nothing, right, It's just depends on the
Speaker 2: fidelity of information you want. But the problem is there's
Speaker 2: two problems actually, But the main one is you've sent
Speaker 2: that data. Okay, you sent it to Prometheus data dog whatever.
Speaker 2: You sent that one data point. So then you look
Speaker 2: in the metrics and you say, holy cah, we're going
Speaker 2: to all these five hundreds. Why is that. I'll sit
Speaker 2: here and wait as long as you want. You're not
Speaker 2: going to be able to tell me the answer to
Speaker 2: the question unless it's blindingly obvious, unless you can say, oh, well,
Speaker 2: this other bit of data over here is like correlates
Speaker 2: with it time wise, and maybe it might be that, yeah, okay,
Speaker 2: it might be that, how do you know it's that? Well,
Speaker 2: we're having to guess. Guessing is not a strategy. Hope
Speaker 2: is not a strategy. I don't really want to be
Speaker 2: bugged by just flipping guessing. I want to know, and
Speaker 2: the only way of knowing is having traces. So the
Speaker 2: way I like to think of it is tracing is
Speaker 2: the pinnacle. Logs can be derived from traces, which is
Speaker 2: why the three rungs of ladder and everything can be
Speaker 2: derived as a metric from the two rungs above. So
Speaker 2: if you've got only logs, you don't have any nested context,
Speaker 2: but you can get metrics from logs. Fine. If you
Speaker 2: just have metrics, I would say you're not in great
Speaker 2: shape because you can't understand why without pure guessing, and
Speaker 2: it amazes me how many people push back on this
Speaker 2: idea and think just having some metrics is enough. It's
Speaker 2: nowhere near enough, not in my experience. If somebody wants
Speaker 2: to refute me and come on this podcast or have
Speaker 2: a chat with me after, I would love to listen
Speaker 2: to how metrics allow you to debug, like very very
Speaker 2: deliberately and get the exact data that you need and
Speaker 2: you can send off dimensions to metrics and then your
Speaker 2: metrics bill explodes within about five seconds, especially if it's
Speaker 2: high cardinality data like IP address is I've made that
Speaker 2: mistake before. We're going to send a dimension of IP
Speaker 2: with our metrics so that we can understand what's going on.
Speaker 2: In a week. My manager usually messages me, usually in
Speaker 2: less than a week, saying, can you you can turn
Speaker 2: that off? We've just got a data dog bill of
Speaker 2: like five grand whoopsies.
Speaker 1: I guess I do have like maybe some specific instances
Speaker 1: where metrics alone can help like identify things. And that's
Speaker 1: more where it's like the grain aler metrics are the
Speaker 1: things that you're actually looking like care about, right Like
Speaker 1: let's say, for example, like back to the Sidekick background
Speaker 1: jobs example, like if you notice like your queues piling up,
Speaker 1: and you happen to have your dashboard of metrics just
Speaker 1: looking at Q size and looking at throughput, like you
Speaker 1: can easily say, oh, like there's something blocking it and
Speaker 1: gives you kind of a point of where to look
Speaker 1: at in this specific instance or as an example, like also,
Speaker 1: you know, you can notice like there's a leak in
Speaker 1: memory by monitoring you know, your memory consumption of the
Speaker 1: app and just looking at the metrics for that and
Speaker 1: getting an alert and saying, why is the memory not
Speaker 1: stopping growing after a certain amount of time. I mean
Speaker 1: these are like, you know, very specific examples that I'm giving,
Speaker 1: but like I agree, like if you're looking for like
Speaker 1: you know, it's not going to tell you, like if
Speaker 1: your users are like back to your like token expiration,
Speaker 1: like are people having a problem with our application that
Speaker 1: we've made? Like uh, you know, and like we keep
Speaker 1: getting these, you know, client you know emails coming in
Speaker 1: like oh I can't like sign into your app, Like
Speaker 1: what's happening? You know you can just like take that
Speaker 1: and be like, oh, yeah, it's obviously the tokens like expiration, right,
Speaker 1: like it's your collect customers emails aren't going to like
Speaker 1: translate directly to that, and you're not going to know
Speaker 1: right away without having your tracing it in it in place.
Speaker 2: So I'm a few things there. Number one, you bring
Speaker 2: up a really good exception I'd forgotten to mention conveniently.
Speaker 2: If it's infrastructure stuff, if it's like memory, hard this space,
Speaker 2: all that kind of stuff, fair game for metrics. Fine.
Speaker 1: Yeah.
Speaker 2: The second thing is I'm quite hyperbolic, so I'm quite
Speaker 2: an extreme person. So when I say they're useless, I
Speaker 2: don't mean literally they're completely useless. I think of metrics
Speaker 2: as a hint, Hey there's something going on over here. Cool,
Speaker 2: that's not useless. Obviously it's useful. But then the next
Speaker 2: question is why. And if you've got a super simple system,
Speaker 2: then it's probably like the three things and you go, well,
Speaker 2: there's only three jobs and the systems so cool. And
Speaker 2: maybe you've segregated your metrics by background jobs, which is fair.
Speaker 2: You know, it gives you a place to look and
Speaker 2: it gives you a starting point. But I've yeah, yeah,
Speaker 2: they're useful in the aggregate, and they're useful at giving
Speaker 2: you a hint. But and yes, they're useful in terms
Speaker 2: of like making sure the infrastructure is still running. But
Speaker 2: I see a lot of people depending on them, and
Speaker 2: I you know, there's a guy I really respect used
Speaker 2: to work with him called Lewis Jones, and Hum and
Speaker 2: I have gone back and forth on this over LinkedIn,
Speaker 2: and he is convinced I'm wrong about this. Is like
Speaker 2: we run everything through metrics. Metrics are awesome, You're just
Speaker 2: on cloud nine if you think you can trace everything.
Speaker 2: And there's also a significant weakness with tracing as well,
Speaker 2: which is you can't trace everything unless you've got relatively
Speaker 2: low through put or even medium through put. You can
Speaker 2: you can make it work. If you trace every single
Speaker 2: request and you're doing millions of requests a day, I
Speaker 2: dread to think what your bill is going to be.
Speaker 2: So and then that's where head tracing and head sampling
Speaker 2: and tail sampling comes into it, and we can get
Speaker 2: into that if you would like.
Speaker 1: I mean, I would love to dig more into tracing
Speaker 1: in general, and maybe more of the distributed aspect of it,
Speaker 1: because I think what you're talking about is very important. Like,
Speaker 1: you know, if we're just talking about tracing through like
Speaker 1: a single request and a reels app, but it's not
Speaker 1: as useful as maybe what where tracing really comes to
Speaker 1: play is where there's multiple things that start happening, Like
Speaker 1: once you start having more than one application, and you know,
Speaker 1: the data starts trickling from one application to the other,
Speaker 1: even in sidekick example, right, if you're throwing stuff into
Speaker 1: the background, how does that data snapshot transition through the
Speaker 1: background jobs, especially if you have ones that start depending
Speaker 1: on each other, how do you then manage the queue
Speaker 1: like and the making sure that you know where it
Speaker 1: started and you know where it's going, because sometimes you
Speaker 1: can catch a problem before it starts by having the
Speaker 1: traces in play and know where it's heading, right, And
Speaker 1: so I would love to dig into those aspects like
Speaker 1: where do you like what tooling or maybe maybe we
Speaker 1: shouldn't talk about tooling specifically, but like what aspects of
Speaker 1: tracing are most important for like holistically looking at your
Speaker 1: system outside of like you know, running through your your
Speaker 1: question Like I think at this point we're beyond like
Speaker 1: having your questions of what you're trying to look at
Speaker 1: and you already know what those questions are. And where
Speaker 1: do you start like setting up tracing because I know,
Speaker 1: like a doctimity we'd use open tracing is like an
Speaker 1: open standard for tracing and observability across like platforms, languages,
Speaker 1: and things like that. Do you find that the industry
Speaker 1: standards are like heading in the right direction or like
Speaker 1: where where are the pitfalls there?
Speaker 2: Like?
Speaker 1: Because I know it's like it just introduces a lot
Speaker 1: of dependencies once, once you start to adopt a lot
Speaker 1: of these things totally.
Speaker 2: So I should say I am singing the praise of tracing,
Speaker 2: but it's a slightly utopian vision that I'm painting because
Speaker 2: ninety percent of the work I've done is with logging
Speaker 2: purely because it's simpler to get going, it's more of
Speaker 2: a known quantity, and a lot of my talks, this
Speaker 2: is why I'm not talking a lot about tracing and
Speaker 2: I'm talking about structured logging because I think structured logging
Speaker 2: gives you this kind of event based mindset that you
Speaker 2: can then start extending to tracing. And the reverse is
Speaker 2: not true, Like you can't take that event based kind
Speaker 2: of mindset into metrics because metrics is just that aggregation. Right. So,
Speaker 2: but I have, like recently, I've been doing a lot
Speaker 2: of queries in our rails up and I've been going
Speaker 2: to we use new relic. Sorry, we use data Dog
Speaker 2: at work, and I've been going to data dogs tracing
Speaker 2: interface and really trying to answer my questions there instead
Speaker 2: of in logging. So we have both tracing and logging.
Speaker 2: Our tracing is hobbled a little bit, just purely because
Speaker 2: of cost reasons, and our logging is not so hobbled.
Speaker 2: So are the standards heading in the right direction? Yes,
Speaker 2: but it's going to take a really long time to
Speaker 2: get there, It's my short answer. There is a lot
Speaker 2: of there's a lot of different ways of going about tracing.
Speaker 2: The most promising, as we all know, is open telemetry.
Speaker 2: But I mean I read some pretty harsh critiques of
Speaker 2: open telemetry. There's kind of as a topic that generally
Speaker 2: divides people. If you don't know anything about open telemetry,
Speaker 2: it sounds absolute utopia, and I got really excited when
Speaker 2: I started researching into it. The more you dig into it,
Speaker 2: the more you realize how much complexity there is to resolve,
Speaker 2: and how many challenges that project faces in order to
Speaker 2: resolve them. And so, I mean, what it's trying to
Speaker 2: resolve is thirty, maybe forty years possibly even more of
Speaker 2: legacy software, right, because that's how long logging has been around,
Speaker 2: and they're trying to aggregate all of that into one
Speaker 2: single standard. Good look. It's a very very difficult problem
Speaker 2: to solve, and they're doing an incredible job, but it's
Speaker 2: very very difficult. So they have open celemetry is where
Speaker 2: I would start with the answer to your question. Open
Speaker 2: celemetry is one hundred percent of the future. I've not
Speaker 2: seen anything that rivals it, and open tracing I believe
Speaker 2: came first and then evolved into open celemetriies my understanding.
Speaker 2: Apologies if I've got that slightly wrong, and so yeah,
Speaker 2: I think there's a few options if you're in Ruby,
Speaker 2: none of which are ideal. So the open telemetry client
Speaker 2: in Ruby is not ready for prime time. It's quite
Speaker 2: behind the current standards in open celemetry. It doesn't obey
Speaker 2: any of the latest semantic standards for example I have.
Speaker 2: I've played around with it in an example project and
Speaker 2: when it's working, it's absolutely incredible. It's next level brilliant
Speaker 2: that there are a few problems with it. It's extremely slow.
Speaker 2: So I tried to use tracing on our test suite
Speaker 2: at work using this open telemetry tracing and it just
Speaker 2: it's like I can't remember the numbers, but it really
Speaker 2: slowed down our test suite to the point where it
Speaker 2: you really just want practical to use because we were
Speaker 2: trying to measure the performance of the test suite. So
Speaker 2: you know, I could have been doing something stupid there.
Speaker 2: It's very possible that I just wasn't using it in
Speaker 2: the right way. So sorry open selemorry folks, if I've
Speaker 2: got I know. I think a lady is called Kayley
Speaker 2: who is from New Relic and she and I'm so
Speaker 2: sorry my name the names escape me. But there's a
Speaker 2: whole bunch of people in the Ruby space who are
Speaker 2: working really hard on open telemetry. But it's just that,
Speaker 2: like the open celemetry project is moving so fast. That's
Speaker 2: the other problem. So that's option number one open telemetry.
Speaker 2: You could maybe fork it and tweak it yourself. And
Speaker 2: the second option, and what we use at work is
Speaker 2: because we're using data Dog, we use data dogs trace tool,
Speaker 2: which is pretty good. But then even with tracing or logging,
Speaker 2: I feel like we're kind of maybe twenty years behind
Speaker 2: where everybody else is in programming in terms of observability
Speaker 2: because one of the questions I often have when I
Speaker 2: look at this stuff and even think about tracing, I
Speaker 2: maybe have like five six seven questions that even I
Speaker 2: can't resolve, which is what do I trace? How much
Speaker 2: detail do I trace in? How much is this going
Speaker 2: to cost me? And we're still in the stone age
Speaker 2: with a lot of this stuff, So I don't have
Speaker 2: any good answers for you in that regard. So we
Speaker 2: use the vendor tooling for tracing. I'm sure you Relic
Speaker 2: has its own version of that. In fact, I know
Speaker 2: they do, I know Century does. There are certain other
Speaker 2: providers that don't have any tracing capabilities at all, So
Speaker 2: I would say for now, the best option we have
Speaker 2: is relying on the the tracing tools. I would say, Yeah.
Speaker 1: It's funny you mentioned David Dog. We've had I have
Speaker 1: one before from dat Dog to talk about a lot
Speaker 1: of the like I think memory profiling. He works on
Speaker 1: a lot of like granular Ruby performance tooling, really interesting stuff.
Speaker 1: But yeah, I mean I would love to see maybe
Speaker 1: some more I don't know, higher level examples of like
Speaker 1: making use of open telemetry in the Ruby space in general,
Speaker 1: because I think that that level, I mean, especially with
Speaker 1: all of the solid Q like our solid trifector or whatever.
Speaker 1: Stuff that's coming around. It would be nice to see
Speaker 1: something like tracing specifically introduced the rails that that would
Speaker 1: make you know, more sense in that ecosystem, because yeah,
Speaker 1: I mean where do you where do you start? Profiling? Stuff?
Speaker 1: Is like kind of like an intro to tracing, right,
Speaker 1: like if you wanted to see like the requests. It
Speaker 1: reminds me of was a rack mini profile or tool
Speaker 1: right where you can just see a little tiny tab
Speaker 1: that says, oh, it took this number of seconds to
Speaker 1: load this particular page you wanted to get, and you
Speaker 1: can click on and expand and see, oh, well what
Speaker 1: did your application do at each step of the way
Speaker 1: and see how long each thing took?
Speaker 2: Right?
Speaker 1: And I think of that as like a trace a
Speaker 1: lot of the times, right, And it's very like useful,
Speaker 1: like even when you're just starting out to see that, right,
Speaker 1: and it helps you visualize the And so I feel
Speaker 1: like maybe that's what's missing, is a lot of like
Speaker 1: visualization aspects of all this tracing stuff, because there's something
Speaker 1: that you look at or find useful when you're starting
Speaker 1: to dig into like structuring the traces and things like that.
Speaker 2: Definitely, that's leading me up to my one of my
Speaker 2: big kind of rumps, passions whatever within the observerity space.
Speaker 2: And I don't see anybody talking about this. I feel
Speaker 2: like it's either I'm onto a really great idea or
Speaker 2: it's an unbelievably idiotic idea for some reason that I
Speaker 2: don't know about. It's usually the latter as a spoiler. Okay,
Speaker 2: So when I'm looking at traces, there's almost never enough information,
Speaker 2: almost never enough information. And this is why Charity Majors
Speaker 2: and the team at Honeycombe and there's Von Jones always
Speaker 2: talk about and have wide context awear events. That's their mantra,
Speaker 2: Wide context awear events and events. We've already talked about context,
Speaker 2: we've already talked about we haven't talked much about the wide.
Speaker 2: So wide means lots of attributes. So their take on
Speaker 2: it is add as many attributes as you can to
Speaker 2: every event and make them high cardinality attributes. What does
Speaker 2: that mean? It took me about three months to outmad
Speaker 2: around what high cardinality means. It means anything ending in
Speaker 2: an ID. There you go, there's an easy there's an
Speaker 2: easy explanation. So a request ID anything that looks sorry
Speaker 2: that was me and my microphone. Anything that looks gooid
Speaker 2: like anything that is a unique identifier for anything, so
Speaker 2: that's user ID, request ID. But anything that is a
Speaker 2: domain object. And this is the real missed opportunity I
Speaker 2: think that we have in the rails community and in
Speaker 2: observability community potentially in general, when there is when something
Speaker 2: goes wrong, even when something goes right. Let's say, let's
Speaker 2: take the Let's take the token as an example. When
Speaker 2: that token is created, the token is a domain object. Okay,
Speaker 2: it's to do with authentication, so it's not it's not
Speaker 2: really a domain object in a way. But let's say
Speaker 2: that customer is signing up for an account. The account
Speaker 2: definitely is a domain object. And if you want to
Speaker 2: understand what I mean by domain object, I just mean
Speaker 2: an object that belongs to the domain, the business domain
Speaker 2: in which you're operating. It's a business object, a domain object.
Speaker 2: Call it what you will. But when the CETO or
Speaker 2: the even better the CEO or somebody in marketing talks
Speaker 2: about this customer account, they talk about people creating accounts,
Speaker 2: they use that word account, that's your first clue that
Speaker 2: it's a really important concept in the domain. So that's
Speaker 2: what I say when I mean domain objects. I mean
Speaker 2: words that non technical people use to describe your app. Right,
Speaker 2: So they're domain objects. Why are we not adding every
Speaker 2: relevant domain object every event? We don't do it, And
Speaker 2: so what you'll see is people do this kind of
Speaker 2: half heart. Oh well, we'll add the ID to the
Speaker 2: current span or the current trace or even the current log.
Speaker 2: We'll add the ID and that's okay, that'll be enough.
Speaker 2: But you're not capturing the state of the object. Why
Speaker 2: not just take the object in this case the account,
Speaker 2: convert it into a hash and attach it to the event.
Speaker 2: Why can't we do that? Now, there's a number of
Speaker 2: reasons why we actually can't do that in some cases.
Speaker 2: If you're build in terms of the size of your event.
Speaker 2: So if you're build on data, obviously that's going to
Speaker 2: get expensive fast. But if you're build on pure events,
Speaker 2: as in your observability provider or observability tooling is saying
Speaker 2: for every x number of events or every x number
Speaker 2: of logs per month, we will charge you this much,
Speaker 2: but the size doesn't matter. Then this is a perfect
Speaker 2: use case to be taking those rich domain objects, converting
Speaker 2: them into a structured format and dumping them in in
Speaker 2: the log or the trace, and so I've kind of
Speaker 2: thought about this quite a lot, and I've come up
Speaker 2: with a few quite simple ideas that people can use
Speaker 2: starting tomorrow in their rail SAPs, not without their problems,
Speaker 2: but the first of which is, I don't know if
Speaker 2: anybody's worked with formatted so two formatted s for daytime strings.
Speaker 2: And we have this idea in Ruby, don't we have
Speaker 2: duct typing. We have an object, and really good O
Speaker 2: design is that you shouldn't understand anything about that object.
Speaker 2: You just know it's got form methods on it. And
Speaker 2: it can be an account, it can be an invoice,
Speaker 2: it can be many different things. So my approach and
Speaker 2: them testing this approach out at work at the moment,
Speaker 2: is instead of having two formatted s, have two formatted
Speaker 2: h What does that mean. It means you're going to
Speaker 2: format the domain object as a hash, and so two
Speaker 2: formatted s allows you to pass in a symbol to
Speaker 2: define the kind of format that you want. So it
Speaker 2: can be short, ordinal, long, human eyes and it will
Speaker 2: output a string. I'll put a stringified version of that
Speaker 2: date in these different formats. So my idea is, why
Speaker 2: can't we have a method on every single domain object
Speaker 2: in our rails app called two formatted H and you
Speaker 2: pass it in a format. That format could be then
Speaker 2: open telemetry, it could be any one of numbers, a
Speaker 2: short compact. And so for every trace, the way I
Speaker 2: like to think of it is, I want to into
Speaker 2: that trace add every object that's related to that, and
Speaker 2: you could format those in open telemetry format for example,
Speaker 2: or you could have a full format or a long format,
Speaker 2: whatever you want, and so that way you can say, oh,
Speaker 2: I just want to I want a representation of the
Speaker 2: account that is short and it's just got the idea
Speaker 2: and that's a totally minimal skeleton, and that's enough for me.
Speaker 2: But actually here the work I'm doing is a bit
Speaker 2: more involved. So I want to call to formatted H
Speaker 2: with full and that will give the full account, like
Speaker 2: the updates that I created, that everything about it, and
Speaker 2: then that will be sent to my logs and traces,
Speaker 2: and I now have a standardized way of observing what's
Speaker 2: going on with all the rich data of my app
Speaker 2: app state at that point, with all the relevant domain
Speaker 2: objects in it. So that's that's my dream that I'm
Speaker 2: headed towards with this GEM. So that's kind of the
Speaker 2: way I think about structuring it, and I think about
Speaker 2: the people I see people doing all this ad hoc
Speaker 2: kind of well, this is this is an idea, and
Speaker 2: then we'll call the job idea a job underscore idea, suppose,
Speaker 2: and we'll what's the account. We can call that account
Speaker 2: underscore idea. And I'd just like to think of it
Speaker 2: as imagine your domain object. So an account has a customer,
Speaker 2: a customer has some bank details. Bank details is a
Speaker 2: bad idea, but address maybe, And so we could have
Speaker 2: these different formats that load nested relationships or not. And
Speaker 2: obviously you've got to be careful about the performance problems
Speaker 2: with that. And so you will have the exact structure
Speaker 2: of your domain object in your logs, in your traces.
Speaker 2: That for me is a dream. And then every single
Speaker 2: time and account is logged, it's in the same structure. Awesome.
Speaker 2: So I know that an account is always going to
Speaker 2: have an ID, it's always going to have a whatever
Speaker 2: other attributes, account at pending status, whatever it is. And
Speaker 2: so therefore I can say, show me every trace where
Speaker 2: the account was pending boom. Yeah.
Speaker 1: I love that idea, and it does. It reminds me
Speaker 1: a little of the introduction of the you know, the
Speaker 1: new rails like you know Blogger where you can tag
Speaker 1: you know, the taglogger was it was kind of like
Speaker 1: a start to kind of this idea of Okay, capture
Speaker 1: all of these pieces with this tag and it's like
Speaker 1: almost a pseudo trace, I call it. But it does
Speaker 1: go along that formatting aspect of like, okay, format all
Speaker 1: the things like this in a specific way. And I
Speaker 1: agree that there's definitely a lot to unwind there. We'll
Speaker 1: have to have you on more if you you know,
Speaker 1: when you you know, put this together as a gem
Speaker 1: or something, because I would love to dig into that.
Speaker 2: Cool.
Speaker 1: Yeah, I mean it definitely is. I love the idea
Speaker 1: of like the domain objects and extracting those out into
Speaker 1: a formatable way that you can then trace and follow
Speaker 1: through because that that design decision has definitely missed a lot,
Speaker 1: and seeing things like packwork as an example, was a
Speaker 1: great step in the right direction, I thought, And I'd
Speaker 1: like to see more of that kind of evolve in
Speaker 1: the real's ecosystem, of abstracting the domains into their own
Speaker 1: kind of segments and then being able to format them
Speaker 1: for traceability and things like that. I think you're on
Speaker 1: to the right, You're onto a lot here, and then.
Speaker 2: I mean, the thing that I think is unbelievably ironic
Speaker 2: is all I'm talking about is convention over configuration. And
Speaker 2: is that not why we all got into rails. I
Speaker 2: know Ruby is a different thing, but Rails is all
Speaker 2: about convention over configuration, and the entire area of observability
Speaker 2: strikes me could do with a massive dollop of convention
Speaker 2: over configuration. And that's what our contemmetry are trying to do.
Speaker 2: The one last thing, and I know the time is
Speaker 2: getting on, but the one last thing I want to
Speaker 2: just say on that is the other huge opportunity is
Speaker 2: adding context to errors. So we have the exception objects
Speaker 2: in Ruby and people store strings with them, and it's like, what,
Speaker 2: how do you suppose how am I supposed to understand
Speaker 2: anything from a string? And then people try and put
Speaker 2: IDs in the strings and you're like, knows not so
Speaker 2: at work, I've made this extremely simple, basically a subclass
Speaker 2: of standard error where you can attach context. So when
Speaker 2: you create the error, you pass in structured context. So
Speaker 2: if our logs are structured, surely our error should be
Speaker 2: structured as well. Makes sense, right, So you can say
Speaker 2: this error happened, and here was the account associated with
Speaker 2: it when that error happened, and here's a user and
Speaker 2: here's this, so it gets attached into the error and
Speaker 2: then using rails is new error handling. Rails do error
Speaker 2: dot handle. If you've not used it before, look it up.
Speaker 2: It's absolutely awesome. It's one of my favorite things that
Speaker 2: they've added to RAILS recently, relatively recently the last few years.
Speaker 2: And you can basically have listeners to these events, to
Speaker 2: these errors. Byby pardon, it will catch the errors. And
Speaker 2: then the context is encapsulated in the error, so you
Speaker 2: can pass these errors around and then you can do
Speaker 2: interesting stuff with that context. And all I do is
Speaker 2: pull out all the context and send it straight into
Speaker 2: the logs. And that has absolutely changed changed the way
Speaker 2: ID bug because when therever there's an error and it
Speaker 2: has all those rich data, you just look in the
Speaker 2: rich data and you're like, oh, that was the count,
Speaker 2: that was the shopify ida, that was a product ID
Speaker 2: I've got it, and then you just look up the
Speaker 2: idea in your externals. Oh right, okay, it's outsinc whatever
Speaker 2: it is. It makes life so much easier. So that's
Speaker 2: something I'm really passionate about as well having domain objects
Speaker 2: encapsulated within errors or we've got structured errors not just
Speaker 2: structured logs.
Speaker 1: Yeah, I mean that's definitely one thing that I look
Speaker 1: for when I'm looking for, you know, installing dependencies, right,
Speaker 1: like does the gem have its own you know, base
Speaker 1: air class that it then can you know, give minta
Speaker 1: data about whatever that it's raising the air is about
Speaker 1: like more than just like a string of some air
Speaker 1: that then you have to figure out what it is
Speaker 1: like having that extra mated data that you could just
Speaker 1: because you can you can just add attributes to a class,
Speaker 1: right and say this error has these attributes like it
Speaker 1: has you know, meaning associated with the air. I think
Speaker 1: more people doing that is definitely going to be making
Speaker 1: that easier to do first of all, but yeah, and
Speaker 1: then also getting more people to take on that convention.
Speaker 1: I completely agree with you there. Yeah, I mean we
Speaker 1: are getting at time here, is there any lass you know,
Speaker 1: pieces you wanted to you know, quickly highlight or mention
Speaker 1: before we you know, move into pics.
Speaker 2: I think the main thing is if you're listening to
Speaker 2: this and anything that I'm saying is resonating, forget about
Speaker 2: the demand object stuff. That's like getting really into the
Speaker 2: nitty gritty. But coming back to the beginning, if you're
Speaker 2: frustrated by your debugging experience, if you're thinking why am
Speaker 2: I not smart enough to understand this, chances are the
Speaker 2: problem is not with you, it's with the tools. So
Speaker 2: if you improve the tools, not only do you make
Speaker 2: your life easier and better, you level up everybody around
Speaker 2: you because all the engineers can use the same tools.
Speaker 2: And that's what we've experienced at Bigger Pockets, and that
Speaker 2: culture of observability has really worked its way into our culture,
Speaker 2: so that now anybody is equipped to go into the
Speaker 2: logs and ask any question that they want. So it
Speaker 2: is a long road, but it all starts with a
Speaker 2: single step. And so if you are feeling that pain,
Speaker 2: feel free to reach out to me. And I can
Speaker 2: go through all my socials in a minute, but feel
Speaker 2: free to reach out to me, ask me any questions.
Speaker 2: I'm happy to jump on a zoom call for half
Speaker 2: an hour and help you for free. But basically, it
Speaker 2: all starts by taking very small steps towards a very
Speaker 2: specific question. Don't try and add observability because you'll still
Speaker 2: be here next Christmas. So take heed, there is hope,
Speaker 2: and if any anything that I say resonates, please feel
Speaker 2: free to reach out to me and I'll help you
Speaker 2: figure it out.
Speaker 1: That's awesome. Yeah, I also echo that sentiment of you know,
Speaker 1: tooling is so important, and you know open tracing definitely
Speaker 1: is a great, great framework, and if we can improve
Speaker 1: that in the Ruby space, that will definitely well, we'll
Speaker 1: be reaping the rewards as well. So let's move into picks. John,
Speaker 1: do you have anything that you want to share first
Speaker 1: or you want me to go?
Speaker 2: Am I limited to one pick because I have many?
Speaker 1: Okay, go ahead.
Speaker 2: Cool. So the first one is a new language, and
Speaker 2: I already I already thoroughly trounced the idea that we
Speaker 2: should be learning one programming language a year, or rather,
Speaker 2: I I just dissed it off without actually giving much justification.
Speaker 2: So I'm going to go back on what I just
Speaker 2: said and say that this language has changed the way
Speaker 2: I think pretty much forever, and it's changed the way
Speaker 2: I see Ruby and rails and just programming in general.
Speaker 2: And the language is called Unison. Now it's a very
Speaker 2: very strange, unusual language. It's maybe not that readable in places,
Speaker 2: and it's also extremely new. I mean it's been going
Speaker 2: for five or six years, but what they're trying to
Speaker 2: do is incredibly ambitious. But look it up. It's Yeah,
Speaker 2: it's an incredibly interesting language and it will expand your
Speaker 2: mind and that's what it's so what it's done for me.
Speaker 2: And so it's kind of a language that's targeted at
Speaker 2: creating programs that are just much much simpler but actually
Speaker 2: more difficult to get your head around. It's a completely
Speaker 2: new paradigm for distributed computing basically, and it's absolutely fascinating.
Speaker 2: So I would highly suggest check that out. I know
Speaker 2: that Dave Thomas at Uruko. When I spoke at Yuruko recently,
Speaker 2: he was on stage and he was championing Unison and
Speaker 2: he called it the future of programming, and I could
Speaker 2: not agree more. It's an incredible language made by some
Speaker 2: incredibly smart people. So that's number one. Number two, there
Speaker 2: is a static site builder. I've used pretty much all
Speaker 2: the static site builders on planet Earth, and this is
Speaker 2: my favorite. It's called eleven T. It's a really odd name.
Speaker 2: But I'm embarking upon this project at work that really
Speaker 2: is exciting me, which is how do you serve UI
Speaker 2: components from a dynamic app rails and melds them into
Speaker 2: a static site builder without having a pile of JavaScript
Speaker 2: that you have to weighe through. So I want to
Speaker 2: author my UI components in rails, and I want to
Speaker 2: deliver them extremely fast through a static site that's just
Speaker 2: a blog without having to run that blog on rails.
Speaker 2: So eleven T is my go to tool for doing
Speaker 2: all that stuff. It also encompass this thing called web C,
Speaker 2: which is my new favorite templating language. Yes, I know,
Speaker 2: another templating language. I promise, I promise, it's really good.
Speaker 2: It's not another retread of all these other templating languages
Speaker 2: that are very, very niche and very whatever. So web
Speaker 2: C is compatible with web components and it's a fantastic
Speaker 2: way of making HTML like components are service side rendered.
Speaker 2: And I would love to see a plug in for
Speaker 2: that come to rails because it is absolutely phenomenal. So
Speaker 2: those are my two favorite things at the moment. If
Speaker 2: anybody is trying to wrestle with you, I components in
Speaker 2: rails and trying to extract them out to rails components.
Speaker 2: Also would love to chat through that with anybody who's
Speaker 2: interested in that kind of area because I think it's
Speaker 2: yeah good, there's a potential to really break new ground,
Speaker 2: how about you.
Speaker 1: Yeah, thanks, I'll definitely be digging into some of those. Yeah.
Speaker 1: I was at in New York City the other day
Speaker 1: for the Ruby AI happy hour that they've been doing
Speaker 1: every couple months. This time they did demos and I
Speaker 1: demoed this real time podcast buddy that I've made. It's
Speaker 1: called podcast Buddy, and it just kind of like listens
Speaker 1: in the background and in real time keeps track of
Speaker 1: the topic and the discussions and some example questions worth
Speaker 1: mentioning or maybe some you know topics that transition to
Speaker 1: and it's a lot of fun. I just did it
Speaker 1: for fun, but I recently refactored it to use the
Speaker 1: async framework and shout out to Samuel Williams just phenomenal,
Speaker 1: like so well put together. The documentation is coming along.
Speaker 1: It is lacking in some areas, but I was able
Speaker 1: to just completely refactor the code so that it works
Speaker 1: with async and runs things, you know, as they come
Speaker 1: in and it's streaming the whisper you know, transcripts. It
Speaker 1: performs actions in the background just like in the same thread.
Speaker 1: All advantaged with ACYNC just I love it, So check
Speaker 1: out podcast Buddy and check out async. You can't go
Speaker 1: wrong async web socket now you can handle even web
Speaker 1: sockets asynchronously just like completely seamless hgd B two and
Speaker 1: one compatible. Love it, so check those out. And John,
Speaker 1: if people want to reach out to you on the
Speaker 1: web or just in general, how can how can they
Speaker 1: reach you?
Speaker 2: Thank you? Yeah? So I'm on LinkedIn. That's a platform
Speaker 2: I'm most active on and my LinkedIn handle is synaptip mishap,
Speaker 2: which is yeah, I really regret that, sorry everybody, but yeah.
Speaker 2: So if you just search for John Gallagher, g A
Speaker 2: doub l A g h E R and maybe rails
Speaker 2: or observability, you should be able to find me. I've
Speaker 2: got quite a cheesy photo black and white photo of
Speaker 2: me in a suit. It's a horrible photo. And I
Speaker 2: blog at Joyfulprogramming dot com. It's a substack. So is
Speaker 2: this still a blog anymore? I have no idea, but
Speaker 2: that's where I write. I'm on Twitter at synaptic missapp
Speaker 2: and my get up handle is John Gallagher or one word.
Speaker 2: So yeah, Joyful Programming is the main source of goodies
Speaker 2: for me. I've also got a fairly minimal YouTube channel
Speaker 2: called Joyful Programming. So feel free to reach out to
Speaker 2: me connection, request me, ask me any question. I would
Speaker 2: love to engage with some reuby folks about observability. Tell
Speaker 2: me your problems and I'll try and help you wherever.
Speaker 1: Reccon awesome. I love it. Keep up the great work
Speaker 1: and keep you know, shouting from the mountaintop about observability,
Speaker 1: pulling those pillows down and just focusing on the important stuff, right,
Speaker 1: I love it. So until next time, everybody, I'm Valentino.
Speaker 1: Thanks John for coming on and look forward to the
Speaker 1: next time.
Speaker 2: Thanks for having me Valentino. It's been amazing, awesome,