WEBVTT

1
00:00:04.519 --> 00:00:07.879
<v Speaker 1>Hey, everybody, Welcome to another episode of the Ruby Rouges podcast.

2
00:00:08.039 --> 00:00:10.759
<v Speaker 1>I am your host today in Valentino, Stall and we

3
00:00:10.800 --> 00:00:15.199
<v Speaker 1>are joined by a very special guest today, John Gallagher. John,

4
00:00:15.320 --> 00:00:19.039
<v Speaker 1>can you introduce yourself and tell everybody a little bit

5
00:00:19.079 --> 00:00:21.239
<v Speaker 1>about yourself from why we heard you on today?

6
00:00:21.719 --> 00:00:25.399
<v Speaker 2>Sure? Thanks for having me on. My name is John Gallaghan,

7
00:00:25.519 --> 00:00:30.879
<v Speaker 2>and I am a senior engineer. Had a company called

8
00:00:30.920 --> 00:00:35.399
<v Speaker 2>Bigger Pockets and we teach how to invest in real

9
00:00:35.560 --> 00:00:39.280
<v Speaker 2>estate based in the US, and I also run my

10
00:00:39.320 --> 00:00:42.960
<v Speaker 2>own business on the side called Joyful Programming to introduce

11
00:00:43.039 --> 00:00:45.719
<v Speaker 2>more joy to the world of programming. And I'm on

12
00:00:45.840 --> 00:00:48.479
<v Speaker 2>today to talk a bit about observability, which is one

13
00:00:48.520 --> 00:00:51.399
<v Speaker 2>of my many passions. I'm a bit of a polymath.

14
00:00:51.920 --> 00:00:54.920
<v Speaker 2>This is one of the things that is really really

15
00:00:55.280 --> 00:00:58.320
<v Speaker 2>important to me and I'm passionate about and particularly passionate

16
00:00:58.359 --> 00:01:01.679
<v Speaker 2>about introducing this into rails apps. So thanks for having

17
00:01:01.679 --> 00:01:01.880
<v Speaker 2>me on.

18
00:01:02.759 --> 00:01:04.920
<v Speaker 1>Yeah, and thank you for all the joy you're bringing

19
00:01:04.959 --> 00:01:10.599
<v Speaker 1>to people. I hope you definitely pick the right language Ruby.

20
00:01:11.159 --> 00:01:15.719
<v Speaker 1>If you're not familiar with this podcast, Ruby is a

21
00:01:15.799 --> 00:01:23.840
<v Speaker 1>very joyful experience personally, so it's very cool. I love

22
00:01:26.280 --> 00:01:29.959
<v Speaker 1>I've loved been digging into all of the observability talk

23
00:01:30.640 --> 00:01:36.599
<v Speaker 1>that you have on joyful programming, And it's a kind

24
00:01:36.599 --> 00:01:39.879
<v Speaker 1>of a very important topic that I feel is definitely

25
00:01:40.480 --> 00:01:46.680
<v Speaker 1>kind of overlooked. If you're starting up, maybe you get

26
00:01:46.840 --> 00:01:50.760
<v Speaker 1>some like you know, bug alerting or something like that

27
00:01:50.879 --> 00:01:56.000
<v Speaker 1>in place as like a standard, but kind of anything

28
00:01:56.519 --> 00:01:59.920
<v Speaker 1>performance monitoring wise is kind of like a oh no,

29
00:02:00.239 --> 00:02:03.680
<v Speaker 1>like something happened, let's look into it now. I feel

30
00:02:03.719 --> 00:02:09.879
<v Speaker 1>like it's like the typical flow of things as people

31
00:02:09.919 --> 00:02:12.520
<v Speaker 1>start up. Do you want to just give us like

32
00:02:12.520 --> 00:02:16.280
<v Speaker 1>a high level like what is observability and why should

33
00:02:16.319 --> 00:02:16.719
<v Speaker 1>we care?

34
00:02:17.879 --> 00:02:17.919
<v Speaker 2>What?

35
00:02:18.479 --> 00:02:21.159
<v Speaker 1>You know? We could drill into the details of it

36
00:02:21.159 --> 00:02:21.840
<v Speaker 1>after it realiant.

37
00:02:22.039 --> 00:02:26.280
<v Speaker 2>Well, I don't actually think anybody should care about observability,

38
00:02:26.319 --> 00:02:29.400
<v Speaker 2>and I don't care about observability as a thing because

39
00:02:29.400 --> 00:02:32.759
<v Speaker 2>it's just a means to an end, and what's the

40
00:02:32.879 --> 00:02:36.879
<v Speaker 2>actual goal. It doesn't matter how you get there, but

41
00:02:36.960 --> 00:02:40.120
<v Speaker 2>the goal is being able to number one understand your

42
00:02:40.240 --> 00:02:44.840
<v Speaker 2>rails app in production, and number two be able to

43
00:02:45.000 --> 00:02:51.280
<v Speaker 2>ask unusual questions, not questions that you've thought of a day,

44
00:02:51.599 --> 00:02:54.759
<v Speaker 2>two days, three weeks ago, because that's not really very

45
00:02:55.159 --> 00:02:59.240
<v Speaker 2>useful or interesting if we knew exactly the questions to ask.

46
00:02:59.680 --> 00:03:02.639
<v Speaker 2>In the future of our apps, everything will be easy.

47
00:03:03.479 --> 00:03:07.599
<v Speaker 2>Just be like, how many two hundreds have we had

48
00:03:07.719 --> 00:03:12.599
<v Speaker 2>in last week? Kind of boring question to ask, maybe

49
00:03:12.680 --> 00:03:16.280
<v Speaker 2>a bit useful. I find the more obvious the question,

50
00:03:16.599 --> 00:03:22.439
<v Speaker 2>the less useful it is. So observability is the practice

51
00:03:22.599 --> 00:03:28.159
<v Speaker 2>of making a black box system more transparent. So I

52
00:03:28.319 --> 00:03:30.639
<v Speaker 2>like to think of it. Imagine your entire rails app,

53
00:03:30.879 --> 00:03:34.360
<v Speaker 2>all the hosting, everything to do with that app is

54
00:03:34.400 --> 00:03:38.360
<v Speaker 2>wrapped up in an opaque black box, and somebody says,

55
00:03:38.960 --> 00:03:41.400
<v Speaker 2>how does it work? And why is this thing going wrong?

56
00:03:41.919 --> 00:03:46.479
<v Speaker 2>You would have no hope of understanding it if the

57
00:03:46.560 --> 00:03:50.719
<v Speaker 2>box is completely translucent and you can see everything, which

58
00:03:50.719 --> 00:03:54.520
<v Speaker 2>of course is completely impossible in software. But in theory,

59
00:03:54.759 --> 00:03:57.719
<v Speaker 2>you'd have this completely translucent box and you can ask

60
00:03:57.800 --> 00:04:01.280
<v Speaker 2>all these questions and you get instant that's like one

61
00:04:01.319 --> 00:04:05.080
<v Speaker 2>hundred percent observability. And of course that is absolutely impossible.

62
00:04:06.240 --> 00:04:09.639
<v Speaker 2>And so what we're trying to do with observability is

63
00:04:10.000 --> 00:04:14.000
<v Speaker 2>understand what is going on, not just when it goes wrong,

64
00:04:14.599 --> 00:04:18.079
<v Speaker 2>although that's the obvious use case. Is we have an incident.

65
00:04:18.560 --> 00:04:22.319
<v Speaker 2>The most you know, the most critical point where observability

66
00:04:22.360 --> 00:04:26.279
<v Speaker 2>comes into play is an exact scenario that I landed

67
00:04:26.319 --> 00:04:30.040
<v Speaker 2>in two weeks into a new role I had, So

68
00:04:30.079 --> 00:04:32.519
<v Speaker 2>it was two weeks in the site had gone down.

69
00:04:33.720 --> 00:04:35.639
<v Speaker 2>I was in the U, I am in the UK,

70
00:04:36.120 --> 00:04:37.720
<v Speaker 2>and my tea, the rest of my team we're in

71
00:04:37.759 --> 00:04:40.319
<v Speaker 2>the US and there were two other engineers in my

72
00:04:40.439 --> 00:04:43.160
<v Speaker 2>time zone, and all of us have been at the

73
00:04:43.160 --> 00:04:46.639
<v Speaker 2>company for a total of five weeks. So we've got

74
00:04:46.639 --> 00:04:50.519
<v Speaker 2>this app. It's down, it's on fire, and we need

75
00:04:50.560 --> 00:04:53.000
<v Speaker 2>to put the fire out. And we just the three

76
00:04:53.079 --> 00:04:56.439
<v Speaker 2>of us looked at each other. We're like, what should

77
00:04:56.439 --> 00:04:59.879
<v Speaker 2>we just restart the dinos? Yeah, so we restarted the dyno.

78
00:05:00.240 --> 00:05:03.720
<v Speaker 2>We crossed our fingers and it was pure. Look that

79
00:05:03.839 --> 00:05:07.879
<v Speaker 2>the app came back up. That is the exact opposite

80
00:05:07.920 --> 00:05:10.240
<v Speaker 2>of what we want. And we've now moved to a

81
00:05:10.360 --> 00:05:14.600
<v Speaker 2>situation where we can ask our app a whole load

82
00:05:14.639 --> 00:05:18.800
<v Speaker 2>of very unusual questions and we will get an answer

83
00:05:18.800 --> 00:05:22.639
<v Speaker 2>to that. Why are there a peak of four h

84
00:05:22.800 --> 00:05:28.160
<v Speaker 2>fours on iOS at three a m. Looks like that

85
00:05:28.519 --> 00:05:30.680
<v Speaker 2>a lot of them are coming from this IP address. Okay,

86
00:05:30.839 --> 00:05:34.160
<v Speaker 2>what's that IP address doing on the site? Okay? Interesting?

87
00:05:34.199 --> 00:05:38.279
<v Speaker 2>How many users are using that IP address? Five? So

88
00:05:38.360 --> 00:05:41.319
<v Speaker 2>only five people using and you keep So that's the

89
00:05:41.319 --> 00:05:43.959
<v Speaker 2>point of observability to me, to be able to ask

90
00:05:44.399 --> 00:05:50.360
<v Speaker 2>unusual questions that you haven't thought of already dynamically and

91
00:05:50.720 --> 00:05:54.279
<v Speaker 2>explore the space and come to some conclusions.

92
00:05:54.560 --> 00:06:00.199
<v Speaker 1>Yeah, I think that's a great overview. And your debugging

93
00:06:01.399 --> 00:06:05.079
<v Speaker 1>reminded me of that. I have the lucky experience of

94
00:06:06.000 --> 00:06:09.839
<v Speaker 1>running rails with Ruby one point eighty seven, and every

95
00:06:09.839 --> 00:06:12.160
<v Speaker 1>once in a while you just had to like give

96
00:06:12.199 --> 00:06:15.319
<v Speaker 1>the give the server a little kick because it started

97
00:06:15.360 --> 00:06:20.319
<v Speaker 1>to grow in memory size and just you know, giving

98
00:06:20.319 --> 00:06:23.120
<v Speaker 1>it a quick little flush like reset things. And you're

99
00:06:23.199 --> 00:06:24.959
<v Speaker 1>just like, oh, I guess that's how we're gonna do

100
00:06:25.000 --> 00:06:30.079
<v Speaker 1>it until we can get like some insight into what's happening, right,

101
00:06:31.040 --> 00:06:35.639
<v Speaker 1>And I think that's definitely underlines the importance of observability

102
00:06:35.680 --> 00:06:38.680
<v Speaker 1>in general, Like, uh, you know, how do you get

103
00:06:38.720 --> 00:06:42.240
<v Speaker 1>those insights to begin with? And maybe that's a great

104
00:06:42.240 --> 00:06:46.439
<v Speaker 1>starting point, Like where do you start like looking at it,

105
00:06:46.839 --> 00:06:49.959
<v Speaker 1>like adding this insights? Right, Like what's the Is there

106
00:06:50.000 --> 00:06:52.240
<v Speaker 1>a modular approach you could take or is it more

107
00:06:52.319 --> 00:06:56.319
<v Speaker 1>of like you should look at doing everything all at

108
00:06:56.319 --> 00:06:57.120
<v Speaker 1>once kind of thing?

109
00:06:57.759 --> 00:07:01.319
<v Speaker 2>You should definitely not look at everything at once as

110
00:07:01.360 --> 00:07:04.240
<v Speaker 2>I think we can all agree in software doing everything

111
00:07:04.240 --> 00:07:06.079
<v Speaker 2>all that wants to the rest be for disaster. No

112
00:07:06.120 --> 00:07:07.000
<v Speaker 2>matter what you're doing.

113
00:07:07.879 --> 00:07:10.079
<v Speaker 1>There's no vendor you could just like pay money to

114
00:07:10.199 --> 00:07:13.920
<v Speaker 1>and like you get onbility.

115
00:07:12.800 --> 00:07:16.120
<v Speaker 2>There are vendors that tell you that you can do that,

116
00:07:16.439 --> 00:07:18.240
<v Speaker 2>whether you actually kind of know, it's a different matter.

117
00:07:19.279 --> 00:07:22.920
<v Speaker 2>Spoiler alert, you can't. So I just want to back

118
00:07:22.959 --> 00:07:25.399
<v Speaker 2>up a little bit and talk about the feelings, because

119
00:07:25.399 --> 00:07:28.160
<v Speaker 2>I think it's the feelings that is where all of

120
00:07:28.160 --> 00:07:33.800
<v Speaker 2>this start for me, pon me. So I got into observability.

121
00:07:33.920 --> 00:07:37.360
<v Speaker 2>And it's funny because for the first kind of year

122
00:07:37.439 --> 00:07:39.439
<v Speaker 2>of my journey doing this, I didn't even realize I

123
00:07:39.480 --> 00:07:43.319
<v Speaker 2>was doing observability. I'd heard about this observability thing and

124
00:07:43.399 --> 00:07:46.600
<v Speaker 2>it was out there in the universe. Okay, maybe I

125
00:07:46.639 --> 00:07:49.000
<v Speaker 2>should learn that. I should learn that, and I kept

126
00:07:49.360 --> 00:07:52.480
<v Speaker 2>using this. Should I should learn this? Have loads of

127
00:07:52.480 --> 00:07:54.279
<v Speaker 2>other stuff to do. I've got loads of other things.

128
00:07:54.439 --> 00:07:56.079
<v Speaker 2>I don't know what it is. I know it comes

129
00:07:56.079 --> 00:07:58.399
<v Speaker 2>from controls here, and there's a Wikipedia page. It's really

130
00:07:58.399 --> 00:08:01.839
<v Speaker 2>complex and really confusing whatever. I've got real work to do.

131
00:08:02.439 --> 00:08:05.120
<v Speaker 2>But what I know is that he kept coming across

132
00:08:05.160 --> 00:08:09.920
<v Speaker 2>these bugs in bugs, Century air Break, choose your error

133
00:08:09.920 --> 00:08:13.600
<v Speaker 2>reporting tool. They all help you to a degree, but

134
00:08:13.639 --> 00:08:17.040
<v Speaker 2>they're not a silver bullet. And I kept coming across

135
00:08:17.079 --> 00:08:20.639
<v Speaker 2>these defects over and the story was exactly the same.

136
00:08:21.279 --> 00:08:23.480
<v Speaker 2>Come across the defect, I'd see the stack trace in

137
00:08:23.759 --> 00:08:26.839
<v Speaker 2>the error reporting tool, and I would look at it

138
00:08:26.879 --> 00:08:31.720
<v Speaker 2>and first first emotion right out of the gate, complete confusion.

139
00:08:33.200 --> 00:08:36.360
<v Speaker 2>What is going on here? No idea? So I dig

140
00:08:36.399 --> 00:08:37.960
<v Speaker 2>a little bit into the code and dig it a

141
00:08:38.000 --> 00:08:40.039
<v Speaker 2>little bit into the stack trass. So it's coming from

142
00:08:40.039 --> 00:08:43.120
<v Speaker 2>here and this thing is nil. Classic right, this thing

143
00:08:43.200 --> 00:08:45.519
<v Speaker 2>is nil? Where is it being passed in this nil?

144
00:08:46.159 --> 00:08:49.720
<v Speaker 2>I don't know. So now I'm like, well, I can't

145
00:08:49.759 --> 00:08:52.360
<v Speaker 2>just say I can't fix this, so I now have

146
00:08:52.440 --> 00:08:56.600
<v Speaker 2>to well do what exactly? I don't have any information

147
00:08:56.720 --> 00:09:00.679
<v Speaker 2>to go off. Well, I guess we'll do that. Let's

148
00:09:00.679 --> 00:09:02.960
<v Speaker 2>look at the next one. And this just kept happening,

149
00:09:03.399 --> 00:09:05.720
<v Speaker 2>and I would find myself going through all the bugs

150
00:09:05.720 --> 00:09:08.159
<v Speaker 2>in the backlog and I couldn't fix any of them.

151
00:09:08.200 --> 00:09:11.279
<v Speaker 2>And I just wasted four hours looking at things, asking

152
00:09:11.360 --> 00:09:15.159
<v Speaker 2>questions that I couldn't explain, looking at things I didn't understand,

153
00:09:16.080 --> 00:09:18.080
<v Speaker 2>And for years I thought the problem was with me.

154
00:09:19.120 --> 00:09:21.399
<v Speaker 2>I honestly thought, I'm just not smart enough. I'm not

155
00:09:21.440 --> 00:09:24.000
<v Speaker 2>a good engineer. Blah blah blah blah blah. Bug fixing

156
00:09:24.080 --> 00:09:26.440
<v Speaker 2>isn't really my thing. It's I'm just not really good

157
00:09:26.480 --> 00:09:30.519
<v Speaker 2>at it. And then, after many, many years of this,

158
00:09:31.600 --> 00:09:34.279
<v Speaker 2>I was in a company and I just got really

159
00:09:34.320 --> 00:09:37.519
<v Speaker 2>sick of this. We just released a brand new app

160
00:09:38.200 --> 00:09:41.480
<v Speaker 2>and it was it was a customer account app, and

161
00:09:41.480 --> 00:09:44.759
<v Speaker 2>we're getting all these weird bug reports, people saying how logging.

162
00:09:44.840 --> 00:09:47.879
<v Speaker 2>People kept saying, I can't reset my password. And every

163
00:09:47.879 --> 00:09:49.639
<v Speaker 2>time we did this, we would add a little bit

164
00:09:49.679 --> 00:09:52.440
<v Speaker 2>of kind of this ad hoc logging and then put

165
00:09:52.480 --> 00:09:55.320
<v Speaker 2>the bug back in the backlog, and then it would

166
00:09:55.320 --> 00:09:57.720
<v Speaker 2>come up again and come up again. And after a while,

167
00:09:57.720 --> 00:09:59.960
<v Speaker 2>I was just like, this is just this is ridicul

168
00:10:00.279 --> 00:10:02.600
<v Speaker 2>We're highly paid engineers. Is it not a better way?

169
00:10:03.120 --> 00:10:07.000
<v Speaker 2>So then I started looking into we were using Cabana

170
00:10:07.080 --> 00:10:08.759
<v Speaker 2>at the time, or rather I should say we were

171
00:10:08.759 --> 00:10:10.960
<v Speaker 2>not using Cabano at the time. Cabana was there, we

172
00:10:10.960 --> 00:10:15.559
<v Speaker 2>were paying for it, and I was like, I've heard

173
00:10:15.559 --> 00:10:17.679
<v Speaker 2>this is something to do with logging, So where do

174
00:10:17.720 --> 00:10:19.600
<v Speaker 2>we do our logging? People like Cabana, I have no

175
00:10:19.639 --> 00:10:21.720
<v Speaker 2>idea what this even is. Let's open it up, and

176
00:10:21.799 --> 00:10:25.679
<v Speaker 2>there was just all this trash, all this rubbish. I

177
00:10:25.720 --> 00:10:28.399
<v Speaker 2>was like, what's this? How is this supposed to be useful?

178
00:10:28.600 --> 00:10:30.399
<v Speaker 2>People like, oh, we don't really look at that. It's

179
00:10:30.399 --> 00:10:33.159
<v Speaker 2>not very useful. So so how do you figure out bugs?

180
00:10:33.840 --> 00:10:36.840
<v Speaker 2>And they're like, well, we just yes, we just figure

181
00:10:36.879 --> 00:10:39.320
<v Speaker 2>it out. Well, yes, but we're not figuring it out.

182
00:10:39.679 --> 00:10:42.519
<v Speaker 2>So all of this was born through frustration. And so

183
00:10:42.559 --> 00:10:44.720
<v Speaker 2>what they did back then is what I recommend everybody

184
00:10:44.759 --> 00:10:48.000
<v Speaker 2>does now to answer to your To answer your question,

185
00:10:48.039 --> 00:10:52.480
<v Speaker 2>come back to the point, John, which is, take a

186
00:10:52.639 --> 00:10:55.679
<v Speaker 2>question that you wish you knew the answer to a

187
00:10:55.879 --> 00:10:59.360
<v Speaker 2>very specific question, not why is our app not performing

188
00:10:59.399 --> 00:11:02.759
<v Speaker 2>as we want? Not as in like why do our

189
00:11:03.480 --> 00:11:06.799
<v Speaker 2>you know? A very very specific question. So take your big,

190
00:11:07.200 --> 00:11:11.399
<v Speaker 2>big question, and the time this was why are people

191
00:11:11.440 --> 00:11:14.120
<v Speaker 2>being locked out of the app? Why can they not

192
00:11:14.240 --> 00:11:17.399
<v Speaker 2>reset the password? They're clicking on this password link and

193
00:11:17.440 --> 00:11:20.840
<v Speaker 2>they're saying it's expired or it goes nowhere or it

194
00:11:20.840 --> 00:11:27.120
<v Speaker 2>doesn't work. Okay, why are those people like? Why is

195
00:11:27.159 --> 00:11:29.679
<v Speaker 2>that happening? So that's quite a general question, and you

196
00:11:29.720 --> 00:11:33.240
<v Speaker 2>want to break it down into some hypotheses. So that's

197
00:11:33.240 --> 00:11:36.639
<v Speaker 2>the first thing. I have a five step process, and

198
00:11:37.120 --> 00:11:39.399
<v Speaker 2>this is step one. I'll go through the five step

199
00:11:39.440 --> 00:11:42.639
<v Speaker 2>process in a minute. So Step one is think of

200
00:11:42.679 --> 00:11:46.799
<v Speaker 2>a specific question. So a specific question. This might case

201
00:11:46.879 --> 00:11:51.960
<v Speaker 2>might be Okay, I've got one customer here. There's many,

202
00:11:52.000 --> 00:11:56.320
<v Speaker 2>many different types of defect. So this one customer here

203
00:11:56.360 --> 00:11:58.440
<v Speaker 2>is saying it was expired. I went to the web

204
00:11:58.480 --> 00:12:03.399
<v Speaker 2>page and the said it had expired. Okay, when did

205
00:12:03.440 --> 00:12:07.320
<v Speaker 2>they click on that link? What response did the app

206
00:12:07.360 --> 00:12:13.120
<v Speaker 2>give to them? And when did the token time out? Right?

207
00:12:13.759 --> 00:12:18.279
<v Speaker 2>So those are three questions. Now, they're not going to

208
00:12:18.320 --> 00:12:21.919
<v Speaker 2>get us to the answer directly, but there are three questions,

209
00:12:22.080 --> 00:12:26.399
<v Speaker 2>very specific questions that we can add instrumentation for. So

210
00:12:26.480 --> 00:12:29.320
<v Speaker 2>I take one of those questions, when did the token

211
00:12:29.440 --> 00:12:33.440
<v Speaker 2>time out? Great question. So in order to do that,

212
00:12:33.519 --> 00:12:36.600
<v Speaker 2>we need to know when the token was created and

213
00:12:36.639 --> 00:12:39.279
<v Speaker 2>what the expire of the token. Well, this is just

214
00:12:39.360 --> 00:12:42.240
<v Speaker 2>a random example off the top of my head. So

215
00:12:42.320 --> 00:12:44.480
<v Speaker 2>you'd be like, Okay, well we need to know the

216
00:12:44.519 --> 00:12:48.919
<v Speaker 2>customer ID. We need to know the token. We don't

217
00:12:48.960 --> 00:12:51.559
<v Speaker 2>actually need to know the exact token, but we need

218
00:12:51.600 --> 00:12:55.480
<v Speaker 2>to know the customer ID. We need to know the

219
00:12:55.519 --> 00:13:00.679
<v Speaker 2>time that the token was created and the spiry time

220
00:13:00.720 --> 00:13:03.480
<v Speaker 2>of that token. Is it fifteen minutes, is it two hours? Whatever?

221
00:13:04.279 --> 00:13:08.039
<v Speaker 2>So I would then look into the code. That's the

222
00:13:08.080 --> 00:13:11.240
<v Speaker 2>next So we've done step two. Step two is define

223
00:13:11.240 --> 00:13:15.519
<v Speaker 2>the data you want to collect, user ID token expiry,

224
00:13:16.919 --> 00:13:21.159
<v Speaker 2>and an event saying the token has been created now

225
00:13:21.320 --> 00:13:24.960
<v Speaker 2>for this user ID. Okay, So that's the second step.

226
00:13:24.960 --> 00:13:27.240
<v Speaker 2>The third step is build the instrumentation to do that.

227
00:13:27.840 --> 00:13:30.919
<v Speaker 2>So whatever you have to do, maybe it's you need

228
00:13:31.000 --> 00:13:34.600
<v Speaker 2>to actually add structured logging to your entire app. I

229
00:13:34.639 --> 00:13:38.000
<v Speaker 2>don't know. Maybe it's that you've got the structured logging fine,

230
00:13:38.039 --> 00:13:41.720
<v Speaker 2>but there's nothing listening to it. Maybe maybe the tool

231
00:13:42.000 --> 00:13:44.480
<v Speaker 2>just can't actually measure what you want it to measure,

232
00:13:44.559 --> 00:13:46.240
<v Speaker 2>So maybe you need to invest in a new tool,

233
00:13:46.559 --> 00:13:49.639
<v Speaker 2>whatever it is. And then you build some code to

234
00:13:50.200 --> 00:13:55.080
<v Speaker 2>instrument just that very small piece of functionality. And then

235
00:13:55.200 --> 00:13:57.200
<v Speaker 2>once you've done that, you wait for it to deploy

236
00:13:57.480 --> 00:13:59.320
<v Speaker 2>and then you look at the graphs, you look at

237
00:13:59.320 --> 00:14:01.840
<v Speaker 2>the logs, at the charts, whatever out but you've got,

238
00:14:02.679 --> 00:14:05.480
<v Speaker 2>and what normally happens is for me, I look at

239
00:14:05.480 --> 00:14:08.279
<v Speaker 2>the charts and I say that is not what I

240
00:14:08.360 --> 00:14:11.840
<v Speaker 2>wanted at all, Actually I've misunderstood the problem. I've misunderstood

241
00:14:11.840 --> 00:14:15.320
<v Speaker 2>the data I want. Now that I see it. Ah,

242
00:14:15.759 --> 00:14:20.200
<v Speaker 2>just like you would with agility, true agility not agile,

243
00:14:20.440 --> 00:14:23.440
<v Speaker 2>because agile mean something else now, But true agility is

244
00:14:23.879 --> 00:14:25.720
<v Speaker 2>you do a little bit of work, You develop a feature,

245
00:14:25.759 --> 00:14:29.159
<v Speaker 2>You show the customer. They say, not quite right, go back,

246
00:14:29.200 --> 00:14:32.679
<v Speaker 2>adjust it closer, but still not quite right. But if

247
00:14:32.720 --> 00:14:36.159
<v Speaker 2>you ask them to describe it exactly right from the beginning,

248
00:14:36.639 --> 00:14:39.120
<v Speaker 2>it doesn't align with what they want at all. You

249
00:14:39.200 --> 00:14:42.120
<v Speaker 2>need to show them, and it's only by showing them

250
00:14:42.480 --> 00:14:45.879
<v Speaker 2>that you get feedback. And the same is true for ourselves.

251
00:14:46.759 --> 00:14:48.799
<v Speaker 2>It's only by looking at the graphs and the logs

252
00:14:49.080 --> 00:14:51.519
<v Speaker 2>that I realized that actually isn't what I wanted to

253
00:14:51.519 --> 00:14:53.960
<v Speaker 2>begin with, or it is, or I'm onto something there,

254
00:14:54.360 --> 00:14:57.320
<v Speaker 2>and so I keep. Then sort of I've used the graph,

255
00:14:57.679 --> 00:15:01.080
<v Speaker 2>maybe it was unusable, maybe I could query the parameter.

256
00:15:01.519 --> 00:15:04.120
<v Speaker 2>Maybe there's all sorts of things that might be happening there.

257
00:15:04.399 --> 00:15:08.039
<v Speaker 2>So then the last stage is improved, and so from

258
00:15:08.080 --> 00:15:09.919
<v Speaker 2>improve you can go back to the very beginning, ask

259
00:15:09.960 --> 00:15:12.159
<v Speaker 2>a different question, or maybe you just want to a

260
00:15:12.200 --> 00:15:15.320
<v Speaker 2>trade on the instrumentation a bit, deploy it again. Oh,

261
00:15:15.360 --> 00:15:17.759
<v Speaker 2>that's more like it. Okay, So now we know the

262
00:15:17.799 --> 00:15:21.879
<v Speaker 2>token expiry, what's the next question we want to ask? Well,

263
00:15:21.919 --> 00:15:24.559
<v Speaker 2>why did like, when did the user actually hit the site?

264
00:15:24.919 --> 00:15:28.600
<v Speaker 2>Was it after the token expiry or before? Hmm? Okay,

265
00:15:28.679 --> 00:15:32.519
<v Speaker 2>sounds like an obvious question. But maybe maybe it's after,

266
00:15:32.720 --> 00:15:38.360
<v Speaker 2>which would indicate the token really had expired. Oh it's before. Huh?

267
00:15:38.679 --> 00:15:41.080
<v Speaker 2>How could it be expired when it was before? Oh?

268
00:15:41.159 --> 00:15:44.559
<v Speaker 2>Hang on, what's the time zone of the token? Now

269
00:15:44.559 --> 00:15:47.440
<v Speaker 2>we're getting into it, right, so you've logged the time zone.

270
00:15:48.039 --> 00:15:51.279
<v Speaker 2>Holy how the time zone of the token is out

271
00:15:51.279 --> 00:15:53.840
<v Speaker 2>of sync with the time zone of the user? That's

272
00:15:53.879 --> 00:15:54.799
<v Speaker 2>what it is. Yeah?

273
00:15:54.840 --> 00:16:00.759
<v Speaker 1>I love that. I love that analogy of identifying the

274
00:16:00.919 --> 00:16:04.639
<v Speaker 1>use case in order to expose what to observe and

275
00:16:05.000 --> 00:16:08.639
<v Speaker 1>where to insert. You know, all of these pieces that

276
00:16:08.720 --> 00:16:11.879
<v Speaker 1>are missing, I identify them really right, not to just

277
00:16:11.919 --> 00:16:14.879
<v Speaker 1>insert them, but to identify them. I think that's very important.

278
00:16:14.960 --> 00:16:18.519
<v Speaker 1>I think in general, is like trying to identify the

279
00:16:18.559 --> 00:16:23.759
<v Speaker 1>actual use cases in order to know what you even

280
00:16:23.759 --> 00:16:26.159
<v Speaker 1>want to capture to begin with. Right, Like, Yeah, we

281
00:16:26.200 --> 00:16:29.960
<v Speaker 1>get to throw a wall of logs at a source

282
00:16:30.120 --> 00:16:34.320
<v Speaker 1>resource like Cabana, and it's not very useful. But once

283
00:16:34.360 --> 00:16:37.679
<v Speaker 1>you start to abstract the ideas and use cases and

284
00:16:37.759 --> 00:16:41.399
<v Speaker 1>how people are actually like using the thing that you've built,

285
00:16:42.159 --> 00:16:45.240
<v Speaker 1>you know, you can definitely isolate what it is that

286
00:16:45.440 --> 00:16:48.480
<v Speaker 1>you actually care about. And I think that I think

287
00:16:48.480 --> 00:16:51.279
<v Speaker 1>you're right, like that is like kind of the whole

288
00:16:51.279 --> 00:16:57.120
<v Speaker 1>importance of observability is identifying the use case and exposing

289
00:16:57.360 --> 00:17:01.240
<v Speaker 1>what you actually care about as far as all of

290
00:17:01.240 --> 00:17:03.320
<v Speaker 1>these things that are because I mean, you know, there's

291
00:17:03.879 --> 00:17:08.359
<v Speaker 1>HTTP logs, there's like all kinds of logs and information

292
00:17:08.400 --> 00:17:11.880
<v Speaker 1>available that's just like emitting all the time. Like how

293
00:17:11.920 --> 00:17:15.440
<v Speaker 1>do you know and identify you know, which are really important?

294
00:17:15.880 --> 00:17:18.599
<v Speaker 1>And I think it just depends, right, like what do

295
00:17:18.680 --> 00:17:20.559
<v Speaker 1>you you know? What are you trying to capture? So

296
00:17:20.599 --> 00:17:23.680
<v Speaker 1>it's a it's a great like step wise way to

297
00:17:23.839 --> 00:17:28.079
<v Speaker 1>just like start to figure that out, right, because yeah,

298
00:17:28.200 --> 00:17:31.640
<v Speaker 1>just depending on your role and depending on what you

299
00:17:31.680 --> 00:17:34.480
<v Speaker 1>know your responsibilities are, that could change and that could

300
00:17:34.480 --> 00:17:37.440
<v Speaker 1>be different, and your observability needs will change with that.

301
00:17:37.599 --> 00:17:42.799
<v Speaker 1>So identifying that is probably most important, I think.

302
00:17:44.319 --> 00:17:47.559
<v Speaker 2>But as with everything else, I would say, if you're

303
00:17:47.599 --> 00:17:51.400
<v Speaker 2>really not feeling any pain, don't bother, Just don't bother.

304
00:17:51.839 --> 00:17:55.880
<v Speaker 2>I'm not into kind of not really interested in telling

305
00:17:55.920 --> 00:17:58.079
<v Speaker 2>people what they should be doing or could be doing.

306
00:17:58.319 --> 00:18:01.079
<v Speaker 2>Havan goodness. Mean we hear off of that in engineering,

307
00:18:01.119 --> 00:18:04.279
<v Speaker 2>don't we. You should really learn a language every year.

308
00:18:04.319 --> 00:18:08.519
<v Speaker 2>You should be blair, you should be blair. You'm sick

309
00:18:08.559 --> 00:18:10.799
<v Speaker 2>of it, absolutely sick of all these gurus telling me

310
00:18:10.880 --> 00:18:12.599
<v Speaker 2>what to do and what I should be learning and

311
00:18:12.599 --> 00:18:16.160
<v Speaker 2>what I and very few of them talk about, well,

312
00:18:16.160 --> 00:18:20.000
<v Speaker 2>what's the benefit to me? And in order for me

313
00:18:20.119 --> 00:18:22.079
<v Speaker 2>to do anything, in order for me to change as

314
00:18:22.079 --> 00:18:24.880
<v Speaker 2>a human being in any way learn anything, I have

315
00:18:24.960 --> 00:18:27.359
<v Speaker 2>to feel the pain of it. If you're not feeling

316
00:18:27.359 --> 00:18:30.680
<v Speaker 2>the pain, don't bother. But if you are feeling the pain,

317
00:18:31.039 --> 00:18:35.920
<v Speaker 2>if deploys are really glitchy, if you keep asking for me,

318
00:18:36.039 --> 00:18:38.880
<v Speaker 2>the kicker is if I keep asking questions I don't

319
00:18:38.920 --> 00:18:45.079
<v Speaker 2>have the answer to, that's a concern. And if they're

320
00:18:45.119 --> 00:18:47.240
<v Speaker 2>just mine, oh, like why did I wake up ten

321
00:18:47.279 --> 00:18:52.119
<v Speaker 2>minutes late today? Who cares? It's not important. But if

322
00:18:52.160 --> 00:18:54.680
<v Speaker 2>the site's gone down for the fourth time this month,

323
00:18:56.039 --> 00:18:59.079
<v Speaker 2>and every time the site goes down, we lose at

324
00:18:59.160 --> 00:19:05.119
<v Speaker 2>least five grand, ten grand, maybe even more, and even worse,

325
00:19:05.279 --> 00:19:07.799
<v Speaker 2>every single type in the site does go down we

326
00:19:07.960 --> 00:19:10.000
<v Speaker 2>just kind of get it back up more by luck

327
00:19:10.119 --> 00:19:13.079
<v Speaker 2>than good judgment, this kind of feeling of, oh, we

328
00:19:13.160 --> 00:19:15.519
<v Speaker 2>kind of got away with it that time. That's okay.

329
00:19:16.039 --> 00:19:19.559
<v Speaker 2>I know there was this weird thing and it's still

330
00:19:19.599 --> 00:19:21.960
<v Speaker 2>not really figured that one out, but that's okay. We'll

331
00:19:22.000 --> 00:19:25.720
<v Speaker 2>just put it in the back blog. It's the operational risk.

332
00:19:25.799 --> 00:19:28.440
<v Speaker 2>You've got to decide are you comfortable with that operational

333
00:19:28.519 --> 00:19:32.119
<v Speaker 2>risk or not? Is it big enough? And in my experience,

334
00:19:32.839 --> 00:19:34.839
<v Speaker 2>you've kind of got to hit rock bottom with this

335
00:19:34.839 --> 00:19:36.799
<v Speaker 2>stuff as I did. There were loads and loads of

336
00:19:36.799 --> 00:19:39.839
<v Speaker 2>bugs that I could have investigated and added logging for

337
00:19:39.960 --> 00:19:42.400
<v Speaker 2>and fixed. But you know, it's pushing a boulder up

338
00:19:42.400 --> 00:19:44.519
<v Speaker 2>a hill. It's not actually worth it. And it was

339
00:19:44.640 --> 00:19:48.240
<v Speaker 2>only when it reached my threshold of pain I was like,

340
00:19:48.359 --> 00:19:50.960
<v Speaker 2>you know what, I have to do something about this now.

341
00:19:51.279 --> 00:19:54.759
<v Speaker 2>This is just ridiculous. We're professional people, were being paid

342
00:19:54.759 --> 00:19:57.839
<v Speaker 2>a lot of money and it's not working. The app

343
00:19:57.880 --> 00:19:59.799
<v Speaker 2>that we've delivered is not working, and what's more, we

344
00:19:59.839 --> 00:20:05.559
<v Speaker 2>don't know why. But also I do just want to add,

345
00:20:06.119 --> 00:20:08.279
<v Speaker 2>and this may broaden out the conversation a little bit.

346
00:20:08.599 --> 00:20:10.640
<v Speaker 2>You may want to maybe we may want to keep

347
00:20:10.680 --> 00:20:15.200
<v Speaker 2>it narrow on rails apps. But I've realized that observability

348
00:20:15.240 --> 00:20:19.720
<v Speaker 2>principles go way beyond how does our web back work.

349
00:20:20.480 --> 00:20:24.720
<v Speaker 2>It applies to any black box. So as an example,

350
00:20:24.720 --> 00:20:27.359
<v Speaker 2>a few years ago, I was working at a company

351
00:20:27.839 --> 00:20:33.839
<v Speaker 2>and their SEO wasn't great, and they just kind of

352
00:20:33.839 --> 00:20:36.440
<v Speaker 2>were like, oh, you know, well, we'll try and fix it.

353
00:20:36.519 --> 00:20:39.519
<v Speaker 2>And they had several attempts to fix it. None of

354
00:20:39.559 --> 00:20:45.279
<v Speaker 2>them really worked, and every attempt was the same. They

355
00:20:45.319 --> 00:20:48.359
<v Speaker 2>would get some expert in the expert would give us

356
00:20:48.400 --> 00:20:50.839
<v Speaker 2>a list of one hundred things to do. We would

357
00:20:50.839 --> 00:20:54.599
<v Speaker 2>do eighty of the hundred, and then nothing would really improve.

358
00:20:56.000 --> 00:20:59.039
<v Speaker 2>And then they'd be like, well, we did everything you said,

359
00:21:00.039 --> 00:21:02.079
<v Speaker 2>and then they move on to another and rin some repeat,

360
00:21:02.160 --> 00:21:06.440
<v Speaker 2>keep doing that, and then one day, within four weeks,

361
00:21:07.440 --> 00:21:11.400
<v Speaker 2>twenty percent of the site traffic disappeared, and nobody could

362
00:21:11.480 --> 00:21:16.839
<v Speaker 2>tell us why. Nobody understood why observability. Now, Google is

363
00:21:16.839 --> 00:21:20.160
<v Speaker 2>a black box, so you know you're not going to

364
00:21:20.200 --> 00:21:23.640
<v Speaker 2>be able to instrument Google. But there's lots of tools

365
00:21:23.640 --> 00:21:26.359
<v Speaker 2>that allow you to peer into the inner workings of Google,

366
00:21:27.359 --> 00:21:31.640
<v Speaker 2>Zen Rush, Screaming Frog, all these kind of tools. They are,

367
00:21:31.799 --> 00:21:36.200
<v Speaker 2>in my opinion, actually in to some degree the observability space.

368
00:21:36.559 --> 00:21:40.920
<v Speaker 2>They're not you know, everybody thinks of them as marketing tools, SERPs,

369
00:21:40.960 --> 00:21:44.640
<v Speaker 2>engine and optimization tools, whatever, whatever whatever. They're allowing you

370
00:21:44.720 --> 00:21:49.160
<v Speaker 2>to make reasoned guesses about why your searchers aren't performing

371
00:21:49.160 --> 00:21:51.720
<v Speaker 2>the way they are, and then you can actually take

372
00:21:51.759 --> 00:21:53.960
<v Speaker 2>action on that because now you have some data. Oh

373
00:21:54.000 --> 00:21:57.680
<v Speaker 2>this keyword dropped from place four to place one hundred,

374
00:21:57.920 --> 00:22:02.759
<v Speaker 2>why is that? Okay, let's try a let's try hypothesis A.

375
00:22:03.119 --> 00:22:06.559
<v Speaker 2>Put that live and see if Google will respond to that.

376
00:22:06.720 --> 00:22:10.039
<v Speaker 2>Oh and now up to you know, position eighty whatever

377
00:22:10.039 --> 00:22:14.279
<v Speaker 2>it is. So the idea of observability goes way way

378
00:22:14.319 --> 00:22:18.240
<v Speaker 2>beyond like data dog and you relic and obviously all

379
00:22:18.279 --> 00:22:21.240
<v Speaker 2>of those people in the observability space. But I see

380
00:22:21.279 --> 00:22:25.440
<v Speaker 2>it as a much much wider and much more applicable topic.

381
00:22:26.160 --> 00:22:31.160
<v Speaker 1>Yeah, I hear you there, and I'm all against. I'm

382
00:22:31.200 --> 00:22:34.359
<v Speaker 1>all also like, you know, let's not just add new

383
00:22:34.400 --> 00:22:38.920
<v Speaker 1>relic to every app that we deploy or you know

384
00:22:39.519 --> 00:22:42.799
<v Speaker 1>it is bugsnag even needed for every app. Like these

385
00:22:42.799 --> 00:22:46.480
<v Speaker 1>are questions that I ask myself too, like what value

386
00:22:46.519 --> 00:22:49.000
<v Speaker 1>are you getting from all these auxiliary services they give

387
00:22:49.039 --> 00:22:54.079
<v Speaker 1>you the observability into like just blanket things right, like

388
00:22:54.279 --> 00:22:57.400
<v Speaker 1>what at what point do you? Like stop like that

389
00:22:57.480 --> 00:23:00.440
<v Speaker 1>kind of mentality and be like, I will you know,

390
00:23:00.599 --> 00:23:03.079
<v Speaker 1>every rails app should at least be able to get

391
00:23:03.079 --> 00:23:06.880
<v Speaker 1>insight into the logs so that you can see what

392
00:23:06.920 --> 00:23:10.359
<v Speaker 1>the application is doing, Like, well, how long do you

393
00:23:10.400 --> 00:23:12.559
<v Speaker 1>capture that? Like what kind of time frame do you have?

394
00:23:12.599 --> 00:23:15.680
<v Speaker 1>Any like default standards where you're like, well, I know

395
00:23:15.839 --> 00:23:18.440
<v Speaker 1>that I'm going to need to look at this at

396
00:23:18.480 --> 00:23:22.359
<v Speaker 1>some point in the application cycle, like what are your defaults?

397
00:23:24.519 --> 00:23:33.839
<v Speaker 2>Great question, I would say if you're if you're making

398
00:23:33.880 --> 00:23:39.079
<v Speaker 2>a small app with very little traffic and it's thresholds

399
00:23:39.200 --> 00:23:41.279
<v Speaker 2>like kind ofthing. Else, if you're making a small app

400
00:23:41.319 --> 00:23:44.480
<v Speaker 2>with very little traffic. I have a client at the

401
00:23:44.480 --> 00:23:48.160
<v Speaker 2>moment I'm consulting for and I've made them an app

402
00:23:48.319 --> 00:23:52.400
<v Speaker 2>and it has maybe flipping twenty visits a day or

403
00:23:52.400 --> 00:23:56.920
<v Speaker 2>something twenty hits a day. So I installed rollbar free

404
00:23:57.039 --> 00:24:00.319
<v Speaker 2>version of roll bar. Anything goes wrong again, an outtion,

405
00:24:00.559 --> 00:24:05.599
<v Speaker 2>It's fine. The further up the stack you move, the

406
00:24:05.640 --> 00:24:11.319
<v Speaker 2>more the defaults change. For a rails app that's mission

407
00:24:11.319 --> 00:24:14.440
<v Speaker 2>critical that I'm not even going to say mission critical,

408
00:24:14.599 --> 00:24:17.799
<v Speaker 2>but just serving a decent number of hits a month

409
00:24:19.279 --> 00:24:25.240
<v Speaker 2>ten twenty thousand. I don't know. I've tried a lot

410
00:24:25.240 --> 00:24:29.119
<v Speaker 2>of observability tools and there's no one that yet that

411
00:24:29.160 --> 00:24:34.519
<v Speaker 2>I can unreservedly recommend. They all got their pros and cons.

412
00:24:36.359 --> 00:24:40.319
<v Speaker 2>Data dog is a good option if money is no object.

413
00:24:41.799 --> 00:24:44.160
<v Speaker 2>I kind of don't want to get into the tooling

414
00:24:44.160 --> 00:24:46.839
<v Speaker 2>debate because it's kind of a bit of a red herring.

415
00:24:46.920 --> 00:24:50.480
<v Speaker 2>I think in many ways there's various costs benefit trade

416
00:24:50.480 --> 00:24:52.599
<v Speaker 2>offs there, but in terms of the defaults, in terms

417
00:24:52.599 --> 00:24:56.880
<v Speaker 2>of what you observe requests, has got to be up there.

418
00:24:56.960 --> 00:25:01.240
<v Speaker 2>So every app that I I have in my care

419
00:25:01.319 --> 00:25:05.640
<v Speaker 2>of any significant size, I would always say installed Semantic Logger.

420
00:25:05.920 --> 00:25:09.400
<v Speaker 2>Semantic Logger is the best logger I've found out does

421
00:25:09.599 --> 00:25:12.880
<v Speaker 2>Jason out of the box. It's quite extensible. There are

422
00:25:12.920 --> 00:25:15.839
<v Speaker 2>many problems with it, but it's the best option that

423
00:25:15.880 --> 00:25:20.759
<v Speaker 2>we've got. So that's number one that will log every

424
00:25:20.799 --> 00:25:24.079
<v Speaker 2>like Rails already logs every request for you that will

425
00:25:24.119 --> 00:25:27.039
<v Speaker 2>form it in Jason for you. There are some notable

426
00:25:27.759 --> 00:25:31.759
<v Speaker 2>missing defaults in semantic Logger, and I'm working on a

427
00:25:31.799 --> 00:25:35.559
<v Speaker 2>gem at the moment that will add some even more

428
00:25:35.599 --> 00:25:40.000
<v Speaker 2>sensible defaults into it. So For example, I believe that

429
00:25:40.240 --> 00:25:43.440
<v Speaker 2>request headers do not get logged out of the box.

430
00:25:45.559 --> 00:25:48.640
<v Speaker 2>Certainly request body does not get logged out of the box.

431
00:25:49.119 --> 00:25:52.720
<v Speaker 2>Request headers might be the user agent doesn't get logged

432
00:25:52.720 --> 00:25:54.920
<v Speaker 2>out of the box. I mean, this is just pretty

433
00:25:54.920 --> 00:25:59.640
<v Speaker 2>basic stuff, right, But so I have a setup that

434
00:25:59.680 --> 00:26:03.720
<v Speaker 2>I use is the logs a whole load of things

435
00:26:03.720 --> 00:26:07.319
<v Speaker 2>about the requests out of the box. I liked adding

436
00:26:07.480 --> 00:26:11.079
<v Speaker 2>user ID out of the box. And it depends what

437
00:26:11.160 --> 00:26:13.440
<v Speaker 2>kind of setup you have for authentication. But at the

438
00:26:13.599 --> 00:26:17.759
<v Speaker 2>very very least, if somebody's logged in the idea of

439
00:26:17.799 --> 00:26:20.759
<v Speaker 2>them should be logged in every single request. That is

440
00:26:20.880 --> 00:26:25.279
<v Speaker 2>absolutely you know, absolutely basic stuff. A request I D

441
00:26:26.000 --> 00:26:30.680
<v Speaker 2>is also a really really useful one. I have a

442
00:26:30.680 --> 00:26:35.599
<v Speaker 2>complex relationship with logs and tracing because tracing is essentially

443
00:26:35.720 --> 00:26:39.240
<v Speaker 2>the pinnacle of observability. I hear a lot of people

444
00:26:39.279 --> 00:26:42.480
<v Speaker 2>say logging look like logging's a be all and end all.

445
00:26:42.720 --> 00:26:45.920
<v Speaker 2>Logging is a great place to start, but tracing is

446
00:26:45.960 --> 00:26:48.200
<v Speaker 2>really where it's at, And I can go into that

447
00:26:48.599 --> 00:26:51.119
<v Speaker 2>why that is in a bit. But logging is a

448
00:26:51.119 --> 00:26:55.200
<v Speaker 2>great default. Logging is a good place to start. Start

449
00:26:55.200 --> 00:27:01.000
<v Speaker 2>with semantic logger. Basically every single thing that's important in

450
00:27:01.079 --> 00:27:05.799
<v Speaker 2>any request should be logged, so that's every header. Obviously

451
00:27:05.799 --> 00:27:08.440
<v Speaker 2>you need to be careful with sensitive data and headers

452
00:27:08.799 --> 00:27:15.079
<v Speaker 2>like do your rails active. I can't remember what it's called,

453
00:27:15.119 --> 00:27:17.720
<v Speaker 2>but there's the filtering module that you can add in

454
00:27:18.400 --> 00:27:22.000
<v Speaker 2>and sometimes semantic logger doesn't give you that by defaults

455
00:27:22.039 --> 00:27:25.839
<v Speaker 2>you need to be a bit careful. A good default

456
00:27:25.880 --> 00:27:31.160
<v Speaker 2>as well is logging all background jobs. Background jobs are

457
00:27:31.240 --> 00:27:36.039
<v Speaker 2>one of the most painful areas of observability that I've experienced,

458
00:27:36.359 --> 00:27:39.680
<v Speaker 2>and we still haven't really cracked it. We have some

459
00:27:39.839 --> 00:27:43.799
<v Speaker 2>very very basic logging out of the box, and semantic

460
00:27:43.839 --> 00:27:46.720
<v Speaker 2>Logger I believe it logs the job class, the job ID,

461
00:27:47.599 --> 00:27:51.119
<v Speaker 2>and a few other things, but it doesn't log the latency,

462
00:27:51.200 --> 00:27:56.480
<v Speaker 2>which is a huge, huge missed opportunity. And it also

463
00:27:56.759 --> 00:28:00.640
<v Speaker 2>I don't believe it logs the request I d from

464
00:28:00.960 --> 00:28:04.680
<v Speaker 2>whence it was in queued. So when a job is

465
00:28:04.680 --> 00:28:09.200
<v Speaker 2>in queued, it will by default semantic Logo will trigger

466
00:28:09.240 --> 00:28:11.680
<v Speaker 2>a little entry in the logs this job is in queued,

467
00:28:11.720 --> 00:28:13.640
<v Speaker 2>and it will tell you what request it came from.

468
00:28:13.799 --> 00:28:16.039
<v Speaker 2>But on the other side, when it's picked up and

469
00:28:16.079 --> 00:28:19.839
<v Speaker 2>the job is performed, that request ID is missing. So

470
00:28:19.920 --> 00:28:21.640
<v Speaker 2>you need to kind of go into the request ID,

471
00:28:22.240 --> 00:28:24.880
<v Speaker 2>find the qued job, find the job I D and

472
00:28:24.920 --> 00:28:27.519
<v Speaker 2>then take that next leap. So I mean it's a

473
00:28:27.519 --> 00:28:31.799
<v Speaker 2>bit clunky, but it's manageable. So in short, semantic logo

474
00:28:31.839 --> 00:28:34.400
<v Speaker 2>gives you you some okay defaults out of the box,

475
00:28:34.839 --> 00:28:38.759
<v Speaker 2>but there's some really basics that it still misses. And

476
00:28:38.839 --> 00:28:44.200
<v Speaker 2>so background jobs requests those are the two really really

477
00:28:44.200 --> 00:28:47.359
<v Speaker 2>big ones to start out with. But as you can imagine,

478
00:28:47.359 --> 00:28:47.920
<v Speaker 2>there are a tunnel.

479
00:28:50.079 --> 00:28:54.480
<v Speaker 1>Yeah, you mentioned kind of some key pieces I always

480
00:28:54.519 --> 00:28:58.720
<v Speaker 1>think of with observability in general, which is like separating

481
00:28:59.440 --> 00:29:04.200
<v Speaker 1>the the pieces into their own puzzle, right, Like we

482
00:29:04.240 --> 00:29:06.839
<v Speaker 1>have logs which are kind of just like our data,

483
00:29:07.480 --> 00:29:11.000
<v Speaker 1>and then we have individual metrics that were like snapshotting

484
00:29:11.400 --> 00:29:15.599
<v Speaker 1>the logs for particular segments like traffic or a number

485
00:29:15.640 --> 00:29:17.960
<v Speaker 1>of people using it, like the number of jobs that

486
00:29:18.000 --> 00:29:22.599
<v Speaker 1>are running. And then there are traces which we could

487
00:29:22.640 --> 00:29:26.559
<v Speaker 1>dig into next. Because I have a lot of I

488
00:29:26.599 --> 00:29:28.680
<v Speaker 1>have a lot of love for all of the standards

489
00:29:28.720 --> 00:29:30.400
<v Speaker 1>that are coming out of this, with open tracing and

490
00:29:30.440 --> 00:29:33.480
<v Speaker 1>things like that. I love to dig in there. But

491
00:29:33.519 --> 00:29:36.720
<v Speaker 1>also like alerting, like you know, how does anybody know

492
00:29:36.759 --> 00:29:44.640
<v Speaker 1>that there's ever a problem Like yeah, I mean and

493
00:29:44.680 --> 00:29:47.559
<v Speaker 1>I love I love like thinking about it in these

494
00:29:47.559 --> 00:29:50.720
<v Speaker 1>separate groups and categories, because I think it also helps

495
00:29:51.240 --> 00:29:53.799
<v Speaker 1>to think about like the overarching theme. We're just like

496
00:29:54.160 --> 00:29:59.039
<v Speaker 1>getting insight but also like getting meaningful insight and like

497
00:29:59.119 --> 00:30:01.799
<v Speaker 1>when you want really like the only the only reason

498
00:30:02.000 --> 00:30:04.720
<v Speaker 1>anybody ever cares about the ability anyway is like when

499
00:30:04.720 --> 00:30:09.480
<v Speaker 1>something goes wrong or something is problematic that causes something

500
00:30:09.480 --> 00:30:11.400
<v Speaker 1>to go wrong, and do you want to either catch

501
00:30:11.400 --> 00:30:18.960
<v Speaker 1>it early or you know, try and remediate and so

502
00:30:19.119 --> 00:30:23.559
<v Speaker 1>like where do you find like, I mean, background jobs

503
00:30:23.599 --> 00:30:26.000
<v Speaker 1>are like kind of like I feel like the first

504
00:30:26.160 --> 00:30:30.359
<v Speaker 1>instance where people realize, like, oh, like we need to

505
00:30:30.359 --> 00:30:34.039
<v Speaker 1>start looking at you know, what it's doing, right, Like

506
00:30:34.359 --> 00:30:37.000
<v Speaker 1>you start throwing stuff in the background, You're like, okay, great,

507
00:30:37.039 --> 00:30:40.200
<v Speaker 1>like it's doing the work. And then you don't maybe

508
00:30:40.319 --> 00:30:43.519
<v Speaker 1>realize if you're on the same node, like well those

509
00:30:43.759 --> 00:30:47.880
<v Speaker 1>you know, slow requests can block the web requests, right

510
00:30:47.960 --> 00:30:50.319
<v Speaker 1>and then okay, well if you split those up, finally

511
00:30:50.480 --> 00:30:54.160
<v Speaker 1>got that result. But then okay, well one problematic you

512
00:30:54.200 --> 00:30:56.559
<v Speaker 1>know job can pack up a queue that it's on,

513
00:30:57.519 --> 00:31:00.720
<v Speaker 1>you know, like where do you uh.

514
00:31:02.480 --> 00:31:02.720
<v Speaker 2>To me?

515
00:31:02.880 --> 00:31:06.279
<v Speaker 1>Like the background processing aspect is like why we have

516
00:31:06.359 --> 00:31:11.359
<v Speaker 1>tracing to begin with, because it does like its concurrency, right,

517
00:31:11.400 --> 00:31:14.599
<v Speaker 1>So it's like that's where everybody like ends up hitting

518
00:31:14.640 --> 00:31:17.680
<v Speaker 1>their pitfalls as soon as you start doing things like

519
00:31:17.759 --> 00:31:20.400
<v Speaker 1>all at once, like and thinking, oh, like we just

520
00:31:20.440 --> 00:31:22.599
<v Speaker 1>throw it in the background and like process things as

521
00:31:22.599 --> 00:31:26.119
<v Speaker 1>they come and as things start to scale, it causes

522
00:31:26.160 --> 00:31:28.799
<v Speaker 1>more problems as you try and figure out timing and

523
00:31:28.799 --> 00:31:31.240
<v Speaker 1>stuff like that, like where do you find the most

524
00:31:31.240 --> 00:31:36.039
<v Speaker 1>important pieces of like making sure that you you know,

525
00:31:36.160 --> 00:31:41.599
<v Speaker 1>are capturing the right segments and the right flows you

526
00:31:41.640 --> 00:31:42.680
<v Speaker 1>know in that process.

527
00:31:43.240 --> 00:31:46.519
<v Speaker 2>Yeah, there's so many things you touched on there. I

528
00:31:46.559 --> 00:31:48.759
<v Speaker 2>want to come back to to answer your question. First

529
00:31:48.799 --> 00:31:53.880
<v Speaker 2>of all, it's the five steps that I walk through. Yeah,

530
00:31:54.079 --> 00:31:57.559
<v Speaker 2>that's the short answer is if you have a specific

531
00:31:57.680 --> 00:32:01.400
<v Speaker 2>question that you cannot answer, what we're really talking about

532
00:32:01.519 --> 00:32:04.799
<v Speaker 2>is the implementation details of how you answer that question.

533
00:32:06.000 --> 00:32:09.680
<v Speaker 2>So what question you pick determines a whole load of

534
00:32:10.359 --> 00:32:12.079
<v Speaker 2>a whole load of stuff. I can't just give you

535
00:32:12.880 --> 00:32:16.359
<v Speaker 2>a bog standard answer because it just it depends. I

536
00:32:16.400 --> 00:32:22.519
<v Speaker 2>hate saying that, but it does. So I think the

537
00:32:22.559 --> 00:32:25.640
<v Speaker 2>first question is to ask the question, figure out what

538
00:32:25.720 --> 00:32:31.240
<v Speaker 2>data is missing, and then choose the right piece to

539
00:32:31.279 --> 00:32:33.960
<v Speaker 2>add into your logs. I feel like I've maybe not

540
00:32:34.039 --> 00:32:34.960
<v Speaker 2>understood your question.

541
00:32:35.079 --> 00:32:38.839
<v Speaker 1>Maybe yeah, I mean it's more of like an open,

542
00:32:39.319 --> 00:32:44.759
<v Speaker 1>open question. I guess like when trying to think about,

543
00:32:44.880 --> 00:32:49.160
<v Speaker 1>like one of my biggest debugging pitfalls is like trying

544
00:32:49.200 --> 00:32:53.799
<v Speaker 1>to like reconstruct the state of what happened when something

545
00:32:53.799 --> 00:32:56.200
<v Speaker 1>went wrong. It's like, I feel like that's like one

546
00:32:56.279 --> 00:33:00.319
<v Speaker 1>of the most typical things. It's like, Okay, something been

547
00:33:00.519 --> 00:33:05.359
<v Speaker 1>like well, like it's the data has changed since something

548
00:33:05.359 --> 00:33:10.279
<v Speaker 1>had happened. Maybe the change resolved the issue, but like

549
00:33:10.720 --> 00:33:12.759
<v Speaker 1>you know, trying to figure out what that is and

550
00:33:13.519 --> 00:33:17.759
<v Speaker 1>going running through those questions, right, like how do you

551
00:33:17.839 --> 00:33:22.279
<v Speaker 1>think about like reconstructing data or reconstructing the state of

552
00:33:22.359 --> 00:33:24.680
<v Speaker 1>an issue? Like is that not the right way to

553
00:33:24.720 --> 00:33:27.960
<v Speaker 1>go about it? Or do you try and like do

554
00:33:28.039 --> 00:33:28.599
<v Speaker 1>something else?

555
00:33:29.359 --> 00:33:34.200
<v Speaker 2>Fantastic question. So and this gets to the route of

556
00:33:34.279 --> 00:33:40.079
<v Speaker 2>why the three pillars are complete nonsense? Okay, so the wait,

557
00:33:40.160 --> 00:33:45.599
<v Speaker 2>what are the three pillars metrics, traces, and logs? Okay, nonsense,

558
00:33:46.200 --> 00:33:49.720
<v Speaker 2>They're not three pillars. And the analogy I like to

559
00:33:49.799 --> 00:33:55.319
<v Speaker 2>use is saying that observability is three pillars, and it's traces, logs,

560
00:33:55.359 --> 00:33:59.599
<v Speaker 2>and metrics is a bit like saying programming is three pillars.

561
00:33:59.759 --> 00:34:04.200
<v Speaker 2>It's to raise integers and strings. It's the same kind

562
00:34:04.240 --> 00:34:08.800
<v Speaker 2>of deals. No, it's nothing to do with those things.

563
00:34:09.079 --> 00:34:12.039
<v Speaker 2>Well it is because you use those every day. Yes,

564
00:34:12.079 --> 00:34:17.159
<v Speaker 2>but you kind of missing the point. So thanks to

565
00:34:17.719 --> 00:34:21.199
<v Speaker 2>some amazing work by people at Honeycomb and Charity Majors

566
00:34:22.440 --> 00:34:27.159
<v Speaker 2>and reading their stuff and reading their incredible work, I've

567
00:34:27.199 --> 00:34:31.400
<v Speaker 2>realized that the metrics tracing logs and missing the point.

568
00:34:31.840 --> 00:34:36.440
<v Speaker 2>The point is we want to see events that happened

569
00:34:36.440 --> 00:34:39.719
<v Speaker 2>at some point in time, and that neatly answers your

570
00:34:39.800 --> 00:34:42.880
<v Speaker 2>question about how do you reconstruct the state of y APP?

571
00:34:43.119 --> 00:34:45.239
<v Speaker 2>I mean, the short answer is, of course you can't.

572
00:34:46.039 --> 00:34:48.320
<v Speaker 2>If you're not in an event driven to the system,

573
00:34:48.519 --> 00:34:50.480
<v Speaker 2>if you're in a crud AUP, if you're storing state

574
00:34:50.559 --> 00:34:53.159
<v Speaker 2>to the database. There is no way you can go

575
00:34:53.280 --> 00:34:56.039
<v Speaker 2>back in time and accurately recreate it. But we can

576
00:34:56.119 --> 00:34:59.559
<v Speaker 2>give it a reasonably good stab. And we can do

577
00:34:59.639 --> 00:35:05.400
<v Speaker 2>this by capturing the state of each event when it was.

578
00:35:05.599 --> 00:35:09.360
<v Speaker 2>Forget about observability tools and logging and structured logging and tracing.

579
00:35:09.480 --> 00:35:14.079
<v Speaker 2>Just now imagine if when that incident happened, let's say

580
00:35:14.199 --> 00:35:18.239
<v Speaker 2>my expired token would be would be maybe potentially a

581
00:35:18.280 --> 00:35:22.559
<v Speaker 2>good example. There are several points in that timeline that

582
00:35:22.599 --> 00:35:26.519
<v Speaker 2>we want to understand. Number one, when the token was created.

583
00:35:26.800 --> 00:35:30.840
<v Speaker 2>Number two when the user hit the website, and maybe

584
00:35:30.920 --> 00:35:33.239
<v Speaker 2>there's a third one when the account was created. Let's

585
00:35:33.280 --> 00:35:36.079
<v Speaker 2>say that. So imagine if at each of those three

586
00:35:36.159 --> 00:35:41.079
<v Speaker 2>points we had a rich event with everything related to

587
00:35:41.199 --> 00:35:44.800
<v Speaker 2>that event in it. So when the account was created,

588
00:35:45.199 --> 00:35:48.599
<v Speaker 2>we had the account i'd, the status of the account,

589
00:35:48.840 --> 00:35:52.280
<v Speaker 2>whether it's pending or not, the creation date, the customer,

590
00:35:52.480 --> 00:35:56.159
<v Speaker 2>the customer ID, blah blah blah, blah blah. And then

591
00:35:56.440 --> 00:35:59.199
<v Speaker 2>when the user visited the site, what was the request?

592
00:35:59.239 --> 00:36:01.360
<v Speaker 2>What was the request idea? What was user I D?

593
00:36:01.559 --> 00:36:04.480
<v Speaker 2>What was the anonymous user I D? Et cetera, et cetera.

594
00:36:04.880 --> 00:36:07.440
<v Speaker 2>And then when the token was created, what was the expiry?

595
00:36:07.480 --> 00:36:09.800
<v Speaker 2>What was this? What was that? What was the user

596
00:36:09.840 --> 00:36:14.400
<v Speaker 2>I D? Okay, So if we have those three events,

597
00:36:15.280 --> 00:36:19.440
<v Speaker 2>and we have enough rick data gathered with each of

598
00:36:19.480 --> 00:36:23.519
<v Speaker 2>the events, we can answer your question does that make

599
00:36:23.559 --> 00:36:25.320
<v Speaker 2>sense so far? And there's a whole load of more

600
00:36:25.360 --> 00:36:27.719
<v Speaker 2>blah blah blah, But does that make sense so far?

601
00:36:28.239 --> 00:36:30.599
<v Speaker 1>I think that you're making some great points of like

602
00:36:30.840 --> 00:36:37.079
<v Speaker 1>capturing the transactional uh like user information or users actions.

603
00:36:37.559 --> 00:36:42.400
<v Speaker 2>Yes, and also also other events that happening in the system.

604
00:36:42.639 --> 00:36:46.840
<v Speaker 2>So there's user did something, computer did something, computer and

605
00:36:46.920 --> 00:36:50.079
<v Speaker 2>cuter background job, performed a job, et cetera, et cetera.

606
00:36:50.119 --> 00:36:51.760
<v Speaker 2>So the way I think about it is everything that

607
00:36:51.840 --> 00:36:54.800
<v Speaker 2>happens in your app, whether it's initialized by the computer

608
00:36:55.079 --> 00:36:58.599
<v Speaker 2>and external data source use its basic events storming stuff

609
00:36:58.639 --> 00:37:03.000
<v Speaker 2>really that creates an event, and that event, if you

610
00:37:03.039 --> 00:37:06.519
<v Speaker 2>don't capture enough data, that is it. The data is

611
00:37:06.559 --> 00:37:09.360
<v Speaker 2>lost forever if you're not in an event and assuming

612
00:37:09.400 --> 00:37:11.400
<v Speaker 2>you're not doing event sourcing, and assuming you're not in

613
00:37:11.400 --> 00:37:14.960
<v Speaker 2>an event driven system. So to the way I think

614
00:37:15.000 --> 00:37:17.760
<v Speaker 2>about it at the most core fundamental level is whether

615
00:37:17.760 --> 00:37:20.599
<v Speaker 2>it's druck logs, traces, metrics, whatever it is, we need

616
00:37:20.639 --> 00:37:24.920
<v Speaker 2>a way of capturing those events. And more importantly, ideally

617
00:37:25.280 --> 00:37:29.000
<v Speaker 2>we need to link the events together. And this is really,

618
00:37:29.079 --> 00:37:34.119
<v Speaker 2>really really important. So if somebody create let's say somebody

619
00:37:34.159 --> 00:37:37.000
<v Speaker 2>hits our app and it creates the token, well there's

620
00:37:37.039 --> 00:37:40.280
<v Speaker 2>two parts that they hit the app. There was a

621
00:37:40.320 --> 00:37:43.320
<v Speaker 2>request to our app, and then in the call stacked

622
00:37:43.400 --> 00:37:46.920
<v Speaker 2>somewhere the token is created. Those two things are two

623
00:37:46.960 --> 00:37:51.239
<v Speaker 2>separate events, but they're nested. We want to capture that

624
00:37:51.360 --> 00:37:55.400
<v Speaker 2>causal relationship one cause the other one is a subset

625
00:37:55.440 --> 00:37:57.119
<v Speaker 2>of the other. One is a parent a child.

626
00:37:57.280 --> 00:37:57.920
<v Speaker 1>However you want to.

627
00:37:57.880 --> 00:38:03.000
<v Speaker 2>Put it without that call link, we're lost again. We

628
00:38:03.000 --> 00:38:07.840
<v Speaker 2>don't know what caused what. So there are some three

629
00:38:08.039 --> 00:38:13.599
<v Speaker 2>or four ideas here. Number one events, number two contextual

630
00:38:13.760 --> 00:38:20.159
<v Speaker 2>data with each of those events, and number three nested events.

631
00:38:20.199 --> 00:38:24.519
<v Speaker 2>If you like causal relationships between events, and with those

632
00:38:24.559 --> 00:38:28.840
<v Speaker 2>three things, you can debug any problem that you would like,

633
00:38:29.440 --> 00:38:33.719
<v Speaker 2>is my claim. And so if you just keep that

634
00:38:33.760 --> 00:38:37.440
<v Speaker 2>model in mind, let's examine traces, logs, and metrics and

635
00:38:37.480 --> 00:38:42.000
<v Speaker 2>see where they fall short. See which one meets those criteria.

636
00:38:42.639 --> 00:38:46.639
<v Speaker 2>So tracing gives us all three you can. So for

637
00:38:46.679 --> 00:38:49.199
<v Speaker 2>those of who I should explain what tracing is, because

638
00:38:49.239 --> 00:38:52.519
<v Speaker 2>I was confused about what tracing even was for absolutely years.

639
00:38:53.400 --> 00:38:58.440
<v Speaker 2>So tracing allows you to when somebody hits your app,

640
00:39:00.480 --> 00:39:04.199
<v Speaker 2>a trace is started. So there are two concepts in tracing.

641
00:39:04.280 --> 00:39:07.360
<v Speaker 2>There's traces and there are spans, and then there's the

642
00:39:07.480 --> 00:39:10.039
<v Speaker 2>data associated with spans. But let's just leave that to

643
00:39:10.039 --> 00:39:13.480
<v Speaker 2>one side. So when somebody hits your app with a request,

644
00:39:13.920 --> 00:39:17.920
<v Speaker 2>a trace is started, and so the trace would be like, okay,

645
00:39:18.320 --> 00:39:21.840
<v Speaker 2>I've started here, I am you can append any data

646
00:39:22.039 --> 00:39:26.079
<v Speaker 2>that you want to me whilst I'm open. It's like

647
00:39:26.199 --> 00:39:29.320
<v Speaker 2>opening the cupboard door and then you keep putting stuff

648
00:39:29.320 --> 00:39:31.719
<v Speaker 2>in the cupboard, and then once a cupboard doors closed,

649
00:39:31.760 --> 00:39:34.199
<v Speaker 2>you can't put any more stuff in it. Very simple analogy.

650
00:39:34.360 --> 00:39:37.280
<v Speaker 2>So we open the door, we start the trace, and

651
00:39:37.320 --> 00:39:39.880
<v Speaker 2>so it goes down to the controller level and the

652
00:39:39.880 --> 00:39:42.880
<v Speaker 2>controller says, oh, I'm going to glom on some data

653
00:39:42.960 --> 00:39:49.039
<v Speaker 2>into whatever the existing traces about the method, the post body,

654
00:39:49.119 --> 00:39:51.840
<v Speaker 2>the request blah blah blah blah blah, headers, whatever it is.

655
00:39:52.320 --> 00:39:55.480
<v Speaker 2>I'm going to glomb that onto the current trace. And

656
00:39:55.519 --> 00:39:57.800
<v Speaker 2>then we get down into maybe you've got a service object.

657
00:39:57.880 --> 00:40:00.320
<v Speaker 2>I know some people hate them. I love themlah blah

658
00:40:00.320 --> 00:40:03.599
<v Speaker 2>blah whatever. That's not the podcast about John. So you

659
00:40:03.719 --> 00:40:06.880
<v Speaker 2>get into a service object and the service object says, oh,

660
00:40:06.920 --> 00:40:09.559
<v Speaker 2>whatever is in the current trace, I want you to

661
00:40:09.599 --> 00:40:13.119
<v Speaker 2>know you hit me, and you hit me with these arguments. Cool,

662
00:40:13.239 --> 00:40:15.000
<v Speaker 2>I'm going to append that to the trace as well,

663
00:40:15.480 --> 00:40:19.000
<v Speaker 2>and then we enqure background job. That event gets added

664
00:40:19.039 --> 00:40:22.480
<v Speaker 2>onto the trace, and then even more excitingly, there's a

665
00:40:22.519 --> 00:40:25.519
<v Speaker 2>setting in open cemetry where when the job is picked

666
00:40:25.559 --> 00:40:29.599
<v Speaker 2>up and performed that the trace is kept open. And

667
00:40:29.599 --> 00:40:31.880
<v Speaker 2>there's a whole load of debate about whether this is

668
00:40:31.880 --> 00:40:33.719
<v Speaker 2>a good idea or not, but you can do it.

669
00:40:33.960 --> 00:40:38.639
<v Speaker 2>You can keep the trace open until that job is started.

670
00:40:39.159 --> 00:40:41.800
<v Speaker 2>And so the job says, ah, I've kicked off. Now

671
00:40:42.000 --> 00:40:43.960
<v Speaker 2>it gloms a whole load must maybe you make an

672
00:40:43.960 --> 00:40:46.639
<v Speaker 2>API request in the job. It gloms a whole load

673
00:40:46.679 --> 00:40:49.800
<v Speaker 2>more stuff into the trace, and then it comes all

674
00:40:49.840 --> 00:40:52.719
<v Speaker 2>the way back up the stat and you have this

675
00:40:52.840 --> 00:40:56.920
<v Speaker 2>trace with all this nested context. And when it's saying

676
00:40:56.960 --> 00:40:59.480
<v Speaker 2>I'm going to glom this data onto the trace, that's

677
00:40:59.480 --> 00:41:02.519
<v Speaker 2>called a span, and a span is nested, so you

678
00:41:02.559 --> 00:41:06.039
<v Speaker 2>can have spans nested inside spans inside span. So essentially

679
00:41:06.079 --> 00:41:08.480
<v Speaker 2>it's this big tree structure. And you might have seen

680
00:41:08.480 --> 00:41:10.880
<v Speaker 2>this before. It's the flame graph that you get in

681
00:41:11.440 --> 00:41:13.760
<v Speaker 2>data Dog and new Relic and all these kind of things.

682
00:41:13.840 --> 00:41:16.480
<v Speaker 2>And everybody looks at these things and things are really pretty,

683
00:41:16.519 --> 00:41:19.280
<v Speaker 2>and they are indeed they are. So that is the

684
00:41:20.199 --> 00:41:23.480
<v Speaker 2>that's the pinnacle of observability. In my head. Traces give

685
00:41:23.519 --> 00:41:26.719
<v Speaker 2>it us all and we can say, as you can

686
00:41:26.719 --> 00:41:29.920
<v Speaker 2>do in any of these observability tools that support tracing,

687
00:41:30.159 --> 00:41:32.800
<v Speaker 2>you can do some really cool stuff. Show me all

688
00:41:32.840 --> 00:41:36.559
<v Speaker 2>the requests that were a two hundred the enqueued a

689
00:41:36.679 --> 00:41:39.920
<v Speaker 2>job where the job lasted for more than three seconds.

690
00:41:40.679 --> 00:41:43.800
<v Speaker 2>Holy cow, now we're cooking with gas. We've got everything

691
00:41:43.800 --> 00:41:48.159
<v Speaker 2>that we need. Show me all the spans that indicated

692
00:41:48.199 --> 00:41:51.639
<v Speaker 2>anything to do with the background job where it was

693
00:41:51.639 --> 00:41:54.440
<v Speaker 2>a five hundred response but the user was logged in

694
00:41:54.639 --> 00:41:58.039
<v Speaker 2>and and so we can start to not only query

695
00:41:58.719 --> 00:42:02.280
<v Speaker 2>the spans, but query the parents of the spans. So

696
00:42:02.320 --> 00:42:04.639
<v Speaker 2>you've got all of these nested cause of relationship and

697
00:42:04.679 --> 00:42:09.119
<v Speaker 2>it gets ridiculously powerful. So that's traces cool. Let's look

698
00:42:09.159 --> 00:42:12.159
<v Speaker 2>at logs. What do logs give us? Well, it gives

699
00:42:12.239 --> 00:42:15.199
<v Speaker 2>us events. That's all logs are. Really, it's a series

700
00:42:15.239 --> 00:42:19.079
<v Speaker 2>of events that happened. Does it give us the ability

701
00:42:19.119 --> 00:42:24.000
<v Speaker 2>to nest events inside one another? Nop, Sorry, your looks out.

702
00:42:24.360 --> 00:42:27.920
<v Speaker 2>You can log causation IDs and you can link them together.

703
00:42:27.960 --> 00:42:32.239
<v Speaker 2>And obviously you can log request IDs and filter everything

704
00:42:32.280 --> 00:42:34.559
<v Speaker 2>by the request I do, but there's no concept in

705
00:42:34.599 --> 00:42:38.199
<v Speaker 2>the log of this log is nested inside this other log.

706
00:42:39.079 --> 00:42:45.719
<v Speaker 2>So that information goodbye, it's gone. Don't have it, but

707
00:42:45.920 --> 00:42:48.880
<v Speaker 2>you have the rich data in every event. Let's look

708
00:42:48.920 --> 00:42:52.159
<v Speaker 2>at metrics. What does metrics give you. Doesn't give you

709
00:42:52.480 --> 00:42:56.039
<v Speaker 2>the events, doesn't give you the nesting, and it just

710
00:42:56.039 --> 00:43:00.599
<v Speaker 2>gives you some aggregated numbers. So I don't think of

711
00:43:00.599 --> 00:43:03.800
<v Speaker 2>them as three pillars. They're three rungs of a ladder.

712
00:43:04.679 --> 00:43:09.960
<v Speaker 2>The very top rung is tracing awesome, The next rung

713
00:43:10.000 --> 00:43:14.119
<v Speaker 2>down is logs. Pretty good. Metrics are useless. Now when

714
00:43:14.159 --> 00:43:16.320
<v Speaker 2>I say metrics are useless, people get upset with me

715
00:43:16.400 --> 00:43:18.360
<v Speaker 2>and say, oh, well, I look at metrics all the

716
00:43:18.360 --> 00:43:21.920
<v Speaker 2>time to understand my app Yeah, okay, but if you

717
00:43:22.079 --> 00:43:27.199
<v Speaker 2>derive metrics from higher rungs, that's totally cool, totally fine.

718
00:43:27.840 --> 00:43:32.440
<v Speaker 2>But what's a really bad idea is to directly say

719
00:43:32.519 --> 00:43:35.599
<v Speaker 2>I'm going to send this metric right now to my

720
00:43:35.599 --> 00:43:37.800
<v Speaker 2>back end. And people do this all the time. People

721
00:43:37.840 --> 00:43:41.280
<v Speaker 2>think this is a good idea. It's okay, I mean,

722
00:43:41.320 --> 00:43:44.480
<v Speaker 2>it's better than nothing, right, It's just depends on the

723
00:43:44.480 --> 00:43:47.159
<v Speaker 2>fidelity of information you want. But the problem is there's

724
00:43:47.159 --> 00:43:49.519
<v Speaker 2>two problems actually, But the main one is you've sent

725
00:43:49.639 --> 00:43:54.199
<v Speaker 2>that data. Okay, you sent it to Prometheus data dog whatever.

726
00:43:54.400 --> 00:43:57.719
<v Speaker 2>You sent that one data point. So then you look

727
00:43:57.719 --> 00:43:59.840
<v Speaker 2>in the metrics and you say, holy cah, we're going

728
00:43:59.920 --> 00:44:03.719
<v Speaker 2>to all these five hundreds. Why is that. I'll sit

729
00:44:03.800 --> 00:44:05.800
<v Speaker 2>here and wait as long as you want. You're not

730
00:44:05.840 --> 00:44:07.679
<v Speaker 2>going to be able to tell me the answer to

731
00:44:07.760 --> 00:44:11.360
<v Speaker 2>the question unless it's blindingly obvious, unless you can say, oh, well,

732
00:44:11.400 --> 00:44:14.760
<v Speaker 2>this other bit of data over here is like correlates

733
00:44:14.800 --> 00:44:18.320
<v Speaker 2>with it time wise, and maybe it might be that, yeah, okay,

734
00:44:18.400 --> 00:44:20.320
<v Speaker 2>it might be that, how do you know it's that? Well,

735
00:44:20.599 --> 00:44:24.119
<v Speaker 2>we're having to guess. Guessing is not a strategy. Hope

736
00:44:24.199 --> 00:44:26.519
<v Speaker 2>is not a strategy. I don't really want to be

737
00:44:26.559 --> 00:44:29.599
<v Speaker 2>bugged by just flipping guessing. I want to know, and

738
00:44:29.639 --> 00:44:32.559
<v Speaker 2>the only way of knowing is having traces. So the

739
00:44:32.559 --> 00:44:34.639
<v Speaker 2>way I like to think of it is tracing is

740
00:44:34.679 --> 00:44:39.719
<v Speaker 2>the pinnacle. Logs can be derived from traces, which is

741
00:44:39.760 --> 00:44:43.039
<v Speaker 2>why the three rungs of ladder and everything can be

742
00:44:43.119 --> 00:44:46.559
<v Speaker 2>derived as a metric from the two rungs above. So

743
00:44:46.599 --> 00:44:49.960
<v Speaker 2>if you've got only logs, you don't have any nested context,

744
00:44:49.960 --> 00:44:52.599
<v Speaker 2>but you can get metrics from logs. Fine. If you

745
00:44:52.719 --> 00:44:55.840
<v Speaker 2>just have metrics, I would say you're not in great

746
00:44:55.880 --> 00:45:00.320
<v Speaker 2>shape because you can't understand why without pure guessing, and

747
00:45:00.360 --> 00:45:02.559
<v Speaker 2>it amazes me how many people push back on this

748
00:45:02.639 --> 00:45:05.719
<v Speaker 2>idea and think just having some metrics is enough. It's

749
00:45:05.719 --> 00:45:08.920
<v Speaker 2>nowhere near enough, not in my experience. If somebody wants

750
00:45:08.960 --> 00:45:11.400
<v Speaker 2>to refute me and come on this podcast or have

751
00:45:11.480 --> 00:45:13.719
<v Speaker 2>a chat with me after, I would love to listen

752
00:45:13.760 --> 00:45:18.480
<v Speaker 2>to how metrics allow you to debug, like very very

753
00:45:18.519 --> 00:45:21.280
<v Speaker 2>deliberately and get the exact data that you need and

754
00:45:21.440 --> 00:45:24.079
<v Speaker 2>you can send off dimensions to metrics and then your

755
00:45:24.400 --> 00:45:27.639
<v Speaker 2>metrics bill explodes within about five seconds, especially if it's

756
00:45:27.719 --> 00:45:30.559
<v Speaker 2>high cardinality data like IP address is I've made that

757
00:45:30.599 --> 00:45:33.639
<v Speaker 2>mistake before. We're going to send a dimension of IP

758
00:45:33.840 --> 00:45:36.079
<v Speaker 2>with our metrics so that we can understand what's going on.

759
00:45:37.280 --> 00:45:40.400
<v Speaker 2>In a week. My manager usually messages me, usually in

760
00:45:40.480 --> 00:45:42.760
<v Speaker 2>less than a week, saying, can you you can turn

761
00:45:42.800 --> 00:45:45.079
<v Speaker 2>that off? We've just got a data dog bill of

762
00:45:45.119 --> 00:45:46.920
<v Speaker 2>like five grand whoopsies.

763
00:45:48.519 --> 00:45:54.320
<v Speaker 1>I guess I do have like maybe some specific instances

764
00:45:54.360 --> 00:45:58.280
<v Speaker 1>where metrics alone can help like identify things. And that's

765
00:45:58.320 --> 00:46:01.519
<v Speaker 1>more where it's like the grain aler metrics are the

766
00:46:01.519 --> 00:46:05.079
<v Speaker 1>things that you're actually looking like care about, right Like

767
00:46:05.159 --> 00:46:08.239
<v Speaker 1>let's say, for example, like back to the Sidekick background

768
00:46:08.280 --> 00:46:12.559
<v Speaker 1>jobs example, like if you notice like your queues piling up,

769
00:46:12.599 --> 00:46:15.480
<v Speaker 1>and you happen to have your dashboard of metrics just

770
00:46:15.559 --> 00:46:19.360
<v Speaker 1>looking at Q size and looking at throughput, like you

771
00:46:19.400 --> 00:46:23.760
<v Speaker 1>can easily say, oh, like there's something blocking it and

772
00:46:24.079 --> 00:46:26.559
<v Speaker 1>gives you kind of a point of where to look

773
00:46:26.599 --> 00:46:31.880
<v Speaker 1>at in this specific instance or as an example, like also,

774
00:46:32.239 --> 00:46:36.400
<v Speaker 1>you know, you can notice like there's a leak in

775
00:46:36.480 --> 00:46:39.679
<v Speaker 1>memory by monitoring you know, your memory consumption of the

776
00:46:39.719 --> 00:46:42.360
<v Speaker 1>app and just looking at the metrics for that and

777
00:46:42.360 --> 00:46:44.800
<v Speaker 1>getting an alert and saying, why is the memory not

778
00:46:44.920 --> 00:46:49.239
<v Speaker 1>stopping growing after a certain amount of time. I mean

779
00:46:49.280 --> 00:46:53.920
<v Speaker 1>these are like, you know, very specific examples that I'm giving,

780
00:46:53.960 --> 00:46:58.239
<v Speaker 1>but like I agree, like if you're looking for like

781
00:46:58.400 --> 00:47:00.559
<v Speaker 1>you know, it's not going to tell you, like if

782
00:47:00.599 --> 00:47:04.079
<v Speaker 1>your users are like back to your like token expiration,

783
00:47:04.320 --> 00:47:07.679
<v Speaker 1>like are people having a problem with our application that

784
00:47:07.719 --> 00:47:12.199
<v Speaker 1>we've made? Like uh, you know, and like we keep

785
00:47:12.239 --> 00:47:16.920
<v Speaker 1>getting these, you know, client you know emails coming in

786
00:47:17.199 --> 00:47:20.400
<v Speaker 1>like oh I can't like sign into your app, Like

787
00:47:20.960 --> 00:47:23.880
<v Speaker 1>what's happening? You know you can just like take that

788
00:47:24.039 --> 00:47:28.559
<v Speaker 1>and be like, oh, yeah, it's obviously the tokens like expiration, right,

789
00:47:28.679 --> 00:47:31.760
<v Speaker 1>like it's your collect customers emails aren't going to like

790
00:47:31.800 --> 00:47:33.800
<v Speaker 1>translate directly to that, and you're not going to know

791
00:47:34.079 --> 00:47:37.440
<v Speaker 1>right away without having your tracing it in it in place.

792
00:47:39.440 --> 00:47:43.760
<v Speaker 2>So I'm a few things there. Number one, you bring

793
00:47:43.840 --> 00:47:47.719
<v Speaker 2>up a really good exception I'd forgotten to mention conveniently.

794
00:47:48.039 --> 00:47:51.280
<v Speaker 2>If it's infrastructure stuff, if it's like memory, hard this space,

795
00:47:51.360 --> 00:47:53.639
<v Speaker 2>all that kind of stuff, fair game for metrics. Fine.

796
00:47:53.840 --> 00:47:55.079
<v Speaker 1>Yeah.

797
00:47:55.239 --> 00:47:58.880
<v Speaker 2>The second thing is I'm quite hyperbolic, so I'm quite

798
00:47:58.920 --> 00:48:01.400
<v Speaker 2>an extreme person. So when I say they're useless, I

799
00:48:01.440 --> 00:48:05.320
<v Speaker 2>don't mean literally they're completely useless. I think of metrics

800
00:48:05.360 --> 00:48:09.920
<v Speaker 2>as a hint, Hey there's something going on over here. Cool,

801
00:48:09.960 --> 00:48:13.719
<v Speaker 2>that's not useless. Obviously it's useful. But then the next

802
00:48:13.800 --> 00:48:17.960
<v Speaker 2>question is why. And if you've got a super simple system,

803
00:48:18.119 --> 00:48:20.880
<v Speaker 2>then it's probably like the three things and you go, well,

804
00:48:20.920 --> 00:48:24.320
<v Speaker 2>there's only three jobs and the systems so cool. And

805
00:48:24.360 --> 00:48:27.760
<v Speaker 2>maybe you've segregated your metrics by background jobs, which is fair.

806
00:48:27.960 --> 00:48:29.639
<v Speaker 2>You know, it gives you a place to look and

807
00:48:29.679 --> 00:48:35.320
<v Speaker 2>it gives you a starting point. But I've yeah, yeah,

808
00:48:35.679 --> 00:48:39.079
<v Speaker 2>they're useful in the aggregate, and they're useful at giving

809
00:48:39.119 --> 00:48:43.519
<v Speaker 2>you a hint. But and yes, they're useful in terms

810
00:48:43.559 --> 00:48:46.400
<v Speaker 2>of like making sure the infrastructure is still running. But

811
00:48:47.519 --> 00:48:50.400
<v Speaker 2>I see a lot of people depending on them, and

812
00:48:50.440 --> 00:48:52.920
<v Speaker 2>I you know, there's a guy I really respect used

813
00:48:52.920 --> 00:48:55.679
<v Speaker 2>to work with him called Lewis Jones, and Hum and

814
00:48:55.679 --> 00:48:58.079
<v Speaker 2>I have gone back and forth on this over LinkedIn,

815
00:48:58.440 --> 00:49:00.559
<v Speaker 2>and he is convinced I'm wrong about this. Is like

816
00:49:00.760 --> 00:49:05.119
<v Speaker 2>we run everything through metrics. Metrics are awesome, You're just

817
00:49:05.239 --> 00:49:07.360
<v Speaker 2>on cloud nine if you think you can trace everything.

818
00:49:07.480 --> 00:49:10.400
<v Speaker 2>And there's also a significant weakness with tracing as well,

819
00:49:10.559 --> 00:49:14.639
<v Speaker 2>which is you can't trace everything unless you've got relatively

820
00:49:14.679 --> 00:49:19.679
<v Speaker 2>low through put or even medium through put. You can

821
00:49:19.840 --> 00:49:21.880
<v Speaker 2>you can make it work. If you trace every single

822
00:49:21.920 --> 00:49:25.239
<v Speaker 2>request and you're doing millions of requests a day, I

823
00:49:25.320 --> 00:49:27.360
<v Speaker 2>dread to think what your bill is going to be.

824
00:49:27.679 --> 00:49:31.719
<v Speaker 2>So and then that's where head tracing and head sampling

825
00:49:31.760 --> 00:49:33.679
<v Speaker 2>and tail sampling comes into it, and we can get

826
00:49:33.760 --> 00:49:34.679
<v Speaker 2>into that if you would like.

827
00:49:35.920 --> 00:49:37.920
<v Speaker 1>I mean, I would love to dig more into tracing

828
00:49:37.920 --> 00:49:40.960
<v Speaker 1>in general, and maybe more of the distributed aspect of it,

829
00:49:41.039 --> 00:49:43.760
<v Speaker 1>because I think what you're talking about is very important. Like,

830
00:49:44.920 --> 00:49:47.599
<v Speaker 1>you know, if we're just talking about tracing through like

831
00:49:47.880 --> 00:49:50.519
<v Speaker 1>a single request and a reels app, but it's not

832
00:49:50.880 --> 00:49:55.039
<v Speaker 1>as useful as maybe what where tracing really comes to

833
00:49:55.159 --> 00:49:58.159
<v Speaker 1>play is where there's multiple things that start happening, Like

834
00:49:58.440 --> 00:50:01.840
<v Speaker 1>once you start having more than one application, and you know,

835
00:50:01.960 --> 00:50:04.800
<v Speaker 1>the data starts trickling from one application to the other,

836
00:50:05.519 --> 00:50:08.360
<v Speaker 1>even in sidekick example, right, if you're throwing stuff into

837
00:50:08.400 --> 00:50:11.880
<v Speaker 1>the background, how does that data snapshot transition through the

838
00:50:11.920 --> 00:50:15.079
<v Speaker 1>background jobs, especially if you have ones that start depending

839
00:50:15.079 --> 00:50:17.119
<v Speaker 1>on each other, how do you then manage the queue

840
00:50:17.199 --> 00:50:19.440
<v Speaker 1>like and the making sure that you know where it

841
00:50:19.599 --> 00:50:23.280
<v Speaker 1>started and you know where it's going, because sometimes you

842
00:50:23.320 --> 00:50:27.159
<v Speaker 1>can catch a problem before it starts by having the

843
00:50:27.199 --> 00:50:31.079
<v Speaker 1>traces in play and know where it's heading, right, And

844
00:50:31.159 --> 00:50:34.320
<v Speaker 1>so I would love to dig into those aspects like

845
00:50:34.360 --> 00:50:37.440
<v Speaker 1>where do you like what tooling or maybe maybe we

846
00:50:37.440 --> 00:50:41.360
<v Speaker 1>shouldn't talk about tooling specifically, but like what aspects of

847
00:50:41.599 --> 00:50:45.760
<v Speaker 1>tracing are most important for like holistically looking at your

848
00:50:45.760 --> 00:50:51.199
<v Speaker 1>system outside of like you know, running through your your

849
00:50:51.280 --> 00:50:53.840
<v Speaker 1>question Like I think at this point we're beyond like

850
00:50:54.159 --> 00:50:56.280
<v Speaker 1>having your questions of what you're trying to look at

851
00:50:56.360 --> 00:50:59.760
<v Speaker 1>and you already know what those questions are. And where

852
00:50:59.800 --> 00:51:03.920
<v Speaker 1>do you start like setting up tracing because I know,

853
00:51:04.159 --> 00:51:07.239
<v Speaker 1>like a doctimity we'd use open tracing is like an

854
00:51:07.280 --> 00:51:13.119
<v Speaker 1>open standard for tracing and observability across like platforms, languages,

855
00:51:13.159 --> 00:51:16.719
<v Speaker 1>and things like that. Do you find that the industry

856
00:51:17.039 --> 00:51:19.679
<v Speaker 1>standards are like heading in the right direction or like

857
00:51:19.760 --> 00:51:21.679
<v Speaker 1>where where are the pitfalls there?

858
00:51:21.719 --> 00:51:21.800
<v Speaker 2>Like?

859
00:51:22.559 --> 00:51:24.920
<v Speaker 1>Because I know it's like it just introduces a lot

860
00:51:25.679 --> 00:51:28.679
<v Speaker 1>of dependencies once, once you start to adopt a lot

861
00:51:28.679 --> 00:51:30.039
<v Speaker 1>of these things totally.

862
00:51:31.239 --> 00:51:35.239
<v Speaker 2>So I should say I am singing the praise of tracing,

863
00:51:36.079 --> 00:51:41.159
<v Speaker 2>but it's a slightly utopian vision that I'm painting because

864
00:51:42.039 --> 00:51:45.119
<v Speaker 2>ninety percent of the work I've done is with logging

865
00:51:45.480 --> 00:51:48.960
<v Speaker 2>purely because it's simpler to get going, it's more of

866
00:51:49.000 --> 00:51:51.760
<v Speaker 2>a known quantity, and a lot of my talks, this

867
00:51:51.800 --> 00:51:54.440
<v Speaker 2>is why I'm not talking a lot about tracing and

868
00:51:54.480 --> 00:51:57.679
<v Speaker 2>I'm talking about structured logging because I think structured logging

869
00:51:57.719 --> 00:52:01.199
<v Speaker 2>gives you this kind of event based mindset that you

870
00:52:01.239 --> 00:52:05.199
<v Speaker 2>can then start extending to tracing. And the reverse is

871
00:52:05.239 --> 00:52:08.119
<v Speaker 2>not true, Like you can't take that event based kind

872
00:52:08.159 --> 00:52:12.039
<v Speaker 2>of mindset into metrics because metrics is just that aggregation. Right. So,

873
00:52:13.119 --> 00:52:17.199
<v Speaker 2>but I have, like recently, I've been doing a lot

874
00:52:17.239 --> 00:52:20.199
<v Speaker 2>of queries in our rails up and I've been going

875
00:52:20.239 --> 00:52:23.119
<v Speaker 2>to we use new relic. Sorry, we use data Dog

876
00:52:23.159 --> 00:52:25.800
<v Speaker 2>at work, and I've been going to data dogs tracing

877
00:52:26.760 --> 00:52:30.920
<v Speaker 2>interface and really trying to answer my questions there instead

878
00:52:30.960 --> 00:52:35.159
<v Speaker 2>of in logging. So we have both tracing and logging.

879
00:52:35.800 --> 00:52:40.159
<v Speaker 2>Our tracing is hobbled a little bit, just purely because

880
00:52:40.159 --> 00:52:44.760
<v Speaker 2>of cost reasons, and our logging is not so hobbled.

881
00:52:46.599 --> 00:52:50.480
<v Speaker 2>So are the standards heading in the right direction? Yes,

882
00:52:51.400 --> 00:52:53.599
<v Speaker 2>but it's going to take a really long time to

883
00:52:53.639 --> 00:52:59.199
<v Speaker 2>get there, It's my short answer. There is a lot

884
00:52:59.320 --> 00:53:02.639
<v Speaker 2>of there's a lot of different ways of going about tracing.

885
00:53:02.920 --> 00:53:05.480
<v Speaker 2>The most promising, as we all know, is open telemetry.

886
00:53:06.039 --> 00:53:11.239
<v Speaker 2>But I mean I read some pretty harsh critiques of

887
00:53:11.360 --> 00:53:14.880
<v Speaker 2>open telemetry. There's kind of as a topic that generally

888
00:53:14.960 --> 00:53:19.599
<v Speaker 2>divides people. If you don't know anything about open telemetry,

889
00:53:19.599 --> 00:53:23.360
<v Speaker 2>it sounds absolute utopia, and I got really excited when

890
00:53:23.360 --> 00:53:26.039
<v Speaker 2>I started researching into it. The more you dig into it,

891
00:53:26.119 --> 00:53:30.400
<v Speaker 2>the more you realize how much complexity there is to resolve,

892
00:53:30.840 --> 00:53:34.760
<v Speaker 2>and how many challenges that project faces in order to

893
00:53:34.800 --> 00:53:38.880
<v Speaker 2>resolve them. And so, I mean, what it's trying to

894
00:53:38.960 --> 00:53:44.119
<v Speaker 2>resolve is thirty, maybe forty years possibly even more of

895
00:53:44.320 --> 00:53:47.800
<v Speaker 2>legacy software, right, because that's how long logging has been around,

896
00:53:48.320 --> 00:53:50.760
<v Speaker 2>and they're trying to aggregate all of that into one

897
00:53:50.840 --> 00:53:54.320
<v Speaker 2>single standard. Good look. It's a very very difficult problem

898
00:53:54.400 --> 00:53:58.400
<v Speaker 2>to solve, and they're doing an incredible job, but it's

899
00:53:59.039 --> 00:54:03.000
<v Speaker 2>very very difficult. So they have open celemetry is where

900
00:54:03.000 --> 00:54:05.199
<v Speaker 2>I would start with the answer to your question. Open

901
00:54:05.199 --> 00:54:07.199
<v Speaker 2>celemetry is one hundred percent of the future. I've not

902
00:54:07.239 --> 00:54:10.519
<v Speaker 2>seen anything that rivals it, and open tracing I believe

903
00:54:10.599 --> 00:54:16.559
<v Speaker 2>came first and then evolved into open celemetriies my understanding.

904
00:54:17.239 --> 00:54:21.920
<v Speaker 2>Apologies if I've got that slightly wrong, and so yeah,

905
00:54:22.000 --> 00:54:25.280
<v Speaker 2>I think there's a few options if you're in Ruby,

906
00:54:25.559 --> 00:54:29.440
<v Speaker 2>none of which are ideal. So the open telemetry client

907
00:54:29.480 --> 00:54:34.800
<v Speaker 2>in Ruby is not ready for prime time. It's quite

908
00:54:34.920 --> 00:54:39.159
<v Speaker 2>behind the current standards in open celemetry. It doesn't obey

909
00:54:39.199 --> 00:54:43.079
<v Speaker 2>any of the latest semantic standards for example I have.

910
00:54:43.719 --> 00:54:46.880
<v Speaker 2>I've played around with it in an example project and

911
00:54:47.719 --> 00:54:52.199
<v Speaker 2>when it's working, it's absolutely incredible. It's next level brilliant

912
00:54:52.760 --> 00:54:56.199
<v Speaker 2>that there are a few problems with it. It's extremely slow.

913
00:54:56.280 --> 00:54:58.360
<v Speaker 2>So I tried to use tracing on our test suite

914
00:54:58.360 --> 00:55:02.159
<v Speaker 2>at work using this open telemetry tracing and it just

915
00:55:02.360 --> 00:55:04.800
<v Speaker 2>it's like I can't remember the numbers, but it really

916
00:55:04.880 --> 00:55:06.920
<v Speaker 2>slowed down our test suite to the point where it

917
00:55:07.039 --> 00:55:09.360
<v Speaker 2>you really just want practical to use because we were

918
00:55:09.360 --> 00:55:11.360
<v Speaker 2>trying to measure the performance of the test suite. So

919
00:55:12.119 --> 00:55:15.559
<v Speaker 2>you know, I could have been doing something stupid there.

920
00:55:15.639 --> 00:55:18.119
<v Speaker 2>It's very possible that I just wasn't using it in

921
00:55:18.119 --> 00:55:21.000
<v Speaker 2>the right way. So sorry open selemorry folks, if I've

922
00:55:21.719 --> 00:55:26.800
<v Speaker 2>got I know. I think a lady is called Kayley

923
00:55:27.599 --> 00:55:33.320
<v Speaker 2>who is from New Relic and she and I'm so

924
00:55:33.360 --> 00:55:37.760
<v Speaker 2>sorry my name the names escape me. But there's a

925
00:55:37.800 --> 00:55:40.039
<v Speaker 2>whole bunch of people in the Ruby space who are

926
00:55:40.039 --> 00:55:43.960
<v Speaker 2>working really hard on open telemetry. But it's just that,

927
00:55:44.199 --> 00:55:47.000
<v Speaker 2>like the open celemetry project is moving so fast. That's

928
00:55:47.000 --> 00:55:50.800
<v Speaker 2>the other problem. So that's option number one open telemetry.

929
00:55:51.079 --> 00:55:54.599
<v Speaker 2>You could maybe fork it and tweak it yourself. And

930
00:55:55.039 --> 00:55:57.199
<v Speaker 2>the second option, and what we use at work is

931
00:55:57.400 --> 00:56:00.559
<v Speaker 2>because we're using data Dog, we use data dogs trace tool,

932
00:56:01.079 --> 00:56:07.320
<v Speaker 2>which is pretty good. But then even with tracing or logging,

933
00:56:07.760 --> 00:56:12.000
<v Speaker 2>I feel like we're kind of maybe twenty years behind

934
00:56:12.000 --> 00:56:15.079
<v Speaker 2>where everybody else is in programming in terms of observability

935
00:56:15.639 --> 00:56:19.280
<v Speaker 2>because one of the questions I often have when I

936
00:56:19.320 --> 00:56:22.280
<v Speaker 2>look at this stuff and even think about tracing, I

937
00:56:22.360 --> 00:56:25.559
<v Speaker 2>maybe have like five six seven questions that even I

938
00:56:25.599 --> 00:56:28.920
<v Speaker 2>can't resolve, which is what do I trace? How much

939
00:56:28.960 --> 00:56:31.039
<v Speaker 2>detail do I trace in? How much is this going

940
00:56:31.119 --> 00:56:33.920
<v Speaker 2>to cost me? And we're still in the stone age

941
00:56:33.960 --> 00:56:35.800
<v Speaker 2>with a lot of this stuff, So I don't have

942
00:56:35.840 --> 00:56:38.960
<v Speaker 2>any good answers for you in that regard. So we

943
00:56:39.119 --> 00:56:44.920
<v Speaker 2>use the vendor tooling for tracing. I'm sure you Relic

944
00:56:45.039 --> 00:56:47.119
<v Speaker 2>has its own version of that. In fact, I know

945
00:56:47.199 --> 00:56:51.480
<v Speaker 2>they do, I know Century does. There are certain other

946
00:56:51.519 --> 00:56:56.000
<v Speaker 2>providers that don't have any tracing capabilities at all, So

947
00:56:56.639 --> 00:56:59.000
<v Speaker 2>I would say for now, the best option we have

948
00:56:59.199 --> 00:57:03.000
<v Speaker 2>is relying on the the tracing tools. I would say, Yeah.

949
00:57:02.800 --> 00:57:04.920
<v Speaker 1>It's funny you mentioned David Dog. We've had I have

950
00:57:05.039 --> 00:57:09.239
<v Speaker 1>one before from dat Dog to talk about a lot

951
00:57:09.280 --> 00:57:13.760
<v Speaker 1>of the like I think memory profiling. He works on

952
00:57:13.800 --> 00:57:19.920
<v Speaker 1>a lot of like granular Ruby performance tooling, really interesting stuff.

953
00:57:21.159 --> 00:57:23.239
<v Speaker 1>But yeah, I mean I would love to see maybe

954
00:57:24.519 --> 00:57:28.039
<v Speaker 1>some more I don't know, higher level examples of like

955
00:57:28.400 --> 00:57:31.719
<v Speaker 1>making use of open telemetry in the Ruby space in general,

956
00:57:32.280 --> 00:57:34.800
<v Speaker 1>because I think that that level, I mean, especially with

957
00:57:34.840 --> 00:57:38.280
<v Speaker 1>all of the solid Q like our solid trifector or whatever.

958
00:57:38.400 --> 00:57:41.840
<v Speaker 1>Stuff that's coming around. It would be nice to see

959
00:57:41.960 --> 00:57:48.079
<v Speaker 1>something like tracing specifically introduced the rails that that would

960
00:57:48.119 --> 00:57:52.519
<v Speaker 1>make you know, more sense in that ecosystem, because yeah,

961
00:57:52.559 --> 00:57:54.639
<v Speaker 1>I mean where do you where do you start? Profiling? Stuff?

962
00:57:54.760 --> 00:57:58.840
<v Speaker 1>Is like kind of like an intro to tracing, right,

963
00:57:58.840 --> 00:58:02.199
<v Speaker 1>like if you wanted to see like the requests. It

964
00:58:02.199 --> 00:58:05.519
<v Speaker 1>reminds me of was a rack mini profile or tool

965
00:58:05.960 --> 00:58:08.519
<v Speaker 1>right where you can just see a little tiny tab

966
00:58:08.599 --> 00:58:11.119
<v Speaker 1>that says, oh, it took this number of seconds to

967
00:58:11.280 --> 00:58:14.400
<v Speaker 1>load this particular page you wanted to get, and you

968
00:58:14.440 --> 00:58:16.360
<v Speaker 1>can click on and expand and see, oh, well what

969
00:58:16.440 --> 00:58:19.639
<v Speaker 1>did your application do at each step of the way

970
00:58:19.760 --> 00:58:21.400
<v Speaker 1>and see how long each thing took?

971
00:58:21.559 --> 00:58:21.719
<v Speaker 2>Right?

972
00:58:22.360 --> 00:58:24.719
<v Speaker 1>And I think of that as like a trace a

973
00:58:24.760 --> 00:58:29.480
<v Speaker 1>lot of the times, right, And it's very like useful,

974
00:58:29.559 --> 00:58:32.760
<v Speaker 1>like even when you're just starting out to see that, right,

975
00:58:32.840 --> 00:58:35.880
<v Speaker 1>and it helps you visualize the And so I feel

976
00:58:35.920 --> 00:58:38.480
<v Speaker 1>like maybe that's what's missing, is a lot of like

977
00:58:39.559 --> 00:58:43.400
<v Speaker 1>visualization aspects of all this tracing stuff, because there's something

978
00:58:43.400 --> 00:58:46.960
<v Speaker 1>that you look at or find useful when you're starting

979
00:58:47.000 --> 00:58:51.880
<v Speaker 1>to dig into like structuring the traces and things like that.

980
00:58:52.440 --> 00:58:56.719
<v Speaker 2>Definitely, that's leading me up to my one of my

981
00:58:56.800 --> 00:59:01.960
<v Speaker 2>big kind of rumps, passions whatever within the observerity space.

982
00:59:02.440 --> 00:59:07.119
<v Speaker 2>And I don't see anybody talking about this. I feel

983
00:59:07.199 --> 00:59:10.880
<v Speaker 2>like it's either I'm onto a really great idea or

984
00:59:10.920 --> 00:59:13.920
<v Speaker 2>it's an unbelievably idiotic idea for some reason that I

985
00:59:13.920 --> 00:59:20.400
<v Speaker 2>don't know about. It's usually the latter as a spoiler. Okay,

986
00:59:20.679 --> 00:59:25.760
<v Speaker 2>So when I'm looking at traces, there's almost never enough information,

987
00:59:26.599 --> 00:59:29.679
<v Speaker 2>almost never enough information. And this is why Charity Majors

988
00:59:30.159 --> 00:59:33.159
<v Speaker 2>and the team at Honeycombe and there's Von Jones always

989
00:59:33.239 --> 00:59:40.519
<v Speaker 2>talk about and have wide context awear events. That's their mantra,

990
00:59:41.199 --> 00:59:46.119
<v Speaker 2>Wide context awear events and events. We've already talked about context,

991
00:59:46.119 --> 00:59:48.840
<v Speaker 2>we've already talked about we haven't talked much about the wide.

992
00:59:49.559 --> 00:59:53.440
<v Speaker 2>So wide means lots of attributes. So their take on

993
00:59:53.480 --> 00:59:58.519
<v Speaker 2>it is add as many attributes as you can to

994
00:59:58.800 --> 01:00:03.559
<v Speaker 2>every event and make them high cardinality attributes. What does

995
01:00:03.559 --> 01:00:06.119
<v Speaker 2>that mean? It took me about three months to outmad

996
01:00:06.159 --> 01:00:09.960
<v Speaker 2>around what high cardinality means. It means anything ending in

997
01:00:10.000 --> 01:00:13.119
<v Speaker 2>an ID. There you go, there's an easy there's an

998
01:00:13.159 --> 01:00:18.320
<v Speaker 2>easy explanation. So a request ID anything that looks sorry

999
01:00:18.360 --> 01:00:21.719
<v Speaker 2>that was me and my microphone. Anything that looks gooid

1000
01:00:21.920 --> 01:00:27.559
<v Speaker 2>like anything that is a unique identifier for anything, so

1001
01:00:27.639 --> 01:00:31.440
<v Speaker 2>that's user ID, request ID. But anything that is a

1002
01:00:31.480 --> 01:00:35.360
<v Speaker 2>domain object. And this is the real missed opportunity I

1003
01:00:35.360 --> 01:00:37.760
<v Speaker 2>think that we have in the rails community and in

1004
01:00:38.079 --> 01:00:43.800
<v Speaker 2>observability community potentially in general, when there is when something

1005
01:00:43.880 --> 01:00:48.280
<v Speaker 2>goes wrong, even when something goes right. Let's say, let's

1006
01:00:48.599 --> 01:00:54.800
<v Speaker 2>take the Let's take the token as an example. When

1007
01:00:54.840 --> 01:01:00.480
<v Speaker 2>that token is created, the token is a domain object. Okay,

1008
01:01:00.559 --> 01:01:03.199
<v Speaker 2>it's to do with authentication, so it's not it's not

1009
01:01:03.239 --> 01:01:06.800
<v Speaker 2>really a domain object in a way. But let's say

1010
01:01:06.800 --> 01:01:09.199
<v Speaker 2>that customer is signing up for an account. The account

1011
01:01:09.239 --> 01:01:12.239
<v Speaker 2>definitely is a domain object. And if you want to

1012
01:01:12.280 --> 01:01:14.599
<v Speaker 2>understand what I mean by domain object, I just mean

1013
01:01:14.679 --> 01:01:19.039
<v Speaker 2>an object that belongs to the domain, the business domain

1014
01:01:19.079 --> 01:01:21.960
<v Speaker 2>in which you're operating. It's a business object, a domain object.

1015
01:01:22.039 --> 01:01:27.119
<v Speaker 2>Call it what you will. But when the CETO or

1016
01:01:27.159 --> 01:01:30.440
<v Speaker 2>the even better the CEO or somebody in marketing talks

1017
01:01:30.480 --> 01:01:34.840
<v Speaker 2>about this customer account, they talk about people creating accounts,

1018
01:01:34.880 --> 01:01:38.360
<v Speaker 2>they use that word account, that's your first clue that

1019
01:01:38.400 --> 01:01:41.119
<v Speaker 2>it's a really important concept in the domain. So that's

1020
01:01:41.159 --> 01:01:43.079
<v Speaker 2>what I say when I mean domain objects. I mean

1021
01:01:43.159 --> 01:01:48.159
<v Speaker 2>words that non technical people use to describe your app. Right,

1022
01:01:48.320 --> 01:01:52.039
<v Speaker 2>So they're domain objects. Why are we not adding every

1023
01:01:52.119 --> 01:01:56.639
<v Speaker 2>relevant domain object every event? We don't do it, And

1024
01:01:56.679 --> 01:01:59.320
<v Speaker 2>so what you'll see is people do this kind of

1025
01:01:59.400 --> 01:02:04.400
<v Speaker 2>half heart. Oh well, we'll add the ID to the

1026
01:02:04.440 --> 01:02:07.440
<v Speaker 2>current span or the current trace or even the current log.

1027
01:02:07.719 --> 01:02:12.639
<v Speaker 2>We'll add the ID and that's okay, that'll be enough.

1028
01:02:12.880 --> 01:02:15.599
<v Speaker 2>But you're not capturing the state of the object. Why

1029
01:02:15.639 --> 01:02:18.159
<v Speaker 2>not just take the object in this case the account,

1030
01:02:19.119 --> 01:02:22.599
<v Speaker 2>convert it into a hash and attach it to the event.

1031
01:02:23.440 --> 01:02:25.679
<v Speaker 2>Why can't we do that? Now, there's a number of

1032
01:02:26.159 --> 01:02:28.679
<v Speaker 2>reasons why we actually can't do that in some cases.

1033
01:02:29.000 --> 01:02:33.599
<v Speaker 2>If you're build in terms of the size of your event.

1034
01:02:33.719 --> 01:02:36.440
<v Speaker 2>So if you're build on data, obviously that's going to

1035
01:02:36.440 --> 01:02:40.440
<v Speaker 2>get expensive fast. But if you're build on pure events,

1036
01:02:40.719 --> 01:02:45.920
<v Speaker 2>as in your observability provider or observability tooling is saying

1037
01:02:46.159 --> 01:02:49.880
<v Speaker 2>for every x number of events or every x number

1038
01:02:49.880 --> 01:02:52.719
<v Speaker 2>of logs per month, we will charge you this much,

1039
01:02:53.039 --> 01:02:56.760
<v Speaker 2>but the size doesn't matter. Then this is a perfect

1040
01:02:56.920 --> 01:03:00.880
<v Speaker 2>use case to be taking those rich domain objects, converting

1041
01:03:00.920 --> 01:03:04.800
<v Speaker 2>them into a structured format and dumping them in in

1042
01:03:04.920 --> 01:03:08.400
<v Speaker 2>the log or the trace, and so I've kind of

1043
01:03:08.480 --> 01:03:10.559
<v Speaker 2>thought about this quite a lot, and I've come up

1044
01:03:10.599 --> 01:03:15.239
<v Speaker 2>with a few quite simple ideas that people can use

1045
01:03:15.320 --> 01:03:19.519
<v Speaker 2>starting tomorrow in their rail SAPs, not without their problems,

1046
01:03:19.559 --> 01:03:23.480
<v Speaker 2>but the first of which is, I don't know if

1047
01:03:23.480 --> 01:03:31.280
<v Speaker 2>anybody's worked with formatted so two formatted s for daytime strings.

1048
01:03:31.719 --> 01:03:33.440
<v Speaker 2>And we have this idea in Ruby, don't we have

1049
01:03:33.519 --> 01:03:38.320
<v Speaker 2>duct typing. We have an object, and really good O

1050
01:03:38.639 --> 01:03:42.400
<v Speaker 2>design is that you shouldn't understand anything about that object.

1051
01:03:42.519 --> 01:03:45.159
<v Speaker 2>You just know it's got form methods on it. And

1052
01:03:45.239 --> 01:03:47.639
<v Speaker 2>it can be an account, it can be an invoice,

1053
01:03:47.679 --> 01:03:51.920
<v Speaker 2>it can be many different things. So my approach and

1054
01:03:51.920 --> 01:03:54.159
<v Speaker 2>them testing this approach out at work at the moment,

1055
01:03:54.559 --> 01:03:58.880
<v Speaker 2>is instead of having two formatted s, have two formatted

1056
01:03:59.239 --> 01:04:01.800
<v Speaker 2>h What does that mean. It means you're going to

1057
01:04:02.000 --> 01:04:06.599
<v Speaker 2>format the domain object as a hash, and so two

1058
01:04:06.719 --> 01:04:09.840
<v Speaker 2>formatted s allows you to pass in a symbol to

1059
01:04:09.920 --> 01:04:13.039
<v Speaker 2>define the kind of format that you want. So it

1060
01:04:13.079 --> 01:04:16.760
<v Speaker 2>can be short, ordinal, long, human eyes and it will

1061
01:04:16.800 --> 01:04:21.079
<v Speaker 2>output a string. I'll put a stringified version of that

1062
01:04:21.280 --> 01:04:26.360
<v Speaker 2>date in these different formats. So my idea is, why

1063
01:04:26.360 --> 01:04:29.920
<v Speaker 2>can't we have a method on every single domain object

1064
01:04:30.000 --> 01:04:33.719
<v Speaker 2>in our rails app called two formatted H and you

1065
01:04:33.880 --> 01:04:36.480
<v Speaker 2>pass it in a format. That format could be then

1066
01:04:36.719 --> 01:04:40.159
<v Speaker 2>open telemetry, it could be any one of numbers, a

1067
01:04:40.280 --> 01:04:45.280
<v Speaker 2>short compact. And so for every trace, the way I

1068
01:04:45.400 --> 01:04:47.800
<v Speaker 2>like to think of it is, I want to into

1069
01:04:47.840 --> 01:04:52.880
<v Speaker 2>that trace add every object that's related to that, and

1070
01:04:52.360 --> 01:04:57.480
<v Speaker 2>you could format those in open telemetry format for example,

1071
01:04:57.760 --> 01:05:01.280
<v Speaker 2>or you could have a full format or a long format,

1072
01:05:01.440 --> 01:05:04.320
<v Speaker 2>whatever you want, and so that way you can say, oh,

1073
01:05:04.360 --> 01:05:07.599
<v Speaker 2>I just want to I want a representation of the

1074
01:05:07.960 --> 01:05:10.840
<v Speaker 2>account that is short and it's just got the idea

1075
01:05:11.119 --> 01:05:14.320
<v Speaker 2>and that's a totally minimal skeleton, and that's enough for me.

1076
01:05:14.599 --> 01:05:17.079
<v Speaker 2>But actually here the work I'm doing is a bit

1077
01:05:17.119 --> 01:05:19.679
<v Speaker 2>more involved. So I want to call to formatted H

1078
01:05:19.800 --> 01:05:23.320
<v Speaker 2>with full and that will give the full account, like

1079
01:05:23.400 --> 01:05:26.280
<v Speaker 2>the updates that I created, that everything about it, and

1080
01:05:26.320 --> 01:05:28.280
<v Speaker 2>then that will be sent to my logs and traces,

1081
01:05:28.559 --> 01:05:32.320
<v Speaker 2>and I now have a standardized way of observing what's

1082
01:05:32.360 --> 01:05:35.559
<v Speaker 2>going on with all the rich data of my app

1083
01:05:35.800 --> 01:05:38.719
<v Speaker 2>app state at that point, with all the relevant domain

1084
01:05:38.760 --> 01:05:42.280
<v Speaker 2>objects in it. So that's that's my dream that I'm

1085
01:05:42.280 --> 01:05:46.920
<v Speaker 2>headed towards with this GEM. So that's kind of the

1086
01:05:46.920 --> 01:05:49.119
<v Speaker 2>way I think about structuring it, and I think about

1087
01:05:49.239 --> 01:05:53.199
<v Speaker 2>the people I see people doing all this ad hoc

1088
01:05:53.360 --> 01:05:56.039
<v Speaker 2>kind of well, this is this is an idea, and

1089
01:05:56.079 --> 01:05:59.559
<v Speaker 2>then we'll call the job idea a job underscore idea, suppose,

1090
01:05:59.599 --> 01:06:02.280
<v Speaker 2>and we'll what's the account. We can call that account

1091
01:06:02.400 --> 01:06:04.880
<v Speaker 2>underscore idea. And I'd just like to think of it

1092
01:06:04.920 --> 01:06:09.039
<v Speaker 2>as imagine your domain object. So an account has a customer,

1093
01:06:09.079 --> 01:06:12.400
<v Speaker 2>a customer has some bank details. Bank details is a

1094
01:06:12.400 --> 01:06:15.760
<v Speaker 2>bad idea, but address maybe, And so we could have

1095
01:06:15.880 --> 01:06:21.440
<v Speaker 2>these different formats that load nested relationships or not. And

1096
01:06:21.480 --> 01:06:24.679
<v Speaker 2>obviously you've got to be careful about the performance problems

1097
01:06:24.679 --> 01:06:27.840
<v Speaker 2>with that. And so you will have the exact structure

1098
01:06:27.880 --> 01:06:31.199
<v Speaker 2>of your domain object in your logs, in your traces.

1099
01:06:31.280 --> 01:06:33.800
<v Speaker 2>That for me is a dream. And then every single

1100
01:06:34.039 --> 01:06:38.440
<v Speaker 2>time and account is logged, it's in the same structure. Awesome.

1101
01:06:38.880 --> 01:06:40.880
<v Speaker 2>So I know that an account is always going to

1102
01:06:40.880 --> 01:06:43.599
<v Speaker 2>have an ID, it's always going to have a whatever

1103
01:06:43.639 --> 01:06:47.320
<v Speaker 2>other attributes, account at pending status, whatever it is. And

1104
01:06:47.360 --> 01:06:50.559
<v Speaker 2>so therefore I can say, show me every trace where

1105
01:06:50.559 --> 01:06:55.519
<v Speaker 2>the account was pending boom. Yeah.

1106
01:06:55.519 --> 01:06:58.679
<v Speaker 1>I love that idea, and it does. It reminds me

1107
01:06:58.719 --> 01:07:03.480
<v Speaker 1>a little of the introduction of the you know, the

1108
01:07:03.519 --> 01:07:07.440
<v Speaker 1>new rails like you know Blogger where you can tag

1109
01:07:07.639 --> 01:07:10.039
<v Speaker 1>you know, the taglogger was it was kind of like

1110
01:07:10.480 --> 01:07:13.880
<v Speaker 1>a start to kind of this idea of Okay, capture

1111
01:07:14.119 --> 01:07:17.599
<v Speaker 1>all of these pieces with this tag and it's like

1112
01:07:18.159 --> 01:07:21.800
<v Speaker 1>almost a pseudo trace, I call it. But it does

1113
01:07:21.840 --> 01:07:25.440
<v Speaker 1>go along that formatting aspect of like, okay, format all

1114
01:07:25.480 --> 01:07:29.559
<v Speaker 1>the things like this in a specific way. And I

1115
01:07:29.599 --> 01:07:33.800
<v Speaker 1>agree that there's definitely a lot to unwind there. We'll

1116
01:07:33.840 --> 01:07:36.639
<v Speaker 1>have to have you on more if you you know,

1117
01:07:37.039 --> 01:07:39.559
<v Speaker 1>when you you know, put this together as a gem

1118
01:07:39.679 --> 01:07:42.119
<v Speaker 1>or something, because I would love to dig into that.

1119
01:07:45.360 --> 01:07:45.639
<v Speaker 2>Cool.

1120
01:07:46.000 --> 01:07:48.320
<v Speaker 1>Yeah, I mean it definitely is. I love the idea

1121
01:07:48.360 --> 01:07:52.280
<v Speaker 1>of like the domain objects and extracting those out into

1122
01:07:52.440 --> 01:07:54.760
<v Speaker 1>a formatable way that you can then trace and follow

1123
01:07:54.800 --> 01:07:59.800
<v Speaker 1>through because that that design decision has definitely missed a lot,

1124
01:07:59.519 --> 01:08:03.159
<v Speaker 1>and seeing things like packwork as an example, was a

1125
01:08:03.199 --> 01:08:06.039
<v Speaker 1>great step in the right direction, I thought, And I'd

1126
01:08:06.119 --> 01:08:08.800
<v Speaker 1>like to see more of that kind of evolve in

1127
01:08:08.840 --> 01:08:14.800
<v Speaker 1>the real's ecosystem, of abstracting the domains into their own

1128
01:08:14.880 --> 01:08:17.640
<v Speaker 1>kind of segments and then being able to format them

1129
01:08:17.800 --> 01:08:20.840
<v Speaker 1>for traceability and things like that. I think you're on

1130
01:08:20.880 --> 01:08:24.880
<v Speaker 1>to the right, You're onto a lot here, and then.

1131
01:08:24.960 --> 01:08:27.600
<v Speaker 2>I mean, the thing that I think is unbelievably ironic

1132
01:08:27.760 --> 01:08:31.880
<v Speaker 2>is all I'm talking about is convention over configuration. And

1133
01:08:32.000 --> 01:08:34.600
<v Speaker 2>is that not why we all got into rails. I

1134
01:08:34.640 --> 01:08:36.960
<v Speaker 2>know Ruby is a different thing, but Rails is all

1135
01:08:36.960 --> 01:08:40.600
<v Speaker 2>about convention over configuration, and the entire area of observability

1136
01:08:40.760 --> 01:08:44.199
<v Speaker 2>strikes me could do with a massive dollop of convention

1137
01:08:44.319 --> 01:08:47.359
<v Speaker 2>over configuration. And that's what our contemmetry are trying to do.

1138
01:08:48.239 --> 01:08:50.319
<v Speaker 2>The one last thing, and I know the time is

1139
01:08:50.359 --> 01:08:51.840
<v Speaker 2>getting on, but the one last thing I want to

1140
01:08:51.880 --> 01:08:56.439
<v Speaker 2>just say on that is the other huge opportunity is

1141
01:08:56.760 --> 01:09:01.079
<v Speaker 2>adding context to errors. So we have the exception objects

1142
01:09:01.079 --> 01:09:05.399
<v Speaker 2>in Ruby and people store strings with them, and it's like, what,

1143
01:09:06.399 --> 01:09:09.199
<v Speaker 2>how do you suppose how am I supposed to understand

1144
01:09:09.199 --> 01:09:11.680
<v Speaker 2>anything from a string? And then people try and put

1145
01:09:11.720 --> 01:09:15.319
<v Speaker 2>IDs in the strings and you're like, knows not so

1146
01:09:15.520 --> 01:09:20.880
<v Speaker 2>at work, I've made this extremely simple, basically a subclass

1147
01:09:20.920 --> 01:09:24.319
<v Speaker 2>of standard error where you can attach context. So when

1148
01:09:24.399 --> 01:09:28.119
<v Speaker 2>you create the error, you pass in structured context. So

1149
01:09:28.159 --> 01:09:30.720
<v Speaker 2>if our logs are structured, surely our error should be

1150
01:09:30.720 --> 01:09:34.720
<v Speaker 2>structured as well. Makes sense, right, So you can say

1151
01:09:35.840 --> 01:09:39.479
<v Speaker 2>this error happened, and here was the account associated with

1152
01:09:39.520 --> 01:09:42.239
<v Speaker 2>it when that error happened, and here's a user and

1153
01:09:42.319 --> 01:09:46.239
<v Speaker 2>here's this, so it gets attached into the error and

1154
01:09:46.239 --> 01:09:50.239
<v Speaker 2>then using rails is new error handling. Rails do error

1155
01:09:50.239 --> 01:09:52.319
<v Speaker 2>dot handle. If you've not used it before, look it up.

1156
01:09:52.359 --> 01:09:55.319
<v Speaker 2>It's absolutely awesome. It's one of my favorite things that

1157
01:09:55.319 --> 01:09:58.800
<v Speaker 2>they've added to RAILS recently, relatively recently the last few years.

1158
01:09:59.159 --> 01:10:04.800
<v Speaker 2>And you can basically have listeners to these events, to

1159
01:10:04.880 --> 01:10:08.319
<v Speaker 2>these errors. Byby pardon, it will catch the errors. And

1160
01:10:08.359 --> 01:10:11.880
<v Speaker 2>then the context is encapsulated in the error, so you

1161
01:10:11.920 --> 01:10:14.079
<v Speaker 2>can pass these errors around and then you can do

1162
01:10:14.159 --> 01:10:17.399
<v Speaker 2>interesting stuff with that context. And all I do is

1163
01:10:17.960 --> 01:10:20.199
<v Speaker 2>pull out all the context and send it straight into

1164
01:10:20.239 --> 01:10:24.600
<v Speaker 2>the logs. And that has absolutely changed changed the way

1165
01:10:24.600 --> 01:10:27.399
<v Speaker 2>ID bug because when therever there's an error and it

1166
01:10:27.439 --> 01:10:29.279
<v Speaker 2>has all those rich data, you just look in the

1167
01:10:29.319 --> 01:10:32.039
<v Speaker 2>rich data and you're like, oh, that was the count,

1168
01:10:32.079 --> 01:10:34.760
<v Speaker 2>that was the shopify ida, that was a product ID

1169
01:10:35.039 --> 01:10:36.520
<v Speaker 2>I've got it, and then you just look up the

1170
01:10:36.520 --> 01:10:39.680
<v Speaker 2>idea in your externals. Oh right, okay, it's outsinc whatever

1171
01:10:39.720 --> 01:10:42.960
<v Speaker 2>it is. It makes life so much easier. So that's

1172
01:10:42.960 --> 01:10:47.039
<v Speaker 2>something I'm really passionate about as well having domain objects

1173
01:10:47.840 --> 01:10:50.960
<v Speaker 2>encapsulated within errors or we've got structured errors not just

1174
01:10:51.000 --> 01:10:51.800
<v Speaker 2>structured logs.

1175
01:10:52.960 --> 01:10:54.760
<v Speaker 1>Yeah, I mean that's definitely one thing that I look

1176
01:10:54.840 --> 01:10:58.720
<v Speaker 1>for when I'm looking for, you know, installing dependencies, right,

1177
01:10:58.760 --> 01:11:01.399
<v Speaker 1>like does the gem have its own you know, base

1178
01:11:01.560 --> 01:11:05.239
<v Speaker 1>air class that it then can you know, give minta

1179
01:11:05.319 --> 01:11:07.520
<v Speaker 1>data about whatever that it's raising the air is about

1180
01:11:07.600 --> 01:11:11.039
<v Speaker 1>like more than just like a string of some air

1181
01:11:11.239 --> 01:11:12.800
<v Speaker 1>that then you have to figure out what it is

1182
01:11:13.199 --> 01:11:14.960
<v Speaker 1>like having that extra mated data that you could just

1183
01:11:14.960 --> 01:11:18.000
<v Speaker 1>because you can you can just add attributes to a class,

1184
01:11:18.119 --> 01:11:22.039
<v Speaker 1>right and say this error has these attributes like it

1185
01:11:22.960 --> 01:11:25.920
<v Speaker 1>has you know, meaning associated with the air. I think

1186
01:11:26.159 --> 01:11:28.840
<v Speaker 1>more people doing that is definitely going to be making

1187
01:11:28.880 --> 01:11:31.399
<v Speaker 1>that easier to do first of all, but yeah, and

1188
01:11:31.399 --> 01:11:34.159
<v Speaker 1>then also getting more people to take on that convention.

1189
01:11:35.039 --> 01:11:38.000
<v Speaker 1>I completely agree with you there. Yeah, I mean we

1190
01:11:38.039 --> 01:11:42.000
<v Speaker 1>are getting at time here, is there any lass you know,

1191
01:11:42.199 --> 01:11:46.680
<v Speaker 1>pieces you wanted to you know, quickly highlight or mention

1192
01:11:46.920 --> 01:11:48.840
<v Speaker 1>before we you know, move into pics.

1193
01:11:50.319 --> 01:11:52.720
<v Speaker 2>I think the main thing is if you're listening to

1194
01:11:52.760 --> 01:11:56.119
<v Speaker 2>this and anything that I'm saying is resonating, forget about

1195
01:11:56.119 --> 01:11:58.960
<v Speaker 2>the demand object stuff. That's like getting really into the

1196
01:11:59.039 --> 01:12:02.479
<v Speaker 2>nitty gritty. But coming back to the beginning, if you're

1197
01:12:02.520 --> 01:12:06.800
<v Speaker 2>frustrated by your debugging experience, if you're thinking why am

1198
01:12:06.840 --> 01:12:10.560
<v Speaker 2>I not smart enough to understand this, chances are the

1199
01:12:10.560 --> 01:12:13.920
<v Speaker 2>problem is not with you, it's with the tools. So

1200
01:12:14.039 --> 01:12:17.039
<v Speaker 2>if you improve the tools, not only do you make

1201
01:12:17.079 --> 01:12:20.479
<v Speaker 2>your life easier and better, you level up everybody around

1202
01:12:20.520 --> 01:12:23.359
<v Speaker 2>you because all the engineers can use the same tools.

1203
01:12:24.039 --> 01:12:27.520
<v Speaker 2>And that's what we've experienced at Bigger Pockets, and that

1204
01:12:27.680 --> 01:12:32.279
<v Speaker 2>culture of observability has really worked its way into our culture,

1205
01:12:32.479 --> 01:12:35.039
<v Speaker 2>so that now anybody is equipped to go into the

1206
01:12:35.039 --> 01:12:38.319
<v Speaker 2>logs and ask any question that they want. So it

1207
01:12:38.479 --> 01:12:41.239
<v Speaker 2>is a long road, but it all starts with a

1208
01:12:41.279 --> 01:12:44.880
<v Speaker 2>single step. And so if you are feeling that pain,

1209
01:12:45.399 --> 01:12:48.359
<v Speaker 2>feel free to reach out to me. And I can

1210
01:12:48.479 --> 01:12:50.600
<v Speaker 2>go through all my socials in a minute, but feel

1211
01:12:50.640 --> 01:12:52.600
<v Speaker 2>free to reach out to me, ask me any questions.

1212
01:12:52.600 --> 01:12:54.319
<v Speaker 2>I'm happy to jump on a zoom call for half

1213
01:12:54.359 --> 01:12:58.479
<v Speaker 2>an hour and help you for free. But basically, it

1214
01:12:58.560 --> 01:13:02.640
<v Speaker 2>all starts by taking very small steps towards a very

1215
01:13:02.680 --> 01:13:07.800
<v Speaker 2>specific question. Don't try and add observability because you'll still

1216
01:13:07.800 --> 01:13:11.880
<v Speaker 2>be here next Christmas. So take heed, there is hope,

1217
01:13:12.560 --> 01:13:16.159
<v Speaker 2>and if any anything that I say resonates, please feel

1218
01:13:16.199 --> 01:13:18.000
<v Speaker 2>free to reach out to me and I'll help you

1219
01:13:18.000 --> 01:13:18.560
<v Speaker 2>figure it out.

1220
01:13:19.439 --> 01:13:27.119
<v Speaker 1>That's awesome. Yeah, I also echo that sentiment of you know,

1221
01:13:27.640 --> 01:13:33.159
<v Speaker 1>tooling is so important, and you know open tracing definitely

1222
01:13:33.239 --> 01:13:36.920
<v Speaker 1>is a great, great framework, and if we can improve

1223
01:13:36.920 --> 01:13:40.479
<v Speaker 1>that in the Ruby space, that will definitely well, we'll

1224
01:13:40.520 --> 01:13:47.880
<v Speaker 1>be reaping the rewards as well. So let's move into picks. John,

1225
01:13:47.920 --> 01:13:50.199
<v Speaker 1>do you have anything that you want to share first

1226
01:13:50.239 --> 01:13:50.960
<v Speaker 1>or you want me to go?

1227
01:13:53.800 --> 01:13:55.880
<v Speaker 2>Am I limited to one pick because I have many?

1228
01:13:56.239 --> 01:13:57.159
<v Speaker 1>Okay, go ahead.

1229
01:13:57.520 --> 01:14:03.279
<v Speaker 2>Cool. So the first one is a new language, and

1230
01:14:04.079 --> 01:14:07.880
<v Speaker 2>I already I already thoroughly trounced the idea that we

1231
01:14:07.880 --> 01:14:11.680
<v Speaker 2>should be learning one programming language a year, or rather,

1232
01:14:11.760 --> 01:14:16.640
<v Speaker 2>I I just dissed it off without actually giving much justification.

1233
01:14:17.079 --> 01:14:19.439
<v Speaker 2>So I'm going to go back on what I just

1234
01:14:19.479 --> 01:14:23.119
<v Speaker 2>said and say that this language has changed the way

1235
01:14:23.159 --> 01:14:26.920
<v Speaker 2>I think pretty much forever, and it's changed the way

1236
01:14:26.960 --> 01:14:30.520
<v Speaker 2>I see Ruby and rails and just programming in general.

1237
01:14:30.560 --> 01:14:33.880
<v Speaker 2>And the language is called Unison. Now it's a very

1238
01:14:34.039 --> 01:14:41.680
<v Speaker 2>very strange, unusual language. It's maybe not that readable in places,

1239
01:14:43.119 --> 01:14:45.920
<v Speaker 2>and it's also extremely new. I mean it's been going

1240
01:14:45.960 --> 01:14:48.079
<v Speaker 2>for five or six years, but what they're trying to

1241
01:14:48.079 --> 01:14:53.800
<v Speaker 2>do is incredibly ambitious. But look it up. It's Yeah,

1242
01:14:53.840 --> 01:14:58.079
<v Speaker 2>it's an incredibly interesting language and it will expand your

1243
01:14:58.159 --> 01:15:00.960
<v Speaker 2>mind and that's what it's so what it's done for me.

1244
01:15:01.960 --> 01:15:05.560
<v Speaker 2>And so it's kind of a language that's targeted at

1245
01:15:05.920 --> 01:15:12.119
<v Speaker 2>creating programs that are just much much simpler but actually

1246
01:15:12.199 --> 01:15:15.079
<v Speaker 2>more difficult to get your head around. It's a completely

1247
01:15:15.119 --> 01:15:20.359
<v Speaker 2>new paradigm for distributed computing basically, and it's absolutely fascinating.

1248
01:15:20.399 --> 01:15:24.039
<v Speaker 2>So I would highly suggest check that out. I know

1249
01:15:24.119 --> 01:15:27.640
<v Speaker 2>that Dave Thomas at Uruko. When I spoke at Yuruko recently,

1250
01:15:27.920 --> 01:15:31.840
<v Speaker 2>he was on stage and he was championing Unison and

1251
01:15:31.920 --> 01:15:34.199
<v Speaker 2>he called it the future of programming, and I could

1252
01:15:34.199 --> 01:15:37.399
<v Speaker 2>not agree more. It's an incredible language made by some

1253
01:15:37.439 --> 01:15:41.920
<v Speaker 2>incredibly smart people. So that's number one. Number two, there

1254
01:15:41.960 --> 01:15:44.600
<v Speaker 2>is a static site builder. I've used pretty much all

1255
01:15:44.640 --> 01:15:47.960
<v Speaker 2>the static site builders on planet Earth, and this is

1256
01:15:48.000 --> 01:15:52.279
<v Speaker 2>my favorite. It's called eleven T. It's a really odd name.

1257
01:15:54.319 --> 01:16:01.000
<v Speaker 2>But I'm embarking upon this project at work that really

1258
01:16:01.079 --> 01:16:05.119
<v Speaker 2>is exciting me, which is how do you serve UI

1259
01:16:05.199 --> 01:16:10.239
<v Speaker 2>components from a dynamic app rails and melds them into

1260
01:16:10.359 --> 01:16:14.159
<v Speaker 2>a static site builder without having a pile of JavaScript

1261
01:16:14.159 --> 01:16:16.039
<v Speaker 2>that you have to weighe through. So I want to

1262
01:16:16.079 --> 01:16:18.840
<v Speaker 2>author my UI components in rails, and I want to

1263
01:16:18.880 --> 01:16:23.560
<v Speaker 2>deliver them extremely fast through a static site that's just

1264
01:16:23.600 --> 01:16:25.840
<v Speaker 2>a blog without having to run that blog on rails.

1265
01:16:27.279 --> 01:16:29.800
<v Speaker 2>So eleven T is my go to tool for doing

1266
01:16:29.840 --> 01:16:33.720
<v Speaker 2>all that stuff. It also encompass this thing called web C,

1267
01:16:35.119 --> 01:16:38.359
<v Speaker 2>which is my new favorite templating language. Yes, I know,

1268
01:16:38.560 --> 01:16:42.680
<v Speaker 2>another templating language. I promise, I promise, it's really good.

1269
01:16:43.279 --> 01:16:46.680
<v Speaker 2>It's not another retread of all these other templating languages

1270
01:16:46.720 --> 01:16:50.520
<v Speaker 2>that are very, very niche and very whatever. So web

1271
01:16:50.600 --> 01:16:55.359
<v Speaker 2>C is compatible with web components and it's a fantastic

1272
01:16:55.479 --> 01:17:01.520
<v Speaker 2>way of making HTML like components are service side rendered.

1273
01:17:01.920 --> 01:17:04.479
<v Speaker 2>And I would love to see a plug in for

1274
01:17:04.600 --> 01:17:08.279
<v Speaker 2>that come to rails because it is absolutely phenomenal. So

1275
01:17:08.319 --> 01:17:11.159
<v Speaker 2>those are my two favorite things at the moment. If

1276
01:17:11.159 --> 01:17:14.960
<v Speaker 2>anybody is trying to wrestle with you, I components in

1277
01:17:15.119 --> 01:17:17.560
<v Speaker 2>rails and trying to extract them out to rails components.

1278
01:17:17.560 --> 01:17:20.560
<v Speaker 2>Also would love to chat through that with anybody who's

1279
01:17:20.680 --> 01:17:24.399
<v Speaker 2>interested in that kind of area because I think it's

1280
01:17:24.680 --> 01:17:27.600
<v Speaker 2>yeah good, there's a potential to really break new ground,

1281
01:17:28.000 --> 01:17:28.600
<v Speaker 2>how about you.

1282
01:17:30.119 --> 01:17:34.279
<v Speaker 1>Yeah, thanks, I'll definitely be digging into some of those. Yeah.

1283
01:17:34.399 --> 01:17:37.399
<v Speaker 1>I was at in New York City the other day

1284
01:17:37.640 --> 01:17:42.479
<v Speaker 1>for the Ruby AI happy hour that they've been doing

1285
01:17:43.319 --> 01:17:47.039
<v Speaker 1>every couple months. This time they did demos and I

1286
01:17:47.800 --> 01:17:52.960
<v Speaker 1>demoed this real time podcast buddy that I've made. It's

1287
01:17:52.960 --> 01:17:56.319
<v Speaker 1>called podcast Buddy, and it just kind of like listens

1288
01:17:56.359 --> 01:17:59.479
<v Speaker 1>in the background and in real time keeps track of

1289
01:17:59.520 --> 01:18:03.880
<v Speaker 1>the topic and the discussions and some example questions worth

1290
01:18:03.960 --> 01:18:06.840
<v Speaker 1>mentioning or maybe some you know topics that transition to

1291
01:18:08.039 --> 01:18:09.640
<v Speaker 1>and it's a lot of fun. I just did it

1292
01:18:09.640 --> 01:18:13.600
<v Speaker 1>for fun, but I recently refactored it to use the

1293
01:18:13.680 --> 01:18:18.239
<v Speaker 1>async framework and shout out to Samuel Williams just phenomenal,

1294
01:18:18.600 --> 01:18:22.560
<v Speaker 1>like so well put together. The documentation is coming along.

1295
01:18:22.800 --> 01:18:26.079
<v Speaker 1>It is lacking in some areas, but I was able

1296
01:18:26.119 --> 01:18:28.720
<v Speaker 1>to just completely refactor the code so that it works

1297
01:18:28.720 --> 01:18:32.640
<v Speaker 1>with async and runs things, you know, as they come

1298
01:18:32.640 --> 01:18:38.560
<v Speaker 1>in and it's streaming the whisper you know, transcripts. It

1299
01:18:38.680 --> 01:18:41.840
<v Speaker 1>performs actions in the background just like in the same thread.

1300
01:18:41.920 --> 01:18:46.159
<v Speaker 1>All advantaged with ACYNC just I love it, So check

1301
01:18:46.159 --> 01:18:49.359
<v Speaker 1>out podcast Buddy and check out async. You can't go

1302
01:18:49.439 --> 01:18:54.000
<v Speaker 1>wrong async web socket now you can handle even web

1303
01:18:54.000 --> 01:18:59.000
<v Speaker 1>sockets asynchronously just like completely seamless hgd B two and

1304
01:18:59.199 --> 01:19:06.079
<v Speaker 1>one compatible. Love it, so check those out. And John,

1305
01:19:06.439 --> 01:19:07.720
<v Speaker 1>if people want to reach out to you on the

1306
01:19:07.760 --> 01:19:12.319
<v Speaker 1>web or just in general, how can how can they

1307
01:19:12.319 --> 01:19:12.720
<v Speaker 1>reach you?

1308
01:19:14.479 --> 01:19:17.560
<v Speaker 2>Thank you? Yeah? So I'm on LinkedIn. That's a platform

1309
01:19:17.600 --> 01:19:24.000
<v Speaker 2>I'm most active on and my LinkedIn handle is synaptip mishap,

1310
01:19:24.359 --> 01:19:30.319
<v Speaker 2>which is yeah, I really regret that, sorry everybody, but yeah.

1311
01:19:30.359 --> 01:19:32.720
<v Speaker 2>So if you just search for John Gallagher, g A

1312
01:19:32.840 --> 01:19:36.319
<v Speaker 2>doub l A g h E R and maybe rails

1313
01:19:36.399 --> 01:19:39.800
<v Speaker 2>or observability, you should be able to find me. I've

1314
01:19:39.800 --> 01:19:43.079
<v Speaker 2>got quite a cheesy photo black and white photo of

1315
01:19:43.119 --> 01:19:47.479
<v Speaker 2>me in a suit. It's a horrible photo. And I

1316
01:19:47.560 --> 01:19:53.199
<v Speaker 2>blog at Joyfulprogramming dot com. It's a substack. So is

1317
01:19:53.199 --> 01:19:55.560
<v Speaker 2>this still a blog anymore? I have no idea, but

1318
01:19:56.119 --> 01:20:01.239
<v Speaker 2>that's where I write. I'm on Twitter at synaptic missapp

1319
01:20:01.239 --> 01:20:03.960
<v Speaker 2>and my get up handle is John Gallagher or one word.

1320
01:20:04.840 --> 01:20:08.359
<v Speaker 2>So yeah, Joyful Programming is the main source of goodies

1321
01:20:08.359 --> 01:20:11.279
<v Speaker 2>for me. I've also got a fairly minimal YouTube channel

1322
01:20:11.319 --> 01:20:14.720
<v Speaker 2>called Joyful Programming. So feel free to reach out to

1323
01:20:14.760 --> 01:20:18.520
<v Speaker 2>me connection, request me, ask me any question. I would

1324
01:20:18.560 --> 01:20:21.880
<v Speaker 2>love to engage with some reuby folks about observability. Tell

1325
01:20:21.920 --> 01:20:24.199
<v Speaker 2>me your problems and I'll try and help you wherever.

1326
01:20:24.319 --> 01:20:28.159
<v Speaker 1>Reccon awesome. I love it. Keep up the great work

1327
01:20:28.279 --> 01:20:33.840
<v Speaker 1>and keep you know, shouting from the mountaintop about observability,

1328
01:20:34.319 --> 01:20:38.840
<v Speaker 1>pulling those pillows down and just focusing on the important stuff, right,

1329
01:20:40.840 --> 01:20:45.039
<v Speaker 1>I love it. So until next time, everybody, I'm Valentino.

1330
01:20:45.760 --> 01:20:48.239
<v Speaker 1>Thanks John for coming on and look forward to the

1331
01:20:48.279 --> 01:20:48.720
<v Speaker 1>next time.

1332
01:20:49.479 --> 01:20:52.680
<v Speaker 2>Thanks for having me Valentino. It's been amazing, awesome,