Speaker 1: Hey, welcome back to another episode of JavaScript Jabber. This week,
Speaker 1: on our panel, we have Dan Shapiir.
Speaker 2: Hello, from this very hot Tel Aviv. Well, isn't that
Speaker 2: a surprise somewhere in Israel? Right?
Speaker 1: It's it's also pretty hot here. It spaghetting up to
Speaker 1: one hundred degrees which is like forty forty one degrees celsius.
Speaker 2: Oh, that's actually hotter than it is here. So we've
Speaker 2: got better.
Speaker 1: It's been that way here for over a month.
Speaker 2: It's been trying your eyeballs.
Speaker 1: Just about anyway, I'm Charles Maxwood from top Endevs, and
Speaker 1: we're going to be talking about monitoring and alerting using
Speaker 1: Prometheus and Gryfauna. Now I've played with Graffana this. This
Speaker 1: was proposed by Dan, and so I figure he's probably
Speaker 1: gonna do more the talking. But yeah, just to give
Speaker 1: a little context, I don't even know what freak this is.
Speaker 2: So yeah, yeah, So to backtrack a little bit before
Speaker 2: we get into specifically what Prometheus and Grafana are, I
Speaker 2: want to talk a little bit about monitoring alerting in general.
Speaker 2: You know, I've been working on stuff like performance monitoring
Speaker 2: and performance optimizations for a good number of years now,
Speaker 2: as you all know if you've been listening to this podcast,
Speaker 2: and probably one of the most important lessons that I've
Speaker 2: learned is that you need to have monitoring in place
Speaker 2: and alerting in place before you start any work on
Speaker 2: improving things. And in this context, I love a quote
Speaker 2: from Peter Drucker. Peter Drucker more or less created the
Speaker 2: field of business management, and the quote is, if you
Speaker 2: can't measure it, you can't improve it. And I'm really
Speaker 2: a big fan of that. If you're unable to measure something,
Speaker 2: there's really no way for you to know if you're
Speaker 2: making progress, if you're improving things, or if you're actually
Speaker 2: degrading things or making no impact at all. And one
Speaker 2: of the things that I've done in this context is
Speaker 2: that whenever I, like say, I join a new company
Speaker 2: or start a new project, I often find myself under
Speaker 2: pressure to start delivering improvements, you know, from the get
Speaker 2: go as quickly as possible, and I always push back
Speaker 2: on that in order to make sure that we have
Speaker 2: proper data collection and proper moneying in place before we
Speaker 2: start making any improvements. By the way, in this context,
Speaker 2: is probably not surprising that I joined Size Sense, which
Speaker 2: is a bi analytics company because I'm really a big
Speaker 2: believer in that. And you know, as an extra benefit
Speaker 2: if you're doing this kind of work, one of the
Speaker 2: big benefits of having some sort of measurement and monitoring
Speaker 2: solution in place is that after you've made improvements, you
Speaker 2: have graphs to show the impact of your hard work.
Speaker 2: And from my experience, that's really beneficial when you're let's
Speaker 2: say you're looking for raise or advancement or something like that.
Speaker 2: But really in order to be successful in any project
Speaker 2: that requires taking a system and trying to let's say,
Speaker 2: improve it, then having some sort of a monitoring capability
Speaker 2: place is crucial. Now the question obviously then becomes what
Speaker 2: do I actually use for that. I mean, you know,
Speaker 2: it's fairly straightforward to collect a lot of data these days,
Speaker 2: but what do you do with it? Where do you
Speaker 2: put it, how do you process it, how do you
Speaker 2: visualize it, et cetera. And And in this context, I
Speaker 2: want to talk specifically about two things, which are Prometheus
Speaker 2: and Gofana. And first I'll start with a riddle for you, Chuck,
Speaker 2: do you know like who Prometheus was?
Speaker 1: Like you know, he was a titan that brought us fire? Right,
Speaker 1: wow mythology, and he was he was punished by having
Speaker 1: what an eagle eat his entrails out.
Speaker 2: Exactly that he was tied to a mountain on the well,
Speaker 2: I think with the Tartarus mountains, and then Zeus's eagle
Speaker 2: will would come once a day and eat his entrails,
Speaker 2: and because he's immortally he cannot die, so he suffers forever.
Speaker 2: By the way, later times Greeks or Romans kind of
Speaker 2: thought that this was like too bad of a fate,
Speaker 2: so they had Hercules release him or something along these lines.
Speaker 2: But the no, I can't prove it. But I think
Speaker 2: that the reason that this project was called Prometheus is
Speaker 2: because it's about bringing knowledge, because fire in this context
Speaker 2: is like synonymous with knowledge, knowledge from the gods to
Speaker 2: the humans. So it's about bringing knowledge to us developers
Speaker 2: about how our systems are operating. So that's the mythology
Speaker 2: of what Prometheus is. Now let's talk about what Prometheus
Speaker 2: the service is. So Prometheus is free software as an
Speaker 2: open source that you can install on premises or you
Speaker 2: can use as a service I think as well that
Speaker 2: is used for event monitoring alerting. It was originally created
Speaker 2: something like twelve years ago at SoundCloud when they came
Speaker 2: to the conclusion that none of the third party solutions
Speaker 2: for monitoring were sufficient for their needs. And after they
Speaker 2: built their system and used it internally, about four years later,
Speaker 2: they donated it to the Cloud Native Computing Foundation, which
Speaker 2: is the same foundation that also hosts the cumbernitese project.
Speaker 2: So you know, this is another successful project from that foundation.
Speaker 2: I think the people who are mostly working on it
Speaker 2: these days are the people from the company that does Grafana.
Speaker 2: But again it's open source and you can see the
Speaker 2: source code on GitHub and there are a lot of contributors.
Speaker 2: I actually personally contributed to one of the satellite projects
Speaker 2: around Prometheus, which is the Prometheus client for node which
Speaker 2: makes it possible to connect node js to Prometheus in
Speaker 2: order to monitor NOE. So I contributed specifically specifically to
Speaker 2: that part of the project.
Speaker 1: I'm gonna stop you just for a minute. I've been
Speaker 1: posting the links in the comments, but they don't go
Speaker 1: to x and that's where most of our live listeners are.
Speaker 1: So Prometheus is Prometheus dot io and Graffana is at
Speaker 1: graffana dot com.
Speaker 2: So anyway, yeah, that's true. Okay, So, so where were
Speaker 2: we So we were talking about a little bit about
Speaker 2: the history of Prometheus, both the pathological figure and the project.
Speaker 2: Now let's talk a little bit about what it actually is.
Speaker 2: So it's a service used for event monitoring, alerting. It
Speaker 2: records real time metrics in something called a time series database,
Speaker 2: which is kind of a special type of database, and
Speaker 2: we'll talk about it in more detail and how it
Speaker 2: differs from the databases that most of us are familiar with.
Speaker 2: It allows for something called high dimensionality, which I also
Speaker 2: will try to explain. It supports flexible queries and real
Speaker 2: time alerting. And as I said, it's free software. It's
Speaker 2: licensed under the Apache two license. So that's what Prometheus is.
Speaker 2: So let's say you want to use Prometheus in your organization.
Speaker 2: What you would do is that you would install the
Speaker 2: Prometheus service and then hook it up to your various
Speaker 2: services that you want to monitor. Now, it's a monitoring
Speaker 2: solution for back end infrastructure, so things like no JS
Speaker 2: or for you know, the JVM, or for something that's
Speaker 2: say written in Go. Because that's not surprising because Prometheus
Speaker 2: itself is actually written in Go. So maybe it's a
Speaker 2: shame that we don't have aj on the show this time.
Speaker 1: Yeah. Maybe, but but your point is is that you know,
Speaker 1: any any language or system could have a driver that yes.
Speaker 2: There's basically a connector for anything. There are also connectors
Speaker 2: for a lot of general services. So if you want
Speaker 2: to monitor, let's say we were talking about Kubernetes, you
Speaker 2: can monitor Kubernetes. Kubernetes has a built in connector for Prometheus,
Speaker 2: so you can look at how pods are functioning, or
Speaker 2: the Kubernetes cluster itself. There are connectors for various AWS
Speaker 2: services and so on and so forth, so you can
Speaker 2: collect a lot of data from third party services and infrastructure,
Speaker 2: and you can also attach it like create applicative level monitoring,
Speaker 2: so you can monitor the behavior of your own applications
Speaker 2: that are running on platforms such as the JVM, such
Speaker 2: as no JS or in Go, etc. Well more or
Speaker 2: less any programming language that you can think of. Now.
Speaker 2: The way that you configure the system, so again not
Speaker 2: very surprising, given perhaps that it's certain in Go the
Speaker 2: configuration are yamal files, and again this is kind of
Speaker 2: correlated with Kubernetes. So Yamal, for those of our listeners
Speaker 2: who somehow don't know, is a configuration format. You can
Speaker 2: think of it to an extent kind of sort of
Speaker 2: similar to what we usually do with Jason files, but
Speaker 2: it's a different format. It has certain advantages over JSON.
Speaker 2: For example, it supports comments. It's used by a lot
Speaker 2: for a lot of administrative stuff. So any DevOps person
Speaker 2: is likely very familiar with Yamo.
Speaker 1: The default configus for rails were all done in Yammo
Speaker 1: as well.
Speaker 2: So yeah, so you basically create configuration files for Prometheus
Speaker 2: in Yamel. And the way that Prometheus works is kind
Speaker 2: of the reverse of what you might expect. So you
Speaker 2: might think that you somehow configure various systems to push
Speaker 2: data into Prometheus, but that's not how it works. The
Speaker 2: way that it actually works is that Prometheus data into itself.
Speaker 2: So in the yamal file, you tell Prometheus the address
Speaker 2: of the various addresses of the various services that you
Speaker 2: wanted to monitor and the rate at which it should
Speaker 2: effectively ping those services, and it effectively does an HTTP
Speaker 2: get to an end points exposed by these services and
Speaker 2: downloads data from them. So it actually pulls the data
Speaker 2: from them into itself. Now, the advantage of this approach
Speaker 2: is that first of all, they don't need to be
Speaker 2: aware of where the Prometheus server is, so it's all
Speaker 2: the configuration is centralized. They just need to open the port,
Speaker 2: you know, listen on it, and that's more or less.
Speaker 2: It also means that they can work with multiple Prometheus
Speaker 2: service servers at the same time because they all just
Speaker 2: pull the data, because pulling the data doesn't clear the data.
Speaker 2: It's not as if, you know, they give the data
Speaker 2: and then forget it it. They retain the data. Those
Speaker 2: services are expected to retain their data in memory, so
Speaker 2: you can just hit any one of them at any
Speaker 2: time and pull from them their current situation state. I
Speaker 2: hope that's clear.
Speaker 1: Yeap makes sense to me. I kind of like it too,
Speaker 1: just from the standpoint of so I've used other systems,
Speaker 1: paid systems, you know, we've been sponsored in the past
Speaker 1: by Century and ray Gun and stuff like that that
Speaker 1: grab a lot of this information. Though I think we're
Speaker 1: talking kind of a level below that right where we're
Speaker 1: not talking specifically about the information that's being sent. We're
Speaker 1: just talking about how the information gets into the system.
Speaker 1: At this point, I like the fact that it's like, Okay,
Speaker 1: I'm gonna periodically check and then I don't have ten
Speaker 1: million hits on the service on the other end, right,
Speaker 1: because I'm not pushing it to it, it's pulling it,
Speaker 1: and so it only does the work that it has
Speaker 1: to do, right Yeah, Yeah, I don't have all this
Speaker 1: extra network crap going on.
Speaker 2: Yeah. So it basically does an AGTP get let's say
Speaker 2: once every minute to any one of those services. The
Speaker 2: response is essentially texts in their own format. It's quite
Speaker 2: readable actually, so you could literally go to one of
Speaker 2: those endpoints and just hit it with your browser and
Speaker 2: see what the response would be. By the way, obviously
Speaker 2: you probably want to make sure that those endpoints are
Speaker 2: not externally exposed, so you know that everything stays behind
Speaker 2: the firewall. The one kind of caveat to that is
Speaker 2: that sometimes you have like short lived services, think about
Speaker 2: something like a lambda. In that case, what they have
Speaker 2: is something called a push gateway, which is like a
Speaker 2: standalone service that those show or lived jobs can push
Speaker 2: data in. Because they are really short lived, so you
Speaker 2: can't assume that they'll hang around until they're they're pulled again.
Speaker 2: So they can push their data into that push gateway,
Speaker 2: which holds onto that data, and then Prometheus pulls that
Speaker 2: data from that push gateway. So it's kind of an
Speaker 2: intermediary service that that you know, for those special cases.
Speaker 2: But in most cases and the cases that I've used
Speaker 2: it in, you know, it was with long lived servers
Speaker 2: or services, and then you know that's just the way
Speaker 2: it worked, right.
Speaker 1: That makes a lot of sense too, just in the yeah,
Speaker 1: like I'm thinking like serviles functions and things like that,
Speaker 1: or you know, if you we've also run background jobs
Speaker 1: a lot in the apps that I have right where
Speaker 1: it pulls it off the queue and then runs it.
Speaker 1: And so yeah, in either of those cases, Yeah, you
Speaker 1: don't want or necessarily need something hanging out so that
Speaker 1: it can say, oh, you're going to query me within
Speaker 1: the next minute or so, it just says poof, I'm
Speaker 1: done and then hands it off, right.
Speaker 2: Yeah. So there's for example, in the case of node JS,
Speaker 2: there's a Prometheus client for node I think it's literally
Speaker 2: the project is really literally as I recall, called Prome client.
Speaker 2: So you just NPM install it and then you know
Speaker 2: it's in there. You just give it to the port
Speaker 2: to listen on and then it does you know, just
Speaker 2: let's say it uses express or something like that, and
Speaker 2: then it basically collects the information and exposes that to
Speaker 2: that port for you, and you don't really need to
Speaker 2: do anything to start monitoring basic system level stuff. Now,
Speaker 2: the Prometheus server gets the data in and then put
Speaker 2: saves it into its own percise database, right, and that
Speaker 2: database is what I call the time series database. And
Speaker 2: what I mean by that is that Prometheus doesn't store data, Like,
Speaker 2: don't think of it something like a database, you know,
Speaker 2: we tend to think of something like a relational database
Speaker 2: or maybe a low sequel database. That's not really that
Speaker 2: sort of a thing. It's something called the time series database.
Speaker 2: So basically it has metrics, and it basically saves the
Speaker 2: value of a metric at every point in time, so
Speaker 2: it's really like keeps on collecting metric data. So there's
Speaker 2: no such thing really as schemas or something like that.
Speaker 2: It just has metrics that it collects data into, and
Speaker 2: it's data collected over time. So like think of I
Speaker 2: don't know, let's say you're let's say you're a farmer
Speaker 2: and you've got a field and you're measuring the temperature
Speaker 2: in the field. So you've got the temperature measurement. Let's
Speaker 2: say every minute that you got from from the whatever,
Speaker 2: you know, the thermonitor device that you use to monitor
Speaker 2: the temperature, and it just gets recorded into that persistent
Speaker 2: storage and you can go backward and forward in time
Speaker 2: and look at any point in time what the temperature was. Right.
Speaker 1: So I'm trying to imagine what this looks like for
Speaker 1: an app. So is it measuring like how much CPU
Speaker 1: it's using and how much memory? Oh we'll get into that.
Speaker 2: Okay, yeah, we'll get into that. But basically, as I
Speaker 2: was saying, you can.
Speaker 1: I'll keep my enthusiasm down for a minute.
Speaker 2: Then it's it's you know what, let's talk about that
Speaker 2: a little bit. So it's really monitoring two types of information.
Speaker 2: One you can think about it as system level stuff.
Speaker 2: So in these context of node a node server, that
Speaker 2: might be CPU usage or memory slash heap usage, or
Speaker 2: the event loop lag or if it's an express service,
Speaker 2: the number of requests per period of time or the
Speaker 2: duration of the responses stuff like that. So those are
Speaker 2: system level things and they're collected for you automatically. So
Speaker 2: as soon as you NPM install the prompt client for
Speaker 2: node and it is loaded into your project, your node server,
Speaker 2: then all that stuff is automatically collected for you, and
Speaker 2: when the promitth your server hits that port that information,
Speaker 2: those metrics are available from the get go. On top
Speaker 2: of that, you can add applicative level metrics that you
Speaker 2: push into Prometheus using an API. So, for example, if
Speaker 2: you've got like your own let's say QUES internal cues
Speaker 2: that you want to monitor the usage of, or your
Speaker 2: own business logic processes that you want to measure the
Speaker 2: duration of, you can measure those as well. Okay, so
Speaker 2: you've got both the system level stuff and the applicative stuff.
Speaker 2: And by the way, one system level stuff that is
Speaker 2: really important in the context of no that may be
Speaker 2: less obvious or familiar to some of our listeners is
Speaker 2: something called the event loop LAG. Are you familiar with that?
Speaker 1: I am not.
Speaker 2: So, as you know, the way the JavaScript works is
Speaker 2: that it's all based on an event loop, be it
Speaker 2: either in the browser or in node JavaScript. The way
Speaker 2: that it works is you've got whenever something happens, like
Speaker 2: if it's a browser, then it's something arrives over the
Speaker 2: network or the user does some sort of an interaction
Speaker 2: a mouse click, a keyboard press. Whenever something happens, an
Speaker 2: event is triggered and that puts the event information in
Speaker 2: a queue, and JavaScript which behaves is in a runs
Speaker 2: kind of in a single threaded type approach, pulls data
Speaker 2: out the most the top data out of the queue,
Speaker 2: processes it, and then moves to the next item in
Speaker 2: the queue, the next item in the queue, and so
Speaker 2: forth until you know. If the queue becomes empty, then
Speaker 2: it effectively idles until another stuff is put into the queue.
Speaker 1: So far, so good, yep.
Speaker 2: Now, the problem is that usually rather than idling, what
Speaker 2: happens really is that information comes into gets into the
Speaker 2: queue at a very rapid pace, at a high clip.
Speaker 2: So it might new new events are placed into the
Speaker 2: queue before the the before the jobscript engine is ready
Speaker 2: to process them, because it's still busy processing other stuff.
Speaker 2: Like you think about, Let's say a node server running
Speaker 2: express the events coming in are are the HDP requests.
Speaker 2: If HDP requests arrive at a too high rate, then
Speaker 2: then the node service might not be able to process
Speaker 2: them quickly enough and will get overloaded. Right, So, what
Speaker 2: the event loop plaque actually measures is the amount of
Speaker 2: time from when a message is placed into the queue
Speaker 2: until it's taken out of the queue in order to
Speaker 2: be processed. So If that period of time is small,
Speaker 2: then you know that your service is really responsive. If
Speaker 2: it gets to be too high, that it means you know,
Speaker 2: your your system is overloaded and and it's not responsive enough.
Speaker 2: You know, if think about think about a service, an
Speaker 2: express service that takes I don't know two seconds to
Speaker 2: get to pull something out of the queue, it means
Speaker 2: that the browser, the client side, is waiting for two
Speaker 2: seconds before it's it's event is even processed. So obviously
Speaker 2: that's a bad thing, and it's especially bad if it
Speaker 2: keeps on growing, because then eventually your service server will
Speaker 2: just become a response right. So that's information that Prometheus
Speaker 2: that the system, the prompt client is actually able to
Speaker 2: extract out of node and exposes that into Prometheus. So
Speaker 2: that's one of the system monitoring things that's really useful
Speaker 2: to look at when you're monitoring a node service. Another
Speaker 2: thing that's really useful to look at is, for example,
Speaker 2: heap usage, because if you've got let's say, some sort
Speaker 2: of memory leak, then you'll see that after a garbage
Speaker 2: collection a GC, rather than going all the way down
Speaker 2: your memory, just utilization keeps going up and up and
Speaker 2: up and up and again. That if it keeps on going.
Speaker 2: What will happen is that the node service will try
Speaker 2: to do GCS more and more and more and more
Speaker 2: in order to free memory. But to do so and
Speaker 2: you get what's known as a GC storm, or effectively,
Speaker 2: the service all it does is just try to free
Speaker 2: up memory that it can't and it becomes totally stuck.
Speaker 2: So that's another thing that you can look at in
Speaker 2: the context of Prometheus.
Speaker 1: Well, it seems like on both of those measures, on
Speaker 1: the lag in what is it node event loop lag,
Speaker 1: it seems like because I'm sitting here and I'm thinking, okay,
Speaker 1: so it's gonna tell me if it doesn't have the
Speaker 1: resources to handle whatever's coming at it. But for me,
Speaker 1: I find it useful because I mean, let's just take
Speaker 1: podcasting for an example, Right, Like I don't go and
Speaker 1: religiously obsess over the numbers, right, I don't go look
Speaker 1: at the metrics. But for an app, if you're checking
Speaker 1: the metrics on a regular basis, it seems like you
Speaker 1: could start to see this lag event loop flag steadily
Speaker 1: increasing and go, Okay, we are getting to the point
Speaker 1: where we need to start looking at right, instead of
Speaker 1: all of a sudden being whoa, whoa, whoa. We're you know,
Speaker 1: we're way over the edge, right, And so it allows
Speaker 1: you to be proactive, right instead of reacting to people
Speaker 1: are complaining it's slow.
Speaker 2: Yeah. And actually, what you really want is to have
Speaker 2: good alerting, and we'll get to that as well, because yeah, yeah,
Speaker 2: because realistically, you're not going to check the graphs for
Speaker 2: all your services every morning or every afternoon. What you
Speaker 2: really want is you want a system that alerts you
Speaker 2: in case something is wrong. And you usually want an
Speaker 2: alert to say not that you know system is broken.
Speaker 2: You want an alert to tell you you know, system
Speaker 2: is running hot, you should you should do something before
Speaker 2: it breaks, right, And yeah, and and Prometheus is great
Speaker 2: for that as well because you can specify alerts. And
Speaker 2: so that's the other part. So I was talking about
Speaker 2: how all the data is collected into the Prometheus service
Speaker 2: and then saved into persistent storage. But then you can
Speaker 2: also do queries on top of that data. Prometheus has
Speaker 2: its own query language. It's called prom ql, and it
Speaker 2: looks nothing like SQL. You know, even though it's a
Speaker 2: query language, it's a totally different query language as well.
Speaker 1: As a query language, and it doesn't look.
Speaker 2: Like exactly, and you can do two sort of things.
Speaker 2: You can have Graffana as a visualization environment. So Graffana
Speaker 2: is the service so that you run in the browser
Speaker 2: and it can show you all sorts of graphs from
Speaker 2: various data sources, and one of the data sources is Prometheus,
Speaker 2: and you can write prom ql queries that extract data
Speaker 2: and then graph this data in whatever dashboard you're using.
Speaker 2: So that's one possible usage. Another possible usage is something
Speaker 2: called the alert Manager, which is another component of Prometheus
Speaker 2: that comes along with it. It's a standalone service that
Speaker 2: does regular that every at regular intervals, it runs uh
Speaker 2: prom ql queries, gets the data from them, and sees
Speaker 2: if they're if alerting criteria are met, and if they are,
Speaker 2: it can then push alerts into emails or Slack or
Speaker 2: page your duty or whatever, so that you can generate
Speaker 2: alerts out of the promit out of the Prometheus data. So,
Speaker 2: for example, going back to that farmer and field and
Speaker 2: thermometer example, you could say that if the temperature goes
Speaker 2: above I don't know, thirty degrees on the ground, send
Speaker 2: an alert something along these thoughts.
Speaker 1: Yeah, that makes sense.
Speaker 2: Yeah, So.
Speaker 1: I just want to throw in here real quick, because
Speaker 1: I think sometimes we kind of treat the time series
Speaker 1: data as kind of monolithic in ways, like treating it
Speaker 1: like for the day or the week.
Speaker 2: Right.
Speaker 1: I was actually looking on the Discord server that we
Speaker 1: use for the hosts and Adventures and DevOps. They were
Speaker 1: going to do an episode where they were talking about
Speaker 1: holiday rushes, right, And so one day to the next,
Speaker 1: it may vary, or you may get a lot of
Speaker 1: your traffic in the morning or the evening, and so
Speaker 1: you know, by having these alerts, you can start to
Speaker 1: pick up some of the patterns. And the other thing
Speaker 1: is is you can turn around and you can say, Okay,
Speaker 1: not only do I know that something's happening now, but
Speaker 1: I can go look at the current state of things
Speaker 1: or get a snapshot from the Prometheus data, and then
Speaker 1: start to solve whatever is the issue is, right, whether
Speaker 1: it's I need more resources or oh I didn't realize,
Speaker 1: but I built something into the application that makes it
Speaker 1: memory heavy, and so my heap size is going out
Speaker 1: of control and I'm running out of memory or whatever.
Speaker 2: For sure, But definitely also the fact that data can
Speaker 2: vary over time, even if regularly, can make some of
Speaker 2: this stuff pretty challenging, but still doable. PROMQL is a
Speaker 2: very sophisticated QUERI language, and I'll give examples of some
Speaker 2: of the challenges that I've run into when creating alerts. So,
Speaker 2: for example, a lot of companies that I've worked for,
Speaker 2: companies like Wix or Next Insurance, there was a lot
Speaker 2: more traffic let's say, over the week days than over
Speaker 2: the week ends, which is not surprising. But the downside
Speaker 2: of that was that data over the weekend would fluctuate
Speaker 2: a lot a lot more because the sampling size was
Speaker 2: significantly lower. When you think about it, let's say you
Speaker 2: have let's say talk about Next Insurance. You might have
Speaker 2: ten thousand sessions in a week, you know, working in
Speaker 2: a week day in a working day, but only one
Speaker 2: hundred over the weekend. And when you've got only one
Speaker 2: hundred sessions, then ten bad sessions can really impact your
Speaker 2: performance numbers. So and then we would see that if
Speaker 2: we weren't careful, we would start getting alerts over the
Speaker 2: weekends because the data was less stable because there were
Speaker 2: just fewer sessions. So we basically kind of had to
Speaker 2: come up with queries that were also dependent on the
Speaker 2: number of sessions, not just the duration of the sessions,
Speaker 2: So if there were too few sessions, we would ignore
Speaker 2: the other criteria of the duration of the sessions. So
Speaker 2: that might be something. You know, it makes the queries
Speaker 2: more challenging. But you want to take these sort of
Speaker 2: things into account. So quickly going back to when is
Speaker 2: Prometheus a good match and when isn't it a good match?
Speaker 2: What type of data would you want to put into
Speaker 2: Prometheus and what kind of data you would probably want
Speaker 2: to put somewhere else. So Prometheus is a good match
Speaker 2: when you're recording pure numeric time series data. It's appropriate
Speaker 2: for machine centric monitoring, you're monitoring systems when it's highly
Speaker 2: dynamic service oriented architectures, because you know, this whole pull
Speaker 2: type mechanism makes it very easy to adjust to additional
Speaker 2: instances coming up or going down in a very dynamic
Speaker 2: sort of a manner. And again i'll talk about it
Speaker 2: assuming we have the time something, but it's very important
Speaker 2: when the data is multi dimensional and both in the
Speaker 2: collection and the querying, and I'll explain what that is
Speaker 2: a bit later on. Now, when is it not a
Speaker 2: good match. First of all, it's not a good match
Speaker 2: when you need one hundred percent accuracy So for example,
Speaker 2: if you're looking at stuff like billing, when the numbers
Speaker 2: have to be perfect, then Prometheus is not a good solution.
Speaker 2: Prometheus in most cases kind of averages out data because again,
Speaker 2: when you're looking at stuff like CPU usage, it doesn't
Speaker 2: really matter if you're running at ninety three or ninety
Speaker 2: three point two or even ninety four percent. But if
Speaker 2: you know, obviously, if you're looking at stuff like, you know,
Speaker 2: your taxes, you probably need to be accurate. Uh, it's
Speaker 2: not approprix.
Speaker 1: Sorry you said, don't talk to me about taxes right now.
Speaker 2: Yeah, yeah, anyway, Uh, it's not. It's also not appropriate
Speaker 2: when you're recording non numeric data. So for example, if
Speaker 2: you're recording stuff like email addresses or phone numbers, even
Speaker 2: even though they seem like they're numeric, they're not really,
Speaker 2: or you know, street addresses, stuff like that, that's not appropriate.
Speaker 2: And it's also not appropriate when the data has to
Speaker 2: be totally persistent, when you know when it's you can't
Speaker 2: afford to lose any data. As I said, Prometheus pulls
Speaker 2: the data, pulls the data out of the very services
Speaker 2: at regular intervals, let's say every one minute. If that
Speaker 2: service crashes during that one minute, you've lost the data
Speaker 2: since you previously pulled it. And that's obviously not something
Speaker 2: that you can live with in a lot of cases
Speaker 2: where you need persistent data and you know, if it's
Speaker 2: if you're dealing with bank accounts and stuff like that,
Speaker 2: you can't afford to lose transactions or stuff like that.
Speaker 1: So yeah, but it seems like most people are pulling
Speaker 1: or using it by putting this client onto their application,
Speaker 1: and so it's it's only recording those specific kinds of data.
Speaker 1: You're talking about a custom use of Prometheus where you
Speaker 1: might push other data into it as.
Speaker 2: Well, exactly exactly. Now, it can be a challenge because again,
Speaker 2: let's say you're pulling, you're pulling the data or pulling
Speaker 2: it in every minute, and if your server has a
Speaker 2: problem that in a certain scenario, memory consumption like rockets
Speaker 2: out of control and then the server crashes, then you
Speaker 2: might not be able to catch that if it, you know,
Speaker 2: if it happens within the spence of the span of
Speaker 2: a few seconds. So you can either then try to
Speaker 2: increase the rate at which you pull and hope that
Speaker 2: you're lucky, or look at for some other solution. Right,
Speaker 2: as I said, it has integrations for no JVM, Go, Python, Ruby,
Speaker 2: and in terms of systems, stuff like Kubernetes, git Lab, AWS, Gira,
Speaker 2: mongo dB, reddis. For visualization, usually you'd use Gofana, even
Speaker 2: though it also has its own built in simple visualization capabilities.
Speaker 2: AH And it's compatible with open Telemetry, so if you're
Speaker 2: using open telemetry for collecting telemetry information, you can also
Speaker 2: use open Telemetry on top of Promethea and have open
Speaker 2: Telemetry put it data, put its data the data that
Speaker 2: it collects into the relevant data, the time series data
Speaker 2: into Prometheus. So there are several types of metrics and
Speaker 2: they're appropriate for different scenarios. So very quickly going over
Speaker 2: the different metric types, you've got something called a counter counter.
Speaker 2: Think about you know, the if you're let's say, think
Speaker 2: about a club. Let's say that you want to go
Speaker 2: into and there's somebody at the door with this kind
Speaker 2: of a counter device clicking it every time somebody goes in,
Speaker 2: like counting the number of people inside, because you know,
Speaker 2: the fire regulations only allow up to x number of
Speaker 2: people to be inside at the same time, and they
Speaker 2: so they need to count the number of people going
Speaker 2: in and the number of number people coming out to
Speaker 2: make sure that you know they don't exceed those limits.
Speaker 2: So counter is really something like that. It's a metric
Speaker 2: that that can only increase and you basically add one
Speaker 2: or add an your end, which is like adding one
Speaker 2: end times, so you basically just add add into it
Speaker 2: and it keeps on getting higher. Can you think about
Speaker 2: things that you would measure using something like like a counter.
Speaker 1: Yeah, like the number of requests that come in or so.
Speaker 2: Funnily that that's that's that's literally the first example that
Speaker 2: I have. So the request count is exactly such a
Speaker 2: such a thing, tasks completed, error count, all these kind
Speaker 2: of things that only go up. They only increase until
Speaker 2: at least until you restart the service, right. And the
Speaker 2: cool thing about a counter is that because of this
Speaker 2: behavior of only increasing, you can compute the rate of increase,
Speaker 2: which makes it possible to have predictions because if you
Speaker 2: can calculate the rate, then you can also predict where
Speaker 2: you'll be in a certain amount of time.
Speaker 1: Makes sense.
Speaker 2: Yeah, So that's the simplest type of metric. Again, it's
Speaker 2: used automatically for things like a request counter or a
Speaker 2: task complete counters, or an error counter or stuff like that,
Speaker 2: But you can also create your own applicative counter if
Speaker 2: you want to count your own stuff. Right. The next
Speaker 2: type of metric is called a gauge, and it records
Speaker 2: a value that can go up or down or literally
Speaker 2: be set at any value that you want. So you
Speaker 2: can literally say, say, the value of this gauge right
Speaker 2: now is x. Again, can you think of when you
Speaker 2: might use a gauge? This would be like Q size
Speaker 2: or shuck, are you looking at my slides? No?
Speaker 3: Because again that's the first item that I have on
Speaker 3: my slide for using it is the Q is Q
Speaker 3: size exactly Q size, memory usage, CPU usage, number of
Speaker 3: requests in currently in progress, all these sort of things that.
Speaker 2: You just set it to a particular value at each
Speaker 2: point in time now the key, so it can go
Speaker 2: up and down or be set to particular value. The
Speaker 2: thing about it, though, is that because it's so arbitrary,
Speaker 2: you can't use it to assess rate of change, because
Speaker 2: if you can just jump between numbers, there's no really
Speaker 2: meaningful rate that you can think about.
Speaker 1: Yeah, you can average it out, but yeah, the next.
Speaker 2: One is slightly more complicated. And when I actually have
Speaker 2: presentations that I do that I so far have done internally,
Speaker 2: I'm kind of looking for conference who wants to talk
Speaker 2: about this. But if you want to be able to
Speaker 2: measure things like histogram like sorry, like averages or even
Speaker 2: more importantly, percentages like the median or the ninety eighth
Speaker 2: percentage or stuff like that, then you use a metric
Speaker 2: called a histogram. Now that might seem surprising why it's
Speaker 2: called a histogram if it's used to measure percentiles for example,
Speaker 2: And I'll explain it. I'll try to explain it in
Speaker 2: a minute. But can you think about when you want
Speaker 2: something like percentiles.
Speaker 1: I would think like memory usage or.
Speaker 2: No memory usage. We talked about counter Well what.
Speaker 1: I what I meant was, you know, or percentage of
Speaker 1: like memory used or percentage of resources used. So what
Speaker 1: they have is something I'm not understanding.
Speaker 2: What it is, so I'll give it. I'll give examples,
Speaker 2: and I think that then it will click for you.
Speaker 2: So think about something like request duration, like oh I gotcha,
Speaker 2: So you want to say, like my median request I
Speaker 2: got yeah, ndering duration is x or the ninetieth percentile
Speaker 2: is why. Another example might be the response size, So
Speaker 2: you might say my average response size or my median
Speaker 2: response size is such and such, it goes up to
Speaker 2: something else when I'm looking looking at the ninety ninth percentile.
Speaker 2: So you kind of want to be able to get
Speaker 2: the measurements but then use them in order to calculate,
Speaker 2: as I said, percentiles right now, though, it's called a
Speaker 2: histogram because the way that it's actually implemented is that
Speaker 2: internally you create buckets. You can say, let's say, like
Speaker 2: let's say you're talking about request duration. You'd say, I
Speaker 2: have a bucket from zero to ten milliseconds, from ten
Speaker 2: million seconds to fifty million seconds, from fifty million seconds
Speaker 2: to one hundred million seconds, and anything above one hundred
Speaker 2: million seconds, okay, And each measurement goes into one of
Speaker 2: these buckets, and then you look at how many fell
Speaker 2: into each one of those buckets, and then you can
Speaker 2: use it to calculate percentiles.
Speaker 1: Right, No, that makes sense.
Speaker 2: So it's like a dist for each point in time. Right.
Speaker 1: The thing that I'm imagining is anything that you would
Speaker 1: put like on a bell curve or something like that, right,
Speaker 1: And so then then you're, yeah, your ninetieth percentile is
Speaker 1: you know, out on the end.
Speaker 2: Right.
Speaker 1: But so these are kind of rare, but they're also
Speaker 1: kind of the extreme that you have to deal with. Right,
Speaker 1: So it's a histogram.
Speaker 2: It's a histogram that gets recorded and updated at every interval,
Speaker 2: so it's like a histogram over time.
Speaker 1: Oh interesting, so you could you could literally see like
Speaker 1: the hump move or whatever exactly.
Speaker 2: You say, this is my histogram, now, this is my
Speaker 2: Instagram a minute later, this is my Instagram another minute later,
Speaker 2: and so interesting. Okay, so hopefully that's kind of clear.
Speaker 2: Now there's another metrical summary. I don't want to get
Speaker 2: into it. It's it's it's more rarely and uncommonly used,
Speaker 2: so we'll skip it for now. You know, we're starting
Speaker 2: to run long in any event, because what I really
Speaker 2: wanted to talk about was the concept of labels and dimensionality.
Speaker 2: So let's go back again to that field example of
Speaker 2: the farmer wanting to measure let's say, humidity and temperature
Speaker 2: in the field in their field. But it's a big field,
Speaker 2: so they're not using just a single measurement device. They've
Speaker 2: got devices spread out in different locations in the field.
Speaker 2: So one way in which you might think about it
Speaker 2: is that we'll create a separate metric for each one
Speaker 2: of those measurement devices. But the way that Prometheus looks
Speaker 2: at it is that we've got a single temperature metric,
Speaker 2: but for each measurement we associate labels with it, and
Speaker 2: that label is textual value. It might be the name
Speaker 2: or the idea of the specific measurement device. So let's
Speaker 2: say I've got four devices in the field. They're called
Speaker 2: ABC and D. I've got the single temperature metric, but
Speaker 2: each measurement is associated with either A or B or
Speaker 2: C or D. Mm hmmm is that clear?
Speaker 1: Yep.
Speaker 2: So now what's the benefit of this approach is that
Speaker 2: it's very dynamic. If if I, if I add and
Speaker 2: if I you know, if I buy some more land
Speaker 2: and I now need also an E N f U measurements,
Speaker 2: I don't need to create new metrics. It's the same metric,
Speaker 2: but it's associated, but there are the new values coming
Speaker 2: in are associated with new label values. So I've got
Speaker 2: a label called you know, the device name, but it
Speaker 2: can come in with any any arbitrary textual value. And
Speaker 2: then when I querry the data, I can say, show
Speaker 2: me just the values for the metric for measurement for
Speaker 2: when the label is equal to A or to A
Speaker 2: or B or A and B because when you do
Speaker 2: queries in the prom ql query language, you can either
Speaker 2: set a label like in the query to have a
Speaker 2: specific value or use a regular expression.
Speaker 1: Okay, So I'm imagining that you could do this if
Speaker 1: you're spinning up say another doctor container or server exactly.
Speaker 2: So you might have the pod name. You would have
Speaker 2: a label called pod name, and the value is the
Speaker 2: name of the pod, right yep. And you can have
Speaker 2: any number of labels associated with each metric, and those
Speaker 2: labels can have any number of values.
Speaker 1: Okay.
Speaker 2: Now that I think highlights the potential issue of dimensionality
Speaker 2: because it makes the system very flexible. But you'll see
Speaker 2: the problem in a minute. So let's say I have
Speaker 2: a metric and I have three labels associated with it.
Speaker 2: So it's actually they defined a three dimensional space for
Speaker 2: that metric because each measurement has a coordinate that is
Speaker 2: specified by the values of those three labels. Okay, is
Speaker 2: that clear? I'm kind of waving my hands, think about it.
Speaker 2: So think about again, going back to the field example,
Speaker 2: and let's say that instead of having each measurement in
Speaker 2: the field have its own aim, it has a coordinate
Speaker 2: in the field, so it has an x access, an
Speaker 2: x value, and a y value, So it's a two
Speaker 2: dimensional space.
Speaker 1: Okay.
Speaker 2: Now, now in reality, like again, when you're measuring things,
Speaker 2: you might have many more labels than that, so it
Speaker 2: becomes an n dimensional space. Right And if for any
Speaker 2: so and the the the for each access, the number
Speaker 2: of points on that access is the number of different
Speaker 2: values that label might have. Okay, So let's say I
Speaker 2: have three labels and each one has ten potential values?
Speaker 2: How many how many points? How many different numbers in
Speaker 2: that space?
Speaker 1: Can I have thousand?
Speaker 2: Exactly? So I need to remember one thousand different numbers.
Speaker 1: For that single metric, right and make sense of it hopefully.
Speaker 2: Yeah, But the problem is that if I'm not careful
Speaker 2: with the number of labels that I used, and especially
Speaker 2: the number of different values that each label might have,
Speaker 2: I can literally blow up my memory. Right and again
Speaker 2: going back to how Prometheus works, that Prometheus pools that
Speaker 2: say you have a note server, and Prometheus pulls the
Speaker 2: data from the node server once every minute. So no,
Speaker 2: that node server keeps all that data in its memory.
Speaker 2: Oh okay, if you're not careful, you'll blow up notes memory.
Speaker 2: If you've got if you've got so I'll given a
Speaker 2: real example. We were using Prometheus to monitor data associated
Speaker 2: with the with the page performance at her next insurance,
Speaker 2: and so one of the labels was the r L.
Speaker 1: H.
Speaker 2: We had something like two thousand different pages, Okay, so
Speaker 2: we had a label call. So one of the dimensions
Speaker 2: was was the r L and it had two thousand
Speaker 2: possible different values. Actually, if we weren't careful, it could
Speaker 2: have had a lot worse if we didn't filter out parameters,
Speaker 2: query parameters and stuff like that exactly. And another label
Speaker 2: that we had was device type because we wanted to
Speaker 2: distinguish between desktop and mobile. And another label that we
Speaker 2: had was the browser the browser type because we wanted
Speaker 2: to distinguish between performance at say on chroman on Safari. Right,
Speaker 2: So think about it. It's two thousand different URLs times
Speaker 2: ten different browsers, times three or four different types of
Speaker 2: devices or five types of devices, and all of a sudden,
Speaker 2: and for each one of those three dimension coordinates, you
Speaker 2: need to remember a metric, so it's a number. So
Speaker 2: it's millions of numbers that I need to that you're
Speaker 2: holding on in memory, and we literally kind of blew
Speaker 2: up the memory space. So yeah, so you need to
Speaker 2: be careful when when you're like Cavalier about the number
Speaker 2: of labels and dimensions that you're using. So it's called cardinality,
Speaker 2: and basically you need to be beware of high cardinality. Right.
Speaker 2: But the the benefit is that that system is extremely flexible.
Speaker 2: Like if there's a new if there's a new r L,
Speaker 2: you don't need to do any modification in the system.
Speaker 2: It automatically adjusts to that because it just associates a
Speaker 2: label value with that particular and another label value with
Speaker 2: that particular metric. Right now, there's all sorts of things
Speaker 2: about naming conventions. I won't go into that about how
Speaker 2: you name your metrics and how you name your labels,
Speaker 2: I won't go into that. I will say that there's
Speaker 2: a really cool API for working with all the different
Speaker 2: UH metric types. So we talked about counter engage and histogram,
Speaker 2: so you can there's a if you import the prom
Speaker 2: UH PROM Client uh NPM package, then you get those
Speaker 2: APIs that you can push your numbers with associated labels
Speaker 2: into the system and they just get recorded. Right. So again,
Speaker 2: you might have let's say, UH some sort of business
Speaker 2: logic process that UH that you know hits does all
Speaker 2: sorts of things go to it goes to one database,
Speaker 2: goes to another database does all sorts of things, and
Speaker 2: you want to measure its duration overall. Or maybe you've
Speaker 2: got your own like custom queue for something and you
Speaker 2: want to measure the size of that queue, or you've
Speaker 2: got something like that or a pool that you want.
Speaker 2: Let's say you're you're you're utilizing your own custom pool
Speaker 2: of resources, and you want to make sure that you've
Speaker 2: made the pool not too big, not too small. Then
Speaker 2: again you could be looking at the size of that
Speaker 2: pool and how many what's what's its utilization at each
Speaker 2: point in time, So you can, through those APIs measure
Speaker 2: your own business logic values. And the example that I
Speaker 2: gave is we were actually using it to measure core
Speaker 2: web vitals. If you remember, we've had a lot of
Speaker 2: uh yeah, exactly. Now, those are metrics that are actually
Speaker 2: measured on the browser side. So you might ask how
Speaker 2: did they make their way into PROMI. Well, we would
Speaker 2: collect this data on the client side, post it to
Speaker 2: the node server, and the node server would use that
Speaker 2: API to put it into Prometheus. Right, So this way
Speaker 2: you can collect data not just from the node side,
Speaker 2: but also from the browser side. Again, it could be
Speaker 2: a system level stuff like corere vitals, or it could
Speaker 2: be applicative stuff. You know, whatever your web application happens
Speaker 2: to be doing that you would like to measure and monitor.
Speaker 1: That's cool.
Speaker 2: Yeah. So now the final piece that I wanted to
Speaker 2: mention is how to get data out of Prometheus. So,
Speaker 2: as I mentioned, there's a query language called prom ql.
Speaker 2: That's prom Ql, and like I said, it's definitely not SQL.
Speaker 2: It can do stuff like factoring data, aggregating data, run
Speaker 2: all sorts of predictive functions tech quantiles, which is a
Speaker 2: fancy name for percentiles and averages and whatnot. And it
Speaker 2: can be used to answer questions like what's the ninety
Speaker 2: fifth pers centile latency in each data center over the
Speaker 2: past months, right, you know, so very sophisticated queries, or
Speaker 2: how full will the discs be in four days? So
Speaker 2: here's an example of a predictive query, or which servers
Speaker 2: are the top five users of CPU? All these sorts
Speaker 2: of queries, and you can either use these queries in
Speaker 2: order to graph things or in order to create alerts.
Speaker 1: M okay, And that's what Rawana does. Yes, it uses
Speaker 1: Promql to do this stuff exactly.
Speaker 2: So in Gafana, what you do is you create a graph,
Speaker 2: and in the graph you specify Prometheus as the data source,
Speaker 2: and you write the prom ql query, and then it
Speaker 2: basically graphs that query over time m and it has
Speaker 2: all sorts of fancy graphs, so you can do like
Speaker 2: line sharts and bar charts and heat maps and all
Speaker 2: sorts of really fancy, fancy graphs, and it's really powerful.
Speaker 2: If you're interested in this sort of stuff, I highly
Speaker 2: recommend just going into YouTube, let's say, and searching for
Speaker 2: for pomql. There's also on the Prometheus website there's like
Speaker 2: there's documentation and tutorials for prom ql as well, so
Speaker 2: we can post links to that later on in the
Speaker 2: in the notes. Obviously, if this was a talk, then
Speaker 2: I would actually be showing examples of a prom ql
Speaker 2: but you know, I can't really do that. But exactly so,
Speaker 2: you would put prom ql in one of two places,
Speaker 2: either in something so actually one of three places. So
Speaker 2: the Prometheus itself has its own simple web interface where
Speaker 2: you can put in prom q ol queries and get
Speaker 2: data either in tabular as basically just numbers tabular representation
Speaker 2: of numbers or in simple graphs. If you want much
Speaker 2: more sophisticated graphs, then you can use gfana, and gfana
Speaker 2: actually has a pretty sophisticated editor for prom ql, so
Speaker 2: it does automatic completion for you. It knows all the
Speaker 2: metrics in the system and the labels and the different
Speaker 2: label values, so it can actually do autocomplete for you
Speaker 2: when you're typing in the queries. Yeah, it's it's it's
Speaker 2: really nice. It's as I think I said, it's kind
Speaker 2: of they're kind of sister projects, so it's it's the
Speaker 2: same people working on both of them, so they made
Speaker 2: sure that the integration is really nice. And you can
Speaker 2: also use the prom ql in the alert manager where
Speaker 2: you would type the queries in the YAMO files and
Speaker 2: then you they get run automatically at periods and if
Speaker 2: they come back with data that matches the query, then
Speaker 2: an alert gets raised. So you run a query. If
Speaker 2: it comes back no data, no alert. If it comes
Speaker 2: back with data, then there's an alert. So the an
Speaker 2: alert query might be do I the number of CPUs
Speaker 2: that are at utilization higher than eighty percent? Right? And
Speaker 2: if the number is zero, then there's no alert. If
Speaker 2: the number is greater than zero than there would be
Speaker 2: an alert makes sense something like that, And it's actually
Speaker 2: even more sophisticated than that because you can say you
Speaker 2: want to avoid like peaks, momentary peaks, so you might
Speaker 2: say it needs to be higher than eighty percent over
Speaker 2: a period of two minutes.
Speaker 1: Right, stuff like that makes sense.
Speaker 2: Yeah, so it's it's a very sophisticated and powerful system.
Speaker 1: So is this something it sounds like something used at
Speaker 1: next Insurance. Is this something that you're using.
Speaker 2: Well, yeah, we're also using it as license. So for example,
Speaker 2: when we deploy our service, our customers have the ability
Speaker 2: to run our services on premises and then they can
Speaker 2: collect monitoring data for our own services inside an instance
Speaker 2: of Prometheus that do that we installed with it.
Speaker 1: That's cool.
Speaker 2: But like I said, it's a really powerful system and
Speaker 2: and it's really flexible. And again if you want to
Speaker 2: do monitoring or alerting, probably want to do both, then
Speaker 2: it's a very good solution for that.
Speaker 1: Nice and that more or less covers it.
Speaker 2: Like I said, I have a talk presentation on it
Speaker 2: in which I, you know, I am able to show
Speaker 2: actual examples and and and and visuals, so it's you know,
Speaker 2: more more more informative if you're interested in that, if
Speaker 2: you're running a conference and you're interested in that, I'm
Speaker 2: shopping this talk out.
Speaker 1: Nice. Well, I mean this is something that I've kind
Speaker 1: of been contemplating figuring out how to put into my
Speaker 1: own systems, right, I mean, I I send data into
Speaker 1: I have a self hosted version of Century is the
Speaker 1: thing I've been using lately, and it collects some of
Speaker 1: this kind of data. But you know, I've gotten more
Speaker 1: and more into self hosting my own kind of thing
Speaker 1: and and running through some of this, right, and it
Speaker 1: sounds like this this is right kind of right up
Speaker 1: that alley too, where it's okay when I deploy right,
Speaker 1: it just you know, it makes sure that I have
Speaker 1: a Grafana and a Prometheus instance setup that it just
Speaker 1: reports to exactly.
Speaker 2: Now, obviously there are overlap between these sort of systems,
Speaker 2: but I think their focus is kind of like not
Speaker 2: exactly the same. So let's say, if you're doing like
Speaker 2: error logging, then you know Century is the tool for you.
Speaker 2: I think I said that Prometheus is not the appropriate
Speaker 2: tool for textual information. So if you're looking and stuff
Speaker 2: like keeping stuff like stack traces and stuff like that,
Speaker 2: then you want to use something like Centric. You know,
Speaker 2: you could theoretically count the occurrences of a particular type
Speaker 2: of error, but yeah, you probably want to use Centric
Speaker 2: for something like that. Also, Centriy has the stuff that's
Speaker 2: specifically intended for performance monitoring a certain scenarios. But if
Speaker 2: you want to do more general purpose monitoring and system
Speaker 2: level monitoring and and monitoring applicative stuff, then Prometheus could
Speaker 2: be the solution for you.
Speaker 1: Right. I also like that, you know, as it aggregates
Speaker 1: that data effectively, what you can learn from it is
Speaker 1: only limited by I guess what it's measuring, but also
Speaker 1: what you can come up with the query out of it.
Speaker 2: Oh yeah, for sure, I created some amazing query using
Speaker 2: prom QA. Like so I'll give that example. So at
Speaker 2: the next insurance, we had something like fifty services in
Speaker 2: a service oriented architecture which we're communicating with each other
Speaker 2: over various API end points, and we wanted alerts to
Speaker 2: be fired when, certainly when any endpoint in the system
Speaker 2: became too slow. But then the question was what does
Speaker 2: too slow mean? Because you know, it can't be absolute numbers,
Speaker 2: because if something it usually takes fifty milliseconds, and then
Speaker 2: all of a sudden it grows to one hundred milliseconds,
Speaker 2: that means it's too slow. But if something always runs
Speaker 2: at one hundred milliseconds and then becomes one hundred and ten,
Speaker 2: well that's a smaller change than you might want to
Speaker 2: ignore it. So you can't look at absolute numbers. So
Speaker 2: we I created really sophisticated queries that basically looked at
Speaker 2: behavior over time and tried to see if something became
Speaker 2: significantly slower compared to its own specific behavior from you know,
Speaker 2: previous durations. And also again wanted to only look at
Speaker 2: those endpoints which were sufficiently used, because if there was
Speaker 2: some endpoint that was hardly ever used and we probably
Speaker 2: don't care about it, right, So yes, so you can
Speaker 2: create really sophisticated queries like that. And it was exactly
Speaker 2: because with all these end points, you don't want to
Speaker 2: have to tell somebody a developer where well, you know,
Speaker 2: you're in charge of a service that has one hundred
Speaker 2: different API en points. You need to look every day
Speaker 2: at one hundred different graphs to figure out if there's
Speaker 2: a problem. You want it so you want you want
Speaker 2: an alert to be sent to their right to the
Speaker 2: Slack channel. Let's say, of that particular team, if any
Speaker 2: one of those end points all of a sudden became
Speaker 2: slower in a way that really impacts potentially impacts the
Speaker 2: system as a whole. Right, and and yeah that's what
Speaker 2: that's one of the projects that I did at Next Insurance,
Speaker 2: and we use cool and we use Prometheus and go
Speaker 2: fun of that.
Speaker 1: Very cool.
Speaker 2: So when when alert like that was sent to the
Speaker 2: Slack channel, the on call member of the team could
Speaker 2: actually click a link see the graph for that the
Speaker 2: performance of that endpoint in GOFANA over time, and say, okay,
Speaker 2: we actually have a problem, or no, this is just
Speaker 2: you know, a fluke or something that we can ignore.
Speaker 1: Yeah, yeah, it spiked for such and such a thing
Speaker 1: and no big deal.
Speaker 2: Yeah we were doing something. We ran some sort of
Speaker 2: a backup service and you know that's why it impacted everything.
Speaker 1: And right, or you migrated some data on the back
Speaker 1: end and it slowed it down for two minutes.
Speaker 4: And then exactly yeah, awesome, all right, Well, we put
Speaker 4: some links into the the comments on twitch and YouTube
Speaker 4: and Facebook if you want to go find those.
Speaker 1: We'll try and get them into the show notes as well.
Speaker 1: But let's go ahead and do some pics.
Speaker 2: For sure, although I don't have that many. Okay, I
Speaker 2: actually do have something. So for some inexplicable reason, I
Speaker 2: decided to start tweeting out my favorite standalone fantasy novels
Speaker 2: or books. You know, fantasy tends to be written as
Speaker 2: a lengthy series of books or at least trilogies, but
Speaker 2: occasionally I just want to read one book that stands
Speaker 2: on its own, merit on its own, because you know,
Speaker 2: it starts and it ends and you can move on
Speaker 2: to something, to the next thing. So I started to
Speaker 2: tweet out the collect my list of the top standalone
Speaker 2: fantasy books, and you know what, I'll use each I'll
Speaker 2: pick each one at the time. So let's see which
Speaker 2: one one was my first? Can you think of one?
Speaker 2: There's the obvious choice, by the way, which is I
Speaker 2: think the Hobbit's I.
Speaker 1: Was thinking that, but then I was like, I don't
Speaker 1: know if he considers that part of a series, because
Speaker 1: it is sort of a prequel to Fellowship of the
Speaker 1: Ring and that series.
Speaker 2: Yeah, yeah, I think it's.
Speaker 1: Yeah.
Speaker 2: Yeah. First of all, it's totally self contained, and the
Speaker 2: second important thing is that it was written well before
Speaker 2: a lot of the Rings was written. Yes, So Tolkien
Speaker 2: actually wrote a lot of the Rings because there the
Speaker 2: publisher was so happy with the success of The Hobbit
Speaker 2: that they wanted more stories in that world, and it
Speaker 2: was his idea to transform it from a child's story
Speaker 2: to a story for grown ups. But so the Hobbit
Speaker 2: was originally written for his written for his children, although
Speaker 2: a lot of adults like that story as well. Oh yeah,
Speaker 2: you know what, I'll tell you another interesting story about
Speaker 2: about the Hobbit. So there's a So, there are a
Speaker 2: couple of translations of The Hobbit into Hebrew. But the
Speaker 2: one of the translations is especially interesting. It's called the
Speaker 2: Pilot's translation because because during so there was a war
Speaker 2: between Israel and Egypt in the early in the early seventies,
Speaker 2: like in between the Sixth Day War and the Young
Speaker 2: Keeper War that was kind of it's called the War
Speaker 2: of Attrition, and various pilots were down and captured by
Speaker 2: Egyptians and held as POWs for several years, kind of
Speaker 2: like the American peow's in Vietnam, and they were looking
Speaker 2: for ways to pass the time, and one of them
Speaker 2: got their hands on the English version of The Hobbit.
Speaker 2: The original Hobbit, and and they decided to translate it
Speaker 2: into Hebrew as a way to pass the time. So
Speaker 2: and when they were released, they they took their the
Speaker 2: translation with them and it literally got published and it's
Speaker 2: called the Pilot's Translation Translation of the Hobbit from from
Speaker 2: English into Hebrew. So anyway, so the Hobbit would be one.
Speaker 2: So but because that's the obvious one, I'll give another one,
Speaker 2: and that that book is called Tigana. It's by Guy
Speaker 2: gravel K. Interestingly, to kind of close the loop. He's
Speaker 2: the person that worked with the with Tolkien's son on
Speaker 2: on bringing it on, you know, publishing the Simarillion. So
Speaker 2: they had a lot of notes over. Yeah, they had
Speaker 2: a lot of notes over from Tolkien. So they actually
Speaker 2: took all those notes and and kind of rounded out
Speaker 2: and filled up the story and then released it as
Speaker 2: a Simarillion after Tolkien had died, and he worked with
Speaker 2: it on it with Tolkien's son, But he also wrote
Speaker 2: several books on his own. One of them is a
Speaker 2: standalone novel called Tigana. You might actually find it especially
Speaker 2: interesting because it's very much inspired by by the Renaissance Italy,
Speaker 2: so the setting there, it's a fictional world, obviously with
Speaker 2: magic and you know, and stuff like that, but it's
Speaker 2: it's the situation. The scenario is very reminiscent of Italy
Speaker 2: in the Renaissance, with the little country of warring principalities
Speaker 2: and influenced by by larger external powers and and so
Speaker 2: on and so forth, and and the culture and whatnot,
Speaker 2: and and it's it's a great book. It's amazing the
Speaker 2: amount of story and settings and scenery and characters that
Speaker 2: he was able to cram into this one book. It's
Speaker 2: a pretty thick book, but still one book. And it's
Speaker 2: very highly recommended. As I said, it's called Tigana. That's
Speaker 2: t I G A n A by Guy gabrielle K
Speaker 2: And that would be my pick for today.
Speaker 1: Awesome. I'll have to check that one out. We were
Speaker 1: watching the Lord of the Rings movies and my daughter
Speaker 1: was saying that she'd never read them, and so I
Speaker 1: had it on audible and so she's been listening to it.
Speaker 2: So but if she's already watched the movie, can she
Speaker 2: can she get into it?
Speaker 1: I think so. She's pretty into other like Harry Potter
Speaker 1: and Percy Accent, and she she got into the books too.
Speaker 1: After she'd seen the movies. So yeah, I'm gonna jump
Speaker 1: in with some picks. Now. I have to say my
Speaker 1: board game group hasn't gotten together in a little bit.
Speaker 1: It's just it's just kind of that season of life.
Speaker 1: But uh, what I'm trying to think of a game?
Speaker 1: I should pick? What was the game that we played
Speaker 1: last Sunday with with the kids? I don't remember anyway
Speaker 1: that I mean, there are all kinds of games out
Speaker 1: there that you can play. I'll just go with one
Speaker 1: of my favorites. It's funny because I've never actually completely
Speaker 1: played through this game. No, I take it back, I
Speaker 1: have played through it once. It's called.
Speaker 2: Monopoly. Does anybody ever play Monopoly all the way through?
Speaker 1: My My kids do all the way through.
Speaker 2: Yeah.
Speaker 1: I haven't played Monopoly in a long time. There there
Speaker 1: are reasons that I don't.
Speaker 2: I played nostalgia, but notably because it's a it's not
Speaker 2: such a fun game. Yeah.
Speaker 1: The game that I'm thinking, yeah, well, there there are
Speaker 1: a number of issues that make it anyway.
Speaker 2: I think I seem to recall that somebody somebody once
Speaker 2: told me that the original motivation for the creation of
Speaker 2: the game Monopoly was to show them to prove the
Speaker 2: futility of capitalism.
Speaker 1: Yeah, well it's not. It's not pure capitalism.
Speaker 2: So yeah.
Speaker 1: Anyway, So the game that I'm gonna pick has called
Speaker 1: Letters from Whitechapel, and I understand it's like Scotland Yard,
Speaker 1: but I've never played Scotland Yard. So one player plays
Speaker 1: as Jack the Ripper, and then the other players play
Speaker 1: as the police deputies. So they're trying to catch Jack.
Speaker 2: Is it kind of like glue or something.
Speaker 1: No, in the sense that you're not you're not trying
Speaker 1: to figure out who did it or anything like that.
Speaker 1: You're it's literally So what the way it works is
Speaker 1: you have the women that Jack the Ripper kills, and
Speaker 1: so when he murders somebody, I just dropped the link,
Speaker 1: but I didn't label it. When he murders somebody in
Speaker 1: the game, he has so many moves that he can
Speaker 1: make to get back to his hideout. And you play
Speaker 1: it in five rounds, and so the police deputies are
Speaker 1: trying to block him off, and so they'll go investigate
Speaker 1: different spots on the board, and there are hundreds of
Speaker 1: spots on the board, and so they'll investigate a spot
Speaker 1: and they'll either find a clue, which means that Jack
Speaker 1: was there, and Jack writes down when he moves through
Speaker 1: a space, he writes it down on his thing. And
Speaker 1: one of my favorite things to do when I'm Jack
Speaker 1: the Ripper is actually to loop back around and so
Speaker 1: they'll find clues. They'll find clues all around this spot
Speaker 1: where I went through twice, and so then they don't
Speaker 1: know which direction I went coming out of there. But anyway,
Speaker 1: if you manage to get back to your height out
Speaker 1: after all five rounds, then you win as Jack the Ripper.
Speaker 1: And then if if the deputies investigate a spot in
Speaker 1: Jack's there and they have to specifically say that they're
Speaker 1: staging in arrest at that spot that if they do
Speaker 1: that properly where Jack is at, then then they win.
Speaker 2: And I have to tell you that the sub the
Speaker 2: subject matter seems a bit dark for a game. I
Speaker 2: have to tell you it is, but.
Speaker 1: The the overall gameplay is is fun. And so what
Speaker 1: happens is you start to narrow down where the hideout
Speaker 1: is right, and so you can you can then start
Speaker 1: to uh, you know, where he's going, and so wherever
Speaker 1: the murder happens at right, then you can start to
Speaker 1: you know, kind of fan out along where he might
Speaker 1: travel through and get get a sense of where he's
Speaker 1: at and then be able to arrest him. And so
Speaker 1: that's that's kind of the gameplay. Let me look it
Speaker 1: up on board game Geek, so I get the weight.
Speaker 1: The reason that I've finished with this guy tried to
Speaker 1: play it with my family, and it's it's kind of
Speaker 1: a longer game. I mean, it can run like two
Speaker 1: hours or longer, and so they just kind of especially
Speaker 1: my kids, just lose interest. And it's not the kind
Speaker 1: of My wife likes to play the kind of light,
Speaker 1: airy games, and this one's a heavier game, and so,
Speaker 1: you know, all this strategizing and stuff, she just doesn't love.
Speaker 1: It has a board game weight of two point sixty four,
Speaker 1: so it is on the heavier side of a game
Speaker 1: that a casual player would play. But anyway, it's it's
Speaker 1: a fun one. So yeah, I'll pick Letters from Whitechapel
Speaker 1: as my board game pick.
Speaker 2: And then.
Speaker 1: I've been doing a whole bunch of stuff with just
Speaker 1: being more. I took way too long on that pick,
Speaker 1: so I'll be quit brief on these other ones. I've
Speaker 1: been trying to be a little bit more efficient with
Speaker 1: my time. And I've also been working on getting back
Speaker 1: into shape. So you know, I picked a marathon and
Speaker 1: started training for it. So a few picks that I've
Speaker 1: got here that I'm just going to put out there.
Speaker 1: So my training program I do in training Peaks and
Speaker 1: that's just training peaks dot com. I'm gonna put it
Speaker 1: in the comments as well. And effectively, it just gives
Speaker 1: you a calendar. You can buy workout plans. The ones
Speaker 1: I've bought cost anywhere from five dollars all the way
Speaker 1: up to like fifty dollars. I think the marathon and
Speaker 1: triathlon ones were more expensive. But it literally just puts
Speaker 1: the workouts into your thing. And then I have a
Speaker 1: garment for runner watch and so it just sinks it
Speaker 1: down to my watch. So all I have to do
Speaker 1: is tell it go into my training calendar, pull the workout,
Speaker 1: run it through the watch. Right, So I did that
Speaker 1: this morning, went for a run.
Speaker 2: It was awesome.
Speaker 1: As far as being more efficient, I've also picked up
Speaker 1: and been using Linear, which seems to be pretty popular
Speaker 1: in in the development circles. Linear dot app I'll put
Speaker 1: that in the comments as well, And I mean it's
Speaker 1: basically a project management board like any of the other
Speaker 1: ones that you're used to. And then what I've been
Speaker 1: doing is I've been anything that I intend to do
Speaker 1: anytime soon out of linear or if it's other stuff
Speaker 1: that I need to be doing. Like for example, when
Speaker 1: Dan and I were talking before the show, he was like, Hey,
Speaker 1: we ought to get people like this on the show,
Speaker 1: or Hey, I was going to reach out to so
Speaker 1: and so, you know, and I was like, oh, I
Speaker 1: know them, and so as we were having that conversation,
Speaker 1: I put them into another system called Motion and I
Speaker 1: kind of I'll look and see if they have an
Speaker 1: affiliate program, but I'm just going to put the link in.
Speaker 1: It's Usmotion dot Com is the app. And the way
Speaker 1: that this works is so you put your tasks in
Speaker 1: let me put it usemotion dot Com. Sorry I'm typing
Speaker 1: in talking at the same time. So you put your
Speaker 1: tasks in and then it pulls like it pulls from
Speaker 1: my Google calendar, and so it knows when I have
Speaker 1: something scheduled like this episode. And then what it does
Speaker 1: is it says, Okay, I'm going to put the other
Speaker 1: tasks that you've put into Motion into your calendar and
Speaker 1: so and you can set it up so that it'll
Speaker 1: actually add them to the Google calendar, which is what
Speaker 1: I did, and then I told it not to market
Speaker 1: as busy, so they show up as free time. So
Speaker 1: it's got tasks set up for Tuesday, Wednesday, Thursday, and
Speaker 1: Friday this week as well, but they show up as
Speaker 1: free time. And so what that means is is that
Speaker 1: if somebody books a time on my calendar, it'll just
Speaker 1: shift everything around it, and so it essentially tells me
Speaker 1: what to work on. And then finally, the last thing
Speaker 1: that I've been using is Focus Blocks and I'm actually
Speaker 1: going to do another premium episode with Manny who's the
Speaker 1: guy that created it. But effectively, what it does is
Speaker 1: you get on Focus, you get on a zoom call
Speaker 1: and that you do a breathing exercise before you start,
Speaker 1: and you commit to this is what I'm going to
Speaker 1: do this hour. And sometimes I hit it. Sometimes I don't,
Speaker 1: and it's things out of my control, or it turns
Speaker 1: out to be harder than I thought it was going
Speaker 1: to be or whatever. But it keeps me on tasks
Speaker 1: because what they make you do is they make you
Speaker 1: put your phone away from your desk and so you
Speaker 1: focus on it. And then at the end of the hour,
Speaker 1: you do another breathing exercise, you report on whether or
Speaker 1: not you got your thing done, and then you do
Speaker 1: it over again. And I usually get two, three, four
Speaker 1: focus blocks in in an afternoon. I try and schedule
Speaker 1: all the regular stuff in the mornings, but yeah, I'll
Speaker 1: do that in the afternoons and then Yeah, so a
Speaker 1: lot of my like prospecting. So if I'm trying to
Speaker 1: find sponsors or stuff you know that's on their first
Speaker 1: thing in the morning, getting through my inbox is a
Speaker 1: first thing in the morning thing. So after I go
Speaker 1: for a run and things like that, that's.
Speaker 2: What we do.
Speaker 1: So anyway, those are my picks. I think his focus
Speaker 1: blocks dot com and I know I have an affiliate
Speaker 1: link for that, but so we'll put that in the
Speaker 1: show notes and it doesn't cost you anything extra, but
Speaker 1: I get a kick back if you use it. But ultimately,
Speaker 1: I mean, if these things save you a bunch of
Speaker 1: time and make you more effective, great, and then if
Speaker 1: I get a kickback because I found an affiliate link
Speaker 1: for it, even better. But ultimately, this is what I'm using.
Speaker 1: So yeah, that's what I've got for picks. So we'll
Speaker 1: wrap it up until next time.
Speaker 2: Folks back sou