Speaker 1: Hey everybody, and welcome back to another episode of Adventures
Speaker 1: in DevOps. This week on our panel we have Will Button.
Speaker 1: What's going on everyone, We have Jeffrey Growman. Hey there,
Speaker 1: I'm Charles Maxwood from Devchat dot TV. And this week
Speaker 1: we have a special guest and it is Simon Tults.
Speaker 2: Hi. Everyone, it's a pleasure to be here. Thank you
Speaker 2: very much for hosting me.
Speaker 3: So fun Yeah, absolutely, I mean you brought all the energy.
Speaker 3: It's it's funny. We were talking beforehand and you're just
Speaker 3: excited and I love it. Do you want to just
Speaker 3: introduce yourself real quick, let people know who you are,
Speaker 3: what you do at? Is it da Tree, Dat Tree.
Speaker 2: The Tree? Yeah? Sure, yep. So yeah. My name is Shimon.
Speaker 2: I've been in the infrastructure R and D space for
Speaker 2: more than ten years, and I've worked at large companies
Speaker 2: like Intel, and i worked at startups from like thirty
Speaker 2: employees until we were a thousand employees. And my previous role,
Speaker 2: I was an engineering manager for a media company and
Speaker 2: I grew together with the organization from thirty employees to
Speaker 2: one thousand, and I really saw how the struggle is
Speaker 2: real when you have four hundred engineers and you're trying
Speaker 2: to make this work while breaking things and moving fast.
Speaker 2: So this is actually what brought us here and what
Speaker 2: brought me to almost four years ago, me and my
Speaker 2: co founder to open the Tree, which helps prevent misconfigurations
Speaker 2: from ever reaching production environments, especially around kugernities.
Speaker 3: All right, so the code that crashes my stuff, that's
Speaker 3: my fault. The misconfigurations that's somebody else's fault. I'm just
Speaker 3: being clear. I hope my boss hears this anyway. So yeah,
Speaker 3: so we were talking before the episode and you said
Speaker 3: that you experienced this outage and this is where you've
Speaker 3: learned a lot of the lessons that led to the tree.
Speaker 3: Do you want to just talk about that? Kind of
Speaker 3: give us the background and the story so that we
Speaker 3: know what you screwed up? I mean how that went?
Speaker 2: Sure, sure thing. So let me describe the scenery first,
Speaker 2: you know, like in a book. So imagine a company
Speaker 2: and you have one thousand employees, four hundred of them
Speaker 2: are developers. The company is born in the cloud, paying
Speaker 2: one point five million dollars for AWS every month, a
Speaker 2: lot of stuff running. No one actually knows what is everything,
Speaker 2: but it kind of works, you know, and moving really
Speaker 2: really really fast working in micro services, a lot of
Speaker 2: different programming languages, and the philosophy of the company is,
Speaker 2: you know, like small speed boats, like Amazon calls it,
Speaker 2: two pizza teams. You know, you have your team, you find,
Speaker 2: you have a problem, you find the best solution, you go,
Speaker 2: you do it, responsible for it, sounds great, removes bottlenecks,
Speaker 2: makes you move fast, and really gives great energy people
Speaker 2: because they don't feel like they're a small, small screw
Speaker 2: in a big organization. And my role there was the
Speaker 2: general manager of the infrastructure division. So my work was
Speaker 2: to find things that are relevant to all the other
Speaker 2: teams and build it as an infrastructure. So for example,
Speaker 2: we built a data collection pipeline that ingested more than
Speaker 2: two hundred billion with the b every month events from
Speaker 2: thirteen regions in AWS. And this is what we would do.
Speaker 2: And every time there's a new technology or something that
Speaker 2: is cross the company, we would be responsible for it.
Speaker 2: Might team and sort of like sort of special ops
Speaker 2: in a way. And one day we had a production outage.
Speaker 2: Now this happens. You know, we're all people. I make mistakes.
Speaker 2: Everyone makes mistakes, besides Charles. It happens, and like every
Speaker 2: company we said, okay, let's post more tom the problem,
Speaker 2: understand what happened, let's find the root cause. And we
Speaker 2: did it. And a developer made a miss configuration in
Speaker 2: one manifest file, and we said, okay, we totally understand
Speaker 2: people make mistakes, and we want we believe in the
Speaker 2: philosophy of, you know, run fast and break things, but
Speaker 2: we don't believe in the philosophy of let's make the
Speaker 2: same mistake five times. You know there is a limit
Speaker 2: to that as well. So we said, okay, so how
Speaker 2: do we make sure that this does not happen again.
Speaker 2: So first of all, you send the post mortem in
Speaker 2: an email to everyone in the company. So we tried that.
Speaker 2: Nice doesn't really work. No one reads it, no one
Speaker 2: remembers it. And I got to tell you from the
Speaker 2: other side as a developer getting emails every day telling
Speaker 2: me like, use this package, use this configuration, check this thing.
Speaker 2: It's it's not scalable, Like how am I supposed to
Speaker 2: remember everything? It's it's just not feasible. And we said, okay,
Speaker 2: we did, you know, internal educational systems, And we did
Speaker 2: an internal meetup and explain to everyone, and everyone agreed,
Speaker 2: and everyone understands but it didn't really work well because
Speaker 2: I think, and this is what we thought, and this
Speaker 2: is what drove us to actually open the company, is
Speaker 2: that it has to happen in an automated way within
Speaker 2: the development flow of the developer. Because every inch, every
Speaker 2: small thing that you do in order to change the
Speaker 2: workflow of the developer, it's crazy. It's almost never going
Speaker 2: to happen, and if it's going to happen, it's going
Speaker 2: to be very very painful for the developer, for the manager,
Speaker 2: and for the company. So he said, how can we
Speaker 2: do something that will be seamless in the flow, Because
Speaker 2: when we spoke with people, they said, I want to
Speaker 2: know when I'm doing something wrong. I don't want to
Speaker 2: be the person that submits secret keys into our public
Speaker 2: GTAB repository. I don't want to be the person that
Speaker 2: takes production down. But sometimes I just don't know. And
Speaker 2: this is what drove us to actually opening the tree
Speaker 2: and building a solution that hooks directly within the development workflow.
Speaker 2: So it's a cli utility. You can run it on
Speaker 2: your laptop, Linux and Mac. Just run the three tests
Speaker 2: on your Kubernities manifest file or helm file, and then
Speaker 2: we provide out of the box pre defined policies. So
Speaker 2: I'll give you simple examples that seems, you know, really trivial,
Speaker 2: but people don't do it. It's like memory limit SIPY.
Speaker 2: You limit a likeness probe, readiness probe, pulling containers from
Speaker 2: a centralized registree, not using the latest doctor tag. Because
Speaker 2: then every time you build it, it's like going to
Speaker 2: the casino. You don't know it's version. You're gonna get what,
Speaker 2: You're gonna have it any productions like you know, and
Speaker 2: next after you have it in your computer, you install
Speaker 2: it in your CICD. And at this point this is
Speaker 2: one of the most powerful things because you get a
Speaker 2: centralized policy management solution. So I, as the develops engineer,
Speaker 2: can identify a problem, think of a policy that I
Speaker 2: want to apply to all of my hundred or fifty
Speaker 2: or five thousand engineers, and with the click of a button,
Speaker 2: I can enable policy. And now all the projects that
Speaker 2: go through this CICD pipeline will actually comply with this
Speaker 2: policy and otherwise it will fail. And the idea is
Speaker 2: that once it fails, it does not notify the develops person.
Speaker 2: It explains the developer what do they have to do
Speaker 2: and shows them and links them to wiki and to
Speaker 2: our dogs and tells them, hey, mister developer, Hey missus developer,
Speaker 2: this is how you can fix it. So we are
Speaker 2: very very proud of it because I really believe that
Speaker 2: this is how I would want my organization to communicate
Speaker 2: those policies and practices to me as a developer, as
Speaker 2: an engineer.
Speaker 1: Quit telling me what to do. No, it makes sense,
Speaker 1: and to be perfectly honest, you know. So, yeah, I
Speaker 1: write web developed. I'm a web developer for a fairly
Speaker 1: large financial firm. And what's nice is a lot of
Speaker 1: this stuff does kind of get pushed into our CICD.
Speaker 1: But the other nice part of it is that generally
Speaker 1: when these kinds of policy changes come down and I
Speaker 1: don't think they're using the tree. I think they're using
Speaker 1: just we're making this policy change and we're configuring Jenkins
Speaker 1: to do it. But they generally are pretty good about
Speaker 1: going in and making the initial move right, so they
Speaker 1: move it to the to match the policy, and then
Speaker 1: from there when it lines up with whatever we're doing.
Speaker 1: That's when we get to the point where it's like, okay,
Speaker 1: So then if we change something that messes it up right,
Speaker 1: then it's on us. Okay, we can roll this back.
Speaker 1: But yeah, they're usually the ones that initially make it comply.
Speaker 1: And I just wanted to add that because I think
Speaker 1: there's some level of responsibility that goes both ways, and
Speaker 1: so that's what I like about this. But if you're
Speaker 1: the one that's making the initial change that's going to
Speaker 1: cause it to fail and SI, then you probably also
Speaker 1: ought to be the person that's either working with somebody
Speaker 1: or doing the work yourself to make it comply in
Speaker 1: the first place.
Speaker 2: I totally agree, and this is why when we designed
Speaker 2: our policy engine, we design I need to have several
Speaker 2: points of granularity. So first of all, you can see
Speaker 2: how am I doing. Now let's scan my GitHub repositors
Speaker 2: and see do I have any violations now or not. Secondly,
Speaker 2: you can enable a rule in a way that we
Speaker 2: call it gradual rollout. So now every time that they
Speaker 2: make a change that is not complainant, to tell them listen,
Speaker 2: on August first, this change will not be complied. Now
Speaker 2: it is passing, and that's okay, totally fine, but just
Speaker 2: so you know, we're going to have a policy in
Speaker 2: place in August first, and this is the policy, and
Speaker 2: here you have time to actually prepare to it, and
Speaker 2: then once August first hits, then it fails as a
Speaker 2: warning and not as enforcement, and then you have a
Speaker 2: great period for adoption of the policy, and only then
Speaker 2: at the end of the end of the end it
Speaker 2: actually goes to full enforcement. And you're totally right. This
Speaker 2: is the feedback that we got from our customers, and
Speaker 2: this is how they designed it, and we built it
Speaker 2: because this is how they wanted.
Speaker 4: Now that's super cool.
Speaker 5: I really like the approach of You mentioned the path
Speaker 5: of how you got here through the emails and the
Speaker 5: meetings and the workshops and stuff, but really all that
Speaker 5: is only relevant at the time. And doing it this way,
Speaker 5: I think one of the key things there is that
Speaker 5: you're meeting the developers where they are, because that's the
Speaker 5: right time to introduce the solution or the information is
Speaker 5: when it's relevant to them. Otherwise it's just out of context.
Speaker 2: I totally agree. You have to get the warning and
Speaker 2: the data in line. I call it in line. And
Speaker 2: this is why we're working on We have a helm
Speaker 2: plug in, working on a cube, cattle plug in, vias,
Speaker 2: code plug in, everything, and this is very important because
Speaker 2: if it's not convenient, and if it's not in the
Speaker 2: developer's workflow, then I'll give you a story. Okay, I
Speaker 2: met with a big enterprise company. The talk to me
Speaker 2: about the certain policy that we have that says like
Speaker 2: pull containers from the centralized registry of the company. So
Speaker 2: it's like m doctor at company ACME dot com, right,
Speaker 2: and he's like, it's it's a good policy. I want
Speaker 2: to use your solution instead of ours. And I'm like, what,
Speaker 2: what's your solution? So what do you do today? Oh,
Speaker 2: we just blocked docer hobbing our firewall and no one
Speaker 2: can ask it. And I'm like, what.
Speaker 5: Problem solved?
Speaker 2: For real? This is the real really, this is what
Speaker 2: he told me. I was and he's like, yeah, well,
Speaker 2: we just block it in the DNS and firewall level
Speaker 2: and that's it and they can't pull it from there.
Speaker 2: And I think that this is absolutely not the way
Speaker 2: to do it. As we go forward, developers want to
Speaker 2: achieve left nice to play solution.
Speaker 4: Yeah, that works until your developers get smart enough to
Speaker 4: figure out how to use a VPN or other ways
Speaker 4: of Yeah. Yeah, I mean this.
Speaker 2: Is security by obscurity.
Speaker 1: That's yeah, it's the wrong way I was going to
Speaker 1: say Jurassic part nature or developers will find a way.
Speaker 4: Absolutely, you know, just a pile on. I totally love
Speaker 4: this idea too, and it's it really speaks I think
Speaker 4: to like the whole DevOps mentality of like flow and
Speaker 4: like pull requests versus push requests, because I think, you know,
Speaker 4: the way you were describing it earlier was that it's
Speaker 4: basically just you know, pushing stuff out, which never really
Speaker 4: works all that well. But if you, as Will said,
Speaker 4: you get timing right so that now developers can pull
Speaker 4: that information as they need it. The timing is right,
Speaker 4: the method is right, the information is there for them
Speaker 4: to pull it.
Speaker 2: Just you know.
Speaker 4: Again, I know I'm just piling on, but I just
Speaker 4: feel like it actually really fits really nicely and elegantly.
Speaker 4: Then the whole DevOps you know sort of methodology.
Speaker 2: But you know what I got to say, of course
Speaker 2: I gave a very radical example. Well, now, but when
Speaker 2: we meet with companies that are at this crossroad, because
Speaker 2: they say, okay, listen, we scale we had thirty developers,
Speaker 2: it was okay, forty fifth, like now we have like
Speaker 2: seventy developers. It's COVID. We're all working from home. You
Speaker 2: can't come to just a room and ask hey, how
Speaker 2: do we do this and that, and it's like we
Speaker 2: need to put something in place. And then I see
Speaker 2: companies choose two different paths. It's like two opposites that
Speaker 2: you can go to. And of course I think that
Speaker 2: the best solution is the middle ground. But like some
Speaker 2: companies go the old way, the whole way too. Okay,
Speaker 2: So DevOps is responsible for the cluster, which is true
Speaker 2: in many organizations, the responsible for the operational excellence of
Speaker 2: the cluster and for the day to day operations. But
Speaker 2: then the developers write the application. Then what they say
Speaker 2: is okay. So now every change that the developer makes
Speaker 2: to a Kuberniti's manifest or helm or anything that touches
Speaker 2: the infrastructure has to go through the opsteam. Now what
Speaker 2: happens at this point is that there's a huge bottleneck. Eventually,
Speaker 2: it frustrates both sides because the developers they have the
Speaker 2: R and D backlog and the product sitting on them
Speaker 2: with timelines that they need to release stuff and they're
Speaker 2: waiting for the up steam to approve it. The OPS
Speaker 2: team they don't want to babysit developers and tell them listen,
Speaker 2: you forget you're pulling the latest image put up in
Speaker 2: down version. No, because it's not interesting. They want to
Speaker 2: do cost reduction, they want to optimize the performance, they
Speaker 2: want to upgrade it, they want to bring the new
Speaker 2: best versions. They you know, do crazy pocs. And then,
Speaker 2: like all all sides are are basically frustrated because they
Speaker 2: babysit developers. The developers don't get autonomy, and at the
Speaker 2: end of the day, it's just bottleneck the develops. And
Speaker 2: not to talk about the fact that SERE and DevOps
Speaker 2: teams are usually like one to ten developers, so you
Speaker 2: might have like ten develops people to one hundred or
Speaker 2: two hundred developers. So that's one side.
Speaker 3: I can definitely identify with this.
Speaker 1: I mean, I'm working on a project right now that's
Speaker 1: on several timelines, right, and yeah, when everything's won't deploy,
Speaker 1: when it doesn't play nicely with the cluster, things like that,
Speaker 1: we get frustrated, right, And then my boss gets frustrated
Speaker 1: and it's like, why isn't this out there?
Speaker 4: Right?
Speaker 1: And then you know, well DevOps, right, and so then
Speaker 1: they go to DevOps and it's the same thing, right,
Speaker 1: and then the DevOps guys. Sometimes it's okay, well, let's
Speaker 1: they'll go figure out what it is and it's something
Speaker 1: that they can fix. And sometimes they're coming back to
Speaker 1: us and saying, well, there's this problem, and they don't
Speaker 1: want to come back to us and manage us, and
Speaker 1: and nobody else is happy because whoever the powers that
Speaker 1: be are for the business needs, they just want it
Speaker 1: out right.
Speaker 3: And so, yeah, what you're talking about. We've run into
Speaker 3: this more than once over the last year.
Speaker 2: And then it's like, what I'm doing to know? I
Speaker 2: heard it from several and several companies and multiple times.
Speaker 2: And just to tell you another example, the most common
Speaker 2: thing is that they come to Develops and tell them
Speaker 2: I have a deadline and then they go like, yeah,
Speaker 2: but the CFO told me to do cost reduction on AWS.
Speaker 2: So what do I do? I listen to the CEO
Speaker 2: to the CFO, or do I listen to the VPR
Speaker 2: and D or like what's more important? And who knows?
Speaker 2: I don't know. It's it's hard. Now let's talk about
Speaker 2: the other side. The other side is actually when this happens,
Speaker 2: but Develops does not assume responsibility and they go the
Speaker 2: path of educating the developers and they say, no, we're
Speaker 2: not going to lock everything. We're not going to lock anything,
Speaker 2: but we're going to put additional efforts into educating the
Speaker 2: developers in order to making the right decisions, which is nice.
Speaker 2: The thing is, while this happens, you're really not sleeping
Speaker 2: at night, both because you're afraid and because you're getting
Speaker 2: like pager paged for things that are happening. And secondly
Speaker 2: the developers are I find them terrified. They go like,
Speaker 2: I'm going to do that's something that is going to
Speaker 2: change production now. And they go like, I'm a Java
Speaker 2: billing engineer, I don't know Docker, I don't know Kubernites.
Speaker 2: I'm expert in Java billing, not in you know, Docker,
Speaker 2: I don't know. And then you find them like almost
Speaker 2: crippled because because they're afraid, they say, I don't know,
Speaker 2: I'm not an expert, and I'm afraid to break it
Speaker 2: and I don't want to do it. And then the
Speaker 2: teams they try to educate them and so on. But
Speaker 2: I think that what goes best, and it's solutions like
Speaker 2: the tree or you can take up open policy agent
Speaker 2: with contest and the gatekeeper and write your own policies
Speaker 2: and what I've heard from developers is that when the
Speaker 2: middle ground they call it, I feel like I have guardrails,
Speaker 2: Like I'm riding the freeway, but I have guardrails so
Speaker 2: I can do it by myself. I'm not bottlenecked by
Speaker 2: de VopS, but if I do something horribly wrong, the
Speaker 2: system will stop me. And then it's like a nice
Speaker 2: middle ground between the two, which I think can greatly
Speaker 2: help both sides of the equation.
Speaker 5: I think that's one of the approaches I try to
Speaker 5: take in specifically in post mortems, you know, because in
Speaker 5: post mortims a lot of the focus is on root
Speaker 5: cause and what went wrong.
Speaker 3: But I try to.
Speaker 5: Take it a little bit further than that and say,
Speaker 5: you know, the failure was not that this code did whatever.
Speaker 5: The failure is that the system didn't warn somebody that
Speaker 5: this was going to happen.
Speaker 4: Right.
Speaker 5: We built an environment where a developer or an engineer
Speaker 5: was able to make a change that they shouldn't have
Speaker 5: been allowed to make. And I think that's what you're
Speaker 5: describing there, is they're free to do whatever they want,
Speaker 5: but there's the guardrails in place to keep them from
Speaker 5: doing something that they didn't intentionally want.
Speaker 2: To do a WS you go delead the resource and
Speaker 2: it's like, there are fifteen resources attached to this security group.
Speaker 2: I guess you don't want to delete the security group.
Speaker 4: So one thing I'm curious about you is when you
Speaker 4: talk about your clients, I'm curious to hear what are
Speaker 4: the comment sort of misconfigurations. You know, I don't know
Speaker 4: if you're to say, you know, if you were to
Speaker 4: ask you here, what are the top five or top ten,
Speaker 4: I'm really curious to hear like what you see commonly.
Speaker 2: Yeah, great question, It's an absolutely great question. And by
Speaker 2: the way, in our docs have that the tree that
Speaker 2: I owe, we list all of the policies that we
Speaker 2: have and you can view everything. So let's go over
Speaker 2: some categories and talk about them and talk about the
Speaker 2: their security. So one one day, one company, very big
Speaker 2: messaging company told me, he told me, I want an
Speaker 2: if in for a safety score. I want this to
Speaker 2: run and for me to be to feel not safe
Speaker 2: in regards to security safe. Also it also has security aspects,
Speaker 2: but I want to know that my safety score is high.
Speaker 2: So it starts from resource management. So in kubernets, for example,
Speaker 2: you can have a CPU requests and CPU limit, memory limit,
Speaker 2: and memory requests. This is very very common, and it
Speaker 2: specially happens because the developer, she codes the app and
Speaker 2: then she sends it to the cluster, she doesn't know
Speaker 2: what is she going to be paired with. Now, the
Speaker 2: develops engineer works on workload management and optimizes the workloads
Speaker 2: on the different nodes so it will be cost effective.
Speaker 2: And the problem starts when you don't have memory in
Speaker 2: CPU limits, and then you have a memory leak in
Speaker 2: one of your containers, and then it starts affecting it
Speaker 2: psych a noisy neighbor, but very very noisy, and then
Speaker 2: Kubernitis starts to it depends on how you configured it
Speaker 2: and so on, but it starts to kill services, starts
Speaker 2: to run out of memory. There are different behaviors that
Speaker 2: none of them is good and none of them is
Speaker 2: as expected. So I'd say this is the like no
Speaker 2: brainer one that you should do.
Speaker 4: Now.
Speaker 2: What we see companies usually do also is that they
Speaker 2: start and they apply cluster wide memory and CPU limit
Speaker 2: because you can do it on the runtime level. The
Speaker 2: problem starts when you know you have different departments. One
Speaker 2: of them needs four jigabytes of memory and it's great,
Speaker 2: but then you have the AI engineers and they're like,
Speaker 2: we need forty jigabytes. And then if you don't configure
Speaker 2: it on a shift left side, then you go like, okay,
Speaker 2: so I need to increase everyone's limit to forty jigabytes,
Speaker 2: and then it's like nothing, it doesn't matter. So it's
Speaker 2: really important to set it on the resources side. So
Speaker 2: that's one. The second one is I would say around
Speaker 2: workload manage in terms of making sure that you have
Speaker 2: a liveness probe, a readiness probe, that your doctor container
Speaker 2: has a health check. It sounds so trivial, it sounds
Speaker 2: so simple, but so many times people go and create
Speaker 2: the workload, don't set those things. They just oh, just
Speaker 2: HTTP to it, ah, HTTP two hundred, it works great,
Speaker 2: and then they don't configure it on the workload level.
Speaker 2: And not to talk about you know, deeper things where
Speaker 2: you have a service and you want the health check
Speaker 2: to include maybe a connection to a database or to
Speaker 2: a cash and I would really really advise to, like,
Speaker 2: in order to increase your safety and stability, really put
Speaker 2: an effort into your health checks, readiness, liveness, because if
Speaker 2: you do it right and correctly, once things fail and
Speaker 2: things always fail. It will be easy for you to
Speaker 2: find the root cause, and it will be easy for
Speaker 2: you to protect yourself and for kubernitis to kill this
Speaker 2: workload and to get another workload running.
Speaker 4: So let me just stop you there, because I was
Speaker 4: to ask the dumb questions. So I think, I think
Speaker 4: I understand what you're saying. But for some of our
Speaker 4: for other people, you know, our listeners, who may not
Speaker 4: have followed that whole train of thought, right, because there's
Speaker 4: a lot there. You just said in two minutes that
Speaker 4: I feel like we could unpack, right, So I'm going
Speaker 4: to say it, and then you need to tell me
Speaker 4: how wrong I am or if I miss something. But
Speaker 4: so let's say we spin up, you know we we
Speaker 4: we send out a new package, we spin it up,
Speaker 4: and I'm the developer. So what I do is I
Speaker 4: just checked to say, hey, can I hit can I
Speaker 4: can I with my browser hit it? And I get
Speaker 4: a two hundred response back saying we're good. So that's
Speaker 4: only a piece of it, because maybe I'm only hitting
Speaker 4: let's say the load balancer, and so the load balancer
Speaker 4: is saying I'm here, right, I'm answering you, but the
Speaker 4: application behind it is dead, or maybe the application is alive,
Speaker 4: but the database behind it is dead. So unless I'm
Speaker 4: doing health checks that are short of going through those
Speaker 4: steps we may have had, we may have just deployed
Speaker 4: something that broke everything and I don't even realize it
Speaker 4: because all I'm doing is pinging the load balancer and
Speaker 4: getting a two hundred response and everything looks good to
Speaker 4: me because I didn't check what's going on behind you know,
Speaker 4: I sort of peeling back the layers of the onion.
Speaker 4: Did I get that right? Orm I missing.
Speaker 2: Something absolutely right. This is one of the most common
Speaker 2: mistakes developers make is they just check the simple front
Speaker 2: end web browser and they don't do the entire process.
Speaker 2: And then when you do have a problem, it's so
Speaker 2: hard to debug it because everything returns a great health check.
Speaker 2: So and you don't understand what is actually the problem.
Speaker 4: So is this something that you know, like the tree
Speaker 4: does forty? Is this something I've got to figure out?
Speaker 4: Like how does that?
Speaker 2: You know?
Speaker 4: How do you build that into a health check? So
Speaker 4: that sounds that There's a lot of steps, and it
Speaker 4: really depends on the architecture your application.
Speaker 2: Yeah, so we can talk about it from an engineering standpoint.
Speaker 2: In terms of the tree. The tree is a tool,
Speaker 2: and it's a tool that you can use in order
Speaker 2: to say, listen, from now on, all of our Kubernites
Speaker 2: workloads are going to have a liveness probe, a readiness probe. Now,
Speaker 2: how you configure this liveness problem and readiness probe is
Speaker 2: up to you. Same thing, you're going to put a
Speaker 2: memory limit. If you put a memory limit of sixty
Speaker 2: four megabytes and your server can't even it's some Java
Speaker 2: huge jar, I don't know, it can't even load up,
Speaker 2: it's your problem. But what we will do is we
Speaker 2: will make sure that a policy exists and that it
Speaker 2: is configured on the resource. The next layer is another
Speaker 2: thing that is like what is the most common you know, policies.
Speaker 2: It's actually labels. Again, it sounds so simple, it sounds
Speaker 2: so trivial to put the label, and there are so
Speaker 2: many and is why to put a label? So I'll
Speaker 2: start with the one that we're talking about now. So
Speaker 2: first of all, you can use labels in order to
Speaker 2: say what type of workload it is in order to
Speaker 2: determine which type of policy in terms of resource management,
Speaker 2: for example, it should use. So then you could say
Speaker 2: this is from type AI and they use those types
Speaker 2: of limits and those are from type back end front end.
Speaker 2: I don't know. Different teams call it in different names,
Speaker 2: and you can use it in order to understand what
Speaker 2: are the relevant policies you should use. This is number one.
Speaker 2: Number two cost management. This is also a very very
Speaker 2: common use case that DevOps people have to deal with,
Speaker 2: which is constantly knowing to assign the cost center because
Speaker 2: they run the shared resources and at the end of
Speaker 2: the day they pay the check to a WS or
Speaker 2: AZURE or whatever you run it, and then the internal
Speaker 2: company goes like, okay, but how much do we need
Speaker 2: to build each business unit inside of organization? And then
Speaker 2: they go like, I don't know. We had like five
Speaker 2: thousand servers and then they go like, okay, now it's
Speaker 2: mandatory everyone should say which department this server belongs to,
Speaker 2: because otherwise we're not going to know how much to
Speaker 2: allocate to it, because then you don't know what is
Speaker 2: the cost of goods of your business, and then the
Speaker 2: CFO doesn't know if the business is profitable or not profitable,
Speaker 2: or can we hire people? Can we not hire people?
Speaker 2: And it's crazy because it's like board of director's decisions
Speaker 2: that go down to the CFO that go down, down, down,
Speaker 2: down down to the simple label that you need to
Speaker 2: put on your comin this workload in order to know
Speaker 2: how much it costs.
Speaker 5: Can you define for us the difference between a liveness
Speaker 5: probe and a readiness probe?
Speaker 2: That is a great question. So liveness probe works on
Speaker 2: I said readiness. Readiness probe is when I'm ready to
Speaker 2: serve traffic. So let's say I'm initializing myself. I need
Speaker 2: to start. I need to go create a cash, make
Speaker 2: sure that they can put it there, and so on.
Speaker 2: And then a liveness probe is when I'm running. Am
Speaker 2: I running correctly? Can I continue communicating, for example, with
Speaker 2: my existing cash or whatever it is. I like the
Speaker 2: I think it's too much. I like the health simple
Speaker 2: health check you know that goes end to end and
Speaker 2: does the check. In addition, by the way, I also
Speaker 2: suggest for companies it has nothing to do with the
Speaker 2: trends on but like to configure outside health checks that
Speaker 2: actually go and do a user activity on your services
Speaker 2: for real, because the worst thing you want is a
Speaker 2: customer calling and saying the service is down. You want
Speaker 2: to be the first to know, so I think that
Speaker 2: those are the main things I would focus on.
Speaker 5: That's a really good point. I've been in a few
Speaker 5: outages where everything was working internally but nothing was working externally.
Speaker 2: Yep, I'll tell you one of my most severe outages.
Speaker 2: It was so hard to debug. It was my previous company.
Speaker 2: It was not an outage, it was even worse. What's
Speaker 2: worse than an outage? Everything slows down and works really
Speaker 2: really bad, and it doesn't break, so you don't really know.
Speaker 2: And it was a data pipeline that collected two hundred
Speaker 2: billion events every month, and it was a geolocation based routing,
Speaker 2: so it would every time someone will click and add,
Speaker 2: it will route to the closest aws region and send
Speaker 2: the event there. And then we had thirteen regions that
Speaker 2: would send everything to a centralized kinnessis and then we
Speaker 2: would have workers that would process it. Now, in order
Speaker 2: to do the duplication and add some attributes, we had
Speaker 2: a ready skesh and this ready so all the workers
Speaker 2: would access the ready skesh in order to put in ideas,
Speaker 2: select ideas and so on. And at some point, with
Speaker 2: the amount of messages increased, you know, slowly, slowly, slowly, slowly,
Speaker 2: and then at some point the memory of the readies
Speaker 2: got filled. So what did it? It switched to swap.
Speaker 2: This is a problem with swap. It's slow. And then
Speaker 2: all the requests started returning really really slowly. And then
Speaker 2: you don't understand. You think, okay, there's a problem, so
Speaker 2: you put on more servers and then they bombard the
Speaker 2: readies even more, and then you put on more workers
Speaker 2: and like you're trying everything from like you're just trying
Speaker 2: to debug everything until finally we're like opening and I
Speaker 2: was like, oh my god, the Reddit is running on swap.
Speaker 2: And then we had to increase their readis memory and
Speaker 2: then and then it fixed it. And if we had
Speaker 2: a check that life is check that said I'm going
Speaker 2: to perform I put events to the readis and I
Speaker 2: expected to take two to four milliseconds. I'm just making
Speaker 2: this up. And if at some point this is more
Speaker 2: than four milliseconds, there's a problem, we would have immediately
Speaker 2: new where is the root cause of this issue? But
Speaker 2: we didn't. That's the truth.
Speaker 4: Yeah, another good reason why I put more are so important.
Speaker 4: I'm curious to know because here's my anecdotal I guess
Speaker 4: experience is that I find that very few organizations do
Speaker 4: post mortems well, and if they're doing them, I don't
Speaker 4: think that they do them in a very effective way.
Speaker 4: I think they do them in more of a finger pointing,
Speaker 4: root cause analysis, who caused the problem and who should
Speaker 4: we fire right? And I just feel like you feel
Speaker 4: that Unfortunately, Listen, I'm on the security side, so a
Speaker 4: lot of the post mortems I'm involved with our security incidents,
Speaker 4: so those might be a little bit, you know, handled
Speaker 4: a little bit differently than like an you know, typical
Speaker 4: outage or or that sort of situation.
Speaker 3: But yeah, we will did it.
Speaker 4: Right, Yeah, I mean so, I guess I should say
Speaker 4: I feel like most of the time the post mortems
Speaker 4: just don't happen. I feel like the times that they do,
Speaker 4: it almost becomes a witch hunt. And those are very rare.
Speaker 4: But when they do happen, again, that's just my experience,
Speaker 4: they just just get nasty. So I'm curious. I want
Speaker 4: to hear a better story because I feel like my
Speaker 4: experience is not good.
Speaker 2: I've never experienced anything like it. Thankfully, the organizations that
Speaker 2: I've worked with, my company, thank god, we did not
Speaker 2: have a security incident that someone stole all of our
Speaker 2: records or something, because then I think you're like obligated
Speaker 2: to take action. And maybe most of the like your
Speaker 2: cases were those type of severe cases where you know,
Speaker 2: it's it's just it's like borderline, like federal, it's like
Speaker 2: it's really a problem. And what I am, yeah, what
Speaker 2: I'm referring to is more of a engineers and like
Speaker 2: like the ready say story, I just told you, who
Speaker 2: are you gonna fire? No one, It's just gonna make
Speaker 2: everyone the better, you know, and tell them and then
Speaker 2: think of how we could have fixed it. And then
Speaker 2: we implemented the check. Believe me, every time there was
Speaker 2: a problem, the first thing everyone checkedes the reddits. Everyone
Speaker 2: went to see that redit is okay. It was like
Speaker 2: a small baby that everyone takes care of. But I
Speaker 2: don't believe in the witch hunts. I really believe in
Speaker 2: the culture where people come and they say I made
Speaker 2: a mistake and people help them understand. And again, as
Speaker 2: long as there was no negligence, I don't know, you know,
Speaker 2: something criminal or something like that, people make mistakes. Another story,
Speaker 2: there was an employee. It was her first day. The
Speaker 2: company was still using SVN and not GET and on
Speaker 2: her first day on the job, she deleted the entire
Speaker 2: SVN three nothing happened to her. Yeah, so I think
Speaker 2: restored the backup and it's okay. But I think this
Speaker 2: is the main difference. I don't know what is your
Speaker 2: experience personally.
Speaker 4: I've seen both ways, you know.
Speaker 5: I remember in years past post mortems were the lynch
Speaker 5: mob had to pitch the pitchforks and the torches trying
Speaker 5: to find out who we were going to grab. But
Speaker 5: I think that's in my experience that's gone away over
Speaker 5: the last few years to people being more willing to
Speaker 5: accept that mistakes happen. But it almost feels like a
Speaker 5: pendulum where now an it's over overly trying too hard.
Speaker 5: I guess to make sure that someone doesn't feel attacked
Speaker 5: in the post mortem, that you never get to the
Speaker 5: root cause either, you know, And so I think you
Speaker 5: got to struggle to find the happy medium there. And
Speaker 5: I mean, ultimately, you know, in a lot of these situations,
Speaker 5: someone did do something incorrect and you've got to point
Speaker 5: that out in order to identify it. And when you
Speaker 5: point it out, you know, you're not like calling that
Speaker 5: person out or attacking their skills. It was just a mistake.
Speaker 5: It happened, but it's important to fully understand what that
Speaker 5: mistake was so that you can build in the systems
Speaker 5: to prevent it from happening again.
Speaker 2: Yeah.
Speaker 1: Yeah, I've been in the situation where and not because
Speaker 1: of a post mortem, but just because of other things.
Speaker 3: You know.
Speaker 1: I had a boss come in once on one of
Speaker 1: the teams, I was team lead, and he basically walked
Speaker 1: in the room and said, somebody's getting fired today, right,
Speaker 1: And you don't want people to feel that, right, because
Speaker 1: I took him outside and I said, I said, if
Speaker 1: you're going to pull this, they're all keeping their jobs.
Speaker 1: I'm just going to quit, right, And it's because nobody
Speaker 1: should live in that kind of fear. Right, We're all
Speaker 1: trying to work on the same thing. But the flip
Speaker 1: side is is, yeah, I mean, if somebody is routinely reckless, right,
Speaker 1: it's always Jim.
Speaker 3: Right.
Speaker 1: It's gone down four times this month, and Jim has
Speaker 1: been the one to mess it up every time, and
Speaker 1: this is all stuff that we've done training on, and
Speaker 1: so Jim should know better, you know. The first time, Hey,
Speaker 1: Jim's a human. Second time, Jim's still a human. Third time, Okay,
Speaker 1: Jim's a human. But Jim is starting to cause some problems.
Speaker 1: You can have the conversation about whether or not Jim
Speaker 1: needs to keep his job. But if people feel like
Speaker 1: they're going to be punished for making a mistake every
Speaker 1: once in a great while, then you're going to slow
Speaker 1: the whole system way down. And the whole point, as
Speaker 1: Shimon keeps pointing out, is we want to keep moving fast.
Speaker 1: We want to move fast, we.
Speaker 3: Want to get stuff out, we want to solve problems
Speaker 3: for our customers as quickly as possible, and at the
Speaker 3: same time maintain some level of stability.
Speaker 4: Yeah, really agree. I think the last point I would
Speaker 4: make is that I think the whole idea of root
Speaker 4: cause analysis, even if it is one person's you know,
Speaker 4: at the end of the day, even if you can
Speaker 4: tie it back to one person's typo or mistake or whatever,
Speaker 4: I personally feel like the root cause analysis is generally
Speaker 4: flawed in that it's rarely one person right. It's it
Speaker 4: might be one person you know again who typed it
Speaker 4: in wrong or did whatever, but there's a process breakdown
Speaker 4: as well, and there was an authority breakdown, or like
Speaker 4: what she was talking about before, the guardrails didn't exist.
Speaker 4: You just can't point it at one person like it's
Speaker 4: the system broke down. Yes, it resulted in somebody's mistake
Speaker 4: in a manifest file or something like that. But if
Speaker 4: you go, you know, if you take it back, you
Speaker 4: look at it and you say, well, wait a second, guys,
Speaker 4: because our process isn't all that great. He was trying
Speaker 4: to do the best he could, he didn't know or
Speaker 4: whatever it was. You can't be an expert in everything
Speaker 4: made a mistake, but it's because the entire process broke down,
Speaker 4: not just because one person made a mistake. And I
Speaker 4: feel like that's the piece that you know, you're trying
Speaker 4: to do the cause analysis, that's the piece that people
Speaker 4: just don't think about.
Speaker 2: I totally agree. Just just to finish on this point,
Speaker 2: the best root cause analysis process that I have ever
Speaker 2: seen in my life is get lab. They went down
Speaker 2: and they've opened a live doc that everyone could see,
Speaker 2: all the customers, all everyone, and they've had a sessions
Speaker 2: that are like open a Google hangout resume. I remember
Speaker 2: what they did, and anyone could join, and it was
Speaker 2: a totally transparent process of them debugging the outage that
Speaker 2: they had, and of course afterwards they published everything like
Speaker 2: including like logs, crazy stuff and like here's what happened.
Speaker 2: Here's for transparency, and here's for you to learn how
Speaker 2: not to make our mistakes. And I really admired it.
Speaker 5: Yeah, I think there's something to be said for gaining
Speaker 5: credibility with your customers whenever they find out that there's
Speaker 5: an outage from you, instead of them telling you that
Speaker 5: there's an outage, and then you provide real time or
Speaker 5: near time updates to them up until the issues resolved.
Speaker 2: Definitely.
Speaker 5: So I think we've all seen scenarios where ABS has
Speaker 5: had an incident and you find out about it either
Speaker 5: personally or on Reddit three or four hours before the
Speaker 5: AWS status page updates.
Speaker 2: That is, if it did not affect the status space
Speaker 2: because that happened as well, you know.
Speaker 1: Yeah, well, and that's interesting to me too, right, is
Speaker 1: that sometimes it's hey, we screwed this stuff up and
Speaker 1: so therefore our app didn't run. And then yeah, we
Speaker 1: see these big companies that use a lot of the
Speaker 1: AWUS or other infrastructure on the out there on the cloud,
Speaker 1: and what winds up happening is yeah, what we're kind
Speaker 1: of talking about, except they take down the entire US
Speaker 1: Eastern one region, right, and everybody goes, why is the
Speaker 1: Internet not working? And yeah, it turns out that, yeah,
Speaker 1: the Internet relied on that that region for a whole
Speaker 1: bunch of stuff and it's gone. And so those kinds
Speaker 1: of externalities too, where it's it goes beyond even your code,
Speaker 1: your company, your infrastructure, your cloud set up. That's fascinating too,
Speaker 1: and those cases, you know, as Will's pointing out, we
Speaker 1: all kind of want to know, right, because it's affecting everybody.
Speaker 2: This is why, you know, it was very interesting when
Speaker 2: Jeffrey said that, like which hunt and so on? And
Speaker 2: I think this is like the define line between security
Speaker 2: and infrastructure, where it's like the culture and infrastrucures like yeah,
Speaker 2: we all like seventeen out the jazz and no problem.
Speaker 2: And then when it crosses this line specifically you know
Speaker 2: a privacy security you know, personally identifying information, and then
Speaker 2: it's like, okay, something's different going to happen here. And
Speaker 2: it's interesting because in organizations, like government organizations, there are
Speaker 2: special ways to investigate what happened. That's saying in a
Speaker 2: military when there was an operation, so they want to
Speaker 2: learn from it. So there are two paths of investigation.
Speaker 2: One path is like the regular path, they investigate and
Speaker 2: like they can put someone to jail and so on.
Speaker 2: And then there's it's called the professional combat review where
Speaker 2: everyone can say whatever. They can say, I killed someone
Speaker 2: and they will not be eligible for anything, like they
Speaker 2: can't do anything to them, and they have one hundred
Speaker 2: percent immunity in this process. And this is done in
Speaker 2: order to make sure that we learn and that everyone
Speaker 2: say what really really happened, and like everything you say
Speaker 2: there is classified, it cannot be used against you and
Speaker 2: so on. So I think it's also an interesting thing
Speaker 2: to think about in our field.
Speaker 4: I totally agree. I feel like the organizations that yeah,
Speaker 4: like I said, I think that which I'm you know,
Speaker 4: mentality is a terrible one regardless of what what happened.
Speaker 4: I mean unless you are talking about like we said before,
Speaker 4: nofeasance or negligence or something like that, or you know,
Speaker 4: beyond negligence, but you know, really criminal negligence like which
Speaker 4: rarely happens. Right, It's it's generally speaking, you know, it's
Speaker 4: a breakdown process and just fix it. I mean, just
Speaker 4: work together and fix it. Nobody wants to. I mean,
Speaker 4: I've just been involved in so many companies post breach,
Speaker 4: and so everybody just wants wants THEMS to go back
Speaker 4: to normal. It's like COVID, right, everyone just wants things
Speaker 4: to go back to normal. Let's just pass this move on,
Speaker 4: you know, do we have to do, but let's stop
Speaker 4: reliving it on a daily basis.
Speaker 1: Yeah, all right, Well, I think we're kind of getting
Speaker 1: towards a place where we can start to wrap up.
Speaker 1: Are there any other kind of big pieces of advice
Speaker 1: that we need to put out there before we go
Speaker 1: to our picks.
Speaker 2: I want to point out one thing which I really
Speaker 2: believe in, which is it's a big word called gee tops.
Speaker 2: But in general, make sure that all of your configuration
Speaker 2: and all of your assets, everything is in code and
Speaker 2: in GIT. And if you live with one thing from
Speaker 2: this podcast is make sure that everything is infrastructure's code
Speaker 2: and in geed because then you will be able to
Speaker 2: at least see what happened and what was the configuration
Speaker 2: and how did we configure it. So this is my
Speaker 2: final small remark here.
Speaker 3: That's good advice.
Speaker 1: All right, Well let's roll into picks then, Jeffrey, do
Speaker 1: you want to startus off with the picks?
Speaker 4: All right? So it's something I was just thinking about.
Speaker 4: I was actually thinking about as we're talking, you know,
Speaker 4: just having our conversation here. So my pick isn't a
Speaker 4: specific thing. It's more of just an approach. So I
Speaker 4: get asked all the time like how do you, you know,
Speaker 4: sort of continue to continuously learn and you know, learn
Speaker 4: new technology is new, you know, sort of stay on
Speaker 4: top of current threats. It's technology in general is just
Speaker 4: that constantly changing space. But I mean, honestly, I think
Speaker 4: that applies beyond technology. Our world is just constantly changing,
Speaker 4: and how do you stay on top of that? And
Speaker 4: how do you do that without spending eight hours a
Speaker 4: day just trying to read or learn or watch or whatever.
Speaker 4: And so a couple of things that I have learned.
Speaker 4: So I think there's more just ideas than actual like
Speaker 4: go go and buy a product or something like that
Speaker 4: is you know, the with it we learn. I think
Speaker 4: that you know, there are different you know, different people
Speaker 4: do learn differently. But what I've seen is that, you know,
Speaker 4: there's so much out there now like on YouTube, for instance,
Speaker 4: I mean, there's so much content out there, but it
Speaker 4: takes a long time to go through, especially now that
Speaker 4: everything has adds in it. So now now every video
Speaker 4: takes much longer to get through, right, But if what
Speaker 4: you're trying to learn is very specific, it's sometimes harder
Speaker 4: to figure out how to learn it because there are
Speaker 4: so many blog posts that are too too generic or
Speaker 4: just repeating what everybody else has already said on the
Speaker 4: topic already, and everyone just wants to put it into
Speaker 4: their blog to try and you know, get whatever it
Speaker 4: is se o or get you know, traction out of it,
Speaker 4: traffic that sort of thing, or you can try and
Speaker 4: you know, pick it out of like a video, but
Speaker 4: you know, you could be going through a sixty minute
Speaker 4: video and trying to figure out where where it is.
Speaker 4: So I think part of it is and there's no
Speaker 4: real answer here, but part of it is just figuring
Speaker 4: out what's the best medium for learning what I'm trying
Speaker 4: to learn? Am I just trying to get an overview
Speaker 4: of it of that subject, then maybe a video is
Speaker 4: good if I'm trying to learn something very specific, maybe
Speaker 4: going to like stack overflowed. I think building that skill
Speaker 4: set in yourself of figuring out what is it that
Speaker 4: I'm trying to learn and what's the best way for
Speaker 4: me to get there is something that we all have
Speaker 4: to just sort of develop. And I think a lot
Speaker 4: of us who've been doing this for years, you're probably thinking, yeah,
Speaker 4: I've been there, I've done that. I think I'm there already.
Speaker 4: But I think for some of the people earlier on
Speaker 4: in their in their in their career, this might be
Speaker 4: something that you should really be thinking about, is just
Speaker 4: how to be most efficient learning something new. And obviously
Speaker 4: it also goes back to figuring out what the best
Speaker 4: sources are, because, like I said, there's a lot of
Speaker 4: content out there, and it's just regurgitating what's what's already
Speaker 4: out there and and sort of dumbing it down sometimes
Speaker 4: like pulling out some of the details. So those sources,
Speaker 4: you know, you want to toss and you want to
Speaker 4: just sort of go to, you know, figure out what
Speaker 4: are the right sources that that you know give you
Speaker 4: their information. So that's one piece. The other thing I
Speaker 4: was going to say is I think sometimes a lot
Speaker 4: of times we are we have this sort of natural
Speaker 4: tendency to look for, you know, when we do have
Speaker 4: to buy something, we think about, what's the cheapest product
Speaker 4: out there? Right, what's and I think so many so
Speaker 4: often the cheapest product actually takes you more time, more energy,
Speaker 4: and you end up having to do things over you know,
Speaker 4: over again or whatever, and it's not the cheapest product,
Speaker 4: and I think, you know it's it's again, you know,
Speaker 4: as you go through that learning and figuring out what
Speaker 4: what is it that I need, don't fall into the
Speaker 4: trap of just buying the cheapest product. Sometimes it's buying
Speaker 4: the more expensive product. I mean, sometimes it is the
Speaker 4: cheapest product. Generally it's a use once type of thing,
Speaker 4: or you know, I'm really going to use it. Great,
Speaker 4: But if it's something you're not going to do that
Speaker 4: you are going to continue toly use, spend some time
Speaker 4: figuring out does it make sense to invest in something
Speaker 4: a little bit you know, better quality. So anyway, those
Speaker 4: are my two picks methodologies. Whatever thoughts for the day?
Speaker 3: Nice Will, what are your picks?
Speaker 4: All right?
Speaker 5: So I have been working my way through this book,
Speaker 5: The Manual from Bictitis. So he was a stoic philosopher,
Speaker 5: and I've actually tried to read Marcus Aurelius's meditations in
Speaker 5: the past and not really sure how much I actually
Speaker 5: retained from that. So I came across this book and
Speaker 5: I really like it because it's just it's very short,
Speaker 5: like each page just has one particular quote or saying
Speaker 5: from Epictetis and it's been really helpful to just kind
Speaker 5: of come to understanding with the whole Stoic philosophy and
Speaker 5: that in combination with daily emails the email list from
Speaker 5: the Daily Stoic dot com, I start each day by
Speaker 5: reading those and it's a really good way to kind
Speaker 5: of level set your mind before you get started in
Speaker 5: a day and put things in perspective, because I think
Speaker 5: that's helpful, especially with the amount of information and if
Speaker 5: you can't avoid the news that's going on every day,
Speaker 5: it kind of helps you temper that message and keep
Speaker 5: things into more of a longer range perspective. So the
Speaker 5: main from Epictetus and the Daily store dot com are
Speaker 5: my picks for today.
Speaker 3: Nice, what do I have for picks?
Speaker 1: So Father's Day, I've got a couple of picks for
Speaker 1: stuff that I did or got for Father's Day. The
Speaker 1: first pick that I have is my wife's like, Hey,
Speaker 1: you get to control the TV, which never happens at
Speaker 1: my house, both because I don't watch a ton of
Speaker 1: TV and because my kids just are on video games
Speaker 1: all day during the summer, so you know, I'll go
Speaker 1: down there and I'll just kind of see what's going on.
Speaker 1: But yeah, So on Sunday afternoon, I watched Willow, which
Speaker 1: is one of my favorite old timey movies.
Speaker 3: So I'm gonna I'm gonna pick that because I enjoyed it.
Speaker 3: I really enjoyed it.
Speaker 1: Of course, all my kids the second we turn it
Speaker 1: on there they sat there for ten to fifteen minutes
Speaker 1: and then just cleared out of the room.
Speaker 3: And I'm just like, my guys, is a good movie. Whatever. Whatever.
Speaker 1: Anyway, the other pick that I have so my wife,
Speaker 1: I've been having issues. My grill has been falling apart
Speaker 1: for a few years, and I like cooking me some meat.
Speaker 1: So my wife got me a trigger smoker for Father's Day.
Speaker 4: Oh nice.
Speaker 1: And it's got a couple of meat probes in it
Speaker 1: and stuff like that, which is super nice because a
Speaker 1: lot of time it's not. It doesn't have bluetooth or
Speaker 1: anything in it. I know some of the more expensive
Speaker 1: models do, but it's nice just because you can kind
Speaker 1: of cook at the temperature and then you know you're
Speaker 1: ready to pull it out right. And so anyway, made
Speaker 1: a brisket on it for Father's Day so good.
Speaker 3: Oh my gosh.
Speaker 1: You know, I've got some baby back ribs in the
Speaker 1: fridge that I need to throw on there sooner rather
Speaker 1: than later. But it's just it's so nice and all
Speaker 1: of the stuff that you kind of cook on the
Speaker 1: slow cook end of things, they just come out so
Speaker 1: so so good.
Speaker 4: Right.
Speaker 1: So the other forms of that I guess are like
Speaker 1: the crockpot or the souvide, But yeah, the smoker's nice
Speaker 1: too because it gets all this flavor in there. And anyway, yeah,
Speaker 1: I am loving having So I'm gonna pick that, Simon.
Speaker 3: What are your picks?
Speaker 2: So I'm gonna have a barbecue now, fifteen of my
Speaker 2: friends are coming and I have an Apoleon grill and
Speaker 2: I really love grilling and also I always measure the
Speaker 2: temperature of the meat and I really really love it
Speaker 2: in terms of my picks. So I found daily dot dev.
Speaker 2: It's something cool that you can daily dot too. Sorry
Speaker 2: that no, I'm I'm mistaking several things here. It's called
Speaker 2: daily dot dev and it's a Chrome homepage extension so
Speaker 2: when you open up a new tub, it actually shows
Speaker 2: you like stuff from news and stuff like that, but
Speaker 2: you know, targeted at dev So it's really really nice
Speaker 2: because it just gives you a like a thumbnail and
Speaker 2: a title and it shows you what's going on. So
Speaker 2: I thought it's it's something nice because it's really targeted
Speaker 2: towards our target audience, so it's nice. So that's my
Speaker 2: small TEP besides the get tops tip that I give
Speaker 2: at the beginning.
Speaker 1: Awesome if people want to connect with you online, where
Speaker 1: do they find you?
Speaker 2: Yeah, so I'm at Shechemon Tolts at the Twitter and
Speaker 2: you can always go to the tree do io and
Speaker 2: they see our website there. You can try to message
Speaker 2: me on LinkedIn, but it's gonna be you know, it's
Speaker 2: it's a we can do a whole session about like
Speaker 2: what is LinkedIn become in that regard. But yeah, so
Speaker 2: Simon Tolds at Twitter, that's the best place to reach out.
Speaker 2: And I look forward to hearing from you and listening
Speaker 2: to feedback from users because this is what we love
Speaker 2: the most. When people come in run our run our
Speaker 2: c l, I get some stuff, and then they write
Speaker 2: to us this is great, but we hate this thing
Speaker 2: and why can't I do this and that? And then
Speaker 2: we talk to them and we hear their feedback and
Speaker 2: this is how we prioritize our roadmap, so I encourage
Speaker 2: you to give us feedback about our product at the tree.
Speaker 2: Do I O d A t r e E do
Speaker 2: I O awesome?
Speaker 3: All right, well we'll go ahead and wrap up here.
Speaker 3: Thanks than for coming. This was a lot of fun.
Speaker 2: Thank you very much for having me. It was really
Speaker 2: really fun being here and gegging out about develops with you.
Speaker 2: I feel at home, so thank you very much for
Speaker 2: having me.
Speaker 3: All right, well, until next time, folks, max out