Speaker 1: Go wild. So that's how the episode starts with Jillian Wild.
Speaker 1: Speaking of which, staying on the internet, Yeah too late,
Speaker 1: it's too late. Yeah, okay, Speaking of which, you are
Speaker 1: back on the show with us. You're back in the US,
Speaker 1: So it's been going on.
Speaker 2: Well, I mean I moved continent, so that was that
Speaker 2: was a pretty big thing.
Speaker 3: Now I'm in the US full time, which.
Speaker 2: Is exciting, Like just two hours north of Boston and
Speaker 2: a little town that probably nobody has ever heard of
Speaker 2: in New Hampshire. But it's been fun.
Speaker 3: It's been it's been a big adjustment, but it's been
Speaker 3: a really good time.
Speaker 1: Yeah. From you were in Doka.
Speaker 3: Right Doha?
Speaker 1: Oh yeah, yeah, in the Middle East, So yeah, Doha
Speaker 1: to New Hampshire. There's some adjustment there.
Speaker 2: There, there is. There's a lot more color here, like
Speaker 2: just in general the story.
Speaker 1: I'm sure. Yeah, I've not been to Dolla, but I
Speaker 1: have been to Dubai several times, and I think they're
Speaker 1: fairly similar.
Speaker 2: Yeah. Yeah, it's like less than an hour's plane ride.
Speaker 2: They're about the same thing.
Speaker 1: Yeah. Cool. So back in the US permanently or is
Speaker 1: this a temporary thing for you?
Speaker 3: Permanently? No living here now?
Speaker 1: Right on. Well cool, Well, welcome back home.
Speaker 3: Thank you. It's been nice.
Speaker 2: I'm very happy to be back, and it's nice that professionally,
Speaker 2: it's so much better for me too. I don't have
Speaker 2: to work split shifts and that's great, and I get
Speaker 2: to be back on the podcast and talk to you guys,
Speaker 2: which is obviously.
Speaker 3: The biggest win.
Speaker 1: Of course.
Speaker 2: Yeah, of course, go harass my clients in Boston. Now
Speaker 2: that's where most of them are at and here we are.
Speaker 4: Oh but two hours like and you know, so you
Speaker 4: said maybe no one knows about that, but I actually
Speaker 4: made that drive fair number of times up to Lake Ossipe,
Speaker 4: So there's there's a whole bunch of town there that
Speaker 4: I could probably rattle off. And maybe maybe you're living
Speaker 4: in one of those. That's a long I get from
Speaker 4: New Hampshire downtown.
Speaker 2: I'm not driving, though, so it doesn't bother me. I
Speaker 2: drive to the bus station and then one of those
Speaker 2: very nice buses takes me into Boston. I'm not trying
Speaker 2: to drive in Boston, Like, that's not a thing that
Speaker 2: we're doing.
Speaker 1: It doesn't really matter, right because you're on the East
Speaker 1: coast where they actually have public transit.
Speaker 2: No, the bus is actually nice, Like I don't actually
Speaker 2: mind taking the bus. It has like the really nice
Speaker 2: big seats and it goes right into South Station and
Speaker 2: then I don't know, most of my clients are like
Speaker 2: ten fifteen minutes from there.
Speaker 1: That's cool. I do kind of miss that. I miss
Speaker 1: that every once in a while of just seeing and
Speaker 1: working with people face to face.
Speaker 2: I do. I actually do like that, Like I wouldn't
Speaker 2: even really want to have a hybrid job, but showing
Speaker 2: up like I don't know quarterly and just going to
Speaker 2: go see everybody. Usually it's for some type of meetup.
Speaker 2: There's like, you know, a big meeting of the mind's
Speaker 2: happening or a conference or a talk or something like that,
Speaker 2: and then I'll go for sure.
Speaker 1: So what was it like moving across continents? Did you
Speaker 1: like bring like couches and mattresses and furniture and did no?
Speaker 3: No, no, it's all I don't know.
Speaker 2: I guess like living overseas, all the furniture and stuff
Speaker 2: got treated as kind of disposable because there was enough
Speaker 2: of those moves. And the last place that we were
Speaker 2: at was the last place that we were at was
Speaker 2: mostly furnished anyway, So I didn't have a lot of furniture.
Speaker 2: So I bought like furniture when I bought the house
Speaker 2: that I'm living in now right on.
Speaker 1: Yeah, cool, We did that whenever we moved to Oregon
Speaker 1: from Arizona. The people that bought our house in Arizona
Speaker 1: wanted to buy most of the furniture too, So when
Speaker 1: we got up here, we had almost no furniture. And
Speaker 1: so the first thing we had to do was, you know,
Speaker 1: go drop thousands of dollars on furniture. And I can't
Speaker 1: say we made the best decisions, you know, because like
Speaker 1: you're like desperate for a couch and a mass so
Speaker 1: you don't really research it or take time to say
Speaker 1: do I really like this? You just like say I
Speaker 1: think I'll like this, and then three months later you're like,
Speaker 1: I probably should have gone a different direction.
Speaker 5: You're an amateur mover.
Speaker 2: That's fun.
Speaker 3: Yeah, I mean I do, because I'm insane.
Speaker 2: Like I would just buy some of the Amazon like
Speaker 2: Futon mattresses, like the kind that I take when I
Speaker 2: go camping, and I would throw those on the floor
Speaker 2: with some pillows, and that's what we would have until
Speaker 2: I found a couch that I obsessed research and I
Speaker 2: have like a whole decision matrix of couch versus size
Speaker 2: in the living room versus how I'm gonna like this
Speaker 2: versus different seasons, and how I'm gonna like it? How
Speaker 2: much can the destroy it? Like, there's a lot.
Speaker 3: That goes into these decisions for.
Speaker 5: Me, So you figured it out.
Speaker 3: That's not how I do anything, ever.
Speaker 4: The futon sofa and I also used to buy a
Speaker 4: queen size blow up mattress because you know that's fairly
Speaker 4: cheap and can help you a lot there with even
Speaker 4: knowing what your situation is going to be. If you
Speaker 4: can't find a quick futon sofa and then yeah, then
Speaker 4: you've got as much time as you're willing to sacrifice
Speaker 4: into living into this terrible situation until you actually define
Speaker 4: the mattress you're looking for, the furniture you're willing to
Speaker 4: I got a lot of black previously because I used
Speaker 4: to use my moving boxes as partial furniture. This box
Speaker 4: turned sideways is like a shelf. You stack some boxes
Speaker 4: on top, and then you have a bureau made out
Speaker 4: of cardboard.
Speaker 5: There, he does.
Speaker 2: Although I am willing to fight you on the air
Speaker 2: mattresses because air mattresses are like one hundred bucks now,
Speaker 2: and my kids break them fairly regularly, so they don't
Speaker 2: last long enough for us, which is why I have
Speaker 2: moved on, which is why I moved on to the
Speaker 2: futon mattress like lifestyle, and I'm willing to commit to that.
Speaker 3: Yeah, for life.
Speaker 2: That's it.
Speaker 3: Nothing bad is going to happen to host mattresses ever,
Speaker 3: right on.
Speaker 1: Yeah, cool, Well, welcome back. Excited to have you back
Speaker 1: on the show.
Speaker 3: And thank you for letting me back.
Speaker 1: Yeah, of course, anytime, especially for today's episode because we're
Speaker 1: talking about the twenty twenty four Door Report. Because I mean,
Speaker 1: by the time this episode releases, I'm sure everyone will
Speaker 1: have read all one hundred and twenty pages and this
Speaker 1: is going to be a pretty boring episode because you
Speaker 1: will have digested all the information already. But for the
Speaker 1: two or three people who didn't read it, we're here
Speaker 1: to get you at the speed so that you can
Speaker 1: converse with everyone else as if you spent just as
Speaker 1: much time reading it as they did.
Speaker 2: So I think instead of people reading it, we're all
Speaker 2: going to have to brains and then the report will
Speaker 2: automatically be wrong because it says there ai ho's and
Speaker 2: how as much have been impact as was expected, except
Speaker 2: in Drug Discovery, where it's been insane. But what could happen?
Speaker 4: I honestly, while I was reading it, I had this
Speaker 4: fear like, oh, you like, our whole episode could have
Speaker 4: just been feeding the report into notebook LM and having
Speaker 4: that running instead of the legitimate adventures and DevOps for
Speaker 4: our you know, faithful viewers here.
Speaker 3: Well, I haven't actually read, so.
Speaker 5: I did go through. I did go through almost all
Speaker 5: of it.
Speaker 4: There were some pages at the end that I skipped
Speaker 4: that was just repeating a little bit about what Dora
Speaker 4: does and for those that actually don't know, Dora was
Speaker 4: a separate company that Google bought in twenty eighteen that
Speaker 4: was focused on the delivery metrics that we all know
Speaker 4: and now love in and around engineering to have.
Speaker 5: I mean, there's nothing else.
Speaker 4: I say, there's nothing else, although there are a bunch
Speaker 4: of companies that have started to experiment with a sort
Speaker 4: of revision of them called Space, which I don't have
Speaker 4: a lot of experience with, but does handle additional aspects
Speaker 4: that some of the Dora metrics are missing.
Speaker 1: Is space another acronym for something?
Speaker 4: Yeah, you know, because people love acronyms and they don't
Speaker 4: mean anything, and I'm sure it stands for something, but
Speaker 4: I think realistically most organizations can do a better job
Speaker 4: by just understanding where their metrics are currently at and
Speaker 4: focusing on improving them irrelevant of what they are, because
Speaker 4: no metric is entirely perfect. Because of the companies that
Speaker 4: I've worked with in my history, often what happens is
Speaker 4: they say, oh, we're we have great Dora metrics, or
Speaker 4: we even have bad door metrics, and we need to
Speaker 4: do better improving them. And then I look at how
Speaker 4: they're capturing those metrics and they couldn't be more wrong.
Speaker 4: Like you look at them and they're say, let's say,
Speaker 4: using feature flags and whatnot, and so you're looking at
Speaker 4: your time to roll out changes in production. But those
Speaker 4: changes in production have never been tested, they're never being used,
Speaker 4: and they're behind flags that aren't even being that don't
Speaker 4: make that functionality available to your customers. And so it's like,
Speaker 4: can you really even say it's in the production. And
Speaker 4: yet these companies are saying, oh, yeah, we roll out
Speaker 4: our code immediately. It's all in craud, it's all being used,
Speaker 4: and so our values for that are really high. So
Speaker 4: I mean, I think in those cases that it doesn't
Speaker 4: matter what the metric is if you're recording it in
Speaker 4: a way which doesn't encourage good engineering practices.
Speaker 2: So I think maybe we should backtrack that a little bit.
Speaker 2: What is a Dora metric?
Speaker 1: Dora is this little girl in a cartoon with a
Speaker 1: monkey that gets chased by a fox, and evidently she
Speaker 1: does reporting in between episodes of making cartoons.
Speaker 3: That's fair. I'll bet she'd be a better reporter than most.
Speaker 1: Yeah, that was a reference to Dora the Explorer. For
Speaker 1: anyone who didn't get that joke, or.
Speaker 5: Maybe before my time, it was gone.
Speaker 1: Yeah, it was when my kids were young, so it's
Speaker 1: probably pretty old at this point.
Speaker 2: My oldest really went through like a phase where she
Speaker 2: really loved Dora.
Speaker 3: She was like three or something.
Speaker 2: That was very cute. It was one of the shows
Speaker 2: that I actually liked too. Not all of them, but
Speaker 2: some of them are adorable.
Speaker 1: I'm sorry, I totally derailed the topic. What was your question, Jillian?
Speaker 2: Yeah, what is it? Door? We're talking about Dora metrics
Speaker 2: and it's like, well, what is a Dora metric? First
Speaker 2: of all, I mean, I suppose we can all kind
Speaker 2: of infer it, right, It's some type of metric that
Speaker 2: we're using to and for how devopsy we really all are,
Speaker 2: Like who's got the highest score? Is there a scoreboard
Speaker 2: that we can we can enter ourselves into, Like what
Speaker 2: are we doing here?
Speaker 4: So there was actually like four official ones that everyone
Speaker 4: could could be using. And this is where the test
Speaker 4: really comes because I don't know if I remember all
Speaker 4: of them off.
Speaker 5: On my head.
Speaker 4: But there is a mean time to resolution when a
Speaker 4: failure happens and you need to go and resolve it
Speaker 4: and get your production state back to working according to
Speaker 4: your SLAS. Then there's cycle time to deploy changes from
Speaker 4: the time you start working on them. There's change failure rate,
Speaker 4: how often failures actually happen because of the changes you're making.
Speaker 4: And I missed one and I'm hoping Will knows which
Speaker 4: one I missed.
Speaker 1: Yeah, we have change lead time, deployment frequency with frequency yeah,
Speaker 1: then the change fail rate and deployment recovery time.
Speaker 5: Yeah.
Speaker 1: Yeah. So it's interesting. It's one hundred and twenty pages
Speaker 1: of texts that just boils down to those four metrics.
Speaker 5: So normally, normally that's what they talk about.
Speaker 4: And so the question is that since they've been doing
Speaker 4: this what's new every year that is relevant because once
Speaker 4: you sort of share that these are the metrics that
Speaker 4: every engineering team could be utilizing to decide like how
Speaker 4: great they are. And they talk about benchmarks and where
Speaker 4: elite teams are versus maybe behind the curve teams that
Speaker 4: have a long way to go, what changes from year
Speaker 4: to year. And so this year they dove a lot
Speaker 4: more into three areas. The first one was they started
Speaker 4: talking about a new metric the amount of rework you have,
Speaker 4: so when you thought something was completed and you have
Speaker 4: to go back. And this is really interesting because it
Speaker 4: was always something that made a lot of sense to
Speaker 4: me and wasn't included something else that they don't really
Speaker 4: talk about. It's like the number of support tickets you get,
Speaker 4: like whether or not, Like there may not be a
Speaker 4: production incident in any way, but your customers may be
Speaker 4: concerned and confused, your users.
Speaker 5: May not know what's going on, and that's not really included.
Speaker 4: So they talk about expanding the metrics and maybe breaking
Speaker 4: them down into different areas that are all independent of
Speaker 4: each other. And that's sort of what's important, right, Metrics
Speaker 4: that depend on each other or have high correlation or
Speaker 4: high cohesion tend not to be as effective because they measure.
Speaker 5: The same thing.
Speaker 4: Then the report moves on to in depth review of
Speaker 4: the impact AI in our industry and the world, really
Speaker 4: about engineering teams and software, and the last part talks
Speaker 4: a lot about platform engineering and its impact.
Speaker 1: Yeah, before we dig into those, I think the thing
Speaker 1: that stood out to me most in here is the
Speaker 1: section applying insights from Dora, and it says the recommended
Speaker 1: approach is to identify the outcome you would like to improve,
Speaker 1: measure your baseline or current state, and then I feel
Speaker 1: like this is where so many places start falling down.
Speaker 1: So the next step is to develop a hypothesis about
Speaker 1: what might get you to your closer state, and then
Speaker 1: agree to a plan for improvement, do the work, and
Speaker 1: then the last one measure the progress that you've made.
Speaker 1: I feel like that's where we fail a lot of times.
Speaker 1: You know, we like say, hey, we're going to make
Speaker 1: this better, and then we start working on it and
Speaker 1: never take a step back to say did we actually
Speaker 1: make it better? And if not, let's rip that out,
Speaker 1: And so over time we end up dragging all of
Speaker 1: these failed attempts with us adding you know, layers of
Speaker 1: complexity or bureaucracy or whatever the addition was, and we
Speaker 1: just keep dragging it along with us, even though it
Speaker 1: doesn't work for us or improve the situation for us.
Speaker 2: Yeah.
Speaker 4: No, I mean I think we bring that up a
Speaker 4: lot in this podcast, and I think the report really
Speaker 4: dives into it in the executive summary section. Which is
Speaker 4: an experiment is a hypothesis that is time bound with
Speaker 4: retrospective at some point. And often I think we do
Speaker 4: see a lot of organizations say, oh, we could try
Speaker 4: this thing, which is great, like you should absolutely experiment,
Speaker 4: but then there is no review, like was it even
Speaker 4: a good thing.
Speaker 5: For us to do?
Speaker 4: And that's like there is no like any experiment is
Speaker 4: a success, even if it rejects your hypothesis, if you
Speaker 4: come out with some learning. The only failed experiment is
Speaker 4: when you start it and never stop. And I think
Speaker 4: that's what a lot of organizations end up doing. They
Speaker 4: try something out, they experiment with a different team structure,
Speaker 4: even something that we believe is good. You know. I
Speaker 4: think it was a few years ago that the concept
Speaker 4: of team toypapologies came out with different types of teams
Speaker 4: and how they will work, stream aligned teams, platform teams,
Speaker 4: complex subsystem teams, etc. And it's a great idea to
Speaker 4: shift an organizational structure to focus on these, But even
Speaker 4: if it's a thing that everyone agrees on, you should
Speaker 4: still be like, we believe that a different org structure
Speaker 4: would make sense for us, and then measure why that
Speaker 4: is the case at some point later, like did it
Speaker 4: actually have an impact? Because if you don't do that,
Speaker 4: someone else is just going to say later, oh, that
Speaker 4: was bad for us, and you're not going to have
Speaker 4: any data to really back that up. A common argument
Speaker 4: against that has been the notion like rapid reteaming or
Speaker 4: dynamic reteaming. I think it's a book on the topic, actually,
Speaker 4: which is this concept ofok, like you swarm on an idea,
Speaker 4: you just get it to completion, and the you, you know,
Speaker 4: di spand the team and create a new team to
Speaker 4: focus on it, and without really understanding when your core
Speaker 4: businesses or why you're making the change, just arbitrarily making
Speaker 4: changes your organization, you don't know if it's going to
Speaker 4: be successful in the end.
Speaker 2: So my favorite way of thinking about applying the scientific
Speaker 2: method is when it involves people, because the only way
Speaker 2: that you can actually do that is to spin off
Speaker 2: parallel universes and have this whole double lunch where you
Speaker 2: have all these simulations of different people, and then you
Speaker 2: have your crystal ball where you're like, well, how did
Speaker 2: this actually work and how many times? And then you
Speaker 2: have the different iterations, and then you would have different
Speaker 2: like branching models of the different iterations too, and since
Speaker 2: you can't actually do that, you just get to guess.
Speaker 2: But at least with technology, we could have isolated environments
Speaker 2: where we really applied the scientific method. But you can't
Speaker 2: so much with people because people are chaos.
Speaker 5: I mean, you're absolutely right, Jillian.
Speaker 4: I feel like most companies don't have the sort of
Speaker 4: scale to be able to do this, and it's a
Speaker 4: mistake to believe, especially if you're like selling to businesses,
Speaker 4: that you have the ability to segregate a clear control
Speaker 4: group and a test and multiple test groups to validate
Speaker 4: what's going on. I do remember there was a not
Speaker 4: great book that referenced one of the major tech companies
Speaker 4: in the world today, and they had talked extensively about
Speaker 4: horse trading, individual countries, populations, as far as users go
Speaker 4: to perform human psychological experiments on the information that the
Speaker 4: company was providing them.
Speaker 5: To see how humans would deal with that sort of.
Speaker 4: Information, and so like at scale, you know, there is something,
Speaker 4: but that's not like most of the companies, Like it's
Speaker 4: not my company. I don't think it's well as company.
Speaker 4: I'm sure it's not yours, Agilian, Like most people don't
Speaker 4: work at a company where this is actually a real
Speaker 4: thing that could be done at a human level scale.
Speaker 2: No, I'm also uncomfortable with idea of like the human experimentation,
Speaker 2: Like my scenario was a lot more fun because I
Speaker 2: would just like to point that out there.
Speaker 1: So what you're saying is it's okay to experiment with
Speaker 1: humans as long as they're in parallel universes, just not
Speaker 1: in this one.
Speaker 2: Yeah, fine, I mean we agreed, that's all fine, Like
Speaker 2: we all like the crossover events and the Arrowverse, like
Speaker 2: this is what we do.
Speaker 5: Okay, I wouldn't.
Speaker 4: I wouldn't have mentioned the arrow Verse, but you know,
Speaker 4: I uh yeah, I mean I I'll just have done
Speaker 4: quite heavily in science fiction and fantasy, for sure. I
Speaker 4: Mean I always liked the question like did you did
Speaker 4: you enjoy.
Speaker 5: Being an older sibling?
Speaker 4: Because I only have younger siblings and I'm like I
Speaker 4: honestly don't know how to answer that question. I've never
Speaker 4: been a younger sibling, so I have.
Speaker 2: To see this is what I mean. We can't spin
Speaker 2: you off like we can't. We can't create like another
Speaker 2: experiment with you as a person where then you have
Speaker 2: the perception of having older siblings, but it doesn't work.
Speaker 2: I could spend it, Listen, I have a neuroscience to
Speaker 2: could spend like a lot of time mark doing about this,
Speaker 2: but we may.
Speaker 3: Want the actual devox portion of the show.
Speaker 4: I mean, I'm just wondering if there's, like you're secretly
Speaker 4: developing some sort of medical treatment here that like would
Speaker 4: cause you to forget your memories and still be able
Speaker 4: to persist in a new hypothetical reality where you could actually.
Speaker 1: My second time, oh that exists, Go smoke some weed,
Speaker 1: wake up in a room with a bunch of empty
Speaker 1: cereal bowls and boxes of fruity pebbles, going Wait, right.
Speaker 5: Is memory loss associated with potsmoking?
Speaker 1: I've heard that it is. I don't know that for
Speaker 1: a fact.
Speaker 4: I mean, maybe maybe we're missing out on this prime
Speaker 4: experimental tool that we could be using in legitimate professional.
Speaker 2: Applications, absolutely legitimate, I'm sure we could get lots of
Speaker 2: sign up for that and it'll be fine.
Speaker 3: Well, ethical dilemmas there.
Speaker 4: Well, what they did in my university is they used
Speaker 4: to run these experiments all the time.
Speaker 5: And all they do is they say that.
Speaker 2: You know, well, you have students, the students like care
Speaker 2: about that.
Speaker 4: And you pay them, Yeah, students that you pay like
Speaker 4: twenty dollars per study, and they will for sure sign up,
Speaker 4: sign away all of their rights and do whatever unethical
Speaker 4: thing you want in your treatment. No co.
Speaker 3: Yeah, that's how that goes.
Speaker 1: Wow, I'm glad we don't get real time analytics of
Speaker 1: our episodes. I don't know. Maybe maybe this would be
Speaker 1: a maybe this would actually increase our ratings. I don't know. Meanwhile,
Speaker 1: don't know.
Speaker 2: But the whole like organizational team building and all that stuff,
Speaker 2: all of those are very interesting to me because like,
Speaker 2: you can't really test it properly, and the only proper
Speaker 2: way to test anything is to use the scientific method,
Speaker 2: so you know. But then it's like, all right, so
Speaker 2: you can't test it properly, but what can you do
Speaker 2: to kind of approximate it? Like you know, obviously this
Speaker 2: is not black and white there, they're like ways you
Speaker 2: can you know, evaluate the performance of you know, of
Speaker 2: team organizations or any of that kind of stuff.
Speaker 3: And for the purposes of.
Speaker 2: This talk, we should probably you know, just just stick
Speaker 2: with reality of like, Okay, these are the things that
Speaker 2: people are doing and what are they doing. So was
Speaker 2: there anything in this report that discusses like, well, this
Speaker 2: is how your team should be organized if you have
Speaker 2: you have a platform team and an AI team and a.
Speaker 5: So it definitely tries to people the team.
Speaker 4: So I think this is one of those things where
Speaker 4: it's research oriented, not so much prescriptive or evaluating what happened.
Speaker 4: So the best it comes out with is this is
Speaker 4: what some people are doing, and this is the data,
Speaker 4: and here's us trying to maybe explain why the data
Speaker 4: is what it is. So there is an aspect of
Speaker 4: a belief that a lot of organizations are creating something
Speaker 4: that they're calling platform engineering or platform engineering teams.
Speaker 5: And obviously some of those organizations just took.
Speaker 4: Their age old infra team, didn't change anything or the
Speaker 4: people or the mindset and just rebrand, you know, slapped
Speaker 4: a new label on it and said this is a
Speaker 4: platform engineering team.
Speaker 5: We're goal then. So I mean, obviously.
Speaker 4: Those people and what they're executing on and their approach.
Speaker 4: It didn't change, and so they're likely going to be
Speaker 4: in the laggard bucket.
Speaker 5: And low performers.
Speaker 4: And then there's other teams that are other companies that
Speaker 4: really focused on how do we elevate what we're doing,
Speaker 4: how do we And one of the things they really
Speaker 4: care about is if we look back at the door metrics,
Speaker 4: is maybe something called developer productivity and what does that
Speaker 4: even mean realistically? But maybe it's the number of lines
Speaker 4: of code that you push out. They don't really talk
Speaker 4: about this, but it's how people feel. So maybe it's
Speaker 4: the number four requests or the number of GERA tickets
Speaker 4: that got completed. And no one uses Gira obviously anymore,
Speaker 4: so maybe they're using like linear, so linear tickets that
Speaker 4: got completed, and we can evaluate number of tickets that
Speaker 4: got completed equals velocity thereby productivity, and so did our
Speaker 4: organizational change that we made cause an increased number of
Speaker 4: tickets that get completed? And when it comes to platform engineering,
Speaker 4: the sort of conclusion is that it doesn't have as
Speaker 4: big of impact as we wanted to or believe it should.
Speaker 5: And even in the.
Speaker 4: Best companies it about averages out to if you have
Speaker 4: some sort of platform engineering team and they're doing a
Speaker 4: great job, it amounts to about five percent productivity increase
Speaker 4: per engineer. So that's five percent across your organization basically,
Speaker 4: and I think this is a really interesting number if
Speaker 4: you think a lot about it, because it means that
Speaker 4: if you have twenty engineers, you can hire one person
Speaker 4: to power your platform engineering team, because if you hire
Speaker 4: more than one person, then you're not going to be
Speaker 4: able to increase the productivity of those engineers any more
Speaker 4: than that. The five percent times twenty is one hundred
Speaker 4: percent of the amount of work that could possibly be done.
Speaker 4: And of course you don't want to have a team
Speaker 4: of one person. You probably have a team of four people.
Speaker 4: So realistically, if we just look at the numbers, having
Speaker 4: a platform engineering team before you have one hundred person
Speaker 4: organization means that you are spending too much time focusing
Speaker 4: on whatever the platform engineering.
Speaker 5: Team is doing and optimizing for things that the.
Speaker 4: Actual who your users are, the engineers, developers, product engineers
Speaker 4: don't get the benefit out of it. You can't increase
Speaker 4: the productctivity.
Speaker 1: Further, Yeah, I would agree with that because I think
Speaker 1: in those like the scenario were describing. You know, before
Speaker 1: you have one hundred engineers. I think for most companies
Speaker 1: in my experience, before that, you don't even really know
Speaker 1: what you're doing as a company. So it's it's hard
Speaker 1: to optimize for something when you don't know what that
Speaker 1: something is.
Speaker 5: Yeah, for sure, Julian, you're gonna say something.
Speaker 2: Oh, I was just going to say, you were talking
Speaker 2: about the difference between platform engineering teams, which is kind
Speaker 2: of a I would say, maybe like a newer phrase,
Speaker 2: and infrastructure teams, and I was just wondering, what do
Speaker 2: you what do you kind of see is the difference
Speaker 2: between the two of those.
Speaker 4: I mean, this is this is where like you know,
Speaker 4: it's because well, I mean no, I mean, I think
Speaker 4: it's a really great question because at least from my
Speaker 4: engineering experience, and I feel like I'm a little bit
Speaker 4: unique here because when I started, some of the things
Speaker 4: I was doing, I would definitely say came into effect
Speaker 4: because of the mindset of DevOps was growing up now
Speaker 4: getting out of it seed stages, and the work I
Speaker 4: was doing was specifically to eliminate teams in the organization
Speaker 4: that we're called platform engineering teams, and now almost twenty
Speaker 4: years later, we're now creating platform engineering teams and calling
Speaker 4: the platform engineers because I think everyone no one remembered
Speaker 4: that this term had been used previously, and they're doing
Speaker 4: similar things.
Speaker 5: But we've automated a lot of.
Speaker 4: Whey so much to the point of where we feel
Speaker 4: like the job now could exist again, but doing different things.
Speaker 4: So when I say infrastructure, what I had meant was
Speaker 4: is maybe writing some scripts to automate the installation of
Speaker 4: some software on bare metal racks for our servers that
Speaker 4: you got installed. Often they may even be responsible for
Speaker 4: setting installing the blades and putting together the server blades
Speaker 4: in the data center, but for sure writing the scripts
Speaker 4: to manage that and maybe send alerts and whatnot. And
Speaker 4: today I doubt any platform engineering team is doing that work.
Speaker 4: I mean, even when they have an on prem solution
Speaker 4: and an on prem data center, they're likely not even
Speaker 4: building the blades. You're outsourcing data center creation. Even if
Speaker 4: you have a physical onpreme data center, you are relying
Speaker 4: on these third party companies that come in build it
Speaker 4: for you, of which then you're likely not even installing
Speaker 4: the base software, but then you're installing some sort of
Speaker 4: native system on top of that, either at Kubernetes.
Speaker 5: Or Coros or something like that.
Speaker 1: I think I think the end product for both of
Speaker 1: those teams is the same, you know, to create a
Speaker 1: place for the engineering teams you support to run their applications.
Speaker 1: I think the big difference between the two is how
Speaker 1: they accomplish that goal or as an infrastructure engineer will
Speaker 1: be more hands on and more tightly interfaced with the
Speaker 1: infrastructure components themselves versus a platform engineering team. It abstracts
Speaker 1: themselves away from those those complexities and will rely on
Speaker 1: like either one or more platform engineering style tools that
Speaker 1: automate or abstract those tasks away from them. So I
Speaker 1: think I think the end product is the same as
Speaker 1: the approach of how you build that end product distinguishes
Speaker 1: the two teams.
Speaker 5: Yeah, for sure.
Speaker 4: I mean I think my CEO and I were talking
Speaker 4: about this recently and she brought up there's like a
Speaker 4: lot of different other things here that probably aren't necessarily
Speaker 4: normally associated with it, but things like building golden paths
Speaker 4: of sets of frameworks and tech stacks that we decide
Speaker 4: to use, but you know, someone has to do the
Speaker 4: difficult work of actually configuring it. Because these things are
Speaker 4: too unopinionated about what you're utilizing is one of them.
Speaker 4: Maybe you're utilizing some sort of shared digital or software
Speaker 4: driven infrastructure, infrastructure as code and coming up with those
Speaker 4: common components that go together would be another one. Sometimes
Speaker 4: there's an integration with IT system so SSO based integration
Speaker 4: from your IT provider, whatever it is in Google Workspace,
Speaker 4: et cetera, to power as SO in your organization, and
Speaker 4: the integration between other apps you have, or the management
Speaker 4: of the cloud accounts if you're using AWS or another
Speaker 4: cloud provider. And obviously those things didn't exist twenty years ago.
Speaker 2: So I'm not opposed to using like the newest buzzword,
Speaker 2: mostly because I do that every couple of years on
Speaker 2: my LinkedIn to get myself a raised Like that's all
Speaker 2: fine with me, But I specifically don't like the word
Speaker 2: platform because it's just it's too general. And then I'm
Speaker 2: in you know, these meetings where you have like the scientists,
Speaker 2: and then you have like the infrastructure people, and then
Speaker 2: you have like the drug discovery, and you have all
Speaker 2: these different groups and they all use the word platform
Speaker 2: and it means something completely different to each group, and
Speaker 2: then everybody's just having their own conversations, and they don't
Speaker 2: seem to realize that there's like four different conversations going
Speaker 2: on because we're all using the same word, but it
Speaker 2: does not mean the same thing. That's my rant about platform,
Speaker 2: which I suppose doesn't maybe have a whole lot to
Speaker 2: do with the Dora report, but I don't like it.
Speaker 3: It's too general a word.
Speaker 2: I think.
Speaker 5: I think you're you're totally on a good point there.
Speaker 4: Actually, So last year I ran a seminar at one
Speaker 4: of the DevOps Days conferences trying to define actually what
Speaker 4: a platform is, and after an hour talking on the subject,
Speaker 4: the best we were able to come up with is
Speaker 4: it's something horizontal that supports something else, which.
Speaker 1: Is just.
Speaker 2: The beams of my house.
Speaker 3: That's it.
Speaker 5: Isn't that the foundation though?
Speaker 3: So I mean, I don't know if the analogy broke
Speaker 3: down what you're talking about.
Speaker 5: Well, we we got platform.
Speaker 4: I think the best analogy we had is if you
Speaker 4: think about the gates for getting on a train, like
Speaker 4: the plat those are the platforms. You stand on the
Speaker 4: platform to get to the height of the entrance to
Speaker 4: the train. So the train if you had to board
Speaker 4: it from the ground layer or floor you know whatever
Speaker 4: from the earth. Uh, it's too far away. It's not
Speaker 4: well designed from that regard, and so the platform lifts
Speaker 4: you up to be able to utilize that technology as
Speaker 4: efficiently as possible. And what it doesn't handle is you
Speaker 4: think about that as a platform, it's very difficult to
Speaker 4: raise up that concrete slab or lower it. If the
Speaker 4: height of the train changes, or if the types of
Speaker 4: the trains change, or if the train is in a
Speaker 4: different location, you may have to extend the platform.
Speaker 5: And so this is I think Will sort of brought
Speaker 5: this up.
Speaker 4: One of the things that's actually highlighting the report is
Speaker 4: that platform engineering teams, what they build at first, tackles
Speaker 4: a lot of the low hanging fruit, and that's things
Speaker 4: that are easy, that have a lot of high return
Speaker 4: on investment that the engineering teams may be struggling with. Right,
Speaker 4: you see the same problem a bunch of times, and
Speaker 4: then you develop a solution to help solve them. Maybe
Speaker 4: it's a script or a bunch of open TOFU files
Speaker 4: that have the same coupled infrastructure together. But since technology
Speaker 4: is not static, the needs of the team slightly change
Speaker 4: over time, and so whatever you built is now out
Speaker 4: of date. It now is actually causing a cost and
Speaker 4: a burden to the teams to continue to use it
Speaker 4: because it's too opinionated, because it's a wrapper on top
Speaker 4: of what was actually available in the world before, and
Speaker 4: so actually platform teams in a one to three year timeframe,
Speaker 4: cost usually if not well maintained overall, actually hinder engineering
Speaker 4: teams for being able to deliver stuff effectively, and then
Speaker 4: until they realize, oh, we're actually getting in the way.
Speaker 4: We need to build a real product to actually deliver effectively,
Speaker 4: and then turn that around and extend the platform and
Speaker 4: improve it over time until it reaches this more effective
Speaker 4: long term vision of being more agnostic and being able
Speaker 4: to just like raise up very small amount but consistently
Speaker 4: across for whatever their user group is seems reasonable.
Speaker 3: Yeah, it seems like a good way to at least
Speaker 3: reason about it.
Speaker 2: If we're going to try to set up mental models
Speaker 2: and not going to use the word I almost use
Speaker 2: the word frameworks, but trying to not anymore.
Speaker 1: I'll take overloaded buzzwords for one thousand anlycs.
Speaker 4: Yeah, so, I mean this actually came up in one
Speaker 4: of the communities I'm in. We're trying to define what
Speaker 4: the difference between code and configuration.
Speaker 3: Is and it's a very blurry line now for sure.
Speaker 4: Right, And so my point was exactly as yours was, Jillian,
Speaker 4: which is like, it's just a word that helps convey
Speaker 4: some sort of point that we're trying to make to
Speaker 4: help get us from where we are to somewhere else.
Speaker 4: We're just using the vocabulary we have to try to
Speaker 4: convey our thoughts and the most conducive way we know.
Speaker 4: And of course different people understand those things to be
Speaker 4: different and apparently also have very strong opinions on what
Speaker 4: those things actually do mean. And if you try to
Speaker 4: challenge it, like you know, doesn't matter, they do sometimes.
Speaker 5: Not too happy about challenge.
Speaker 4: You know, they associate themselves with who they are based
Speaker 4: off of the tools they're using. And if you said, oh,
Speaker 4: you're just a configurator, you know, that may drive a
Speaker 4: stake in some people's you know, personal perspective of themselves.
Speaker 1: Yeah, for sure, that's that could go very deeply into
Speaker 1: a whole other topic. But a lot of people in
Speaker 1: our industry, they do tie their professional identity to the tools,
Speaker 1: like I'm a terrorform person or or I'm an antiable person,
Speaker 1: or i use AWS and it's one of the things
Speaker 1: I try to share with people whenever I talk to
Speaker 1: them about their careers is don't tie yourself to that technology.
Speaker 1: Tie yourself to the problems that you're solving, because in
Speaker 1: the course of your career, those technologies are going to change,
Speaker 1: but the problems you solve will be the same. And
Speaker 1: it's a very subtle but effective trick to making sure
Speaker 1: that you stay relevant for additional career options over the years.
Speaker 2: That's why my title is not Biopearl software developer anymore.
Speaker 2: I don't know what it is right now, but it's
Speaker 2: something that it's not biopearl. It's not catalysts.
Speaker 3: I don't know.
Speaker 2: I don't even remember what any of the old biogragmatics
Speaker 2: frameworks now, but they're all pearl based.
Speaker 1: Nice. How could you let that go?
Speaker 2: I don't.
Speaker 3: I don't know. I don't know.
Speaker 2: Yeah, I would think most people will kind of. I
Speaker 2: guess if you're talking to like newer professionals, I could
Speaker 2: see how that could be a problem.
Speaker 3: I don't know.
Speaker 2: I don't think people in the field for a long
Speaker 2: time would associate themselves with the particular tool, but I
Speaker 2: could be wrong about that.
Speaker 1: That Pearl is one of those, like the old school.
Speaker 3: Pearl for like a long long time.
Speaker 1: Oh yeah, and they'll still argue with you today about
Speaker 1: how great it is is and how humanity is doomed
Speaker 1: because we've moved on to something else.
Speaker 2: That was the first programming class I actually ever took.
Speaker 2: I had no idea what they were talking about, and
Speaker 2: the professor just launched into this whole Pearl versus Python thing,
Speaker 2: and I was like, I just want to program some robots,
Speaker 2: Like I don't even know what you're talking about, but
Speaker 2: it stuck with me.
Speaker 3: You know. Twenty something years later, here we are.
Speaker 1: It's worring. You were going to chime in on Pearl.
Speaker 5: Oh well, there is a joke there.
Speaker 4: There's like some OCR that converts artwork into text, and
Speaker 4: it was like a terrible OCR program, like it did
Speaker 4: not do a very good job at all, but if
Speaker 4: you threw an image at it, some famous artwork, and
Speaker 4: it would generate just arbitrary tax which was like periods
Speaker 4: and semicolons and dashes and ad signs and every literally
Speaker 4: everything had generated was a valid Pearl program.
Speaker 2: It's great gobbledygook, but Pearl will take it.
Speaker 1: Pearl is the honey Badger of programming languages.
Speaker 2: I still check in on like the Pearl six News
Speaker 2: sometimes and I'm like, ah, such hope, such hope, and
Speaker 2: such despair and like one small space on the internet,
Speaker 2: it's fascinating.
Speaker 1: We'll get that when we get half left three. So
Speaker 1: let's talk about artificial intelligence and how that factored into
Speaker 1: the Dora metrics this year.
Speaker 2: Because oh so excited for this part, I get to
Speaker 2: disagree with the whole thing, the whole one.
Speaker 1: I read, well, never let that lack of knowledge be
Speaker 1: a reason that you can't argue a point right.
Speaker 2: Lack of knowledge never form my opinions.
Speaker 1: So in looking at it, this is a card graph
Speaker 1: to read, I found the graphs to be challenging.
Speaker 4: So anyone who's actually looking at the report, what you
Speaker 4: have to remember is that at the bottom access it's
Speaker 4: often like zero point one point two point three, And
Speaker 4: what it really means is percentage of the total population
Speaker 4: or the actual percentage increase overall comparating the options. So
Speaker 4: really it's just often ten to twenty percent for the
Speaker 4: different areas.
Speaker 5: There's the elites are you.
Speaker 4: Know, somewhere around ten percent, and the lowest minority is
Speaker 4: also ten percent and is about twenty to thirty for
Speaker 4: the mid pieces. I don't know if that's the grapher
Speaker 4: looking at well, but I found personally the UX on
Speaker 4: the graphs not to be the best.
Speaker 1: Yeah, that's that's very confusing. So I'm going to go
Speaker 1: with the text up above saying that eighty one percent
Speaker 1: reported they're shifting their priorities to increase incorporating AI into
Speaker 1: their applications. What's not clear is, well, I guess given
Speaker 1: the scope of the audience, we can assume that eighty
Speaker 1: one percent of DevOps professionals are incorporating AI into the
Speaker 1: develop services.
Speaker 5: I think it's actually businesses.
Speaker 4: Is the total population that they're talking about here, Okay,
Speaker 4: eighty one percent of businesses are trying to incorporate AI
Speaker 4: into the final product that they're utilizing. I think there's
Speaker 4: also a different metric for the amount that are trying
Speaker 4: to utilize it as a tool to do some sort
Speaker 4: of software development. But I'm not sure that's totally highlighted there.
Speaker 4: I don't know which one the eighty one percent was.
Speaker 4: I'm so totally curious what the thing was that Jillian
Speaker 4: found that also Viamilly disagreed with I am just waiting
Speaker 4: in suspense here.
Speaker 1: So well, so now.
Speaker 3: I'm looking at the report and I can't find the sentence.
Speaker 2: But I thought that it was something like, you know,
Speaker 2: AI adoption was kind of less than predicted, or AI
Speaker 2: significance was less than predicted, And I found that to
Speaker 2: be completely opposite, especially in drug discovery companies, where everybody
Speaker 2: was like, yeah, whatever, Like we've seen these before, right,
Speaker 2: Like Microsofts came out with their AI that was for
Speaker 2: healthcare a few years ago that was supposed to revolutionize everything,
Speaker 2: and it revolutionize like nothing pretty much, and so on
Speaker 2: and so forth, and that has absolutely not been the
Speaker 2: case with these kind of with the newer models, with
Speaker 2: the chat ChiPT the cloud ones haven't gotten to try
Speaker 2: out the new reasoning ones yet. But in particular, the
Speaker 2: AI seems to have this kind of scary good knowledge
Speaker 2: of how chemistry works. And it's great because my brain
Speaker 2: is like equally divided in half on this opinion where
Speaker 2: it's like, well, the AI sort of understands language and
Speaker 2: human language and that's what it's trained on, and our
Speaker 2: representation of chemistry is also humans creating language to represent something.
Speaker 2: So this also makes sense, but it shouldn't it shouldn't
Speaker 2: make sense the way that it does. And like the
Speaker 2: results that you get, everybody you know, just looks at
Speaker 2: and says this should like this should not work. This
Speaker 2: is so you know, this is so scary good what
Speaker 2: the AI can do where you give it like a
Speaker 2: string representation of these compounds or these drugs or whatever,
Speaker 2: and you can take it and you can like start
Speaker 2: asking the AI to generate you Knew drugs, Like you
Speaker 2: can say like, hey, here's you know, I ran this
Speaker 2: experiment and here's all of these compounds that docked against
Speaker 2: this other compound. Make me, you know, make me fifty
Speaker 2: new ones that are going to do like you know,
Speaker 2: that are gonna be comparative or something like that. And
Speaker 2: then you can also with these bigger context windows, you
Speaker 2: can throw like all kinds of public data sets at it.
Speaker 2: You can take open targets, you can take the therapeutic
Speaker 2: like there's just a crazy amount of data that you
Speaker 2: can incorporate into it. And it seems to be doing really,
Speaker 2: really well. And so that's my very narrow view of AI.
Speaker 2: That is the only thing that I actually care about,
Speaker 2: which has nothing to do with this report. But I
Speaker 2: found the adoption to be much greater than was expected
Speaker 2: across the board, Like I had somebody say like, oh,
Speaker 2: you know, it was just this toy that I was
Speaker 2: playing with a couple of days ago, and now it's
Speaker 2: my whole job.
Speaker 3: So I just find this very interesting.
Speaker 5: I think that what is going to.
Speaker 2: Be making our medications and I don't know how I
Speaker 2: feel about that it's coming.
Speaker 4: I mean there is like I always had this perspective
Speaker 4: of humans make mistakes, and I think we forget that
Speaker 4: and we assume that if we design an AI process
Speaker 4: to handle something, that it has to be perfect and
Speaker 4: realistically it just has to be equal or better than
Speaker 4: what humans are doing, or even worse if there are
Speaker 4: some other trade offs. And often when we evaluate a process,
Speaker 4: there's a very common aspect for engineers who say, oh,
Speaker 4: we're looking at the code or the output and it's
Speaker 4: not actually one hundred percent.
Speaker 5: Correct, so we should reject it.
Speaker 4: And I don't think that's necessarily the right mentality to
Speaker 4: have here. But what you're messaging you're talking about, Jillian,
Speaker 4: I totally agree with. I think outside of the software
Speaker 4: engineering space, the usage in built in product that are
Speaker 4: being built or the end product can be hugely valuable.
Speaker 5: Almost everywhere.
Speaker 4: I think, as far as the report is concerned, we
Speaker 4: should look at the impact of the product the company
Speaker 4: is creating versus the adoption by the software engineering teams,
Speaker 4: and so I think overall, what we are seeing and
Speaker 4: what the report says is that everyone's optimistic or most
Speaker 4: people are optimistic about the usage of AI for doing
Speaker 4: the development, doing software development, using it in the teams.
Speaker 4: But when it comes to the actual product delivery, success, performance, durability, reliability,
Speaker 4: that actually decreases with the usage of AI. And I
Speaker 4: think this actually makes a lot of sense where we're
Speaker 4: employing very knowledgeable, inexperienced engineers, a lot of them to
Speaker 4: build us our products, and so of course we can.
Speaker 5: Do it faster, but the outcome of what we get
Speaker 5: is going to be worse, and it's.
Speaker 4: Going to be the case for I think a very
Speaker 4: long time unless we overcome the barrier of making truly
Speaker 4: more intelligent lms, and the nature of lms, I think
Speaker 4: is incredibly limited there in.
Speaker 2: That regard, I think the lms a little bit they're
Speaker 2: going to have the same problem specifically for writing code
Speaker 2: that the package managers do, because when I've tried AI
Speaker 2: is out for writing code it like it mostly works,
Speaker 2: but it'll mix in different syntax versions of different of
Speaker 2: like different programs. So like if you're using like PANDAS
Speaker 2: for example, that's kind of a common data science library,
Speaker 2: and they like to change their syntax quite a bit,
Speaker 2: so you'll get like different you know, different changes from
Speaker 2: different versions and things like that. So I think that'll
Speaker 2: be kind of the next I see that as sort
Speaker 2: of the next big change that will have to come
Speaker 2: along for the coding AIS to really take off.
Speaker 4: Yeah, I mean what I've seen is the is sort
Speaker 4: of two shotting the output where you keep a collective
Speaker 4: dock of some of the learnings that you want to
Speaker 4: have passed into every code generation you do. So, like,
Speaker 4: if you know there are differences in PANDACE, what you
Speaker 4: should do is have a doc that says and you
Speaker 4: can use your package manager as the source, like you know,
Speaker 4: only give me output that matches recommendation based off of
Speaker 4: the versions that I have or specifically say Panda's you
Speaker 4: know whatever, the version one point three five, and then
Speaker 4: it will help to make sure that it generates output
Speaker 4: that matches this index from that version. And if you
Speaker 4: don't give that, then of course it's going to pull
Speaker 4: out the most relevant thing based off of the problem
Speaker 4: that you've asked, which may not be in the version
Speaker 4: that you're utilizing.
Speaker 2: Well, I think that's going to be interesting with the
Speaker 2: newer models having such large context windows.
Speaker 3: Is that I want for PI.
Speaker 2: Charm to reach the point where I set up my
Speaker 2: interpreter and then the AI like it's just like this,
Speaker 2: this is the interpreter. These are all the versions and
Speaker 2: things that we should be using. Just stick with this
Speaker 2: the same way that if you have like the code
Speaker 2: Complete or anything like that set up, it takes from
Speaker 2: the It takes from the dock strings and stuff of
Speaker 2: the functions of the libraries that you're using. So I
Speaker 2: kind of want for it to do that.
Speaker 1: I think that that's one of the hidden benefits of
Speaker 1: using AI for specifically for things like writing code, is
Speaker 1: you'll give your chatbot or whatever you're using a prompt
Speaker 1: and then it comes back with an am answer and
Speaker 1: you find scenarios like that and you're like, oh, okay,
Speaker 1: that's one hundred percent my fault because I didn't tell
Speaker 1: you all of the constraints. And for me personally, I've
Speaker 1: found that it's helping me to become a better communicator
Speaker 1: because I would have the same problems interfacing with humans.
Speaker 1: I would say, hey, go do this task, and they
Speaker 1: come back and I'm like, ah, well, okay, I forgot
Speaker 1: to tell you it has to run on you know, Linux,
Speaker 1: or I need sixty four giga member. You know, all
Speaker 1: these little details that I should have communicated up front,
Speaker 1: and so the same mistakes apply. Is just more rapid
Speaker 1: feedback for me to understand that I wasn't clear in
Speaker 1: my request.
Speaker 5: Yeah, it's my fault.
Speaker 4: I should have mentioned that we were using a sixty
Speaker 4: four bit operating system here instead of a thirty two bit.
Speaker 4: I mean that's a that's a new invention, I know,
Speaker 4: like just last year.
Speaker 5: So I can.
Speaker 1: I'll you know, got.
Speaker 4: Confused here, but yeah, for sure that is exactly what's happening.
Speaker 1: And that was some of the looking at the task
Speaker 1: reliance on AI here seventy four point nine percent of
Speaker 1: people responding or using it for code writing, sixty two
Speaker 1: for code explanation. I use it a lot for that.
Speaker 2: What do you mean do you like highlight your code
Speaker 2: and then say write me a doc string or what
Speaker 2: do you do?
Speaker 1: No, No, I'll copy some code into it and say
Speaker 1: what is this thing doing? And then it comes back
Speaker 1: and tells me, especially in like whenever you know it's
Speaker 1: a language I don't work with very frequently, you know,
Speaker 1: if I'm helping someone debug arrust application or whatever, you know,
Speaker 1: I'll drop that in there and say what does this
Speaker 1: thing do? And it breaks it down.
Speaker 4: See if you if you get in that situation, I
Speaker 4: almost recommend taking the next two steps, which are first
Speaker 4: then ask it to generate unit tests for it if
Speaker 4: they don't exist. Where the unit tests are the documentation
Speaker 4: for the code, and then tell it to rewrite the
Speaker 4: code so that it's simpler and more understandable. And you
Speaker 4: have the unit tests as well, so that the next
Speaker 4: verst netlooks of the code knows.
Speaker 5: That it what it does.
Speaker 1: Oh wow.
Speaker 4: And also you have the unit tests that were generated
Speaker 4: and validated regressions for the previous version, so you know
Speaker 4: the new version doesn't break either. And yeah, it's extra steps,
Speaker 4: but it reduces the loop there.
Speaker 1: Damn level up, Warren, I go, I got.
Speaker 5: Some good ones here.
Speaker 4: So I think the doc where you actually have a
Speaker 4: bunch of recommendations in that you keep including its context
Speaker 4: over and over again is one of them. Another one
Speaker 4: is that we're going to very quickly run into this
Speaker 4: place where LMS are trying to generate code in languages
Speaker 4: that is not optimized for like the code the software
Speaker 4: languages we've created, we're all created by humans to do
Speaker 4: human related understanding things, and over time we're going to
Speaker 4: if we lean more on LMS, we're going to get
Speaker 4: to the point where we actually want an LM focused
Speaker 4: language where it will be able to better understand how
Speaker 4: to generate code that does the correct thing, understand the
Speaker 4: context of what code has been written and documentation around
Speaker 4: it in order to do the right thing, Which means
Speaker 4: we're going to end up creating software languages which are
Speaker 4: more less understandable for human and more understandable for machines,
Speaker 4: until we get to a point where what we're writing
Speaker 4: is almost completely not understandable by us at all.
Speaker 5: But I think that's still a.
Speaker 4: Little ways out, but we're going to get new software
Speaker 4: languages for sure that better incorporate this this concept.
Speaker 1: Yeah, and I think that's the big like, that's the
Speaker 1: basis of every sci fi AI takes over the world
Speaker 1: story ever, right, is that AI starts improving things until
Speaker 1: it's beyond the point of us being able to understand
Speaker 1: what it's doing.
Speaker 4: Yeah, but I mean look at like there's very few
Speaker 4: of us. I mean I did do this in my
Speaker 4: academic career. But you go look at the bytecode that
Speaker 4: was generated or the assembly by the assembler, and you
Speaker 4: don't like you are looking at that and be like, oh, yeah,
Speaker 4: I know. I think shift like three registers here, that's
Speaker 4: clearly a crypto you know, hashing algorithm obviously with a
Speaker 4: bunch of jumps, and this is making an HDP call.
Speaker 4: You see the register here and and the requests in
Speaker 4: the socket map.
Speaker 5: Ya of course, like I know exactly what that's.
Speaker 1: Doing, right, binary in your head.
Speaker 4: Already looking at the minified javascriptsss for websites and you're like, oh, yeah,
Speaker 4: I know exactly which file this came from and what
Speaker 4: the stack trace was. When this is going to no,
Speaker 4: it's just like random letters that make no sense, you know,
Speaker 4: with parentheses and periods between them. So, I mean we're
Speaker 4: already at that stage. I think there's just another one
Speaker 4: above here, which is between using LMS as the input.
Speaker 4: They're still not the input to software development. It's gonna
Speaker 4: but it is going to change. We're going to change
Speaker 4: the languages that we're utilizing so that they match better
Speaker 4: with our non organic coding partners. I remember, like years
Speaker 4: ago when the browser wars were like like at their peak.
Speaker 4: We were as a group of people and we were
Speaker 4: talking about, you know, different browsers and everybody was watching
Speaker 4: for their for a favorite, you know. And one of
Speaker 4: the guys in a group he says, a real man
Speaker 4: just uses curl and parses the HTML in his head.
Speaker 2: But it's a fun one. But see that's why, like
Speaker 2: I'm a little bit surprised that I don't like the
Speaker 2: the lms don't do better with code because it is
Speaker 2: still humans like creating models out of language, right, I mean,
Speaker 2: like the code isn't completely that, but it can. You know,
Speaker 2: it's close enough. It's as close as chemistry is anyways.
Speaker 2: So why do we get these really good results in
Speaker 2: some of these fields that have these representations of data
Speaker 2: that at least in part use something that kind of
Speaker 2: looks like a language, and not so much with the code.
Speaker 4: So someone's going to call me out on this because
Speaker 4: I am for sure not the expert. So there's that caveat,
Speaker 4: and I'm.
Speaker 2: Sure that's nobody is the expert that's the thing nobody knows.
Speaker 3: Every meeting I'm in, we're all like, why.
Speaker 2: Is it doing this?
Speaker 4: Like that part doesn't get Yeah, I mean, if it
Speaker 4: gets cut out of the podcast, someone will definitely call
Speaker 4: me out. So handling human speech in written words has
Speaker 4: a very single directionality to it, forward directionality as far
Speaker 4: as linear goes in time, you sort of read the context,
Speaker 4: you get the result, whereas software development has a need
Speaker 4: for the output to be sort of understood from a
Speaker 4: huge context of a single class, an object oriented or
Speaker 4: multiple functions, et cetera. And so there's at least a
Speaker 4: bi directionality things going forward and back. Like by the
Speaker 4: end of function or a file of which you have
Speaker 4: some source code, there's maybe some braces at the end, etc.
Speaker 4: Which aren't just based on the previous lines. They do
Speaker 4: impact what should have been written almost and same across files.
Speaker 4: Whereas you could have a many page book where each
Speaker 4: page in the book only follow from the previous one,
Speaker 4: but it's software development is more of like a choose
Speaker 4: your own adventure.
Speaker 5: Some of those functions are used outside.
Speaker 4: Of the context of that file, and so there's a
Speaker 4: lot that goes into it that isn't just the same
Speaker 4: as natural language, and so there's actually different models that
Speaker 4: are being used in order, like built up and design
Speaker 4: and how we're actually orchestrating the consumption of that data
Speaker 4: and the neural net that we're creating to process that
Speaker 4: data is fundamentally different depending on the output. It's much
Speaker 4: closer to say, creating an image than it is a
Speaker 4: human language, like a written text.
Speaker 2: That's interesting, Yeah, you're right, all, but the probability space
Speaker 2: is like much greater with code than it is with
Speaker 2: human language, right, Like if you look at how the
Speaker 2: AI kind of takes thing, it's part of a very
Speaker 2: simplified explanation is that you know, it sort of creates
Speaker 2: like the probability that the next character in the sentence
Speaker 2: will be this thing, and it does that sometimes. And
Speaker 2: with natural language, if people speak king, I mean, I
Speaker 2: don't know, but I don't know that the severb and
Speaker 2: documented anywhere. But I would guess that it's much much
Speaker 2: smaller for natural language than it is for code. You
Speaker 2: can investigate your own adventure.
Speaker 4: For you can understand a lot more with mistakes i'll
Speaker 4: call them in human natural language, things that don't make sense,
Speaker 4: sentences that run on or curtail or miss the predicate,
Speaker 4: whereas with code it's going to get executed, and so
Speaker 4: there is some fundamental difference here. I don't know how
Speaker 4: it is in chemistry. I always can't speak to that.
Speaker 4: It's possible that the models are highly both fine tuned
Speaker 4: with that in regard, or maybe it is just the
Speaker 4: obvious result of a bunch of natural language as well.
Speaker 4: It lends itself well to that. Whereas but I could
Speaker 4: imagine experiment diagrams or state diagram transfers I forgot they're
Speaker 4: called in chemistry, you know, it can convert from a
Speaker 4: couple of different molecules to different other kind of molecules
Speaker 4: like that. You know, of course, of say, a chemistry
Speaker 4: book could be challenging to get right, like it could
Speaker 4: definitely swap from talking about one kind of molecule to
Speaker 4: a different one later. And where a natural language that
Speaker 4: may not totally matter within the context of running experiment,
Speaker 4: are actually designing a product that someone could use, Uh,
Speaker 4: would not make sense at all.
Speaker 2: That is always a little bit freaky about the AI
Speaker 2: doing things, especially with like medicine and drug discovery, where
Speaker 2: you're like, listen, one compound is safe for you to
Speaker 2: take the mirror image of that compound is going to
Speaker 2: kill you.
Speaker 3: Like, and these are you know, they're the same thing.
Speaker 2: They're just they're just mirror images of each other. We've
Speaker 2: got to make sure that the AI, that the AI
Speaker 2: gets this. Sometimes. I know the ones that I've experimented
Speaker 2: with have just been the general AI models. I know,
Speaker 2: like new ones are coming out all the time, right,
Speaker 2: and I'm sure there are going to be ones that
Speaker 2: are more targeted, you know, specifically to certain fields like
Speaker 2: drug discovery or astronomy or you know, whatever the things are.
Speaker 4: I have to know, now, does it often misdate getting
Speaker 4: the el chairo and ar chylal molecules mixed up, or
Speaker 4: is it like does it know that it should mostly
Speaker 4: be l chiral.
Speaker 2: I don't know, because I haven't been running those types
Speaker 2: of experiments. That was just kind of the easiest thing
Speaker 2: that I could pop up with in my head to say,
Speaker 2: these are the stakes for the AI getting things wrong,
Speaker 2: and they can be very simple switches, right, Like this
Speaker 2: has happened in human labs where there was I don't know,
Speaker 2: there was like a type of medication given to women
Speaker 2: during pregnancy for nausea that caused a bunch of birth
Speaker 2: defects and it was just a chiral compound, so your
Speaker 2: chances were, you know, like almost fifty to fifty if
Speaker 2: you've taken it things like that.
Speaker 3: So I don't know, Sorry, I lost my train of
Speaker 3: thought there. I don't know what I was going on about.
Speaker 4: I mean, this was always so rivetting for me that
Speaker 4: the molecular compound equations could result in al chiral and
Speaker 4: ar chylal compounds. And in the human world at least
Speaker 4: like ninety nine percent of things that we encounter are
Speaker 4: all l chiral and they're all mostly harmless to us,
Speaker 4: Like the archylo versions of them, which almost don't exist
Speaker 4: in nature, are all like incredibly poisonous for us to
Speaker 4: consume in any regard, causing instant death.
Speaker 5: And I don't know, this is just always so interesting
Speaker 5: to me.
Speaker 3: Chemistry is pretty wild. Like chemistry too.
Speaker 4: I'm trying to think of there's anything left in the
Speaker 4: in the report that we haven't talked about.
Speaker 1: There's one thing I want to bring up because I
Speaker 1: found this to be really interesting. The drivers of adoption
Speaker 1: for AI number one, marketing number two, for GO bureaucracy
Speaker 1: number three. What if our competitor implements AI before us,
Speaker 1: and I find those interesting because I think those are
Speaker 1: all the exact wrong reasons to implement AI. Like none
Speaker 1: of those say, oh, because it's going to help us
Speaker 1: to live a better product to our customers, which I
Speaker 1: think is should be like the primary goal, well to
Speaker 1: do to do that at a at a profitable margin,
Speaker 1: and that's just kind of the end result.
Speaker 2: And people tend to, you know, drill down more and
Speaker 2: get more into the particulars of what they're doing on
Speaker 2: their day to day.
Speaker 3: But I think that like if the.
Speaker 1: But you can do all three of those things and
Speaker 1: not make a better product for your customer the way
Speaker 1: that they're stated here. Anyways, it could be.
Speaker 3: It could just be screwing around with AI, and it
Speaker 3: couldn't you know, not mean anything.
Speaker 5: I think there is right now. I think what's happening
Speaker 5: is it can be.
Speaker 4: It's always a challenge for you when you're marketing your
Speaker 4: product to explain concisely what the value is that you're
Speaker 4: offering so that you capture your customers to come and
Speaker 4: utilize your things. And because it's such a challenge, I
Speaker 4: think a lot of people are jumping to just saying
Speaker 4: that AI is included in it because it is being
Speaker 4: used as a proxy for all the things in which
Speaker 4: AI could hypothetically be doing to make it better. But
Speaker 4: then there is this sort of death spiral where that
Speaker 4: encourages the use of AI in the product to do
Speaker 4: those things, even though you don't fully understand the value
Speaker 4: it's offering. And then you have to say that it's
Speaker 4: using AI to do that thing, because people are concerned
Speaker 4: that if you're using AI, the output may not actually
Speaker 4: be what it should be. It could be making more
Speaker 4: mistakes than you would expect if it was well coded
Speaker 4: and validated, and so you end up with this death cycle.
Speaker 5: And I think that is certainly part of the problem
Speaker 5: for sure.
Speaker 2: Yeah, I think especially on these experiments where they're large
Speaker 2: enough scale that you're not going to want to replicate
Speaker 2: them with people. So I see a lot of literature
Speaker 2: mining projects where it's like, we want to know all
Speaker 2: the literature that's not available on whatever this new cancer,
Speaker 2: okay cancer, And I mean, I don't I wonder how
Speaker 2: much it would cost if I was going to be like,
Speaker 2: I'm going to download three thousand articles dust them. I
Speaker 2: know how much it costs for the AI to do it,
Speaker 2: But you know, just them with the AI, I'm going
Speaker 2: to ask it a bunch of questions and get some results.
Speaker 2: How much would the same experiment costs to be doing
Speaker 2: with people.
Speaker 3: I don't know, but.
Speaker 2: I'm going to guess, you know, I don't know how
Speaker 2: many how many papers can you really read before your
Speaker 2: brain turns to mush. I'm going to put it in
Speaker 2: about ten, so you know, you need a good number
Speaker 2: of people to do that.
Speaker 3: And then I don't know of anybody who's doing that
Speaker 3: kind of experiment, so I don't know.
Speaker 2: That's always like the interesting thing for me with like
Speaker 2: the marketing and the more like wiki style experiments is
Speaker 2: that I can't really can't really have a control group there,
Speaker 2: not one that I'm paying for anyways, not going to
Speaker 2: do that.
Speaker 4: I mean, I think it would be and it would
Speaker 4: be interesting if you're talking about trying to come up
Speaker 4: with like real conclusions based off of having downloaded or
Speaker 4: consumed all of those articles. Because if we look at
Speaker 4: what a human would do, like let's say intelligent software
Speaker 4: engineer one of our viewers, right probably write some sort
Speaker 4: of script to go and curl all of the documents,
Speaker 4: get them downloaded, and do some sort of rejex matching
Speaker 4: to find words in the document that you care about
Speaker 4: and then utilize that and investigate each one of those
Speaker 4: more specifically. Right, you know, you start like a very
Speaker 4: high level approach to filter out the information you don't
Speaker 4: care about into only the parts you do.
Speaker 5: And I think that can help a lot.
Speaker 4: Now we're actually talking about consuming all of them and
Speaker 4: coming up with a collective, unified epiphany on the topic.
Speaker 4: I'm not sure that the AI would be able to
Speaker 4: do much better job than basically regurgitate some parts of
Speaker 4: the individual documents itself. I don't think it's going to
Speaker 4: be able to pull from a second corpus of like
Speaker 4: sets of epiphanies you can have and like do a match, right,
Speaker 4: you know, I read these documents and then like, you know,
Speaker 4: there's six possible conclusions you can come.
Speaker 5: To, and you know, pick one of those.
Speaker 4: I think is where humans are still generating new ideas,
Speaker 4: whereas we've limited the LMS on what they will actually generate.
Speaker 4: If they generate stuff from not the documents, we call
Speaker 4: it wrong and usually it is. And if it generates
Speaker 4: up just from the documents, then I think it's losing
Speaker 4: some sort of insight that we would get even if
Speaker 4: you took the human friend brain approach.
Speaker 2: I think that's very true. The LLM is not going
Speaker 2: to innovate the way that humans are. Like presumably you
Speaker 2: could get some very smart human who looks at all
Speaker 2: those papers and says, this is what they all missed,
Speaker 2: and then we'd have, you know, some crazy new scientific
Speaker 2: thriller novel that the punch of nerds reading papers or something.
Speaker 3: I don't know.
Speaker 1: All right, it feels like time to do some picks. Okay,
Speaker 1: all right, Gillian, welcome back to the show. What's your
Speaker 1: first pick?
Speaker 2: I don't know.
Speaker 3: So the show got me thinking about neuroscience again.
Speaker 2: So I'm going to pick the book The Man who
Speaker 2: Mistook His Wife for a Hat by Oliver Something. But
Speaker 2: that's the title, pretty ditinctive, so you can you know.
Speaker 2: It's on Amazon, I promise. But it's really interesting because
Speaker 2: a lot of the experiments in neuroscience that are done
Speaker 2: you can't you can't just run around doing.
Speaker 3: Those two people.
Speaker 2: Okay, we have like ethical committees for stuff like that.
Speaker 2: So instead what happens is that you know, people who
Speaker 2: have suffered certain brain injuries or had strokes or some
Speaker 2: you know, like something like that has happened. They are
Speaker 2: very extensively studied, and then we say, oh this, you know,
Speaker 2: these parts of this person brain got knocked out, and
Speaker 2: then this is this is what happened. So then you
Speaker 2: get very interesting cases like you know, somebody saying, you know,
Speaker 2: this is my.
Speaker 3: Wife over here is actually a hat, and you.
Speaker 2: Just you have all kinds of crazy stuff that goes
Speaker 2: on when you start messing with the human brain. So
Speaker 2: that's that, and then you can learn all about why
Speaker 2: you should not be experimenting on people.
Speaker 1: Seems like a worthy lesson to learn.
Speaker 3: It's important why mess with the human brain?
Speaker 1: There may be some data points. We open up a
Speaker 1: couple of history books that provide additional context. There why
Speaker 1: that's a bad idea?
Speaker 2: There are there definitely are really really maybe I don't know,
Speaker 2: just throwing stuff out here.
Speaker 1: What about you, Warren? What do you got?
Speaker 5: Yeah? So I didn't pick a book this week.
Speaker 4: My pick is going to be QCon San Francisco, which
Speaker 4: should be happening this week. My my CEO of authors
Speaker 4: is actually giving the only security talk I believe at
Speaker 4: the whole conference, which is titled security or Convenience?
Speaker 5: Why not both?
Speaker 4: And so if you if you're there and you have
Speaker 4: the opportunity to go and see it, I highly recommend it.
Speaker 4: There's a lot of interesting insights where innovative companies are
Speaker 4: focusing some of their opportunities and what is almost now
Speaker 4: considered legacy.
Speaker 1: Right on. Nice cool for me. I'm going to pick
Speaker 1: a book that we we were talking about before, just
Speaker 1: before we recorded this episode, and I think it's with
Speaker 1: the conversation we had about AI. I think it ties
Speaker 1: into that and we'll have some relevance there. It's a
Speaker 1: book called Trust Me I'm Lying by Ryan Holiday. You
Speaker 1: may know him from The Daily Stoic, but prior to
Speaker 1: that venture, he was a marketing professional and did a
Speaker 1: bunch of different things in the marketing world that he's
Speaker 1: not proud of, and this book is his admission to
Speaker 1: doing those things. But it's really interesting to read to
Speaker 1: understand how corporate marketing works and the things that they're
Speaker 1: willing to do and will do to promote their brand.
Speaker 1: So yeah, trust Me on Lying by Ryan Holiday. Super
Speaker 1: interesting read. It will forever change every commercial website, ad
Speaker 1: and blog posts that you ever read. And do you
Speaker 1: guys hear that clicking in the background? Is it just me?
Speaker 1: I do Okay, I think that's.
Speaker 3: My computer, right, n yeah, that's good.
Speaker 1: Yeah. Well sometimes it's just in my head and then
Speaker 1: I ask and everybody stares at me awkward, going uh no, dude,
Speaker 1: you just go over there in the corner and do
Speaker 1: your thing, and I'm like, okay, all right, Well with that,
Speaker 1: we have an episode. Thank you Warren, thank you Jillian,
Speaker 1: and to all the listeners out there, thank you for
Speaker 1: hanging out listening to our rants and ramblings and misdirection.
Speaker 1: I hope you enjoyed the show. If you did or
Speaker 1: if you didn't, you know how to find us on
Speaker 1: various social media platforms, so give us a shout and
Speaker 1: let us know. We'll see in the next episode.