Well, hello, hello, and welcome back for another wonderful episode of Jobscript
Jabber. On today's show, we have first our guest Connor books Bronston.
Oh, I'm sorry I spelled your name wrong. I spelled with two wins.
It happens, man, We'll fix it and post. All good.
Not something I'll worry about too much, given that both of my names you
could assume I'm spelling them a little wrong, so peple usually assume double ends
the first name Bronze sun flast name which just happened. Don't worry about all
right, cool, and then you give us the thirty second who we are
and yeah, pardon me here? So I'm the coast the Devin Rupted podcast.
We talk with engineering leaders every week like AJ and Steve and leaders at
you know, Slack, et cetera, about topics like debex. Jen Ai.
I also work at Linear B and one thing we've really done is dived
into data reports around house software engineering and software to the life cycle works.
So I'm excited to share some insights early insights actually from our Jena I report
what's out tomorrow, along with talking debax and some of the impacts we've seen
from things like the Dora study which we worked with Google on this last year,
so it should be incredible conversation and excited to dive in these insights to
you two, all right, And then we also have trusty Rusty Steve Hello,
Hello, Hello a j. We got to work on your posting skills
here. You always do the panelists before the guests say the best for last,
right, so we got to work on this. You're kind to me
on that one. We'll take. And then Connor, I had to keep
a straight face when you put me an engineering leader in the same sentence.
You know my general approach with work and podcasts, I always bring on people
smarter than me, and so my I am the leader in good dad jokes.
But we'll leave it at that, and that's a crucial contribution. So
that's very crucial, Yes, and I will show leadership at the end of
this episode in that area. That's right, leaders joke last, But no,
that's not how it goes now anyway. But with that all the way,
we'll go ahead and introduce today's topic. We're going to be talking about
jen AI, which is like Generation Z except not at all related. It's
just generative AI. And then developer experience and you've got those insights you're talking
about, So where where should we start? Connor, Yes, I think
a great place to start is a definition phase. So you mentioned generative AI.
Let's just make sure we level start with the audience and showcase that.
Yeah, when we say JENAI and software development, we're talking about generating code
through the use of an AI tool like Copilot, like these other tools out
there. And the cool thing that linear B has done is we have done
a study following up to the great work that the Dora team did this last
year and said, Okay, you know the Dora team when I met with
Nathan Harvey and we've partnered with them on their door report in twenty twenty three,
they found that there weren't early efficiency impacts from Jennai that were showing up
in their survey audience. However, what they did see was that teams that
were using Jenni and about fifty percent of the respondents were, saw improved happiness
on their deb teams. And we correlated that with, oh, you know,
maybe we don't have to necessarily spend as much time making tests. We
can have the AI helveles with that generating those or maybe the annoying parts of
the code can get kind of abstracted out, so I can then focus on
these strategic pieces. And that's where I think there's this huge opportunity. Initially
is like, hey, how can we remove these blockers? How can we
improve here? And now what we've done is we've followed this up and we've
done targeted surveys with over one hundred and fifty different development teams across the world
and are now adding in quantitative data as well from our linear By tooling.
Linear By for Context is a platform for software delivery management and software engineering insights.
We helped both provide quantitative and quality of metrics and automation for improvement.
And one of the things that we're going to be rolling out here shortly is
a beta of how you can actually track gen AI pole requests or partially generated
partially GENI generated pole requests and then compare them over time to code that is
not being affected by GENI so you can actually see are we getting efficiency gains?
Is my GENI initiative at the executive level making an impact or is this
kind of more hype than anything else? And we have some really interests,
Yeah, please define GENAI poll requests. Does that mean that the AI wrote
the description or that the AI wrote the code or the code contribution. So
we're we're working on a tool to automatically detect geni's contribution there, so linking
up with Copilot or other different GENAI code generation tools to say, hey,
this pole request is partially written or was written in concert with kind of a
GENAI code assistant, and then we can evaluate code quality down the line and
say, did we have to refractor it more with our issues that were caused
by it? Do we need to maybe add an addition test early on to
make sure that everything that is working. How does this fit? And we're
seeing some fascinating early insights. Okay, so before we get going too much
farther, we got to find some terms here. As a parent, when
I hear Dora, I'm thinking swiper, no swiping, you know, Dora
the explorer? Absolutely, So who is Dora? What is Dora? Why
is Dora important? And et CETERA great question. I am using a lot
of acronyms, and I apologize to anyone who's fallen along and knocketed them.
So Dora is the accelerate state of DevOps team at Google Cloud, and they've
done more than ten years now of seminal research in software development and software development
metrics to understand how productive engineering teams are, how to make teams happier and
help build high performing engineering teams. Okay, so moving on, we've established
dora. So what were you about to ask aj? So you Connor,
you just said something about results or insights that you were kind of summarizing that.
Yeah, it was probing for more detail. Happy, happy to dive
into some more. So I'll say a couple of things. One, across
our survey set, we saw that eighty seven percent of organizations are already planning
to invest or have already invested in the Genai tool in twenty twenty four.
So the idea that this is something that is very broad across the industry is
absolutely true. We're seeing that rapidly accelerating our data set. Compared to fifty
percent in survey respondents last year for the Door survey, We're not seing this
continue to increase, and it's it's a key area of focus, and I
think part of that is that efficiency gains people are starting to see, and
then the happiness gains. I mentioned that are clearly being shown across dev teams
that are labaging this. You know, a great example of a success around
this is Uber, and Uber has actually started to help not only generate code
internally using Genai coding tools, but also to generate comments for codes. So,
what's one of the most annoying things that happens in a software development team's
life cycle? It's like, Oh, I have to go review someone else's
code, right, Like okay, Like I have to, you know,
pause what I'm doing, maybe jump out of a meeting and I'm like,
hey, this, what do I have to understand this PR? Or what's
the context I need? We can start leveraging automations like the guests through automation
that linear B has to provide context, and then Uber's actually saying, hey,
we're going a step farther and we're actually going to help generate comments,
suggested comments that people can then evaluate immediately in that PR to help improve the
PR's quality. So huge opportunities there, And I know Ubers Developer Experience and
Developer Productivity teams have estimated they saved about ten million dollars in time saved for
engineers last year alone. And that doesn't even consider the potential positive benefits on
that happiness piece we talked about earlier. Of like, if I'm a dev
on a team and these annoying tasks are starting to get automated away, or
at least it's getting easier for me to do them, whether that's tests or
reviewing code, We're having better tools to help me that are enabled by generative
AI. This is a huge opportunity to improve retention, improve de have happiness.
And we've already seen a million correlations across various studies with happiness and high
performing teams. So it's a really exciting thing for the industry. Okay,
so we've got a we've got a gambit from there are people who literally do
not start coding without some sort of AI integration. And then on the other
side, we've got people who vehemently refuse or are banned from opening such such
tool. So can you tell us anything about that spectrum and it was any
of that type of data captured in the report? Yeah, And this report
will be out for everyone to read here the next few days, so definitely
go to linear beat on io to check out more information here. It should
be out forty eight hours, is ish from now. But the exciting thing
here, as you mentioned, Aja, is that you know, there's these
opportunities for improvement, but there are risks. And this is one of the
key things that we captured in this report, which is the risks are perceived
as very much compliance quality. IP security is a top concern for devs broadly,
but particularly with generative AI tools, where they can be kind of a
black box of what's this code that's coming out of here, We're you know,
giving access to our code base, what's the risk level here? And
what we saw is that risk levels or perceived risk levels drop across the board
as adoption grows. But IP and compliance are major concerns for frontline managers and
executives. And interestingly, something we saw is that the frontline managers are actually
more concerned about these risks, maybe because they have more in depth understanding of
what the tools are actually doing, maybe simply because they know they feel more
accountable to it than the directors and executives, who I think are feeling a
lot more pressure to go, oh, hey, I have an initiative I
need to do. My board is telling me we have to have an efficiency
initiative, we have to do more with less. I'm sure we've all heard
that terminology thrown around, and so there's this odd change between you know,
the business side which is really pushing this and says, you know, we
have to improve our software development with AI and the managers who are are to
your point at AGA concerned, and I think frontline devs as well in some
cases, who are saying, hey, is this the right thing to be
doing? Does this actually help me? In and AJ Before we started recording,
I know you mentioned that in your experiments with genet of AI code you
felt like there are times where maybe it just wasn't getting things right and you
weren't as tightly aligned to what you actually needed here, and that is definitely
something we saw as well. So there's a huge opportunity for improvement where like
basic tasks, automations, tests, these are things can really be helped.
But as code gets more and more complicated, it's harder to get your GENAI
code assistant to actually match the in depth needs of very particular coding needs.
So there's very much still a need for senior developers to come in and say,
hey, I understand this code based, let me help guide this,
let me make this work. And you know, one of the things that
we're predicting is that that need for senior dev is actually going to really increase,
because, particularly as junior devs start leveraging Genai code to just move faster
deliver more code into the system, we're going to need a lot more senior
devs who can help parse through that and make sure the quality is there.
And there's a huge risk that as we just simply increase the speed which which
we generate code, that there are quality issues in that review cycle, in
that actual delivery cycle, because we designed all of our software development systems to
have a certain amount of code going through them, and now we're saying maybe
we're have twenty percent more code, maybe have fifty percent more code. Are
the different systems of review, systems of tests that we expected to have here
going to work or are we going to have these huge blockages that maybe then
people don't spend as much time reviewing. And this is where there's this need
for us next level out of me like what I mentioned Uber so, I
like what we're doing with our free get stream tool, say okay, how
can we help these blockages in the software development life cycle? These area where
the pipe kind of tightens up, particularly code review move faster. So how
is code review going to be affected by AI? Because to me, like,
okay, so my experience with it, I use both Olama with the
code Up I think it was code up and code Wizard. There's two models
that I've switched between. With Olama and then with GPT chat GPT I use
just the main website. I don't have any VIM integration for code gen in
part because I've from what I've seen and experienced thus far, having it pop
up bad code half the time and me having to do the mental context switch
of like am I writing? Am I correcting code? Back and forth?
That's like that, I'm that's frustrating to me. So I'd rather just take
a few extra seconds to type things out and just type out what I know
I need to do rather than like approve or deny and retry on something that's
relatively you know simple on cogen. So for like little snippets and stuff,
I'm not doing it, but I will. I'll ask a GPT a question.
And again it's like it's like fifty percent of the time it's it's worth
it, I guess, because you know, it'll get it right, it'll
get it wrong. But I'll ask you the question, and it seems like
the closer my question is to Python code to start, the better. It's
going to be a generating go or job. But but there is some value
there because there are some things that are relatively common boilerplate that are you know,
it might be thirty lines of boilerplate and I'll ask it to do something
that I know is maybe in that realm, and it can generate it.
It's very good at that. Yeah, it is. It is. I
also asked it to write a backup routine and it wrote a function that would
delete all files in the folder. Yeah. That tracks as I think we've
both experienced here. The more complicated your use case, the farther away Jenai
is from being ready to really solve it for you without deep fine tuning.
This is where major enterprises have the opportunity to say, hey, we really
want to fine tune this tool internally so that it fits our code. Basic
it fits our needs, but right now it's really good at the basics helping
solve these kind of pain points. And where we're seeing it. The interesting
code review, as you asked, is that we have just more code moving
through the system. Because the basics are all easier now, people can now
just deliver code faster. And so now we're seeing this major pain point which
was frankly already a pain point for many software development teams, which is,
not only are there more code reviews to be done, there's more code to
be reviewed. People are still spending most of their time on generating code.
So what we've seen is this already existent pain point get exacerbated by the increased
speed which code is generated. Yeah. Yeah, but oh so what I
was gonna say is what I see it do is I'll see it add comments
to code and asking it to summarize code. It can it can often do
a really good job at that, even even if it's something pretty niche.
Like there's been cryptographic stuff that I didn't understand and I was working in some
cryptographic library and I paste in some code and I say, you know,
what does what does MKW stand for? Here? And it's like that's the
Mills Buiser kill Awowsen curve. And I'm like, oh, okay, is
that a real thing? And then you know, it turns out it actually
is, and I'm like, oh, okay, that's that. And so
it's done really good at generating comments, but then most of the time it's
pretty dumb. It's you know, because it's this adds two and four to
get six. I'm like, yeah, yeah. And so when I'm when
I'm thinking about in code review, I just most of what it would do.
I you know, if it was something for a cryptographic library, I
would you know, want somebody who's at a really high level to be handling
that because it's something that's really sensitive and we don't want to take a chance
that it's a hallucination, right, And if it's something that's really basic,
and then it's just going to spit out, Okay, this you know,
this really benign or banal description of code or something. Where where does it
Where is it helpful? What is it doing? That's that's really what's the
sweet spot for that. So I actually think you nailed the opportunity here,
which is context and and saying okay, how can we autogenre generate the right
piece of context. So maybe it's things like something that we're doing is PR
labeling. How long is this PR going to take? We can estimate based
off of a little machine learning on the back end and say oh, this
this PR is x number of lines of code long, one hundred four lines
of code long. This will take five minutes, or this will take thirty
minutes to review. And it'll can also give you, hey, here's a
few things that you might not know what this means to your point, you
know, the MKAV or whatever, and give you that context so that you
can hopefully jump in. And it's a lot easier to get acclimated because too
often, and I'm sure you've experienced this someone you know, you get assigned
as a reviewer for a PR for someone else and you used to go and
you're like, Okay, what are you doing here? You know, maybe
they didn't give you a great comment from the start with of like, oh,
here's the context, and that's where I think, Jenny, I can
start start to help us. So it's helping us very much in the Hey,
let's code let's code through the kind of annoying boiler plate faster. Let's
code test faster. Great, and then let's give context faster so that we
can then make more strategic decisions and spend our time on kind of the higher
leverage opportunities of Okay, does this all work. Let me apply my knowledge.
Let's talk about test for a second. My experience in testing code when
I really feel like I need tests is usually one of two situations. It's
usually either a regular expression or it's something where I actually am not certain what
the result is supposed to be. So I'm writing a test kind of to
say I think the result is supposed to be this, and then if I
find a case where it falls outside of those parameters, then I need to
manually review and say, Okay, is it that the code is not handling
an edge case or is it there was an edge case in the input that
needs to be handled in the test I'm typically not doing. I I don't
think that the type of when I'm when I'm reaching for tests, I mean,
I guess the regular expression stuff? Could I mean the test is really
there more to explain what the regular expression does. It's more like a comment
at that point. But then again, so is you know that other thing?
So? So what kind of test generation again? Is is where the
value is being brought to the table. Yeah, so I would say two
things, And I'll admit here that I haven't spent a ton of time generating
test with Jenny Echo, this is very much something that I'm hearing secondhand from
people that I'm seeing in our data. I've heard it. I've heard it.
I just haven't seen it. I haven't seen I totally will say,
oh yeah, I'm generating my tests, but I haven't seen the tests that
were generated to to be like, oh yeah, that brings a lot of
value to the table I should I for two examples, so so one that
regular expression when you mentioned totally a great use case like this is something that's
very parsible. It's I don't want to call it basic, but it's like,
okay, it's boiler plate that we've kind of talked about. They can
get this, let's let's help new things faster. Gave me a little time,
and the edge cases I think are still figured out. Though maybe you'll
have someone who's going to come on next week who's going to be an expert
in edge case testing with Jenna I and they can help help lead us there.
But the other area here is that when you have a large enough database,
So I'm going to keep using this Uber example because it's something that they
looked at and they're saying, Okay, now we have thousands of devs.
We have an idea of basically what our codebaseys we know a functions. Let's
take all of those tests and make them artifacts that help train this AI.
It's particularly saying, okay, let's suggest tests for you right off the bat
for some of those EDU cases. Here's things that we saw as potential educases
that are issues in previous similar areas of the code base. Let's just suggest
some right off the bat for you. And that's an area that I think
is a huge opportunity for companies that and particularly like B to B companies that
are working with a variety of clients. We can say, hey, can
we train a great database to improve testing off of this? And I think
there's a lot to be done there still, but we're hearing great results from
individual devs and small dev teams, and we're seeing some enterprise companies succeeded that.
So I expect there to be an explosion of these tools moving forward.
Yeah. So, I think one really interesting thing we saw on the report
was one of the things we asked Serve of respondents was hey, do you
prefer qualitative metrics or quantitative metrics? Ie, do you want hard data of
like this is ten percent better or do you want Hey, we saw developer
happiness improve. And what we saw is that, particularly with larger companies who
are usually farther along in their adoption journey for AI, they just have the
resources to do it and add it more than the software development process. They
were ready to start saying, hey, we want these hard metrics. We
want to see. Eight percent efficiency gain is the ROI of our Genai initiative,
you know, seven percent of efficiency gain. Whereas earlier on the initial
signals all, hey, our devs are happier, they feel like they're moving
faster. We see this gain in total lines of code created, but those
are all kind of bad mesh. Well, I mean, death happiness isn't
It's an amazing depax measure. We can definitely talk about that. But like
lines of code for example, is I think we all know not necessarily indicative
of great code or you know what you actually want and there are some terrible
favorite there. My favorite prs are more red lines than green lines. Yep.
And this is where I think companies now looking to say, Okay,
what's the actual impact of the code that we generated with gen Ai, And
this is where we're running our beta with linear B we're saying, okay,
let's label every PR that has been code generated with genera of AI or fully
generated with generative AI, and then track its life cycle to see its impact
on the codebase. And we're we're very much just starting to get data there.
We're going to be putting some out in a workshop here in a couple
of weeks. But I'm excited about the long term potential there because I think
we're going to have this transition over the next couple of months from you know,
survey data which is super useful, to survey data backed by these hard
data metrics, these quantitative data metrics that that can really dive into it.
And this is where you also see this in developer experience, which kind of
like was the early indicator here of success. So, as I mentioned in
our work with Google Clouds State of Software Development Report last year, we saw
these efficiency gains for teams because team members were happier, and it was kind
of correlating this positive feedback cycle for dev teams based off of the leveraging gen
AI, whether it's for tests or coding, those basics. And now I
think we're starting to see this wave and are going to continue to see this
wave of here's the actual impact of that org long term, here's where you
know they delivered X number of futures faster. And that's what we're really excited
about, is that next step, and that's where we're building towards. Okay,
and then the developer experience, and I want to allow us to take
this conversation outside of just the AI portion totally, but with the developer experience.
What were the areas in which that happiness is improving or that and well
and also I want to know, I want to know what what tools actually
before I go into the question about the happiness what what tools are people using?
Because I'm I'm familiar with Chad, GBT and Olama those are the ones
that I use the most. And then I know that a lot of people
are using vs code with the Microsoft uh CHATGBT plug in, the co pilot
plug in. What what else is there out there that people are using.
Yeah, so obviously Copilot's the one hundred pound gorilla here. They've got I
think one hundred and fifty thousand organizations across the world using it Chat GPT being
the more broadly available for every individual dev and everyone else, code whispers and
other that's out there that we saw. It was quite popular in our survey.
Codium, dot AI and TAB nine were a few others that we saw.
But you named both of the key ones there, okay, and then
Tab nine man, oh they were doing they were doing a marketing push.
And then I think two weeks later, uh, Microsoft announced the GPT copilot.
I was so sad for them that their timing was tough. Yeah,
yeah, but it's hard to compete with that, a company like Microsoft when
they decided to roll something all like that, well, and I mean,
and you don't know that it's coming right, like, yeah, they they
had they probably they've been working on their platform for however long, you know,
a few years, finally getting out of beta and and then just overnight.
Oh that's so rough. That's so rough. I did use Tab nine
and I kind of had the same you know, feeling feeling about it like
it was. It was. It was cool, it did. It was
so good with refactoring, so good in terms of you know, I need
to go change the next ten lines to be from I don't know grams to
kilograms or whatever, you know, some sort of I need to add sql
tags to the comments I needed whatever, like, it was so good at
that sort of thing. And I imagine code Pilot probably is too, although
or co Pilot, although I haven't used that. And then Codium I think
I've heard of this one. Is this any relation to VS Codeum Yes,
Codeum not AI is its own website, but I do believe it's related.
If I'm not miss speaking here, maybe Steve's gonna correct me here. Well,
no, I just said a panelist co panelist to mine on another podcast
was using Codium and he really liked it. He didn't get into details,
but it was just the way that AI was just sort of built into everything
in the ide was was the way he described it is very helpful. So
I haven't used it myself for seen it, but I just remember his description
of it. I know they have uh well the jet Brains and VS code
Free versions, so I do. I do believe they're related there, Oh,
je brands, I use PHP storm. I checked that out. Well,
you know, I'd be happy to pay for their their their VIM plug
in when that comes out. Okay, is this one more privacy focused or
something? Or what makes it? Do you know what makes it different I'll
be honest, I don't. I don't know a ton about what makes that
makes them different. I've been pretty focused on I'm looking at copy it because
the data sets a lot bigger, and kind of evaluating our metrics across three
areas, So like adoption AKA, are you opening more pole requests? Are
you getting more poll requests merged? Are we just seeing more code flowing through?
And then the benefits piece of are you merging more frequently? Is coding
time going down? Are more stories getting completed? Are you being more accurate
when you plan stories? And then also we're starting to look at the risks
too, like our PR size is bloating, is rework rate up? Is
review depth down? Because people are just kind of skimming our PR has emerged
that review that kind of thing. What about do you have anything in place
for license violations? Because I think that it's it's been shown that in many
cases it's almost a direct copy and paste from the source material, And I
think that I haven't experienced this myself, but I've heard that people have complained
that it's generated GPL license code as like a whole block, which you know
has which is really weird because Microsoft, for the longest time, you know
that they wouldn't even they would not allow their employees to look at GPL code.
And to think that we've got an AI scanning all of GitHub where I
don't know what five percent of the code, I'm just making up a number,
you know, But it's like MIT and Apache. You're probably the highest
BSD, but there's a lot of GPL code out there. Any any thoughts
or observations on the licensing concern, I think it's a huge concern to your
point, like, this is one of the areas where we saw significant serns
this compliance side of things. We don't have a full solve for it yet.
I think this is something well, you can expect us to have an
automation out on our get stream platform within linear b here in the next month
or two that will hopefully help track that. But it's a it's definitely a
risk that I think companies are going to need to solve in it. I
mean, we're seeing lawsuits over this kind of thing, not just in the
code space, but look at the New York Times suing open ai for use
of their content and models. So the legal side of this is going to
be a massive thing. To unfurl over the coming months, no question.
Yeah, I can remember. I can remember the big stink when when Gethub's
copilot first came out that I remember seeing on Hacker News or other places that
there were authors that had private repos you know, with non GPO licenses,
so it should not have been annexed by by copilot, and they would type
in and here comes their code, just you know, straight out of the
way they read it coming up from copilot when it wasn't supposed to be.
So that was I can remember that stink, and I haven't really followed it
since then as to what actually happened with those instances. But it's just struck
me as odd as that I would think that, you know, co pilot
being a GitHub product and GitHub having access to all of the repos, how
you don't have a flag in there says okay, if this isn't private,
don't going into exist. Code didn't seem real complex if you own the whole
back in. But you know, obviously I'm not sure how that works,
but that was a big issue. Yeah, it's a great question, Steeve,
and it's really interesting too because we've talked about these like three use cases
for it right of you know, writing new code, writing tasks, assisting
in code reviews, and then there's this whole other side of i'll call it
like generating documentation, and some of those it's totally fine. Like genering documentation.
Sure, I don't I don't care if I'm copying something probably like maybe
there's some compliance concerns, but that writing new code piece, there are absolute
clients concerns. And and I know Copilot is working on this, but to
your point, it's surprising they're not a little farther along given that it is
a Microsoft for Lata product. Yeah, and I do wonder, Okay,
I've seen one of these reports that came out where it generated like entire paragraph
of paragraphs or text, or it would generate images that looked almost identical to
the original but with basically noise on top of it. And in that case,
I think that that is truly a copyright concern. But I also wonder
how much of this is boy cried wolf, where how many ways are there
to implement the you know, area of a triangle function? And so I
do want to, you know, just put in my own caution there,
even though I'm the one that brought this up. I haven't looked at what
these people are saying to verify, Oh, that's a legitimate copyright problem as
opposed to, well, how many ways in Java can you implement a JWT
siding function? You know, maybe it pulled from some quote unquote pulled from
GPL code, But maybe that's because the GPL code is also the same as
the MIT code and the Apache code for those three lines that everybody has in
common. And I think that that was one of the things in that Oracle
versus Google lawsuit over Java that if I recall correctly, ultimately they ruled in
favor of Google, saying yeah, like for common mathematical functions and you know
inputs and outputs of an API that you can't copyright pat in that stuff because
it's just too basic. So you know, there's there's probably some mix in
there of people crying wolf and making a big della nothing and then and then
some actual you know like what Steve was saying, where you know, there's
some sort of proprietary code but regular to a company, and then boom,
all of a sudden, it's been shared. And actually, that's one question
I had, is that how much of this code because with chat GPT,
the license with chat GPT, and I wish they would make it easier to
get a business account. I'm trying to get a business account and it is
it is really hard to get a business account with OpenAI. But anytime you
use chat GPT, anything that you type in or paste in goes into the
pool for model learning. But on the business account, if you use the
API and then the it doesn't go into the pool for model learning, you
keep it proprietary. Do you know anything about that? Well, I would
just say that I think this is a huge opportunity for open ai and all
these other companies here, and this is why you're seeing them get these crazy
valuations, is because every business is going to want to secure their own code
base and data and they don't want to extend it into the main chat GPT
data. So to your point, AJ like, this is a reason for
every business to look at that business business style version of it and paid that
extra dollar. Is that compliance use case? And it's a classic B two
B one right where it's like, oh, you know, anyone can get
our basic tooling, but you know you have to get our enterprise package in
order to add the compliance piece add you know, role based access control,
these other things, and I think you're absolutely going to see that. I
mean, we already are with all these AI companies, all right, So
now now let's take the turn. Unless, Steve, do you have any
more questions on this topic? Now, let's let's take the turn and go
into the developer experience the developer happiness topic genre. So we go ahead.
Let's start with the AI what what are the biggest improvements, what are people
excited about, What's what's making them happy? Yeah, So I mentioned Dora's
report from last year, and I kind of I'll just say the twenty twenty
three accelerate State of depop Support is what I'm saying when I'm referencing that.
So I interviewed Nathan Harvey, who's the head of Dora's kind of outreach team,
on our dev Rupted podcast last year and we were also partners Lenard View
as partners on the report data. And one of the really interesting things we
saw in that report is set they saw among the three thousand plus organizations they
surveyed that last year that about fifty percent of them were using AI tools already
and most were already considering an adoption, but of those folks, there was
a correlation with improved developer happiness on dev teams, and so this is where
we've started to see these great examples of teams that are saving time on documentation,
saving time on code review, saving time on tests, saving time on
codgeneration, and it's helping devs get rid of these annoying blockers in their time
and spend more time on the strategic inputs. And so it's a really key
piece of how you can potentially make your team both more efficient and happier.
And there's plenty of research, whether from Google or many others, that has
shown that happier teams perform better. There's this like positive feedback loop of hey,
I'm feel good, I have more energy, I maybe feel like I'm
being more impactful on my work. And this is where I think a lot
of companies are starting to wake up to the importance of that developer experience idea.
You can look at LinkedIn, which uses a developer experience index, which
is a specialized metric. LinkedIn provides to teams like aggregate scores based on a
number of different metrics such as you know local build times and you know happiness
scores on surveys and a variety of qualitative and quantitative measures, and they use
that to help kind of calibrate the complexity of developing within their orgon. And
there's plenty of the companies that Google does this, Peloton does this. Obviously
they're different pieces. Like for Peloton, for example, they look at engagement,
velocity, quality, and stability. And it's really important that the organization
starts to think about this in the context of am I making my devs perform
better because we're giving them the right tools, We're giving them the right time,
we're right frameworks, and am I going to be able to recruit and
retain dabs because of that? And so whether whether you're an individual developer or
you know, a major corporation is just looking at some of dollars and cents.
Because STEBAC should be something people are thinking about. I would like to
categorize this under the category of the sky is blue and the grass is green
as well. I mean, if you're you have developers and you want them
to do your work, it would seem logical you're going to do best when
you give them the tools that they need. This is I mean it extends
to more than developer tools. There was a recent article that came out and
this has been along the same lines, although it's not specifically code related.
There's whole studies that have been done about the workspace that you give engineers,
and that you will you find that engineers need an office space instead of being
crammed together in cube forms for various reasons. Has mostly to do with focus
and you know, not being interrupted and so on, and that's one of
those things that that to this day is a no brainer. You know,
anybody who's been an engineer or has worked in environments like that will tell you
you, oh, yeah, that I can focus better, I can concentrate,
I can get more done, and yet you still have people that cram
them all in the queue farms, and there's you know, there's various reasons
for that in terms of office space and having only so much space, you
know, versus being able to give everybody their own individual office. I can
still remember being out at Intel. Intel has a lot of their engineering offices
here on the West side of the Portland area and being out there one time
and holy cow, you literally had to have a map to know where you're
going in this gigantic queue farm, and I was like, how do people
get stuff done in here? So, you know, sorry to go off
on a tangent, but you know what you were saying struck to me?
Is you know it struck me as Yeah, Okay, that goes without saying
it doesn't have to be a dead job. Any job here, Go do
this job, but I'm not going to give you the tools you need to
do the job. Well, I output it. What kind of output are
you going to expect from them? Then? The challenge, I think is
that too often, particularly lately, software development and other R and D teams
are being beholden more to I'll call them CFO metrics without thinking about what's actually
happening on teams. How can we improve it? And so it's saying,
okay, well we can save money by doing X. We can you know,
cut hat count on this team and that'll make us more efficient on our
cap table and we can return more value to shareholders. And you want to
know the classic the poster child for this right now? Everybody knows about Boeing.
Yes, okay, Portland two weeks ago, plane takes off, door
panel comes flying off mid flight, two seats get checked out. A twelve
year old sitting next to them has his shirt ripped off. The plane comes
around and is able to to make a landing. It was an Alaska Airline's
flight, which happens to be an airline I fly quite a bit. And
this has been an ongoing thing with Boeing ever since nineteen ninety seven, especially
with well, especially with the seven thirty seven Max line of planes. Where
it used to be. I've seen articles written ad nauseum where Boeing used to
be an engineer driven company and everything was about quality, didn't matter the cost,
Let's get a good plane made, and that's how they developed reputation.
And then in ninety seven they merged with with it Pratt and Whitney or who
was it that they merged with. There was another giant AERONAYM. Mc donald
douglas. I think it was McDonald douglas. And the joke was that McDonald
douglas bought Boeing with their own money because they came in and pretty soon everything
was driven by cost and cost accounting and let's save money. Let's not do
anything that's going to cost a lot of money. So Boeing decides, hey,
we need to keep up with Airbus. But we don't want to take
ten years to develop a whole new product that's going to actually compete, So
let's just take our seven thirty seven and modify it and tweak it. So
right out of the gate, you've got planes crashing because of software that overrode
the pilots and forced the plane to dive into the ground. That happened at
least two times. And now you've got they started outsourcing production, outsourcing building
of a lot of this, which is where this particular problem was tracked to,
and a lot of QA stuff that happened there all because quality went out
the window. And we've got c suite people who weren't engineers making decisions solely
based on money and not on quality. So sorry, that's my rant,
But no, that comes right to mind as soon as I hear that.
That's an incredible example, Steve, and I think it really correlates to something
we're seeing a lot in software engineering right where we could name layoffs that have
happened in January ad nauseum, and a lot of the reason for them,
you know, whether it's Amazon layoffs, YouTube, just score, whatever else
might be. Because Section one seventy four of the US tax code got changed,
which has now made it so that you cannot expend right off R and
D expenses in the United States as part of tax You have to instead cost
capitalize them over five or fifteen years, making it more expensive to develop.
And companies are saying, oh, well, let's just we can outsource this,
Well, let's not actually pay attention to the quality piece. They're making
those same decisions that Boeing did. To your point, see Eve, and
I think you brought us something else that is really interesting, which is the
idea of the space that we built for developers. And I think it's important
to think not only about you know, the office space, the meat space
i'll call it, but also the digital space. Like platform engineering teams can
significantly enhance developer experience by building a tight information loop that identifies pain points and
gathers DEV feedback. And if they use you know, the right internal developer
portals to do this, you can you know, actively solicit develop or input
via surveys and use quantitative measures like local build times and cycle time to under
standing track blockers and broadly improve developer experience. So like thinking about both how
we're setting up devs in person and setting up devs in digital space is really
crucial, and I think you guys are doing great job highlighting that, because
if we instead spend our time thinking, okay, what's the most cost effective
solution, well, it's probably you know, pushing our dev team off to
some other country and cutting our US team entirely. And there are major risks
that come with that, but they're not usually risks in the moment. They're
usually long term risks. And too often companies are thinking, well, I'm
gonna I can cut costs right now, and I think Wall Street or my
investors are really going to like that. You know, my vcs will love
seeing our cap table look better here because we're spending less money on our R
and D costs. And you have to grapple with the whether or not that's
going to be good for your company and the product you're building, whether you
can actually succeed long term. And this is where I think a lot of
companies are making a major mistake is they aren't tracking developer experience metrics. They
aren't tracking their engineering metrics, and oddly so they could see, hey,
there's an efficiency drop, or you know, the pr quality that is being
merged is lower now. And yeah, this example you gave Steve of the
McDonnell Douglas merger with Boeing, I think is a perfect example of this in
an engineering heavy field. And I'll also say, like I was supposed to
be flying a seven thirty seven Max nine the next morning that got canceled,
so right there with you on the Alaska flights getting canceled and the impact there,
and they then ended up finding like bolts loose on multiple planes. What
could happen? Again, very scary yep. And then there was another one
in Japan where uh, there's four layers apparently to the windshield in the cockpit,
and one of them was cracked as they took off. So they came
back around and said, uh, yeah, no, So as far as
I know, all of the that particular seven thirty seven Max not als like
them seven thirty seven Max nine something I think had been grounded. But anyway,
yeah, I think they're bringing some back into service once they've been like
fully checked over. But definitely it's a super scary thing, and I think
we need to have that same risk concern in software engineering, like sure,
maybe we're not going to have a plane blow out its window. But maybe
the software you're building is impacting you know, whether that plan will dive into
the ground, as you mentioned, Steve, or maybe it's impacting people's bank
account information. They're they are very real world concerns with both the compliance and
security risks that we have to consider and also the quality of our code based
generally, and this is why I think it's so crucial that every software engineering
team above a few people needs to have some basic visibility on how is our
team performing? How is our company getting value out of this? And it
shouldn't be about, hey, is this individual developer delivering X lines of code?
Because it's a terrible metric of success, but it needs to be are
we, you know, delivering on our promises to the business? Are we
having to refactor a bunch or or in fact, are we, you know,
how, helping drive revenue by what we're doing. And I think any
engineering leader today, any dev leader, needs to be stepping up and saying,
hey, like, we have to help influence this discussion at all the
way up to the board level, or else we're going to see more CFOs
make that decision that you mentioned, Steve of saying, hey, let's cut
costs, let's just focus on shareholder value and not necessarily customer value. And
when you start getting away from customer value, that's when these huge risks come
into organizations. All right, So you've you've kind of touched on my favorite
topic there, which is in the world of sas, the customer comes last,
because you're taking money that was almost free printed from the federal reserve that's
given to venture capitalists that are only interested in growth and don't necessarily care about
profitability. So what happens on the black mirror side of things here? Do
you have any thoughts on that? How do we how do we how do
we uh incentivize that we actually care about the customer and the developer or you
know what risks are there with AI being Yeah, Well, so for example,
you know, people for hundreds of years have been saying, oh,
this technological improvement X is going to make everybody's quality of life go up without
having to work more and we're gonna, you know, we're gonna work less
hours. And this is you know, this has been going since you know,
probably the era of the factory. What actually happens every time, well,
oh, by the way, Steve, do you have a thing?
Did you put that thing up on the screen? I can't take that down?
I don't think Okay, yes I did, right? Oh sorry,
anyway, I just got distracted because the the I was expecting comments fade away
after twenty seconds or something, and it didn't. Oh no, I got
to take it down. Yeah. So so the you know what, what's
how do we what what might the abuse situation look like? And how do
we prevent it or curb it? Oh? Man, that's a that's a
big one. So I'll say I think what you're talking about here is that
a lot of the value that gets generated from technological innovations like AI or you
know, improved machinery that helps assembly lines or anything else has happened over the
last one hundred two hundred years, and during industrialization has accrued to capital because
uh, you know, shareholders are are driving this and they accrue most of
the gains, and not most of those gains aren't necessarily accruing to the working
class. I know, I'm swear, I'm not a Marxist, y'all,
I'll tell you, but I'm just this is we're talking about here. To
be fair, the rising Tide raises all ships. Like the total Amazon exists,
I have access to products. I personally have access to products that I
would not be able to get access to otherwise without having much more specialized knowledge
and having to drive much further to get to a specialized store. And as
your points to fake crap, and I can't always distinguish between the product I'm
trying to get in the fake version. But like, I have access to
more products, So it's not it's not like it's all bad. But in
particular, what I was pointing out is that they said, oh, you're
going to work less and you're going to have more. I do think that
we have more. We're certainly not working less. Are we work differently?
Yeah, I don't know. If we're happier, well, I think has
a lot of other impacts, right right, Yeah, sure, I think
we I think we work differently. Is really what it turns out to you
right where it's like, Okay, maybe now with these AI changes, we're
going to spend less time writing tests, We're gonna spend less time generating basic
code. We're gonna spend a lot more time strategizing on the deeper pieces of
code, or how does this all fit together, or understanding making sure these
systems don't break. Maybe like we're seeing an increased need for platform engineering and
internal developer portals and you know dev X teams at major companies because we need
to make sure all these systems we've built to improve development speed work. And
the other piece I think of what you're saying is like, you know,
this kind of goes back to our discussion of hey, dev teams need to
have these conversations. Dev leaders need to have these conversations with their CFOs,
with their partners in finance and their partners in sales and all these other place
parts of the company, and we need to be part of influencing the business
discussion and saying, hey, make sure some of these benefits do accrue to
dev teams, like we're helping provide more efficiency and more value, Like we
need to be able to work on things we're passionate about. We need to
be able to be you know, and sure we're have DEVAX metrics that look
at whether those teams engage and talk to the value of that. So I
do think you're spot on there, and it has absolutely allowed us to create
these incredible technological improvements But what I think we need to be really careful on
is assuming that will just mean we work less. Usually it just means we
work more efficiently, and then we try to increase overall value over all GDP
through that and hopefully you produce more value that then gets you the next pay
raise, the next piece up. And I think as an industry, if
we aren't helping shape that discussion by talking about DEVX, by talking about developer
productivity, by being part of that discussion, we're gonna let, to Steve's
point, the accountants be the ones who make the decisions, and that's where
we get into major trouble. Shout out good accountants. Ever, by the
way, I'm not saying accounts are bad. So one thing that was funny.
You're talking about devs teams. Originally that was called DevOps. The purpose
of DevOps was to make the developer experience better. Then we just rebranded ops
to DevOps without changing anything. I do agree with you. So what's the
new it's gonna happen with THEVAX team stuff? Well, now, platform engineering.
When you heard so, I mean to your point, it's all it's
all a bit of you know, six of one and a half dozen of
another. We kind of use these new terms to speak to new eras of
how we think about soccerlent, but we typically retain it right where it's like,
hey, the agile movement this huge thing, and then DevOps this huge
like layer on top of it. Now we're talking developer experience and platform engineering
and building on this, and really, I'll say my take is that it's
it's basically, how can we improve on the last standardized way we did this
and how can we keep leveling up? But is the role very similar?
Absolutely? Are many of the concepts the exact same? Absolutely? All right,
Well, it is about time for us to wrap up here. Is
there anything else that you wanted to share? Any any topics we didn't discuss.
No, I really enjoyed the conversation. AJ Steve has been super fun.
Thanks so much for having me on. Well we do picks at the
end. Hopefully you were informed, which is, you know, basically can
be anything just stuff that you're interested in, could be tech related, not
tech related. I'll go ahead and start to give you some time to think
about something. If you if you hadn't because I I'm guessing if you if
you got the email with the wrong calendar link. You probably didn't get the
email with the description anyway. So there's a couple of things that I still
that came to my mind a few weeks ago that I still haven't gone through
all of it. So let me see if I can scroll to where this
was. So one thing that I was going to talk about that I mean,
I've mentioned, just like like I did today, this whole issue with
you know, we've got the investors and they're focused on growth and and that's
a lot of what's driving SaaS companies is not profitability but just growth. And
then what happens when you've reached the market that the full market potential of the
growth? You know, what what happens when everybody who wears shoes has a
bluetooth smart shoe. What happens when everybody who needs a calendar link has calendarly?
What happens when everybody who wants to scroll infinitely and starve to death is
on Facebook? Right? And there's a video that let me see what the
title is. The thumbnail was something like ninety eight point nine market saturation.
Now what oh whoa, whoa, whoa whoa? That was really loud?
Sorry about that I I need to reconfigure my don't auto play plug in.
Yeah, okay, So the title of the video is market saturation equals ninety
eight point nine percent? Now what? So? I think this is an
interesting video and it just talks about this issue of these these SaaS companies are
starting to kind of cumulative. We all get to the point where growth is
no longer possible, and many of them still haven't figured out how to be
profitable. So what's going to give as there's no more investment to get and
there's no more people to reach with the product. I'm not it seems like
there should lead to a collapse. I don't think that it will. I
think that something else is going to proxy and take it place. I fear
it might be something government related or you know, I don't know. But
but it's this is a question that I think is I got I don't know.
I guess it's not really important thing about because nobody can do anything about
it. Nobody, you know, unless you're unless you're on the investor capital
side. We're getting really existential today. You're just going to experience it.
But I don't know. It's it's something that that I I wish more people
were aware of and and I really would like to see us get back to
having profitable businesses rather than having growth businesses, because certainly profitable, profitable businesses
are better for the customer by definition, because then, I would argue,
is the one who benefits. They can often be better for the employee too,
because you aren't having these kind of wild growth expectations. Instead, it's
how do we build a strong business. So I've been thinking about as too
lately, actually, AJ, which is you know, I've considered kind of
in the process of starting my own business. I've previously been a founder business
owner, and I'm very much looking at the bootstrap side of things and saying,
I don't want to take VC funding because the pressure that it provides to
me as a founder, to me as an employee is so substantial and some
of the goals that get set by our kind of perverse incentives as you put
out there, and there's a there's I think a big push now for folks
to say, hey, I'm going to bootstrap, I'm or I'm going to
take very minimal funding, or I'll take and I think this is a new
way for starting to see I'll take funding, but you get to share future
profits instead of hey, I just get a percentage of the company. And
I think we're going to see a continued wave of that going forward. Yeah,
I think that that. I hope I hope to see that. I
hope that the dah revolution uh kicks off strong and hard and we get back
to making products that deliver value to customers rather than than deliver poker face uh.
For for investors, I mean, it's it's really a Ponzi scheme.
It's a hot potato hot Ponzi scheme between investors right now and needs to get
back to profitability because because it's just going to lead to government bailouts and anyway,
I'll go further down that route. But other thing really cool keychain,
keychain pin tool that I got that just turns out to be useful all the
time. I don't know if I pull this out real quick, not show
my keys on video where you can easially copy them. But I got a
little keychain tool. It's got little silicon wrapper or the pentle like sim cards,
reset or router, you know, hit the reset button on the game
controller. You name it. There's all sorts of things that have that little
reset button on them. And now I just have this on my keychain.
It's a little and and because it's in a silicon sleeve, it's not poking
into my you know, like ripping my jeans or anything I need one,
or poking in my skin. So yeah, I mean they're they're overpriced only
in the sense that in order to ship it, like the cost of shipping
is more than the cost of the product. But you can get a pack
of three or five of them on Amazon with the silicon sleeves, so I'll
put a link to that. There also open audible. I don't remember when
we were talking about audible last time, but thank Heavens, that's not a
trademarkable word for whatever reason I don't I don't quite know, maybe because it's
a dictionary word, and I don't think you can trademark dictionary words. But
anyway, open audible is you can you can download your audible books and then
you can convert them into a format so that you actually own them. So
rather than being a rental where when the license with that author ends you lose
access to your books, you can give your grandchildren your audio books. And
so I often buy the MP three CDs when they're available, and at a
at a you know, good price, within the same range of you know
what i'd pay on audible, the fifteen dollars, twenty dollars, ten dollars
something like that. But I really I would not be using Audible, and
I would not be listening to as many audiobooks as I do if it weren't
for open audible giving me peace of mind that this stuff isn't just going to
be ripped out from underneath me one day, that I do have it preserved
in a permanent format. I check that. Lastly, Yeah, yeah,
it's it's it's twenty bucks or so, but totally worth it to be able
to have that peace of mind that you know, the stuff you bought you
actually own for you know, for real. And then the last thing was,
because we were talking about the plane stuff, there's this YouTube channel mentor
Pilot, and he he goes over every accident report of every plane incident and
all of history across the globe, and I just love listening to it.
It's one of those things that you can also listen to while you're falling asleep,
you know, kind of half pay attention if if you want to just
need need to hear something. But it's so it's so interesting to learn about
the different plane accidents and the different things that go into play, the different
parts of the report, what people on the ground and the control center reporting,
what was going on in the black box, you know, what other
pilots were seeing on the radar. Whatever. You know, he takes It's
not like that for every single incident, but you know, for all the
incidents that have the external data points, he links it together. He talks
about manufacturing process. He talks about you know that the different grades of lubricant
that are used on a particular shaft and why it you know it failed because
it wasn't they ship that they switch from this brand to this brand, and
now it should have been on a six month rotation, but they kept it
on the eighteenth month rotation. You know, any of that stuff. It's
not too technical, I don't think. I mean like it's enjoy I'm not
a pilot, and it's very enjoyable to me. It is technical, but
it's not like you don't have to be a pilot to appreciate it. But
he does go into detail on things and I really like that. So anyway,
those are those are my picks for the day. Steve Harmley, I
left you a little time for dad jokes. Hopefully I'll take the time whether
you left it or not. That's the most important part of the episode.
No. Along the line of pilots, I'm working on getting my drone pilot
certification and for the FAA. It's called Part one O seven And I've been
taken a class that was recommended to me by somebody else who's a fixed wing
pilot as well. And remember our drone program for our fire district, and
it's called Pilot Institute and really really quite good. The guy's name, I
forget the guy's name is Greg something. He's French that he lives in Prescott,
Arizona and does a lot of teaching and training. But everything I've been
seeing from those who have taken his course and then taken the Part one oh
seven FA tests, he said, you just need to know the course.
Everything you need is in there, and it's been really good. A lot
of a lot of information to cover, for sure. But pilot Institute dot
com is the u r L really quite good. All right. So now
for the the jokes of the day, I got to give Kudo's shout out
to the best stand up communitian of all time, without dispute, and his
name is Stephen Wright, deadpan comedian, was actually actolutely hysterical, and recently
he was on with Stephen Colbert and you know leveluce a couple jokes there that
were that were fantastics. So he thought this one that he told, this
one that had me laughing out loud, and he have to imagine him telling
it in his deadpan style. I'll do my best, he says. My
friend Jimmy brought an electric car and he was very excited about it. Then
he bought an electric blanket, Then he bought an electric guitar, and then
he bought an electric chair. And I haven't heard from him in a while.
Oh that loop got turned on again. Sorry, So right, So
one time when I was in school, I went to my teacher and I
said, would you find a me for something I did not do? She
said no, of course not. I said, okay, I did not
do my home. And then a simple question, what do skunks say at
church? Let us spray? All right, I got to give you prophecy.
This are some solid ones there, thank you. I have high standards
when it comes to the dad jokes I tell, so I'd do my best.
I don't have a good dad joke to share, but I do have
a couple of good carve outs, so AJ shout out for the audiobooks.
Huge fan also recommend, in addition to trying to get your open audible licenses
set up, public libraries have a fantastic audiobook selections. I'm in Seattle area,
Seattle Public Libraries has an amazing connection. Very much recommend it. Most
local libraries, whether it's a county library, city libook library, will be
part of a larger collective that has a ton of audiooks available to you.
So if you want to get an audiobook through your public libraries, fantastic way
to go about it. I'll also show out to Ali Abdell, one of
my favorite YouTubers who I'm currently reading his new book which is called Feel Good
Productivity, How to Do More of what matters to You, Very much enjoying
it, talking about a lot of the concepts that we've kind of brought up
more broadly around de experience and those other pieces of Hey, how can I
be happier in what I'm doing and get to do more things that make me
happy and really enjoying the book to kind of start off my year. And
then I'll close by just saying hey, if you haven't checked out the Devinrupted
Substack or our podcast, would love to hear feedback from everyone on this.
Always great to come on these other shows and really learn from Aj and Steve
here. It has been so much fun. I really appreciate you guys having
me on, and you can check us out at Devenrupted dot substack dot com
and maybe Ajarstevil join us for a guest article this year. I don't know
you talk you guys into it. We'd love to feature something from you guys
if it's interesting later in the year. Sure, I'm up for that.
Like I said, I'm not exactly what you call an engineering leader, but
I could do my best. Well. I'm stoked to continue to collaborate with
you guys, and thanks again for having me on. It's been a ton
of fun. Yeah, thanks for coming. I'm I'm glad that we had
you and we got to talk about these topics and and get a preview of
that report from Linear Body. Yeah, super exciting stuff. It'll be out
here probably by the time this pod goes fully live. On all channels and
lots of cool insights coming. So yeah, linear b dot io to check
out the report, and then Devin op dot substack dot com to check out
our weekly guest articles and the podcast. So yeah, thanks guys, been
a wonderful conversation, really really enjoy all right, well, have a good
one. Audios ideos ideas,