Speaker 1: Hey, folks, welcome back to another episode of JavaScript Jabber.
Speaker 1: This week, I'm your host, Charles max Wood, and I'm
Speaker 1: here with Itamar Friedman.
Speaker 2: Now it tomorrow.
Speaker 1: Do you want to let people know who you are
Speaker 1: and what you do? I see your shirt it says quoto,
Speaker 1: So you want to talk about what they do? And yeah,
Speaker 1: and then we dive in and talk about whether or
Speaker 1: not to trust your age AI generated code or code review.
Speaker 3: Really happy to talk about that topic and being here,
Speaker 3: Charles like, really a pleasure. Developing community are really awesome,
Speaker 3: especially as they're more.
Speaker 4: Specific you get into the details. COODO stands you mentioned.
Speaker 3: The name CODO stands for Quality off Development, and the
Speaker 3: most most focused of our platform is around AI code review.
Speaker 3: In general, we deal with quality, different AI quality workflows,
Speaker 3: et cetera. Basically helping enterprise professional a depth teams to
Speaker 3: standardize their quality via the review process or.
Speaker 4: Shift left, cold review and testing.
Speaker 3: It's okay, yeah, serving for example thousand around the world.
Speaker 3: But I think, like really really exciting, for example, just
Speaker 3: to talk is cold review important and all right now
Speaker 3: and in the future.
Speaker 4: Forget about CODO. That's I think like a cool topic.
Speaker 1: Yeah, absolutely, Well you know, you kind of hit on
Speaker 1: something that I think a lot of people are talking
Speaker 1: about with the AI stuff, you know, whether it's a
Speaker 1: code review or AI generated code. Maybe it's AI generated
Speaker 1: code that's code reviewed by AI.
Speaker 2: I mean, I don't know.
Speaker 1: It seems like there's a lot of concern as well
Speaker 1: as far as Okay, well, if I've got you know,
Speaker 1: an AI n LLM generating code, am I even going
Speaker 1: to have a job or you know, maybe they just
Speaker 1: downsize most of my team and so I have to
Speaker 1: be the most elite of the awesome elite at my company.
Speaker 1: And so yeah, there is a lot of concern there.
Speaker 1: And then the other piece of it is is, okay,
Speaker 1: I've got this powerful tool, Am I using it right?
Speaker 1: And so let's talk about the code reviews first, since
Speaker 1: that's where you're kind of living these days. And I
Speaker 1: don't think we've really gotten into the AI code reviews,
Speaker 1: to be perfectly honest. I saw that GitHub does some
Speaker 1: like you can turn this on for some of your
Speaker 1: repos and I haven't even tried it because it's just
Speaker 1: like I don't know, I mean, at work, you have
Speaker 1: to have a human code review your stuff anyway.
Speaker 2: And then on my personal stuff.
Speaker 1: It's like, like, I guess I did turn it on
Speaker 1: on one project and I was not that impressed. So
Speaker 1: so tell me where this fits in and maybe where
Speaker 1: I'm not using it to its full potential.
Speaker 3: Yeah, these are good pointers. So you touch an interesting point.
Speaker 3: They're related and different. So it's suckles almost theim like
Speaker 3: first about losing your job, Like you get to gut
Speaker 3: that point really quickly. I think like, first of all,
Speaker 3: the next few years. I'm not talking about fifteen years.
Speaker 3: That's it's hard to predict, especially the future. So let's
Speaker 3: focus on five years and we can we can go
Speaker 3: to further and out. I do have like my opinion,
Speaker 3: strong opinion about about the five next year. Forget about it.
Speaker 4: Sorry.
Speaker 3: Dario froment Tropic and Sam Altman cleaning. I don't know
Speaker 3: half a year ago, one year ago to the twenty
Speaker 3: twenty five. You don't need more developers. Ninety percent of
Speaker 3: code generated by developers. Yea sweet bench if you know
Speaker 3: this benchmark software engineering benchmark going to be ninety nine
Speaker 3: by nine of the year.
Speaker 4: We're far from all of that.
Speaker 3: And now I claim that while they decrease their predictions
Speaker 3: for twenty twenty six lower than twenty twenty five. I
Speaker 3: think even there they're wrong. So but that's not doesn't
Speaker 3: mean that the future is not going to be strong
Speaker 3: right now. It all already is, and it's going to
Speaker 3: be more and more AI empowered. And yes, at some
Speaker 3: point there is going to be a flection point where
Speaker 3: we're actually going to see end to end automation of
Speaker 3: software certain aspect of software development. But we can talk
Speaker 3: about it via a core review specifically about you know, Copata,
Speaker 3: et cetera. We are actually good friend, Like we have
Speaker 3: multiple clients that we are partnering together. They focus more
Speaker 3: on code generation and agentic workflow out of or pushing
Speaker 3: like get up issue into a PR much quicker. We're
Speaker 3: more focused on how do you note that that you
Speaker 3: can trust that line code? And I'll elaborate about that now.
Speaker 3: Sometimes there are some you know, feature like think about
Speaker 3: the cloud, like cloud have cloud observability tools, but still
Speaker 3: you have the data dog off the world. And that's
Speaker 3: a difference between like a few features that the cloud
Speaker 3: have around observability to a full fledged platform that gives
Speaker 3: you the confidence. So I won't go deeper on that
Speaker 3: competition because it's actually quite a symbiotic like, the more
Speaker 3: people use copilot, the more you need CODO in order
Speaker 3: to trust the code, So it's actually good.
Speaker 4: But code review specifically, I think like some might.
Speaker 3: Think about a synonym for where you deal with software quality,
Speaker 3: but actually difference like it is part of it. But
Speaker 3: let's think about quote review the purpose. It's meaningful purposes
Speaker 3: first like owning the code and learning two different very
Speaker 3: close relate buckets, like as a team. First the developer
Speaker 3: mostly single handed, like developing a feature or a sub feature.
Speaker 3: And then there's that moment where the team take responsibility
Speaker 3: if that person is going to be in vacation, who's
Speaker 3: going to do root code analysis if something happens right,
Speaker 3: So it's a moment where you you like, learn to
Speaker 3: each other code, learn the software, and own it together,
Speaker 3: even if an AI generated it completely one hundred percent
Speaker 3: from one prompt to a PR which is rare. But
Speaker 3: let's assume we're going to maybe talk about aspector and development,
Speaker 3: et cetera. But let's say you did you respect. Yeah,
Speaker 3: let's say you did write respect completely like perfect man,
Speaker 3: and you've got a feature. Even then a human needs
Speaker 3: to take over at least for the next few years
Speaker 3: and take responsibility. So that's one one thing about Court
Speaker 3: of View, and the second thing is that like until now,
Speaker 3: and it's going to be in the next few years,
Speaker 3: even if you do the best spec and the best
Speaker 3: like work, you're still going to have AI that needs sorry,
Speaker 3: code that needs to be reviewed. And there's a different
Speaker 3: between a tool that is meant to help you review
Speaker 3: and startulize quality to the one that is trying to
Speaker 3: help you generate code. Same off, how there's a difference
Speaker 3: between code of reserveability and thought. So to till the
Speaker 3: r hey, don't worry, in my opinion about your job.
Speaker 3: I would even recommend people as different than all other I.
Speaker 4: Think nine and a half the industry.
Speaker 3: Do go learn like you know computer science, but do
Speaker 3: like work with AI as one of your main tools
Speaker 3: throughout the STLC and code generation, Code of view will
Speaker 3: cot analysis, generating your respect everything and you will just
Speaker 3: alleviate together with with the profession, and that I think
Speaker 3: my point about that.
Speaker 1: Yeah, I agree with a lot of what you said there,
Speaker 1: and I'm going to back up through some of it,
Speaker 1: and then I've got a couple of questions as we
Speaker 1: go through. But one is you know, because yeah, you
Speaker 1: kind of wove in the am I going to lose
Speaker 1: my job?
Speaker 2: Or where where do I go with my job? Along
Speaker 2: with hey, where.
Speaker 1: Does the code you stuff fit in? And and that's
Speaker 1: where I have more questions. But I just want to
Speaker 1: reiterate a couple of things. One is is that we're
Speaker 1: not getting away from people having to be involved in
Speaker 1: the process. Is what I heard you say. And so
Speaker 1: to a certain degree, yeah, you may have tools that
Speaker 1: take on certain part aspects of the job or you know,
Speaker 1: do some of the things that are involved in you know,
Speaker 1: understanding and figuring out what's going on. But at the
Speaker 1: end of the day, yeah, you need a human that's
Speaker 1: you know, that's going to take responsibility for this stuff
Speaker 1: and and shepherd it through. And if you're looking to
Speaker 1: enhance your career, you need you need to understand and
Speaker 1: be able to use these tools because they do make
Speaker 1: people more productive. And so the adoption of this stuff
Speaker 1: is inevitable basically. And so those are the triitions and
Speaker 1: the role right and the role of soffer interrupting of
Speaker 1: software developer right now in five year is completely different.
Speaker 1: I love the word that you shepherd, like like basically
Speaker 1: you're going to deal a lot with writing spects, writing
Speaker 1: your rules, writing your best practices, following down like navigating
Speaker 1: your army of agents, specializing in how to deal with
Speaker 1: specific problems that might.
Speaker 3: Come around with that. How everything I said is going
Speaker 3: to be evolved, et cetera. So it's going to be
Speaker 3: completely different, like higher level, more architecture, guardrails stuff. And
Speaker 3: it's a process that happened throughout the last twenty thirty years.
Speaker 4: Right.
Speaker 3: We used to punch cards and write notes and assembly
Speaker 3: and etcetera. So it evolved. So it's going to be different.
Speaker 3: But there's a lot of things for a human to
Speaker 3: take responsibility and ownership.
Speaker 1: Yeah, and wherever we end up, right, because I think
Speaker 1: it's optimistic of anybody to say, well, this is really
Speaker 1: where I think we're going to be even in two
Speaker 1: or three years, because things are moving so quickly, right,
Speaker 1: And you pointed out, you know, Sam Altman and some
Speaker 1: of these other folks, they thought they knew where this
Speaker 1: was going, and they just you know, it's impossible to
Speaker 1: really predict it. But if you're on top of what's
Speaker 1: going on today, then it's a whole lot easier to
Speaker 1: adapt to whatever comes later. And then as far as
Speaker 1: the like the AI run code reviews and things like that.
Speaker 1: So you talked a little bit about somebody's going to
Speaker 1: have to understand and maintain and take ownership of the code,
Speaker 1: and a lot of that knowledge has passed through a
Speaker 1: code review, right, So are you are you advocating that
Speaker 1: you have a human reviewer and an AI reviewer or so? So, yeah,
Speaker 1: how does this fit into the life cycle of my
Speaker 1: features and things like that.
Speaker 3: Yeah, we'll talk about a bit about nostalogy just before,
Speaker 3: Like you remember those days, like I think me and
Speaker 3: you are like sorry for iman selling old model enough
Speaker 3: to remember today's where we actually use books.
Speaker 4: Do you remember before.
Speaker 3: We went to Google or whatever was other websites, we
Speaker 3: were actually using books to learn about things, et cetera.
Speaker 4: And I still use books.
Speaker 1: That's kind of my maybe I'm going to sound old,
Speaker 1: but and mostly yeah, if it's technology, I do spend
Speaker 1: a lot of time on the internet kind of picking
Speaker 1: up the new things, right because the books get out
Speaker 1: of date kind of fast, depending on how quickly technology moves.
Speaker 1: But yeah, I prefer picking up the books and just
Speaker 1: getting kind of the classic ideas that don't change it.
Speaker 2: Everything runs on.
Speaker 4: So so we are nostalgic about it.
Speaker 3: But if I have to force you, like choose one,
Speaker 3: I think I know what you're going to choose. And
Speaker 3: think about how much we delegate to Google or other
Speaker 3: technologies on the Internet and interanet to that we're not
Speaker 3: going to validate right now to the deepest like part
Speaker 3: of it. So I think it's right now already like
Speaker 3: a very human digital we call it I or not
Speaker 3: like technology based like development already as it is today.
Speaker 3: And I think like the same thing is going to
Speaker 3: happen with court review, where actually if you have a
Speaker 3: one hundred and we are seeing customers that have more
Speaker 3: than that, like a set of guardrails, rules, standards, and
Speaker 3: I can give you an example like having a human
Speaker 3: reviewer doing that like all at once in the time
Speaker 3: the limited time is actually problematic and if you can
Speaker 3: free the time to focus on what you know where
Speaker 3: could be the most useful for that courde review, it's
Speaker 3: actually going to have better learning, especially if the tool
Speaker 3: AI tools that are using cares about learning. So even
Speaker 3: if it caught something, it's just like, oh I fix it.
Speaker 3: It's rather hey, this is the reasoning, and that's what
Speaker 3: we learned from it, and by the way, this is
Speaker 3: how we changed our second brain like that. And so
Speaker 3: let me give you an example, like there could be
Speaker 3: that someone in the company really cares about you know,
Speaker 3: eat your flags, and they didn't manage to tell everyone no.
Speaker 3: And the five hundred developers seven thousand developer organizations, that's
Speaker 3: what you care about. You can configure that in the
Speaker 3: system and I system. And now if you developed a
Speaker 3: new feature, you get an alert, Hey you didn't put
Speaker 3: a feature flag. Oh show you shouldn't you know? It's
Speaker 3: this is also learning, like it's like, hey, oh, the
Speaker 3: company cares about feature flag. I have a one hundred
Speaker 3: example like this, but just one, you know, very I
Speaker 3: think famous, funny or scary or not funny. Min I'm
Speaker 3: a developer. This is like I'm a developers. I wake
Speaker 3: up in the morning, I have a slot forty minutes
Speaker 3: to review a PR, a pull request. If you give
Speaker 3: me five lines of code in that PR, I'll give
Speaker 3: you fifty comments. If you give me fifty lines, I'll
Speaker 3: give you five comments. If it's five hundred lines, looks
Speaker 3: good to me? Right, Like, in forty minutes, how can
Speaker 3: I five on lines of Code's that's a reality. And
Speaker 3: by the way AI could even help us with classifying
Speaker 3: like which review which pr?
Speaker 4: You can actually do it for forty minutes.
Speaker 3: That's how Also doctor works like they get an AI
Speaker 3: tool that helps them prioritize, not necessarily tell them if
Speaker 3: that person is whatever diagnostic, but just serve facing. Right,
Speaker 3: So the opportunities are awesome. Actually, the problem is real
Speaker 3: already existing around learning and catching issues. Tell me if
Speaker 3: you didn't have a major issue, which developer did not
Speaker 3: have a major issue?
Speaker 4: And production?
Speaker 3: I can tell a few horror stories myself, and so
Speaker 3: the opportunity is not reducing the learning or increasing bugs,
Speaker 3: is actually trying to to you know, get this better.
Speaker 3: But we do need to write you xui and the
Speaker 3: right mindset of the team. That's the building the dev tool.
Speaker 4: So obviously like.
Speaker 3: Sorry, shameless plug bug code, but it's not only us, okay,
Speaker 3: like or not going to be the only one who
Speaker 3: cares about it.
Speaker 2: Yeah, I think I think the how do I put it?
Speaker 4: So?
Speaker 1: Programmatic checking of your code has been around for a
Speaker 1: long time, right, So you have the static analysis tools,
Speaker 1: You've got the linters and stuff like that. The electronic
Speaker 1: with those is that typically at least the ones that
Speaker 1: I've seen they break your code down with the abstract
Speaker 1: syntax tree and then they look for patterns.
Speaker 2: Is basically what they do.
Speaker 1: And the AI systems they kind of do that. But yeah,
Speaker 1: you can be a lot more I guess, a lot
Speaker 1: less prescriptive as far as what the patterns are, right,
Speaker 1: and so you can train it on how to look
Speaker 1: at code and what to look for, but you can
Speaker 1: be a lot more broad and it'll figure out how
Speaker 1: to do some of the stuff that you have to
Speaker 1: explicitly tell the other the static analysis tools and the
Speaker 1: linters how to do right. And so you can say,
Speaker 1: like your feature flag example, right, you know it can
Speaker 1: you know, I mean, depending on the training of the
Speaker 1: system and you know how good the data set is
Speaker 1: and things like that, and how good your prompt is.
Speaker 4: But you can teach it.
Speaker 1: Essentially, this is how you identify a new feature and right,
Speaker 1: and then you have to make sure that there's a
Speaker 1: feature flag around it. And it can figure out the
Speaker 1: other steps, like it can infer what to do and
Speaker 1: so on a lot of those things where you're saying
Speaker 1: you set up a rule. Effectively, what you're saying is
Speaker 1: is my code has to conform to these ideas, and
Speaker 1: then it can with its latent space, you know, the
Speaker 1: stuff that it's been trained on, anything else that you
Speaker 1: add to the mix, with its context and things like that,
Speaker 1: it can then go in and intelligently figure a lot
Speaker 1: of that stuff out where trying to figure out how
Speaker 1: to explicitly program it to say, here's how you find
Speaker 1: something like this in the abstract syntax tree, or here's
Speaker 1: how you break down this idea. Especially on you mentioned
Speaker 1: like the five hundred line pr right, it's like across
Speaker 1: all of these files and all of these changes. That's tricky.
Speaker 1: But the LLM can consume it and figure it out
Speaker 1: much much easier than you can figure out how to
Speaker 1: explicitly program it for every case you're going to run into.
Speaker 1: And it's not going to be perfect, but it can
Speaker 1: do a lot better job, and it can do it
Speaker 1: a lot more quickly and a lot more thoroughly.
Speaker 2: Than I can.
Speaker 1: Yeah, just kind of browsing through it and going, you know,
Speaker 1: hoping my pattern matching brain goes, yeah, that's a problem,
Speaker 1: and that's also a problem. And hey, there's so much
Speaker 1: context in here that I'm just trying to understand that.
Speaker 1: You know, I'm going to pick out all these little things.
Speaker 1: But then when I go in as an experienced programmer
Speaker 1: to do the code review, you know, I can see
Speaker 1: what it caught and I can say, yeah, ninety nine
Speaker 1: percent of this is great. You don't have to worry
Speaker 1: about these couple of things that you know, it may
Speaker 1: not be quite right on that or it's close, but
Speaker 1: you should do this instead of that. And then the
Speaker 1: other thing that I can see with it is that
Speaker 1: there may be things that are just kind of aesthetic
Speaker 1: or organizational or other things that we do that we
Speaker 1: haven't codified into the rules where it's like, no, we
Speaker 1: do things this way, not that way. And then you
Speaker 1: can also go back and you can retrain your you know,
Speaker 1: you can rewrite your prompt to include those on the
Speaker 1: next one. But yeah, I can see where the LM's
Speaker 1: looking at the code could do a much broader and
Speaker 1: more nuanced analysis than you can get out of some
Speaker 1: of these other tools and be more thorough than a
Speaker 1: programmer doing it.
Speaker 3: I'd love to relate to the you know, traditional quote
Speaker 3: unquote old world satical analysis versus the semantic AI. But
Speaker 3: just just before that, about philosophy around like, how would
Speaker 3: you how would you actually exploit l lams like AI
Speaker 3: to work with rules or standard standards that are written down.
Speaker 3: So it's for those who are watching the video I'm
Speaker 3: actually having I have here like a black shirt that
Speaker 3: it's really where you really I wear purple. And I'll
Speaker 3: explain why, because I think the world's roughly sticking divide
Speaker 3: into too and I'm relating to how you use rules
Speaker 3: in the standard and mark down, et cetera. I'm going
Speaker 3: to explain it's connected, I swear. So there's the blue
Speaker 3: team and the Red team. Like principle at CODO, we
Speaker 3: see yourself like more in the red with mixed. That's
Speaker 3: one of the reasons we chose the purple. So I'll
Speaker 3: explain the blue team, which we had like the Winter
Speaker 3: of the Cursor of the World, et cetera. Right QUOD
Speaker 3: code compile everyone Basically they take those rules you can
Speaker 3: write them in different and they mostly put it as
Speaker 3: part of their context, which in many cases is actually
Speaker 3: listening to.
Speaker 4: But there's two problems. One I said many cases.
Speaker 3: The second is that maybe it took it as part
Speaker 3: of the context. But as you usually it doesn't work.
Speaker 3: You don't finish a feature like at least not a
Speaker 3: meaningful feature from promptu code, even if it's an agent
Speaker 3: that runs for ten minutes usually or one hour, and
Speaker 3: how sometimes you usually like get to prompt it over like.
Speaker 4: Like navigate it.
Speaker 3: Then at your tenth you know, prompt it actually very
Speaker 3: very probably like missed something from your spec or or
Speaker 3: your rules or your even your own prompt. And the
Speaker 3: second one that's a blue way of thinking, would they
Speaker 3: prioritized like their KPI is like u x ui. We
Speaker 3: admire on them on that and speed from prompt to code,
Speaker 3: et cetera. The red team is how we take every
Speaker 3: fact that is there as an intent they're functional or
Speaker 3: non functional, and verify it. That's a totally different process.
Speaker 3: So you take the list of your one hundred rules,
Speaker 3: whether it's ten or or one thousand by the way,
Speaker 3: and you and you check them with and write lam
Speaker 3: to turn to really increase. I can't say one hundred percent,
Speaker 3: but really ninety nine whatever and free is a chance
Speaker 3: that they're actually actually checked. I just want to you
Speaker 3: know that it's important to differentiate between different philosophies and
Speaker 3: it actually leads to a different uxui, et cetera. And
Speaker 3: that relates to the first topic of ASTs. I think
Speaker 3: that they're actually very powerful tools like sonar for example.
Speaker 3: I would definitely I have like high expectation of that company.
Speaker 1: And but yeah, they do a great job within the
Speaker 1: set that they're you know, they're capable of doing.
Speaker 3: They do two and I think like eventually, if you
Speaker 3: want to exploit AI, really explore, move fast with confidence,
Speaker 3: you want that mix of you know LLLM AI empowered
Speaker 3: code review, code quality with that static and it would
Speaker 3: be the best if you can actually mix them together
Speaker 3: and you would see integrations like we have integrations with
Speaker 3: with those tools because they might catch in many cases
Speaker 3: the same thing and you don't want that annoying like
Speaker 3: double double reviewing double thing. So but but the basic
Speaker 3: of the technology, you want this technology to to to
Speaker 3: work together because otherwise you don't have a confidence to
Speaker 3: really like, okay, there's one to clock code. In my
Speaker 3: experience like you often I wouldn't say daily but weekly,
Speaker 3: Like with one prompt, I can can get one thousand
Speaker 3: line of code in five minutes being changed. Go review
Speaker 3: that right and right and and now and now, like
Speaker 3: you want the maximum help of AI also or sorry,
Speaker 3: any technology to help you navigate to getting confidence about
Speaker 3: that code quality, correctness, maintainability, et cetera. So I have
Speaker 3: like a high expectation of these twols working working together,
Speaker 3: being that hunter part the red versus the blue. So
Speaker 3: you can get a purple as a dev organization.
Speaker 1: So I mean going into this, right, and I kind
Speaker 1: of I mean, we may change the title when we publish,
Speaker 1: but I kind of tongue incheek put the title is
Speaker 1: can I trust AI generated code? And so now you're
Speaker 1: talking about this other end of things right where it's
Speaker 1: like you've got the copilots or cloud codes or Google
Speaker 1: Gemini or chat GPT.
Speaker 2: Or whatever, right or whatever?
Speaker 1: Right, Yeah, and so you know they're using these models
Speaker 1: to generate code, right and then yeah Kiro we did
Speaker 1: an episode with Eric Hanchett where you know, in Kiro
Speaker 1: it helps you generate the entire spec and then basically
Speaker 1: it kind of iterates through step by step and does it.
Speaker 1: It was funny because we did that episode. I'm sitting
Speaker 1: there going, well, I've been using Cursor and vs Code
Speaker 1: with Copilot and neither of them do it, and literally
Speaker 1: a week later is showed up in Cursor. And so
Speaker 1: so I've used that feature a handful of times and
Speaker 1: it's really nice. The other thing that I'm going to
Speaker 1: point out because you're like, yeah, you know you have
Speaker 1: claud code generate you a thousand lines of code or whatever.
Speaker 1: I was having a conversation yesterday with Obi Fernandez. He's
Speaker 1: a Ruby developer, but I've seen the same thing with
Speaker 1: from JavaScript friends, where you know, they use claud code
Speaker 1: and they have it generate a bunch of the code.
Speaker 2: Right, and then they kind of review it on their own.
Speaker 1: And the thing that's he basically said because he used
Speaker 1: to run a consultancy, one of the bigger consultancies out
Speaker 1: there in the Ruby space, and he was saying, I
Speaker 1: built this app in a month. That it would have
Speaker 1: taken us six months and probably five hundred thousand dollars
Speaker 1: of developer salary to build this, and I've done it
Speaker 1: by myself in a month with AI help. And so
Speaker 1: I'm looking at it and go, well, so to a
Speaker 1: certain level, I guess the answer to can I trust
Speaker 1: AI generated code is yeah, I mean to it because
Speaker 1: it works, right, But yeah, so if you have the
Speaker 1: AI generating the code, I mean, where do you run
Speaker 1: into problems there? And then you know, especially with what
Speaker 1: you're seeing, where you have this system that then reviews
Speaker 1: the code. So if you have the LLM generate the
Speaker 1: code and the LLM review the code, right, is that
Speaker 1: kind of a circular dependency that does or doesn't work?
Speaker 1: I mean, this is where I'm starting to get into. Okay,
Speaker 1: you know, I've got these tools that are supposed to
Speaker 1: empower me, but do they actually work nicely together to
Speaker 1: give me that red team blue team work out?
Speaker 4: Yeah?
Speaker 1: And I guess the other concern is is you know,
Speaker 1: am I actually moving faster or am I moving faster
Speaker 1: with a gun into my foot.
Speaker 2: The whole time?
Speaker 4: Yeah, we heard a lot last point.
Speaker 3: I think, like short, what I'm mostly want to relate
Speaker 3: about the main question off the topic of today, like
Speaker 3: should you trust AI generated code? I do have an answer,
Speaker 3: but I just say, like I do think the future
Speaker 3: is already happening now, like pol people of a client
Speaker 3: connecting CODO with cursor, CODO with COPAAD or or whatever,
Speaker 3: and that's more.
Speaker 4: Yeah. I just want to like it, Okay, Yeah, I
Speaker 4: just don't want to like to spend too much time
Speaker 4: with that.
Speaker 3: I guess it's a lot of commercial today's, so it's
Speaker 3: possible it's going through that direction.
Speaker 4: More and more.
Speaker 3: I do want to answer the question, but with a
Speaker 3: metaphor before if I ask, do you you do you
Speaker 3: trust human code? Wait, way, don't answer, let me give
Speaker 3: you give you a second, but just think about a
Speaker 3: for a second. Do you trust human the like code?
Speaker 3: Think like, let's let's well think about a little bit. Sorry, no,
Speaker 3: because because you're you know, it's a it's a good point.
Speaker 1: The flip side, though, is that because I don't know
Speaker 1: that it's necessar do you trust human generated code? A
Speaker 1: lot of it depends on the human right, and you know,
Speaker 1: it's like, Okay, how much experience do they have? Have
Speaker 1: they done this before?
Speaker 4: You know?
Speaker 1: Are they taking security and scaling and all the other
Speaker 1: things into account?
Speaker 2: Right?
Speaker 1: But the other thing is is that I don't think
Speaker 1: that that's the baseline that we have to look at
Speaker 1: AI from because it's not whether or not I trust
Speaker 1: human code and whether or not I trust AI code,
Speaker 1: because at this point, the human generated code is the baseline.
Speaker 2: We've been doing that for years and years and years
Speaker 2: and years.
Speaker 1: Is the AI code better or the other way you
Speaker 1: look at it from a business standpoint, is is it
Speaker 1: close enough? Given what I had to do to generate it,
Speaker 1: which is usually a prompt and not nearly as much time. Right,
Speaker 1: so maybe it's not as good as human code in
Speaker 1: a number of ways, but it's good enough and it's.
Speaker 2: A lot cheaper, and so it may be worth it anyway.
Speaker 4: I'll use you on it.
Speaker 3: But still still bear of me with the metaphor, like
Speaker 3: if I ask, and I'll borrow something that you said
Speaker 3: to you to make my point. If you ask, like
Speaker 3: do you trust human the developed generated code, written code
Speaker 3: or coded code, then then I think that the minute
Speaker 3: answer is like, yes, that's what we do. But then
Speaker 3: you think about it a bit more like you just did,
Speaker 3: and you're thinking, if I trust that person, even if
Speaker 3: it's a senior developer, to write the code and noteped
Speaker 3: plus plus on the airplane and then push it to production.
Speaker 3: I'm not saying I never did that, but but but
Speaker 3: but push a production. I think your answer is no,
Speaker 3: And then you're what you are saying, like a senior
Speaker 3: developer that is going through the processes that is required,
Speaker 3: uh checking also for security reviewing for the standards and
Speaker 3: all that and going through the review process and checking
Speaker 3: the CI results and et cetera. You do trust if
Speaker 3: you want generated code. That's my answer, Like you should
Speaker 3: you trust AI generated code? Spit it out, you write
Speaker 3: a you wrote it, prompt you via code, trust it. No,
Speaker 3: I suggest you don't also do that. For the most
Speaker 3: senior developer, you have a process, the process could be quick,
Speaker 3: and I think that's what.
Speaker 4: We're going to see.
Speaker 3: Also for the AI generated code, you're going to see
Speaker 3: AI being used to automate more and more and more parts,
Speaker 3: and then AI generated code is going to look like
Speaker 3: more more a generated code like AI generated software development,
Speaker 3: AI software development, where AI is going to do a
Speaker 3: proper process. Humans, by the way, in certain percent, which
Speaker 3: in the beginning would be very big, very large, sorry,
Speaker 3: are going to be involved throughout the process, and over
Speaker 3: time the process is more like you know, like verifying
Speaker 3: that the pipeline works, you know, like like we do
Speaker 3: with the MANUF in a manufacturing like a lab. Right
Speaker 3: like it started with human labor and slowly like more
Speaker 3: and more and more automatic humans are still involved there,
Speaker 3: et cetera. It took like fifty years to do that.
Speaker 3: It will take also like fifteen twenty years. We've talked about.
Speaker 4: Predicting for totally being machines.
Speaker 3: Meanwhile, like I just invent in a number like fifteen years,
Speaker 3: like twenty forty one just to have to guess, like
Speaker 3: the four and one looks like AI, so I guess
Speaker 3: it's a stayer for me. But until then, like we're
Speaker 3: just going to see like more and more portioned being
Speaker 3: automated and maybe some portion like automated end to end,
Speaker 3: and at that point you will trust it because it's
Speaker 3: not just like spitting statistically code and even if it's
Speaker 3: trained really really well, but it's also going to validate itself,
Speaker 3: verify itself going through the process, et cetera.
Speaker 4: And that's a future. That's a future we believe that.
Speaker 3: Code of right, that's where we focus on code quality,
Speaker 3: codelorification and etc.
Speaker 2: Right.
Speaker 1: I think the point is well taken that because a
Speaker 1: lot of people they conflate where things are with where
Speaker 1: we're going to end up. And what you're saying is
Speaker 1: is that, yeah, we're going to get better and better
Speaker 1: tools to do more more things, and they're going to
Speaker 1: manage more and more pieces of the process, right, And
Speaker 1: and yeah, I think that that is absolutely true. It's
Speaker 1: funny because for several years, you know, I've been using
Speaker 1: rock or chat, GPT, or Claude. I've used all of
Speaker 1: them for different things.
Speaker 2: Right.
Speaker 1: It's like, hey, I'm trying to explore this thing, right,
Speaker 1: and so'll it'll give me all kinds of feedback on
Speaker 1: you know, health or whatever, right, and so I kind
Speaker 1: of use it as a coach or at least, you know,
Speaker 1: and then sometimes I'll go fact check it or verify
Speaker 1: this or that, or you know, refine whatever it gave me.
Speaker 1: But you know, a lot of times it shortcuts a
Speaker 1: whole bunch of research that I would have to do,
Speaker 1: and so then I can just justify the pieces where
Speaker 1: I'm like, it doesn't seem quite right, but it's it's
Speaker 1: gotten more and more correct the longer, you know, the
Speaker 1: longer we go, because the models get better, the data
Speaker 1: that's you know, in that latent space gets better.
Speaker 2: And so that's I definitely see with software.
Speaker 1: Yeah, But as far as conflating where we are with
Speaker 1: where you know, a few years ago, I think it
Speaker 1: was just last year actually, I was having a conversation
Speaker 1: with my father in law. Now, granted he's a general contractor,
Speaker 1: and by general contractor, I mean like he fixes crap
Speaker 1: in people's houses general contractor, and you know, and so
Speaker 1: he just heard about like the goof ups in the
Speaker 1: news where it was like, well, I heard somebody ask
Speaker 1: chat GPT this thing and it told him this bogus thing.
Speaker 1: And I'm like, yeah, dad, but we've moved like four
Speaker 1: models ahead since then and it doesn't do that anymore.
Speaker 1: And he's like, yeah, well, you just can't trust it
Speaker 1: for anything, and you know, and again I'm looking at
Speaker 1: him and going, well, actually, I use it all the
Speaker 1: time for this other stuff because you can trust it, right,
Speaker 1: But yeah, I go and fact check stuff. It's like,
Speaker 1: you know, I think you have a bias here, and
Speaker 1: so I'm going to go fact check these pieces because
Speaker 1: I think I think the data that you were trained
Speaker 1: on isn't one hundred percent in line with my worldview
Speaker 1: or the way I think things are. But at the
Speaker 1: same time, yeah, it's gotten way, way, way more accurate,
Speaker 1: especially when it gets into a lot of the you know,
Speaker 1: like easier for like meal planning and you know, hey,
Speaker 1: I got to modify this workout or this or that,
Speaker 1: and it you know, it's terrific because it's got all
Speaker 1: that data in it. And I think that's where we
Speaker 1: go with the software. Is My point is so even
Speaker 1: where we see it kind of fall short. Is it
Speaker 1: a much better and much more accurate, much more thorough
Speaker 1: than I can be as a code review reviewer. And
Speaker 1: it looks like the answer to that question is undoubtedly.
Speaker 4: Yes.
Speaker 1: It may give me some false positives and some false negatives,
Speaker 1: but it's going to be more thorough and much faster,
Speaker 1: and I can pick through that and it's still going
Speaker 1: to save me a whole bunch of time and effort,
Speaker 1: and it's only going to get better. Right, So where
Speaker 1: we end up in a few years it may be
Speaker 1: completely different, but it's almost certainly.
Speaker 2: Going to have better data to run on and make
Speaker 2: the process better.
Speaker 3: I think, like I truly believe there is meaningful improvement
Speaker 3: in the LLLM. Like some people over time claim that
Speaker 3: it's like diminishing the velocity of right because it consumes
Speaker 3: what it's been putting out. I've heard that for example,
Speaker 3: for that reason, et cetera, I think the are meaningful improvement.
Speaker 3: We have internal benchmarks around quality of code, et cetera.
Speaker 3: It's it's it's going up. And having said that, I
Speaker 3: think there's other reasons where like extracted value is bigger. First,
Speaker 3: we use it in more areas the l ll ms
Speaker 3: and more areas. But specifically I want to relate to
Speaker 3: what you said. I think it's also we're learning how
Speaker 3: to use it.
Speaker 4: So you know, like as a developer, you learn how
Speaker 4: to Google, or you you learned how.
Speaker 3: To use stack overflow, but now I guess, but you
Speaker 3: learn how to have a good Google. That's still very
Speaker 3: relevant and I think like we are learning how to
Speaker 3: you know, prompt or use it could be one of
Speaker 3: the differences in between you and the front or family,
Speaker 3: et cetera. And that's really actually to for example, what
Speaker 3: we talked about w as Hero and Spectra and development
Speaker 3: that like we we're learning that the better we the
Speaker 3: more information should be concise and accurate. But the more
Speaker 3: information we provide as part of the prompt if we're
Speaker 3: talking about self development, like maybe spec in most cases, the.
Speaker 4: Better job it will do.
Speaker 3: By the way, I am like once in a while,
Speaker 3: like giving a disclaimer or trying to be careful because
Speaker 3: because there's a lot of research down for example and
Speaker 3: Tropic are really good in it is that if you
Speaker 3: think that you can push like as much as context
Speaker 3: and instruction as you want and expected to really work well,
Speaker 3: they're actually seeing it diminishing returns even like worse if
Speaker 3: you give it like a spec of like a full book,
Speaker 3: even if the context is bigger. But putting that aside,
Speaker 3: it is a really good idea to.
Speaker 4: Learn how to use these tools.
Speaker 3: And I think we're actually consciously and consciously like like
Speaker 3: like doing doing that and and for software development specifically,
Speaker 3: and I think even JavaScript, where you know the language
Speaker 3: is maybe not that descriptive, et cetera. Like having a
Speaker 3: proper spec is a good idea. And although I have
Speaker 3: to say I'm not a big believer that spectrum of
Speaker 3: development is going to be the last thing that survived.
Speaker 3: It's going to be the biggest thing that actually make
Speaker 3: the difference. Is an important concept, but it's not going
Speaker 3: to be the what's solved everything.
Speaker 2: No, I think.
Speaker 1: And I'm just going to piggyback on what you're saying
Speaker 1: because I think you're correct in one way how we
Speaker 1: use the tool, right, and so you know, spec druven
Speaker 1: development is one way that we've you know, this is
Speaker 1: a new way to use.
Speaker 2: The LM and you know, maybe have a wider.
Speaker 1: Context on what it's doing and give it a step
Speaker 1: by step cohesive plan. But yeah, I don't think that's
Speaker 1: where we end up. I mean, we're going to invent
Speaker 1: other ways of using these tools, and this may be
Speaker 1: a stepping stone to something else. The other thing, though,
Speaker 1: is that it's not just for me. Hey, we're getting
Speaker 1: better at using these tools, but also as it takes
Speaker 1: things off of our plate, we're able to refine in
Speaker 1: other areas, and I think those get better because the
Speaker 1: next versions of the models pick up some of those
Speaker 1: changes to the way we do things outside of how
Speaker 1: we use the LLM and make it better that way too.
Speaker 1: And so at some point does it kind of you know,
Speaker 1: are we getting smaller increments of value?
Speaker 2: Maybe?
Speaker 1: But again I I just see the ingenuity of people
Speaker 1: as we go continue to just be really cool and awesome,
Speaker 1: and so for the time being, we just see these
Speaker 1: astronomical leaps every time we get a major version update
Speaker 1: on these llms.
Speaker 3: Yeah. By the way, I think once upon a time.
Speaker 3: My background is the machine learning since two thousand and six, oh, okay,
Speaker 3: annual networks in twenty ten, so I allow myself talk
Speaker 3: about a history. I think once upon a time until
Speaker 3: roughly speaking GPTs three three point five. Every time you
Speaker 3: train them all like really like nine nine percent of
Speaker 3: the cases. You try it to be better than the
Speaker 3: others in a specific niche, right, even if it's a
Speaker 3: big niche, still GPT three point five, I think we
Speaker 3: had like a year or two or more that we
Speaker 3: were under assumption level matters. You're not the market that wow, Like,
Speaker 3: look at this GPT three point five winning every benchmark,
Speaker 3: even human benchmark if you remember those graphs, amazing graph
Speaker 3: on opening eye exacts as like different professions like from
Speaker 3: lawyer stuff to history and the waxes like the percentile
Speaker 3: on their official tests, and like GPT three point five
Speaker 3: like cross every model on all of these like fifteen
Speaker 3: different professions, and then GBT four the same, et cetera.
Speaker 3: At this point, it's not the case anymore. I think
Speaker 3: since roughly speaking, so that's three point five if you're
Speaker 3: familiar from on Tropic, et cetera, then and I think
Speaker 3: that model was suddenly better. Some claim much better than
Speaker 3: GPTs on coding, but probably not at all.
Speaker 4: Quite a few cases.
Speaker 3: Now, like there was a moment where seeing people thought
Speaker 3: only open air on Tropic or whatever, Google are going
Speaker 3: to generate like foundation models, but I'm seeing like dedicated
Speaker 3: foundation models and in health medicine, customer success, et cetera.
Speaker 4: And and like we do do see like GPT.
Speaker 3: Five for example, and maybe there's new new versions the
Speaker 3: the are coming that are that are better on specific
Speaker 3: aspect of software development, like specific in software develop right,
Speaker 3: So so like there is evidence that that is it's incremental,
Speaker 3: and I think that's actually somewhat meaning that we're not maturing.
Speaker 3: And I'm not sure if that's what people thought. Aren't
Speaker 3: going to say, like we're maturing. Okay, let's get to
Speaker 3: let's get to those EDU cases. Let's get Okay, we
Speaker 3: probably need a specific LLM and a specific agent for
Speaker 3: specific quality measure that we want to track or help with,
Speaker 3: et cetera. So I think I think like it doesn't like,
Speaker 3: it doesn't mean that the whole solution doesn't keep evolving
Speaker 3: upwards right in the same or even bigger speed, you know,
Speaker 3: like the same.
Speaker 4: As uh uh.
Speaker 3: And book we talked about book like carts will that
Speaker 3: similarity is near. We're seeing like this is a notion
Speaker 3: that we're seeing like exponential growth and technology. But if
Speaker 3: you're zooming in, then you're seeing like skurve and and
Speaker 3: the thing is that each escort the time. The difference
Speaker 3: between that each escort just slower and smaller and smaller, right,
Speaker 3: and if you zoom out, it looks like exponential. So like, yeah,
Speaker 3: maybe that specific GPT sorry, specific LLM attention architecture and
Speaker 3: specific training, et cetera, is slightly like the low hanging
Speaker 3: fruits are over, but we are we will. We are
Speaker 3: seeing in the igentic world and and other like technologies
Speaker 3: more breakthroughs and I can mention more and and overall
Speaker 3: like the we're going to see AI like keep keep
Speaker 3: going upwards and so so I wouldn't like, like say,
Speaker 3: you know, like incremental LLM the solution we're getting, it's
Speaker 3: going to get better and better and we should adopt it.
Speaker 1: Yeah. All right, Well we're getting toward the end of
Speaker 1: our time. I hate to cut this short because I
Speaker 1: could sit here and talk about this forever. But yeah,
Speaker 1: so we're gonna just roll into our picks. I've got
Speaker 1: like five minutes before my work, Yeah, my work anyway,
Speaker 1: so I'm going to jump in. I have a work meeting,
Speaker 1: so I'm going to jump in and move to picks.
Speaker 2: Now.
Speaker 1: Picks are just shout outs about whatever it is that
Speaker 1: we've been up to and enjoying lately. So the first
Speaker 1: pick that I have is, so on Friday, I'm going
Speaker 1: to be teaching board games. I do this periodically. Hang on,
Speaker 1: So I yeah, I'm teaching board games at a board
Speaker 1: game conference, and I've picked most of the games we're teaching.
Speaker 1: The one game I haven't picked is well, there are
Speaker 1: two of them. One of them I'm learning tonight and
Speaker 1: the other one earned last week. And this one's called
Speaker 1: far Away. It has a board game weight on board
Speaker 1: game Geek of one point nine to one, which means
Speaker 1: that it's fairly approachable for the average board game player.
Speaker 1: And so what it is is you how do I explain?
Speaker 1: It's it's mostly cards. So you're playing cards in front
Speaker 1: of you, and you have like eight slots, and so
Speaker 1: you play your first slot and then your second slot,
Speaker 1: and then your third slot. But when you score it,
Speaker 1: you score it back the other way. And so the
Speaker 1: last card you put down is the first one you
Speaker 1: played to score, and it's it's available for scoring on
Speaker 1: all the other cards that you played, and then you
Speaker 1: flip over the next to last card that you played
Speaker 1: and you score it against the two cards that you
Speaker 1: have down.
Speaker 2: If you play cards. So let's say you play.
Speaker 1: The the twelve, and then you play the fifteen after
Speaker 1: the twelve, then you also get some other cards. I
Speaker 1: can't remember what they're called, but those ones count through
Speaker 1: the whole scoring process and anyway, so you just kind
Speaker 1: of build up this deck and then you score it
Speaker 1: back up the other way it is. It was really fun.
Speaker 1: I think it took us like a half hour. There
Speaker 1: were four of us playing. It says that you can
Speaker 1: play it with ten ages ten plus. You probably can
Speaker 1: if you're going to play competitively, as far as like, hey,
Speaker 1: you know, I'm stacking all these cards up so that
Speaker 1: all the resources on the earlier cards play nicely on
Speaker 1: the later cards. A ten year old might struggle with,
Speaker 1: you know, planning ahead that far figuring out what to do,
Speaker 1: but they definitely play the game right, and enough of
Speaker 1: it is common sense enough to where they can probably
Speaker 1: at least wrangle their way through a lot of it.
Speaker 1: So so yeah, so I'm going to pick that it's
Speaker 1: called far Away on the board game.
Speaker 2: Pick, and then let's see other picks.
Speaker 1: So I think I might have I think I might
Speaker 1: have picked this last time, but I'm going to just
Speaker 1: pick it again there's a movie that came out. It's
Speaker 1: called Truth and Treason by Angel Studios. My wife and
Speaker 1: I are members of Angel Guild, so we pay every
Speaker 1: month to be part of that. We get to vote
Speaker 1: on the movies that they make, and it's also part
Speaker 1: of our subscription to the Angel app that Angel where
Speaker 1: you know, you can watch videos and you can say,
Speaker 1: I don't want any of this kind of profanity or
Speaker 1: any of this kind of content, right, so it'll cut
Speaker 1: all the sex scenes out of your movies and stuff
Speaker 1: like that. But so we wind up getting tickets as
Speaker 1: part of our Angel Guild membership to all of these
Speaker 1: movies when they come out to theater. So this one
Speaker 1: is a World War II film. It's the story of
Speaker 1: three young men become disaffected with the Nazi regime during
Speaker 1: World War Two after their Jewish friend gets disappeared by
Speaker 1: the SS and so they start distributing leaflets by putting
Speaker 1: them in mailboxes and stuff and on cars and things
Speaker 1: around Hamburg and they get caught, and the movies about
Speaker 1: them and you know what happened to them, and so anyway,
Speaker 1: it was it was really really good. One of the
Speaker 1: things I like about these kinds of movies. I mean,
Speaker 1: it's it's a sad story, you know, in the end,
Speaker 1: you know the way, the way that it all goes
Speaker 1: for them. But it's like, look, you know, how how
Speaker 1: willing are you to stand up for what's right and
Speaker 1: what's true?
Speaker 2: And I think in.
Speaker 1: Today's world, in certain parts of the world, yeah, you
Speaker 1: may be risking your life, you know where I live. Yeah,
Speaker 1: I guess they do kill people for that because it
Speaker 1: killed Charlie Kirk. But I don't feel like somebody's going
Speaker 1: to kill me for standing up. But I've had people
Speaker 1: come after, you know, my reputation and things for things
Speaker 1: that I've said. But again it's down to how, you know,
Speaker 1: are you willing to stand up for truth?
Speaker 2: Are you willing to.
Speaker 1: You know, do the right thing even if it costs you.
Speaker 1: So anyway, it's called truth and treason. I don't know
Speaker 1: if it's still in theaters or not. I think it
Speaker 1: still is probably until like Thanksgiving, So yeah, definitely worth seeing. Terrific,
Speaker 1: terrific film. And then my wife, My wife and I
Speaker 1: are still playing Jaws of the Lion, which is one
Speaker 1: of the gloom Haven board game setups, so and it's
Speaker 1: basically without a Dungeon Master, self directed D and D
Speaker 1: kind of game, and so you know, uh, anyway, the
Speaker 1: difference being that you're it's not as free form. You
Speaker 1: actually have cards to give you your abilities and so
Speaker 1: you play the cards. Yeah, anyway, very fun. So I'm
Speaker 1: gonna pick that as well tomorrow.
Speaker 4: What are yours? Okay?
Speaker 3: Like, I just saw the movie Good Fortune, and uh,
Speaker 3: the first time I'm seeing a movie in a in
Speaker 3: a theater in New York. I just relocated here, and
Speaker 3: I felt that there was a somewhat funny mix of uh,
Speaker 3: you know, today's a tech tech bro you know thing,
Speaker 3: just this cushion related to together with like all the
Speaker 3: apps that we're using daily and how how like people influence?
Speaker 3: How are these apps that are designed by supposedly tech
Speaker 3: bros Our influence day to day? Kenor is like playing
Speaker 3: there as like an angel I don't give like ten
Speaker 3: out of ten, but coming from the tech industry, et cetera,
Speaker 3: I thought like it's an interesting movie also to see
Speaker 3: a bit a little bit how like other people like
Speaker 3: see our industry, et cetera.
Speaker 4: So I definitely like.
Speaker 2: Very cool.
Speaker 1: All right, Well, one last thing and then I have
Speaker 1: to jump off for a work media, I'm already late
Speaker 1: for if people want to check in see what you're
Speaker 1: working on.
Speaker 2: Check out codo. Where do people go for any of
Speaker 2: that stuff?
Speaker 4: Yeah, totally so first of all qodeo, dot ai.
Speaker 3: From there, we have like everything we are at social
Speaker 3: obviously as well personally, I a mar underscore. Mar is
Speaker 3: my handler at x Twitter, and we have multiple open sources.
Speaker 3: We're actually going to contribute some of them to one
Speaker 3: of the open source foundations.
Speaker 4: Still learning which one is the best one.
Speaker 3: So for example, we have like a pull request code
Speaker 3: review agent, so you can find it.
Speaker 4: It's called pr agent.
Speaker 3: It's very different than our main product, by the way,
Speaker 3: very very different, but it is part of like our
Speaker 3: collaboration with the community. So this is a bunch of
Speaker 3: ways to reach out and we love like hearing the
Speaker 3: community code reviews subjective quality is subjective, while we do
Speaker 3: need to standardize that. People think about it differently, so
Speaker 3: hearing everyone, please reach out and anything and we'll be
Speaker 3: in touch.
Speaker 2: All right, cool, Well, thanks for coming. This was fun.
Speaker 4: Yeah, same here. I really loved it.
Speaker 2: All right, folks, we'll wrap it here till next time.
Speaker 4: Max Out