Speaker 1: Welcome to the Occult Green Jacks. Tonight, we've got a
Speaker 1: very special guest, Ashih, who is going to tell us
Speaker 1: a lot of really deep intelligent stuff about AI and
Speaker 1: all all the rest of the stuff. He'll get into it,
Speaker 1: but first let's go to Nick. Nick, how you doing
Speaker 1: what you got coming out? Tell us about your herbal
Speaker 1: video that just popped out?
Speaker 2: Oh, thank you, thank you?
Speaker 3: Yeah that it seems like I seemed to be getting
Speaker 3: like you even mentioned good reviews are right, ab, we'll
Speaker 3: getting some good comments.
Speaker 2: It's a different way of looking at it.
Speaker 3: I do cover the like kind of the occult angle
Speaker 3: of the witchcraft angle, maybe even like a little bit
Speaker 3: of history, like I forgot what it might have been,
Speaker 3: peppermin or something, spearman something I was mentioning, like even
Speaker 3: I think in Greek Greek history of Greece they'd wear
Speaker 3: it around their heads, is like to help them study
Speaker 3: or something when they.
Speaker 2: Took tests, some random shit like that.
Speaker 3: Like and I'm throwing all that stuff in there, and
Speaker 3: then I'll go into how it actually affects your twelve
Speaker 3: cranial nerves and when you saw it seeing like when
Speaker 3: it's actually going to you your your you know your
Speaker 3: vegas system and your uh you're glad uh if it
Speaker 3: basically like the some effect like up to three at
Speaker 3: a time, and that gets pretty interesting when it starts
Speaker 3: doing that. You know your trigeminal, your vegas system, and
Speaker 3: maybe one or two one other one.
Speaker 2: But uh yeah, long story short.
Speaker 3: I think it's interesting for people to see like what
Speaker 3: some of these things actually do when you just inhale them,
Speaker 3: what kind of effects it's gonna have on your body
Speaker 3: and how you're gonna feel afterwards, and some of them
Speaker 3: that I even show, Uh, if you decide to just
Speaker 3: all of a sudden going crazy dosages with it, you'll
Speaker 3: notice it starts to shut your system down. And I
Speaker 3: start to wonder about that with maybe shamanistic rituals.
Speaker 2: You know, that's why you were fumigating that stuff, you know.
Speaker 3: So it's just interesting to see the kind of science
Speaker 3: with these things because you start to wonder, did they
Speaker 3: know this back then, because like the effects that you'll see,
Speaker 3: like that would be in like Scott Cunningham's book about
Speaker 3: what these herbs are gonna do for you, you can say, well,
Speaker 3: that's what it's doing to you fucking physically, just from
Speaker 3: hill or putting it on your skin, you know, so
Speaker 3: you almost wonder like were they were still getting it twisted?
Speaker 2: You know what I'm saying.
Speaker 3: It is still more science unfortunately with that stuff, and
Speaker 3: somehow it got woo wooed.
Speaker 2: You know.
Speaker 3: I don't know did we lose did we lose the original?
Speaker 3: I guess a cult cult idea behind it, something like that.
Speaker 3: So check it out. It is getting good reviews. I
Speaker 3: was a little surprised that actually has as many comments
Speaker 3: as it does now. So, uh yeah, it's got about
Speaker 3: three parts one drop today. We got a few other
Speaker 3: ones that will be coming out. I got part two
Speaker 3: of the Colors coming out. I should have Johannes Kepler
Speaker 3: dropping in like two weeks. I should be editing that
Speaker 3: up and that will be out. And I got into
Speaker 3: my park that should be dropping any within the week,
Speaker 3: I'm hoping. So yeah, we got a bunch of shit
Speaker 3: coming out. I have a bunch of stuff edited. I
Speaker 3: just haven't actually wrapped it all up yet. But uh yeah,
Speaker 3: then we do have some stuff going forward. We'll be
Speaker 3: covering the Stargate project next week. Again, it's gonna be
Speaker 3: a big group of the whole cult reject team we'll
Speaker 3: be here covering more of the Stargate or Gateway project,
Speaker 3: PaperWorks whatever was Yeah, so that should be interesting.
Speaker 2: So we got a lot of stuff coming.
Speaker 1: Thank you all right, sounds awesome, Ahi, Ish tell the
Speaker 1: people about what you've been through, tell them about who
Speaker 1: you are and what you do.
Speaker 4: Hi. Good to be back on guys. Thanks for having
Speaker 4: me in case you don't know me from the last
Speaker 4: times I was on here. My background, my name's Ashiesh.
Speaker 4: My background is in artificial intelligence and in finance. I
Speaker 4: spent ten years working in finance as a trader for
Speaker 4: Edward Jones Investments, and part of my responsibilities were covering
Speaker 4: tech at the time. After that, I transitioned full time
Speaker 4: into AI and I was specifically trying to apply AI
Speaker 4: to computer vision. And the presentation I want to give
Speaker 4: today is related to that because AI is a hot
Speaker 4: topic and I think one of the things that's happening,
Speaker 4: just like every other subject we've seen, there's a massive
Speaker 4: amount of distortion that's happening in AI, And what I
Speaker 4: want to do is I want to help people. So
Speaker 4: a lot of people think that AI is like a
Speaker 4: super advanced subject that's impossible to tackle or understand unless
Speaker 4: you have some kind of like background in computer science,
Speaker 4: and to be honest, it's not really necessary. I hope
Speaker 4: I can make this accessible for people to understand real AI.
Speaker 4: And I think it's really important that people understand this
Speaker 4: because there's so many profound things that are actually true
Speaker 4: about AI that are not really publicly discussed. And so
Speaker 4: one of the big topics that is kind of a
Speaker 4: big deal with AI and a lot of people who
Speaker 4: are like doomers on AI, they think that there's some
Speaker 4: kind of artificial life form that's about to be created,
Speaker 4: and this artificial life form will turned into the terminator
Speaker 4: or some apocalyptise scenario or something like that. Hopefully, what
Speaker 4: I can do here is bring a little bit of
Speaker 4: rationality this subject without getting overly technical. But there are
Speaker 4: a couple of technical concepts that I want to explain, no,
Speaker 4: I'll make it. And I also want to be able
Speaker 4: to show what real AI is and how you know
Speaker 4: what real AI is with versus what is not because
Speaker 4: there's a lot of junk AI out there. So let's
Speaker 4: just kind of start with like what is AI? Like fundamentally,
Speaker 4: at a kindergarten level, what is AI? AI is a
Speaker 4: mathematical function, and it is primarily based on a reward
Speaker 4: and cost. There's a function for reward that can be
Speaker 4: basically anything you want, and then there's a function for
Speaker 4: cost for that same reward, which can also basically be
Speaker 4: whatever you want. And the objective in AI is to
Speaker 4: make these two functions resemble real life as much as possible.
Speaker 4: So this is really what AI is. It's a type
Speaker 4: of pattern recognition map. And this pattern recognition map is intense.
Speaker 4: It's extremely large. Some of these models can have billions
Speaker 4: of weights, and I'll kind of explain a little bit
Speaker 4: about that. And because of the massive size of these models,
Speaker 4: they are also energy intents and the energy you may
Speaker 4: know in you know, in America, there's this huge push
Speaker 4: for energy investment. Weightly, we're spending trillions upon trillions of dollars,
Speaker 4: multiple trillions of dollars into both our energy grid and
Speaker 4: energy utilities. And then on top of that, AI as
Speaker 4: well is also independently receiving or trillions of dollars are
Speaker 4: independently being invested in AI. So there are two major
Speaker 4: energy intense aspects of AI. There is the training side,
Speaker 4: which is like what you do to build the model,
Speaker 4: and then there's the inference side, which is what happens
Speaker 4: after the model's already done you deployed it. There's people's
Speaker 4: hands and then they do a querry on something like chat,
Speaker 4: GPT or whatever, and then that quarry has a small
Speaker 4: amount of energy that's required to do this thing called inference,
Speaker 4: which then outputs your answer output to your result. Now,
Speaker 4: the training side is massive, and the inference side individually
Speaker 4: is very small, But where the massive energy constraint comes
Speaker 4: from is the scale. So once these developed models get
Speaker 4: deployed out into the world, the energy demand from the
Speaker 4: sum total of them all combined. That's where the energy
Speaker 4: intensity goes through the roof. So two sides that are
Speaker 4: heavily energy intense that are required in order to make
Speaker 4: AI work. So what you can see from this though,
Speaker 4: and what you should kind of interpret in your head
Speaker 4: just starting with this, is that if an AI model
Speaker 4: requires that much energy in order to generate the model
Speaker 4: and then be able to use it, you're probably in
Speaker 4: the vacuum stage, a vacuum tube stage, because this is
Speaker 4: nowhere near the efficiency of the compute that happens in
Speaker 4: the human brain. And I'm going to.
Speaker 1: Start relating hold on because I think this is a
Speaker 1: two way street. It's it's not only just taking everything
Speaker 1: that you're you know, sort of, it's not only taking
Speaker 1: your questions. It's taking all of the information together. So
Speaker 1: this is a massive data gathering tool as well as
Speaker 1: a data you know, output tool. Yeah, and that's the
Speaker 1: main sort of correlation that's being drawn here. I think
Speaker 1: they're sucking up a lot of core information, psychological information
Speaker 1: about each each one of us, how we interact with it.
Speaker 4: Yep. And the kind of that kind of description that
Speaker 4: you just gave that is most relevant to the concept
Speaker 4: of l ms, which are large language models. So this
Speaker 4: is like your chat Ebt, Gemini, crock whatever, take your
Speaker 4: flavor of the month. There's so many these things. This
Speaker 4: is a good, good thing you mentioned that, because what
Speaker 4: I want to show here, and one of the things
Speaker 4: that I'm hoping to promote, is that computer vision is
Speaker 4: a way better, more accurate version of AI than a
Speaker 4: large language model. So what I was going to say
Speaker 4: earlier was that I'm relating AI to the brain. And
Speaker 4: what I was originally trying to do here and when
Speaker 4: I was working on AI, was to model artificial intelligence
Speaker 4: off the human brain. So you're going to hear me
Speaker 4: relate to the human brain many times in this presentation.
Speaker 4: I'll get to that because that's a big part of
Speaker 4: what I'm doing here. So one of the things to
Speaker 4: know about that is that in a large language model,
Speaker 4: an LM is a pattern recognition math built around literally
Speaker 4: language text. And if you think about the human brain,
Speaker 4: the language center of the brain is a very small
Speaker 4: part of the brain right here over here. It's a
Speaker 4: very small part actually, And I'm not saying it's not important.
Speaker 4: A large language model and that technology is important, but
Speaker 4: that is not what's going to get you to this
Speaker 4: artificial life form. And that's controversial to say because there's
Speaker 4: so many people that think that a large language model
Speaker 4: will just suddenly become this artificial life form. But language
Speaker 4: is not a life that's not how language is a
Speaker 4: form pattern recognition math. And the easy way to understand
Speaker 4: this without having any understanding of the math behind it
Speaker 4: is if you walk into a room and there are
Speaker 4: ten people speaking different languages that you've never heard before,
Speaker 4: not languages that you don't understand but have heard before. No, no, no,
Speaker 4: languages that you've never heard before, and you will not
Speaker 4: be able to distinguish between all that noise one voice
Speaker 4: from another. However, if one of those ten people start
Speaker 4: speaking English in that room, all of a sudden, you'll
Speaker 4: be able to zone in on that wavelength and filter
Speaker 4: it out, and you'll be able to hear the English
Speaker 4: from the noise. This is how you know that language
Speaker 4: is a pattern recognition math and you're not born with language.
Speaker 4: Another thing to understand about language, two people in this
Speaker 4: world can hear the same sound and interpret something totally
Speaker 4: different from that sound. In other words, there can be
Speaker 4: a word in English and then another word in a
Speaker 4: different language that sounds exactly the same. That mean two
Speaker 4: totally different things. Two different people can understand two different
Speaker 4: things from this same sound. So this is how you
Speaker 4: know that language is an acquired skill. That is a
Speaker 4: form of pattern recognition. And this is not anything that
Speaker 4: you would call a life form. This is a talent,
Speaker 4: you could say, but it isn't a life form. And
Speaker 4: one of the other things is and I actually brought
Speaker 4: this up just so you could see in case anyone
Speaker 4: forgot from their seventh grade science class, but the definition
Speaker 4: of life to this day still does not include intelligence.
Speaker 4: And in my opinion, intelligence is equal to life these definitions.
Speaker 4: If you read what this is saying here on the screen,
Speaker 4: the actual definition of life, this sounds like primates trying
Speaker 4: to describe humans horribly. By the way, this is not
Speaker 4: the definition of life. If you take away reproduction, for example,
Speaker 4: is part of life. It is an essential element of
Speaker 4: what is considered life. If you took away reproduction from
Speaker 4: everything on Earth tomorrow, does that mean there's no life
Speaker 4: on Earth? That is how life works. So what I'm
Speaker 4: trying to argue here is that intelligence is part of
Speaker 4: the definition of life. And in case you're interested, there's
Speaker 4: something else that I want to say before I get
Speaker 4: into the actual formal presentation as a little set up here,
Speaker 4: So if you're interested in understanding more about intelligence and perception,
Speaker 4: my home basis perception. By the way, AI my focus
Speaker 4: area was perception. If you want to understand perception, it
Speaker 4: turns out, and what I'm going to demonstrate in this presentation,
Speaker 4: it turns out that computer vision by accident implicitly understands neuroscience.
Speaker 4: And I'll describe this in detail as I go through
Speaker 4: the presentation. But if you want to learn more about
Speaker 4: what I'm talking about on the neuroscience side, which I'm
Speaker 4: not going to do too much about. I'll talk mostly
Speaker 4: on the AI side, but if you want to learn
Speaker 4: more about the neuroscience side, I highly recommend this lecture series,
Speaker 4: and this one will break it down easy enough for
Speaker 4: you to understand some basic neuroscience about vision. So computer vision,
Speaker 4: in my opinion, doesn't get the rep that it deserves
Speaker 4: in AI. And here's something to note about all of this.
Speaker 4: So as I was saying, intelligence is not part of
Speaker 4: the definition of life, my dad was a neurologist and
Speaker 4: a psychiatrist, my mom is a pharmacist with a bunch
Speaker 4: of degrees and all this stuff. There's still disappointed that
Speaker 4: I'm not a doctor. And what what is interesting and
Speaker 4: what I've learned over the years is a two major things.
Speaker 4: So number one in evolution, one of the things, one
Speaker 4: of the first structures that was ever evolved was a
Speaker 4: photoreceptor attached to a flagella. A photoreceptor just says, so
Speaker 4: a flagella is a tail, and it was a photoreceptor
Speaker 4: was attached to it, and a photoreceptor it just says
Speaker 4: whether light exists or not, yes or no, one or zero.
Speaker 4: And so you can see that this is the beginning
Speaker 4: of binary map, whether light exists or does not exist.
Speaker 4: So another thing that is commonly known but often forgotten
Speaker 4: is that eyeballs are brain cells. So this is very
Speaker 4: important to understand. In the human eye, the ratio of
Speaker 4: cones uh they're there are these two types of uh self.
Speaker 4: There's these uh raw shaped cylinders which are gray and
Speaker 4: they determine grayscale. They determine basically opacity light light, the
Speaker 4: gradient of light, brightness of light, and grayscale. And then
Speaker 4: they have these cones RGB red, red, green, blue. The
Speaker 4: ratio on this is roughly twenty six three one. Roughly
Speaker 4: if you're a female, you'll probably have a little bit
Speaker 4: more blue, so twenty for twenty. The ratio of cones
Speaker 4: in the human eye is overweighted gray cylinders. Ratio of
Speaker 4: twenty on the gray, six on the red, three on
Speaker 4: the green, and then one or a little bit higher
Speaker 4: than one if you're female on the blue cones. So
Speaker 4: that's interesting. And what are the part that I'm going
Speaker 4: to skip in a neuroscience is about V one through
Speaker 4: V four cortex and the visual cortex. And this is
Speaker 4: outside of the you know, the purview of the presentation.
Speaker 4: But if you want to go deeper down the eyeball route,
Speaker 4: recommend after watching this presentation on the AI side. For me,
Speaker 4: if you want to understand what I was saying earlier
Speaker 4: about how computer vision experts implicitly, whether they know it
Speaker 4: or not, accidentally already understand the visual cortext just from
Speaker 4: computer vision, because computer vision is doing exactly what the
Speaker 4: visual cortex is doing exactly. It is literal, and I'm
Speaker 4: going to show published papers that demonstrate this. So what
Speaker 4: I'm saying is that computer vision in the modern day
Speaker 4: is mimicking the brain, and it is published and known,
Speaker 4: and computer vision experts, whether they know it or not,
Speaker 4: accidentally figured out the visual cortext and understand it implicitly.
Speaker 4: An example of this in computer vision, we understand why
Speaker 4: you want to use grayscale to detect motion. First, it's
Speaker 4: energy efficient and that's what happens in V one and
Speaker 4: that's why you have predominantly gray cylinders and your eyeball.
Speaker 4: And then there are functional purposes for RGB as well,
Speaker 4: and this is also discussed in computer vision. Okay, so
Speaker 4: that was a good like little preamble for the setup,
Speaker 4: and hopefully you're seeing that this is shit that no
Speaker 4: one's ever talked about. And this is real.
Speaker 2: AI, have you spent a lot of time? Go ahead,
Speaker 2: I was just gonna say, this is right up your all.
Speaker 1: He's upgraded all of these symbols in occultism directly to
Speaker 1: the eyeball, and the eyeball is so central to occultism
Speaker 1: in general. I mean it's in the word itself. And
Speaker 1: he breaks down all of these symbols and how they
Speaker 1: relate to stuff, and it's just like, yeah, it's brain tissue,
Speaker 1: this is what's programming.
Speaker 4: I also know the connection and spirituality, and I wasn't
Speaker 4: going to discuss it that much, but here's something to know.
Speaker 4: In the beginning, there was light and another thing about Yeah,
Speaker 4: I know a lot about that, but that's a whole
Speaker 4: wormhole and if I say something about it, we'll have
Speaker 4: to oh for like an hour. But in other words, yeah,
Speaker 4: there is it's it's there's a lot of indication that
Speaker 4: in religion that there was an understanding of evolution, especially
Speaker 4: in Hinduism, there was an understanding of evolution. And there's
Speaker 4: a lot of not just written documentation of this, but
Speaker 4: there's also archaeology that demonstrates this. Uh literal you can
Speaker 4: see it with your own eyes, thousands and thousands of
Speaker 4: years old, and they're demonstrating that they understood evolution and
Speaker 4: more beyond that.
Speaker 2: You know, it's real quick, you know, no, go ahead,
Speaker 2: let's go ahead, let's okay.
Speaker 1: Thees zinc calcium reaction at conception. It's a flash of light.
Speaker 4: Yeah, there's a Yeah, it's kind of like the Paho
Speaker 4: electric effect.
Speaker 2: But real quick, is she? I think it's interesting.
Speaker 5: You know.
Speaker 3: There was when a boy I think actually friends with
Speaker 3: them pravine. Uh, he was a cosmic summit. I went
Speaker 3: up to him and I said something to him. I
Speaker 3: was trying to explain to him. I was like, I know,
Speaker 3: like what you were showing on the screen, you were
Speaker 3: showing with something else. I said, but have you ever wondered,
Speaker 3: like if those are depictions of inside your rible?
Speaker 2: And he like like lost it.
Speaker 3: Like he like I looked like I literally like freaked
Speaker 3: him out to where.
Speaker 2: He like sat down and sat sat down to study
Speaker 2: eating his food.
Speaker 3: He's like, oh yeah, I was like, yo, I literally
Speaker 3: just bugged this guy the fuck out.
Speaker 2: But like you get it, you get it.
Speaker 4: Yeah, Pravine is one of my most favorite.
Speaker 2: Yeah, yeah, maybe you could maybe you could. Maybe you
Speaker 2: can tell that and he'll take it a little bit
Speaker 2: more serious, like.
Speaker 4: Oh, I've I've talked his ear off about this stuff.
Speaker 4: He's fully aware of everything.
Speaker 2: Oh, I don't know. He looked like he was bugged
Speaker 2: out when I said something to him, like he.
Speaker 4: Looked at that time. Oh okay if I had just
Speaker 4: met Pravine at that time, so he didn't know that
Speaker 4: much about me at that time. But we became good
Speaker 4: friends over time. Oh. Nice, I'm actually gonna go to
Speaker 4: Cambodia with him and a couple of months or a
Speaker 4: few months.
Speaker 2: Nice, that's awesome.
Speaker 1: Be careful of those planes.
Speaker 4: A lot of yeah, yeah, yoh.
Speaker 2: We'll have to get you on with Day to talk
Speaker 2: about the eyeball. I did like a five or six.
Speaker 3: I don't have many plots series with the cult symbolism
Speaker 3: in the eye, But if you ever want to come
Speaker 3: back on and talk about that ship, we're talking about.
Speaker 5: The eyeball at all. Day Man.
Speaker 2: A lot of my lot of all.
Speaker 3: My art, believe and I see the crossing of the arms.
Speaker 3: That's that's the crossing of the optic nerve. Like a
Speaker 3: lot of my art literally is depicting the inside of
Speaker 3: the eyeball, the one that I have like a chick.
Speaker 2: It looks like she's on a sea show.
Speaker 3: That's like the hyloid collac canal and the oriserata and
Speaker 3: then the ac humor underneath. I totally all my ship's
Speaker 3: all basically depicted on an eyeball on the brain man.
Speaker 4: And don't forget about the pineal gland, right, yeah, yeah,
Speaker 4: so we got all and then the third eye and everything. Yeah,
Speaker 4: there's quite a lot to do with vision. And uh,
Speaker 4: one of the how do I say so? One of
Speaker 4: the there's another thing that I want to discuss about
Speaker 4: evolution before I get into the formal presentation, and then
Speaker 4: we'll go from there. But one more thing about evolution.
Speaker 4: And I'm not sure how this happened, but for some
Speaker 4: reason in this zeitgeist, it is commonly thought or said
Speaker 4: that DNA is the cause of evolution. And this is
Speaker 4: just not true. There is no scientists saying that. The
Speaker 4: people who discovered DNA didn't say that. Darwin didn't say that.
Speaker 4: As far as I know, there is no official source
Speaker 4: anywhere that is saying that DNA causes evolution. DNA is
Speaker 4: the product of evolution, and that's why they're correlated, but
Speaker 4: correlation is not Causation is understood at kindergarten level, and
Speaker 4: as far as I know, no scientist is actually saying that.
Speaker 4: But for some reason in the psycheiasis is commonly thought
Speaker 4: that DNA causes evolution. This is not true, and there
Speaker 4: are it is academically understood and known. It is academically
Speaker 4: accepted that the brain has known mechanisms for controlling DNA.
Speaker 4: And even every in your everyday life, your DNA is
Speaker 4: not static. Things turn different, Switches turn on and off
Speaker 4: in your DNA all the time, and so and this
Speaker 4: mechanism is controlled by the brain, and often environmental factors
Speaker 4: are the reason for why this happens. So again, and
Speaker 4: another way, just like if you if you're just really
Speaker 4: thinking about it from a high level, you've got to
Speaker 4: understand that, like, uh, there's so much infrastructure that's required
Speaker 4: in order to make DNA work. In the first place,
Speaker 4: you need to have the hisstones to zip the file,
Speaker 4: because I look at DNA as being software. You need
Speaker 4: to have the zip hisstones to zip the file. And
Speaker 4: then you need to have a way of unraveling it.
Speaker 4: You need to have a way of splitting it. You
Speaker 4: need to have the transcription enzymes to transcribe the DNA.
Speaker 4: You need to have all the different proteins already there
Speaker 4: in order to even transcribe them in the first place.
Speaker 4: So there's and and then so much more. In order
Speaker 4: to make DNA work, it needs all of these other
Speaker 4: things as well, and so in order of operations. But
Speaker 4: how is like DNA by itself doesn't do anything. There's
Speaker 4: no magic that happens with DNA by itself, And so
Speaker 4: in order of operations, you would need to have all
Speaker 4: of this other infrastructure available or at least functional to
Speaker 4: some degree before DNA becomes useful. So in terms of
Speaker 4: order or operations, DNA can't be at the front of
Speaker 4: the line. This is the software. Software doesn't come before
Speaker 4: the hardware. So this is kind of you know, you
Speaker 4: can kind of understand this stuff at a pretty basic level.
Speaker 4: This is all commonly understood and all this stuff, but
Speaker 4: for some reason, there's just a different idea of how
Speaker 4: it works out there. So let's get into the presentation.
Speaker 4: So what I was doing at Arizona State University, as
Speaker 4: I was working on computer vision applied to artificial intelligence
Speaker 4: in automated vehicles, and so I was trying to apply
Speaker 4: camera based systems to automated vehicles. And one of the
Speaker 4: issues that I ran into in automated vehicles, especially at ASU,
Speaker 4: is that a overwhelming amount of investment and research in
Speaker 4: R and B effort, especially in academia but also out
Speaker 4: in the real business world, has been dedicated to the
Speaker 4: task of making liedar part of the automated vehicle apparatus system.
Speaker 4: And what I'm sure I'm going to be arguing in
Speaker 4: this presentation is that LDAR is completely useless. It's actually work.
Speaker 4: It makes your system worse, not better. And I want
Speaker 4: to be clear upfront. Lightar is the real technology and
Speaker 4: it does have real world application. It's used in space,
Speaker 4: it's used to map layout. There are actual, real functional
Speaker 4: purposes for lightar. But what I am going to hit
Speaker 4: hard on in this presentation is it LDAR makes no
Speaker 4: sense at all in an automated vehicle. Doesn't matter what
Speaker 4: your philosophy is, there is no reason to have lightar
Speaker 4: in a vehicle. So automated vehicle so me since maybe
Speaker 4: not everyone is familiar with how light our works, just
Speaker 4: to set it up the way that light our works
Speaker 4: is so the major companies that you may have seen
Speaker 4: that have light or Weimo is the big one in
Speaker 4: the US and Waimo is owned by Google. And then
Speaker 4: in the past there were other companies that have also
Speaker 4: used them. There was General Motors GM, but they ended
Speaker 4: up having a problem a couple of years ago in
Speaker 4: California where they ended up using a light our based
Speaker 4: system and this car dragged a person underneath their vehicle
Speaker 4: for like twenty or thirty feet, and so then they
Speaker 4: had to end basically operations. They were even caught lying
Speaker 4: like trying to deceive about it too when they were
Speaker 4: doing it. It was all over the news. People probably
Speaker 4: remember another company that was doing it was Uber and
Speaker 4: I live in Phoenix, Arizona. This is a heavy testing
Speaker 4: ground for automated vehicles. And the reason why Arizona is
Speaker 4: a heavy testing ground or Phoenix specifically is a heavy
Speaker 4: testing ground for automate vehicles is because we have three
Speaker 4: hundred and sixty days of sunshine and we have some
Speaker 4: of the widest and best roads in the US because
Speaker 4: we have massive trucking that comes through here, and so
Speaker 4: this is and we have a very grid like road system,
Speaker 4: very squared, grid like road system is very clean at organized,
Speaker 4: so we're pretty much ideal set up for this, and
Speaker 4: so WEIMO eventually came here, and what they ended up
Speaker 4: doing is they ended up going to ASU. WEIMO did,
Speaker 4: and they ended up investing a huge amount of money
Speaker 4: into their engineering department. And that was very problematic for me,
Speaker 4: which I didn't know about at the time, because I
Speaker 4: was going in there applying with this presentation for a
Speaker 4: PhD in computer vision to use it for automated vehicles.
Speaker 4: And so you'll see why this gets to be a
Speaker 4: big problem when you're in the hornet's nest with WEIMO.
Speaker 4: So before I get to that, let's kind of build
Speaker 4: it all up and kind of set up what I'm
Speaker 4: talking about here. My home basis perception. But and by
Speaker 4: the way, if this math is scaring you, don't worry.
Speaker 4: I'm not going to get to mathavy. I'm breaking everything down.
Speaker 4: You don't need a degree to understand any of this stuff.
Speaker 4: But what is important here is that there are three
Speaker 4: major elements to UH automated vehicles for doing the doing
Speaker 4: controlling automated vehicles. So the first element is perception, and
Speaker 4: then dynamics and then control and then basically what I
Speaker 4: have on this side is intro to robotics. So this
Speaker 4: is this is the functional math for how you do robotics.
Speaker 4: And what I have here on the left side, this
Speaker 4: big old crazy looking equation. All this, all this is
Speaker 4: saying is what is the least wrong answer? So based
Speaker 4: off of these three equations on the right side, the
Speaker 4: left side is trying to figure out what is the
Speaker 4: least wrong answer based off of that, And what you'll
Speaker 4: see is that perception, which is ZT. Perception is used
Speaker 4: in all three equations, is fundamental to all three equations.
Speaker 4: So perception is the most important task in robotics. Perception.
Speaker 4: So here are some of the major brands. So remember
Speaker 4: that this presentation was done in twenty twenty three, so
Speaker 4: some of this is a little bit dated, but almost
Speaker 4: all of it is still relevant today. This is a
Speaker 4: company called Argo on the top left here. This was
Speaker 4: owned by Ford. This is a now defunct company. But
Speaker 4: they were also trying to force fit lighter into their cars.
Speaker 4: They weren't able to do it, so they went bankruptor
Speaker 4: they closed on the company. But what I wanted to
Speaker 4: point out about this is that they have these light
Speaker 4: our things on here. But notice the cameras that they
Speaker 4: used are ring cameras, literal ring cameras, and so you
Speaker 4: can tell that they went way out of their way
Speaker 4: to do go all in on cameras. Huh. Of course,
Speaker 4: Weimo's the big one. This company down here in the
Speaker 4: center is this should be Wave if I recall correctly,
Speaker 4: it's been a little while since I've done this. And
Speaker 4: then there's a couple others. It's fine. There's a Chinese
Speaker 4: company over here called Xpaying, which people should know about
Speaker 4: if they care about this stuff. So LDAR is so
Speaker 4: prevalent and overwhelmingly so. Almost every major company has been
Speaker 4: trying to force fit LDAR into their automated vehicle, and
Speaker 4: almost every single one of them is failing. The only
Speaker 4: one that is achieving scale, despite what narratives are, the
Speaker 4: only one that is achieving scale on the autonomous vehicles
Speaker 4: in this world is Tesla. And so Tesla is a
Speaker 4: camera based system that does not use any other perception sensor,
Speaker 4: just cameras and way moos use lightar and camera. So
Speaker 4: here's another thing about LDAR. Even if if you do
Speaker 4: use a light ar system, no matter what, you still
Speaker 4: need a camera based system. Doesn't matter who you are,
Speaker 4: what philosophy you believe in it doesn't matter anything. You
Speaker 4: still need a camera based system. So I'm going to
Speaker 4: discuss why that's important or why that is. This is
Speaker 4: like Chinese companies. I'm going to skip some of this
Speaker 4: stuff because this presentation is actually quite long, and so
Speaker 4: I'm going to skip it to condense it quite a
Speaker 4: lot and stick to really the most important points. So
Speaker 4: one of the issues is that lighter has got this
Speaker 4: spinny thing that's going around. You've probably seen it if
Speaker 4: you've seen a Waymo vehicle or something like that. Zeukes
Speaker 4: I've just seen in Las Vegas has also started to deploy,
Speaker 4: which I believe it's owned by Amazon. This is a
Speaker 4: relatively new addition, but again still using widar. And what
Speaker 4: a light oar system does is this little spinny thing
Speaker 4: that I'm talking about. It shoots a laser beam and
Speaker 4: it shoots a ton of these all over the and
Speaker 4: then there's like you're supposed to get a reflection back.
Speaker 4: This is an imperceptible beam of light, laser beam of light,
Speaker 4: and then based on that time and distance, they're able
Speaker 4: to get a point return of information on depth. So,
Speaker 4: in other words, this laser beams functional purpose and what
Speaker 4: it's doing, is it's trying to measure depth of the
Speaker 4: things around it. The actual literal laser precise depth calculation,
Speaker 4: and it is laser precise, it is super precise depth calculation.
Speaker 4: But this is effectively the entire in its entirety, the
Speaker 4: full and complete explanation of the functional purpose of why
Speaker 4: these light oar systems exist on these cars is for
Speaker 4: precise measurement of depth. But lightar is like a dumber
Speaker 4: version of sonar, that's the way to think of it.
Speaker 4: And it creates a point cloud return kind of a
Speaker 4: grip around it in point cloud return. And I'm going
Speaker 4: to show videos and pictures in all this of that.
Speaker 4: But what I want to show first before I get
Speaker 4: into the videos of that, is I want to show
Speaker 4: why a few different cases of why lid are doesn't
Speaker 4: make any sense in an automated vehicle. Why am I
Speaker 4: saying that so strongly? So this what I have on
Speaker 4: the screen, And by the way, every slide should have
Speaker 4: a source on it, So like down here, this is
Speaker 4: the actual paper, academically accepted paper, and i E. The
Speaker 4: two major conferences I'm going to be talking about are
Speaker 4: i EE and CVPR. I EE is like basically the
Speaker 4: biggest engineering conference in the world, and CVPR is the
Speaker 4: largest computer vision conference in the world. So all of
Speaker 4: these papers and slides and presentations that I'm going to
Speaker 4: be talking about are all from the largest possible conferences,
Speaker 4: the most academically accepted conferences that there are in the world,
Speaker 4: not just in America, in the world. So this paper
Speaker 4: that I have on the screen is a little dated.
Speaker 4: It's from twenty twenty one, but this is considered originally
Speaker 4: the foundational paper on identifying what is known as VRUS.
Speaker 4: VRUS stands for Vulnerable road user and this is a
Speaker 4: regulatory term, not an AI term. This is a regulatory
Speaker 4: term and it is basically a catch all phrase for pedestrians, dogs, cats,
Speaker 4: et cetera. And what this paper is showing is that
Speaker 4: it at that time and it identified thirty two attributes
Speaker 4: on identifying a pedestrian, which is like one of the
Speaker 4: most important tasks that a computer, any automated vehicle needs
Speaker 4: to do. And if you look at what these attributes
Speaker 4: are that they're talking about, like a handbag, clothes and
Speaker 4: things like that, the bags that they're carrying male female.
Speaker 4: The issue with a point cloud return is that it
Speaker 4: only gives you the shape. It does not give you
Speaker 4: the color quality that you get from a camera. So
Speaker 4: liedar is like a one dimensional point. It's just a
Speaker 4: point that tells you death and that's it. And you
Speaker 4: get a whole bunch of points, millions of points, and
Speaker 4: then hopefully you can identify the shape of an object
Speaker 4: with these points. That's what light our does. Well. It
Speaker 4: turns out that if this is what you're prioritizing, and
Speaker 4: this is what you're trying to maximize for, which is
Speaker 4: what a lot of pretty much all these light our
Speaker 4: companies are doing well, you miss out on all of
Speaker 4: those extra fine quality details that are actually most relevant,
Speaker 4: the most important aspect is this stuff. And this is
Speaker 4: academically understood. So this is in twenty twenty one, so
Speaker 4: this is dated. So this has actually been improved significantly
Speaker 4: since then. But this is just a cited This is
Speaker 4: known for like a long time in the industry. I'm
Speaker 4: gonna kind of skip this part and go to some
Speaker 4: more of the meat. So right here is something called
Speaker 4: this is Argo verse, which is Argo that the company
Speaker 4: that was that company that was owned by Ford that
Speaker 4: was also trying to force fit LDAR into their into
Speaker 4: their cars. They're now defunct, but in twenty twenty three
Speaker 4: at CVPR they presented this information that I'm showing on
Speaker 4: the screen. Now. One of the things that has happened
Speaker 4: in academia, and I'm pretty sure, at least as far
Speaker 4: as i'm aware to this day, is still true. One
Speaker 4: of the things that has happened in academia is that
Speaker 4: research around light our degradation over range seems to be
Speaker 4: heavily suppressed. And before ARGO went defuncts they published this
Speaker 4: and this is, as far as I'm aware, the only
Speaker 4: major company that has ever published anything like this. What
Speaker 4: this is showing on the screen here is the accuracy
Speaker 4: on object and classification identification using light ar based systems.
Speaker 4: So you can see in their best case scenario they're
Speaker 4: getting about fifty one percent accuracy at zero to fifty
Speaker 4: meters of range, but once you go to fifty to
Speaker 4: one hundred meters of range, it drops by half, and
Speaker 4: then once you go one hundred to one hundred and
Speaker 4: fifty meters a range, it drops by half again. So
Speaker 4: what this is showing is that it's showing a log
Speaker 4: to degradation in accuracy every fifty meters using a light
Speaker 4: ur based system. And I just did the math down here.
Speaker 4: This is some basic maths. This is about as intense
Speaker 4: as the math is going to get. At sixty miles
Speaker 4: per hour, that's twenty six point eight two meters per second.
Speaker 4: That means fifty meters of accurate range. Gives you two
Speaker 4: seconds of forecasting time, so you're able to see at
Speaker 4: two seconds of what ahead of you with with this
Speaker 4: system at sixty miles per.
Speaker 1: Hour, this completely takes automated trucking off of the board.
Speaker 1: You cannot have those type of tolerances if you're going
Speaker 1: to actually have a truck that's now commanded by AI
Speaker 1: or whatever.
Speaker 4: No doubt about that. Yeah, that's why there isn't a
Speaker 4: light hour based truck as far as I'm aware, it's
Speaker 4: not one that carries weight. But there is the Tesla
Speaker 4: semi which does exist and is on the roads. It
Speaker 4: is working, and there are millions of vehicles Tesla vehicles,
Speaker 4: automated vehicles on the road. In fact, I have one.
Speaker 4: Oh I should also mention this. I've been beta testing
Speaker 4: the Tesla automated system since twenty eighteen. I have more
Speaker 4: than one hundred thousand miles driven on their system since then.
Speaker 4: Over multiple cars and it's night and day. Comparatively to
Speaker 4: anything else that you can never see, it's like ridiculous,
Speaker 4: how much better it is. And again, Tesla has millions
Speaker 4: and millions of these vehicles on the road and they're
Speaker 4: all over the country doing this and actually the elsewhere
Speaker 4: in the world. But Waimo only operates geo fence spaces
Speaker 4: and a few select cities. They only have two thousand
Speaker 4: vehicles in their fleet. And I'm going to get into
Speaker 4: more about all that, but it's that that's worth keeping
Speaker 4: in your mind for now, and it's gonna come up
Speaker 4: later again why that's relevant.
Speaker 3: So I think I did want to ask you though,
Speaker 3: like what do you think about like light art kind
Speaker 3: of being used on the ground to service, Like I mean,
Speaker 3: you aren't even using that as a way of like
Speaker 3: kind of like a ground truthing for like these satellite
Speaker 3: images like even on the Pyramids or other stuff.
Speaker 1: That's fine, all right, yeah, ye think about this. So
Speaker 1: they're only taking in one wavelength of light, so they
Speaker 1: don't know what what the fuck anything is. It's just
Speaker 1: a distance. You've only got one dimension of vision. It's
Speaker 1: it's worse than gray scale.
Speaker 4: Yeah, And I'm setting up a few things here on
Speaker 4: purpose before I show you the video of what the
Speaker 4: light are actually looks like, because I want you out
Speaker 4: kind of see how many different problems problem scenarios there are.
Speaker 4: And then when I show you the video, you'll see
Speaker 4: you'll have all the context you need for the video
Speaker 4: to make sense. So you're going to see that, don't worry,
Speaker 4: it's coming. You're going to see why how much of
Speaker 4: a difference there is between light R and basically vision.
Speaker 4: It's like not even close. So what I was saying
Speaker 4: earlier though, was that light our based systems require geofencing,
Speaker 4: meaning they have to map to a specific area and
Speaker 4: they can only operate within this specific area, whatever this
Speaker 4: area block is, and they have to go through this
Speaker 4: area over and over and over again, and then they
Speaker 4: map it. It's effectively memorizing the area. And that's what
Speaker 4: geo fencing is called, and that's what WEIMO does. They
Speaker 4: don't operate everywhere in Phoenix. They operate within confined regions
Speaker 4: and that's for a reason. And another thing about this
Speaker 4: is that that means that you're only allowed to do
Speaker 4: in that system. In a light our based system, you're
Speaker 4: only at this point as far as I'm aware, and
Speaker 4: there's no technology that can beat this. Are only allowed
Speaker 4: to do point to point travel, specific point pre determined
Speaker 4: to specific point predetermined. And this is different than say
Speaker 4: at Tesla, which can do anywhere to anywhere travel. And
Speaker 4: so that means you can start from anywhere go to
Speaker 4: anywhere Whereas a way, MO has to start in pre
Speaker 4: defined spots only and go to pre defined spots only,
Speaker 4: and these have to be these are finite. It's not
Speaker 4: anywhere in that space. So even within the geofence region,
Speaker 4: they can only go from specific place to specific place,
Speaker 4: not anywhere anywhere you want in there. And so one
Speaker 4: of the things that has happened, and one of the
Speaker 4: distortions that has happened, this is infected the regulatory space
Speaker 4: at a kindergarten level, this is broken somehow, some way.
Speaker 4: Waimo and other light our companies have convinced everyone that
Speaker 4: point to point travel is more difficult than anywhere to
Speaker 4: anywhere travel, And to this day these systems are still
Speaker 4: classified that way, where point to point travel, which is
Speaker 4: an ADS system in regulatory terms, is somehow, some way
Speaker 4: from a regulatory perspective, it is somehow more difficult than
Speaker 4: anywhere to anywhere travel, which is an ADAS system. Sort
Speaker 4: of well, I should be careful what's saying ADS versus eights. Actually,
Speaker 4: let me come back to ADS versus ADS later. But
Speaker 4: the point is that somehow they've convinced regulatory people that
Speaker 4: point to point is harder than anywhere to anywhere, when
Speaker 4: it's clearly the other way around. By far, it's exponentially
Speaker 4: more difficult to do anywhere to anywhere travel than point
Speaker 4: to point, especially when you're not confined by geofencing. One
Speaker 4: of the other things is that these systems, these light
Speaker 4: our based systems, are extremely expensive. Now I'm sure the
Speaker 4: cost has come down over the past couple of years,
Speaker 4: but in twenty twenty three, the average cost on those
Speaker 4: weay moths for the lighter based system that apparatus that
Speaker 4: they put on there, that cost is fifty thousand dollars
Speaker 4: just for the system alone. Then you have to at
Speaker 4: that time roughly maybe it's come down a little, but
Speaker 4: it's still stupidly expensive. And then for whatever reason, they're
Speaker 4: putting these things on Jaguars, So that's another eighty thousand.
Speaker 4: So just between these two things alone, you have more
Speaker 4: than one hundred and twenty thousand dollars being spent on
Speaker 4: these systems one hundred and twenty one hundred and thirty thousand,
Speaker 4: and that's not including a safety driver, that's not including
Speaker 4: R and B, that's not including any kind of testing or anything.
Speaker 4: That's just the hardware alone with nothing else. No software
Speaker 4: is even put in it, No AI has even been
Speaker 4: put in there yet, So already this stuff is automatically
Speaker 4: absurdly overpriced to begin with, whereas the Tesla vehicles are
Speaker 4: thirty forty thousand dollars by comparison all in with everything
Speaker 4: already done, and so the economics here is something that
Speaker 4: people need to understand is that there is nothing like
Speaker 4: the scale is not reasonably possible with a light our
Speaker 4: based system because the expense on light our based systems,
Speaker 4: even after the costs have come down, is still absurd.
Speaker 4: By comparison, a camera costs about one hundred dollars. The
Speaker 4: cameras that are on the Teslas that they've been doing
Speaker 4: for years and years that do all this stuff anywhere
Speaker 4: to anywhere, travel just one hundred bucks each, so the
Speaker 4: cost difference is absurd. One of the other issues with
Speaker 4: geofencing is that if geo fencing and basically memorizing the
Speaker 4: area means that you have to expect that the area
Speaker 4: is static, because as soon as the area changes in
Speaker 4: the material way, your training data goes out the window.
Speaker 4: And so what construction becomes a problem because these space
Speaker 4: says will continuously change. So as soon as construction arise,
Speaker 4: all your training data for that area is gone, and
Speaker 4: then you have to retrain with the construction there. And
Speaker 4: then as soon as the construction is done, the whole
Speaker 4: area changed again, and so you have to redo the
Speaker 4: training with new data all over again. So it is
Speaker 4: not easily adaptable. So one of the big problems with
Speaker 4: interacting in the real world is that it's an open
Speaker 4: set environment. What that means in mathematical terms, what that
Speaker 4: means is that there's infinite possibilities. So if you're trying
Speaker 4: to memorize an answer, which is what basically every light
Speaker 4: our system is trying to do, if you memorize an answer,
Speaker 4: you cannot possibly memorize infinity. And the problem they're trying
Speaker 4: to solve is an open set problem, which means there's
Speaker 4: infinite possibilities, So you cannot memorize infinity, so you are
Speaker 4: destined to fail from the beginning if your system is
Speaker 4: dependent on geofencing or memorization, which it's basically the same thing.
Speaker 4: So just to kind of demonstrate other issues that come
Speaker 4: up with light AR, this is an intersection in China,
Speaker 4: And if you look at this intersection in China, just
Speaker 4: look at the heavy complexity that's involved here, the path
Speaker 4: and complexity, all the lands, all the turning, all the
Speaker 4: different conditions that would be required, and the amount of
Speaker 4: traffic that you would have normally going through this. How
Speaker 4: intense it would be to navigate and manage all that. Well,
Speaker 4: this is yet another situation where light R doesn't do
Speaker 4: shit for you, nothing at all. Light R does not
Speaker 4: help you with this problem in any way, shape or form,
Speaker 4: exactly zero. It's useless, doesn't do anything. Same thing as
Speaker 4: true in America. I was just showing this one because
Speaker 4: one of the issues is that you want to be
Speaker 4: able to generalize an answer. So if you try to
Speaker 4: train on just US infrastructure, then you're pigeonholing yourself to
Speaker 4: US infrastructure and you can't work anywhere else. And one
Speaker 4: of the things I'm trying to get it is I'm
Speaker 4: trying to model off the human brain, so we need
Speaker 4: to find a generalizable answer that works anywhere regardless of
Speaker 4: where you are a situation. Here's another problem. You have
Speaker 4: all kinds of different signs, all kinds of different lights,
Speaker 4: and they're constantly evolving, and there's so many different versions
Speaker 4: of them, and they constantly change. And this is another
Speaker 4: situation where light ar doesn't do shit for you. Exactly zero.
Speaker 4: There's nothing about lightar that helps you with anything here
Speaker 4: on the screen, zero exactly zero.
Speaker 1: So they ishud on our streets. I mean, if they
Speaker 1: can't read a light, what the fuck are we doing
Speaker 1: letting it drive around?
Speaker 4: Not a clue? I mean, I will make the argument
Speaker 4: for I will a little bit later give the fundamental
Speaker 4: argument about why light R is used, which is about
Speaker 4: depth perception. This is the fundamental functional purpose that light
Speaker 4: our serves. But we need to get a little more
Speaker 4: involved into this presentation before I really hit it. But
Speaker 4: the argument for lightar is it is accurate death. And
Speaker 4: I'll get into that a little later. Let me kind
Speaker 4: of do all the setup for everyone, because these are
Speaker 4: all going to be new concepts, so we're gonna want
Speaker 4: We're going to do this right. Let's put it that way. So,
Speaker 4: as I was saying before the primary issues that we
Speaker 4: have an open set environment. If we're trying to interact
Speaker 4: in the universe, in the real world, then there's going
Speaker 4: to be an infinite number of things that can happen.
Speaker 4: So there are three major categories of problems. There are
Speaker 4: the known common problems that we always know about. That's
Speaker 4: this green bubble. Then there's the long tail known cases,
Speaker 4: the rare events but we know about them. Those are
Speaker 4: that's the blue circle. And then there is this much
Speaker 4: larger orange circle, which is the long tail unknown events,
Speaker 4: the unusual events that we don't know about, which is
Speaker 4: the actual larger set of problems, way larger than the
Speaker 4: other two. So just to give you some examples, at
Speaker 4: least once a year, in this top left photo, at
Speaker 4: least once a year, there might be a dinosaur cross
Speaker 4: the road. And on the top right photo you have
Speaker 4: a literal it's I know it's kind of hard to see,
Speaker 4: sorry small, but that's a literal plane on the road.
Speaker 4: And then you have a few other different issues here,
Speaker 4: like on the bottom that and we'll hit on a
Speaker 4: lot more of this stuff, but you can see how
Speaker 4: there's a lot of unusual things that happen in fact,
Speaker 4: the number of weird things that can happen is probably
Speaker 4: way higher than you really are common Like, even if
Speaker 4: you drive a lot, you'd be surprised how many weird
Speaker 4: things happen that you've probably personally never seen before. So
Speaker 4: in particular, what I want to highlight here is the
Speaker 4: center top video and the center bottom video. So in
Speaker 4: the center top video you have a bunch of small
Speaker 4: mammals all over the road. In the center bottom video,
Speaker 4: you have like flying pages. Now I was mentioning earlier
Speaker 4: that the way light our works is that lied Our
Speaker 4: tries to do object detection and classification by shape. So
Speaker 4: what really is problematic is if you're trying to classify
Speaker 4: by shape, what ends up happening is if you get
Speaker 4: this wrong, meaning you reverse these two scenarios. In the
Speaker 4: top scenario, you may go very cautiously because of these mammals.
Speaker 4: In the bottom scenario, you don't want to you're on
Speaker 4: the highway. On this scenario, you don't want to force
Speaker 4: the break right away. You would actually go really aggressively
Speaker 4: because you would know that these are pages. However, if
Speaker 4: you're classifying by shape, you might reverse these two scenarios,
Speaker 4: and that's when the horrible stuff happens, and so this
Speaker 4: is another example where shape can easily be fooled and
Speaker 4: where the quality, the quality of color becomes very relevant
Speaker 4: and important. And in fact, not long ago I think
Speaker 4: it was actually when we were at the comp for
Speaker 4: instant weimo run over a cat and so uh yeah,
Speaker 4: so I mean that also became really I'm a big
Speaker 4: cat fan, So that's that really hurt. And of course
Speaker 4: you have a lot of unusual scenarios.
Speaker 1: Top right, what's the difference between a cat and a
Speaker 1: bag blowing in the wind. If a cat's at full sprint,
Speaker 1: how do you tell the difference in shape from that?
Speaker 1: And maybe a bag that's now in the shape of
Speaker 1: a cat.
Speaker 4: Yeah, you would be surprised at how much better computers
Speaker 4: are at computer vision than humans when you really want
Speaker 4: to do And one of the things that I'm not
Speaker 4: going to hit on it, but maybe it's worth mentioning
Speaker 4: since you asked, is that once you can do the
Speaker 4: visual spectrum, why can't you do everything else? There's nothing
Speaker 4: stopping you. And so that's exactly what the evolutionary path
Speaker 4: will look like when you get really advanced. Is that
Speaker 4: once you can do wavelength light there's nothing stopping. That's
Speaker 4: a very tight that's like the smallest liver of the
Speaker 4: wavelength available to you. Once you can do that, you
Speaker 4: can do all the other stuff. So just add infrared
Speaker 4: for example, and you can automatically see through smoke, you
Speaker 4: can see through fog, you can see through you can
Speaker 4: see at night. You'll be able to do all kinds
Speaker 4: of stuff. So this is all and it's all vision,
Speaker 4: not anything to do with light, ar or anything like that.
Speaker 4: And of course you'll have some unusual situations like in
Speaker 4: the far right scenario that's a random washer rolling down
Speaker 4: the road, and so that can also be problematic. Sometimes
Speaker 4: you'll have trees that are in the middle of the road,
Speaker 4: like a bottom left and then this situation I'm going
Speaker 4: to bring up now discuss it a little more. But
Speaker 4: second from the bottom right, this is a trash can
Speaker 4: that's blowing from the wind through the road. Now, the
Speaker 4: reason why in AVS for a brief period of time
Speaker 4: this scenario became complicated was because there's this thing called
Speaker 4: kinematic bits. These things that kinematic bits is just motion bits,
Speaker 4: motion information data about motion. And so so people started
Speaker 4: training on kinematic bits in addition to the wavelength light
Speaker 4: and stuff like that. And so what they did was
Speaker 4: they trained the system to be able to recognize every
Speaker 4: single trash can there is, and it did it. But
Speaker 4: one of the problems was that the training that they
Speaker 4: did was on trash cans that didn't move, and so
Speaker 4: when they deployed it, there was this trash can that
Speaker 4: started moving and so it had kinematic bits, but every
Speaker 4: trash can it was trained on had zero, no kinematic bits.
Speaker 4: So trash can is not supposed to move. And so
Speaker 4: you can see how a detail like that will mess
Speaker 4: up the system. And so one of the things that's
Speaker 4: important is to understand where are all the advantages and
Speaker 4: the disadvantages, and kinematic bits are actually an advantage, you
Speaker 4: just need to make sure you train on them. And
Speaker 4: so this is earlier. This scenario that I'm talking about
Speaker 4: has already been figured out. Don't worry. This is you know,
Speaker 4: long ago. So people have figured this dumb shit out.
Speaker 4: But all we had to go through the evolutionary process.
Speaker 4: And I think in some of the that I've seen,
Speaker 4: you know, because there's so many companies that are trying
Speaker 4: to do light ar at this point and I see
Speaker 4: all kinds of horrible stuff, and I see a lot
Speaker 4: of cheats as well, And I'll talk a little bit
Speaker 4: about one of those cheats right now, which is a
Speaker 4: bounding box. So this is a in computer vision, this
Speaker 4: is not a big deal. But in basically any other
Speaker 4: like what do you call it, any other modality, this
Speaker 4: bounding box problem I've seen is a big problem. And
Speaker 4: even in twenty twenty three, I was still seeing papers
Speaker 4: that were being submitted to CVPR or to IEEE that
Speaker 4: we're using bounding boxes. Now, that paper that I was
Speaker 4: showing before about the pedestrians the vr us in twenty
Speaker 4: twenty one, that's a found I was saying that the
Speaker 4: foundational paper heavily cited, well understood, academic accepted, you know,
Speaker 4: big deal. That paper also demonstrated that bounding boxes are
Speaker 4: a cheat. And what a bounding box is is that
Speaker 4: not only do you need to be able to identify
Speaker 4: what an object is, but you also need to be
Speaker 4: able to position that object on the frame, be able
Speaker 4: to identify that position. And what a bounding box is
Speaker 4: basically a box that goes around the object you're trying
Speaker 4: to identify. And the issue with this is that a
Speaker 4: bounding box is going to always be larger than the
Speaker 4: actual object itself, and it's not going to be the
Speaker 4: shape of the object. The bounding box is just going
Speaker 4: to like for example, in this circle that you're seeing here,
Speaker 4: the bounding box in this example is way larger than
Speaker 4: the actual truck itself, but so you can see how
Speaker 4: much larger it is. So in the scenario like you
Speaker 4: have in the middle here with this overturned truck, if
Speaker 4: you imagine a bounding box with this kind of error
Speaker 4: rate on it in terms of the size of its shape,
Speaker 4: and apply it to this truck, the bounding box is
Speaker 4: going to be bigger than the frame. Now, why that's
Speaker 4: important is that it's easier to see on this bottom
Speaker 4: left photo. That's important is because if you're using a
Speaker 4: bounding box and it's larger than the object, you can
Speaker 4: see that there's bounding box on the side here. You
Speaker 4: can see here this side here is covering up the road.
Speaker 4: So from a from the AI's perspective, there is no
Speaker 4: road to travel on, and so you'll see the AI
Speaker 4: systems start to stutter here because it doesn't think there's
Speaker 4: actually anywhere to go, any room to go, and so
Speaker 4: it has no way of path projecting through a bounding
Speaker 4: box because you're not supposed to interact with this object.
Speaker 4: So this is why bounding boxes are a cheat. And
Speaker 4: this is still, as far as I know, this is
Speaker 4: still actually happening. Unfortunately, in computer vision, I think AI
Speaker 4: specialists have figured this dumb shit out, but in everything else,
Speaker 4: I think they haven't. And this is a real shame.
Speaker 4: And I'll show you the solution to this problem, how
Speaker 4: you solve the bounding box problem. But I just want
Speaker 4: to bring it up right now that I've seen many
Speaker 4: light our based systems use this, and this is a cheat.
Speaker 4: This is a known cheat, and this is academically published
Speaker 4: as a cheat, and it's unfortunate that this thing is
Speaker 4: still happening. So here's the WEIMO system on the on
Speaker 4: the right here, and you can see the little spinny
Speaker 4: thing that's a lighter. And like I was saying, no
Speaker 4: matter what your philosophy is, you still need to have
Speaker 4: cameras anyway. And what this is doing is it's kind
Speaker 4: of showing the cleaning system that they're using. So this
Speaker 4: whole apparatus is being put on top of jaguars in
Speaker 4: Phoenix and in La San France and maybe a couple
Speaker 4: other cities now Austin, I think also, and so what
Speaker 4: I want. So there's a couple major issues with this.
Speaker 4: So number one, there's a reason why we do wind
Speaker 4: tunnel testing. And so if you're putting all these light
Speaker 4: oar systems on it, and it's not just this little
Speaker 4: thing on top, it's also got light oars on the front,
Speaker 4: on the two corners on the back, and the two
Speaker 4: corners I think on the center back as well. So
Speaker 4: you're increasing the amount of mass and you're increasing the
Speaker 4: amount of surface area on the exterior of the vehicle
Speaker 4: in order to do this. And there's a reason why
Speaker 4: we do windsital testing. So if you do all this,
Speaker 4: you're gonna basically invalidate all that and this is going
Speaker 4: to affect your range efficiency. So you're gonna have range
Speaker 4: problems as a result of this. And this is especially
Speaker 4: going to become a problem at speed. And so the
Speaker 4: higher speed you go, the more wind resistance you're going
Speaker 4: to get in, the more problems you're gonna have with range.
Speaker 4: Another issue with this, and what I was showing here
Speaker 4: with this video is the cleaning system. And if you
Speaker 4: just apply one brain cell. Another thing you can do
Speaker 4: is you can just take these cameras and put them
Speaker 4: behind the windshield and then you don't ever have to
Speaker 4: do this dumb shit ever again. And then you can
Speaker 4: use the cabin features like the windshield wiper and the
Speaker 4: defrost settings that are already there and you don't have
Speaker 4: to build anything new and it only will cost you
Speaker 4: like a couple hundred dollars and that's it. So that's
Speaker 4: the kind of dumb we're dealing with. So this is
Speaker 4: why I needed to do this setup for it before
Speaker 4: I get into the more complicated stuff, because I need
Speaker 4: a level set on what kind of level of dumb
Speaker 4: we're dealing with. So I'm gonna skip this. This is
Speaker 4: called a distribution shift. I don't want to get too technical,
Speaker 4: but this is basically saying some stuff about time and
Speaker 4: how it's relevant. I want to point this company out
Speaker 4: just briefly. I'm not going to go deep into it,
Speaker 4: but I just want to just so that people don't
Speaker 4: think I'm just promoting Tesla whatever, and that's it. This
Speaker 4: is another company out of the UK. This is the
Speaker 4: company's name is wave Wayve and this is run by
Speaker 4: a guy named Alex Kendall, and they're trying to do
Speaker 4: a monocular based system. So a monocular based system is
Speaker 4: one camera and they're using like other things like HD
Speaker 4: maps and probably a couple other things. But the point
Speaker 4: is their system is one camera and that's all they're using,
Speaker 4: and this is based out of the UK, and they're
Speaker 4: trying to build it up. I don't think they're fully
Speaker 4: deployed yet, or at least not that I'm aware of.
Speaker 4: But what is important to understand here is that even
Speaker 4: with one camera, automated vehicles can be achieved. And they've
Speaker 4: done a really good job with this, Like I'm skipping
Speaker 4: a lot here, but like it's actually impressive what they're
Speaker 4: able to do with one camera and how money abstract
Speaker 4: ways that they've been able to improve on the system
Speaker 4: that are unconventional. And so the important aspect to understand
Speaker 4: here is that again, the camera's one hundred dollars and
Speaker 4: this camera based, this singular camera based system is still
Speaker 4: capable of doing automated vehicles, and Waimo's spending one hundred
Speaker 4: and twenty thousand dollars one hundred and thirty thousand dollars
Speaker 4: per vehicle to not really achieve that so it's one
Speaker 4: thing for me to say that Waymo's messing up and stuff,
Speaker 4: but it's another thing to act. Showed the hardcore data
Speaker 4: on it. So I just want to bring this up
Speaker 4: real quick. Well where to go there it is. This
Speaker 4: is from the NITSA dot gov website. So this is
Speaker 4: official crash reporting data. This is a standing general Order
Speaker 4: on crash reporting. You're required to report and this is
Speaker 4: NITSA dot gov if you can see at the top.
Speaker 4: And this is national reporting. So any automated vehicle that
Speaker 4: is operating on public roads you're required to report if
Speaker 4: it's involved in the collision. And what this is again
Speaker 4: national reporting. So I want to iterate again. WEAIMO only
Speaker 4: operates in geofen spaces within specific cities in the country,
Speaker 4: and that's it. WEAIMO. This is this data is being
Speaker 4: reported on. The last reported on November seventeen, twenty twenty five.
Speaker 4: This is continuously updated every couple months, every few months.
Speaker 4: I've been following it since twenty twenty three, and WEIMO
Speaker 4: has about two thousand vehicles on the road. And you
Speaker 4: can see here it's kind of tiny. Here, I'll read
Speaker 4: it for you. It's one thousand four hundred and twenty
Speaker 4: six collisions. So in their fleet of two thousand vehicles
Speaker 4: across only a few different cities, they're reporting one thousand,
Speaker 4: four hundred and twenty six collisions. Now, another thing to
Speaker 4: know about this, Weimo recently expanded to a couple new cities,
Speaker 4: so that fleet of two thousand vehicles is recently like
Speaker 4: increased recently. In May of this year, I saw they
Speaker 4: had like twelve hundred vehicles on the road, fourteen hundred vehicles,
Speaker 4: something like this, and they were reporting roughly eighty percent
Speaker 4: of their fleet has already been involved in a collision.
Speaker 4: And I was I've been watching this chart since twenty
Speaker 4: twenty three, and on average it's eighty percent and sometimes
Speaker 4: I've even seen more than one hundred percent. In other words,
Speaker 4: on average more than one collision per vehicle in their
Speaker 4: fleet on Waimo. Now, let's compare this is Tesla's collisions.
Speaker 4: Tesla has I think five million vehicles on the road
Speaker 4: with the eight ass system, millions and millions of vehicles,
Speaker 4: and also around the world, not just in the United States,
Speaker 4: and they're reporting two thousand I think that says eight
Speaker 4: hundred and forty five collisions with millions of vehicles on
Speaker 4: the road, so pretty large, massive magnitude of difference between
Speaker 4: the two. The collision rate with Waimo's is pretty insane.
Speaker 4: I mean, any other vehicle company that you've ever heard
Speaker 4: of having a sixty eighty hundred percent collision rate and
Speaker 4: still operating after all these years, I've never heard of one,
Speaker 4: but Waimo seems to be able to get away with it.
Speaker 4: And so one of the other issues.
Speaker 3: Here, well, one thing I did want to ask you
Speaker 3: it might be a little too conspiratorial. I was wondering
Speaker 3: you think the LIGHTO is maybe the actually map the area,
Speaker 3: Like it's an excuse, that's just one of the excuses
Speaker 3: it's there.
Speaker 4: There is actually that excuse it is made, and that
Speaker 4: is true.
Speaker 3: Like I wonder if it's actually like collecting data and
Speaker 3: you're thinking like, oh, it's on top of the car
Speaker 3: to help get it somewhere.
Speaker 4: It's too Actually, yeah, so this is actually true. So
Speaker 4: you're you're making a good point. There is actually a
Speaker 4: temporary use for light ar in automated vehicles, and this
Speaker 4: is known so one of them and I will get
Speaker 4: into it a little bit later. But one of the
Speaker 4: major issues is calculating depth This is pretty much one
Speaker 4: of the big problems that needed to be solved in
Speaker 4: AVS is depth calculation. And one of the things that
Speaker 4: they learned is that no matter what your system is,
Speaker 4: even if you're using a computer vision only or into
Speaker 4: system like Tesla, one of the things that they learned
Speaker 4: is that you can temporarily put a light our system
Speaker 4: on the vehicle along with the vision and as you're
Speaker 4: training the model, and you can map the area to
Speaker 4: measure depth calculations around you, and you can use that
Speaker 4: to help train the vision system on how to train depth.
Speaker 4: So but then you would still take the light our
Speaker 4: system off after that. The only reason you would have
Speaker 4: it on is temporarily to get depth calculation to get
Speaker 4: the system kickstarted into training, but then you would remove
Speaker 4: it later on and it would be able to calculate
Speaker 4: depth off of the vision system alone. Now even this
Speaker 4: is not required anymore, but there was. This is an
Speaker 4: argument that has made this is sometimes done and this
Speaker 4: is supposed to be temporary, but this is not what
Speaker 4: Wemo's doing. Weymo's making it, and they've multiple times set
Speaker 4: this in public ands including their executives. They're all saying
Speaker 4: that light oar is here to stay for them, meaning
Speaker 4: that they're trying to make light are a fundamental element
Speaker 4: of their solution. The fund, I should say, the fundamental
Speaker 4: element of their solution, which is nonsense, complete nonsense.
Speaker 1: The vital solution for the Silicon Valley nazis.
Speaker 4: He well, well there's stuff there. It's important to understand
Speaker 4: that Weimo is owned by Google and there are numerous
Speaker 4: incentives for doing this. What I had started off by
Speaker 4: saying was that when Weimo came to Phoenix, they went
Speaker 4: to ASU and they started dumping enormous amounts of money
Speaker 4: into the engineering department at ASU. I didn't know that
Speaker 4: at the time, but that's what ended up getting me
Speaker 4: basically in a lot of problems. And so when I
Speaker 4: was applying to my PhD at ASU, I was using
Speaker 4: this presentation. I presented this to all the lab directors
Speaker 4: at ASU and the Perception and Robotics group, which is
Speaker 4: a five floor building filled with PhDs, and basically none
Speaker 4: of them are working on computer vision. They had one
Speaker 4: robot arm with a single camera and that was like
Speaker 4: their computer vision product. Everything else was light our or
Speaker 4: light our related technologies that they were working on. And
Speaker 4: in fact I had to teach my professors all this
Speaker 4: stuff that I'm talking about. They had no clue about
Speaker 4: computer vision and even some of the basic stuff. At
Speaker 4: a certain point it became completely pointless for MEDI even
Speaker 4: b at ASU, because not the professors, every single person
Speaker 4: in the perception of robotics group, not a single person
Speaker 4: at that time had a clue about anything I'm saying here.
Speaker 4: And it's how do I say this? These problems are
Speaker 4: too dumb for somebody that has a PhD and not
Speaker 4: know about in AI and let alone an entire university
Speaker 4: filled with PhD is not knowing a single clue about
Speaker 4: any of this. That's corruption, that's not a mistake. So
Speaker 4: what I was so earlier, I was saying I wanted
Speaker 4: to do this setup so you can understand some of
Speaker 4: the issues that come with this. Now I'll show you
Speaker 4: what Waimo's light our based system actually sees. So here's
Speaker 4: a visual demonstration of what I'm talking about. So this
Speaker 4: is a point cloud representation of what the Weimo vehicle sees.
Speaker 4: You have the literal cameras on top, and then below
Speaker 4: it is what the light our system is seeing, creating
Speaker 4: a point cloud return representation. This is what I'm showing
Speaker 4: on the bottom is what it means to have a
Speaker 4: point cloud return representation. And like you were saying earlier,
Speaker 4: it's just one dimensional point of light. It's not even
Speaker 4: really the wavelength. It's just that one point and that's it.
Speaker 4: Let me play it again. So this is like a
Speaker 4: person walking across with a box. You can see the
Speaker 4: point cloud representation here again, these people move in this stuff.
Speaker 4: You can see it kind of misses the second person,
Speaker 4: it looks like, but it's very low quality, right, Like
Speaker 4: you would never trade the top for the bottom, right though,
Speaker 4: But somehow they've convinced you that this is exactly what
Speaker 4: you should do. One more time, just to iterate the point.
Speaker 4: This is what a point cloud representation means. Now remember
Speaker 4: this video because I'm going to show you what you
Speaker 4: can do with computer vision, and anybody that has understands
Speaker 4: video games and how video games work and all that stuff,
Speaker 4: you're about to have your mind blown because it's exactly
Speaker 4: the same thing. So whatever you can do in video games,
Speaker 4: that's what you can do with computer vision, and you
Speaker 4: cannot do that dumb shit with light art. Okay, this
Speaker 4: is in my opinion, the most important slide, and this
Speaker 4: is the slide that got me in trouble. I know
Speaker 4: there's a lot on here. I'm going to break it down.
Speaker 4: It's not that hard, but this is really worth knowing.
Speaker 4: And in my opinion, like I sh, I want to
Speaker 4: mention that there's a lot of things that I'm not saying.
Speaker 4: Some of it is to condense, but a lot of
Speaker 4: it is also to avoid creating even more problems for myself.
Speaker 4: I'm happy to talk about things if you want to,
Speaker 4: but I'm gonna skip some of the really bad stuff
Speaker 4: that could get me in a lot of trouble. But
Speaker 4: this is this is good. This got me in enough
Speaker 4: trouble what I have here on the screen, This got
Speaker 4: me in enough trouble. But I already did this, so
Speaker 4: I'll go ahead and talk about it. So this is
Speaker 4: the state. So if you ever see the phrase soda
Speaker 4: Sota state of the art, soda state of the art,
Speaker 4: So this is the state of the art. Way MOO
Speaker 4: leaderboard results. On June twenty ninth, twenty twenty three WHIPS.
Speaker 4: This was presented at CVPR twenty twenty three, the Major
Speaker 4: Computer Vision Conference by Chen Wu who's for WAMEL. I
Speaker 4: have the link here and this is the paper that
Speaker 4: is associated with that presentation on the left side. So
Speaker 4: this is the reference paper that I'm talking about on
Speaker 4: the left side. Here on the left side you have
Speaker 4: this table. This table represents what is industry standard for
Speaker 4: object classification around the world. Everyone in the world that
Speaker 4: is doing object identification and classification in AI, they all
Speaker 4: use MIOU mean intersection over union in this task. This
Speaker 4: is industry standard, not just in America, everywhere in the world.
Speaker 4: This is well understood, well known. This is standard. What
Speaker 4: MIOU is that there is a minimum. It's at the
Speaker 4: time it was a minimum sixteen classes, but actually the
Speaker 4: numbers in significantly since then. I think in this case
Speaker 4: WEIMO was doing twenty two which at the time in
Speaker 4: twenty twenty three was acceptable to do twenty two classes.
Speaker 4: Since then this number has increased to even more classes.
Speaker 4: But just know at the time this is actually industry
Speaker 4: standard and accepted. You're going to have two sets of
Speaker 4: data here, validation set and the test set. The validation
Speaker 4: set is the performance of the model on the training data,
Speaker 4: and then the test set, which is the real measure
Speaker 4: of performance. The test set is on data after the
Speaker 4: model has been trained and is being tested on scenarios
Speaker 4: that it hasn't ever seen before. So on average, you
Speaker 4: would expect that the validation set would be or I mean,
Speaker 4: the test set would be lower performance than the validation set,
Speaker 4: So that's supposed to be it. So it's all these
Speaker 4: different classes. They measure the performance on each one and
Speaker 4: then they do an intersection over union on them, and
Speaker 4: then they average it, and that's what this is table
Speaker 4: is saying. So what you're getting here is forty six
Speaker 4: point eighty two percent accuracy on MIOU from WEIMO at
Speaker 4: this one and they presented us this table was on
Speaker 4: the last page of their paper here and it was
Speaker 4: not referenced in the paper, and this table was hidden
Speaker 4: at the very end, not talked about. Everything else on
Speaker 4: this slide is what weimo's actually doing. This table is
Speaker 4: industry standard. Everything else is not. This is what weimo's doing,
Speaker 4: So here we go. What you need to know is this.
Speaker 4: Look at this line that says swformer dot threef hours,
Speaker 4: referring to WEIMO, and then also down here s w
Speaker 4: former threef hours. These two lines are the lines you
Speaker 4: need to look at. What you'll see is they're showing
Speaker 4: the performance using this metric apaph on pedestrians, for example,
Speaker 4: Whatever the hell this apaph metric is not miou and
Speaker 4: I have never seen this metric before, so I had
Speaker 4: to go find it. What the hell is this apaaph
Speaker 4: thing they're showing on this whatever this metric is apaph,
Speaker 4: whatever the hell that means? This thing is showing an
Speaker 4: eighty two point nine percent accuracy rate up here or
Speaker 4: maybe down here, like eighty two point one three on
Speaker 4: the pedestrian L one whatever ap me and aph. So
Speaker 4: I went and looked for where the hell did this
Speaker 4: come from? And that's what's on the right. And it
Speaker 4: turns out this metric apaph it was also made by
Speaker 4: Waimo in a paper from twenty nineteen, which I'm showing
Speaker 4: here on the right side. And what this effectively does
Speaker 4: this is the formula for it on the right side. Whoops.
Speaker 4: What this does is it takes this data on the left,
Speaker 4: runs it through this magic formula on the right, and
Speaker 4: outputs these results. That's literally what it does. I'm not exaggerating.
Speaker 4: That's what it does. It takes this data, runs it
Speaker 4: through that formula, manufactures this data and as you can see,
Speaker 4: the performance that they're showing off of this metric, which
Speaker 4: is what they're using for their safety reports and what
Speaker 4: they published in their safety data in twenty twenty three
Speaker 4: and twenty twenty four. This data here is almost double
Speaker 4: the performance of the industry standard, and they totally ignored
Speaker 4: the industry standard forty six percent versus eighty two point
Speaker 4: nine percent or whatever on a metric that they invented.
Speaker 4: So let me make it clear here. WEIMO invented the
Speaker 4: metric that nobody in the world uses, and then they
Speaker 4: use that metric for their safety data, and that metric
Speaker 4: that they invented is showing a performance that is nearly
Speaker 4: double the actual performance that is known as industry standard
Speaker 4: around the world. So this is important to know. Now,
Speaker 4: one other thing that I was saying earlier and the
Speaker 4: previous side was the degradation of lid or over range.
Speaker 4: So what you're getting these point cloud returns. One of
Speaker 4: the things that happens is that the number of points
Speaker 4: that you get from the light our system the farther
Speaker 4: you go out is lower. That's why the quality goes down.
Speaker 4: So when I was saying fifty meters one hundred meters,
Speaker 4: one hundred and fifty meters, the argoverse results were showing
Speaker 4: degradation at a log two rate half every fifty meters.
Speaker 4: This this thing that I just described, this is called sparsity.
Speaker 4: In AI terms, this is referred to as sparsity. How
Speaker 4: sparse are the points? You want to have higher resolution data?
Speaker 4: More points? Well, these results were showing degradation at range,
Speaker 4: and I was also stating that it seems like research
Speaker 4: on this specific topic about degradation over range seems to
Speaker 4: be heavily suppressed because I just can't find anything anyone
Speaker 4: really doing any serious research on it. And it's not,
Speaker 4: in my opinion, it's impossible that nobody thought about that.
Speaker 4: For how prevalent and common, and how many businesses, major
Speaker 4: multi billion dollar businesses have been using lightar and failed,
Speaker 4: the graveyard of businesses that have tried to make lightar
Speaker 4: work and failed, there is no possible way that nobody
Speaker 4: thought about degradation of lightar over range. I promise you
Speaker 4: this is like from an engineering perspective, you have to
Speaker 4: be completely idiot. You have to be a complete idiot
Speaker 4: to not think about that. This is like even in cameras,
Speaker 4: you think about like video games, you know about like
Speaker 4: how you have lower quality pixels. Further down you go
Speaker 4: and in order to say, remember, you have to understand
Speaker 4: this stuff. It's like at an elementary level. There's no
Speaker 4: possible way people didn't think about that. So why is
Speaker 4: that important? I want to read this thing that came
Speaker 4: from the paper. Some of the latest commercial light ours
Speaker 4: can sense up to two hundred and fifty and three
Speaker 4: hundred meters in all directions around the vehicle, leading to
Speaker 4: a large range of point clouds. Okay, so they're fully
Speaker 4: aware of point how point clouds work, and they're talking
Speaker 4: about how long range lightar can go up to this range.
Speaker 4: But notice they don't tell you how sparse it is
Speaker 4: at that point. So let's go to this bottom right here,
Speaker 4: I'm going to read this. Our experiments are primarily based
Speaker 4: on the challenging WEAIMO open data set. Okay, One thing
Speaker 4: to know about this WEIMO open data set. At the
Speaker 4: time that I published this, there were a total which
Speaker 4: is November twenty twenty three, at the time I was
Speaker 4: publishing this, which is four or five months after CVPR.
Speaker 4: This CVPR, they had a total of six submissions to
Speaker 4: the WEIMO open data set. By comparison, in Nvidia at
Speaker 4: on their data set had more than four hundred submissions
Speaker 4: and I'm bringing that up because the next line they
Speaker 4: say is which has been adopted in many recent state
Speaker 4: of the art three D detection methods. As far as
Speaker 4: I'm aware, basically nobody in the world uses the Waymo
Speaker 4: open data set other than weymo, and even the submissions,
Speaker 4: the six submissions that I saw in November twenty twenty three,
Speaker 4: even those submissions were from nobody's I've never heard of
Speaker 4: any of them. But on the in Vidia data set,
Speaker 4: with more than four hundred, every major computer vision every
Speaker 4: major university that competes in computer vision has competed there,
Speaker 4: including in Vidia and all other major computer vision participants,
Speaker 4: it's like the standard. Effectively, the data set contains one
Speaker 4: hundred and fifty seven split into seven hundred and ninety
Speaker 4: eight training, two hundred two validation one hundred fIF Each
Speaker 4: scene has about two hundred frames, where each frame captures
Speaker 4: the full three hundred and sixty degrees around the EGO vehicle.
Speaker 4: The data set has one long range lighter with the
Speaker 4: range capped at seventy five meters, four near range lightar,
Speaker 4: and five cameras, so it doesn't matter that the long
Speaker 4: range lighter can since up to two hundred and fifty
Speaker 4: to three hundred meters. They're capping it at seventy five.
Speaker 4: And this helps support what I was saying earlier and
Speaker 4: what Argoverse results were showing at the same timeframe as this,
Speaker 4: that lighter degrades over range. And notice the seventy five
Speaker 4: meter cap just a little beyond the fifty meter degradation.
Speaker 1: How much you would have bet that that seventy five
Speaker 1: meter cap that they're giving it is probably a way
Speaker 1: over estimation as well.
Speaker 4: Yeah, I'm pretty sure. Yeah, you're going to see some stuff.
Speaker 4: I'm just being a little I'm not trying to be.
Speaker 1: Yeah, they're overestimated. Even in that seventy five meters. They
Speaker 1: would have gone higher if they could have.
Speaker 4: Yeah, they would have gone high there. I mean, the
Speaker 4: seventy five meters is sixty miles an hour, gives you
Speaker 4: less than three seconds of forecasting time, you know, So
Speaker 4: that's still not much better, you know. So let me
Speaker 4: talk a little bit. I mean, I'll skip all this
Speaker 4: technical stuff, but if anyone cares and wants to know
Speaker 4: about the architecture and how it really works for the
Speaker 4: camera based system, this is actually a really useful slide.
Speaker 4: It pretty much lays out the actual architecture if anyone
Speaker 4: actually cares. But I'm going to try and skip all
Speaker 4: the really technical stuff. I do want to just mention
Speaker 4: something from this paper, which was a big deal. This
Speaker 4: is called one Former. This was a big deal in
Speaker 4: computer vision and all this stuff. The element here that
Speaker 4: I'm just going to briefly is that, as you would expect,
Speaker 4: it turns out that vision also improves large language models semantics,
Speaker 4: and it's actually a two way road. They kind of
Speaker 4: go hand in hand. But the point is that vision
Speaker 4: is an augment to an LLM. So another way of
Speaker 4: saying that is being able to see a red wagon
Speaker 4: allows you to be able to describe a red wagon
Speaker 4: in language better. And this one form is kind of
Speaker 4: showing all that. Let me skip some of this stuff.
Speaker 4: This is technical stuff. It's kind of cool, shows some
Speaker 4: really interesting things, but I kind of want to get
Speaker 4: away from it. Segment anything for anyone in computer vision
Speaker 4: is totally worth knowing. This is a This was made
Speaker 4: by Meta and when it came out, it was a
Speaker 4: big deal because it did it did object classification pretty
Speaker 4: robustly across a wide variety of topics, not just avs,
Speaker 4: like not just like useful in the av industry, but
Speaker 4: useful across a variety of tasks, and it was a
Speaker 4: big deal. So this is where we'll get into the
Speaker 4: argument for depth and why light oars were adopted. So again,
Speaker 4: the primary and significant function that lightar provides avs is
Speaker 4: a depth calculation. And the reason why this is completely
Speaker 4: useless because number one, there's no reason to have super
Speaker 4: precise death depth. There's no reason to have laser precision
Speaker 4: on depth. The argument went like this, the major problem
Speaker 4: in avs is collision, and because lightar does accurate depth perception,
Speaker 4: highly accurate, this is how you have collision. This was
Speaker 4: the crux and fundamental purpose of the argument for lidar.
Speaker 4: But if you think about your human eyes and how
Speaker 4: you do it in real life, you're not trying to
Speaker 4: calculate the distance between you and an object in front
Speaker 4: of you. The only thing you care about when you
Speaker 4: in your everyday real life is you care about this
Speaker 4: object is in front of that object, but behind that object.
Speaker 4: This is called relative distance. So this is how the
Speaker 4: brain does it, and this is actually the better way
Speaker 4: of doing it. It's way more efficient. So one of
Speaker 4: the reasons why we have binocular vision two eyes is
Speaker 4: because to having two eyes aka two cameras is how
Speaker 4: our brain does depth perception on a relative basis. When
Speaker 4: you have two fixed points that you're using as a
Speaker 4: camera with known dimensions, you can create triangles off of this,
Speaker 4: and then everything becomes geometry and that's it. You don't
Speaker 4: you can even make up then like the actual real
Speaker 4: precise numbers don't even matter. You can be as long
Speaker 4: as they're relatively correct. It doesn't actually matter what the
Speaker 4: numbers are. You can even make up numbers as you go.
Speaker 4: The only thing that matters is relative to death.
Speaker 1: Now, insects, look at the eyes on insects. It's just
Speaker 1: massive eyes, just over and over again, just so that
Speaker 1: it has perfect you know, it's it's got depth perception
Speaker 1: and everything else. Yeah, all all you know, wired together
Speaker 1: with those eyes.
Speaker 4: And you'll even notice that the eyes on say a fly,
Speaker 4: for example, are like they look complicated, but they're actually
Speaker 4: just repeating the same thing over and over again, which
Speaker 4: one fundamental concept and what they care about. A fly
Speaker 4: cares about the most is motion detection. And that's why
Speaker 4: the eyes bulge out and they can see behind them
Speaker 4: and all around. That's why the globe shape and they're
Speaker 4: doing object detection sorry, motion motion de and that's the
Speaker 4: fundamental reason why it is. And you can even get
Speaker 4: hints of evolutionary purpose evolutionary progress. Sorry, you can get
Speaker 4: hints of evolutionary progress just by looking at the eye
Speaker 4: throughout all the species. The more evolved the eye, probably
Speaker 4: the more intelligent the species. If all the eye that
Speaker 4: they could this is some animal or or whatever it
Speaker 4: can do, is look do objects or motion detection, then
Speaker 4: it's probably a primitive eye that probably has gray cylinders
Speaker 4: or something equivalent, right, gray scale. But the more evolved
Speaker 4: the eye is, with more color perception and more quality,
Speaker 4: then this is probably a more intelligent species. And one
Speaker 4: of the things that I kind of I'm not going
Speaker 4: to hit too hard on this, but one of the
Speaker 4: things I want to demonstrate is that consciousness is a gradient.
Speaker 4: One of the things is that there's not a real
Speaker 4: definition of consciousness. And why is it difficult to measure?
Speaker 4: It's because consciousness is a gradient. It's not one. It's
Speaker 4: not like you just suddenly have a certain level that
Speaker 4: you hit and then all of a sudden you're conscious.
Speaker 4: Nonsciousness is a gradient, and it goes from simple to complex,
Speaker 4: and it can get even more complex than we even are.
Speaker 4: So I want to get to the real fun stuff.
Speaker 4: So I was just talking about depth perception. So and
Speaker 4: earlier I had mentioned that bounding boxes are a cheat. Well,
Speaker 4: this is the solution to that. To those problems, depth
Speaker 4: perception and bounding boxes are both solved by this. So
Speaker 4: in the AV industry it has been completely adopted, including Waimo,
Speaker 4: every single major company or philosophy. It doesn't matter what
Speaker 4: your belief system is. Every participant in the AV industry
Speaker 4: has adopted this thing called an occupancy network. And this
Speaker 4: is one of two AI cons that I'm going to
Speaker 4: be demonstrating. This is when earlier I was saying I
Speaker 4: was hoping to teach people what real AI is and
Speaker 4: distinguish that from fake AI. This is the one of
Speaker 4: the two things you need to know in order to
Speaker 4: understand what is real AI and what is not. An
Speaker 4: occupancy network is one, and the next thing I'll talk
Speaker 4: about is something called nerve. These two concepts you must
Speaker 4: know if you want to understand anything about AI going forward.
Speaker 4: And this is what distinguishes real AI from all the
Speaker 4: other junk. So an occupancy network, what it is is
Speaker 4: it's a grid pattern that is made around the vehicle itself,
Speaker 4: and this grid pattern is filled with these Minecraft looking
Speaker 4: boxes all around it. And these Minecraft looking boxes, these
Speaker 4: boxes are called voxels, and these voxels are made are
Speaker 4: heat mapped all around the vehicle. So it's basically a grid.
Speaker 4: And then you put objects on this grid and it
Speaker 4: doesn't matter the exact precise distance as long as you
Speaker 4: get it pixel correct. Whatever your pixel resolution is will
Speaker 4: be the resolution of your occupancy network and these voxels.
Speaker 4: What is really simple and powerful about an occupancy network
Speaker 4: and why it was adopted by the industry is because
Speaker 4: it gives you a simple mechanism, a zero and one
Speaker 4: binary mechanism for drive and not drive. So you can
Speaker 4: see here in blue, this blue area is where you
Speaker 4: can drive, and then everything else is no drive, don't drive.
Speaker 1: If you're gonna if you're gonna relate this to a sense,
Speaker 1: it would have to be touch, right, because each and
Speaker 1: every one of these things have different bumps and different layers,
Speaker 1: so that as you're reaching out, you're touching these boxes,
Speaker 1: which is a generalized form, and you're getting the feel
Speaker 1: for it how far or close it is. So it's
Speaker 1: a very tactile sais with this form right here.
Speaker 4: Good, So you're absolutely right, and so eventually I'm going
Speaker 4: to show the evolution of this. So this is way.
Speaker 4: This is an old version. I know that says twenty
Speaker 4: twenty three, but this is actually an old version of
Speaker 4: Tesla's occupancy network. I think it is. Actually this is
Speaker 4: the twenty twenty version, if I recall correctly. So this
Speaker 4: is an old version, and I'm just showing this to
Speaker 4: show you the evolution of what happened in the industry
Speaker 4: the occupancy because this is fundamentally important an occupancy network,
Speaker 4: and you're right. You are going to get the texture
Speaker 4: quality as a result of this, and there's a few
Speaker 4: there's actually a number of other benefits. But first and foremost,
Speaker 4: the thing to understand here is that the reason why
Speaker 4: this was adopted into the av industry. No matter your philosophy,
Speaker 4: whether you're using lidar or not, it doesn't matter, you're
Speaker 4: still going to be doing this is because it provided
Speaker 4: an automatic implicit collision avoidance system. This through software alone,
Speaker 4: this occupancy network automatically created an implicit collision avoidance system,
Speaker 4: and that alone was what was a lot enabled it
Speaker 4: which made it attractive to put into avs. Now, there
Speaker 4: are many other benefits that I'm going to talk about
Speaker 4: that ended up happening, and this evolved even further. But
Speaker 4: important to understand here that basically, think of Minecraft. You
Speaker 4: have a zero one mechanism for where you can drive
Speaker 4: and where not to drive. And in the you know,
Speaker 4: in the zero part, the part you don't drive these
Speaker 4: little boxes. What they've done is they've added in a
Speaker 4: bunch of little data points, so they add further data
Speaker 4: into the not drive part. And then the drive part
Speaker 4: is basically the road. That's it, just the road, and
Speaker 4: that's simple, binary and super powerful. And it turns out
Speaker 4: and this is this is the solution to the bounding problem.
Speaker 4: And I'm going to get into that. So the bounding
Speaker 4: box problem I was saying was a was a cheat.
Speaker 4: Oh shit, I'm sorry, I need to talk about nerves first.
Speaker 4: Pause on the bounding box problem. I need to talk
Speaker 4: about this first. I got a mixed up. So the
Speaker 4: I was saying, there are two things that we need
Speaker 4: to know in real AI. The occupancy network is one,
Speaker 4: and then the second thing you need to know. And
Speaker 4: this is as technical as I'm going to get there
Speaker 4: is nerves now NERF, n ERF, NERF. This thing was invented.
Speaker 4: This is a technology that was invented in twenty nineteen,
Speaker 4: I think or twenty eighteen, twenty nineteen, and this stands
Speaker 4: for neural radiance field. And now, if you play video games,
Speaker 4: you're going to have an advantage in understanding all this because
Speaker 4: a neural radiance field is very similar in concept to
Speaker 4: raytracing in a video game. And so what a radiance
Speaker 4: field is is that it's the ray from a pixel
Speaker 4: on a frame. It's the ray vector that is pretty
Speaker 4: much identical to what ray tracing is in video games,
Speaker 4: and it's each pixel doing that ray tracing. And that's
Speaker 4: what a NERF is. That's originally what a NERF was.
Speaker 4: So NERF started out in twenty nineteen as a new technology,
Speaker 4: and what happened is this thing that's called a radiance field,
Speaker 4: this NERF thing, it evolved into a whole branch of technologies. Now,
Speaker 4: this whole branch of technologies in AI is just colloquially
Speaker 4: collectively referred to as NERF. But they haven't since then.
Speaker 4: Since then, they have made like thousands and thousands of
Speaker 4: different evolutions of this thing. Called nerve. Now, earlier I
Speaker 4: was showing the occupancy network is these like Minecraft looking
Speaker 4: boxes fossils. What happened oh time is that those voxels
Speaker 4: reduced in size over time and eventually became single points.
Speaker 4: And what you were able to do with that is
Speaker 4: you were able to make an occupancy network and the
Speaker 4: NERF effectively the same thing. So occupancy networks and nurse
Speaker 4: initially started out as two independent, completely different things, and
Speaker 4: over time they came together and became basically the same thing.
Speaker 4: So another way of understanding this is that ray tracing
Speaker 4: in video games and occupancy networks in aves basically the
Speaker 4: same thing. And so this is like if you play
Speaker 4: video games, that should be nuts because that means that
Speaker 4: everything that is happening in vehicles in robotics is a
Speaker 4: video game. It is like literally it is that. It
Speaker 4: is not like that, it is that equal to that.
Speaker 4: So these two things became the same thing. This is
Speaker 4: why I'm saying you need to know these two things,
Speaker 4: not that you have to understand everything about it. You
Speaker 4: just need to know what this is because this is
Speaker 4: real AI and you're gonna see what happens with this.
Speaker 4: So I'll even see that word. So NERF is a
Speaker 4: new concept, I'm sure to people. So I'm just gonna
Speaker 4: show like a few examples so that people get an
Speaker 4: understanding of what it is. So initially NERF. So initially
Speaker 4: NERF was slow, you know, like all new things. But
Speaker 4: then eventually there was this paper that was called plinoxiles
Speaker 4: and they figured out how to do it way faster,
Speaker 4: especially with GPUs. And you'll see in this video here
Speaker 4: the difference. So the left one is like what it
Speaker 4: was originally trying to do. This is what NERF is
Speaker 4: originally trying to do. It's trying to render the object.
Speaker 4: But you can see that it was like slow and
Speaker 4: has some problems and has some difficulty. But then of
Speaker 4: course they basically sped this thing up like really fast.
Speaker 4: So what I'm going to eventually build up to is
Speaker 4: can you do this in real time? Yes you can,
Speaker 4: And so I'm going to build up to, Yes, you
Speaker 4: can render NERVE aka ocupency network in real time. If
Speaker 4: you're an AI person, this matters to you because if
Speaker 4: you're an av navigating through the world, it's one thing
Speaker 4: to be able to build of the environment around you,
Speaker 4: but can you do it in real time? Which is
Speaker 4: an even harder right, you have to do it fast.
Speaker 4: So can you do radiance fields aka ocupency networks in
Speaker 4: real time? The answer is going to be yes, and
Speaker 4: I'm going to demonstrate that these are the results if
Speaker 4: you need to care about this. So let me show
Speaker 4: you one step of evolution. This is not the modern
Speaker 4: version of the occupens the network in the tessel system.
Speaker 4: This is like what twenty twenty two, So you can
Speaker 4: see just in a couple of years how the evolution
Speaker 4: is lower resolution I mean sorry, higher resolution voxels, smaller vauxels,
Speaker 4: and what you're getting here. You can see that this
Speaker 4: is still a zero one mechanism, just like I was
Speaker 4: saying before for drive, no drive. But look at this.
Speaker 4: So these are the cameras on top, the literal cameras
Speaker 4: on top, and then this is what the car sees.
Speaker 4: So this is the evolution of the occupancy network over
Speaker 4: probably a couple of years. So a little bit higher
Speaker 4: resolution you're seeing, and you can see the car shapes
Speaker 4: are starting to become more defined. And remember how I
Speaker 4: was saying, uh, bounding boxes are a cheat. These little
Speaker 4: shapes right here, this is how you get around the
Speaker 4: bounding box problem. Once you apply them to a grid
Speaker 4: and you make the voxels as small as possible to
Speaker 4: individual points. Then the task becomes getting the shapes pixel perfect,
Speaker 4: and you have a pixel perfect shape through an occupancy
Speaker 4: network for free. And still so many people are using
Speaker 4: bounding boxes, which is a total cheat.
Speaker 1: This reminds me of Blender.
Speaker 4: Blender, Oh Blender, yeah, right.
Speaker 1: The three D software, Like you could just plug just
Speaker 1: straight into there and have a total model of the
Speaker 1: entire place, and then I guess share it between different vehicles.
Speaker 1: Isn't that what's also happening as well, Like there's a
Speaker 1: sharing between different vehicles with all these.
Speaker 4: I think that, I mean, this is that's more theoretical.
Speaker 4: How do I say it's it's not commercially available like
Speaker 4: Tesla's don't do it yet, but this is part of
Speaker 4: the plan that they're working on. Yeah, and yeah, that
Speaker 4: that you can assume that that'll be obvious. It's yeah.
Speaker 4: I would say that in terms of priorities, like what
Speaker 4: I'm talking about here, making this perfect, getting it exactly
Speaker 4: right becomes is a way higher priority for everyone in
Speaker 4: the world, And that's what they're This is where we're at.
Speaker 4: We're still in the vacuum tube stage of AI. Every
Speaker 4: once acond we're like, no, no, no, we're not. And I'm
Speaker 4: trying to show some of the evolution in the progress
Speaker 4: because this is the state of the art stuff. And
Speaker 4: if you understand video games, you understand that we're still right,
Speaker 4: like this is like not even nineteen nineties graphics yet, right,
Speaker 4: So we're getting there. So, as I was saying before,
Speaker 4: you get object shapes for free, and this is the
Speaker 4: solutions to the bounding box problem. And this was a
Speaker 4: major problem in the industry and still probably is for
Speaker 4: some dumb people. But this is how you solve it.
Speaker 4: And this is a really really important thing, getting object
Speaker 4: shapes for free, Like it's up there with collision avoidance
Speaker 4: in terms of importance one of the and this is
Speaker 4: like kind of important too, especially if you're an AI
Speaker 4: you understand how significant this is. Probably most people won't
Speaker 4: understand how big of a deal this is. But this
Speaker 4: is a one of the other things you get out
Speaker 4: of an ocupency network is that it reasons well within
Speaker 4: about occluded objects, So objects that are traveling behind other
Speaker 4: objects and partially occluded. This system inherently and automatically just
Speaker 4: understands that because it's just a grid network around you.
Speaker 4: Very simple heat map. But again, as I was saying,
Speaker 4: the most important thing about the occupancy network for avs
Speaker 4: is a collision avoidance system. It implicitly has this. Now
Speaker 4: that that is not to say that it will automatically
Speaker 4: go on the correct path. No, it just means it
Speaker 4: will consistently choose a collision avoidant path. So, as you
Speaker 4: can see here, this car went out of control. It
Speaker 4: almost collided in the right corner, but the collision avoid
Speaker 4: the occupancy network avoided it moved over to the left.
Speaker 4: This is clearly the wrong side of the road, but
Speaker 4: it avoided another collision. It stabilized and that's the value originally,
Speaker 4: initially that was the major value, and it was sufficient
Speaker 4: to sell it into the av industry just for this alone,
Speaker 4: because there's a software alone doing this.
Speaker 5: Close your words, looking to the darkness far the blazing
Speaker 5: start focus on it be called the don't feel let
Speaker 5: the should be done.
Speaker 4: And does
Speaker 1: You the shoes