Speaker 1: Welcome to the Sentient Code, where intelligence is engineered, autonomy
Speaker 1: is emerging, and a line between human and machine grows thinner.
Speaker 1: Each episode, we decode the algorithms, explore the robotics, and
Speaker 1: examine the ideas shaping the future of artificial minds.
Speaker 2: I want to start today by asking you to do
Speaker 2: something that feels incredibly simple, almost you know, trivial, but
Speaker 2: it's actually a miracle of biology. Right now, just pause
Speaker 2: for a second and notice exactly what you're doing. You're
Speaker 2: listening to my voice, obviously, but maybe you're also driving,
Speaker 2: so your eyes are scanning the road, watching for break lights.
Speaker 2: You feel the texture of the steering wheel under your hands.
Speaker 2: Maybe you're drinking coffee and you can smell the roast.
Speaker 3: It's the sensory soup. We're swimming in it.
Speaker 2: Exactly, it's a soup. But here's the thing, and I
Speaker 2: really want you to catch this. You aren't toggling between
Speaker 2: these senses like you're switching apps on a phone. No,
Speaker 2: you don't stop hearing to start seeing. You don't pause
Speaker 2: your sense of smell to process the texture of the wheel.
Speaker 2: Your brain is this incredible fluid mixing board. It takes
Speaker 2: audio visual tactile and textual inputs and weaves them into
Speaker 2: this single, seamless narrative we call reality.
Speaker 3: And it's completely effortless for us. I mean, it is
Speaker 3: the defining feature of biological consciousness. So we don't really
Speaker 3: think about modalities, do we. We just think about the world.
Speaker 2: But and this is the big concept we're unpacking today.
Speaker 2: Until very very recently, artificial intelligence was not like that
Speaker 2: at all. In fact, it was the exact opposite.
Speaker 3: Oh, it was completely fragmented. You look at the history
Speaker 3: of AI really from the nineteen fifties up until well
Speaker 3: the early twenty twenties, we were building a fractured mind.
Speaker 3: We had what we call the island problem.
Speaker 2: The island problem. I like that image paid the picture
Speaker 2: for us.
Speaker 3: Okay, so picture an archipelago on one island. You have
Speaker 3: these brilliant computer vision systems. They were specialists. They could
Speaker 3: look at a photo of a cat and tell you
Speaker 3: that's a tabby with like ninety nine percent accuracy. Superhuman
Speaker 3: vision in some respects. Right. But if you showed that
Speaker 3: same system a handwritten note that said this is a cat,
Speaker 3: it was blind. It couldn't read. It had no concept
Speaker 3: of what letters were.
Speaker 2: Okay, so that's island one the eye they cannot read exactly.
Speaker 3: Then on the next island over you have the text spots.
Speaker 3: The ancestors of you know, chatchypt and the like. They
Speaker 3: could write you a sonnet about a cat. They could
Speaker 3: define the biology of a feline. They could translate cat
Speaker 3: into fifty languages. But if you showed them a picture
Speaker 3: of a kitten, nothing, just static. They were effectively brains
Speaker 3: in a jar that only knew the world through symbols.
Speaker 2: So you have the eye that cannot read and the
Speaker 2: brain that cannot see precisely.
Speaker 3: And the worst part, they were built by different people.
Speaker 3: The computer vision engineers didn't hang out with the natural
Speaker 3: language processing engineers.
Speaker 2: They were in different departments.
Speaker 3: They used different math, they used different architectures. They were
Speaker 3: effectively different species of intelligence.
Speaker 2: So for fifty sixty years we were building these savants.
Speaker 2: One savann could see perfect pixels, one savant could pars
Speaker 2: perfect grammar. But they couldn't have a conversation.
Speaker 3: They couldn't even acknowledge each other's existence.
Speaker 2: And today, because the reason we're doing this show is
Speaker 2: that something fundamental changed.
Speaker 3: Today. The bridges have been built the water between the
Speaker 3: Islands is gone. We are witnessing the rise of multimodal
Speaker 3: AI and I want to be really clear to everyone listening,
Speaker 3: this isn't just a feature update. This isn't just now
Speaker 3: your chatbot has a camera icon.
Speaker 2: It feels much much bigger than that.
Speaker 3: It is fundamental. We are moving from the era of
Speaker 3: the specialist to the era of the generalist. We are
Speaker 3: giving machines the ability to integrate senses in a way
Speaker 3: that well, it mimics that human sensory soup we started with.
Speaker 2: That's our mission for this discussion. We've pulled together a
Speaker 2: stack of research, technical papers, and industry analysis to figure
Speaker 2: out how this happened, because it seemed like for decades
Speaker 2: we were stuck and then in the last few years
Speaker 2: everything just collided right.
Speaker 3: We're going to look at the architecture, the actual aha
Speaker 3: moment that let machines see and read. At the same time,
Speaker 3: we'll look at the superpowers this unlocks, like reading X
Speaker 3: rays while reading patient notes.
Speaker 2: And we absolutely have to talk about the limitations because
Speaker 2: the research shows that all these machines can see, they
Speaker 2: hallucinate in brand new.
Speaker 3: Ways they do, and we need to ask the big
Speaker 3: philosophical question if a machine looks at a photo of
Speaker 3: a funeral and writes a palm that makes you cry,
Speaker 3: does it actually understand grief or is it just really
Speaker 3: really good at math.
Speaker 2: Let's get into the mechanics then, Section one. How did
Speaker 2: we get here? Because I remember reading about AI in
Speaker 2: say twenty fifteen, and it was all about these specialized tools.
Speaker 2: You had one tool for chess, one tool for translating French.
Speaker 2: When did the walls come down? Was there like a
Speaker 2: single invention.
Speaker 3: To understand the solution, you really have to understand why
Speaker 3: the walls were there in the first place. And it
Speaker 3: all boils down to the architecture, the literal shape of
Speaker 3: the neural networks.
Speaker 2: Okay, break that down for us, you know, don't go
Speaker 2: too heavy on the jargon, but give us the reality.
Speaker 2: Why couldn't vision bot talk to the text butt?
Speaker 3: Okay? So, for a long time, the king of computer
Speaker 3: vision was something called a CNN, a convolutional neural network.
Speaker 2: We've touched on these before. These are the ones that
Speaker 2: scan an image like a grid, right, looking for edges
Speaker 2: and shapes. Right.
Speaker 3: Imagine a sliding window moving over a picture. It looks
Speaker 3: at a tiny patch of pixel, say a three x
Speaker 3: three square and asks is there an edge?
Speaker 2: Here?
Speaker 3: Is there a curve? Is there a color gradient? It
Speaker 3: builds up from lines to shapes, to ears to eventually
Speaker 3: a cat. It is designed mathematically to process grids of
Speaker 3: spatial data. It understands space.
Speaker 2: Okay, so that's the eye. It deals in grids. It
Speaker 2: thinks and grids exactly.
Speaker 3: But for text, text is in a grid. Text is
Speaker 3: a stream, It's a sequence the quickbat brown dot fox.
Speaker 3: The order matters immensely. Of course, you can't just look
Speaker 3: at brown without knowing quick came before it. So for
Speaker 3: that we used Ours were current neural networks. These were
Speaker 3: designed to remember the past. They process the word fox
Speaker 3: while trying to hold onto the memory of the.
Speaker 2: So you have one kind of math designed for grids,
Speaker 2: which is space, and a completely different kind of math
Speaker 2: designed for streams, which is time.
Speaker 3: You've got it. And you couldn't just plug one into
Speaker 3: the other. They spoke different languages. It was like trying
Speaker 3: to put a VHS tape into a toaster. The inputs
Speaker 3: just didn't match the machinery.
Speaker 2: So what changed? I know the answer involves transformers, because
Speaker 2: that seems to be the answer to everything in AI. Lately.
Speaker 2: But why what did the transformer do that the others couldn't.
Speaker 3: Well, the date was twenty seventeen. The paper was attention
Speaker 3: is all you need. We talk about it all the
Speaker 3: time on this show. But the hidden revolution in that
Speaker 3: paper wasn't just that it was better at language. It
Speaker 3: was that the transformer was a universal substrate.
Speaker 2: Universal substrate, Yeah, that sounds impressive, but what does it
Speaker 2: actually mean? In practice?
Speaker 3: It means it's a structure that can process any kind
Speaker 3: of information as long as you can turn that information
Speaker 3: into a sequence.
Speaker 2: So text is obviously a sequence, word after word after
Speaker 2: word that fits right.
Speaker 3: In AI terms, we call those tokens. But then the
Speaker 3: researchers have this, this real aha moment. They realized, wait
Speaker 3: a minute, we can treat an image as a sequence too.
Speaker 2: Hold on, how do you turn a picture into a sequence.
Speaker 2: A picture is a flat two D object. It doesn't
Speaker 2: have a start and end like a sentence does.
Speaker 3: That was the stroke of genius. The researchers asked, what
Speaker 3: if we forced it to be a sequence?
Speaker 2: Forced it? How?
Speaker 3: Imagine taking a photo of a dog. Now imagine taking
Speaker 3: a pair of scissors and cutting it up into a
Speaker 3: grid of little squares. Let's say sixteen by sixteen pixel squares.
Speaker 3: You have a pile of these tiny patches.
Speaker 2: Okay, I'm with you.
Speaker 3: Now, you just line them up square one, score two,
Speaker 3: scure three from top left to bottom right.
Speaker 2: You flatten the grid into a line exactly.
Speaker 3: They turn the image into a sentence of visual words.
Speaker 3: They call them patches. And once you did that, once
Speaker 3: you turn the image into a sequence of patches, the
Speaker 3: transformer looked at it and said, I know what to
Speaker 3: do with this.
Speaker 2: Because of the transformer, a patch of pixels is just
Speaker 2: another token, the same way a word is a token.
Speaker 3: Precisely, that is, the everything is a token realization. And
Speaker 3: it didn't stop at images. Audio that's just a sequence
Speaker 3: of spectrogram slices. Video that's just a sequence of frames
Speaker 3: in temporal order, even code or molecules.
Speaker 2: So the machine stops seeing image versus text versus audio,
Speaker 2: and just start seeing data stream versus data strue.
Speaker 3: Correct. It was like discovering that French, Mandarin and mathematics
Speaker 3: are all actually dialects of the same underlying language. Once
Speaker 3: they realized that the transformer could handle all of these
Speaker 3: as sequences, the barrier between the senses just it evaporated.
Speaker 2: That is wild. So the architecture was the lock, and
Speaker 2: this idea of tokenization was the key that fit everything.
Speaker 3: That's a beautiful way to put it. And once that
Speaker 3: architectural problem was solved, the floodgates opened. We moved into
Speaker 3: this phase of connecting the dots, of teaching these different
Speaker 3: senses to talk to each other.
Speaker 2: Okay, I get the architecture. That makes sense. We can
Speaker 2: now feed everything into the same kind of machine. But
Speaker 2: I'm still stuck on the understanding part. Just because I
Speaker 2: feed a picture of a dog and the word dog
Speaker 2: into the same machine, how does the machine know they
Speaker 2: refer to the same thing. Surely it's not just looking
Speaker 2: it up in a dictionary.
Speaker 3: No, no, it's not a lookup table at all. It's geometry.
Speaker 2: Geometry. You're gonna have to explain that one. How does
Speaker 2: a picture of a dog become geometry?
Speaker 3: This is where we have to talk about vectors and
Speaker 3: high dimensional space, and to do that we have to
Speaker 3: talk about how these things are actually trained. The most
Speaker 3: famous example is a model called clap from open ai clip.
Speaker 2: I've seen that mentioned contrast of language image pre training.
Speaker 3: It's a mouthful, but the concept is really elegant. Imagine
Speaker 3: you have a massive bucket of data, and I'm talking
Speaker 3: four hundred million images scraped from the Internet and the
Speaker 3: text captions that came with them.
Speaker 2: So like IMG zero zero one dot jpeg and the
Speaker 2: alt text that says a golden retriever catching a frisbee
Speaker 2: on the beach.
Speaker 3: Right now, you start with a blank brain. It knows nothing.
Speaker 3: You show at the image and you show at the text. Initially,
Speaker 3: the machine thinks these are totally unrelated things. It turns
Speaker 3: the image into a set of numbers. We call that
Speaker 3: a vector, and it turns the text into another set
Speaker 3: of numbers. And those numbers are in this mathematical space,
Speaker 3: nowhere near each other. The strangers in the map total strangers.
Speaker 3: But then you apply something called contrastive loss. This is
Speaker 3: the training mechanism. You essentially punish the machine. You say, hey,
Speaker 3: these two sets of numbers, they belong together. Pull them closer.
Speaker 2: You're forcing them to be neighbors exactly.
Speaker 3: And simultaneously you show at the text a golden retriever
Speaker 3: catching a frisbee and a picture of a toaster, and
Speaker 3: you say, push these apart. These are not the same.
Speaker 3: These live on opposite sides of the universe.
Speaker 2: So it's this constant game of hot and cold, pushing
Speaker 2: and pulling.
Speaker 3: Done billions and billions of times, over and over, and
Speaker 3: eventually the machine builds a map. We call it a
Speaker 3: high dimensional vector space. Imagine a graph, but instead of
Speaker 3: two or three axes, it has thousands. In this map,
Speaker 3: the coordinates for the visual pattern of fur, floppy, ears
Speaker 3: and tail end up located at the exact same coordinates
Speaker 3: as the linguistic pattern for the word dog.
Speaker 2: Wow. So it's not using a dictionary. It's not looking
Speaker 2: up dog equals animal. It's mapping the concept of dogness
Speaker 2: to a specific location in this massive, invisible space.
Speaker 3: Yes, and this is why it feels like it understands
Speaker 3: because that space has geometry. It has a kind of logic.
Speaker 2: Okay, give me an example of that logic, because logic
Speaker 2: implies it can do reasoning, not just matching.
Speaker 3: Okay, think about the classic relationship between king and queen
Speaker 3: in text. If you take the math vector for the
Speaker 3: word king, subtract the vector for man, and then add
Speaker 3: the vector for woman, you land almost perfectly on the
Speaker 3: vector for queen.
Speaker 2: Right. That's the famous example King minus man plus woman
Speaker 2: equals queen. It's like vector arithmetic.
Speaker 3: Now do it with images. If you take the visual
Speaker 3: vector of a king a photo of a guy in
Speaker 3: a crown, subtract the visual features that represent man, and
Speaker 3: add the visual features that represent woman, the machine generates
Speaker 3: an image of a.
Speaker 2: Queen that is mind blowing. The logic, the geometric relationship
Speaker 2: it holds up across the senses it does.
Speaker 3: It means the machine has found a concept layer that
Speaker 3: sits deeper than language and deeper than pixels. It has
Speaker 3: found the meaning that connects them.
Speaker 2: It's performing analogical reasoning.
Speaker 3: It is. That's how the system can look at a
Speaker 3: photo of a funeral and connect it to the text
Speaker 3: A moment of grief. It's not because it memorized that
Speaker 3: specific photo and caption pair. It's because the visual information
Speaker 3: in the funeral photo and the concept of grief from
Speaker 3: the text live in the same emotional region of this
Speaker 3: mathematical space.
Speaker 2: It's math the geometry of sadness.
Speaker 3: In a mathematical sense. Yes, it has aligned the visual
Speaker 3: features of sadness with the linguistic features of sadness.
Speaker 2: That explains so much about why these systems feel like
Speaker 2: they get it. They aren't just matching keywords. They are
Speaker 2: navigating a map of meaning.
Speaker 3: And usually the architecture that runs this map, the sort
Speaker 3: of central brain, is a large language model. You have
Speaker 3: these specialized encoders. You can think of them as the
Speaker 3: eyes and ears that project all this information into the brain.
Speaker 3: The LM does the reasoning in that shared space, and
Speaker 3: then it can send information back out.
Speaker 2: So the LLEN is the conductor of the orchestras, making
Speaker 2: sure the strings and the woodwinds are all playing from
Speaker 2: the same sheet music. Ceaseely, all right, So we have
Speaker 2: the history. The silos are gone with the science. It's
Speaker 2: a geometry of concepts. Now I want to talk about
Speaker 2: the utility, because cool math is great, but what can
Speaker 2: this actually do?
Speaker 3: The capabilities are substantial, and I think we should start
Speaker 3: with what we can call vision language nuance, because we're
Speaker 3: not just talking about identifying objects anymore.
Speaker 2: Right. This isn't just drawing a box around a cat
Speaker 2: and saying cat ninety nine percent confidence. That was like
Speaker 2: twenty fifteen era. AI.
Speaker 3: No, No, Now, it's about identifying relationships emotional tenor. It
Speaker 3: can look at a scene and say this is a
Speaker 3: tense negotiation happening in a corporate boardroom based on the
Speaker 3: body language, the lighting, the arrangement of people. But one
Speaker 3: of the most practical superpowers is something called OCR integration
Speaker 3: optical character recognition.
Speaker 2: But OCR has been round since the nineties. My scanner
Speaker 2: came with it. Why is this a big deal? Now?
Speaker 3: Old OCR was dumb. It just scraped text off a page.
Speaker 3: It didn't know where the text was or what it
Speaker 3: meant in context. Multimodal AI reads the text in context.
Speaker 3: It can look at a street sign in a photo,
Speaker 3: read the sign, look at the cars, look at the
Speaker 3: time of day, and tell you if parking is legal right.
Speaker 2: Now, or to go back to our earlier point. It
Speaker 2: could read a handwritten note on a medical scan and
Speaker 2: understand how that note relates to the X ray.
Speaker 3: Itself exactly, which segues perfectly into the second big capability
Speaker 3: document understanding. This is what some people are calling the
Speaker 3: ultimate office assistant.
Speaker 2: This is the one that I think is going to
Speaker 2: change a lot of white collar work. I want you
Speaker 2: to walk me through a scenario here, because I deal
Speaker 2: with PDFs all day and they are where data goes
Speaker 2: to die.
Speaker 3: Okay, picture this. You have a fifty page annual report.
Speaker 3: It's got three columns of text, complex bar charts, photos
Speaker 3: with captions, footnotes. For old AI, that was a complete nightmare.
Speaker 3: The text would get jumbled, the chart was invisible.
Speaker 2: I was soup. You'd copy paste it into a text
Speaker 2: file and just get absolute garbage. Right.
Speaker 3: But a multimodal system sees the document like a human does.
Speaker 3: It understands the layout. It can look at the bar chart,
Speaker 3: extract the data from the visual bars, I mean literally
Speaker 3: measuring the pixels of the bars, read the surrounding text,
Speaker 3: understand what that data means, and answer a question like
Speaker 3: based on the chart on page three, which quarter had
Speaker 3: the highest revenue.
Speaker 2: Without a human having to manually turn that chart into
Speaker 2: an Excel sheet first zero preprocessing.
Speaker 3: It just looks and understands.
Speaker 2: That's incredible. It basically unlocks all the information that is
Speaker 2: trapped inside images, within documents. What about audio and video
Speaker 2: You mentioned earlier that video is just a sequence of frames.
Speaker 3: Audio and video are huge frontiers now. In audio, we
Speaker 3: aren't just transcribing speech to text anymore. We are analyzing
Speaker 3: the vocal characteristics. The system can detect emotion. Is the
Speaker 3: speaker angry, nervous, sarcastically happy?
Speaker 2: You can hear the scare quotes in your voice.
Speaker 3: It can absolutely and it can analyze music, not just
Speaker 3: the genre, but the rhythm, the mood, the instrumentation. When
Speaker 3: you combine that with video, you get narrative understanding. It
Speaker 3: can track events over time and start to build a story.
Speaker 2: But the real magic, and the research we looked at
Speaker 2: was really emphatic about this is the killer app of
Speaker 2: true integration. Is it not just being good at video
Speaker 2: or good at text. It's the combo.
Speaker 3: It is the synthesis. That's where the real power is.
Speaker 3: Let's look at a coding scenario. Imagine you're a developer.
Speaker 3: You're stuck. You get some cryptic error message. You take
Speaker 3: a screenshot of your error message. You just paste it
Speaker 3: into the AI. The AI reads the screenshot, looks at
Speaker 3: your actual code file, consults the official software documentation online,
Speaker 3: and synthesizes an answer.
Speaker 2: So it's using its eyes and it's reading comprehension at
Speaker 2: the exact same time to solve one problem.
Speaker 3: Or take medicine, the radiologist assistant idea. It looks that
Speaker 3: the CT scan, that's vision. It reads the patient's history notes,
Speaker 3: that's text. It checks the latest research papers for medical journals.
Speaker 3: More text, and it synthesizes a potential diagnosis based on
Speaker 3: all three modalities.
Speaker 2: It becomes the ultimate second opinion engine.
Speaker 3: Right or a final example in design, you sketch a
Speaker 3: rough idea for an app on a napkin, You take
Speaker 3: a photo, you upload it, and you say, make this
Speaker 3: look like a sleek modern app interface, but use our
Speaker 3: official brand colors from this attached pdf. It sees the sketch,
Speaker 3: it reads your brief, it consults the PDF for the
Speaker 3: color codes, and it generates the final image.
Speaker 2: It's closing the loop between idea, instruction, and creation. It
Speaker 2: feels like we're getting closer to that Jarvis from Ironman Fantasy,
Speaker 2: the as system that just handles them.
Speaker 3: We are getting closer. But and this is a very
Speaker 3: very big up. We have to talk about where it
Speaker 3: breaks because it is not Jarvis yet, and you and
Speaker 3: I need to be clear that this isn't magic. It
Speaker 3: breaks in some surprisingly dumb ways.
Speaker 2: You don't want to play the skeptic here for a minute,
Speaker 2: because it sounds perfect, but I know it's not. Where
Speaker 2: does the machine stumble? What trips it up?
Speaker 3: It stumbles in some surprisingly fundamental areas. The first one,
Speaker 3: and this is almost ironic is spatial reasoning, which.
Speaker 2: Is funny, right because you'd think a computer vision system
Speaker 2: would be great at space.
Speaker 3: It sees pixels, you would think, But remember these systems
Speaker 3: are trained on flat two D images from the Internet.
Speaker 3: They struggle to build an intuitive three D model of
Speaker 3: the world. If you show it a picture of a
Speaker 3: table with a messy pile of objects, and you ask
Speaker 3: is the apple behind the book or in front of it,
Speaker 3: it often gets really confused.
Speaker 2: It sees the pixels of the apple and the pixels
Speaker 2: of the book, but it doesn't get the depth the
Speaker 2: physics of one object including another.
Speaker 3: It lacks a physics engine in its head. It doesn't
Speaker 3: intuitively understand that solid objects occupy space and can't pass
Speaker 3: through each other. And this is a massive problem for robotics.
Speaker 3: If you want a robot to clean your kitchen, it
Speaker 3: needs to know exactly where the cup is relative to
Speaker 3: the edge of the table. Close enough isn't good enough.
Speaker 3: When you're hands fine china.
Speaker 2: That makes a ton of sense. It sees the picture,
Speaker 2: but it doesn't understand the physical reality behind the picture exactly.
Speaker 3: Then there is temporal understanding. We can process video as
Speaker 3: a stream of frames, but Understanding causality over time is
Speaker 3: really hard for these models.
Speaker 2: Causealit you mean, like the glass broke because the ball.
Speaker 3: Hit it exactly that. Or even following a complex argument
Speaker 3: in a lecture, if I say A happened which led
Speaker 3: to B, but then C came along and prevented D
Speaker 3: from occurring, the AI might track all the nouns but
Speaker 3: lose the thread of the logic that question what happened
Speaker 3: because of what is a surprisingly difficult cognitive task for it.
Speaker 2: It's the difference between seeing a series of snapshots and
Speaker 2: understanding a story with a plot precisely.
Speaker 3: And then we have to talk about the big one hallucination.
Speaker 2: We know text models lie, they make up facts, they
Speaker 2: make up sources. Do multimodal models lie in the same.
Speaker 3: Way, they lie in new and excitingly creative ways. We
Speaker 3: call it multimodal hallucinating.
Speaker 2: That sounds terrifying. Give me an example of what that
Speaker 2: looks like.
Speaker 3: It can be. A system might read the text in
Speaker 3: a financial report perfectly, but then look at a graph
Speaker 3: on the same page and completely hallucinated trend that isn't there.
Speaker 3: It might say sales are trending upwards when the line
Speaker 3: is clearly going down whoa. Or it might describe a
Speaker 3: photo and just invent details about objects that are partially hidden.
Speaker 2: So it sees half a car behind a building and says,
Speaker 2: there is a red convertible with a dog in the
Speaker 2: back seat, even though it can't possibly see the back seat.
Speaker 3: Right, it's confabulating based on probability things well, usually cars
Speaker 3: have things in back seats, and it just fills in
Speaker 3: the blanks with a plausible story. In a creative context,
Speaker 3: you might call that imagination. But in a medical or
Speaker 3: illegal context that's malpractice.
Speaker 2: That's a crucial distinction. It's just guessing, and sometimes a
Speaker 2: guess is wrong with total unwavering confidence.
Speaker 3: And there's also the problem of compositionality. This is what
Speaker 3: I like to call the catenoid at the dog problem.
Speaker 3: Explain that one identifying cat and dog in a picture
Speaker 3: is easy, that's basic object recognition, but understanding their relationship
Speaker 3: the cat is annoyed at the dog requires understanding subtle
Speaker 3: cues and the interaction between the two. Current systems often
Speaker 3: struggle to bind those attributes correctly. They might see a
Speaker 3: happy dog in an angry cat and output the sentence
Speaker 3: an angry dog and a happy cat. They mix up
Speaker 3: who owns which emotion, so.
Speaker 2: They get all the ingredients right, but they get the
Speaker 2: recipe completely wrong.
Speaker 3: A perfect analogy, and finally, the deepest and most philosophical
Speaker 3: limitation grounding, or what we can call the fire problem.
Speaker 2: This was the part of the research that really stuck
Speaker 2: with me, the idea that the AI knows fire, but
Speaker 2: it doesn't know fire.
Speaker 3: It's the gap between data and experience. The AI has
Speaker 3: seen a billion pixels of fire, it has read a
Speaker 3: trillion words about heat, burning, smoke, and danger, but it
Speaker 3: has never felt heat. It has never reflexively pulled its
Speaker 3: hand away from a hot stove. It lacks sensory consequence.
Speaker 2: But does that matter. I've ever been to Mars, but
Speaker 2: I feel like I know a lot about it. I
Speaker 2: learned it all from books and pictures. If the AI
Speaker 2: tells me don't touch the fire it's dangerous, does it
Speaker 2: matter that it's never been burned itself.
Speaker 3: That is the big counter argument. Maybe you don't need
Speaker 3: a body to understand, but there is a very strong
Speaker 3: hypothesis in cognitive science that true intelligence requires embodiment. That
Speaker 3: you can't really think about the physical world unless you
Speaker 3: have a body that risks being hurt by it. If
Speaker 3: you don't fear the fire, do you really understand danger
Speaker 3: or do you just know the statistical correlation between the
Speaker 3: token danger and the token fire.
Speaker 2: That's deep. We should definitely circle back to that later,
Speaker 2: but first let's look at where this is hitting the
Speaker 2: ground right now, despite the hallucinations and the lack of
Speaker 2: a body, where are these systems actually transforming the world today.
Speaker 3: Medicine is the big one. I really can't overstate this.
Speaker 3: Medicine has always always been multimodal.
Speaker 2: Right, you go to the doctor, they look at you,
Speaker 2: they listen to your lungs, they re your chart, they
Speaker 2: look at your lab results. It's a mix of everything exactly.
Speaker 3: A doctor is at their core an information integrator. Multimodal
Speaker 3: AI is the first tool that really matches that workflow.
Speaker 3: It acts as a high speed second opinion. It's not
Speaker 3: replacing the doctor's judgment, but it's synthesizing the data faster
Speaker 3: than any human could ever hope to.
Speaker 2: It's the ultimate intern.
Speaker 3: It's an intern that has read every single medical paper
Speaker 3: ever published and fields like pathology, radiology, genomics. It's starting
Speaker 3: to find patterns that humans miss. It might see a
Speaker 3: faint correlation between a genetic marker mentioned in the text
Speaker 3: of a patient's file and a specific cell shape in
Speaker 3: a microscopy image that a human would never connect because
Speaker 3: the data is just too vast.
Speaker 2: Then there is accessibility. This feels like one of the
Speaker 2: most immediate and unambiguously positive impacts.
Speaker 3: It is democratization on a massive scale. Think about what
Speaker 3: this fluidity between senses means. If you are blind, the
Speaker 3: world is opaque to visual signals. Multimodal AI can describe
Speaker 3: the visual world to you in text or audio.
Speaker 2: There's a blue car approaching on your left, or the
Speaker 2: light just turned.
Speaker 3: Green exactly, or even you are holding the can of
Speaker 3: soup upside down. It gives you eyes. For deaf users,
Speaker 3: it can translate audio to text, but also describe the
Speaker 3: emotion in the speaker's voice. It bridges the gap between
Speaker 3: the sense you have and the information you need.
Speaker 2: It completely removes the friction of format.
Speaker 3: Domain number three science and education. In science, we are
Speaker 3: absolutely drowning in data. We have microscopy images, protein structures,
Speaker 3: satellite data, research papers. No single human can read it all.
Speaker 2: So the AI becomes a kind of research partner.
Speaker 3: It becomes an active participant. It can read all the
Speaker 3: latest papers, look at all the new experimental slides, and say, hey,
Speaker 3: this pattern in the satellite data over the Amazon matches
Speaker 3: this obscure theory from a paper published in nineteen ninety.
Speaker 3: It connects dots that are separated by decades and disciplines.
Speaker 2: And in education, how does it play out there?
Speaker 3: Responsive tutoring imagine AI that watches a student solve a
Speaker 3: math problem on a piece of paper, literally watches the
Speaker 3: pen move through the camera and at the same time
Speaker 3: listens to them talk through their reasoning. It can pinpoint
Speaker 3: exactly where the logic broke down. It doesn't just say
Speaker 3: wrong answer, It says you forgot to carry the one
Speaker 3: in the tens column right here.
Speaker 2: That's the difference between a textbook and a real teacher.
Speaker 2: A teacher watches the process, not just the result it is.
Speaker 3: And finally, of course, creative work. This is the controversial one.
Speaker 2: Text to image, text to video. We see this everywhere
Speaker 2: now it's exploded.
Speaker 3: It has democratized visual creation in a way we've never seen.
Speaker 3: You don't need to know how to draw or paint
Speaker 3: to create the stunning image anymore. But it creates this
Speaker 3: massive tension regarding the displacement of professionals and the ethics
Speaker 3: of using copyrighted work in training.
Speaker 2: Data artists are rightfully saying, Hey, you train this model
Speaker 2: on my entire life's work without my permission, and now
Speaker 2: it's competing with me for jobs, and it's.
Speaker 3: A completely valid conflict. There's no easy answer. But the
Speaker 3: tenential is also there for using these tools as extensions
Speaker 3: of human vision. It's a tool that can amplify a creativity,
Speaker 3: not just replace it. A director can visualize an entire
Speaker 3: storyboard in seconds. An architect can iterate on a dozen
Speaker 3: building facades instantly.
Speaker 2: I want to pivot back to that philosophical moment we
Speaker 2: touched on earlier, the grief example. So this is the
Speaker 2: one that really keeps me up at night.
Speaker 3: Let's go back to it. It's the most important question,
Speaker 3: I think.
Speaker 2: Okay, So if the machine recognizes the funeral, it correctly
Speaker 2: identifies the sadness in people's faces, and it writes a
Speaker 2: poem about loss that makes me cry, has it understood grief?
Speaker 3: This is the critical distinction between what we call behavioral
Speaker 3: performance versus experiential understanding.
Speaker 2: Behavior versus experience. Okay, break that down for me.
Speaker 3: Behaviorally, yes, absolutely, it performed the task of understanding grief perfectly.
Speaker 3: It recognized the symbols, It generated the appropriate linguistic response.
Speaker 3: It passed the Turing test for sadness with flying colors.
Speaker 3: But experientially, experientially know it is a hollow shell. It
Speaker 3: processes the symbols of grief without the reference. It has
Speaker 3: the map, but it has never visited the territory. It
Speaker 3: has never lost anyone. It has never felt that hollow
Speaker 3: ache of absence in its chest.
Speaker 2: So why does this distinction matter? I mean, if the
Speaker 2: poem is good, who cares that the poet is a
Speaker 2: sad robot or a sad human? If the output is
Speaker 2: the same, why does the internal state matter so much?
Speaker 3: For writing a poem, maybe it doesn't matter. For generating
Speaker 3: ad copy it definitely doesn't matter. But for moral judgment,
Speaker 3: for empathy, for wisdom, that gap is critical. If we
Speaker 3: ask an AI to make decisions about elder care or
Speaker 3: legal sentencing or childcare, do we want a system that
Speaker 3: just mimics wisdom or one that actually has it?
Speaker 2: That is a chilling thought. We shouldn't mistake a high
Speaker 2: confidence output for lived experience.
Speaker 3: Exactly, A human radiologist brings years of seeing patients, of
Speaker 3: knowing the fear in their eyes when they get a
Speaker 3: bad diagnosis, of understanding the weight of that responsibility. The
Speaker 3: AI brings pattern matching on a massive data set. Those
Speaker 3: are not the same thing. Even if the diagnosis it
Speaker 3: gives is correct. The AI doesn't care if the patient
Speaker 3: lives or dies. It just cares about minimizing the loss
Speaker 3: function in its training.
Speaker 2: So where does this all go next? If this is
Speaker 2: where we are now on the frontier, what's beyond the frontier?
Speaker 3: The trajectory is becoming very clear. We are moving from
Speaker 3: processing information to acting on it.
Speaker 2: The agentic shift. I keep hearing this term popping up
Speaker 2: more and more.
Speaker 3: Yes, right now, most people interact with AI by chatting,
Speaker 3: write this for me, analyze this data. The next wave
Speaker 3: is agents that do things.
Speaker 2: So not just tell me how to book a flight,
Speaker 2: but literally book me the cheapest flight to Chicago next Tuesday.
Speaker 3: Book the flight, email my boss to let them know
Speaker 3: I'll be out, update my calendar, and order a car
Speaker 3: to take me to the airport. These are systems that
Speaker 3: will browse the web, operate software on your computer, and
Speaker 3: execute code to accomplish goals. They will have eyes to
Speaker 3: see the screen, and hands whether virtual or robotic, to
Speaker 3: click the.
Speaker 2: Buttons, and robots real physical robots in the world.
Speaker 3: That's the physical manifestation of the same idea. Robots that
Speaker 3: watch a human demonstrate a task, say folding laundry or
Speaker 3: assembling a circuit board, and then replicate it. The visual
Speaker 3: understanding guides the motor control. The senses are connected to
Speaker 3: the limbs.
Speaker 2: This raises the stake significantly.
Speaker 3: It completely changes the risk profile. A chat butt that
Speaker 3: writes a bad poem is embarrassing. A robot that misunderstands
Speaker 3: the command clean up the kitchen and throws out your
Speaker 3: vital medication is dangerous.
Speaker 2: Or an autonomous software agent that misunderstands a financial instruction
Speaker 2: and executes a code that deletes a critical.
Speaker 3: Database exactly when perception leads directly to physical or digital action.
Speaker 3: In the world, safety isn't just about content moderation anymore.
Speaker 3: It's about physical safety and operational security. We are giving
Speaker 3: these systems hands, we need to be very very sure
Speaker 3: about the brain that's guiding them.
Speaker 2: It really feels like we are standing on the edge
Speaker 2: of a profoundly different world.
Speaker 3: We are the walls between the senses are gone, the
Speaker 3: silo are broken.
Speaker 2: So to kind of summarize our journey today, we started
Speaker 2: with the island problem. AI was completely fragmented. We moved
Speaker 2: to the universal substrate, transformers and tokens connected everything. We
Speaker 2: saw the magic of vector space, where meanings are mapped
Speaker 2: out geometrically. We looked at the superpowers, the medical assistant,
Speaker 2: the coder's buddy. We acknowledged the very real limitations no body,
Speaker 2: no real spatial sense, the problem of hallucination. And we've
Speaker 2: just looked ahead at the agentic future.
Speaker 3: That is the arc. It's the collapse of separation.
Speaker 2: What's the final thought here which we walk away thinking
Speaker 2: about as we go about our day.
Speaker 3: I think it's this. The machines are learning to see
Speaker 3: and hear, They are developing senses. For decades, we have
Speaker 3: spent so much time worrying about whether they can think
Speaker 3: that we maybe haven't paid enough attention to what it
Speaker 3: means that they can perceive, perceive us, perceive the world.
Speaker 3: We are building a new kind of observer. It's not human,
Speaker 3: but it's not blind anymore. The question we all need
Speaker 3: to ask is are we paying enough attention to what
Speaker 3: that means for the world we are building.
Speaker 2: That is a question I think we will be wrestling
Speaker 2: with for a very long time. Next time you look
Speaker 2: at your phone, just remember it might be looking back
Speaker 2: at you and it's finally starting to understand what it sees.
Speaker 2: Thanks for joining us on this exploration A pleasure, as always,