Speaker 1: Welcome to the Sentient Code, where intelligence is engineered, autonomy
Speaker 1: is emerging, and a line between human and machine grows thinner.
Speaker 1: Each episode, we decode the algorithms, explore the robotics, and
Speaker 1: examine the ideas shaping the future of artificial minds.
Speaker 2: Welcome back to the deep Dive.
Speaker 3: You know.
Speaker 2: We spend so much time obsessing over the software side
Speaker 2: of AI.
Speaker 3: Oh, we absolutely do. It is all anyone talks about,
Speaker 3: right We look.
Speaker 2: At the chatbots and the image generators, the flashy demos,
Speaker 2: the code. It is always look what this new model
Speaker 2: can do. But today I want to physically unplug all
Speaker 2: of that.
Speaker 3: If we are going down to the basement.
Speaker 2: We are going to the basement, we are stepping away
Speaker 2: from the cloud and we're going to talk about the
Speaker 2: actual physical machine that makes the cloud exist. We are
Speaker 2: talking about the iron.
Speaker 3: The iron. You know, it is funny you use that
Speaker 3: term because for the first really the first fifty years
Speaker 3: of computing history, software people look down on hardware people.
Speaker 2: It was just plumbing to them.
Speaker 3: Exactly. Hardware was a commodity. It was dirt. It was
Speaker 3: just the physical thing you ran your brilliant code on.
Speaker 3: And if your code was running slow. You didn't rewrite it.
Speaker 3: You just waited two years for Intel to make a
Speaker 3: faster chip, and your problem was magically solved.
Speaker 2: Moore's law was basically a free lunch for developers.
Speaker 3: It was a free lunch. But that free lunch is over.
Speaker 3: The script is completely flipped. Now we have moved from
Speaker 3: this era of code dominance to an era of compute dominance.
Speaker 2: Because in the AI world today, the smartest algorithm doesn't
Speaker 2: necessarily win anymore.
Speaker 3: No, it doesn't. The team with the biggest, most specialized
Speaker 3: pile of silicon wins, period.
Speaker 2: And we are calling this the compute race. I think
Speaker 2: what is so surprising to people tuning into this is
Speaker 2: that this isn't just Apple versus Microsoft anymore. This is
Speaker 2: arguably the central geopolitical conflict of the twenty twenties.
Speaker 3: It is the absolute bottleneck of the modern world. You
Speaker 3: look at the supply chain for advanced AI chips, it
Speaker 3: is terrifyingly concentrated. Right we were talking about a single
Speaker 3: company in the Netherlands that makes the manufacturing machines, a
Speaker 3: single island Taiwan that actually manufactures the chips, and a
Speaker 3: handful of US companies that design them and if.
Speaker 2: Any single link in that chain breaks, the.
Speaker 3: AI revolution doesn't just slow down, it stops entirely.
Speaker 2: So today we are going to trace that chain. We
Speaker 2: are going to figure out how a piece of hardware
Speaker 2: designed to let teenagers play Call of Duty somehow became
Speaker 2: the brain of modern civilization.
Speaker 3: Is an incredible pivot, and.
Speaker 2: We are going to look at why Google panic built
Speaker 2: their own secret chip factory, and we will get into
Speaker 2: the physical limits too, because apparently we are literally running
Speaker 2: out of atoms to work with.
Speaker 3: It is a story about physics, it is a story
Speaker 3: about economics, and ultimately it is a story about war.
Speaker 2: Let us start with the origin story, because this is
Speaker 2: the part that always feels like a massive accident of
Speaker 2: history to me. If you look at the trillion dollar
Speaker 2: club today, Nvidia is sitting right there at the top,
Speaker 2: But twenty years ago they were not trying to build
Speaker 2: artificial intelligence.
Speaker 3: Not even close. If you walked into Nvidia headquarters in
Speaker 3: say nineteen ninety nine or two thousand and five, they
Speaker 3: were obsessed with one single thing, polygons video games, specifically
Speaker 3: rendering three D graphics. They wanted to make better explosions,
Speaker 3: realistic textures, dynamic lighting, shadows and to understand why that
Speaker 3: matters for AI, we really have to get a little
Speaker 3: technical here.
Speaker 2: Okay, let's break it down. We need to distinguish between
Speaker 2: the chip in your standard laptop, which is the CPU,
Speaker 2: and the chip in a gaming card, the GPU. Because
Speaker 2: I think most people hear the word processor and they
Speaker 2: just picture a little black square. What is the actual
Speaker 2: architectural difference inside that square?
Speaker 3: Okay, let's unpack this. Imagine a CPU, the central processing unit.
Speaker 3: This is your Intel Core I nine or your AMD risin.
Speaker 3: A CPU is like a team of incredibly smart mathematicians.
Speaker 3: Let us say a tight knit team of twelve geniuses.
Speaker 2: Small team, very high IQ.
Speaker 3: Extremely high i Q, and highly versatile. If you give
Speaker 3: one of these CPU cores a really complex problem, something
Speaker 3: like run this operating system, then open Excel, then calculate
Speaker 3: this complex formula, then check for incoming email. It can
Speaker 3: handle that context switching perfectly.
Speaker 2: Because it is designed for serial processing.
Speaker 3: Exactly Step A, then step B, then step C. It
Speaker 3: has massive amounts of complex logic built in just to
Speaker 3: handle branching paths like if this happens, then do that.
Speaker 2: So a CPU is optimized for logic and sequence.
Speaker 3: Correct now look at a GPU, the graphics processing unit.
Speaker 3: A GPU is not a team of twelve geniuses. It
Speaker 3: is a stadium filled with ten thousand average high school students.
Speaker 2: Okay, I like where this analogy is going.
Speaker 3: Individually, those students aren't that smart. They cannot run a
Speaker 3: modern operating system. They will completely freeze up if you
Speaker 3: give them complex branching logic chains.
Speaker 2: But they have numbers on their side.
Speaker 3: Exactly, if you give them a task that is simple
Speaker 3: and repetitive, like take these two numbers and add them together,
Speaker 3: and you tell all ten thousand of them to do
Speaker 3: it at the exact same.
Speaker 2: Time, they will completely obliterate the CPU.
Speaker 3: They will leave it a in the dust.
Speaker 2: And this is the core concept of parallelism.
Speaker 3: Specifically data parallelism. And this is exactly where video games
Speaker 3: come into the picture. Think about your computer screen right now.
Speaker 3: It is a grid of pixels. A standard monitor is
Speaker 3: nineteen twenty y ten eighty, which is roughly two million pixels.
Speaker 3: To render just one single frame of a video game,
Speaker 3: you need to calculate the exact color for every single
Speaker 3: one of those two million pixels based on the virtual lighting,
Speaker 3: the texture of the wall, the geometry of the character.
Speaker 2: And the crucial part here is that the color of
Speaker 2: the pixel and the top left corner generally does not
Speaker 2: depend on the color of the pixel in the bottom
Speaker 2: right corner.
Speaker 3: Precisely, they are mathematically independent. You don't need to calculate
Speaker 3: pixel one and then wait to calculate pixel two and
Speaker 3: then pixel three. You can calculate all two million of
Speaker 3: them simultaneously. Computer scientists actually have a great term for this.
Speaker 3: They call it an embarrassingly parallel problem.
Speaker 2: I love that term so much. It is so parallel
Speaker 2: it is actually embarrassing not to do it all at once, and.
Speaker 3: That is exactly why the GPU was invented. It actively
Speaker 3: sacrifices individual core speed and complex logic in exchange for
Speaker 3: raw massive parallelism. It uses thousands of tiny, relatively dumb
Speaker 3: cores instead of a few really smart ones.
Speaker 2: Okay, so we have this chip that was fundamentally designed
Speaker 2: to run games like Doom and Quake. How do we
Speaker 2: make the jump from rendering a virtual shotgun to running
Speaker 2: chat GBT?
Speaker 3: This is where we hit the great convergence. Around the
Speaker 3: late two thousands, AI researchers were hitting a massive wall.
Speaker 3: They had these theoretical ideas about neural networks, which are
Speaker 3: basically mathematical structures inspired by the biological human brain, but
Speaker 3: actually training them was agonizingly slow because they were trying
Speaker 3: to run them on those CPUs. The twelve Geniuses, right,
Speaker 3: and the geniuses were just getting bogged down because a
Speaker 3: neural network, at its very core is just a massive
Speaker 3: grid of numbers. In math we call the matrices. To
Speaker 3: train in AI, you have to multiply these giant grids
Speaker 3: of numbers together, adjust the results slightly, and then do
Speaker 3: it again billions of time, literally billions of times.
Speaker 2: And I'm guessing matrix multiplication is.
Speaker 3: It is embarrassingly parallel. Multiplying a massive matrix is really
Speaker 3: just performing the exact same simple multiplication operation on thousands
Speaker 3: of numbers at the exact same time. It turns out
Speaker 3: the math required to simulate a photon of light bouncing
Speaker 3: off a three D wall in a video game is
Speaker 3: almost identical to the math required to simulate a virtual
Speaker 3: neuron firing in an artificial brain.
Speaker 2: That is just such a wild coincidence to me.
Speaker 3: It is the happy accident that gave us the entire
Speaker 3: modern world.
Speaker 2: So when did the industry actually realize this? Was there
Speaker 2: a specific moment where the light bulb suddenly went on
Speaker 2: for everyone, there was.
Speaker 3: A big bang moment. It was twenty twelve. A competition
Speaker 3: called image.
Speaker 2: Net set the scene for us. What exactly was image.
Speaker 3: Net Imaget was basically the Olympics of computer vision. You
Speaker 3: had this massive data set of millions of images pictures
Speaker 3: of cats, dogs, airplanes, strawberries, and researchers had to write
Speaker 3: software that could look at the pixels and identify what
Speaker 3: was actually in.
Speaker 2: The picture, which is incredibly hard for a computer, very hard.
Speaker 3: For years, the best teams in the world, mostly using
Speaker 3: traditional hand coded logic techniques, were stuck getting air rates
Speaker 3: around twenty six percent.
Speaker 2: That is not great, That is missing one out of
Speaker 2: every four pictures.
Speaker 3: It was the best we had At the time. Progress
Speaker 3: was agonizingly slow. Most people thought true human level computer
Speaker 3: vision was decades away. But then in twenty twelve, this
Speaker 3: small team from the University of Toronto shows up. Alex Krzewski,
Speaker 3: Ilia Sitzkaver, and Jeffrey Hinton, and.
Speaker 2: Those Nate I mean is Ilia Setzkaver and Jeffrey Hinton.
Speaker 2: These are the absolute titans of AI today, but back
Speaker 2: then they were kind of the outsiders.
Speaker 3: Right, they are the crazy ones. Neural networks were widely
Speaker 3: considered a dead end by most serious computer scientists, but
Speaker 3: this team entered a neural network they called alex net,
Speaker 3: and it didn't just win the competition, it utterly destroyed
Speaker 3: the field. They dropped the air rate from twenty six
Speaker 3: percent down to fifteen percent in.
Speaker 2: A single year. That is unprecedented for that competition.
Speaker 3: In one year. It was a mathematical massacre. The entire
Speaker 3: conference room when completely silent when they presented. But here's
Speaker 3: the specific detail that matters for a story today. To
Speaker 3: train alex Net, they didn't use a massive government supercomputer.
Speaker 3: They didn't use a giant server cluster.
Speaker 2: What did they use.
Speaker 3: They literally went to a consumer electronics store and bought
Speaker 3: two Nvidia GTX five eighty graphics cards.
Speaker 2: Two gamer cards, the exact kind of thing you would
Speaker 2: put in a Dusktop PC to play Skyrim.
Speaker 3: Exactly two cards that cost maybe five hundred dollars each
Speaker 3: at the time. They shoved them into a standard PC.
Speaker 3: They wrote some custom code to move the math off
Speaker 3: the CPU and onto the GPU, and they suddenly realized
Speaker 3: they could train their model in a matter of days instead.
Speaker 2: Of months and That is the true Aha moment, because
Speaker 2: if you can iterate in days, you can actually learn
Speaker 2: and adapt. If an experiment takes six months, you are
Speaker 2: just stuck.
Speaker 3: Exactly. Speed is intelligence in this field. If you can
Speaker 3: run one hundred experiments in the time it takes your
Speaker 3: rival to run one one, you get smarter incredibly fast. Honestly,
Speaker 3: in vidia stock price chart should basically have a little
Speaker 3: bronze statue of alex Net next to it. But this
Speaker 3: is the deep dive of nuance we need to hit.
Speaker 3: It wasn't just the physical hardware that made.
Speaker 2: This possible, right, because you cannot just plug a video
Speaker 2: card into a motherboard and tell it to learn English.
Speaker 2: It inherently speaks graphics, doesn't speak math.
Speaker 3: Correct And this is where in Nvidia's CEO Jensen Huong
Speaker 3: showed just incredible almost prophetic foresight years before alex Net,
Speaker 3: way back in two thousand and six, and Video released
Speaker 3: a software platform called CUA ce UA.
Speaker 2: I see this acronym constantly when reading about this space.
Speaker 2: It is usually described as Invidia's massive mote. What is
Speaker 2: it actually doing under the hood?
Speaker 3: Well before CD existed, If you wanted to use a
Speaker 3: GPU for scientific math, you basically had to hack it.
Speaker 3: You literally had to trick the graphics card into thinking
Speaker 3: your math problem was actually a texture or a shadow
Speaker 3: on a polygon. It was an incredibly painful process.
Speaker 2: Please render this massive spreadsheet as an explosion.
Speaker 3: Basically, yes, it was a total nightmare for researchers. CUDA
Speaker 3: changed all of that. It was a software layer that
Speaker 3: let normal programmers write standard code like C plus plus
Speaker 3: that ran directly on the GPU. It exposed the raw
Speaker 3: mathematical power of the chip without all the annoying graphics baggage.
Speaker 2: So Nvidia essentially built the translation layer before anyone even
Speaker 2: really knew what language they wanted to speak.
Speaker 3: Jensen Wong practically bet the entire company on it now
Speaker 3: and Wall Street absolutely hated it at the time. Investors
Speaker 3: were furious. They said, why are you spending billions of
Speaker 3: dollars on R and D for a feature that only
Speaker 3: a few academic weirdos in universities use.
Speaker 2: And then a few years later those weirdos invented modern AI.
Speaker 3: And because all those weirdos learned to code specifically in CUDA,
Speaker 3: the entire foundation of modern AI was built on top
Speaker 3: of Nvidia's proprietary software. Now If you are a hardware
Speaker 3: startup today and you want to build a brand new
Speaker 3: chip to beat in Nvidia, you have a massive, massive problem.
Speaker 2: Because nobody knows how to program your new chip exactly.
Speaker 3: All the libraries, all the developer tools, all the research
Speaker 3: they all natively speak CUDA. It is the classic ecosystem
Speaker 3: lock in. It is like Windows in the nineties or
Speaker 3: the iPhone app store today. It is incredibly difficult to
Speaker 3: break that habit.
Speaker 2: Let us fast forward to today then, because we are
Speaker 2: obviously not using five hundred dollars GTX five eighties anymore.
Speaker 2: We're using the H one hundred. This is the chip
Speaker 2: that companies are fighting over, the one Mark Zuckerberg is
Speaker 2: supposedly buying three hundred and fifty thousand.
Speaker 3: Of H one hundred is It is a monster. It
Speaker 3: is a true marvel of human engineering.
Speaker 2: Give me the physical stats. What are actually looking at here?
Speaker 3: It is a slab of silicon that has eighty billion
Speaker 3: individual transistors carved into it using a four and nanimeter
Speaker 3: manufacturing process. Just wrap your head around that eighty billion
Speaker 3: on one chip. But honestly, the raw transistor count isn't
Speaker 3: even the most impressive part. It is how highly specialized.
Speaker 3: The architecture has become specialized in what way? Remember how
Speaker 3: the old GPUs were fairly general purpose for graphics. The
Speaker 3: H one hundred is designed specifically from the ground up
Speaker 3: for the math of transformers DASH, which is the t
Speaker 3: in chat GPT. It has specific hardware units inside it
Speaker 3: called tensor.
Speaker 2: Cores tensor course.
Speaker 3: Think of them as dedicated calculator services that do nothing
Speaker 3: but matrix multiplication. They cannot render graphics, they cannot run
Speaker 3: an operating system. They just do that one specific math
Speaker 3: operation incredibly fast. The H one hundred can perform roughly
Speaker 3: four thousand trillion floating point operations per second if you
Speaker 3: use the right precision levels.
Speaker 2: Four thousand trillion operations per second. That is unfathomable.
Speaker 3: But here's the crazy part. Raw compute speed is actually
Speaker 3: the easy part of chip design. Now, the real bottleneck,
Speaker 3: the thing that actually keeps chip architects up at night,
Speaker 3: is memory.
Speaker 2: This is the memory wall concept I keep reading about, right.
Speaker 3: It simply does not matter if your process or brain
Speaker 3: can think of billion thoughts a second, if you cannot
Speaker 3: get the data into the brain fast enough, A super
Speaker 3: SaaS chip with slow memory is like a Ferrari with
Speaker 3: a clogged fuel line, it just stalls out. So the
Speaker 3: H one hundred uses a brand new type of memory
Speaker 3: called HBM High bandwidth memory.
Speaker 2: How does that solve the fuel line problem?
Speaker 3: It is stacked vertically. They literally build skyscrapers of memory
Speaker 3: chips directly on top of the processor itself to physically
Speaker 3: shorten the distance the electrical signals have to travel.
Speaker 2: So they are building three D towers of memory right
Speaker 2: next to the logic cores just to save the fractions
Speaker 2: of a nanosecond it takes for the signal to travel
Speaker 2: across a standard motherboard.
Speaker 3: We are actively fighting this speed of light. At this point.
Speaker 3: The H one hundred has a memory bandwidth of over
Speaker 3: three point three terabytes per second. To put that in perspective,
Speaker 3: that is like downloading thousands of full four K movies
Speaker 3: in a single second. It is an absolute fire hose
Speaker 3: of data.
Speaker 2: And they use something called envylink to string them together. Right.
Speaker 3: Yes, Envylink is their proprietary interconnect because one H one
Speaker 3: hundred isn't enough. You need thousands of them to function
Speaker 3: as one giant unified brain. Envylink is the nervous system
Speaker 3: that lets them talk to each other fast enough to
Speaker 3: stay synchronized.
Speaker 2: And yet despite all of that insane power and the
Speaker 2: CDA mode, and VIDIA is not the only player in
Speaker 2: town anymore. Which brings us to this sleeping giant that
Speaker 2: suddenly woke up Google.
Speaker 3: This is truly one of my favorite corporate history stories
Speaker 3: because while everyone else in the world was just blindly
Speaker 3: buying in vidio chips, Google looked at their internal usage
Speaker 3: data and absolutely freaked out.
Speaker 2: This was back around twenty thirteen r twenty thirteen.
Speaker 3: Yeah, yeah. Google engineers did a back of the n
Speaker 3: appting calculation that terrified them. They looked at the rapid
Speaker 3: rise of voice search on Android phones and they realized
Speaker 3: that if every single Android user used voice search for
Speaker 3: just three minutes a day.
Speaker 2: Just three minutes, that is like two quick searches.
Speaker 3: Exactly almost nothing. But they calculated that those three minutes
Speaker 3: would require so much compute power to process the speech
Speaker 3: recognition that it would completely double Google's entire global data center.
Speaker 2: Footprint, doubled their entire footprint.
Speaker 3: They would have had to build twice as many data
Speaker 3: centers as they had built in their entire corporate history
Speaker 3: just to support three minutes of voice search. They realized
Speaker 3: instantly that if they relied on buying standard Intel CPUs
Speaker 3: and Nvidia GPUs, they would literally go bankrupt. The economics
Speaker 3: just flat out did not work at that scale.
Speaker 2: So, in classic Google fashion, they just decided, we will
Speaker 2: build our own hardware.
Speaker 3: They launched a highly secret internal project to build the TPU,
Speaker 3: the tensor processing unit, and their design philosophy was incredibly
Speaker 3: radical compared to Nvidia. Because in Vidia sells GPUs to everyone, right,
Speaker 3: they have to be good at gaming, cryptomning, self driving cars,
Speaker 3: scientific simulations. Google said, we do not care about gaming,
Speaker 3: We do not care about graphics at all. We want
Speaker 3: a chip that does deep learning and absolutely nothing else.
Speaker 2: So they stripped the sports car all the way down
Speaker 2: to the chassis. No AC, no radio, just a massive engine.
Speaker 3: Even the engine itself is totally different. They used a
Speaker 3: specific architecture called a systolic array.
Speaker 2: Systolic like blood pressure, like a.
Speaker 3: Heart beat, exactly like a heart beat, and a normal CPU.
Speaker 3: With GPU, the chip acts kind of like a library.
Speaker 3: You go to the shelf to get a book which
Speaker 3: is your data. You bring it to the desk the processor,
Speaker 3: you read it, and then you walk all the way
Speaker 3: back to put it on the shelf. That walking back
Speaker 3: and forth accessing the memory takes a massive amount of
Speaker 3: energy and time.
Speaker 2: And as we just established, energy and memory are the
Speaker 2: ultimate enemies.
Speaker 3: Here, right, So in a systolic array, you do not
Speaker 3: put the book back on the shelf. You process it,
Speaker 3: and then you immediately hand it to the person sitting
Speaker 3: right next to you. That data physically flows through the
Speaker 3: grid of the chip in a wave. One calculation finishes
Speaker 3: and simply pushes the result directly to the next math unit.
Speaker 3: It heavily mimics a continuous flow of blood through a
Speaker 3: circulatory system.
Speaker 2: So the data just enters one side of the chip,
Speaker 2: flows through this massive grid of math units getting multiplied,
Speaker 2: and just pops out the other side as a finished result.
Speaker 3: Exactly. It drastically reduced the need to constantly access external memory,
Speaker 3: and the result was staggering. That first internal TPU they
Speaker 3: deployed was roughly fifteen thirty times more efficient per watt
Speaker 3: than anything else available on the commercial market at the time.
Speaker 2: That is an insane leap inefficiency. And Google didn't stop there.
Speaker 2: They kept iterating on it.
Speaker 3: Oh yeah. Version two came out in twenty seventeen, and
Speaker 3: that was a huge deal because the first one could
Speaker 3: only run models, they couldn't train them. V two added
Speaker 3: full training capabilities, and today we are on V four
Speaker 3: and V five. They are entirely liquid cooled now and
Speaker 3: they deploy them in massive clusters they call pods.
Speaker 2: Thousands of chips all wired together.
Speaker 3: Right, and this brings up a massive advantage Google has
Speaker 3: over almost everyone else. It is their interconnect system called ICI.
Speaker 2: How is that different from in videos and vlink?
Speaker 3: Because Google completely controls their own data centers, they can
Speaker 3: wire these TPUs directly to each other in what is
Speaker 3: called a torus topology. Think of it like a giant
Speaker 3: three D donut shape. They use direct optical links between
Speaker 3: the chips. They don't have to route the data through
Speaker 3: standard bulky networking switches. It makes those thousands of TPUs
Speaker 3: act flawlessly as one single brain.
Speaker 2: And this is why today when you look at Google,
Speaker 2: they don't really buy in Vidia chips for their core
Speaker 2: internal AID training. They use their own hardware.
Speaker 3: They use TPUs for almost everything. Gemini, which is their
Speaker 3: massive competitor to GPT four, was trained entirely on TPUs.
Speaker 3: It gives Google this incredible vertical integration.
Speaker 2: They own the whole stack.
Speaker 3: They own the chip design, the physical server rack, the
Speaker 3: custom cooling systems, the softwaware framework which is TensorFlow or jx,
Speaker 3: and the final AI model itself. It is essentially the
Speaker 3: Apple iPhone strategy, but applied to a warehouse sized data center.
Speaker 3: They completely control their own destiny.
Speaker 2: Which is an amazing position to be in. So we
Speaker 2: have the raining commercial champion in Nvidia. We have the independent,
Speaker 2: vertically integrated superpower Google. But looking at the market right now,
Speaker 2: it feels like the floodgates have totally opened. Every major
Speaker 2: tech company is suddenly announcing their own custom chip.
Speaker 3: It is the great me too wave of Silicon and
Speaker 3: it is driven by very simple, very ruthless economics. If
Speaker 3: you are Amazon Aws or Microsoft Azure, you are currently
Speaker 3: spending tens of billions of dollars a year buying chips
Speaker 3: from Nvidia. That just vaporizes your profit margins.
Speaker 2: And worse than that, you are entirely dependent on a
Speaker 2: single supplier who literally cannot manufacture the chips fast enough
Speaker 2: to meet your needs exactly.
Speaker 3: So let's look at the broader landscape. You have Amazon Aws,
Speaker 3: who took a very smart, bifurcated approach. They split the
Speaker 3: AI problem in half. They built a chip called Trainium
Speaker 3: specifically for training models, and a separate chip called Inferentia
Speaker 3: for running them.
Speaker 2: We actually really need to pause here and defind this clearly.
Speaker 2: Because the difference between training and inference comes up constantly
Speaker 2: in this space. What is the actual practical difference.
Speaker 3: The best way to think about it is think of
Speaker 3: training as graduate school.
Speaker 2: Okay, graduate school, years of intense work, massive amounts of coffee,
Speaker 2: incredibly expensive exactly.
Speaker 3: Training is the phase where the AI is actively learning
Speaker 3: from scratch. You are feeling it essentially the entire written
Speaker 3: text of the Internet. It takes months of continuous run time.
Speaker 3: You need massive, incredibly expensive compute clusters with thousands of
Speaker 3: GPUs working in perfect unison. You need high precision maths
Speaker 3: so the model can learn tiny nuances. This is the
Speaker 3: graduate school phase, and this is where Nvidia absolutely dominates.
Speaker 2: But eventually the model finishes its exams, it.
Speaker 3: Graduates, It graduates, and it has to go get a job.
Speaker 3: That job is inference, okay. Inference is what happens when
Speaker 3: you open an app, ask chat GPT a question and
Speaker 3: it types out an answer. The model is no longer learning.
Speaker 3: Its weights are frozen. It is simply applying the knowledge
Speaker 3: had already gained in grad school to a new prompt.
Speaker 2: And that happens in real time instantly, right.
Speaker 3: It happens in milliseconds, and it happens billions of times
Speaker 3: a day across the world. And the hardware needs for
Speaker 3: that day job are completely different than grad school. For inference,
Speaker 3: you don't need massive precision. You care about latency, how
Speaker 3: fast can I serve this answer to the user? And
Speaker 3: you care deeply about costs and power consumption.
Speaker 2: Because you only train the model once maybe twice a year,
Speaker 2: but you run inference on it constantly every single second
Speaker 2: of every.
Speaker 3: Day, precisely as AI applications explode and get integrated into
Speaker 3: every piece of software. The overall market for inference chips
Speaker 3: is actually projected to grow much faster than the market
Speaker 3: for training chips. That is exactly why Amazon built Inferentia
Speaker 3: is designed to be a cheap, highly efficient chip just
Speaker 3: for that day job workload. Microsoft is doing the exact
Speaker 3: same thing with their Meya accelerator for Azure.
Speaker 2: But what about Meta? Because Facebook and Instagram they aren't
Speaker 2: selling cloud server space to startups like Amazon and Microsoft do.
Speaker 2: Why are they spending billions to design their own chip.
Speaker 3: Meta is a fascinating outlier here. Their core AI problem
Speaker 3: is fundamentally different from open ai or Google. They aren't
Speaker 3: primarily building conversational text chatbots. Their entire trillion dollar business
Speaker 3: depends on recommendation engines.
Speaker 2: Right, the algorithm deciding exactly which reel or add to
Speaker 2: show me next, so I do not close the app exactly.
Speaker 3: And computationally speaking, a recommendation engine is a very weird
Speaker 3: mathematical problem. It relies on something called embedding tables. These
Speaker 3: are just astronomy massive databases that map out user preferences
Speaker 3: and content features.
Speaker 2: So how does a chip process that differently?
Speaker 3: Well, when you are generating text with a language model,
Speaker 3: the math is very dense and predictable. But when you
Speaker 3: are pulling from embedding tables for an ad recommendation, the
Speaker 3: memory access pattern is random, sparse, and chaotic. You're jumping
Speaker 3: all over the place pulling bits of user history.
Speaker 2: So a standard in video GPU just isn't efficient for
Speaker 2: that kind of chaotic memory access.
Speaker 3: It is massive overkill in some compute areas and horribly
Speaker 3: inefficient in memory access for others. So Meta designed their
Speaker 3: own chip. The MTIA, the Meta Training and Inference Accelerator.
Speaker 3: It is tuned specifically to handle the chaotic memory demands
Speaker 3: of massive embedding tables. It really shows that the industry
Speaker 3: is moving rapidly away from this idea of one chip
Speaker 3: fits all and moving toward the right custom chip for
Speaker 3: the specific math problem.
Speaker 2: And Apple is doing this too, right with the Neural
Speaker 2: Engine on iPhones, but their goal is keeping the AI
Speaker 2: on the physical phone for privacy rather than sending it
Speaker 2: to a cloud data center.
Speaker 3: Exactly on device inference. Everyone is carving out their own
Speaker 3: specialized niche.
Speaker 2: Okay, we cannot talk about chip design without talking about
Speaker 2: the rebels in the room, the startup landscape, because honestly,
Speaker 2: it takes a certain level of sheer insanity to try
Speaker 2: to start a hardware company from scratch to compete against
Speaker 2: a giant like Nvidia, but people are actually doing it.
Speaker 3: It is notoriously the hardest game of Silicon Valley. But
Speaker 3: there are two startups right now that really highlight the
Speaker 3: extreme physical limits we are pushing in chip architecture. Cerebras
Speaker 3: and groc let Us.
Speaker 2: Start with Cerebra Systems. These are the wafer scale guys,
Speaker 2: So to.
Speaker 3: Understand what Cerebras did you have to look at how
Speaker 3: chips are normally made. Normally, a factory takes a silicon wafer,
Speaker 3: which is basically a shiny disc of silicon roughly the
Speaker 3: size of a dinner plate. They print hundreds of identical
Speaker 3: small chips onto that plate, and then they slice the
Speaker 3: plate up into little squares.
Speaker 2: And then you take those little individual squares, put them
Speaker 2: in protective plastic packages and wire them all back together
Speaker 2: on big green motherboard.
Speaker 3: Right, But that wiring them back together part, that is
Speaker 3: the ultimate bottleneck. Moving data across copper wires between different
Speaker 3: chips is painfully slow and burns a ton of energy.
Speaker 3: So the founders of Cerebras just asked a seemingly crazy question,
Speaker 3: why are we cutting the wafer it all?
Speaker 2: They just use the entire dinner plate as one single chip.
Speaker 3: The whole plate the Cerebra's wafer scale engine is a
Speaker 3: single chip roughly the size of an iPad. It contains
Speaker 3: four trillion transistors.
Speaker 2: Four trillion on one piece of silicon. Visually, it is
Speaker 2: just such a cool concept. But practically, I mean, manufacturing
Speaker 2: at the atomic level is not perfect. Usually, if a
Speaker 2: tiny speck of dust ruins one chip on a wafer,
Speaker 2: you just throw that one small square away and keep
Speaker 2: the other three hundred. If the entire wafer is the chip,
Speaker 2: doesn't one single manufacturing defect ruin the whole multi million
Speaker 2: dollar plate.
Speaker 3: You just nailed the exact reason no one ever successfully
Speaker 3: did this before. It's called the yield problem. Cerebras had
Speaker 3: to invent a completely novel networking architecture to route around
Speaker 3: the physically broken parts on the fly. If a microscopic
Speaker 3: section of the wafer has a manufacturing defect, the internal
Speaker 3: software simply ignores it and rotes the data to the
Speaker 3: healthy neighbors.
Speaker 2: It is like having a biological brain with a few
Speaker 2: dead neurons. The overall network just adapts and rotes around
Speaker 2: the damage exactly.
Speaker 3: And the massive benefit of doing this is unprecedented bandwidth.
Speaker 3: Because everything, all the memory, and all the compute cores
Speaker 3: is physically located on the exact same piece of unbroken silicon,
Speaker 3: communication is instantaneous. You never have to wait for data
Speaker 3: to travel across a slow external wire. Is an absolute
Speaker 3: beast for training massive models.
Speaker 2: And then on the other end of the spectrum there
Speaker 2: is grock spelled with a queue I have seen their
Speaker 2: inference demos online where it just prints out paragraphs of
Speaker 2: text instantly. It genuinely feels faster than human thought.
Speaker 3: Grok took a completely different, almost philosophical approach. They looked
Speaker 3: at the modern GPU and said, there is way too
Speaker 3: much chaotic management going on inside this chip. The standard
Speaker 3: GPU dedicates a massive amount of physical hardware and energy
Speaker 3: just to scheduling dynamically deciding which core should do which
Speaker 3: math problem next. Grock stripped all of that dynamic management
Speaker 3: out completely.
Speaker 2: They made it strictly deterministic.
Speaker 3: Yes, deterministic architecture in their system called an LPU or
Speaker 3: language processing unit. The software compiler maps out exactly what
Speaker 3: every single transistor will do it every single clock cycle
Speaker 3: before the program even starts running.
Speaker 2: Like a perfectly choreographed dance routine where everyone knows their
Speaker 2: steps in advance, so you don't need a director shouting
Speaker 2: orders in real time exactly.
Speaker 3: There is zero hesitation. And crucially, they do not use
Speaker 3: the massive slow external memory like HPM that Nvidia uses.
Speaker 3: They exclusively use something called SRAM. What is shram static ram?
Speaker 3: It is a type of incredibly fast memory that lives
Speaker 3: directly on the processor die itself right next to the
Speaker 3: logic gates. It is vastly more expensive to manufacture, and
Speaker 3: you physically cannot fit very much of it on a ship,
Speaker 3: but it completely a limit is the delay of fetching data.
Speaker 3: That is exactly why growth is so blindingly fast at
Speaker 3: generating text tokens. It is perfectly optimized for the inference
Speaker 3: day job, where raw speed is everything.
Speaker 2: But again, all these incredible hardware startups face the exact
Speaker 2: same invisible wall we talked about earlier software.
Speaker 3: The Cuda ecosystem. You can literally build the fastest, most
Speaker 3: beautiful chip in human history, but if a researcher's standard
Speaker 3: PyTorch code does not run on it effortlessly out of
Speaker 3: the box, nobody's going to buy it. And Vidia has
Speaker 3: a fifteen year head start on software momentum. Breaking that
Speaker 3: psychological and technical walk in is arguably harder than breaking
Speaker 3: the laws of physics.
Speaker 2: Speaking of breaking things, let us zoom out to the
Speaker 2: global map, because up until now we've just been talking
Speaker 2: about corporate rivalries. But this isn't just about companies anymore.
Speaker 2: It is about.
Speaker 3: Countries, geopolitics. This is where the story gets genuinely scary.
Speaker 2: You mentioned at the very beginning that the supply chain
Speaker 2: is highly concentrated, walk us through just how fragile this
Speaker 2: map actually is.
Speaker 3: Imagine if all the oil in the world, every single drop,
Speaker 3: was pumped from just three buildings. That is the modern
Speaker 3: semiconductor industry. Let us start with the machine that actually
Speaker 3: prints the chips ASML, headquartered in the Netherlands, they are
Speaker 3: the sole manufacturer of extreme ultraviolet or EUV lithography machines.
Speaker 3: To even begin to understand how insanely complex this machine is,
Speaker 3: these machines generate the UV light needed to print atomic
Speaker 3: level circuits by dropping a microscopic droplet of molten tin
Speaker 3: inside a vacuum cham molten tins falling through a vacuum, yes,
Speaker 3: and as it falls, they hit that microscopic droplet with
Speaker 3: a high powered laser. The impact flattens the droplet into
Speaker 3: a pancake shape, and then a microsecond later they hit
Speaker 3: it again with a second, infinitely more powerful laser. This
Speaker 3: instantly vaporizes the tin into a plasma which amids a
Speaker 3: very specific thirteen point five nanometer wavelength of extreme ultraviolet light.
Speaker 2: That sounds like a weapon from Star Trek. And how
Speaker 2: often is it doing this? Laser plasma explosion fift.
Speaker 3: T thousand times a single second.
Speaker 2: That is just incomprehensible.
Speaker 3: It does that continuously fifty thousand times a second to
Speaker 3: generate enough light to etch physical features onto silicon that
Speaker 3: are literally the size of a few strands of human DNA.
Speaker 2: And you are telling me only one single company on
Speaker 2: planet Earth knows how to build this machine.
Speaker 3: Only one ASML. If their primary factory in the Netherlands
Speaker 3: experience is a major flood or a fire, Moore's law
Speaker 3: simply ends period nobody else can make them.
Speaker 2: And then once you have that two hundred million dollar
Speaker 2: ASML machine. The chips themselves are mostly manufactured in Taiwan.
Speaker 3: By TSMC, the Taiwan semiconductor manufacturing company. They manufacture upwards
Speaker 3: of ninety percent of the world's most advanced logic chips,
Speaker 3: all of Apple's chips, all of Nvidia's advanced chips. They
Speaker 3: all come out of TSMC fabs in Taiwan. This creates
Speaker 3: a massive, glaring geopolitical vulnerability for the rest of the world.
Speaker 3: If China were to blockade the island of Taiwan, or
Speaker 3: if there was just a catastrophic earthquake there, the entire
Speaker 3: global economy would lose its primary computing engine overnight.
Speaker 2: And this sheer panic over that vulnerability is exactly why
Speaker 2: the United States initiated the chip war.
Speaker 3: Exactly the US government looked closely at this supply chain
Speaker 3: map and realized that AI is fundamentally a dual use technology.
Speaker 3: The exact same H one hundred chip that runs a
Speaker 3: friendly customer service chatbot can easily be used to model
Speaker 3: the aerodynamics of hypersonic nuclear missiles or orchestrate massive cyber
Speaker 3: warfare campaigns at a global scale.
Speaker 2: So the US government stepped in with heavy export controls.
Speaker 3: Starting heavily in twenty twenty two, the US Department of
Speaker 3: Commerce outright banned the sale of Viba's absolute top tier
Speaker 3: frontier chips, the eight one hundred and AGE one hundred,
Speaker 3: to any entity inside China. The explicit geopolitical goal was
Speaker 3: to freeze China's AI progress in place, keeping them permanently
Speaker 3: a generation or two behind American labs.
Speaker 2: But in Vidia's a publicly traded company, they obviously did
Speaker 2: not want to lose out on the massive Chinese tech market.
Speaker 3: No China represents billions of dollars in revenue for them,
Speaker 3: so Nvidia engineers went back to the drawing board and
Speaker 3: quickly to design new chips, the eight hundred and the
Speaker 3: eight eight hundred. These were specific, slightly modified versions of
Speaker 3: their flagship chips, designed specifically to comply with the exact
Speaker 3: letter of the new US law.
Speaker 2: How do they manage to cripple it enough to make
Speaker 2: it legal? Did they just turn down the clock speed
Speaker 2: and make the math slower.
Speaker 3: No, And that is what was so incredibly clever about it.
Speaker 3: They kept the raw compute speed exactly the same. The
Speaker 3: H eight hundred could crunch matrix math just as fast
Speaker 3: as the H one hundred. But what they did was
Speaker 3: cut the interconnect speed, the envy link communication speed completely in.
Speaker 2: Half, the wire speed between the chips exactly.
Speaker 3: Why. Because if you cannot connect thousands of chips together
Speaker 3: fast enough, you physically cannot build a cohesive supercomputer. You
Speaker 3: cannot train a massive trillion parameter frontier model like GPT four.
Speaker 3: If the chips can't talk to each other rapidly, the
Speaker 3: data just bottlenecks. You can still do basic inference, you
Speaker 3: can run smaller AI models locally, but you cannot create
Speaker 3: the next generation of frontier AI that.
Speaker 2: Is such a hyper specific surgical restriction, just snipping the
Speaker 2: communication cables.
Speaker 3: Essentially, it was brilliant engineering. But then the US government
Speaker 3: looked at the age eight hundred, realized it was still
Speaker 3: too powerful, and they abruptly tighten the rules again the
Speaker 3: following year to ban even those workaround chips. It has
Speaker 3: become this intense, high stakes game of regulatory cat and mouse.
Speaker 2: And what is China doing in response to all this?
Speaker 2: They surely aren't just throwing their hands up and giving
Speaker 2: up on AI.
Speaker 3: Not at all. And this brings up the massive unintended
Speaker 3: consequence argument that policy experts are debating right now. By
Speaker 3: aggressively cutting China off from the best Western hardware, we
Speaker 3: basically forced them to heavily subsidize and build their own
Speaker 3: completely independent supply chain. Huawei, despite massive sanctions, has released
Speaker 3: an AI chip called the Ascend nine ten B.
Speaker 2: Is it actually as good as an Nvidia H one hundred.
Speaker 3: No, it is generally considered slower and less efficient. But
Speaker 3: is it good enough to train large language models? Yes,
Speaker 3: it absolutely is. And Smick, which is China state back
Speaker 3: to semi conductor manufacturer, is figuring out brilliant ways to
Speaker 3: use older non EUV machines to painstakingly manufacture advanced seven
Speaker 3: nanometer chips.
Speaker 2: So by trying to completely starve them up chips, we
Speaker 2: might have accidentally accelerated the exact thing we were terrified of.
Speaker 2: Full Chinese independence and self sufficiency in cutting edge silicon.
Speaker 3: Is a very real possibility. We've pushed an economic superpower
Speaker 3: into a desperate corner and they are actively engineering their
Speaker 3: way out of it.
Speaker 2: Meanwhile, the US is desperately trying to bring manufacturing back
Speaker 2: onto domestic soil with the Chew GPS Act, which.
Speaker 3: Is a massive piece of industrial policy. The US government
Speaker 3: is spending over fifty two billion dollars to subsidize companies
Speaker 3: like Intel to build fabs in Ohio and TSMC to
Speaker 3: build fabs in Arizona.
Speaker 2: But fifty two billion, I mean earlier you said one
Speaker 2: single modern fab cost twenty billion dollars to build.
Speaker 3: Exactly. The money goes fast, and it's not just about
Speaker 3: the money. It is a severe talent deficit. We haven't
Speaker 3: built leading edge silicon factories at scale in the US
Speaker 3: for decades. We simply do not have the thousands of
Speaker 3: specialized PhD ease or the massive workforce of highly trained
Speaker 3: clean room technicians required you can pour concrete in a year,
Speaker 3: but it takes a generation to rebuild that specialized human capacity.
Speaker 2: Okay, we have covered the geopolitical wall, but there are
Speaker 2: two other massive walls we are currently slamming into at
Speaker 2: full speed, physics and energy.
Speaker 3: Let us tackle physics first. Simply put, we are running
Speaker 3: out of atoms.
Speaker 2: The famous death of Moore's law.
Speaker 3: Right. Moore's Law, for decades depended on our ability to
Speaker 3: just keep physically shrinking transistors so we could pack more
Speaker 3: of them onto the same sized chip. But we are
Speaker 3: currently manufacturing at the three nanometer and two nanometer scale.
Speaker 3: To give you perspective, a single silicon atom is roughly
Speaker 3: point two nanometers.
Speaker 2: Wide, So the physical wires inside these nude chips are
Speaker 2: what ten or fifteen atoms across exactly.
Speaker 3: We're dealing with structures that are literally counted in dozens
Speaker 3: of atoms. And when you get down to that microscopic
Speaker 3: quantum scale, classical physics breaks down. Quantum mechanics completely takes over.
Speaker 3: Electrons stop behaving like predictable solid particles and they start
Speaker 3: behaving like waves.
Speaker 2: And what does an electron wave do inside a transistor?
Speaker 3: It ignores the walls. It does something called quantum tunneling.
Speaker 3: The electron literally teleports through the physical barrier that is
Speaker 3: supposed to hold it back. You get massive electrical leakage,
Speaker 3: you get uncontrollable heat. You physically legally cannot shrink the
Speaker 3: silicon gate much further because the laws of the universe
Speaker 3: won't let you.
Speaker 2: So how on earth do we keep making computers faster
Speaker 2: every year? If we cannot make the microscopic parts any.
Speaker 3: Smaller, we completely change how we package them. This is
Speaker 3: called the chiplet revolution chiplets.
Speaker 2: So instead of putting one giant, monolithic chip, you use
Speaker 2: lots of little pieces.
Speaker 3: Exactly go back to that yield problem we talked about
Speaker 3: with cerebras. If you try to print a giant, complex
Speaker 3: chip the size of a cracker, the mathematical odds of
Speaker 3: a microscopic dust particle or a defect landing somewhere on
Speaker 3: that large surface are extremely high. Maybe only forty percent
Speaker 3: of the chips on your wafer actually work. That makes
Speaker 3: them astronomically expensive to produce.
Speaker 2: But if you intentional print very tiny, simple chips, the
Speaker 2: odds of.
Speaker 3: A defect landing on a tiny footprint are very low.
Speaker 3: You might get a ninety or ninety five percent yield.
Speaker 3: So now companies like AMD and Intel are pivoting entirely.
Speaker 3: They are printing small, specialized functional tiles, a compute tile,
Speaker 3: a separate memory tile, and input output tile, and they
Speaker 3: are stitching them closely together.
Speaker 2: After the fact, it is exactly like building with Legos.
Speaker 3: It is just like legos, but they use incredibly advanced
Speaker 3: packaging techniques, sometimes stacking them in three D so that
Speaker 3: electrically to the software they act exactly like one single
Speaker 3: unified chip. This modular chiplet design is realistically the only
Speaker 3: way we can keep performance scaling up now that traditional
Speaker 3: transistor shrinking is effectively dead.
Speaker 2: And then there is the energy wall. Honestly, this one
Speaker 2: feels the most tangible and immediate to me.
Speaker 3: It is the most immediate constraint on the entire AI
Speaker 3: industry right now. The power draw statistics are genuinely frightening.
Speaker 3: A single standard server rack full of Nvidia H one
Speaker 3: hundreds can draw four eighty to fifty kilowants of continuous.
Speaker 2: Power just to ground that. Compare that to a standard
Speaker 2: suburban house.
Speaker 3: An average American home might draw one to two kilowatts
Speaker 3: on average, So one single metal rack of AI servers
Speaker 3: is using the power of an entire neighborhood. A full
Speaker 3: AI data center is drawing the power of a medium
Speaker 3: sized city.
Speaker 2: I have heard these wild rumors that big tech companies
Speaker 2: are literally looking into buying nuclear power plants just for AI.
Speaker 3: I can assure you it is not a rumor. Microsoft
Speaker 3: is actively hiring directors of nuclear strategy right now. Amazon
Speaker 3: just bought a data center campus in Pennsylvania that is
Speaker 3: physically plugged directly into an existing nuclear power plant. They're
Speaker 3: actively lobbying for SMR small modular reactors because our aging
Speaker 3: national power grids simply cannot support the projected AI energy demand.
Speaker 3: We're talking about future training runs that will require gigawatts
Speaker 3: scale power.
Speaker 2: Gigawatts. We are going to have to construct dedicated nuclear
Speaker 2: power plants just to train GPT six.
Speaker 3: Literally, yes, If we do not fund and mentally improve
Speaker 3: the energetic efficiency of these chips, the entire AI revolution
Speaker 3: will just stall out because we will physically trip the
Speaker 3: breakers of the electric grid.
Speaker 2: That puts an unimaginable premium on completely new ideas. If
Speaker 2: traditional silicon is hitting a quantum wall and energy consumption
Speaker 2: is hitting an absolute ceiling, we clearly need new physics.
Speaker 2: What does the sci fi future of computing actually look like?
Speaker 3: There are three major alternative computing paths being researched right
Speaker 3: now that get me incredibly excited. The first is photonic.
Speaker 2: Computing, computing with actual light instead of electricity, right right now.
Speaker 3: To do math, we forcefully push electrons through solid copper wires.
Speaker 3: That physical friction generates heat, it creates electrical resistance, But photons,
Speaker 3: actual particles of light, have zero mass and experience zero
Speaker 3: electrical resistance. Innovative startups like light Matter are currently building
Speaker 3: chips that use microscopic devices called interferometers.
Speaker 2: How does an interferometer actually calculate anything?
Speaker 3: You take a single laser beam of light and you
Speaker 3: split it down two different tiny optic channels. By precisely
Speaker 3: controlling the phase of those light waves, basically shifting how
Speaker 3: the peaks and valleys of the waves line up when
Speaker 3: they recombine, you can actually perform complex matrix multiplication natively
Speaker 3: in the light itself. You are doing advanced aimth with
Speaker 3: pure light beams moving at the literal speed of light
Speaker 3: while generating almost zero heat.
Speaker 2: That sounds like actual magic. If it is that fast
Speaker 2: and cold, why aren't we ripping out our GPUs and
Speaker 2: doing it right now.
Speaker 3: Because controlling microscopic light waves precisely on a tiny silicon
Speaker 3: chip is outrageously difficult, and the big catches. You still
Speaker 3: have to convert the light signals back into slow electricity
Speaker 3: anytime you want to store the data in standard memory,
Speaker 3: but as a coprocessor just for doing the heavy matrix math.
Speaker 3: It is a massive potential game changer.
Speaker 2: Okay, so light is past number one? What is past
Speaker 2: number two? Neuromorphic computing neuro meeting modeling it after the
Speaker 2: biological brain.
Speaker 3: The human brain is, without a doubt, the most efficient
Speaker 3: computer in the known universe. Your brain runs on about
Speaker 3: twenty watts of power, basically incandescent light bulb, and with
Speaker 3: those twenty watts it manages complex continuous learning, real time
Speaker 3: stereoscopic vision, complex language processing, and fine motor control all simultaneously.
Speaker 2: And meanwhile, an AI trying to do just one of
Speaker 2: those things needs its own personal nuclear reactor. What exactly
Speaker 2: is the biological brain doing so differently?
Speaker 3: It uses an event based architecture. In a normal digital
Speaker 3: computer chip, there is a global clock that ticks billions
Speaker 3: of times a second, and with every single tick, every
Speaker 3: single transistor on the chip gets flooded with power, even
Speaker 3: if it has absolutely zero useful work to do. In
Speaker 3: the biological brain, neurons stay completely dark and only consume
Speaker 3: energy when they actively need to fire. We call these spikes.
Speaker 2: So the brain is just incredibly lazy in a highly
Speaker 2: optimized good way.
Speaker 3: Efficiently lazy. Yes, Neuromorphic chips like Intel's Loy Project or
Speaker 3: IBM's Northpole are attempting to physically mimic this spiking neural
Speaker 3: network architecture in silicon. If the incoming data isn't actively
Speaker 3: changing the relevant parts, the chips simply shut down and
Speaker 3: consume zero power. It is wildly efficient.
Speaker 2: And what is the third alternative path?
Speaker 3: Analog computing?
Speaker 2: Analog like going backwards to the nineteen fifties sort of.
Speaker 3: Yes, modern computing is purely digital. It uses strict ones
Speaker 3: and zeros. It is perfectly precise, but it takes a
Speaker 3: massive amount of physical transistors just to represent a long
Speaker 3: decimal number like zero point three four five two. In
Speaker 3: an analog system, you can represent that exact same complex
Speaker 3: number as a single continuous voltage level, just a fluid
Speaker 3: electrical wave.
Speaker 2: So you can actually do the math natively inside the
Speaker 2: continuous wave itself, rather than breaking it down into digital bits.
Speaker 3: Exactly, you can perform massive matrix multiplications instantly simply by
Speaker 3: passing different electrical currents through a grid of physical resistors.
Speaker 3: It is incredibly fast and saves a massive amount of energy.
Speaker 3: The main downside, however, is noise. Analog systems are affected
Speaker 3: by temperature variations and slight manufacturing defects. They aren't profecularly precise.
Speaker 3: The math gets a little fuzzy.
Speaker 2: But with ai ural networks isn't a little fuzzy. Actually, Okay,
Speaker 2: they are basically probability engines anyway.
Speaker 3: That is the exact billion dollar bet these analog startups
Speaker 3: are making. A neural network doesn't always need mathematically perfect
Speaker 3: precision to tell you it's looking at a picture of
Speaker 3: a cat. It just needs to be close enough. If
Speaker 3: we can accept just a tiny bit of analog noise
Speaker 3: in exchange for a one thousand x gain and energy efficiency,
Speaker 3: the entire industry will gladly make that trade, and ultimately
Speaker 3: that ties into the final evolution of all this, which
Speaker 3: is deep software hardware code design.
Speaker 2: Where the code and the silicon are actually designed together
Speaker 2: from day one.
Speaker 3: Exactly, we are moving away from writing general software for
Speaker 3: general chips. Future architectures will feature algorithms totally explicitly co
Speaker 3: designed alongside the cuts and physical hardware squeezing absolutely every
Speaker 3: last drop of performance out of the silicon.
Speaker 2: Wow. So, just stepping back for a second, we have
Speaker 2: gone from two kids buying a five hundred dollars GTX
Speaker 2: five to eighty in a Best Buy to play video games,
Speaker 2: all the way to massive H one hundred data center.
Speaker 2: We've gone through Google's secret TPU panic, through an escalating
Speaker 2: global trade war over tiny EUV machines, and now we
Speaker 2: are talking about computers running on actual light beams and
Speaker 2: analog waves. It has been quite a journey, it really.
Speaker 3: Has been, and I think the major synthesis here, the
Speaker 3: big takeaway, is crucial. We started this entire conversation by
Speaker 3: observing that software code used to be the undisputed king
Speaker 3: of tech, but today raw access to this unbelievable level
Speaker 3: of compute is all that really matters. And unfortunately that
Speaker 3: access is heavily centralizing.
Speaker 2: Right because you and I cannot just decide to build
Speaker 2: a ten thousand chip H one hundred cluster in our
Speaker 2: garage this weekend. This kind of power is exclusively concentrating
Speaker 2: in the hands of the massive hyperscalers and a select
Speaker 2: few sovereign nations.
Speaker 3: Exactly, the barrier to entry has moved from needing a
Speaker 3: clever idea and a laptop to needing billions of dollars
Speaker 3: in capital in your own dedicated power plant.
Speaker 2: So, as we wrap up what does this all mean
Speaker 2: for the listener, we always like to leave everyone with
Speaker 2: a provocative thought to chew on after the deep dive.
Speaker 3: Say this, the compute race that we've been discussing is
Speaker 3: not just a standard market competition between a few tech giants.
Speaker 3: It is quite literally a race to harness the fundamental
Speaker 3: laws of physics. We are currently pushing atomic structures and
Speaker 3: global energy grids to their absolute breaking point, all in
Speaker 3: an attempt to build synthetic minds.
Speaker 2: And whoever gets there first, whoever wins that race, gets
Speaker 2: all the spoils.
Speaker 3: Because whoever builds the fastest, most scalable, and most efficient
Speaker 3: hardware doesn't just win in a temporary commercial sector. They
Speaker 3: gain the power to permanently decide what artificial intelligence actually
Speaker 3: looks like for the rest of human history. They will
Speaker 3: fully control the fundamental infrastructure of thought for the twenty
Speaker 3: first century. That is the true stakes of what is
Speaker 3: being painstakingly etched onto those tiny silicon wafers in Taiwan.
Speaker 2: The infrastructure of thought. That is a wonderfully heavy concept
Speaker 2: to end on. Thank you so much for breaking all
Speaker 2: this down. We will be back next time with another
Speaker 2: deep dive. Thanks for listening, Thanks
Speaker 3: For having me.