Speaker 1: So if you take a look at the screen in
Speaker 1: front of you right now, you might think you're seeing,
Speaker 1: I don't know, a deep three dimensional landscape or high
Speaker 1: speed motion.
Speaker 2: Solid objects that cast actual real shadow.
Speaker 1: Right exactly, but you are, well, you're actually hallucinating because
Speaker 1: what you are really staring at is a completely flat
Speaker 1: grid of tiny, physically static colored lights and they're updating,
Speaker 1: you know, sixty to ninety times a second.
Speaker 2: It's an incredibly flawless optical illusion, it really is.
Speaker 1: And beneath the glass of that screen there is this relentless,
Speaker 1: just hyper fast factory. It's an assembly line processing literally
Speaker 1: millions of mathematical calculations every single millisecond just to keep
Speaker 1: that illusion from shattering.
Speaker 2: Yeah, it's arguably one of the most complex slights of
Speaker 2: hand in modern human history. We were talking about taking
Speaker 2: abstract and visible mathematics inside a processor and transforming it
Speaker 2: instantly into a fully realized, immersive visual world.
Speaker 1: And that is exactly our mission for this deep dive.
Speaker 1: We are going to figure out out how that factory
Speaker 1: actually works, like how do we go from raw math
Speaker 1: to the real time three D environments of massive video
Speaker 1: games or virtual reality or high end architectural simulations.
Speaker 2: Right to guide us today, we're pulling from a really
Speaker 2: excellent twenty twenty two textbook called Developing Graphics Frameworks with
Speaker 2: Java and OpenGL. It's by Lease Demkowski and James Kona.
Speaker 1: Yeah, and the source doesn't just talk about the theory,
Speaker 1: which is great. It gets right down into the absolute
Speaker 1: nuts and bolts of the code that makes a graphics card.
Speaker 2: Tick.
Speaker 1: Okay, let's unpack this. Before we can start constructing a
Speaker 1: three D universe. We kind of have to understand the
Speaker 1: canvas we are painting on, right, the engine itself.
Speaker 2: Right, the physical canvas. Because the end goal of all
Speaker 2: this incredibly dense graphics processing is a concept called rendering,
Speaker 2: and the final output of rendering is well, it's surprisingly basic.
Speaker 1: It's just a raster, a raster being simply a two
Speaker 1: D grid of pixels, just a digital mosaic.
Speaker 2: Essentially precisely, and on modern systems, every single one of
Speaker 2: those pixels in the mosaic defines its color using just
Speaker 2: three floating point numbers, right, And these numbers range from
Speaker 2: zero to one, and they represent red, green, and blue light.
Speaker 2: The classic RGB triples.
Speaker 1: Which, if I remember correctly, is designed specifically to mimic
Speaker 1: the photoreceptors in our human.
Speaker 2: Eyes, right, yes, exactly that.
Speaker 1: So if the computer feeds a pixel the numbers zero, zero, zero,
Speaker 1: it means zero red, zero green, zero blue. That is
Speaker 1: pure black, no light being emitted at.
Speaker 2: All, spot on. And if it feeds it at one one, one,
Speaker 2: you get maximum white across all three channels, which your
Speaker 2: eye perceives as pure white.
Speaker 1: Oh I see, yeah, And.
Speaker 2: If you put in point five for all three you
Speaker 2: get a medium gray. Basically, every color you have ever
Speaker 2: seen on a screen is just a specific mathematical mix
Speaker 2: of those three values. But you know how convincing that
Speaker 2: final images depends heavily on the balance between resolution and precision.
Speaker 1: Now, resolution is something most people are pretty familiar with.
Speaker 1: It is just the raw number of pixels on your grid.
Speaker 1: A four K screen has more pixels than a ten
Speaker 1: eighty p screen. But precision is a bit different, right right.
Speaker 2: Precision is about the memory allocated to each individual pixel.
Speaker 2: It is how many bits of data you are allowed
Speaker 2: to use to describe that RGB mix, and that dictates
Speaker 2: your overall color count.
Speaker 1: Got it. So, if you look back at retrogaming consoles
Speaker 1: like the old eight bit era, the hardware only had
Speaker 1: enough memory to use eight bits of precision per pixel.
Speaker 2: Yeah, and two to the power of eight means the
Speaker 2: entire system could only ever display two hundred and fifty
Speaker 2: six different colors.
Speaker 1: That is why those old games have that chunky, hyperstylized look.
Speaker 1: The system literally didn't have the vocabulary to draw a
Speaker 1: smooth gradient.
Speaker 2: It just couldn't do it. But today the standard is
Speaker 2: twenty four bit precision, which is often called true color.
Speaker 2: That gives the computer over sixteen million possible color combinations
Speaker 2: for every single pixel.
Speaker 1: Wow, which introduces an enormous computational hurdle I'd imagine because
Speaker 1: the screen isn't static, it has to update all of
Speaker 1: those sixty million colors constantly to create the illusion of.
Speaker 2: Motion exactly, which brings us to the frame rate. Movies
Speaker 2: in a cinema traditionally run at twenty four frames per second,
Speaker 2: which gives them that classic cinematic blur. The computer monitor
Speaker 2: or phone you're looking at right now probably refreshes at
Speaker 2: sixty frames per second. But the text points out something
Speaker 2: fascinating about virtual reality.
Speaker 1: Oh yeah, this was wild. If you are building a
Speaker 1: VR simulation, sixty frames per second is completely unacceptable. You've
Speaker 1: absolutely must hit a minimum of ninety frames per second.
Speaker 2: Because in VR, the screen is strapped to your face.
Speaker 2: It's literally replacing your entire visual field, right, So if
Speaker 2: the simulation drops below ninety frames per second, your physical
Speaker 2: head turns, but the digital world updates just a tiny
Speaker 2: fraction of a second too.
Speaker 1: Late, and your inner ear feels the motion, but your
Speaker 1: eyes don't see it instantly exactly.
Speaker 2: That sensory conflict causes genuine physical disorientation. It will literally
Speaker 2: induce nausea in the user.
Speaker 1: So if you are a programmer, you have to ensure
Speaker 1: the computer can calculate and draw a new grid of
Speaker 1: millions of pixels perfectly ninety times every single second, without fail.
Speaker 2: It's a massive undertaking, and in the.
Speaker 1: Early days of computing, the CPU, the central processing unit,
Speaker 1: had to do all this work. But using a CPU
Speaker 1: for graphics is a terrible mismatch. I like to look
Speaker 1: at the CPU like it's a single, incredibly brilliant mathematician.
Speaker 2: That's a good way to put it, right.
Speaker 1: If you give that mathematician an impossibly complex, multi layered
Speaker 1: word problem, they will solve it flawlessly. But if you
Speaker 1: ask that same brilliant mathematician to take a crayon and
Speaker 1: color an eight million individual boxes on a piece of
Speaker 1: paper one by one, ninety times a second, they are
Speaker 1: going to fail. It is just the wrong tool.
Speaker 2: For the job, which is exactly what forced the tech
Speaker 2: industry to invent a completely new piece of architecture, the GPU,
Speaker 2: or the graphics processing unit.
Speaker 1: I really love this piece of hardware history. It was
Speaker 1: Sony that actually coined the term GPU back in nineteen
Speaker 1: ninety four when they were developing the original PlayStation because
Speaker 1: they just needed a name for the specialized circuitry handling
Speaker 1: their three D polygons.
Speaker 2: Oh wow, nineteen ninety four.
Speaker 1: Yeah, and then in Nvidia popularized the term in nineteen
Speaker 1: ninety nine with the release of the g Force two
Speaker 1: fifty six.
Speaker 2: What's fascinating here is the economic driving force behind this
Speaker 2: hardware revolution. The relentless demand of the entertainment and video
Speaker 2: game industries actually bankrolled this technology. Oh total, millions of
Speaker 2: kids wanting cooler explosions and better textures in their games
Speaker 2: paid for the research and development of these massive, highly
Speaker 2: parallel processing chips, and today that exact same GPU architecture
Speaker 2: built for rendering video game dragons is the foundational engine
Speaker 2: running vital life saving technology like MRI visualizations right exactly,
Speaker 2: MRI is DNA folding, CT scans, climate modeling, AI training.
Speaker 2: It all runs on the back of gaming hardware.
Speaker 1: Because if the CPU is that single brilliant mathematician, the
Speaker 1: GPU is an army of thousands of basic calculators. They
Speaker 1: aren't doing advanced calculus, but they don't need to. They
Speaker 1: can all work simultaneously in parallel calculating the math for
Speaker 1: millions of individual pixels at the exact same moment, and.
Speaker 2: That parallel processing power is the entire secret to making
Speaker 2: real time graphics possible. But you know, having an army
Speaker 2: of calculators is only half the battle. If you don't
Speaker 2: organize them perfectly, all you get is chaos.
Speaker 1: So how does a programmer actually organize that army?
Speaker 2: By utilizing an abstract architectural model known as the graphics pipeline.
Speaker 1: Right, It is quite literally a digital assembly line. You
Speaker 1: pour raw numbers into one end, and a fully rendered
Speaker 1: three D world comes out the other.
Speaker 2: Exactly. The pipeline in this OPENNGL framework operates in four
Speaker 2: major stages, and the first stage doesn't even happen on
Speaker 2: the GPU.
Speaker 1: Wait really, yeah, it's the.
Speaker 2: Application stage and it actually runs on your CPU. This
Speaker 2: is the overarching program, the Java code. In the context
Speaker 2: of our source text. It is reading the player's controller input,
Speaker 2: calculating the physics of gravity, and then bundling up all
Speaker 2: the raw data like the coordinates of a three D character,
Speaker 2: and shipping it over the motherboard to the GPU.
Speaker 1: Okay, so once the GPU receives that blue print, we
Speaker 1: enter stage two, which is geometry processing. This is where
Speaker 1: we deal with vertices, which are simply the points in
Speaker 1: three D space that make up our shape. Right, the
Speaker 1: raw points, But the computer has a massive perspective problem here.
Speaker 1: When a three D model is created, its points are
Speaker 1: plotted in what the text calls world space. The coordinates
Speaker 1: are fixed within the absolute center of the virtual universe.
Speaker 2: But the player isn't looking at the entire universe at once.
Speaker 2: The players looking through a very specific window, a virtual camera.
Speaker 2: So the very first thing the GPU have to do
Speaker 2: is a view transformation.
Speaker 1: Gotcha.
Speaker 2: It takes every single coordinate in the entire virtual world
Speaker 2: and recalculates it to figure out where that point is
Speaker 2: relative to the camera lens. It moves everything into viewspace.
Speaker 1: And that camera doesn't just see everything in front of
Speaker 1: it infinitely. The camera's actual viewing region is a specific
Speaker 1: three D shape called a frustum.
Speaker 2: Yes, the frustum.
Speaker 1: Imagine a traditional four sided pyramid, but you take a
Speaker 1: sword and slice off the very top point. Your virtual
Speaker 1: camera is sitting at that sliced off top, looking down
Speaker 1: through the expanding base of the pyramid.
Speaker 2: And only the objects that happen to fall inside the
Speaker 2: mathematical boundaries of that specific frustum volume are going to
Speaker 2: be processed further.
Speaker 1: Here's where it gets really interesting. The concept of clipping
Speaker 1: the computer literally discards and throws away the math for
Speaker 1: any object outside that.
Speaker 2: Frustum, just completely ignores it.
Speaker 1: Yeah, if you are playing a game and your character
Speaker 1: is standing in the middle of a sprawling, densely populated
Speaker 1: virtual city, but you turn your camera to stare closely
Speaker 1: at a blank brick wall, the GPU is not rendering
Speaker 1: the rest of the city behind you. It completely stops
Speaker 1: calculating the traffic, the skyscrapers, the pedestrians behind your back.
Speaker 1: It is a total illusion. The world only exists exactly
Speaker 1: where you are looking the exact millisecond you look at it.
Speaker 2: It is a brilliant optimization. If the GPU tried to
Speaker 2: process the geometry for an entire city when you can
Speaker 2: only see a wall, system would instantly crash, so it
Speaker 2: clips away.
Speaker 1: The unseen world maankes total sense.
Speaker 2: Then whatever is left inside the frustum goes through a
Speaker 2: projection transformation. It mathematically squishes that three D volume into
Speaker 2: a standardized two D cube shape so the next stage
Speaker 2: can understand it. This is usually a perspective projection, which
Speaker 2: artificially compresses things that are further away so they look smaller,
Speaker 2: mimicking how human eyes perceive depth.
Speaker 1: Okay, so we have our points. They are positioned relative
Speaker 1: to the camera, and we have applied realistic perspective, but
Speaker 1: they are still just invisible math points. How do we
Speaker 1: actually paint them?
Speaker 2: That takes us to stage three rasterization.
Speaker 1: Rasterization, the GPU groups these floating points into primitives, which
Speaker 1: are basic structural shapes, mostly lines or triangles. Any three
Speaker 1: D model you have ever seen, whether it is a
Speaker 1: hyperrealistic sports car or an alien monster, is really just
Speaker 1: a hollow mesh made out of thousands or millions of
Speaker 1: tiny flat triangles.
Speaker 2: And during rasterization, the GPU analyzes every single triangle in
Speaker 2: the scene and mathematically maps it all onto the final
Speaker 2: two D grade of your screen. It figures out exactly
Speaker 2: which pixels fall inside the borders of that specific triangle.
Speaker 1: That's filling it in.
Speaker 2: Essentially, yes, for every single pixel inside, it generates a
Speaker 2: package of data called a fragment. A fragment is basically
Speaker 2: a pixel in waiting. It holds all the geographic data
Speaker 2: needed to eventually figure out what color that pixel should.
Speaker 1: Be, which brings us to stage four pixel processing. This
Speaker 1: is where the actual coloring happens. But reading through the text,
Speaker 1: there is this incredibly elegant piece of math that happens
Speaker 1: here called interpolation.
Speaker 2: Oh. Interpolation is arguably where the GPU shows its true genius.
Speaker 2: Imagine you are programming a single flat triangle on the screen.
Speaker 2: You don't have to write code to color every single
Speaker 2: pixel inside that shape. You only assign colors to the
Speaker 2: three corner points the vertices.
Speaker 1: So let's say I make the top corner pure red,
Speaker 1: the bottom left corner pure green, and the bottom right
Speaker 1: corner pure blue. I haven't told the computer what to
Speaker 1: do with the empty space in the middle.
Speaker 2: You haven't, but the GPU handles it automatically during rosterization.
Speaker 2: It looks at a pixel right in the middle of
Speaker 2: the triangle, measures how physically close that pixel is to
Speaker 2: the red corner, the green corner, in the blue corner,
Speaker 2: and then calculates a weighted average of those three colors
Speaker 2: based on the distance.
Speaker 1: It just silently does the gradient mass exactly a pixel
Speaker 1: a little closer to the top corner it gets a
Speaker 1: little more red mixedtem It creates a perfectly smooth, blended
Speaker 1: rainbow across the entire face of the triangle, and the
Speaker 1: programmer didn't have to write a single line of code
Speaker 1: for the interior pixels.
Speaker 2: The hardware just handles the transition naturally. But you know,
Speaker 2: while a single, flat, colorful triangle is nice, it isn't
Speaker 2: the three D world. The real challenge is depth.
Speaker 1: Oh for sure.
Speaker 2: When you have thousands of objects overlapping each other, how
Speaker 2: does a flat two D screen know what is in
Speaker 2: front and what is hiding in the back?
Speaker 1: Right, Because if I write code to draw a massive
Speaker 1: gray mountain, and then I write code to draw a
Speaker 1: green tree standing in front of the mountain, the computer
Speaker 1: needs to know not to paint the mountain over the tree.
Speaker 2: Right.
Speaker 1: This is managed by something called the frame buffer, which
Speaker 1: is the chunk of memory holding the image data as
Speaker 1: it is being built. The frame buffer has a color
Speaker 1: buffer to hold the RGB values, but crucially, it also
Speaker 1: has a depth buffer.
Speaker 2: Yeah. The depth buffer is essentially a bouncer at the
Speaker 2: door of the pixel grid. It keeps track of the
Speaker 2: distance of every single fragment from the camera. Whenever the
Speaker 2: GPU tries to color a pixel, the bouncer steps in.
Speaker 2: The GP says, hey, I have a green pixel for
Speaker 2: a tree, and its distance from the camera is ten.
Speaker 2: The depth buffer checks its records and says, well, the
Speaker 2: current color saved in this spot is a gray mountain,
Speaker 2: but its distance is fifty. Because ten is closer than fifty,
Speaker 2: the bouncer allows the GPU to overwrite the gray with green,
Speaker 2: and it updates the depth record to ten.
Speaker 1: That is a rock solid system for solid, opaque objects.
Speaker 1: But as I was reading this, I realized transparency must
Speaker 1: be an absolute nightmare for a programmer.
Speaker 2: Oh. It is.
Speaker 1: If I am looking through a glass window, I need
Speaker 1: to see the window itself, but I also need to
Speaker 1: see the mountain behind it. So the color buffer isn't
Speaker 1: just RGB, it is actually RGB where the A stands
Speaker 1: for alpha the transparency channel exactly.
Speaker 2: An alpha value of zero is completely invisible, and one
Speaker 2: is totally solid. If you set the glass alpha to
Speaker 2: point six, the GPU knows to blend sixty percent of
Speaker 2: the glass color with forty percent of whatever color is
Speaker 2: already sitting behind it in the buffer.
Speaker 1: But wait, if the depth buffer just ruthlessly overwrites whatever
Speaker 1: is further away. What happens if I mess up the
Speaker 1: drawing order in my code? So if I code a
Speaker 1: glass window and an opaque brick wall, but tell the
Speaker 1: computer to draw the glass first, does the wall just disappear?
Speaker 2: This raises an important question, and it highlights what we
Speaker 2: can call the transparency paradox. If you draw the glass first,
Speaker 2: the illusion shatters completely. The brick wall will simply cease
Speaker 2: to exist.
Speaker 1: It completely vanishes. Why the computer knows the wall is there?
Speaker 2: The application knows, but the depth buffer doesn't care. You
Speaker 2: draw the glass window first. The depth buffer updates its record.
Speaker 2: We now have an object to distance five A mill
Speaker 2: a second later, the GPU tries to draw the opaque
Speaker 2: brick wall at distance twenty. The bouncer checks the record.
Speaker 2: Oh no, the bouncer sees that distance five is already occupied.
Speaker 2: Distance twenty is further away, meaning it must be blocked.
Speaker 2: The bouncer immediately discards all the map and colors for
Speaker 2: the wall. The depth buffer has no way of knowing
Speaker 2: that the object at distance five happens to be see
Speaker 2: through glass.
Speaker 1: I just threw away the data. Didn't even try to
Speaker 1: blend it.
Speaker 2: It can't, the data is gone. To prevent this, programmers
Speaker 2: have to enforce an incredibly tedious strict rendering rule. You
Speaker 2: must draw every single opaque object in your entire virtual
Speaker 2: world first, get them all settled in the depth buffer.
Speaker 2: Only then can you begin drawing the transparent objects, the water,
Speaker 2: the glass, the smoke, and you must sort them and
Speaker 2: draw them strictly in order from the furthest away to
Speaker 2: the closest.
Speaker 1: Which means if a player is walking through a dense
Speaker 1: virtual forest with multiple layers of transparent fog and overlapping
Speaker 1: glass windows, the CPU has to mathematically recalculate the distance
Speaker 1: of every single pane of glass and puff of smoke
Speaker 1: relative to the player every single frame, just to figure
Speaker 1: out what order to hand them to the GPU. That
Speaker 1: is staggering.
Speaker 2: He is a massive computational bottleneck which is why poorly
Speaker 2: optimized transparency often causes video games to lag. But we've
Speaker 2: been talking about all these rules to frustum, the interpolation,
Speaker 2: the depth buffering. How does a human programmer actually communicate
Speaker 2: these specific rules to that army of calculators on the GPU.
Speaker 1: We use shads, and the name is a bit of
Speaker 1: a historical artifact. Originally they were just little programs written
Speaker 1: to figure out how light shaded an object, but today
Speaker 1: they are the absolute core of graphics programming. They are
Speaker 1: small custom programs that you compile and send to run
Speaker 1: directly on the GPU itself. In this textbooks framework, there
Speaker 1: are two main ones, vertex shaders which handle where the
Speaker 1: geometry points are in space, and fragment shaders, which calculate
Speaker 1: the final RGBA color output.
Speaker 2: And the language you use to write these custom programs
Speaker 2: in open gl is called GLSL, the OPENNGL shading language.
Speaker 2: It looks very similar to the C programming language, but
Speaker 2: it is built entirely around processing geometry and color rapidly.
Speaker 1: To do that, GLSL uses these very specific data structures
Speaker 1: called vectors. In the code, you'll see things like vec two,
Speaker 1: vec three, or vec four. These are basically just tight
Speaker 1: little packages of two, three, or four floating point numbers.
Speaker 1: But what I found amazing was the syntax they use
Speaker 1: to access that data.
Speaker 2: The dot notation syntax, or the life saver for cognitive load.
Speaker 1: It really is. If a programmer creates a vec fourzo
Speaker 1: a group of four numbers, they can pull those numbers
Speaker 1: out using different letters depending on the context of what
Speaker 1: they're doing. If they're working with a position in three
Speaker 1: D space, they can type xyz or W to get
Speaker 1: the coordinates. But if that exact same package of four
Speaker 1: numbers is being used for color, they can just type
Speaker 1: RGB or A. It is pulling from the exact same
Speaker 1: memory slots, but the language allows the programmer to change
Speaker 1: the labels on the fly so they don't lose their
Speaker 1: mind trying to remember what the numbers represent.
Speaker 2: It is a very human centric design in an otherwise
Speaker 2: highly machine oriented process. But you touched on something brussal there,
Speaker 2: the W coordinate for position.
Speaker 1: Yes, I am so glad you brought that up, because
Speaker 1: I was staring at the text trying to figure out
Speaker 1: this mathematical quirk. Physical space only has three dimensions up
Speaker 1: and down, left and right, forward and backward. The x,
Speaker 1: y and z axis. Wait, physical space is three D.
Speaker 1: Why on earth do we need a fourth coordinate, the
Speaker 1: W and a vex four just to specify a position.
Speaker 2: It is entirely a mathematical trick, and it is the
Speaker 2: lynchpin of real time graphics. Yeah. In linear algebra, if
Speaker 2: you want to scale a three D object to make
Speaker 2: it bigger or rotate it, you can easily do that
Speaker 2: by multiplying its x, y, and z coordinates by a
Speaker 2: three x three grid of numbers called a matrix. Okay,
Speaker 2: but here's the problem. You cannot translate an object, meaning
Speaker 2: you can't slide it to a new location using a
Speaker 2: three x three matrix multiplication. It physically doesn't work. Nor
Speaker 2: can you do that complex perspective projection we talked about.
Speaker 1: Wait, so if the math doesn't work, you'd have to
Speaker 1: write different slower types of mathematical equations for every different
Speaker 1: type of movement, like one equation to spin, a different one.
Speaker 2: To sluck exactly, which would bottleneck the GPU instantly. But
Speaker 2: mathematicians realize to workaround. If you artificially invent a fourth dimension,
Speaker 2: this W coordinate, and you just set it to the
Speaker 2: number one, you can upgrade your math from a three
Speaker 2: x three matrix to a four by four matrix, and
Speaker 2: by doing that, suddenly translation, rotation, scaling, and camera perspective
Speaker 2: can all be mathematically combined into one single four x
Speaker 2: four master matrix.
Speaker 1: That is wild. The CPU just crunches all the player's
Speaker 1: movements into one master four by four grid of numbers,
Speaker 1: hands it to the GPU, and the GPU just multiplies
Speaker 1: the object by that one matrix, and all the spinning, sliding,
Speaker 1: and zooming happens in a single stroke. It is a
Speaker 1: math hack that makes real time three D computationally affordable.
Speaker 2: It is a profound optimization. So at this point, the
Speaker 2: GPU has done the matrix, math clipped the frustum, interpolated
Speaker 2: the colors, resolve the depth buffer, and run the GLSL shaders.
Speaker 2: We have a finished frame, right, But how does the
Speaker 2: overarching application safely deliver that finished image to your monitor
Speaker 2: sixty times a second without it looking like a glitchy,
Speaker 2: flashing mess.
Speaker 1: This comes down to the application life cycle. The overarching
Speaker 1: Java program has a very specific heartbeat, start up, the
Speaker 1: main loop, and ship down. Startup gets everything loaded into memory,
Speaker 1: but the main loop is where the game lives over
Speaker 1: and over again. It processes the player's input updates the
Speaker 1: scenes map and triggers the render.
Speaker 2: And to prevent the rendering from becoming a glitchy mess,
Speaker 2: the framework utilizes a technique called double buffering.
Speaker 1: So what does this all mean? The best way I
Speaker 1: could picture double buffering in my head was to imagine
Speaker 1: a live theater production. You have the main stage where
Speaker 1: the curtain is open and the audience is smoothly watching
Speaker 1: the actors perform the current scene, but hidden right behind
Speaker 1: a heavy velvet curtain, a crew of stage hands is
Speaker 1: frantically dragging props around, pinking scenery, and setting up the
Speaker 1: next scene in total chaos. Right right, the audience never
Speaker 1: sees the mess. The moment the new scene is perfectly ready,
Speaker 1: the director drops the curtain on the main stage, instantly
Speaker 1: swaps the two sets and opens the curtain again.
Speaker 2: That is a perfect analogy. In the computer, you have
Speaker 2: two separate image buffers in the memory. The screen is
Speaker 2: currently reading and displaying the front buffer the main stage. Meanwhile,
Speaker 2: the TPU is acting as the stage hands, frankly writing
Speaker 2: millions of pixels of new data to the back buffer.
Speaker 1: Because if you only had one buffer, the screen would
Speaker 1: be trying to display the image while the GPU was
Speaker 1: still halfway through painting it. You'd see the top half
Speaker 1: of the screen showing the new frame where you turned
Speaker 1: your character's head, but the bottom half of the screen
Speaker 1: would still be stuck on.
Speaker 2: The old frame, and that is a visual artifact called
Speaker 2: screen tearing. By using double buffering and carefully synchronizing the
Speaker 2: swapping of those buffers with the physical refresh rate of
Speaker 2: your monitor, you ensure the user never sees the stage
Speaker 2: hands at work. They only ever see the flawless finished product.
Speaker 1: And when those stage hands are actually drawing the shapes.
Speaker 1: The code uses things called vertex array objects to manage
Speaker 1: all the geometric data. The programmer triggers a drawing by
Speaker 1: using commands like gl draw rays, but they have to
Speaker 1: specify exactly how the points connect. Yes, they can use
Speaker 1: something like a line loop which just plays connect the
Speaker 1: dots from point A to B two C and loops
Speaker 1: back to A to draw a hollow outline. Or they
Speaker 1: can use a triangle fan, which picks a single point
Speaker 1: as a central hub and draws solid triangles radiating outward
Speaker 1: like spokes on a wheel to fill the shape in completely.
Speaker 2: If we connect this to the bigger picture, this entire architecture,
Speaker 2: from the jav application loop down to the low level
Speaker 2: of triangle fans and GLSL vectors is about maintaining an
Speaker 2: impossibly delicate synchronization. It is the synchronization between cold hardware
Speaker 2: constraints and the fluid nature of human perception. If any
Speaker 2: single part of that typeline stalls, the framework drops, the
Speaker 2: illusion breaks, and you are jarringly reminded that you are
Speaker 2: just looking at a machine.
Speaker 1: It really is incredible when you step back and look
Speaker 1: at it. To wrap this all up for you, just
Speaker 1: think about the epic journey a single pixel takes before
Speaker 1: it reaches your eye. It starts as a mundane variable
Speaker 1: assignment in a JavaScript. It gets bundled into a buffer,
Speaker 1: shot across the motherboard to the GPU, and multiplied by
Speaker 1: a four x four matrix just to figure out where
Speaker 1: it exists in the universe. It survives the ruthless clipping
Speaker 1: of the frustum, gets grouped into a triangle, has its
Speaker 1: colors automatically interpolated, gets tested against a depth bouncer, blended
Speaker 1: with transparency values, and finally loaded into a hidden back buffer,
Speaker 1: just waiting for the curtain to drop so it can
Speaker 1: be swapped onto your screen, all of that millions of
Speaker 1: times a second, just so a glowing yellow pixel can
Speaker 1: appear in a video game.
Speaker 2: It is a staggering amount of invisible computation, and it
Speaker 2: leads you with a thought to moll over long after
Speaker 2: you turn your screen off. Today, as these GLSL shaders
Speaker 2: become exponentially more sophisticated, and as GPUs learn to trace
Speaker 2: the actual physical path of light rays in real time,
Speaker 2: at what point does the mathematical hallucination of a three
Speaker 2: D reality on a flat two D screen become completely
Speaker 2: indistinguishable from the real world