WEBVTT 1 00:00:00.080 --> 00:00:01.480 So if you take a look at the screen in 2 00:00:01.480 --> 00:00:03.319 front of you right now, you might think you're seeing, 3 00:00:03.359 --> 00:00:06.480 I don't know, a deep three dimensional landscape or high 4 00:00:06.480 --> 00:00:07.200 speed motion. 5 00:00:07.280 --> 00:00:10.599 Solid objects that cast actual real shadow. 6 00:00:10.599 --> 00:00:14.279 Right exactly, but you are, well, you're actually hallucinating because 7 00:00:14.320 --> 00:00:17.000 what you are really staring at is a completely flat 8 00:00:17.039 --> 00:00:21.559 grid of tiny, physically static colored lights and they're updating, 9 00:00:21.600 --> 00:00:23.399 you know, sixty to ninety times a second. 10 00:00:23.559 --> 00:00:26.760 It's an incredibly flawless optical illusion, it really is. 11 00:00:26.879 --> 00:00:30.679 And beneath the glass of that screen there is this relentless, 12 00:00:31.120 --> 00:00:35.719 just hyper fast factory. It's an assembly line processing literally 13 00:00:35.880 --> 00:00:40.759 millions of mathematical calculations every single millisecond just to keep 14 00:00:40.759 --> 00:00:41.880 that illusion from shattering. 15 00:00:41.960 --> 00:00:45.759 Yeah, it's arguably one of the most complex slights of 16 00:00:45.799 --> 00:00:48.320 hand in modern human history. We were talking about taking 17 00:00:48.399 --> 00:00:52.719 abstract and visible mathematics inside a processor and transforming it 18 00:00:52.759 --> 00:00:55.840 instantly into a fully realized, immersive visual world. 19 00:00:56.119 --> 00:00:59.119 And that is exactly our mission for this deep dive. 20 00:00:59.200 --> 00:01:01.079 We are going to figure out out how that factory 21 00:01:01.119 --> 00:01:03.799 actually works, like how do we go from raw math 22 00:01:03.920 --> 00:01:07.319 to the real time three D environments of massive video 23 00:01:07.400 --> 00:01:13.560 games or virtual reality or high end architectural simulations. 24 00:01:13.200 --> 00:01:15.439 Right to guide us today, we're pulling from a really 25 00:01:15.480 --> 00:01:19.000 excellent twenty twenty two textbook called Developing Graphics Frameworks with 26 00:01:19.120 --> 00:01:22.719 Java and OpenGL. It's by Lease Demkowski and James Kona. 27 00:01:22.920 --> 00:01:24.920 Yeah, and the source doesn't just talk about the theory, 28 00:01:24.920 --> 00:01:27.560 which is great. It gets right down into the absolute 29 00:01:27.640 --> 00:01:30.120 nuts and bolts of the code that makes a graphics card. 30 00:01:30.239 --> 00:01:30.480 Tick. 31 00:01:31.519 --> 00:01:34.239 Okay, let's unpack this. Before we can start constructing a 32 00:01:34.280 --> 00:01:36.480 three D universe. We kind of have to understand the 33 00:01:36.480 --> 00:01:39.280 canvas we are painting on, right, the engine itself. 34 00:01:39.040 --> 00:01:41.879 Right, the physical canvas. Because the end goal of all 35 00:01:41.920 --> 00:01:46.200 this incredibly dense graphics processing is a concept called rendering, 36 00:01:46.560 --> 00:01:49.760 and the final output of rendering is well, it's surprisingly basic. 37 00:01:49.799 --> 00:01:52.519 It's just a raster, a raster being simply a two 38 00:01:52.599 --> 00:01:55.439 D grid of pixels, just a digital mosaic. 39 00:01:55.079 --> 00:01:58.040 Essentially precisely, and on modern systems, every single one of 40 00:01:58.079 --> 00:02:01.040 those pixels in the mosaic defines its color using just 41 00:02:01.319 --> 00:02:04.959 three floating point numbers, right, And these numbers range from 42 00:02:05.079 --> 00:02:07.879 zero to one, and they represent red, green, and blue light. 43 00:02:08.080 --> 00:02:10.240 The classic RGB triples. 44 00:02:10.120 --> 00:02:13.000 Which, if I remember correctly, is designed specifically to mimic 45 00:02:13.080 --> 00:02:15.240 the photoreceptors in our human. 46 00:02:15.000 --> 00:02:16.439 Eyes, right, yes, exactly that. 47 00:02:16.800 --> 00:02:20.360 So if the computer feeds a pixel the numbers zero, zero, zero, 48 00:02:20.439 --> 00:02:24.000 it means zero red, zero green, zero blue. That is 49 00:02:24.120 --> 00:02:27.039 pure black, no light being emitted at. 50 00:02:26.919 --> 00:02:29.960 All, spot on. And if it feeds it at one one, one, 51 00:02:30.039 --> 00:02:33.199 you get maximum white across all three channels, which your 52 00:02:33.199 --> 00:02:34.919 eye perceives as pure white. 53 00:02:34.919 --> 00:02:36.319 Oh I see, yeah, And. 54 00:02:36.280 --> 00:02:37.960 If you put in point five for all three you 55 00:02:38.000 --> 00:02:40.759 get a medium gray. Basically, every color you have ever 56 00:02:40.800 --> 00:02:43.639 seen on a screen is just a specific mathematical mix 57 00:02:43.639 --> 00:02:47.080 of those three values. But you know how convincing that 58 00:02:47.120 --> 00:02:50.639 final images depends heavily on the balance between resolution and precision. 59 00:02:50.960 --> 00:02:53.479 Now, resolution is something most people are pretty familiar with. 60 00:02:53.840 --> 00:02:56.159 It is just the raw number of pixels on your grid. 61 00:02:56.319 --> 00:02:58.000 A four K screen has more pixels than a ten 62 00:02:58.039 --> 00:03:01.960 eighty p screen. But precision is a bit different, right right. 63 00:03:02.599 --> 00:03:06.479 Precision is about the memory allocated to each individual pixel. 64 00:03:07.000 --> 00:03:09.439 It is how many bits of data you are allowed 65 00:03:09.439 --> 00:03:13.199 to use to describe that RGB mix, and that dictates 66 00:03:13.240 --> 00:03:14.439 your overall color count. 67 00:03:14.759 --> 00:03:17.840 Got it. So, if you look back at retrogaming consoles 68 00:03:17.919 --> 00:03:20.599 like the old eight bit era, the hardware only had 69 00:03:20.719 --> 00:03:23.319 enough memory to use eight bits of precision per pixel. 70 00:03:23.520 --> 00:03:25.520 Yeah, and two to the power of eight means the 71 00:03:25.680 --> 00:03:27.919 entire system could only ever display two hundred and fifty 72 00:03:27.960 --> 00:03:28.800 six different colors. 73 00:03:29.080 --> 00:03:32.439 That is why those old games have that chunky, hyperstylized look. 74 00:03:32.719 --> 00:03:35.800 The system literally didn't have the vocabulary to draw a 75 00:03:35.800 --> 00:03:36.639 smooth gradient. 76 00:03:36.719 --> 00:03:39.680 It just couldn't do it. But today the standard is 77 00:03:39.719 --> 00:03:42.520 twenty four bit precision, which is often called true color. 78 00:03:42.759 --> 00:03:45.960 That gives the computer over sixteen million possible color combinations 79 00:03:46.000 --> 00:03:47.120 for every single pixel. 80 00:03:47.280 --> 00:03:51.759 Wow, which introduces an enormous computational hurdle I'd imagine because 81 00:03:51.759 --> 00:03:54.199 the screen isn't static, it has to update all of 82 00:03:54.199 --> 00:03:57.360 those sixty million colors constantly to create the illusion of. 83 00:03:57.319 --> 00:04:00.879 Motion exactly, which brings us to the frame rate. Movies 84 00:04:00.919 --> 00:04:03.400 in a cinema traditionally run at twenty four frames per second, 85 00:04:03.400 --> 00:04:08.199 which gives them that classic cinematic blur. The computer monitor 86 00:04:08.280 --> 00:04:11.400 or phone you're looking at right now probably refreshes at 87 00:04:11.400 --> 00:04:14.520 sixty frames per second. But the text points out something 88 00:04:14.560 --> 00:04:16.360 fascinating about virtual reality. 89 00:04:16.480 --> 00:04:19.480 Oh yeah, this was wild. If you are building a 90 00:04:19.560 --> 00:04:23.800 VR simulation, sixty frames per second is completely unacceptable. You've 91 00:04:23.959 --> 00:04:28.079 absolutely must hit a minimum of ninety frames per second. 92 00:04:27.759 --> 00:04:30.480 Because in VR, the screen is strapped to your face. 93 00:04:30.519 --> 00:04:33.759 It's literally replacing your entire visual field, right, So if 94 00:04:33.759 --> 00:04:36.600 the simulation drops below ninety frames per second, your physical 95 00:04:36.600 --> 00:04:40.079 head turns, but the digital world updates just a tiny 96 00:04:40.120 --> 00:04:41.199 fraction of a second too. 97 00:04:41.160 --> 00:04:43.399 Late, and your inner ear feels the motion, but your 98 00:04:43.480 --> 00:04:45.480 eyes don't see it instantly exactly. 99 00:04:45.839 --> 00:04:50.839 That sensory conflict causes genuine physical disorientation. It will literally 100 00:04:50.879 --> 00:04:52.199 induce nausea in the user. 101 00:04:52.680 --> 00:04:54.879 So if you are a programmer, you have to ensure 102 00:04:54.920 --> 00:04:57.759 the computer can calculate and draw a new grid of 103 00:04:57.879 --> 00:05:02.680 millions of pixels perfectly ninety times every single second, without fail. 104 00:05:02.800 --> 00:05:04.800 It's a massive undertaking, and in the. 105 00:05:04.759 --> 00:05:08.000 Early days of computing, the CPU, the central processing unit, 106 00:05:08.079 --> 00:05:11.000 had to do all this work. But using a CPU 107 00:05:11.040 --> 00:05:14.160 for graphics is a terrible mismatch. I like to look 108 00:05:14.199 --> 00:05:17.879 at the CPU like it's a single, incredibly brilliant mathematician. 109 00:05:18.000 --> 00:05:19.199 That's a good way to put it, right. 110 00:05:19.319 --> 00:05:23.199 If you give that mathematician an impossibly complex, multi layered 111 00:05:23.279 --> 00:05:26.519 word problem, they will solve it flawlessly. But if you 112 00:05:26.600 --> 00:05:29.639 ask that same brilliant mathematician to take a crayon and 113 00:05:29.759 --> 00:05:32.480 color an eight million individual boxes on a piece of 114 00:05:32.480 --> 00:05:36.120 paper one by one, ninety times a second, they are 115 00:05:36.199 --> 00:05:38.759 going to fail. It is just the wrong tool. 116 00:05:38.519 --> 00:05:41.360 For the job, which is exactly what forced the tech 117 00:05:41.399 --> 00:05:45.399 industry to invent a completely new piece of architecture, the GPU, 118 00:05:45.720 --> 00:05:47.120 or the graphics processing unit. 119 00:05:47.199 --> 00:05:49.519 I really love this piece of hardware history. It was 120 00:05:49.560 --> 00:05:52.279 Sony that actually coined the term GPU back in nineteen 121 00:05:52.319 --> 00:05:55.279 ninety four when they were developing the original PlayStation because 122 00:05:55.319 --> 00:05:57.759 they just needed a name for the specialized circuitry handling 123 00:05:57.759 --> 00:05:58.839 their three D polygons. 124 00:05:58.839 --> 00:06:00.000 Oh wow, nineteen ninety four. 125 00:06:00.160 --> 00:06:03.040 Yeah, and then in Nvidia popularized the term in nineteen 126 00:06:03.079 --> 00:06:05.120 ninety nine with the release of the g Force two 127 00:06:05.199 --> 00:06:05.839 fifty six. 128 00:06:06.160 --> 00:06:09.600 What's fascinating here is the economic driving force behind this 129 00:06:09.720 --> 00:06:13.480 hardware revolution. The relentless demand of the entertainment and video 130 00:06:13.519 --> 00:06:17.959 game industries actually bankrolled this technology. Oh total, millions of 131 00:06:18.040 --> 00:06:21.160 kids wanting cooler explosions and better textures in their games 132 00:06:21.720 --> 00:06:25.360 paid for the research and development of these massive, highly 133 00:06:25.360 --> 00:06:31.279 parallel processing chips, and today that exact same GPU architecture 134 00:06:31.439 --> 00:06:35.279 built for rendering video game dragons is the foundational engine 135 00:06:35.319 --> 00:06:39.759 running vital life saving technology like MRI visualizations right exactly, 136 00:06:39.879 --> 00:06:44.000 MRI is DNA folding, CT scans, climate modeling, AI training. 137 00:06:44.319 --> 00:06:46.319 It all runs on the back of gaming hardware. 138 00:06:46.720 --> 00:06:50.360 Because if the CPU is that single brilliant mathematician, the 139 00:06:50.399 --> 00:06:53.399 GPU is an army of thousands of basic calculators. They 140 00:06:53.439 --> 00:06:56.000 aren't doing advanced calculus, but they don't need to. They 141 00:06:56.040 --> 00:06:59.279 can all work simultaneously in parallel calculating the math for 142 00:06:59.600 --> 00:07:02.439 millions of individual pixels at the exact same moment, and. 143 00:07:02.399 --> 00:07:05.759 That parallel processing power is the entire secret to making 144 00:07:05.839 --> 00:07:08.959 real time graphics possible. But you know, having an army 145 00:07:08.959 --> 00:07:11.000 of calculators is only half the battle. If you don't 146 00:07:11.040 --> 00:07:13.040 organize them perfectly, all you get is chaos. 147 00:07:13.319 --> 00:07:16.040 So how does a programmer actually organize that army? 148 00:07:16.199 --> 00:07:20.680 By utilizing an abstract architectural model known as the graphics pipeline. 149 00:07:20.879 --> 00:07:23.920 Right, It is quite literally a digital assembly line. You 150 00:07:23.959 --> 00:07:27.000 pour raw numbers into one end, and a fully rendered 151 00:07:27.040 --> 00:07:28.560 three D world comes out the other. 152 00:07:28.800 --> 00:07:33.439 Exactly. The pipeline in this OPENNGL framework operates in four 153 00:07:33.480 --> 00:07:36.600 major stages, and the first stage doesn't even happen on 154 00:07:36.639 --> 00:07:37.240 the GPU. 155 00:07:37.439 --> 00:07:38.759 Wait really, yeah, it's the. 156 00:07:38.720 --> 00:07:42.519 Application stage and it actually runs on your CPU. This 157 00:07:42.600 --> 00:07:46.079 is the overarching program, the Java code. In the context 158 00:07:46.160 --> 00:07:49.480 of our source text. It is reading the player's controller input, 159 00:07:49.720 --> 00:07:52.959 calculating the physics of gravity, and then bundling up all 160 00:07:52.959 --> 00:07:55.319 the raw data like the coordinates of a three D character, 161 00:07:55.639 --> 00:07:57.920 and shipping it over the motherboard to the GPU. 162 00:07:58.040 --> 00:08:00.560 Okay, so once the GPU receives that blue print, we 163 00:08:00.720 --> 00:08:03.639 enter stage two, which is geometry processing. This is where 164 00:08:03.639 --> 00:08:06.040 we deal with vertices, which are simply the points in 165 00:08:06.079 --> 00:08:08.079 three D space that make up our shape. Right, the 166 00:08:08.160 --> 00:08:12.120 raw points, But the computer has a massive perspective problem here. 167 00:08:12.639 --> 00:08:15.319 When a three D model is created, its points are 168 00:08:15.360 --> 00:08:19.319 plotted in what the text calls world space. The coordinates 169 00:08:19.360 --> 00:08:22.920 are fixed within the absolute center of the virtual universe. 170 00:08:23.000 --> 00:08:25.399 But the player isn't looking at the entire universe at once. 171 00:08:25.560 --> 00:08:29.279 The players looking through a very specific window, a virtual camera. 172 00:08:29.959 --> 00:08:32.080 So the very first thing the GPU have to do 173 00:08:32.200 --> 00:08:33.399 is a view transformation. 174 00:08:33.600 --> 00:08:34.000 Gotcha. 175 00:08:34.120 --> 00:08:37.440 It takes every single coordinate in the entire virtual world 176 00:08:37.799 --> 00:08:40.519 and recalculates it to figure out where that point is 177 00:08:40.639 --> 00:08:44.440 relative to the camera lens. It moves everything into viewspace. 178 00:08:44.159 --> 00:08:46.440 And that camera doesn't just see everything in front of 179 00:08:46.480 --> 00:08:50.600 it infinitely. The camera's actual viewing region is a specific 180 00:08:50.639 --> 00:08:52.480 three D shape called a frustum. 181 00:08:52.799 --> 00:08:53.840 Yes, the frustum. 182 00:08:54.120 --> 00:08:57.399 Imagine a traditional four sided pyramid, but you take a 183 00:08:57.440 --> 00:09:00.600 sword and slice off the very top point. Your virtual 184 00:09:00.639 --> 00:09:03.279 camera is sitting at that sliced off top, looking down 185 00:09:03.360 --> 00:09:05.679 through the expanding base of the pyramid. 186 00:09:05.519 --> 00:09:08.360 And only the objects that happen to fall inside the 187 00:09:08.399 --> 00:09:11.360 mathematical boundaries of that specific frustum volume are going to 188 00:09:11.360 --> 00:09:12.120 be processed further. 189 00:09:12.399 --> 00:09:15.679 Here's where it gets really interesting. The concept of clipping 190 00:09:16.399 --> 00:09:19.840 the computer literally discards and throws away the math for 191 00:09:19.919 --> 00:09:21.120 any object outside that. 192 00:09:21.080 --> 00:09:22.759 Frustum, just completely ignores it. 193 00:09:22.919 --> 00:09:25.840 Yeah, if you are playing a game and your character 194 00:09:25.919 --> 00:09:28.960 is standing in the middle of a sprawling, densely populated 195 00:09:29.039 --> 00:09:32.159 virtual city, but you turn your camera to stare closely 196 00:09:32.240 --> 00:09:36.039 at a blank brick wall, the GPU is not rendering 197 00:09:36.039 --> 00:09:39.000 the rest of the city behind you. It completely stops 198 00:09:39.000 --> 00:09:42.639 calculating the traffic, the skyscrapers, the pedestrians behind your back. 199 00:09:43.039 --> 00:09:46.639 It is a total illusion. The world only exists exactly 200 00:09:46.639 --> 00:09:49.440 where you are looking the exact millisecond you look at it. 201 00:09:49.440 --> 00:09:52.120 It is a brilliant optimization. If the GPU tried to 202 00:09:52.159 --> 00:09:54.559 process the geometry for an entire city when you can 203 00:09:54.600 --> 00:09:57.840 only see a wall, system would instantly crash, so it 204 00:09:57.840 --> 00:09:58.679 clips away. 205 00:09:58.440 --> 00:09:59.960 The unseen world maankes total sense. 206 00:10:00.200 --> 00:10:02.799 Then whatever is left inside the frustum goes through a 207 00:10:02.840 --> 00:10:07.279 projection transformation. It mathematically squishes that three D volume into 208 00:10:07.279 --> 00:10:10.039 a standardized two D cube shape so the next stage 209 00:10:10.039 --> 00:10:13.279 can understand it. This is usually a perspective projection, which 210 00:10:13.399 --> 00:10:16.519 artificially compresses things that are further away so they look smaller, 211 00:10:16.919 --> 00:10:18.919 mimicking how human eyes perceive depth. 212 00:10:19.120 --> 00:10:22.399 Okay, so we have our points. They are positioned relative 213 00:10:22.440 --> 00:10:26.240 to the camera, and we have applied realistic perspective, but 214 00:10:26.399 --> 00:10:29.519 they are still just invisible math points. How do we 215 00:10:29.559 --> 00:10:30.559 actually paint them? 216 00:10:30.759 --> 00:10:33.519 That takes us to stage three rasterization. 217 00:10:33.960 --> 00:10:38.639 Rasterization, the GPU groups these floating points into primitives, which 218 00:10:38.639 --> 00:10:43.120 are basic structural shapes, mostly lines or triangles. Any three 219 00:10:43.200 --> 00:10:45.720 D model you have ever seen, whether it is a 220 00:10:45.799 --> 00:10:49.240 hyperrealistic sports car or an alien monster, is really just 221 00:10:49.279 --> 00:10:52.559 a hollow mesh made out of thousands or millions of 222 00:10:52.639 --> 00:10:54.600 tiny flat triangles. 223 00:10:54.360 --> 00:10:58.039 And during rasterization, the GPU analyzes every single triangle in 224 00:10:58.080 --> 00:11:00.639 the scene and mathematically maps it all onto the final 225 00:11:00.679 --> 00:11:03.080 two D grade of your screen. It figures out exactly 226 00:11:03.120 --> 00:11:06.519 which pixels fall inside the borders of that specific triangle. 227 00:11:06.200 --> 00:11:06.879 That's filling it in. 228 00:11:07.159 --> 00:11:09.919 Essentially, yes, for every single pixel inside, it generates a 229 00:11:09.960 --> 00:11:12.360 package of data called a fragment. A fragment is basically 230 00:11:12.399 --> 00:11:15.039 a pixel in waiting. It holds all the geographic data 231 00:11:15.080 --> 00:11:17.679 needed to eventually figure out what color that pixel should. 232 00:11:17.480 --> 00:11:21.559 Be, which brings us to stage four pixel processing. This 233 00:11:21.600 --> 00:11:25.080 is where the actual coloring happens. But reading through the text, 234 00:11:25.200 --> 00:11:28.399 there is this incredibly elegant piece of math that happens 235 00:11:28.399 --> 00:11:29.559 here called interpolation. 236 00:11:29.879 --> 00:11:34.879 Oh. Interpolation is arguably where the GPU shows its true genius. 237 00:11:35.320 --> 00:11:39.240 Imagine you are programming a single flat triangle on the screen. 238 00:11:39.600 --> 00:11:41.519 You don't have to write code to color every single 239 00:11:41.559 --> 00:11:45.120 pixel inside that shape. You only assign colors to the 240 00:11:45.240 --> 00:11:47.159 three corner points the vertices. 241 00:11:47.519 --> 00:11:49.639 So let's say I make the top corner pure red, 242 00:11:49.799 --> 00:11:52.159 the bottom left corner pure green, and the bottom right 243 00:11:52.200 --> 00:11:55.039 corner pure blue. I haven't told the computer what to 244 00:11:55.039 --> 00:11:56.440 do with the empty space in the middle. 245 00:11:56.639 --> 00:11:59.960 You haven't, but the GPU handles it automatically during rosterization. 246 00:12:00.600 --> 00:12:01.960 It looks at a pixel right in the middle of 247 00:12:02.039 --> 00:12:04.840 the triangle, measures how physically close that pixel is to 248 00:12:04.879 --> 00:12:07.200 the red corner, the green corner, in the blue corner, 249 00:12:07.679 --> 00:12:10.639 and then calculates a weighted average of those three colors 250 00:12:10.679 --> 00:12:11.600 based on the distance. 251 00:12:11.799 --> 00:12:14.639 It just silently does the gradient mass exactly a pixel 252 00:12:14.679 --> 00:12:16.440 a little closer to the top corner it gets a 253 00:12:16.480 --> 00:12:20.519 little more red mixedtem It creates a perfectly smooth, blended 254 00:12:20.600 --> 00:12:23.840 rainbow across the entire face of the triangle, and the 255 00:12:23.879 --> 00:12:26.399 programmer didn't have to write a single line of code 256 00:12:26.440 --> 00:12:27.720 for the interior pixels. 257 00:12:27.919 --> 00:12:31.679 The hardware just handles the transition naturally. But you know, 258 00:12:32.080 --> 00:12:36.360 while a single, flat, colorful triangle is nice, it isn't 259 00:12:36.360 --> 00:12:38.960 the three D world. The real challenge is depth. 260 00:12:38.919 --> 00:12:39.639 Oh for sure. 261 00:12:39.840 --> 00:12:43.639 When you have thousands of objects overlapping each other, how 262 00:12:43.679 --> 00:12:45.840 does a flat two D screen know what is in 263 00:12:45.879 --> 00:12:47.679 front and what is hiding in the back? 264 00:12:47.879 --> 00:12:50.200 Right, Because if I write code to draw a massive 265 00:12:50.200 --> 00:12:52.240 gray mountain, and then I write code to draw a 266 00:12:52.240 --> 00:12:54.559 green tree standing in front of the mountain, the computer 267 00:12:54.639 --> 00:12:56.840 needs to know not to paint the mountain over the tree. 268 00:12:57.000 --> 00:12:57.240 Right. 269 00:12:57.480 --> 00:12:59.840 This is managed by something called the frame buffer, which 270 00:12:59.879 --> 00:13:03.320 is the chunk of memory holding the image data as 271 00:13:03.320 --> 00:13:05.879 it is being built. The frame buffer has a color 272 00:13:05.919 --> 00:13:09.279 buffer to hold the RGB values, but crucially, it also 273 00:13:09.399 --> 00:13:10.480 has a depth buffer. 274 00:13:10.639 --> 00:13:13.080 Yeah. The depth buffer is essentially a bouncer at the 275 00:13:13.080 --> 00:13:15.279 door of the pixel grid. It keeps track of the 276 00:13:15.279 --> 00:13:18.200 distance of every single fragment from the camera. Whenever the 277 00:13:18.240 --> 00:13:20.919 GPU tries to color a pixel, the bouncer steps in. 278 00:13:21.480 --> 00:13:23.840 The GP says, hey, I have a green pixel for 279 00:13:23.879 --> 00:13:26.320 a tree, and its distance from the camera is ten. 280 00:13:27.120 --> 00:13:29.519 The depth buffer checks its records and says, well, the 281 00:13:29.559 --> 00:13:32.120 current color saved in this spot is a gray mountain, 282 00:13:32.399 --> 00:13:36.039 but its distance is fifty. Because ten is closer than fifty, 283 00:13:36.279 --> 00:13:39.519 the bouncer allows the GPU to overwrite the gray with green, 284 00:13:40.000 --> 00:13:41.720 and it updates the depth record to ten. 285 00:13:42.159 --> 00:13:46.039 That is a rock solid system for solid, opaque objects. 286 00:13:46.279 --> 00:13:49.080 But as I was reading this, I realized transparency must 287 00:13:49.080 --> 00:13:51.000 be an absolute nightmare for a programmer. 288 00:13:51.080 --> 00:13:51.559 Oh. It is. 289 00:13:51.960 --> 00:13:53.840 If I am looking through a glass window, I need 290 00:13:53.879 --> 00:13:55.480 to see the window itself, but I also need to 291 00:13:55.480 --> 00:13:57.759 see the mountain behind it. So the color buffer isn't 292 00:13:57.799 --> 00:14:01.200 just RGB, it is actually RGB where the A stands 293 00:14:01.240 --> 00:14:03.840 for alpha the transparency channel exactly. 294 00:14:04.000 --> 00:14:07.080 An alpha value of zero is completely invisible, and one 295 00:14:07.120 --> 00:14:09.399 is totally solid. If you set the glass alpha to 296 00:14:09.519 --> 00:14:12.639 point six, the GPU knows to blend sixty percent of 297 00:14:12.639 --> 00:14:15.440 the glass color with forty percent of whatever color is 298 00:14:15.440 --> 00:14:17.039 already sitting behind it in the buffer. 299 00:14:17.240 --> 00:14:20.960 But wait, if the depth buffer just ruthlessly overwrites whatever 300 00:14:21.039 --> 00:14:23.679 is further away. What happens if I mess up the 301 00:14:23.759 --> 00:14:26.679 drawing order in my code? So if I code a 302 00:14:26.720 --> 00:14:30.039 glass window and an opaque brick wall, but tell the 303 00:14:30.080 --> 00:14:33.639 computer to draw the glass first, does the wall just disappear? 304 00:14:33.960 --> 00:14:37.000 This raises an important question, and it highlights what we 305 00:14:37.039 --> 00:14:40.399 can call the transparency paradox. If you draw the glass first, 306 00:14:40.440 --> 00:14:43.879 the illusion shatters completely. The brick wall will simply cease 307 00:14:43.919 --> 00:14:44.399 to exist. 308 00:14:44.440 --> 00:14:48.240 It completely vanishes. Why the computer knows the wall is there? 309 00:14:48.519 --> 00:14:51.840 The application knows, but the depth buffer doesn't care. You 310 00:14:51.919 --> 00:14:55.159 draw the glass window first. The depth buffer updates its record. 311 00:14:55.799 --> 00:14:58.879 We now have an object to distance five A mill 312 00:14:58.960 --> 00:15:01.600 a second later, the GPU tries to draw the opaque 313 00:15:01.600 --> 00:15:04.399 brick wall at distance twenty. The bouncer checks the record. 314 00:15:04.559 --> 00:15:07.799 Oh no, the bouncer sees that distance five is already occupied. 315 00:15:08.039 --> 00:15:10.720 Distance twenty is further away, meaning it must be blocked. 316 00:15:11.120 --> 00:15:13.559 The bouncer immediately discards all the map and colors for 317 00:15:13.600 --> 00:15:16.080 the wall. The depth buffer has no way of knowing 318 00:15:16.120 --> 00:15:18.320 that the object at distance five happens to be see 319 00:15:18.360 --> 00:15:18.960 through glass. 320 00:15:18.960 --> 00:15:20.759 I just threw away the data. Didn't even try to 321 00:15:20.799 --> 00:15:21.399 blend it. 322 00:15:21.399 --> 00:15:25.320 It can't, the data is gone. To prevent this, programmers 323 00:15:25.320 --> 00:15:29.480 have to enforce an incredibly tedious strict rendering rule. You 324 00:15:29.559 --> 00:15:33.000 must draw every single opaque object in your entire virtual 325 00:15:33.039 --> 00:15:35.720 world first, get them all settled in the depth buffer. 326 00:15:36.120 --> 00:15:39.799 Only then can you begin drawing the transparent objects, the water, 327 00:15:40.200 --> 00:15:42.519 the glass, the smoke, and you must sort them and 328 00:15:42.600 --> 00:15:44.840 draw them strictly in order from the furthest away to 329 00:15:44.879 --> 00:15:45.919 the closest. 330 00:15:45.600 --> 00:15:47.960 Which means if a player is walking through a dense 331 00:15:48.039 --> 00:15:51.840 virtual forest with multiple layers of transparent fog and overlapping 332 00:15:51.879 --> 00:15:56.080 glass windows, the CPU has to mathematically recalculate the distance 333 00:15:56.120 --> 00:15:58.639 of every single pane of glass and puff of smoke 334 00:15:58.720 --> 00:16:01.600 relative to the player every single frame, just to figure 335 00:16:01.600 --> 00:16:03.879 out what order to hand them to the GPU. That 336 00:16:04.240 --> 00:16:04.919 is staggering. 337 00:16:04.960 --> 00:16:08.000 He is a massive computational bottleneck which is why poorly 338 00:16:08.039 --> 00:16:12.320 optimized transparency often causes video games to lag. But we've 339 00:16:12.360 --> 00:16:15.000 been talking about all these rules to frustum, the interpolation, 340 00:16:15.120 --> 00:16:18.799 the depth buffering. How does a human programmer actually communicate 341 00:16:18.840 --> 00:16:22.440 these specific rules to that army of calculators on the GPU. 342 00:16:22.679 --> 00:16:25.240 We use shads, and the name is a bit of 343 00:16:25.240 --> 00:16:28.360 a historical artifact. Originally they were just little programs written 344 00:16:28.399 --> 00:16:31.120 to figure out how light shaded an object, but today 345 00:16:31.159 --> 00:16:34.360 they are the absolute core of graphics programming. They are 346 00:16:34.440 --> 00:16:37.720 small custom programs that you compile and send to run 347 00:16:37.759 --> 00:16:42.120 directly on the GPU itself. In this textbooks framework, there 348 00:16:42.120 --> 00:16:44.840 are two main ones, vertex shaders which handle where the 349 00:16:44.840 --> 00:16:48.240 geometry points are in space, and fragment shaders, which calculate 350 00:16:48.279 --> 00:16:50.600 the final RGBA color output. 351 00:16:50.799 --> 00:16:53.399 And the language you use to write these custom programs 352 00:16:53.399 --> 00:16:57.399 in open gl is called GLSL, the OPENNGL shading language. 353 00:16:57.759 --> 00:17:00.360 It looks very similar to the C programming language, but 354 00:17:00.399 --> 00:17:03.799 it is built entirely around processing geometry and color rapidly. 355 00:17:03.919 --> 00:17:07.039 To do that, GLSL uses these very specific data structures 356 00:17:07.079 --> 00:17:09.839 called vectors. In the code, you'll see things like vec two, 357 00:17:10.000 --> 00:17:12.440 vec three, or vec four. These are basically just tight 358 00:17:12.440 --> 00:17:15.240 little packages of two, three, or four floating point numbers. 359 00:17:15.680 --> 00:17:17.799 But what I found amazing was the syntax they use 360 00:17:17.880 --> 00:17:19.240 to access that data. 361 00:17:19.119 --> 00:17:22.359 The dot notation syntax, or the life saver for cognitive load. 362 00:17:22.640 --> 00:17:25.319 It really is. If a programmer creates a vec fourzo 363 00:17:25.359 --> 00:17:28.000 a group of four numbers, they can pull those numbers 364 00:17:28.000 --> 00:17:30.160 out using different letters depending on the context of what 365 00:17:30.240 --> 00:17:32.720 they're doing. If they're working with a position in three 366 00:17:32.839 --> 00:17:35.720 D space, they can type xyz or W to get 367 00:17:35.759 --> 00:17:38.920 the coordinates. But if that exact same package of four 368 00:17:39.000 --> 00:17:41.160 numbers is being used for color, they can just type 369 00:17:41.480 --> 00:17:45.720 RGB or A. It is pulling from the exact same 370 00:17:45.799 --> 00:17:49.400 memory slots, but the language allows the programmer to change 371 00:17:49.400 --> 00:17:51.240 the labels on the fly so they don't lose their 372 00:17:51.279 --> 00:17:53.079 mind trying to remember what the numbers represent. 373 00:17:53.440 --> 00:17:56.279 It is a very human centric design in an otherwise 374 00:17:56.359 --> 00:18:00.559 highly machine oriented process. But you touched on something brussal there, 375 00:18:01.000 --> 00:18:02.640 the W coordinate for position. 376 00:18:02.799 --> 00:18:04.839 Yes, I am so glad you brought that up, because 377 00:18:04.839 --> 00:18:06.519 I was staring at the text trying to figure out 378 00:18:06.519 --> 00:18:10.279 this mathematical quirk. Physical space only has three dimensions up 379 00:18:10.359 --> 00:18:13.000 and down, left and right, forward and backward. The x, 380 00:18:13.119 --> 00:18:15.440 y and z axis. Wait, physical space is three D. 381 00:18:15.799 --> 00:18:17.640 Why on earth do we need a fourth coordinate, the 382 00:18:17.759 --> 00:18:20.160 W and a vex four just to specify a position. 383 00:18:20.440 --> 00:18:23.119 It is entirely a mathematical trick, and it is the 384 00:18:23.240 --> 00:18:26.880 lynchpin of real time graphics. Yeah. In linear algebra, if 385 00:18:26.920 --> 00:18:28.640 you want to scale a three D object to make 386 00:18:28.640 --> 00:18:31.839 it bigger or rotate it, you can easily do that 387 00:18:32.079 --> 00:18:34.759 by multiplying its x, y, and z coordinates by a 388 00:18:34.799 --> 00:18:37.359 three x three grid of numbers called a matrix. Okay, 389 00:18:38.400 --> 00:18:42.839 but here's the problem. You cannot translate an object, meaning 390 00:18:42.880 --> 00:18:45.519 you can't slide it to a new location using a 391 00:18:45.559 --> 00:18:49.519 three x three matrix multiplication. It physically doesn't work. Nor 392 00:18:49.559 --> 00:18:52.319 can you do that complex perspective projection we talked about. 393 00:18:52.359 --> 00:18:54.000 Wait, so if the math doesn't work, you'd have to 394 00:18:54.000 --> 00:18:58.119 write different slower types of mathematical equations for every different 395 00:18:58.160 --> 00:19:00.839 type of movement, like one equation to spin, a different one. 396 00:19:00.680 --> 00:19:04.480 To sluck exactly, which would bottleneck the GPU instantly. But 397 00:19:04.559 --> 00:19:08.680 mathematicians realize to workaround. If you artificially invent a fourth dimension, 398 00:19:08.799 --> 00:19:11.000 this W coordinate, and you just set it to the 399 00:19:11.119 --> 00:19:13.279 number one, you can upgrade your math from a three 400 00:19:13.400 --> 00:19:15.759 x three matrix to a four by four matrix, and 401 00:19:15.839 --> 00:19:19.440