WEBVTT 1 00:00:01.199 --> 00:00:06.200 Welcome to the Sentient Code, where intelligence is engineered, autonomy 2 00:00:06.280 --> 00:00:10.439 is emerging, and a line between human and machine grows thinner. 3 00:00:10.800 --> 00:00:15.359 Each episode, we decode the algorithms, explore the robotics, and 4 00:00:15.439 --> 00:00:19.000 examine the ideas shaping the future of artificial minds. 5 00:00:23.839 --> 00:00:27.239 Hello, and welcome back. We are looking at something today 6 00:00:27.559 --> 00:00:30.559 that on the surface feels a little bit like a 7 00:00:30.600 --> 00:00:34.320 classic David and Goliath's story. But the more I look 8 00:00:34.359 --> 00:00:38.560 at the research, the more I start to think maybe 9 00:00:38.880 --> 00:00:42.719 David is actually being secretly funded by another Goliath. 10 00:00:42.960 --> 00:00:47.280 That is a surprisingly, surprisingly accurate way to put it. 11 00:00:47.280 --> 00:00:51.119 It's definitely not the fairy tale version people might imagine, right. 12 00:00:51.119 --> 00:00:54.399 It's way more complicated. Yeah, we're talking about what is 13 00:00:54.439 --> 00:00:57.200 and I don't think this is an exaggeration arguably the 14 00:00:57.280 --> 00:01:00.960 defining technological rivalry of our era. It is absolutely talking 15 00:01:00.960 --> 00:01:03.640 about the battle between open source AI and big tech. 16 00:01:03.840 --> 00:01:07.760 It's huge, and honestly, the landscape is shifting so fast 17 00:01:07.799 --> 00:01:09.519 that by the time you're listening to this, the battle 18 00:01:09.519 --> 00:01:10.920 lines might have already moved again. 19 00:01:11.200 --> 00:01:13.359 That's how fast this whole thing is going. And I 20 00:01:13.359 --> 00:01:15.519 think for a lot of people, you know, the narrative 21 00:01:15.560 --> 00:01:18.879 feels pretty simple It's the scrappy underdogs, right, the hackers, 22 00:01:18.879 --> 00:01:22.560 the independent researchers, the global open source community. 23 00:01:22.200 --> 00:01:25.920 Right, the rebels, exactly, the rebels versus the big bad 24 00:01:26.079 --> 00:01:29.439 corporate giants. But looking at the material we have today, 25 00:01:29.799 --> 00:01:32.760 the reality is, well, it's a lot messier. 26 00:01:32.959 --> 00:01:36.319 It's so much messier and frankly, so much more interesting 27 00:01:36.920 --> 00:01:39.959 because this isn't just about who sells the most software 28 00:01:40.120 --> 00:01:43.000 or you know, who has the highest stock price. This 29 00:01:43.079 --> 00:01:46.040 is really about the fundamental structure of power. It's about 30 00:01:46.120 --> 00:01:50.000 who holds the knowledge, who owns the infrastructure, and ultimately 31 00:01:50.120 --> 00:01:51.879 who gets to shape the twenty first century. 32 00:01:51.959 --> 00:01:54.920 That is a heavy way to open the structure of power. 33 00:01:55.000 --> 00:01:56.799 I think it needs to be, because we have to 34 00:01:56.799 --> 00:02:00.480 move past this simple binary idea of open versus closed. 35 00:02:00.719 --> 00:02:03.840 It's not that simple. Both sides are winning in different arenas, 36 00:02:03.840 --> 00:02:06.439 and you know, both sides are facing these huge, almost 37 00:02:06.480 --> 00:02:08.719 existential threats they might not even fully see yet. 38 00:02:08.759 --> 00:02:10.560 Okay, so let's set the board. Then. If we're looking 39 00:02:10.560 --> 00:02:12.759 at this as a kind of conflict or maybe a game, 40 00:02:13.000 --> 00:02:16.719 who are the main players. Let's start with the heavy hitters. 41 00:02:16.680 --> 00:02:19.039 The big tech camp For sure, these are the names 42 00:02:19.080 --> 00:02:21.800 everyone knows, you have open Ai, which is of course 43 00:02:22.159 --> 00:02:25.280 heavily heavily backed by Microsoft the tens of billions of 44 00:02:25.319 --> 00:02:26.759 dollars camp that's a good name for. 45 00:02:26.719 --> 00:02:30.400 Them, exactly. Then you have Google deep Mind, which is 46 00:02:30.639 --> 00:02:34.280 the force behind the Gemini models. You've got Anthropic, which 47 00:02:34.360 --> 00:02:36.840 is a really interesting one because it was founded by 48 00:02:36.919 --> 00:02:41.080 ex OpenAI people, but now it's backed by both Amazon and. 49 00:02:40.960 --> 00:02:42.319 Google at a bit of a tangled web. 50 00:02:42.360 --> 00:02:44.800 There, a very tangled web. And then of course you 51 00:02:44.919 --> 00:02:49.280 have the quiet giant Apple doing its own thing, mostly 52 00:02:49.319 --> 00:02:51.439 focused on on device intelligence. 53 00:02:51.479 --> 00:02:54.560 Okay, so that's the fortress, that's the establishment. And on 54 00:02:54.599 --> 00:02:56.960 the other side, who are the rebels. 55 00:02:56.680 --> 00:02:59.840 The open source ecosystem? And this is where it gets much. 56 00:03:00.000 --> 00:03:04.439 It's more diffuse. It's not one company. It's a global network. 57 00:03:04.759 --> 00:03:07.680 You have incredible startups like Mistral and Friends. You have 58 00:03:07.719 --> 00:03:10.879 academic researchers and universities all over the world. You have hobbyists, 59 00:03:11.080 --> 00:03:14.560 and then you have this massive community coalescing on platforms 60 00:03:14.639 --> 00:03:15.800 like hugging face. 61 00:03:15.960 --> 00:03:18.560 Hugging Face I always love that name. It just sounds 62 00:03:18.599 --> 00:03:21.719 so friendly and innocent for something that is essentially an 63 00:03:21.800 --> 00:03:23.639 arms depot for AI models. 64 00:03:23.800 --> 00:03:26.159 It is a charming name. Isn't it. But the scale 65 00:03:26.719 --> 00:03:30.719 is anything but charming. It's deadly serious. They host hundreds 66 00:03:30.759 --> 00:03:35.120 of thousands of models. We're talking billions of downloads. It is, 67 00:03:35.479 --> 00:03:38.919 for all intents and purposes, the GitHub of the AI revolution. 68 00:03:39.199 --> 00:03:41.639 So we have the giants and we have the swarm, 69 00:03:42.560 --> 00:03:45.719 and the mission for us today is to figure out, well, 70 00:03:45.719 --> 00:03:49.879 who's actually winning, because looking through all the source material, 71 00:03:49.919 --> 00:03:53.960 the answer seems to depend entirely on how you define winning. 72 00:03:54.240 --> 00:03:57.479 Precisely, Winning the performance benchmark is one thing. Winning the 73 00:03:57.599 --> 00:04:01.280 enterprise market is another. Winning the hearts minds of developers 74 00:04:01.280 --> 00:04:03.039 that's a third. They're all different battles. 75 00:04:03.080 --> 00:04:05.479 So let's start with the giants. Let's talk about why 76 00:04:05.560 --> 00:04:08.439 big tech is so dominant. What is this fortress they've built? 77 00:04:08.759 --> 00:04:09.879 What are the walls made of? 78 00:04:10.039 --> 00:04:12.919 The first, and by far the thickest wall is what's 79 00:04:12.960 --> 00:04:14.000 called the compute mote. 80 00:04:14.039 --> 00:04:16.399 The compute mote. Okay, I want to pause on this 81 00:04:16.439 --> 00:04:18.959 because I think when people hear expensive, they think, okay, 82 00:04:19.040 --> 00:04:21.639 like a really nice car expensive, or maybe a house expensive. 83 00:04:22.000 --> 00:04:24.800 But looking at the numbers here, we are talking about 84 00:04:24.879 --> 00:04:26.879 nation state level spending, aren't we. 85 00:04:26.639 --> 00:04:31.079 We really really are to even begin to visualize what 86 00:04:31.120 --> 00:04:35.240 it takes to train a frontier model, the absolute bleeding 87 00:04:35.399 --> 00:04:39.399 edge systems like GPT four, Gemini Ultra. You have to 88 00:04:39.439 --> 00:04:42.000 start with the hardware, right, we aren't talking about the 89 00:04:42.000 --> 00:04:45.720 graphics card in your gaming PC. We're talking about the 90 00:04:45.759 --> 00:04:49.000 top of the line in Vidia H one hundreds, the 91 00:04:49.160 --> 00:04:50.879 AI chips. 92 00:04:50.519 --> 00:04:52.920 The chips that there's a global shortage of the ones 93 00:04:52.959 --> 00:04:54.120 everyone is fighting. 94 00:04:53.800 --> 00:04:57.000 Over the very same These chips are, you know, almost 95 00:04:57.000 --> 00:05:00.000 a strategic national resource at this point. A single one 96 00:05:00.000 --> 00:05:01.879 one of these costs as much as a luxury car. 97 00:05:02.000 --> 00:05:02.759 A single chip. 98 00:05:02.879 --> 00:05:05.240 Yes, now imagine you need to buy it, not one, 99 00:05:05.480 --> 00:05:08.600 but maybe twenty five thousand of them, twenty five thousand, 100 00:05:08.720 --> 00:05:11.199 and you have to tie them all together with specialized 101 00:05:11.240 --> 00:05:14.720 high speed networking cabling that costs more than most people's houses. 102 00:05:15.199 --> 00:05:17.639 You stick all that in a massive, custom built data 103 00:05:17.639 --> 00:05:21.000 center and you run it at one hundred percent capacity 104 00:05:21.079 --> 00:05:22.040 for ninety. 105 00:05:21.759 --> 00:05:23.920 Days straight, ninety days, NonStop. 106 00:05:23.519 --> 00:05:25.639 NonStop, twenty four to seven. And this is where the 107 00:05:25.680 --> 00:05:28.240 moat gets very, very physical. It's not just about buying 108 00:05:28.240 --> 00:05:30.120 the chips. It's about the physics, the heat. 109 00:05:30.360 --> 00:05:32.240 I saw a note here about thermal densities. 110 00:05:32.240 --> 00:05:34.040 That what you mean, that's exactly. 111 00:05:34.079 --> 00:05:34.199 It. 112 00:05:34.240 --> 00:05:36.319 If you put that much compute in one room, the 113 00:05:36.360 --> 00:05:39.160 air doesn't just get hot, it turns into a blast furnace. 114 00:05:39.480 --> 00:05:43.199 The racks would literally melt. So you need these incredible 115 00:05:43.600 --> 00:05:48.399 industrial scale liquid cooling systems, pipes of chilled water running 116 00:05:48.680 --> 00:05:50.759 to every single server rack. 117 00:05:51.040 --> 00:05:52.879 So it's not just a server room. It's a power 118 00:05:52.920 --> 00:05:55.720 plant and a plumbing project it is, and you. 119 00:05:55.759 --> 00:05:58.600 Need access to a power grid that can handle the 120 00:05:58.680 --> 00:06:02.439 load of a small city. We're talking hundreds of megawatts 121 00:06:02.480 --> 00:06:06.639 of sustained power. The electricity bill alone for a single 122 00:06:06.720 --> 00:06:09.879 training run, just the electricity can be tens of millions 123 00:06:09.920 --> 00:06:10.399 of dollars. 124 00:06:10.439 --> 00:06:12.120 Get to wrap my head around that. So when we 125 00:06:12.160 --> 00:06:14.199 say the kid in the garage is locked out of 126 00:06:14.199 --> 00:06:16.800 this game, it's not because the GID isn't smart enough. 127 00:06:16.959 --> 00:06:19.879 It's because the garage would literally melt and the entire 128 00:06:19.920 --> 00:06:21.879 neighborhood grid would blow a transformer. 129 00:06:21.920 --> 00:06:24.600 The garage would melt, the transformer would blow, and they'd 130 00:06:24.600 --> 00:06:28.079 get a bill for ten million dollars. So, yes, this 131 00:06:28.240 --> 00:06:30.959 is why the list of companies that can actually train 132 00:06:31.040 --> 00:06:34.759 a Frontier model from scratch is so incredibly short. It's 133 00:06:34.800 --> 00:06:40.079 basically Microsoft, Google, Meta, and Amazon. The barrier to entry 134 00:06:40.199 --> 00:06:43.160 is a capital expenditure of billions of dollars before you've 135 00:06:43.199 --> 00:06:45.279 written a single line of useful code. 136 00:06:45.439 --> 00:06:48.959 You use the analogy of a really talented carpenter in 137 00:06:49.000 --> 00:06:52.759 their backyard workshop versus a massive industrial factory that spans 138 00:06:52.800 --> 00:06:54.920 ten city blocks. It feels like that. 139 00:06:54.720 --> 00:06:56.639 That's a perfect analogy, and it doesn't even stop at 140 00:06:56.639 --> 00:06:59.720 the hardware. It's also the talent the people. Yes, the 141 00:06:59.759 --> 00:07:03.160 huge expertise required to orchestrate one of these massive training 142 00:07:03.199 --> 00:07:06.920 runs is incredibly rare. It's a kind of dark art. 143 00:07:07.160 --> 00:07:08.199 What do you mean by that. 144 00:07:08.360 --> 00:07:10.920 Well, when you have twenty thousand GPUs all trying to 145 00:07:10.959 --> 00:07:14.600 work in perfect parallel, things break all the time. Mysterious 146 00:07:14.639 --> 00:07:15.959 failures just happen. 147 00:07:15.920 --> 00:07:18.920 Like a cable comes loose or a network switch fails. 148 00:07:19.199 --> 00:07:21.560 It can be that simple, or it can be much weirder. 149 00:07:22.240 --> 00:07:25.839 A few GPUs might just start overheating and produce infinitesimally 150 00:07:25.879 --> 00:07:29.800 small errors in their calculations bad math. Now, in a 151 00:07:29.839 --> 00:07:33.319 normal computer program, you might get an error message it crashes. 152 00:07:34.079 --> 00:07:37.360 In a training run like this, those tiny errors can 153 00:07:37.399 --> 00:07:40.639 act like a poison. They can slowly destabilize the entire 154 00:07:40.639 --> 00:07:45.000 mathematical process. You might lose days of progress, or even worse, 155 00:07:45.319 --> 00:07:48.639 the model might end up subtly brain damaged in a 156 00:07:48.639 --> 00:07:51.040 way you don't even detect until weeks later when it 157 00:07:51.040 --> 00:07:52.639 starts giving bizarre answers. 158 00:07:52.720 --> 00:07:56.319 So the engineers running these things, yeah, they aren't just coders. 159 00:07:56.319 --> 00:08:01.639 They're more like nuclear reactor operators staring at dials trying 160 00:08:01.680 --> 00:08:02.720 to prevent a meltdown. 161 00:08:02.920 --> 00:08:05.920 They are absolutely high stakes problem solvers. They are staring 162 00:08:05.959 --> 00:08:09.360 at hundreds of graphs looking for tiny, almost imperceptible anomalies 163 00:08:09.360 --> 00:08:12.439 that suggest that AI's brain is getting sick. That level 164 00:08:12.439 --> 00:08:15.240 of operational expertise is a huge part of the moat. 165 00:08:15.560 --> 00:08:17.319 You can't just buy the chips and plug them in. 166 00:08:17.600 --> 00:08:19.639 You need the priesthood of engineers who know how to 167 00:08:19.720 --> 00:08:20.519 keep the beasts alive. 168 00:08:20.639 --> 00:08:23.560 And I'm guessing those people are very, very expensive. 169 00:08:23.319 --> 00:08:26.120 Extremely and they almost all work for the giants. 170 00:08:26.399 --> 00:08:28.360 Okay, so that's the hardware in the talent. The first 171 00:08:28.399 --> 00:08:31.319 wall of the fortress is thick. But there's another piece, right, 172 00:08:32.200 --> 00:08:33.240 the data. 173 00:08:32.840 --> 00:08:35.399 The data advantage. And this is where it gets a 174 00:08:35.399 --> 00:08:37.919 lot more subtle, but maybe even more. 175 00:08:37.759 --> 00:08:40.600 Powerful, because I think most people's first thought is, well, 176 00:08:40.600 --> 00:08:43.720 the Internet is open, right, can anyway, just scrape the 177 00:08:43.720 --> 00:08:45.600 web and get the same data you can. 178 00:08:46.320 --> 00:08:49.799 And the open source models do rely heavily on public 179 00:08:49.919 --> 00:08:53.080 data sets like the common crawl, but big tech has 180 00:08:53.159 --> 00:08:57.720 access to vast oceans of proprietary data that is completely 181 00:08:57.799 --> 00:09:00.720 invisible to the public web, like what for exams. Okay, 182 00:09:00.759 --> 00:09:03.600 just think about Google. They have over two decades of 183 00:09:03.720 --> 00:09:06.200 indexed search queries. They don't just have the text of 184 00:09:06.240 --> 00:09:08.960 the Internet. They know what billions of people are looking for, 185 00:09:09.080 --> 00:09:12.799 how they ferze their questions, and critically, what results they 186 00:09:12.799 --> 00:09:13.480 actually click on. 187 00:09:13.759 --> 00:09:16.159 Right, they don't just see the words, they see the 188 00:09:16.240 --> 00:09:18.000 intent behind the words. 189 00:09:18.080 --> 00:09:22.039 Precisely the intent. Think about Microsoft. They have deep integration 190 00:09:22.120 --> 00:09:25.799 into enterprise workflows. They'll see how businesses write emails and outlook, 191 00:09:25.840 --> 00:09:29.039 how they structure documents and word how they build presentations 192 00:09:29.080 --> 00:09:33.120 in PowerPoint. That's a huge unique data set about professional 193 00:09:33.159 --> 00:09:38.279 communication and meta the social graph decades of behavioral data, 194 00:09:38.320 --> 00:09:40.679 how people interact, what they like, what they share, the 195 00:09:40.759 --> 00:09:43.879 nuances of casual human to human conversation. And then you 196 00:09:43.879 --> 00:09:48.039 have Apple with behavioral patterns from millions of devices, understanding 197 00:09:48.080 --> 00:09:49.279 how people use apps. 198 00:09:49.399 --> 00:09:51.840 It seems like the difference between having a library of 199 00:09:51.919 --> 00:09:54.639 every book ever written, which is the public web, and 200 00:09:54.720 --> 00:09:57.879 having that same library plus a secret video recording of 201 00:09:57.960 --> 00:10:00.519 every single person who ever read a book, yeah, showing 202 00:10:00.840 --> 00:10:03.759 what pages they lingered on, what they highlighted, what made 203 00:10:03.759 --> 00:10:04.200 them laugh. 204 00:10:04.360 --> 00:10:07.399 That is a profound and perfect distinction. It's the quality 205 00:10:07.480 --> 00:10:10.679 versus quantity aspect. But there's an even bigger data advantage 206 00:10:10.679 --> 00:10:12.679 that builds on top of that, the feedback loop. 207 00:10:12.840 --> 00:10:14.919 Explain that, what do you mean by feedback loop? 208 00:10:15.039 --> 00:10:18.039 Big tech sees how users interact with their finished models 209 00:10:18.080 --> 00:10:20.759 in real time. Every time you use chat GPT and 210 00:10:20.799 --> 00:10:24.080 you don't like an answer, so you hit regenerate response. 211 00:10:24.600 --> 00:10:26.200 Open ai sees that they log that. 212 00:10:26.480 --> 00:10:29.039 They log it as a failure case every time you 213 00:10:29.080 --> 00:10:32.639 give a thumbs down to a response from Gemini. Google's 214 00:10:32.639 --> 00:10:36.360 reinforcement learning systems take note. They have this incredibly tight 215 00:10:36.399 --> 00:10:40.159 feedback loop with hundreds of millions of users constantly telling 216 00:10:40.159 --> 00:10:43.879 them what works and what doesn't. They are perpetually refining 217 00:10:43.919 --> 00:10:47.759 the model based on massive real world. 218 00:10:47.480 --> 00:10:50.039 Usage, and the open source community doesn't have that same 219 00:10:50.120 --> 00:10:51.240 direct line, not. 220 00:10:51.240 --> 00:10:53.320 In the same way. Now, when you download a model 221 00:10:53.320 --> 00:10:55.440 from hugging face and run it on your own laptop, 222 00:10:55.519 --> 00:10:57.799 the developers of that model have no idea what you're doing. 223 00:10:57.799 --> 00:10:59.600 They don't get that data back, because that's the whole. 224 00:10:59.480 --> 00:11:01.759 Point, right privacy. You run it locally so they can't 225 00:11:01.759 --> 00:11:03.000 see your data exactly. 226 00:11:03.039 --> 00:11:05.360 It's a feature, not a bug. But the cost of 227 00:11:05.360 --> 00:11:09.879 that privacy is that the model creators lose that invaluable telemetry. 228 00:11:10.360 --> 00:11:13.840 They're flying blind compared to Google or open ai, who 229 00:11:13.879 --> 00:11:16.000 have a god's eye view of how their creations are 230 00:11:16.000 --> 00:11:17.519 performing in the wild, so. 231 00:11:17.960 --> 00:11:22.440 Hearing all of that, the billions in compute, the rare talent, 232 00:11:22.720 --> 00:11:27.000 the proprietary data, the real time feedback loops, it honestly 233 00:11:27.039 --> 00:11:30.799 feels like game over. How can open source possibly compete? 234 00:11:31.279 --> 00:11:33.679 It feels like you're bringing a well crafted knife to 235 00:11:33.720 --> 00:11:34.879 a thermonuclear war. 236 00:11:35.120 --> 00:11:38.559 It absolutely looks that way on paper. The structural disadvantages 237 00:11:38.600 --> 00:11:41.440 are immense. But this is where the rebellion gets really 238 00:11:41.480 --> 00:11:44.799 really interesting, because despite all of that, open source is 239 00:11:44.840 --> 00:11:48.240 not only surviving in some very important areas, it's actually thriving. 240 00:11:48.399 --> 00:11:50.240 Oh how is that even possible? Is it just the 241 00:11:50.279 --> 00:11:51.679 sheer number of people working on it. 242 00:11:51.720 --> 00:11:54.200 That's a big part of it. But the main technical driver, 243 00:11:54.440 --> 00:11:57.679 the core reason they can compete, is a relentless focus 244 00:11:57.759 --> 00:12:01.720 on efficiency efficiency. The open source community, precisely because they 245 00:12:01.720 --> 00:12:05.200 don't have unlimited resources, has become world class at doing 246 00:12:05.240 --> 00:12:06.679 more with less necessity. 247 00:12:06.720 --> 00:12:08.759 Is the mother of invention, as they say. 248 00:12:08.519 --> 00:12:11.679 It's the perfect embodiment of that saying, when you don't 249 00:12:11.679 --> 00:12:14.240 have a billion dollars for your next training run, you 250 00:12:14.360 --> 00:12:17.840 have to get incredibly clever. And they've developed these amazing 251 00:12:17.879 --> 00:12:19.320 techniques like quantization. 252 00:12:19.759 --> 00:12:22.799 Okay, I keep seeing this word in the research quantization. 253 00:12:23.720 --> 00:12:25.919 It sounds like something out of a sci fi movie, 254 00:12:26.360 --> 00:12:30.480 but it seems absolutely critical to the open source survival guide. 255 00:12:30.799 --> 00:12:32.600 What is actually happening there. 256 00:12:32.559 --> 00:12:34.840 It is absolutely critical. The best way to think of 257 00:12:34.879 --> 00:12:37.240 it as like image compression. You know how a raw 258 00:12:37.279 --> 00:12:40.399 photo file from a professional camera can be huge, maybe 259 00:12:40.440 --> 00:12:41.399 fifty megabytes. 260 00:12:41.440 --> 00:12:43.720 Sure, it has all the raw sensor data, every bit of. 261 00:12:43.679 --> 00:12:46.600 Information exactly, But if you convert that photo to a 262 00:12:46.679 --> 00:12:50.559 JPIG it shrinks down to maybe two or three megabytes. 263 00:12:51.000 --> 00:12:53.799 You lose a tiny, tiny bit of detail. Maybe if 264 00:12:53.840 --> 00:12:56.240 you zoom in one thousand percent, the shadows aren't quite 265 00:12:56.240 --> 00:12:59.200 as perfect, But to the human eye the picture looks 266 00:12:59.279 --> 00:13:00.879 basically identicle and it. 267 00:13:00.799 --> 00:13:03.120 Loads way faster. In a website and takes up a 268 00:13:03.159 --> 00:13:04.519 fraction of the space on your hard drive. 269 00:13:04.559 --> 00:13:08.200 Precisely, quantization is basically doing that same trick to the 270 00:13:08.200 --> 00:13:12.559 AI's brain. These huge models are usually stored with incredibly 271 00:13:12.679 --> 00:13:15.759 high precision numbers. Think of every connection in the neural 272 00:13:15.799 --> 00:13:20.080 network having a weight that's like sixteen decimal places of accuracy, 273 00:13:20.159 --> 00:13:20.799 which takes. 274 00:13:20.679 --> 00:13:24.519 Up a ton of computer memory and processing power massive amounts. 275 00:13:25.320 --> 00:13:28.799 Quantization is a process that basically says, hey, do we 276 00:13:28.919 --> 00:13:32.519 really need sixteen decimal places of precision? What if we 277 00:13:32.600 --> 00:13:34.679 just round it down to four or even two? 278 00:13:35.159 --> 00:13:37.519 And the model doesn't just get stupid when you do that. 279 00:13:37.519 --> 00:13:40.879 That's the magic. It turns out for many of these models, 280 00:13:40.879 --> 00:13:44.080 you can round down those numbers very aggressively and the 281 00:13:44.120 --> 00:13:47.639 model only loses a tiny fraction of its overall intelligence. 282 00:13:48.159 --> 00:13:51.919 But suddenly a model that required a forty thousand dollars 283 00:13:51.919 --> 00:13:54.440 server with a ton of RAM can now run on 284 00:13:54.480 --> 00:13:57.159 a two thousand dollars gaming laptop. 285 00:13:57.600 --> 00:14:00.279 So they are literally shrinking the giant's brain so it 286 00:14:00.320 --> 00:14:01.919 fits inside a regular person's head. 287 00:14:02.039 --> 00:14:04.240 That is exactly what they're doing, and that's not the 288 00:14:04.240 --> 00:14:07.120 only trick. Then there's something called distillation, which is think 289 00:14:07.120 --> 00:14:10.440 of it like a teacher student relationship. You take a huge, smart, 290 00:14:10.480 --> 00:14:13.240 expensive model like GPT four and you use it to 291 00:14:13.279 --> 00:14:15.159 teach much smaller, cheaper model. 292 00:14:15.519 --> 00:14:17.440 How does that actually work in practice? 293 00:14:17.519 --> 00:14:20.039 You can, for instance, ask the big teacher model to 294 00:14:20.159 --> 00:14:23.480 generate thousands of perfect answers to questions on a specific 295 00:14:23.559 --> 00:14:27.159 topics a customer service. Then you take those thousands of 296 00:14:27.159 --> 00:14:29.919 perfect question answer pairs and you use them as the 297 00:14:30.000 --> 00:14:32.480 training data for the small student model. 298 00:14:32.639 --> 00:14:36.000 Ah, so the student learns from the master's work exactly. 299 00:14:36.559 --> 00:14:39.279 The student model might not know everything the professor knows 300 00:14:39.320 --> 00:14:42.639 about physics and poetry, but for that one specific task 301 00:14:42.679 --> 00:14:45.440 of customer service, it can get ninety percent of the 302 00:14:45.480 --> 00:14:47.879 way there for one percent of the cost to run. 303 00:14:47.960 --> 00:14:50.279 That's a huge leverage point. It's like you're borrowing the 304 00:14:50.320 --> 00:14:52.840 intelligence of the giant to train your own litle army. 305 00:14:53.200 --> 00:14:56.799 It is and because of techniques like this, the performance 306 00:14:56.840 --> 00:15:00.399 gap between the massive closed models and the nimble open 307 00:15:00.440 --> 00:15:03.919 models has narrowed again and again. The community is just 308 00:15:04.000 --> 00:15:05.600 relentless about optimization. 309 00:15:06.080 --> 00:15:07.919 And this all leads to what one of our sources 310 00:15:07.919 --> 00:15:11.840 calls the deployment victory. I found this concept fascinating. The 311 00:15:11.879 --> 00:15:14.799 idea that winning isn't just about having the single smartest 312 00:15:14.799 --> 00:15:17.720 brain in a lab somewhere, but about being the one 313 00:15:17.720 --> 00:15:19.879 that actually gets used out in the real world. 314 00:15:20.320 --> 00:15:24.039 This is such a critical distinction. There's a massive difference 315 00:15:24.080 --> 00:15:28.559 between the frontier, the absolute smartest, most capable model possible, 316 00:15:29.120 --> 00:15:33.919 and deployment what a company actually feels safe and comfortable 317 00:15:33.919 --> 00:15:36.960 putting into their production systems. Let's say you're a healthcare 318 00:15:37.000 --> 00:15:41.200 company you're handling sensitive patient records, or you're a bank 319 00:15:41.639 --> 00:15:44.799 and you're handling financial data. Are you really going to 320 00:15:44.840 --> 00:15:48.080 send all of that incredibly sensitive private data over the 321 00:15:48.120 --> 00:15:51.720 Internet to a third party API owned by big tech? 322 00:15:52.159 --> 00:15:54.759 Probably not, if you can avoid it. Your compliance department 323 00:15:54.759 --> 00:15:56.559 would have a heart attack. You want to keep that 324 00:15:56.559 --> 00:15:59.000 stuff locked down on your own servers. You definitely don't 325 00:15:59.039 --> 00:16:03.159 want opening ISIS and IS potentially reading your patient files exactly. 326 00:16:03.200 --> 00:16:05.399 You want to own the an entire stack. You need privacy, 327 00:16:05.440 --> 00:16:07.360 You demand security and control. 328 00:16:07.360 --> 00:16:09.360 I would imagine total control. 329 00:16:09.120 --> 00:16:11.919 Massive control. If you build your entire product on top 330 00:16:11.960 --> 00:16:15.279 of a proprietary model from a single company, you are 331 00:16:15.320 --> 00:16:18.879 taking on an enormous strategic risk. What happens if they 332 00:16:18.919 --> 00:16:20.720 decide to double the pricing next year? 333 00:16:20.840 --> 00:16:21.360 You're stuck. 334 00:16:21.559 --> 00:16:24.480 What if they change their roadmap and deprecate the version 335 00:16:24.519 --> 00:16:26.519 of the model you rely on. What if they go 336 00:16:26.600 --> 00:16:28.200 out of business or get acquired. 337 00:16:28.440 --> 00:16:32.320 You're building your entire house on land that you're just renting. 338 00:16:32.399 --> 00:16:33.840 They could evict you at any time. 339 00:16:34.159 --> 00:16:38.879 Precisely, open source offers stability and sovereignty. You download the model, 340 00:16:38.879 --> 00:16:41.919 weights there are yours. You run it on your own infrastructure, 341 00:16:41.960 --> 00:16:44.559 behind your own firewall. No one can ever take it 342 00:16:44.600 --> 00:16:45.120 away from you. 343 00:16:45.600 --> 00:16:49.519 So for the enterprise, you know, the big serious companies 344 00:16:49.559 --> 00:16:52.519 that are actually paying to implement this stuff in the 345 00:16:52.559 --> 00:16:56.120 real world, open source is often the better strategic bet. 346 00:16:56.200 --> 00:16:59.480 For many regulated industries. It's often the only acceptable option. 347 00:17:00.159 --> 00:17:02.519 There's another factor here too, which is just the sheer 348 00:17:02.559 --> 00:17:06.160 speed of innovation, the swarm intelligence. I want to dig 349 00:17:06.200 --> 00:17:07.799 a little deeper into that. Can you give me a 350 00:17:07.839 --> 00:17:10.680 concrete example of a time the open source community just 351 00:17:11.160 --> 00:17:12.880 completely outran the giants? Oh? 352 00:17:12.920 --> 00:17:15.920 Absolutely. The perfect example is the rise of mixture of 353 00:17:15.960 --> 00:17:17.920 experts architecture or MOE. 354 00:17:18.559 --> 00:17:20.640 Sounds fancy, but what is it? In simple terms? 355 00:17:20.839 --> 00:17:23.400 Okay, So, a traditional model is what we call dense. 356 00:17:23.880 --> 00:17:27.839 It's one single giant brain. Every time you ask it 357 00:17:27.880 --> 00:17:30.680 a question, the entire brain has to light up and think. 358 00:17:30.519 --> 00:17:32.799 About it, which is computationally expensive. 359 00:17:33.079 --> 00:17:36.960 Very and MOE model is different. Instead of one giant brain, 360 00:17:37.359 --> 00:17:41.119 the model is made up of many smaller expert submodels. 361 00:17:41.200 --> 00:17:44.039 Maybe you have eight different experts, and when you ask 362 00:17:44.079 --> 00:17:46.960 a question, there's a tiny router network at the front 363 00:17:46.960 --> 00:17:49.720 that decides which one or two experts are best suited 364 00:17:49.759 --> 00:17:50.599 to answer it. 365 00:17:50.519 --> 00:17:53.000 Like a receptionist at a big hospital directing calls. 366 00:17:53.119 --> 00:17:55.160 Exactly like that. Oh you have a coding question, I'll 367 00:17:55.160 --> 00:17:56.759 send you to the python expert. You have a question 368 00:17:56.799 --> 00:17:59.039 about ancient Rome, let's talk to the history expert. 369 00:17:59.119 --> 00:17:59.960 And why is that better? 370 00:18:00.480 --> 00:18:04.200 It's massively more efficient for any given thought. You're only 371 00:18:04.240 --> 00:18:06.880 activating a small fraction of the total brain, so it's 372 00:18:06.960 --> 00:18:09.680 much faster and much much cheaper to run, but you 373 00:18:09.720 --> 00:18:12.119 still get the benefit of the combined knowledge of all 374 00:18:12.160 --> 00:18:12.759 the experts. 375 00:18:12.799 --> 00:18:17.440 Okay, I get it. So how did the swarm outrun 376 00:18:17.480 --> 00:18:18.319 the giants on this? 377 00:18:18.720 --> 00:18:21.519 Well? Big tech has been using this technique internally for 378 00:18:21.559 --> 00:18:24.799 a while, but when the open source company Mistral released 379 00:18:24.839 --> 00:18:29.200 their first MOEE model, the community just exploded. They dissected it, 380 00:18:29.240 --> 00:18:31.960 they understood it, and they started building their own improved 381 00:18:32.039 --> 00:18:36.240 variants instantly. Within weeks, we had people figuring out clever 382 00:18:36.440 --> 00:18:41.960 hacks to run these incredibly complex architectures on consumer grade hardware, on. 383 00:18:42.000 --> 00:18:48.160 MacBooks, a laptop, a model with multiple expert brains. How 384 00:18:48.240 --> 00:18:49.279 is that even possible. 385 00:18:49.400 --> 00:18:53.079 It's possible because the community became obsessed with optimization. They 386 00:18:53.119 --> 00:18:56.319 found these brilliant ways to load the different experts in 387 00:18:56.359 --> 00:18:58.759 and out of your computer's memory so fast that you 388 00:18:58.799 --> 00:19:01.319 could run a model that on paper should never be 389 00:19:01.359 --> 00:19:02.599 able to fit on your machine. 390 00:19:02.599 --> 00:19:04.559 As that classic hacker spirit I love. 391 00:19:04.759 --> 00:19:05.000 Yeah. 392 00:19:05.000 --> 00:19:07.599 Big text answer is buy a bigger, more expensive server. 393 00:19:07.720 --> 00:19:10.079 The open source answer is no, let's rewrite the code 394 00:19:10.119 --> 00:19:11.559 to make it fit on the server we already have. 395 00:19:11.839 --> 00:19:14.200 That is the fundamental cultural difference right there. 396 00:19:14.240 --> 00:19:17.000 And look at the timeline in a big tech company. 397 00:19:17.079 --> 00:19:19.480 If a researcher has a brilliant new idea like. 398 00:19:19.440 --> 00:19:21.759 That, it has to be written up in a proposal, 399 00:19:21.920 --> 00:19:24.519 It has to go through a committee review, it has 400 00:19:24.559 --> 00:19:28.640 to get prioritized and put into the next quarters product roadmap. 401 00:19:29.200 --> 00:19:32.039