WEBVTT 1 00:00:00.120 --> 00:00:02.600 Welcome to the GROC three presentation. 2 00:00:03.200 --> 00:00:08.000 The mission of Xai and Grock is to understand the universe. 3 00:00:08.000 --> 00:00:10.160 We want to understand the nature of the universe so 4 00:00:10.199 --> 00:00:12.240 we can figure out what's going on. Where are the aliens, 5 00:00:12.679 --> 00:00:14.640 what's the meaning of life, how does the universe end? 6 00:00:14.720 --> 00:00:17.120 How did it start? All these fundamental questions. 7 00:00:17.519 --> 00:00:20.199 We're driven by curiosity about the nature of the universe, 8 00:00:20.839 --> 00:00:24.039 and that's also what causes us to be a maximally 9 00:00:24.079 --> 00:00:28.440 truth seeking AI, even if that truth is sometimes at. 10 00:00:28.359 --> 00:00:30.079 Odds with what is politically correct. 11 00:00:31.000 --> 00:00:33.799 In order to understand the nature of the universe, you 12 00:00:33.880 --> 00:00:37.719 must absolutely rigorously pursue truth or you will not understand 13 00:00:37.719 --> 00:00:41.000 the universe. You'll be suffering from some amount of delusion 14 00:00:41.079 --> 00:00:44.280 or error. So that is our goal, figure out what's 15 00:00:44.280 --> 00:00:47.679 going on. And we're very excited to present GROC three, 16 00:00:47.840 --> 00:00:51.439 which is we think, in order of magnitude, more capable 17 00:00:51.439 --> 00:00:53.679 than GROC two in a very short period of time. 18 00:00:54.359 --> 00:00:58.200 And that's thanks to the hard work of an incredible team. 19 00:00:58.600 --> 00:01:00.719 My monitor to work with such a great team, and 20 00:01:00.799 --> 00:01:03.320 of course we'd love to have so the smartest humans 21 00:01:03.320 --> 00:01:04.760 out there join our team. 22 00:01:05.319 --> 00:01:07.840 So let's go, Hi Ron. 23 00:01:07.879 --> 00:01:11.319 My name is Igor lead Engineering at xi I GBPA 24 00:01:11.560 --> 00:01:15.560 leading research. I'm Tony working on the reasoning team, all right, Ele, 25 00:01:15.640 --> 00:01:18.879 And I don't do anything. I just show up occasionally. 26 00:01:19.799 --> 00:01:20.200 Yeah. 27 00:01:20.239 --> 00:01:23.079 So, like I mentioned, GROK is the tool that we're 28 00:01:23.079 --> 00:01:25.280 working on. Grock is our AI that we're building here 29 00:01:25.319 --> 00:01:27.719 at XAI, and we've been working extremely hard over the 30 00:01:27.760 --> 00:01:29.680 last few months to improve GROC as much as we 31 00:01:29.719 --> 00:01:31.640 can so we can give all of you access to it. 32 00:01:32.159 --> 00:01:34.519 We think it's going to be extremely useful. Do we 33 00:01:34.560 --> 00:01:36.200 think it's going to be interesting to talk to a 34 00:01:36.280 --> 00:01:39.400 funny really really funny and we're going to explain to 35 00:01:39.400 --> 00:01:41.359 you how we've improved GROC over the last few months. 36 00:01:41.359 --> 00:01:44.120 We've made quite a jump in capabilities. Actually we should 37 00:01:44.120 --> 00:01:46.319 explain maybe also what is why do we call it groc? 38 00:01:46.599 --> 00:01:49.480 So grog is a word from a Heinland novel Stranger 39 00:01:49.519 --> 00:01:52.519 in a Strange Land, and it's used by a guy 40 00:01:52.519 --> 00:01:55.920 who was raised on Mars. And the word grock is 41 00:01:55.959 --> 00:01:59.680 to sort of fully and profoundly understand something. That's what 42 00:01:59.719 --> 00:02:02.400 the word grog means, fully and profoundly understand something. 43 00:02:02.920 --> 00:02:04.799 And empathy is important. True. 44 00:02:04.920 --> 00:02:08.719 So yeah, if we charged xs progress in the last 45 00:02:08.759 --> 00:02:12.159 few months, it's only been seventeen months since we started 46 00:02:12.439 --> 00:02:17.360 kicking off our very first model. GROCK one was almost 47 00:02:17.479 --> 00:02:19.800 like a toy by this point on the two hundred 48 00:02:19.800 --> 00:02:22.400 and fourteen billion parameters. And now if we're prout the 49 00:02:22.439 --> 00:02:27.080 progress the time on XXI is the performance of favorite 50 00:02:27.080 --> 00:02:31.800 benchmark numbers at MLU on the yaxis, we're literally progressing 51 00:02:32.039 --> 00:02:36.599 at the unprecedent speed across the whole field. And then 52 00:02:36.639 --> 00:02:38.759 we kick off GROG one point five right after GUARG 53 00:02:38.800 --> 00:02:42.879 one released after November twenty twenty three, and then GROG two. 54 00:02:43.439 --> 00:02:46.680 So if you look at where all the performance coming from. 55 00:02:46.719 --> 00:02:49.639 We have a very correct engineering team and all the 56 00:02:49.639 --> 00:02:53.280 best AII talent. The only one thing we need is 57 00:02:53.639 --> 00:02:57.520 a big intelligence comes from big cluster, so we can 58 00:02:57.560 --> 00:03:01.560 reconvert the entire progress and makes AI now replacing the 59 00:03:01.560 --> 00:03:04.680 benchmarket of waxes to the total amount of training flops. 60 00:03:04.879 --> 00:03:07.520 That is how many GPS we can run at any 61 00:03:07.520 --> 00:03:11.319 given time to trail all large language models to compress 62 00:03:11.439 --> 00:03:12.240 the entire Internet. 63 00:03:12.800 --> 00:03:16.039 So after old human knowledge. Really that's right. 64 00:03:16.120 --> 00:03:17.879 Yeah, the Internet is being part of it, but it's 65 00:03:17.879 --> 00:03:19.400 really all human knowledge everything. 66 00:03:19.639 --> 00:03:22.280 Yeah, the whole Internet fits into USB stick at this 67 00:03:22.360 --> 00:03:24.599 point it's like all the human tokens very soon into 68 00:03:24.599 --> 00:03:27.599 the real world. We had so much trouble actually training 69 00:03:27.599 --> 00:03:29.039 Grock two back in the days. 70 00:03:29.199 --> 00:03:31.680 We kickoff the model around February. 71 00:03:31.960 --> 00:03:34.639 And we thought we had a large amount of chips, 72 00:03:34.680 --> 00:03:37.639 but turned out we can barely get AK training chips 73 00:03:37.840 --> 00:03:42.000 running coherently at any given time, and we had so 74 00:03:42.080 --> 00:03:44.639 many cooling and power issues. 75 00:03:45.199 --> 00:03:46.919 I think you were there in the data center. 76 00:03:47.120 --> 00:03:50.159 Yeah, it was like really sort of more like AK 77 00:03:50.360 --> 00:03:53.080 chips on average at eighty percent efficiency, more like like 78 00:03:53.080 --> 00:03:58.159 sixty five hundred effective h one hundreds training for near 79 00:03:58.199 --> 00:04:00.680 several months, but you know we're the one hundred k. 80 00:04:00.960 --> 00:04:05.759 So yeah, that's right, that's right. So what's the next step? 81 00:04:06.000 --> 00:04:10.400 Right, so after Groark too, So if we all continue accelerate, 82 00:04:10.879 --> 00:04:12.520 we have to take the matter into our own hands. 83 00:04:12.560 --> 00:04:15.039 We have to solve all the coolings, all the power 84 00:04:15.120 --> 00:04:16.360 issues and everything. 85 00:04:16.839 --> 00:04:19.279 So it so on April of last year, Elan decided 86 00:04:19.319 --> 00:04:22.439 that really the only way for XAI to succeed, for 87 00:04:22.600 --> 00:04:25.040 XI to build the best AI out there, is to 88 00:04:25.120 --> 00:04:28.000 build our own data center. So really we realize we 89 00:04:28.079 --> 00:04:31.120 have to build the data center in about four months. 90 00:04:31.360 --> 00:04:33.240 It turned out it took US one hundred and twenty 91 00:04:33.279 --> 00:04:36.240 two days to get the first one hundred KGPUS up 92 00:04:36.240 --> 00:04:38.920 and running, and that was a monumental effort to be 93 00:04:38.920 --> 00:04:41.879 able to do that. We believe it's the biggest fully 94 00:04:41.920 --> 00:04:45.639 connected h one hundred cluster of its kind. We actually 95 00:04:45.639 --> 00:04:47.920 decided that we need to double the size of the 96 00:04:47.959 --> 00:04:50.639 cluster pretty much immediately if we want to build the 97 00:04:50.720 --> 00:04:52.040 kind of way that we want to build. 98 00:04:52.600 --> 00:04:55.399 So we then had another phase. 99 00:04:55.279 --> 00:04:57.319 Which we haven't talked about publicly, so this is the 100 00:04:57.360 --> 00:04:59.920 first time that we're talking about this, where we doubled 101 00:05:00.319 --> 00:05:03.319 the capacity of the data center yet again. And that 102 00:05:03.319 --> 00:05:06.040 not only took us ninety two days. So we've been 103 00:05:06.079 --> 00:05:08.480 able to use all of diseributse, use all of this 104 00:05:08.600 --> 00:05:11.879 compute to improve GROG in the meantime. And basically today 105 00:05:11.920 --> 00:05:14.800 we're going to present the results of that, the fruits 106 00:05:14.879 --> 00:05:15.480 that came from that. 107 00:05:16.199 --> 00:05:20.079 That's yeah, So all the paths, all the roads leads 108 00:05:20.079 --> 00:05:23.279 to GROD three ten x more compute, more than ten 109 00:05:23.439 --> 00:05:27.720 x really maybe fifteen x compared to our private generation model, 110 00:05:28.439 --> 00:05:32.600 and GROCT finished the pre trading early January and we started, 111 00:05:32.839 --> 00:05:35.279 you know, the model is still currently trading. Actually, so 112 00:05:35.360 --> 00:05:39.360 this is a little preview of our benchmine numbers. So 113 00:05:39.560 --> 00:05:43.360 we evaluate the GROCK three on you know, three different 114 00:05:43.360 --> 00:05:49.680 categories on general mathematical reasoning, on general knowledge about STEM 115 00:05:49.759 --> 00:05:54.439 and science, and then also on computer science coding. So 116 00:05:55.360 --> 00:06:00.480 amy American Invitational Math Examination hosted you know a year. 117 00:06:00.879 --> 00:06:04.720 If we evaluated model performance, we can see that the 118 00:06:04.800 --> 00:06:07.079 GROD three across the board is in the league of 119 00:06:07.120 --> 00:06:12.040 its own, even as little brother Groctor Mini is reaching 120 00:06:12.439 --> 00:06:14.439 the frontier across. 121 00:06:14.160 --> 00:06:15.199 All the other competitors. 122 00:06:15.839 --> 00:06:18.519 You would say, well, at this point, all these benchmarks 123 00:06:18.680 --> 00:06:21.639 you're just evaluating, you know, the memorization of the textbooks, 124 00:06:22.279 --> 00:06:25.920 memorization of the GitHub repos, how about real time usefulness? 125 00:06:26.199 --> 00:06:28.759 How about we actually use those models in our product. 126 00:06:29.680 --> 00:06:33.160 So what we did instead is we actually kicked off 127 00:06:33.600 --> 00:06:37.680 a blind test of our garacy model code named Chocolate, 128 00:06:38.199 --> 00:06:42.560 Pretty Hot Chocolate. I've been running on this platform called 129 00:06:42.600 --> 00:06:46.160 chap Arena for two weeks. I think the entire x 130 00:06:46.199 --> 00:06:48.959 platform at some point speculated this might be the next 131 00:06:49.000 --> 00:06:54.160 generation of AI coming away. So how this chap arena 132 00:06:54.240 --> 00:06:58.600 works is that it's stripped away the entire product service, right, 133 00:06:58.680 --> 00:07:01.800 It just raw comparison of the engine of those Asia's 134 00:07:01.839 --> 00:07:05.240 the language models themselves and place interface. 135 00:07:04.720 --> 00:07:07.959 Where the user will submit one single querry and you. 136 00:07:07.959 --> 00:07:10.639 Get to show two responses, you don't know which model 137 00:07:10.680 --> 00:07:13.920 they come from, and India you make the vote. So 138 00:07:14.000 --> 00:07:17.240 in this blind test, GROX three, an early version of 139 00:07:17.240 --> 00:07:22.240 GROX three already reached like fourteen hundred. No other models 140 00:07:22.360 --> 00:07:25.079 had reached an ELO score had to have comparison to 141 00:07:25.120 --> 00:07:26.879 all the other models at this score. 142 00:07:27.399 --> 00:07:30.000 And it's not just one single category. 143 00:07:30.399 --> 00:07:35.480 It's fourteen hundred aggregated across all the categories in chapbock capabilities, 144 00:07:35.480 --> 00:07:39.639 in struction, following coding, So it's number one across the 145 00:07:39.639 --> 00:07:40.319 board in this. 146 00:07:40.319 --> 00:07:42.959 Blind test, and it's still climbing, so we actually keep 147 00:07:43.040 --> 00:07:45.720 to keep updating it. So it's forty fourteen hundred, but 148 00:07:45.839 --> 00:07:46.959 fourteen hundred and climbing. 149 00:07:47.199 --> 00:07:48.639 Yeah, And in fact, we have a version of the 150 00:07:48.680 --> 00:07:50.879 model that we think is already much better than the 151 00:07:50.879 --> 00:07:54.839 one that we tested here. Yeah, we'll see, I guess, 152 00:07:55.240 --> 00:07:59.040 but that's the one that we're working on talking about today. 153 00:07:59.199 --> 00:07:59.399 Yeah. 154 00:07:59.439 --> 00:08:02.439 So actually, thing, if you're using BARG three, you I 155 00:08:02.439 --> 00:08:05.959 think you may notice improvements almost every day because we're 156 00:08:06.000 --> 00:08:10.120 continuously improving the model. So literally even within twenty four hours, 157 00:08:10.120 --> 00:08:10.879 you'll see improvements. 158 00:08:11.480 --> 00:08:11.759 Yep. 159 00:08:13.199 --> 00:08:16.079 But we believe here at the XAI, getting the best 160 00:08:16.399 --> 00:08:19.199 pre training model is not enough. That's not enough to 161 00:08:19.240 --> 00:08:22.079 build the best AI and the best A. I need 162 00:08:22.120 --> 00:08:25.279 to think like a human to contemplate about all the 163 00:08:25.319 --> 00:08:31.160 possible solutions, self critique, verify all the solutions, backtrack, and 164 00:08:31.240 --> 00:08:32.799 also think from the first principle. 165 00:08:33.279 --> 00:08:34.720 That's a very important capability. 166 00:08:35.480 --> 00:08:38.519 So we believe that as we take the best PREA 167 00:08:38.600 --> 00:08:43.360 training model and continue training with reinforcement learning, it will 168 00:08:43.480 --> 00:08:47.120 enlicit the additional reasoning capabilities that allows the model just 169 00:08:47.440 --> 00:08:50.480 becomes so much better and scale, not just in the 170 00:08:50.480 --> 00:08:53.039 training time, but actually in the test time as well. 171 00:08:53.559 --> 00:08:56.240 So we already found the model is extremely useful internally, 172 00:08:56.519 --> 00:08:59.799 saving one hundreds of hours of coding time. So you 173 00:08:59.799 --> 00:09:02.960 are the user of our these model, what does the 174 00:09:03.039 --> 00:09:03.639 use cases yea. 175 00:09:03.720 --> 00:09:06.960 So, like Jimmy said, we've added advanced reasoning capabilities to 176 00:09:07.000 --> 00:09:09.399 GROG and we've been testing them pretty heavily over the 177 00:09:09.480 --> 00:09:11.200 last few weeks in order to give you a little 178 00:09:11.200 --> 00:09:12.679 bit of a taste of what it looks like when 179 00:09:12.720 --> 00:09:15.759 GROG is solving hard reasoning problems. So we've prepared two 180 00:09:15.759 --> 00:09:18.600 little problems for you. One that comes from physics and 181 00:09:18.679 --> 00:09:20.720 one is actually a game that God is going to 182 00:09:20.720 --> 00:09:23.000 write for us. When it comes to the physics problem, 183 00:09:23.039 --> 00:09:25.120 you know, what we want Grog to do is to 184 00:09:25.240 --> 00:09:29.080 plot a viable trajectory to do a transfer from Earth 185 00:09:29.120 --> 00:09:31.879 to Mars and then at a later point in time 186 00:09:31.919 --> 00:09:34.519 a transfer back from Mars to Earth. And that requires, 187 00:09:34.919 --> 00:09:37.200 you know, some physics that Grog will have to understand. 188 00:09:37.519 --> 00:09:39.440 So we're going to challenge Grock, you know, come up 189 00:09:39.440 --> 00:09:43.639 with a viable trajectory, calculate it, and then plot it 190 00:09:43.639 --> 00:09:45.919 for us so we can see it. And yeah, this 191 00:09:46.039 --> 00:09:49.120 is totally unscripted, by the way, this is the Grog interface, 192 00:09:49.519 --> 00:09:52.200 and we've typed in this text that you can see 193 00:09:52.240 --> 00:09:54.799 here generate code for an animated three D plot of 194 00:09:54.879 --> 00:09:58.159 a launch from Earth, landing on Mars, and then back 195 00:09:58.200 --> 00:10:01.120 to Earth at the next launch window. And we've not 196 00:10:01.240 --> 00:10:03.799 kicked off or the query and you can see Grog spinking. 197 00:10:04.600 --> 00:10:08.039 So out of Rock's advanced reasoning capabilities are these thinking 198 00:10:08.080 --> 00:10:10.320 traces that you can see here. You can even go 199 00:10:10.399 --> 00:10:13.279 inside and actually read what Grock is thinking as it's 200 00:10:13.320 --> 00:10:15.200 going through the problem, as it's trying to solve it. 201 00:10:15.480 --> 00:10:18.519 Yeah, weld say like we are doing some obscuration of 202 00:10:18.559 --> 00:10:22.240 the thinking so that our model doesn't get totally copied instantly. 203 00:10:22.759 --> 00:10:27.399 So there's more to the thinking than is displayed. 204 00:10:27.799 --> 00:10:30.639 And because this is totally unscripted, there's actually a chance 205 00:10:30.679 --> 00:10:33.080 that Grock might make a little courting mistake and it 206 00:10:33.159 --> 00:10:35.519 might not actually work. So just in case, we're going 207 00:10:35.559 --> 00:10:38.559 to launch two more instances of this, so if something 208 00:10:38.600 --> 00:10:41.679 goes wrong, we were able to search to those enshow 209 00:10:41.679 --> 00:10:44.960 you something that's presentable, So we're kicking off the other 210 00:10:45.039 --> 00:10:47.559 two as well, And like I said, we have a 211 00:10:47.639 --> 00:10:51.159 second problem as well. Actually, one of our favorite activities 212 00:10:51.159 --> 00:10:55.039 here XCI is having rock right games for us, not 213 00:10:55.120 --> 00:10:57.879 just any now any old game, any game that you 214 00:10:57.919 --> 00:11:00.559 might already be familiar with, but actually creating new games 215 00:11:00.600 --> 00:11:03.960 on the spot and being creative about us. So one 216 00:11:04.039 --> 00:11:07.720 example that we found was really really fun is create 217 00:11:07.720 --> 00:11:11.000 a game that's a mixture of the two games Tetris 218 00:11:11.000 --> 00:11:12.000 and Jewels. 219 00:11:12.600 --> 00:11:14.759 So this is maybe an important thing like that this 220 00:11:15.399 --> 00:11:18.000 obviously if you if you ask an ai to create 221 00:11:18.000 --> 00:11:20.080 a game like Tetris, there's there are many examples of 222 00:11:20.120 --> 00:11:21.399 Tetris on the on the internet. 223 00:11:21.399 --> 00:11:25.200 The air or game like Dueled whatever, this, it can 224 00:11:25.240 --> 00:11:25.720 copy it. 225 00:11:26.600 --> 00:11:30.960 What's interesting here is it achieved a creative solution combining 226 00:11:31.320 --> 00:11:36.080 the two games that actually works and is a good game. Yeah, 227 00:11:36.240 --> 00:11:39.960 that's the it's created. We're seeing the beginnings of creativity. 228 00:11:40.480 --> 00:11:42.720 Fingers crossed that we can recreate that, but hopefully it works. 229 00:11:43.200 --> 00:11:45.519 Hopefully it's actually because this is a bit more challenging, 230 00:11:45.519 --> 00:11:47.840 we're going to use something special here, which we call 231 00:11:47.919 --> 00:11:51.480 big Brain. That's our mode in which we use more computation, 232 00:11:51.840 --> 00:11:54.639 which was more reasoning of our rock, just to make 233 00:11:54.639 --> 00:11:56.399 sure that you know, there's a good chance here that 234 00:11:56.799 --> 00:11:59.679 actually might actually do it. So we're also going to 235 00:11:59.720 --> 00:12:04.559 fire of free attempts here at solving this game, at 236 00:12:04.639 --> 00:12:07.919 creating this game that's a mixture of Tetris and the Jewels. 237 00:12:09.080 --> 00:12:10.519 Let's let's see what god comes out. 238 00:12:10.279 --> 00:12:14.639 I've played the game. It's pretty good. Like it's like, wow, okay, 239 00:12:14.720 --> 00:12:15.320 this is something. 240 00:12:16.000 --> 00:12:19.720 Yeah, So while Grog is thinking in the background, we 241 00:12:19.759 --> 00:12:23.000 can now actually talk about some concrete numbers. How well 242 00:12:23.080 --> 00:12:25.559 is Groug doing across tons of different tasks that we've 243 00:12:25.559 --> 00:12:28.960 tested on. So we'll hand it over to Tony to talk 244 00:12:28.960 --> 00:12:29.320 about that. 245 00:12:30.279 --> 00:12:33.360 Yeah, Okay, so let's see how Greg has on those 246 00:12:33.559 --> 00:12:38.440 interesting challenging benchmarks. So yeah, so reasoning again refers to 247 00:12:38.480 --> 00:12:41.600 those models that actually thinks quite for quite a long 248 00:12:41.639 --> 00:12:45.360 time before it tries to solve a problem. So in 249 00:12:45.399 --> 00:12:48.440 this case, you know, around the months ago the growth 250 00:12:48.440 --> 00:12:52.679 three apprecionning finishes, so after that we worked very hard 251 00:12:52.720 --> 00:12:56.840 to put the reasoning capability into the current Growth three model. 252 00:12:57.279 --> 00:12:59.840 But again this is very early days, so the model 253 00:13:00.120 --> 00:13:03.000 still currently in training. So right now, what we're going 254 00:13:03.080 --> 00:13:05.960 to show to people is this beta version of the 255 00:13:06.000 --> 00:13:09.799 growth three reasoning model. Alongside, we also are training a 256 00:13:09.799 --> 00:13:12.720 mini version of the reasoning model. So essentially on this 257 00:13:12.799 --> 00:13:16.279 plot you can see the growth three reasoning beta and 258 00:13:16.279 --> 00:13:19.759 then Growth three mini reasoning. The reasoning mini reasoning is 259 00:13:19.759 --> 00:13:21.919 actually a model that we train for much longer time, 260 00:13:22.320 --> 00:13:25.200 and you can see that sometimes you actually perform slightly 261 00:13:25.240 --> 00:13:28.840 better compared to the Growth three reasoning. This also just 262 00:13:28.879 --> 00:13:31.279 means that there's a huge potential for the Growth three 263 00:13:31.279 --> 00:13:32.639 reasoning because it's trained for. 264 00:13:32.679 --> 00:13:33.360 Much less time. 265 00:13:34.639 --> 00:13:37.600 So all right, so let's actually look at how it 266 00:13:37.639 --> 00:13:42.000 does on those three benchmarks. So Jimmy also introduced already, 267 00:13:42.039 --> 00:13:46.080 so essentially we're looking at three different areas mathematics, science, 268 00:13:46.200 --> 00:13:49.759 and coding. And for math we're picking this high school 269 00:13:49.799 --> 00:13:54.519 competition math problem. For science we actually picked those PhD 270 00:13:54.679 --> 00:13:56.279 level science questions. 271 00:13:56.919 --> 00:13:59.039 And for coding, it's also actually pretty challenging. 272 00:13:59.120 --> 00:14:02.720 It's competitive code and also some eco, which is some 273 00:14:03.440 --> 00:14:07.360 interview problems that people usually get when they interview for companies. 274 00:14:07.600 --> 00:14:10.159 So on those benchmarks, you can see that the growth 275 00:14:10.159 --> 00:14:14.159 three actually performed quite well across the board compared to 276 00:14:14.240 --> 00:14:18.480 other competitors. Yeah, so it's pretty promising. These models are 277 00:14:18.600 --> 00:14:19.080 very smart. 278 00:14:19.639 --> 00:14:23.440 So totally what what are those shaded bars? Yeah, so okay, 279 00:14:23.519 --> 00:14:25.519 so I'm you asked this question. 280 00:14:25.639 --> 00:14:29.240 So for those models, because it can reason, it can think, 281 00:14:29.679 --> 00:14:32.840 you can also ask them to even think longer. You 282 00:14:32.879 --> 00:14:36.759 can spend more what we call test and compute, which 283 00:14:36.799 --> 00:14:40.120 means you can spend more time to reason, to think 284 00:14:40.240 --> 00:14:43.759 about the problem before you spit out the answer. So 285 00:14:43.799 --> 00:14:46.720 in this case, the shaded bar here means that we 286 00:14:46.879 --> 00:14:50.440 just asked the model to spend more time. You know, 287 00:14:50.519 --> 00:14:53.480 it can solve the same problem many many times before 288 00:14:53.919 --> 00:14:57.080 it tries to conclude what is the right solution, And 289 00:14:57.240 --> 00:15:00.200 once you give this compute or this kind of budget 290 00:15:00.519 --> 00:15:02.799 to the model. It turns out the model can even 291 00:15:02.840 --> 00:15:06.600 perform better. So this is it's honly the shaded part 292 00:15:06.840 --> 00:15:07.639 in those. 293 00:15:07.519 --> 00:15:10.720 Plots, right, So I think this is really exciting, right 294 00:15:10.799 --> 00:15:13.519 because now instead of just doing one chain of thoughts 295 00:15:13.720 --> 00:15:17.240 with AI while not the multiplex at once. Yes, so 296 00:15:17.279 --> 00:15:19.600 that's a very powerful technique that allows to continue to 297 00:15:19.840 --> 00:15:25.240 scale the model capabilities after training. And you know, people 298 00:15:25.240 --> 00:15:28.120 often ask were actually just overfitt into the benchmarks? 299 00:15:28.600 --> 00:15:29.720 So how mogulization? 300 00:15:30.039 --> 00:15:33.240 So yes, I think, yeah, this is definitely a question 301 00:15:33.360 --> 00:15:36.840 that we are asking ourselves whether we're overfitting into those 302 00:15:36.919 --> 00:15:37.919 current benchmarks. 303 00:15:38.480 --> 00:15:40.720 Luckily, we have a real test. 304 00:15:41.240 --> 00:15:45.279 So about five days ago, AIM twenty twenty five just finished. 305 00:15:45.480 --> 00:15:48.519 This is where high school come students compete in this 306 00:15:48.559 --> 00:15:52.279 particular benchmark. So we got this very fresh new competition, 307 00:15:52.759 --> 00:15:55.039 and then we ask our two models to compete on 308 00:15:55.120 --> 00:15:58.639 the same benchmark, the same exam, and it turns out 309 00:15:59.480 --> 00:16:03.720 very interest the gross three reasoning the big one actually 310 00:16:03.799 --> 00:16:08.279 does better on this particular new, fresh exam. This also 311 00:16:08.320 --> 00:16:11.919 means that the generalization capability of the big model is stronger, 312 00:16:12.080 --> 00:16:15.039 much stronger compared to the smaller model. If you compare 313 00:16:15.080 --> 00:16:17.840 to the last year's exam. Actually this is the opposite. 314 00:16:18.000 --> 00:16:23.000 The smaller model kind of learns the previous exams better. 315 00:16:23.840 --> 00:16:25.759 So yeah, so this this actually shows some kind of 316 00:16:25.759 --> 00:16:27.559 true generalization from the model. 317 00:16:28.080 --> 00:16:31.039 Right, So, seventeen months ago our Rock zero and Rock 318 00:16:31.120 --> 00:16:34.120 Web baret solved any high school problems. That's right, And 319 00:16:34.159 --> 00:16:37.720 now we have a kid that just already graduate. The 320 00:16:37.720 --> 00:16:39.120 Grock is right to go to college? 321 00:16:39.159 --> 00:16:42.039 Is that right? I mean, it's won't belong before. It's 322 00:16:42.000 --> 00:16:42.759 simply perfect. 323 00:16:43.080 --> 00:16:45.919 The human exams won't be part it'll be too easy. 324 00:16:46.240 --> 00:16:50.720 Yeah, and the internally we actually as a rocket continually evolved. 325 00:16:51.240 --> 00:16:53.080 We're going to talk about, you know what we're. 326 00:16:52.960 --> 00:16:56.000 Excited about, but very soon there will be no more 327 00:16:56.000 --> 00:16:56.799 benchmark left. 328 00:16:57.039 --> 00:16:59.440 Yeah. Yeah. One thing that's quite fascinating. 329 00:16:59.600 --> 00:17:03.000 I think that we basically only trained Grock's reasoning abilities 330 00:17:03.000 --> 00:17:07.200 on math problems and competitive coding problems. It's so very 331 00:17:07.279 --> 00:17:11.000 very specialized kinds of tasks, but somehow it's able to 332 00:17:11.039 --> 00:17:14.759 work on all kinds of other different tasks, so including 333 00:17:14.799 --> 00:17:18.200 creating games. No, lots and lots of different things. And 334 00:17:18.279 --> 00:17:21.079 what seems to be happening is that basically Grog learns 335 00:17:21.160 --> 00:17:24.079 this ability to detect its own mistakes and it's thinking, 336 00:17:24.160 --> 00:17:27.680 correct them, persist on a problem, try lots of different variants, 337 00:17:27.759 --> 00:17:30.200 pick them one that's best. So there are is generalized 338 00:17:30.440 --> 00:17:35.039 generalizing abilities that learns from mathematics and from coding, which 339 00:17:35.039 --> 00:17:37.279 it can then use to solve all kinds of other problems. 340 00:17:37.279 --> 00:17:39.720 So that's yeah, that's pretty I mean, reality is the 341 00:17:39.720 --> 00:17:42.240 instantiation of mathematics. That's right. 342 00:17:43.400 --> 00:17:45.480 And one thing we're actually really excited about that going 343 00:17:45.519 --> 00:17:48.559 back to our faulty mission, is what if one day 344 00:17:48.839 --> 00:17:52.519 we have a computer just like the thought that utilize 345 00:17:52.559 --> 00:17:53.240 our entire. 346 00:17:53.119 --> 00:17:55.839 Cluster just for the one very important problem. 347 00:17:56.039 --> 00:17:58.759 In the test time, all the deply turned out right, 348 00:17:58.839 --> 00:18:02.119 So I think back then we'll building the GBU clusters together. Uh, 349 00:18:02.440 --> 00:18:06.920 you're applobing cables, and I remember that when we turned 350 00:18:06.920 --> 00:18:09.640 on the first initial test, you can hear all the 351 00:18:09.680 --> 00:18:11.640 GPS humming in the hallway. 352 00:18:12.119 --> 00:18:13.559 That's almost feel like spiritual. 353 00:18:14.519 --> 00:18:16.960 Yeah, that's actually a pretty cool thing that we're able 354 00:18:16.960 --> 00:18:18.839 to do. That we can go into the data center 355 00:18:19.160 --> 00:18:21.799 and tinker with the machines there. So for example, we 356 00:18:22.200 --> 00:18:25.160 went in and we unplucked a few of the cables 357 00:18:25.359 --> 00:18:28.079 and just made sure that our training setup is still running, 358 00:18:28.359 --> 00:18:31.039 running stably. So that's something that's you know, I think 359 00:18:31.440 --> 00:18:34.400 most AI you know teams out there don't usually do. 360 00:18:34.559 --> 00:18:37.319 But it's actually totally unlocks like a new level of 361 00:18:37.319 --> 00:18:40.599 reliability and what you're able to do with the hypers. 362 00:18:40.640 --> 00:18:42.759 So okay, so when are we going to solve remont? 363 00:18:43.279 --> 00:18:47.759 So the easiest solution is to enumerate over all possible 364 00:18:47.759 --> 00:18:51.759 strains and as all you have a verifier, en up compute, 365 00:18:51.960 --> 00:18:52.839 you'll be able to do it. 366 00:18:52.960 --> 00:18:56.440 Okay, my projection will be what your guess, what is 367 00:18:56.480 --> 00:18:57.599 your neural edge? Calculate? 368 00:18:58.359 --> 00:19:00.920 So my my boat for the so so three years 369 00:19:00.920 --> 00:19:04.559 ago I told you that I think not two years later, 370 00:19:05.200 --> 00:19:06.240 two things is gonna happen. 371 00:19:06.480 --> 00:19:09.200 We're gonna see machines win some battles. 372 00:19:09.480 --> 00:19:14.359 Yes, two ways a word, fields metal, Globel price with 373 00:19:14.519 --> 00:19:16.240 probably some expert in the loop. 374 00:19:16.319 --> 00:19:19.319 Right, so the expert uplifting. So this year or next year, 375 00:19:20.640 --> 00:19:22.279 that's what it comes down to you. Yeah, So it 376 00:19:22.319 --> 00:19:24.680 looks like GROG finished all of its thinking on the 377 00:19:24.759 --> 00:19:26.880 on the two problems. So let's take a look at 378 00:19:26.880 --> 00:19:30.200 what it said. All right, so this was the little physics. 379 00:19:29.759 --> 00:19:30.640 Problem we had. 380 00:19:31.559 --> 00:19:34.400 You know, we we've collapsed the thoughts here, so they're 381 00:19:34.720 --> 00:19:37.599 you know, they're hidden. And then we see Grock's answer 382 00:19:37.680 --> 00:19:40.039 below that, so it explains it wrought a Python script 383 00:19:40.039 --> 00:19:42.839 here using map plot Lip then gives us all of 384 00:19:42.880 --> 00:19:45.759 the code. So let's take a quick look at the code. 385 00:19:45.839 --> 00:19:48.319 You know, it seems like it's doing reasonable things here, 386 00:19:48.480 --> 00:19:53.240 not not totally of the mark, solf Capitler says here, 387 00:19:53.319 --> 00:19:58.359 so maybe it's solving capitalist laws, capital capitalist law and americally. Yeah, 388 00:19:58.400 --> 00:20:00.599 there's really only one way to find out if this 389 00:20:00.640 --> 00:20:01.160 thing is working. 390 00:20:01.240 --> 00:20:03.079 I would say, let's let's give it a try. Let's run. 391 00:20:03.400 --> 00:20:07.119 Let's run the code all right, and we can see, yeah, 392 00:20:07.119 --> 00:20:11.000 I've got animating two different planets, Earth and Mars here, 393 00:20:11.039 --> 00:20:15.960 and then the green ball is the vehicle that's transiting 394 00:20:16.039 --> 00:20:19.799 the spacecraft that's transitioning between Earth and Mars, and you 395 00:20:19.839 --> 00:20:22.960 could see the journey from Earth to Mars and looks like, yeah, 396 00:20:23.119 --> 00:20:26.880 indeed the astronauts were turned safely, you know, at the 397 00:20:26.960 --> 00:20:31.519 right moment in time. So obviously this was just generated 398 00:20:31.559 --> 00:20:33.440 on the spot, so we can't tell you if that 399 00:20:33.599 --> 00:20:36.039 was actually correct solution. So we're going to take a 400 00:20:36.039 --> 00:20:37.599 close a look. Now, maybe we're going to call some 401 00:20:37.640 --> 00:20:40.880