WEBVTT 1 00:00:00.160 --> 00:00:04.559 Welcome to our deep dive into the world of Linux 2 00:00:04.559 --> 00:00:05.400 system programming. 3 00:00:05.480 --> 00:00:09.160 Yeah, we're going to be exploring the core the Linux 4 00:00:09.199 --> 00:00:12.919 operating system. Okay, using this book is our guide. O'Reilly 5 00:00:13.000 --> 00:00:16.440 dot Linux dot system dot Programming, dot second edition dot 6 00:00:16.480 --> 00:00:18.160 pdf as our guide. 7 00:00:18.239 --> 00:00:20.800 So we're really getting down into the nuts and. 8 00:00:20.760 --> 00:00:23.920 Bolts we are. We're going to be uncovering any gritty Yeah, 9 00:00:24.239 --> 00:00:25.719 how things work at the lowest level. 10 00:00:25.879 --> 00:00:26.320 Awesome. 11 00:00:27.120 --> 00:00:29.800 And you know, it's not just about code, it's about understanding, 12 00:00:29.839 --> 00:00:33.359 like those fundamental building blocks that make Linux tick. Okay, 13 00:00:33.439 --> 00:00:36.560 So think system calls. Yeah, those are the direct lines 14 00:00:36.600 --> 00:00:38.359 to the heart of the OS, to the kernel, to 15 00:00:38.399 --> 00:00:41.039 the kernel exactly, okay, and the logic that drives it all. 16 00:00:41.119 --> 00:00:43.600 So we're not just driving the car. We're actually seeing 17 00:00:43.640 --> 00:00:45.119 how the engine works exactly. 18 00:00:45.200 --> 00:00:47.759 And here's a fun fact for you to kick things off. 19 00:00:47.799 --> 00:00:48.280 Are you ready? 20 00:00:48.320 --> 00:00:48.679 I'm ready? 21 00:00:48.759 --> 00:00:52.759 Okay. Linux actually uses way fewer system calls than something 22 00:00:52.799 --> 00:00:53.399 like Windows. 23 00:00:53.719 --> 00:00:54.079 Really. 24 00:00:54.719 --> 00:00:57.799 Yeah, we're talking hundreds versus potentially thousand. 25 00:00:58.000 --> 00:01:01.200 Wow. I would have thought a complex system like Linux 26 00:01:01.240 --> 00:01:04.480 would need way more ways to talk to its kernel. 27 00:01:04.560 --> 00:01:06.280 That's where the elegance of Linux comes in. 28 00:01:06.439 --> 00:01:06.879 Okay. 29 00:01:07.000 --> 00:01:11.000 It achieves a lot with a surprisingly streamlined set of tools. 30 00:01:11.640 --> 00:01:12.439 That's pretty cool. 31 00:01:12.519 --> 00:01:12.799 Yeah. 32 00:01:12.879 --> 00:01:15.599 So before we get too deep into the weeds, though, 33 00:01:16.120 --> 00:01:18.920 let's take a step back and defind something. Okay, what 34 00:01:19.079 --> 00:01:20.840 exactly is system programming? 35 00:01:21.040 --> 00:01:22.239 Oh that's a great question. 36 00:01:22.840 --> 00:01:25.040 When I think about using a computer, I don't think 37 00:01:25.040 --> 00:01:26.239 about this level. 38 00:01:26.239 --> 00:01:28.640 Right, You're not really meant to You're using all the 39 00:01:28.719 --> 00:01:31.840 nice applications and things, exactly. The system programming is kind 40 00:01:31.879 --> 00:01:34.879 of like, think of it like this. It's like being 41 00:01:34.879 --> 00:01:36.120 the architect of a building. 42 00:01:36.719 --> 00:01:37.120 Okay. 43 00:01:37.200 --> 00:01:39.680 So you're working with the raw materials, the foundation, the 44 00:01:39.719 --> 00:01:41.359 structural framework. 45 00:01:41.000 --> 00:01:43.359 Not just like decorating the rooms exactly. 46 00:01:43.439 --> 00:01:45.159 Yeah, you're building the load bearing walls. 47 00:01:45.359 --> 00:01:47.159 Ooh, I like that metaphor. 48 00:01:47.239 --> 00:01:51.120 Yeah. So we're writing software that interacts directly with the 49 00:01:51.159 --> 00:01:55.560 core of the operating system, the kernel. Okay, and it's 50 00:01:55.799 --> 00:01:59.959 closest partners okay, like the C library known as glib online. 51 00:02:00.840 --> 00:02:04.439 It's the bedrock upon which all those user friendly applications 52 00:02:04.480 --> 00:02:04.879 are built. 53 00:02:05.079 --> 00:02:07.359 So if I'm writing a game or a web browser, 54 00:02:07.359 --> 00:02:11.159 that's like decorating the rooms. But system programming is making 55 00:02:11.199 --> 00:02:11.719 the rooms. 56 00:02:12.280 --> 00:02:13.280 That's a great way to put it. 57 00:02:13.319 --> 00:02:14.599 Okay, I like it. I'm with you. 58 00:02:15.039 --> 00:02:18.000 And just like a good architect needs the right tools, right, 59 00:02:18.080 --> 00:02:21.159 absolutely so do system programmers of course. So the g 60 00:02:21.240 --> 00:02:24.000 and UC compiler okay, where GCC is one of our 61 00:02:24.039 --> 00:02:25.159 main tools in this world. 62 00:02:25.240 --> 00:02:27.319 So that's the blueprint and our hammer. 63 00:02:27.560 --> 00:02:29.240 Yes, like that our tool belt. 64 00:02:29.319 --> 00:02:32.680 And let's not forget about system calls. Ooh, yes, these 65 00:02:32.680 --> 00:02:36.120 are like the language we use to communicate with the kernel. Okay, 66 00:02:36.479 --> 00:02:39.479 So need to open a file, yep, send data over 67 00:02:39.520 --> 00:02:42.840 the network, even just allocate some memory. It all starts 68 00:02:42.840 --> 00:02:45.639 with a system call, got it. So you're directly asking 69 00:02:45.639 --> 00:02:47.080 the kernel to do something for you. 70 00:02:47.400 --> 00:02:51.599 Those are those direct lines of communication we talked about exactly. 71 00:02:51.919 --> 00:02:54.240 And the book also spends a lot of time talking 72 00:02:54.240 --> 00:02:54.960 about files. 73 00:02:55.080 --> 00:02:55.680 Yes it does. 74 00:02:55.960 --> 00:02:59.039 Why are files so central to Linux? Yeah? I was 75 00:02:59.080 --> 00:03:00.960 wondering that too, because I know, I know we have them. 76 00:03:00.800 --> 00:03:03.800 But well this might surprise you, but in Linux, pretty 77 00:03:03.840 --> 00:03:05.599 much everything is represented as a file. 78 00:03:05.879 --> 00:03:08.000 Wait a minute, what everything is a file? 79 00:03:08.120 --> 00:03:11.319 We're talking devices, directories, even your terminal window. 80 00:03:11.360 --> 00:03:14.439 Hold on, So my keyboard is a file. Yep, my 81 00:03:14.560 --> 00:03:15.639 hard drive is a file. 82 00:03:16.199 --> 00:03:16.599 It is. 83 00:03:16.759 --> 00:03:18.599 That's kind of blowing my mind a little bit. 84 00:03:18.639 --> 00:03:21.719 It is. It's a beautifully unifying concept it is. 85 00:03:21.879 --> 00:03:24.639 But how does the kernel keep track of all of that. 86 00:03:24.639 --> 00:03:26.759 That's where inodes and file descriptors come in. 87 00:03:26.919 --> 00:03:29.319 Okay, I'm intrigued, tell me more. 88 00:03:29.439 --> 00:03:32.919 So. Think of an inode like a file's unique ID card. 89 00:03:33.080 --> 00:03:33.400 Okay. 90 00:03:33.680 --> 00:03:39.800 It holds all this essential information, size, permissions, ownership, time stamps. 91 00:03:39.840 --> 00:03:41.360 It's like the file's permanent record. 92 00:03:41.800 --> 00:03:44.159 It's it's the file's permanent record. A file descripture, on 93 00:03:44.199 --> 00:03:47.039 the other hand, is like a temporary handle your program 94 00:03:47.159 --> 00:03:48.639 gets to interact with that file. 95 00:03:48.719 --> 00:03:51.639 Okay. So the inode is the files master record, and 96 00:03:51.759 --> 00:03:55.520 the descriptor is my program's key to access it precisely. 97 00:03:55.759 --> 00:03:57.680 I like it. Okay. Now, all these files they live 98 00:03:57.719 --> 00:04:00.560 in a file system, right, But unlike some other systems 99 00:04:00.680 --> 00:04:03.120 where you might have different drives and they have their 100 00:04:03.159 --> 00:04:05.759 own separate spaces, like your C drive and your D drive. 101 00:04:05.879 --> 00:04:09.000 Yeah yeah, yeah, Linux uses a unified name space. 102 00:04:09.120 --> 00:04:11.360 So it's all just one big, happy family exactly. 103 00:04:11.560 --> 00:04:14.919 It's like having one giant filing cabinet for everything, okay. 104 00:04:15.000 --> 00:04:17.519 And this makes life much easier for programs. They don't 105 00:04:17.560 --> 00:04:20.720 need to worry about where a file is physically located. 106 00:04:21.079 --> 00:04:23.759 That makes sense now, Historically, this name space was shared 107 00:04:23.800 --> 00:04:26.519 by everyone. Okay, but Linux has evolved. 108 00:04:26.800 --> 00:04:28.279 It always does, it always does. 109 00:04:28.399 --> 00:04:31.800 It's always changed to support per process name spaces. 110 00:04:31.839 --> 00:04:33.680 Poopresss name spaces. Now, what is that? 111 00:04:34.040 --> 00:04:37.040 It means that each process can have its own customized 112 00:04:37.120 --> 00:04:40.319 view of the file system ooh if needed. So it's 113 00:04:40.360 --> 00:04:43.720 like having your own personal filing cabinet within that larger one. 114 00:04:43.920 --> 00:04:46.600 Okay, that's a cool feature. Yeah, all right, so we've 115 00:04:46.600 --> 00:04:49.399 got files, we've got this unified name space. But what 116 00:04:49.480 --> 00:04:50.480 about processes? 117 00:04:50.720 --> 00:04:52.040 Ah? Yes, processes. 118 00:04:52.079 --> 00:04:54.759 Those are those mysterious things that make our programs actually 119 00:04:54.839 --> 00:04:55.560 come to life. 120 00:04:55.800 --> 00:04:59.040 You're exactly right. A process is essentially a running instance 121 00:04:59.079 --> 00:05:03.279 of your program. Imagine it's like this little virtualized world 122 00:05:03.680 --> 00:05:06.560 with its own memory space, got it resources and the 123 00:05:06.600 --> 00:05:08.839 illusion that it has the whole computer to itself. 124 00:05:08.920 --> 00:05:12.120 So each program gets its own little sandbox to play in. 125 00:05:12.240 --> 00:05:14.680 Yes, and it's unaware of the other kids playing in 126 00:05:14.680 --> 00:05:18.399 the other sandboxes. Right, and that isolation is key for 127 00:05:18.560 --> 00:05:19.680 stability and security. 128 00:05:19.839 --> 00:05:20.399 Makes sense. 129 00:05:20.920 --> 00:05:23.560 Now, A process goes through a whole life cycle, right, 130 00:05:24.120 --> 00:05:27.759 it's born, It lives its life executing your code. 131 00:05:27.839 --> 00:05:28.720 It has a purpose. 132 00:05:29.000 --> 00:05:31.680 It has a purpose, and eventually it meets its end. Okay, 133 00:05:31.839 --> 00:05:34.759 Sometimes it leaves behind a temporary ghost known as a 134 00:05:34.839 --> 00:05:35.720 zombie process. 135 00:05:35.959 --> 00:05:38.560 A zombie process that sounds a little ominous. 136 00:05:38.720 --> 00:05:41.120 It does sound ominous, but don't worry. They're usually harmlessness. 137 00:05:41.160 --> 00:05:41.759 Okay, good. 138 00:05:41.879 --> 00:05:44.319 It just means that the process is finished running, but 139 00:05:44.439 --> 00:05:47.279 its parent hasn't yet acknowledged its demise. 140 00:05:47.160 --> 00:05:48.519 So it's still kind of hanging around. 141 00:05:48.600 --> 00:05:50.759 Yeah, it's just waiting for that final goodbye. 142 00:05:51.079 --> 00:05:51.720 Interesting. 143 00:05:51.879 --> 00:05:55.879 But during its life, a process has certain permissions that 144 00:05:56.000 --> 00:06:00.879 dictate what it can and can't do. It receive signals, okay, 145 00:06:01.199 --> 00:06:03.720 those are like interprocess messages, and it has to handle 146 00:06:03.800 --> 00:06:04.720 errors gracefully. 147 00:06:04.959 --> 00:06:07.240 So every program needs some rules in some ways to 148 00:06:07.279 --> 00:06:08.439 communicate exactly. 149 00:06:08.480 --> 00:06:17.759 It needs some etiquette, some manners. Yeah, out reading and 150 00:06:17.759 --> 00:06:18.519 writing data. 151 00:06:19.199 --> 00:06:22.480 That's a great question, because, yeah, how do they do that. 152 00:06:23.040 --> 00:06:25.759 Well, that's where input output are IO comes in the 153 00:06:25.839 --> 00:06:28.759 art of reading and writing data, and in Linux this 154 00:06:28.879 --> 00:06:32.160 relies heavily on those system calls we talked about. 155 00:06:31.959 --> 00:06:33.839 Earlier, those direct lines to the kernel. 156 00:06:33.920 --> 00:06:34.360 Exactly. 157 00:06:34.480 --> 00:06:34.879 Awesome. 158 00:06:34.920 --> 00:06:37.319 So imagine you're downloading a movie. Okay, the first thing 159 00:06:37.360 --> 00:06:38.800 you do is you find the right url. 160 00:06:39.360 --> 00:06:41.839 I find it on my favorite streaming service, Yes. 161 00:06:41.680 --> 00:06:44.079 Exactly, that's like the open system call. 162 00:06:44.160 --> 00:06:44.439 Okay. 163 00:06:44.800 --> 00:06:47.920 Then you start receiving those data packets yep, that's read it. 164 00:06:48.160 --> 00:06:50.720 And finally you save those packets to your hard drive. 165 00:06:51.000 --> 00:06:53.319 That's right, right, okay, and when you're done, you close 166 00:06:53.360 --> 00:06:54.360 the connection. That's closed. 167 00:06:54.639 --> 00:06:54.959 Okay. 168 00:06:55.000 --> 00:06:56.920 That makes a lot more sense now, Yeah, it's nice 169 00:06:56.920 --> 00:06:57.680 little analogy. 170 00:06:57.759 --> 00:06:58.519 Yeah, I like it. 171 00:06:58.920 --> 00:07:01.360 So those are the basics of got it. What about 172 00:07:01.360 --> 00:07:02.639 this buffer io i've heard about? 173 00:07:02.959 --> 00:07:04.800 Yeah? Is that just like a fancy name for the 174 00:07:04.839 --> 00:07:05.360 same thing. 175 00:07:05.720 --> 00:07:08.279 Not exactly. Buffer io is all about performance. 176 00:07:08.399 --> 00:07:10.480 Ooh performance. We always like that. 177 00:07:10.600 --> 00:07:13.920 We always like that. So remember each system call has 178 00:07:13.959 --> 00:07:14.920 a bit of overhead. 179 00:07:15.079 --> 00:07:15.480 Okay. 180 00:07:15.600 --> 00:07:17.480 So if you're reading a file like one bite at 181 00:07:17.519 --> 00:07:19.879 a time, yeah, you're making a lot of system calls. 182 00:07:19.959 --> 00:07:21.680 That sounds inefficient, it is. 183 00:07:21.720 --> 00:07:22.720 That's where buffering comes in. 184 00:07:23.040 --> 00:07:24.480 We're bundling those calls together. 185 00:07:24.519 --> 00:07:27.519 They get it, Okay. The c standard io library, which 186 00:07:27.560 --> 00:07:32.480 you access using stideo, right, it uses this technique. Ok 187 00:07:32.720 --> 00:07:36.879 So it uses functions like fopen, fred, fright and f 188 00:07:36.959 --> 00:07:39.879 close to work with files. But it does so in 189 00:07:40.000 --> 00:07:42.399 chunks okay, and it reduces the number of times it 190 00:07:42.439 --> 00:07:44.079 needs to talk directly to the kernel. 191 00:07:44.560 --> 00:07:46.720 That makes sense. It's just being a little bit smarter 192 00:07:46.839 --> 00:07:47.279 about it. 193 00:07:47.439 --> 00:07:48.759 Exactly. It's being efficient. 194 00:07:49.040 --> 00:07:51.959 It's like ordering in bulk exactly. I like it. 195 00:07:52.000 --> 00:07:54.920 But what about when we have multiple threads uh oh 196 00:07:55.040 --> 00:07:56.279 accessing the same file. 197 00:07:56.519 --> 00:07:58.959 Yeah, that sounds like it could get a little messy. 198 00:07:59.000 --> 00:08:01.519 You could get messy. That's where thread safety comes in. Okay, 199 00:08:01.560 --> 00:08:05.360 So when you have multiple threads accessing shared resources like files, 200 00:08:05.920 --> 00:08:09.600 you need mechanisms like locking to ensure that they don't 201 00:08:09.720 --> 00:08:12.839 corrupt data, of course, by trying to write at the 202 00:08:12.879 --> 00:08:13.399 same time. 203 00:08:13.879 --> 00:08:15.240 Yeah, you don't want them to step in on each 204 00:08:15.240 --> 00:08:16.480 other's toes exactly. 205 00:08:16.560 --> 00:08:18.199 That's like having a traffic light system. 206 00:08:18.360 --> 00:08:19.120 Oh. I like that. 207 00:08:19.279 --> 00:08:22.319 Yeah, to control access to a busy intersection, keep. 208 00:08:22.199 --> 00:08:26.240 Things flowing smoothly exactly. Okay, so buffer io is great, 209 00:08:26.639 --> 00:08:30.120 but we need to be careful when multiple threads are involved. 210 00:08:30.120 --> 00:08:30.839 We do, we do. 211 00:08:31.120 --> 00:08:34.200 This is getting pretty deep. Yeah, what about techniques that 212 00:08:34.320 --> 00:08:38.480 go beyond the basics of io Ooh okay, it's lurking 213 00:08:38.519 --> 00:08:40.159 in those advanced waters. 214 00:08:40.279 --> 00:08:42.200 Get ready to level up your IO skills. 215 00:08:42.279 --> 00:08:43.240 I'm ready. Let's go. 216 00:08:43.279 --> 00:08:45.799 We're going to talk about techniques like scatter gather io, 217 00:08:45.919 --> 00:08:50.200 event pulling, and even memory mapping, each with their own superpowers. 218 00:08:50.399 --> 00:08:53.240 Okay, I'm ready for those superpowers. All right, let's start 219 00:08:53.279 --> 00:08:57.120 with scatter gather io. What kind of magic does that involve? 220 00:08:57.320 --> 00:09:00.519 Imagine you have data coming from three different sources, okay, 221 00:09:01.000 --> 00:09:03.519 and you need to write it all to a single 222 00:09:03.600 --> 00:09:07.279 file instead of making three separate right calls. 223 00:09:07.480 --> 00:09:08.720 Yeah, that sounds inefficient. 224 00:09:08.840 --> 00:09:11.320 It is scatter gather io. Let's you do it in 225 00:09:11.360 --> 00:09:14.879 one shot, okay, combining those data chunks into a single 226 00:09:14.879 --> 00:09:15.840 efficient operation. 227 00:09:16.080 --> 00:09:19.480 So it's like optimizing a delivery route by combining multiple 228 00:09:19.480 --> 00:09:20.639 packages exactly. 229 00:09:20.759 --> 00:09:23.759 I like it. Linux provides system calls like rehobar and 230 00:09:23.919 --> 00:09:27.759 writeev and they use a special structure called iovec to 231 00:09:27.840 --> 00:09:29.679 define those vectors of buffers. 232 00:09:29.759 --> 00:09:30.159 Got it? 233 00:09:30.200 --> 00:09:31.240 Making this magic happen. 234 00:09:31.279 --> 00:09:31.840 That's awesome. 235 00:09:32.000 --> 00:09:33.840 But what if you're dealing with a lot of files 236 00:09:33.879 --> 00:09:36.919 and you need to be notified when something happens, like 237 00:09:37.039 --> 00:09:38.080 new data arriving? 238 00:09:38.240 --> 00:09:40.679 Ooh yeah, that sounds like a lot to keep track of. 239 00:09:41.159 --> 00:09:43.519 Is that's where event polling comes in, like having a 240 00:09:43.559 --> 00:09:45.080 surveillance system for your files. 241 00:09:45.279 --> 00:09:45.919 I like it. 242 00:09:45.960 --> 00:09:49.159 So the older system calls select and poll Okay. There 243 00:09:49.159 --> 00:09:52.039 were early attempts at this okay, but they had limitations 244 00:09:52.159 --> 00:09:54.200 when you had a huge number of files to watch. 245 00:09:54.360 --> 00:09:55.320 Yeah, that makes sense. 246 00:09:55.480 --> 00:09:59.600 Thankfully, Linux introduced the epole facility, okay, which is much 247 00:09:59.600 --> 00:10:01.600 more effe at monitoring multiple files. 248 00:10:01.679 --> 00:10:04.559 So it's like upgrading from a security guard watching grainy 249 00:10:04.639 --> 00:10:08.360 CCTV footage, yes, to like a high tech system with 250 00:10:08.559 --> 00:10:09.759 AI powered alerts. 251 00:10:10.519 --> 00:10:13.639 That's a great analogy. And finally, we have memory mapping, 252 00:10:14.440 --> 00:10:16.440 which is where things get really interesting. 253 00:10:16.600 --> 00:10:17.879 Okay, I'm intrigued. 254 00:10:18.240 --> 00:10:21.440 Tell me mar Imagine you have this massive file I 255 00:10:21.440 --> 00:10:24.799 think a huge database or a high resolution image. With 256 00:10:24.960 --> 00:10:28.759 memory mapping, you can map that entire file into your 257 00:10:28.799 --> 00:10:31.759 process's address space. Oh wow, you can treat it as 258 00:10:31.840 --> 00:10:34.320 if it were already loaded into memory. No need to 259 00:10:34.360 --> 00:10:35.080 read it in chunks. 260 00:10:35.120 --> 00:10:36.879 So I have instant access. 261 00:10:36.519 --> 00:10:37.720 You have direct access. 262 00:10:37.879 --> 00:10:39.240 Why wouldn't we always do this. 263 00:10:39.320 --> 00:10:42.519 It's a fantastic tool for performance, especially with large files. 264 00:10:43.720 --> 00:10:49.000 The system calls map and mun map are your keys 265 00:10:49.039 --> 00:10:49.879 to this kingdom. 266 00:10:50.039 --> 00:10:51.039 I'm ready to rule. 267 00:10:51.679 --> 00:10:54.240 But it's important to understand that there are different ways 268 00:10:54.279 --> 00:10:57.559 memory mapping can work, and you can use flags like 269 00:10:57.679 --> 00:10:59.279 map shared and map private. 270 00:10:59.440 --> 00:11:01.159 Interesting, what's the difference So with. 271 00:11:01.240 --> 00:11:05.679 Map shared If multiple processes map the same file, any 272 00:11:05.799 --> 00:11:08.720 changes one process makes are visible to the others. Okay, 273 00:11:08.799 --> 00:11:11.440 it's like a shared Google doc got it map Private, 274 00:11:11.519 --> 00:11:15.480 on the other hand, creates a private copy for each process, 275 00:11:15.960 --> 00:11:17.759 so changes are isolated. 276 00:11:17.440 --> 00:11:20.320 So it's like working on my own personal copy exactly. Okay, 277 00:11:20.360 --> 00:11:20.840 that makes sense. 278 00:11:20.919 --> 00:11:23.279 Yeah, so memory mapping incredibly powerful. 279 00:11:23.360 --> 00:11:24.440 Wow, very cool. 280 00:11:24.480 --> 00:11:27.080 But all this talk about memory makes me realize we 281 00:11:27.120 --> 00:11:29.879 haven't really talked about how Linux actually manages memory. 282 00:11:30.039 --> 00:11:30.960 Yeah. That's a big one. 283 00:11:31.039 --> 00:11:31.840 Yeah, it's a big one. 284 00:11:31.840 --> 00:11:33.639 How does it ensure everything runs smoothly? 285 00:11:33.960 --> 00:11:37.399 Well? Memory management is crucial, I would think. So. Yeah, 286 00:11:37.440 --> 00:11:40.799 it's the unsung hero of any operating system. Okay, making 287 00:11:40.840 --> 00:11:44.080 sure that each process gets fair share of memory. Okay, 288 00:11:44.120 --> 00:11:45.720 and that everything plays nicely together. 289 00:11:45.799 --> 00:11:47.200 So it's like a master conductor. 290 00:11:47.600 --> 00:11:50.080 It's orchestrating this complex symphony. 291 00:11:50.480 --> 00:11:51.000 I like that. 292 00:11:51.320 --> 00:11:53.919 So first we need to understand the process address space. 293 00:11:54.200 --> 00:11:57.399 Remember those isolated sandboxes we talked about earlier. Yes, well, 294 00:11:57.440 --> 00:12:00.679 the sandboxes are further divided into pages, which are these 295 00:12:00.799 --> 00:12:04.919 chunks of memory. Okay, not all pages are created equal, though, Oh, 296 00:12:05.039 --> 00:12:08.759 you have valid pages that correspond to actual data in 297 00:12:08.840 --> 00:12:12.440 RAM okay, or on disc and you have invalid pages 298 00:12:13.000 --> 00:12:15.000 that represent unallocated regions. 299 00:12:15.320 --> 00:12:18.120 So accessing an invalid page is like falling into a 300 00:12:18.200 --> 00:12:19.600 trapdoor in the sandbox. 301 00:12:19.840 --> 00:12:21.559 That's a great way to put it. And trying to 302 00:12:21.600 --> 00:12:25.120 access memory that's not yours is a big no no. Yeah, 303 00:12:25.159 --> 00:12:29.200 that's bad and usually results in a segmentation violation, a 304 00:12:29.240 --> 00:12:30.240 programmer's nightmare. 305 00:12:30.399 --> 00:12:32.799 Oh, segmentation violation. I've heard that term. 306 00:12:32.960 --> 00:12:35.559 It's not good. That's bad, it's not good. 307 00:12:35.639 --> 00:12:36.240 I don't like that. 308 00:12:36.279 --> 00:12:39.879 But thankfully, goodness, we have the memory management unit or MMU. 309 00:12:40.039 --> 00:12:41.279 Okay, the MMU. 310 00:12:41.399 --> 00:12:44.039 The MMU to keep things in check. It acts as 311 00:12:44.080 --> 00:12:48.240 a translator, converting those virtual addresses used by our program 312 00:12:48.480 --> 00:12:52.360 into physical addresses okay, where that data actually resides in RAM. 313 00:12:52.600 --> 00:12:55.799 So the MMU is like a postal worker who knows 314 00:12:55.840 --> 00:12:58.360 the real address behind every po box. 315 00:12:58.559 --> 00:13:00.279 Another perfect analogy. 316 00:13:00.320 --> 00:13:00.799 I'm falling. 317 00:13:00.879 --> 00:13:04.679 So the MMU also plays a vital role in memory protection, 318 00:13:05.600 --> 00:13:09.039 making sure that processes don't wander into memory territory that 319 00:13:09.120 --> 00:13:10.159 doesn't belong to them. 320 00:13:10.519 --> 00:13:11.559 They stay in their own lanes. 321 00:13:11.600 --> 00:13:13.639 They stay in their own lane. Okay, but what about 322 00:13:13.679 --> 00:13:16.240 when a program needs to allocate memory on the fly. 323 00:13:16.759 --> 00:13:20.039 Yeah, like when I'm adding things to my shopping cart online. 324 00:13:19.519 --> 00:13:22.600 Exactly, and the website needs to store that information somewhere 325 00:13:22.759 --> 00:13:26.279 right exactly. That's where dynamic memory allocation comes in, okay, 326 00:13:26.879 --> 00:13:29.240 and it's handled through a special region of memory called 327 00:13:29.279 --> 00:13:33.240 the heap. The heap, you use functions like malik, calic, 328 00:13:33.639 --> 00:13:37.639 realic and free to request and release memory from this 329 00:13:37.759 --> 00:13:38.759 heap as needed. 330 00:13:39.039 --> 00:13:41.639 So the heap is like a self storage facility. It 331 00:13:41.720 --> 00:13:44.679 is where programs can rent units of various sizes. 332 00:13:44.840 --> 00:13:46.200 That's a great way to visualize it. 333 00:13:46.240 --> 00:13:47.519 Okay, I'm picturing it. 334 00:13:47.559 --> 00:13:50.759 But managing the heap can get tricky. You can end 335 00:13:50.840 --> 00:13:54.240 up with something called fragmentation. Fragmentation okay, where you have 336 00:13:54.360 --> 00:13:57.759 enough free space overall, ok but it's scattered around small chunks. 337 00:13:57.799 --> 00:13:59.279 You can't fit that big couching. 338 00:13:59.360 --> 00:14:01.200 You can't fit that big couchin exactly. 339 00:14:01.320 --> 00:14:05.919 Now you technically have enough space exactly. That's frustrating. 340 00:14:06.039 --> 00:14:09.399 It is frustrating. But Linux has some clever ways to 341 00:14:09.440 --> 00:14:12.480 deal with that, okay. Good. For larger chunks of memory, 342 00:14:12.519 --> 00:14:15.480 it can use something called anonymous memory mapping. 343 00:14:15.759 --> 00:14:17.639 Anonymous memory mapping. 344 00:14:17.360 --> 00:14:21.080 Yes, okay, reserving a contiguous block of memory. It's like 345 00:14:21.120 --> 00:14:22.639 renting a whole warehouse. 346 00:14:22.279 --> 00:14:26.039 Unit instead of trying to squeeze into those few small lockers. 347 00:14:26.360 --> 00:14:27.559 Exactly, that makes sense. 348 00:14:27.759 --> 00:14:30.759 Yeah, now, I remember you mentioned memory locking earlier. Yes, 349 00:14:30.879 --> 00:14:32.240 when would we want to use that? 350 00:14:32.720 --> 00:14:37.000 Memory locking is particularly useful when you have a critical 351 00:14:37.039 --> 00:14:41.399 piece of code that needs to access memory with absolutely 352 00:14:41.440 --> 00:14:42.159 no delays. 353 00:14:42.279 --> 00:14:43.519 Of course, you. 354 00:14:43.440 --> 00:14:45.559 Wouldn't want the kernel to swap that memory. 355 00:14:45.279 --> 00:14:47.519 Out to disc right, No, that would be terrible. 356 00:14:47.519 --> 00:14:49.000 It'd be like pausing a movie right in the middle 357 00:14:49.000 --> 00:14:52.840 of an action scene. So memory locking done through the 358 00:14:52.919 --> 00:14:58.480 lock system call okay. It ensures that those critical memory 359 00:14:58.480 --> 00:14:59.799 pages stay in. 360 00:14:59.799 --> 00:15:00.919 Rank they're locked in. 361 00:15:00.960 --> 00:15:04.159 They're locked in guaranteeing predictable performance. 362 00:15:04.200 --> 00:15:06.320 So we're putting a do not disturb sign on those 363 00:15:06.360 --> 00:15:07.559 memory pages, you got. 364 00:15:07.440 --> 00:15:11.120 It, preventing the kernel from relocating them. Awesome. It's a 365 00:15:11.159 --> 00:15:15.399 powerful tool. It sounds like it for performance sensitive applications. Yeah, 366 00:15:15.399 --> 00:15:19.279 but it's important to use it judiciously, as over using 367 00:15:19.399 --> 00:15:20.519 it can have drawbacks. 368 00:15:20.759 --> 00:15:22.519 Of course, too much of a good thing. 369 00:15:22.639 --> 00:15:24.799 Exactly, especially in multi threaded. 370 00:15:24.519 --> 00:15:26.639 Environments right where things are already complicated. 371 00:15:26.759 --> 00:15:27.279 Exactly. 372 00:15:27.399 --> 00:15:30.480 This whole deep dive into memory management has been fascinated. 373 00:15:30.559 --> 00:15:31.840 It is a fascinating topic. 374 00:15:31.919 --> 00:15:35.440 It really is. I'm starting to realize just how much 375 00:15:35.480 --> 00:15:39.399 complexity is hidden beneath the surface of something as seemingly 376 00:15:39.440 --> 00:15:40.759 simple as running a program. 377 00:15:40.799 --> 00:15:43.879 And we've only just begun. Oh No, there's a whole 378 00:15:44.000 --> 00:15:47.279 universe of fascinating concepts and techniques to explore. Bring it on. 379 00:15:47.519 --> 00:15:50.279 I'm real, but I'm glad you're starting to appreciate the 380 00:15:50.320 --> 00:15:52.639 elegance and power of Linux system programming. 381 00:15:52.679 --> 00:15:55.799 Absolutely i am. I'm already looking forward to uncovering more 382 00:15:55.840 --> 00:15:59.320 of these hidden gems right as we continue our deep dive. 383 00:15:59.639 --> 00:16:04.039 Awesome, Now that we've explored memory management, let's shift gears 384 00:16:04.039 --> 00:16:06.600 a bit, okay and talk about something a bit more tangible. 385 00:16:06.919 --> 00:16:07.679 All right, I'm with you. 386 00:16:07.840 --> 00:16:08.840 Files and directories. 387 00:16:09.120 --> 00:16:10.960 Okay, I can definitely relate to those. 388 00:16:11.080 --> 00:16:12.279 Yeah, we all have to deal with those. 389 00:16:12.360 --> 00:16:16.559 I spend way too much time organizing files on my computer. So, right, 390 00:16:16.759 --> 00:16:19.360 what kind of insights can system programming give us about 391 00:16:19.399 --> 00:16:20.440 files and directories? 392 00:16:20.519 --> 00:16:22.320 Well, first of all, we need a way to gather 393 00:16:22.440 --> 00:16:25.600 information about a file. Okay, think of it like a 394 00:16:26.000 --> 00:16:30.480 detective doing some reconnaissance. That's where the STAT family system 395 00:16:30.519 --> 00:16:33.159 calls comes in. They give you all sorts of details 396 00:16:33.200 --> 00:16:36.720 about a file, file sized, permissions, timestamps. 397 00:16:37.679 --> 00:16:39.320 So it's like a file's fingerprint. 398 00:16:39.480 --> 00:16:42.840 Exactly. It's a file's fingerprint, okay. So you can find 399 00:16:42.840 --> 00:16:44.879 out if it's a regular file or a directory, when 400 00:16:44.919 --> 00:16:47.279 it was last modified, who owns it, and even what 401 00:16:47.279 --> 00:16:48.720