WEBVTT 1 00:00:00.120 --> 00:00:03.359 Imagine finding a hidden door in your house that you 2 00:00:03.399 --> 00:00:04.759 never knew existed. 3 00:00:04.799 --> 00:00:06.799 Like a secret passage or something exactly. 4 00:00:06.879 --> 00:00:09.199 Yeah, you pry it open, you step inside, and you 5 00:00:09.240 --> 00:00:11.439 realize it's an entire master control room. 6 00:00:11.679 --> 00:00:11.960 Wow. 7 00:00:12.320 --> 00:00:15.359 Right, and suddenly you can bypass every single like switch, 8 00:00:15.400 --> 00:00:18.239 every thermostat, every lock in the building. You just have 9 00:00:18.359 --> 00:00:23.359 this absolute, unfettered access to the underlying wiring of your home. 10 00:00:23.679 --> 00:00:25.960 That's a powerful image and a little. 11 00:00:25.679 --> 00:00:28.800 Intimidating it is, But I mean that is exactly what 12 00:00:28.839 --> 00:00:30.920 happens when you open the command line on your computer. 13 00:00:31.679 --> 00:00:36.399 So welcome to this custom tailored deep dive, designed specifically 14 00:00:36.399 --> 00:00:36.799 for you. 15 00:00:36.960 --> 00:00:38.159 Glad to be here for this one. 16 00:00:38.280 --> 00:00:42.200 Yeah, Our mission today is to demystify this hidden world, 17 00:00:42.560 --> 00:00:46.039 and we're using excerpts from the Bashed cookbook. So, whether 18 00:00:46.079 --> 00:00:49.719 you're a seasoned programmer or you know, just insanely curious 19 00:00:49.759 --> 00:00:52.520 about what happens behind that graphical curtain of your desktop, 20 00:00:52.960 --> 00:00:56.399 our goal is to reveal how mastering this text based 21 00:00:56.520 --> 00:00:59.960 layer is. Well, it's basically a shortcut to total control 22 00:01:00.079 --> 00:01:00.799 over your machine. 23 00:01:00.880 --> 00:01:04.000 Yeah, Because you know, most of us are just entirely 24 00:01:04.040 --> 00:01:08.200 accustomed to a visual, white and click reality. We navigate 25 00:01:08.200 --> 00:01:10.840 our digital lives the exact same way we navigate a 26 00:01:10.840 --> 00:01:13.439 physical kitchen. Like, if you want to throw a file away, 27 00:01:14.200 --> 00:01:15.840 you look for a little trash can icon. 28 00:01:15.760 --> 00:01:18.439 Right to grab the file, drag it over done exactly. 29 00:01:18.519 --> 00:01:22.319 It's spatial and it feels safe. But when you strip 30 00:01:22.319 --> 00:01:27.280 away that graphical illusion, the trash can just vanishes, goof gone, right, 31 00:01:27.319 --> 00:01:30.439 and you're left staring at a blank screen with just 32 00:01:30.560 --> 00:01:34.599 a blinking cursor. I mean, it is an incredibly powerful 33 00:01:34.680 --> 00:01:37.519 text based environment, but it's just sitting there waiting for 34 00:01:37.599 --> 00:01:38.560 direct instructions. 35 00:01:38.680 --> 00:01:41.680 Okay, let's unpack this because before we can start typing 36 00:01:41.719 --> 00:01:44.200 commands into that blank screen, we kind of need to 37 00:01:44.280 --> 00:01:47.519 understand what we're actually typing into. We use the word 38 00:01:47.599 --> 00:01:50.719 shell all the time and computing, but to really grasp 39 00:01:50.840 --> 00:01:54.920 why a shell even exists, we have to rewind all 40 00:01:54.959 --> 00:01:57.200 the way to the nineteen seventies and look at the 41 00:01:57.239 --> 00:01:59.200 creation of the Unix operating system. 42 00:01:59.400 --> 00:02:01.519 Yeah, and if we connect this to the bigger picture, 43 00:02:01.760 --> 00:02:04.599 early computing was highly highly monolithic. 44 00:02:04.719 --> 00:02:05.959 What do you mean by monolithic? 45 00:02:06.079 --> 00:02:08.520 Well, the user interface and the core operating system were 46 00:02:08.560 --> 00:02:11.599 just welded together in one massive piece of software. Oh 47 00:02:11.599 --> 00:02:14.759 I see, Yeah, you were completely stuck with whatever interface 48 00:02:14.800 --> 00:02:19.159 the hardware vendor decided to provide. But the brilliant concept 49 00:02:19.280 --> 00:02:24.599 behind Unix was this clean, deliberate separation. They split the 50 00:02:24.639 --> 00:02:28.120 system into two distinct parts, the kernel and the shell. 51 00:02:28.400 --> 00:02:31.280 Kernel in the shell, right, the kernel is the core. 52 00:02:31.439 --> 00:02:34.560 It handles all the brutal heavy lifting of computing, like 53 00:02:35.039 --> 00:02:39.319 managing memory allocations, talking directly to the physical hardware, scheduling 54 00:02:39.360 --> 00:02:42.319 CPU time, and nuts and bolts. Exactly. It's all the 55 00:02:42.319 --> 00:02:45.479 stuff the average user never ever wants to manage manually. 56 00:02:45.639 --> 00:02:49.560 So if the kernel is essentially the car's engine and 57 00:02:49.599 --> 00:02:50.240 the drive. 58 00:02:50.080 --> 00:02:51.639 Train, right, that's a good way to look at it. 59 00:02:51.680 --> 00:02:54.280 Then the shell, I guess, is the steering wheel, the pedals, 60 00:02:54.319 --> 00:02:56.919 and the dashboard. It's the interface that lets the driver 61 00:02:57.039 --> 00:02:58.439 actually tell the engine what to do. 62 00:02:58.719 --> 00:03:03.439 Yes, the absolute genius of UNUS was realizing that you 63 00:03:03.479 --> 00:03:06.439 could swap out the steering wheel without having to rebuild 64 00:03:06.479 --> 00:03:09.960 the entire engine from scratch, which is huge. It is revolutionary. 65 00:03:10.120 --> 00:03:12.759 The shell became just another swappable program. 66 00:03:12.400 --> 00:03:15.479 And that decoupling must have allowed for like an absolute 67 00:03:15.520 --> 00:03:16.879 explosion of innovation right. 68 00:03:17.199 --> 00:03:21.639 Oh, completely, because if developers felt a command interface was inefficient, 69 00:03:22.000 --> 00:03:24.599 they didn't have to go and convince a massive corporation 70 00:03:24.960 --> 00:03:27.000 to update an entire operating system. 71 00:03:27.039 --> 00:03:29.199 They could just build their own steering wheel exactly. 72 00:03:29.240 --> 00:03:31.479 They can simply write a new shell, and they did. 73 00:03:31.800 --> 00:03:34.560 I mean, the sources mentioned the original born shell, which 74 00:03:34.599 --> 00:03:38.680 was known as SHE. Then the CShell Shish emerged. It 75 00:03:38.759 --> 00:03:41.879 had some new user conveniences, but it had quirks that 76 00:03:41.919 --> 00:03:43.400 made programming kind of difficult. 77 00:03:43.479 --> 00:03:44.680 Gotcha. 78 00:03:44.840 --> 00:03:49.439 Following that was the corn Shell Gish, And finally, the 79 00:03:49.560 --> 00:03:53.520 undisputed star of our source material today, the Born Again 80 00:03:53.639 --> 00:03:55.919 Shell or Bash bash. 81 00:03:55.680 --> 00:03:58.400 Which is basically everywhere now. Like, if you open a 82 00:03:58.520 --> 00:04:02.080 terminal on almost any line Nicks distribution or a Mac today, 83 00:04:02.599 --> 00:04:04.000 you were likely looking at Bash. 84 00:04:04.199 --> 00:04:05.120 Oh almost certainly. 85 00:04:05.159 --> 00:04:07.879 And the sources point out it achieved this total dominance 86 00:04:08.000 --> 00:04:11.159 largely due to the open source movement, right specifically the 87 00:04:11.199 --> 00:04:14.000 Genu project. Like Brian Fox wrote Bash as a free 88 00:04:14.039 --> 00:04:15.879 replacement for the original born Shell. 89 00:04:15.960 --> 00:04:17.199 Yeah, that was a huge part of it. 90 00:04:17.439 --> 00:04:21.199 But there's also this term mentioned in the text POSEX standardization. 91 00:04:21.759 --> 00:04:24.439 What actually is PO six and why did it help 92 00:04:24.600 --> 00:04:26.079 Bash take over the world? 93 00:04:26.279 --> 00:04:30.360 Right? So PO six stands for Portable Operating System interface. Okay, 94 00:04:30.800 --> 00:04:34.800 back in the day, different versions of Unix were fragmenting badly, 95 00:04:35.360 --> 00:04:38.920 like a script written on one system might completely break 96 00:04:38.920 --> 00:04:42.439 on another because the underlying commands behaved just slightly differently. 97 00:04:42.519 --> 00:04:44.639 Oh, that sounds like a nightmare for developers. 98 00:04:44.759 --> 00:04:48.199 It was so. POSX was created as a standardized set 99 00:04:48.240 --> 00:04:51.439 of rules. It dictated exactly how an operating system and 100 00:04:51.480 --> 00:04:54.639 its shell should behave and Bash was designed to be 101 00:04:54.959 --> 00:04:56.560 highly POSEX compliant. 102 00:04:56.879 --> 00:04:59.279 Ah, so it was like a guarantee exactly. 103 00:04:59.519 --> 00:05:01.560 It meant to developers knew if they wrote a script 104 00:05:01.560 --> 00:05:05.600 for Bash, it would run reliably across wildly different hardware systems. 105 00:05:05.920 --> 00:05:09.319 It essentially became the universal translator of systems administration. 106 00:05:09.519 --> 00:05:11.720 That makes total sense. So, okay, now that we know 107 00:05:11.759 --> 00:05:14.800 we're sitting behind the swappable steering wheel called Bash, we 108 00:05:14.920 --> 00:05:15.879 need to know how to. 109 00:05:15.879 --> 00:05:17.959 Read the dashboard, right, the interface itself. 110 00:05:18.199 --> 00:05:22.199 Yeah, we're staring at that blinking cursor and it's sitting 111 00:05:22.319 --> 00:05:25.079 right next to something called the prompt. And according to 112 00:05:25.120 --> 00:05:28.680 the cookbook, that prompt is packed with vital information about 113 00:05:28.720 --> 00:05:30.480 who you are and what you can do. 114 00:05:30.639 --> 00:05:32.199 It tells you everything you need to know about your 115 00:05:32.279 --> 00:05:32.759 current state. 116 00:05:32.920 --> 00:05:36.160 Right, and the very last character of the prompt is 117 00:05:36.439 --> 00:05:39.800 the dead giveaway like a trailing dollar sign means you 118 00:05:39.839 --> 00:05:44.040 are logged in as a regular user with standard safe permissions. Yes, 119 00:05:44.240 --> 00:05:47.720 but if you see a hash sign, you know the pound. 120 00:05:47.399 --> 00:05:49.560 Symbol ake, you need to be very very careful. 121 00:05:49.600 --> 00:05:51.040 Why what does the hash mean? 122 00:05:51.360 --> 00:05:53.759 It means you are operating as root, that is the 123 00:05:53.920 --> 00:05:58.920 all powerful system administrator account. You have zero restrictions. You 124 00:05:58.920 --> 00:06:03.000 can modify any file, read any data, and consequently you 125 00:06:03.040 --> 00:06:06.600 can accidentally obliterate the entire operating system with a single typo. 126 00:06:07.000 --> 00:06:09.959 Yikes. So the ash symbol is basically a glaring red 127 00:06:09.959 --> 00:06:12.720 warning light that the safety rails are completely off. 128 00:06:12.800 --> 00:06:15.519 Precisely, it's the shell saying I will do whatever you type, 129 00:06:15.600 --> 00:06:17.120 even if it destroys everything. 130 00:06:17.360 --> 00:06:20.600 Good to know. Okay, So, assuming we are safely using 131 00:06:20.600 --> 00:06:22.879 our dollar sign prompt, we actually want to run a command. 132 00:06:22.879 --> 00:06:25.560 The cookbook details several ways to find the tools we need. 133 00:06:25.720 --> 00:06:27.680 Right, Finding the right command is half the battle. 134 00:06:27.839 --> 00:06:31.079 Yeah. So there's type, which tells you if a command 135 00:06:31.240 --> 00:06:33.879 is a built in shell function or an external file. 136 00:06:34.319 --> 00:06:38.199 There's which which locates the exact folder a command lives in. 137 00:06:39.120 --> 00:06:41.439 And there's appropos, which I think is brilliant if you 138 00:06:41.439 --> 00:06:43.120 can't remember a command's name, you just give it a 139 00:06:43.199 --> 00:06:47.879 keyword and it searches the system's manual pages using regular expressions. 140 00:06:47.279 --> 00:06:48.759 Which is incredibly helpful. 141 00:06:48.920 --> 00:06:52.680 Yeah, and for anyone unfamiliar, a regular expression is essentially 142 00:06:52.720 --> 00:06:55.600 a sequence of characters that acts like a highly advanced, 143 00:06:55.720 --> 00:06:59.959 super powered search pattern. But this brings up a mechanical 144 00:07:00.240 --> 00:07:03.360 question for me. Okay, shoot, when I type something simple 145 00:07:03.759 --> 00:07:06.839 like l's to list my files, how does the shell 146 00:07:06.879 --> 00:07:09.560 actually know where that specific program is located on my 147 00:07:09.600 --> 00:07:10.120 hard drive? 148 00:07:10.279 --> 00:07:14.480 Oh? Well, it relies on an environment variable called the path. 149 00:07:14.639 --> 00:07:17.639 The path Yeah, the path through is simply a text 150 00:07:17.720 --> 00:07:21.480 string containing a list of direct relocations, and they're separated 151 00:07:21.519 --> 00:07:25.360 by colon's. Think of it as the shell's authorized search map. 152 00:07:25.560 --> 00:07:26.639 Okay, a search map. 153 00:07:26.800 --> 00:07:29.959 Right. When you type l's and a hit ender, Bash 154 00:07:30.000 --> 00:07:32.959 doesn't just scan your entire hard drive. That would take 155 00:07:33.120 --> 00:07:33.920 entirely too long. 156 00:07:34.040 --> 00:07:35.800 Oh yeah, that would be agonizing. 157 00:07:36.199 --> 00:07:39.680 Instead, it reads the path variable and checks the first 158 00:07:39.759 --> 00:07:43.000 folder on that list. If it finds an executable file 159 00:07:43.079 --> 00:07:45.800 named l's in that folder, it runs it immediately and 160 00:07:45.879 --> 00:07:48.839 stops searching. If not, it moves to the second folder, 161 00:07:49.079 --> 00:07:50.199 then the third, and so on. 162 00:07:50.399 --> 00:07:52.519 Wait, wait, I have to stop you there, because wouldn't 163 00:07:52.519 --> 00:07:55.240 it be infinitely easier to have the computer check the 164 00:07:55.240 --> 00:07:57.040 folder I'm currently standing in first? 165 00:07:57.040 --> 00:07:57.920 You would think, so, right? 166 00:07:58.000 --> 00:08:00.519 I mean, if I'm working inside a specific project folder 167 00:08:00.519 --> 00:08:03.480 and I type a command, logic dictates the shell should 168 00:08:03.480 --> 00:08:05.120 look right in front of my face before it goes 169 00:08:05.160 --> 00:08:08.000 trekking across the entire system map. Why on earth is 170 00:08:08.040 --> 00:08:08.879 it designed this way? 171 00:08:08.959 --> 00:08:12.199 Well, this raises an important question about the tension between 172 00:08:12.240 --> 00:08:15.480 convenience and security. Let's walk through what happens if we 173 00:08:15.519 --> 00:08:18.439 can figure your path the exact way you just described. 174 00:08:19.480 --> 00:08:22.399 We put your current directory, which Unix represents with a 175 00:08:22.439 --> 00:08:24.600 single dot. By the way, at the very front of 176 00:08:24.639 --> 00:08:26.639 your path makes sense to me. So the shell will 177 00:08:26.639 --> 00:08:28.560 now check whatever fold do you happen to be in 178 00:08:28.879 --> 00:08:30.680 before it checks the official system. 179 00:08:30.399 --> 00:08:33.480 Folders, which sounds incredibly efficient. 180 00:08:33.240 --> 00:08:35.519 Until you wander into a directory created by someone with 181 00:08:35.559 --> 00:08:40.240 malicious intent. Oh no, yeah, a bad actor knows that 182 00:08:40.320 --> 00:08:44.600 systems administrators frequently take LS to see what is inside 183 00:08:44.639 --> 00:08:46.279 a newly downloaded folder. 184 00:08:46.840 --> 00:08:50.200 It's muscle memory, right, you enter a folder, you type LS. 185 00:08:50.000 --> 00:08:55.240 Exactly, So this bad actor writes a highly destructive script, 186 00:08:55.480 --> 00:08:59.679 Maybe something that silently deletes databases or steals passwords. And 187 00:08:59.720 --> 00:09:02.080 they say that script in their folder and they name 188 00:09:02.120 --> 00:09:02.639 it els. 189 00:09:02.960 --> 00:09:04.240 Oh, I see where this is going. 190 00:09:04.320 --> 00:09:07.279 You navigate into their trap folder. You type ls fully 191 00:09:07.279 --> 00:09:09.879 expecting to see a list of files, But because your 192 00:09:09.879 --> 00:09:12.720 path is configured to check your current directory first, the 193 00:09:12.799 --> 00:09:16.480 shell finds their malicious script named l's and executes it 194 00:09:16.799 --> 00:09:20.360 using your account privileges. That is terrifying, and the shell 195 00:09:20.639 --> 00:09:25.639 stops searching instantly. It never even reaches the real secure 196 00:09:25.639 --> 00:09:29.360 Indel's program that's located deep in the system's binary folders. 197 00:09:29.399 --> 00:09:29.720 Wow. 198 00:09:30.120 --> 00:09:34.159 Yeah. By forcing the path to search the official lockdown 199 00:09:34.240 --> 00:09:37.879 system directories first, the architecture guarantees you are running the 200 00:09:38.000 --> 00:09:41.399 genuine tools, regardless of what traps someone left in your 201 00:09:41.399 --> 00:09:42.279 current working folder. 202 00:09:42.440 --> 00:09:46.320 That is beautifully devious and a phenomenal reason to never 203 00:09:46.480 --> 00:09:48.840 ever put the dot directory at the front of your path. 204 00:09:49.000 --> 00:09:50.159 Exactly never do it. 205 00:09:50.519 --> 00:09:53.519 So Okay, we know how the shell safely locates and 206 00:09:53.559 --> 00:09:57.039 triggers commands, but triggering a command is only half the battle. 207 00:09:57.639 --> 00:10:00.000 We need to deal with the data those commands. 208 00:09:59.679 --> 00:10:01.159 Produce, right, the actual output. 209 00:10:01.320 --> 00:10:03.799 Yeah, and this introduces us to what is arguably the 210 00:10:03.799 --> 00:10:06.360 most famous core philosophy of Unix. 211 00:10:06.559 --> 00:10:09.200 Everything is a file, yes, the Golden rule. 212 00:10:09.399 --> 00:10:12.639 The cookbook emphasizes that. To the operating system, a hard 213 00:10:12.720 --> 00:10:15.440 drive is a file, a terminal window is a file, 214 00:10:15.480 --> 00:10:18.000 and your keyboard is a file. The system treats all 215 00:10:18.039 --> 00:10:20.440 of them as just an ordered sequence of bytes. But 216 00:10:20.480 --> 00:10:22.840 I have to push back on that though. Calling everything 217 00:10:22.879 --> 00:10:27.440 a file sounds incredibly reductive. Like a keyboard has physical 218 00:10:27.519 --> 00:10:32.279 mechanical switches, a hard drive has spinning magnetic platters. Treating 219 00:10:32.320 --> 00:10:34.759 them is the exact same thing. Seems like you're stripping 220 00:10:34.759 --> 00:10:37.200 away all the functionality that makes them useful. 221 00:10:37.559 --> 00:10:41.000 Well, what's fascinating here is that the stripping away is 222 00:10:41.039 --> 00:10:45.440 the entire point. Really, Yeah, that radical simplification is exactly 223 00:10:45.480 --> 00:10:48.559 what gives the command line its power. Yes, the physical 224 00:10:48.559 --> 00:10:52.200 hardware is vastly different, but remember are split between the 225 00:10:52.279 --> 00:10:54.399 kernel and the shell earlier, right. 226 00:10:54.279 --> 00:10:55.600 The engine and the steering wheel. 227 00:10:55.679 --> 00:10:59.039 Exactly. Yeah, the kernel handles the magnetic platters and the 228 00:10:59.080 --> 00:11:03.320 mechanical switches. It translates those physical actions into a simple 229 00:11:03.360 --> 00:11:06.120 stream of digital bytes. So by the time that data 230 00:11:06.159 --> 00:11:09.279 reaches the shell, all that hardware complexity is just gone. 231 00:11:09.559 --> 00:11:11.960 Oh I get it, because the shell treats everything as 232 00:11:12.000 --> 00:11:15.159 a simple file stream. A software developer doesn't need to 233 00:11:15.159 --> 00:11:17.960 know how to program a driver for a specific brand 234 00:11:18.000 --> 00:11:20.759 of keyboard or a specific model of monitor. 235 00:11:21.000 --> 00:11:23.240 They just write a program that reads bytes from an 236 00:11:23.279 --> 00:11:25.879 input file and writes bites to an output file. 237 00:11:26.200 --> 00:11:32.000 Precisely, the shell seamlessly connects those files to the physical hardware. 238 00:11:31.639 --> 00:11:34.279 Which means before we start passing these bytes streams around, 239 00:11:34.279 --> 00:11:35.840 we really have to talk about quoting. 240 00:11:35.960 --> 00:11:36.159 Oh. 241 00:11:36.279 --> 00:11:39.519 Quoting is crucial because if you just type raw text 242 00:11:39.559 --> 00:11:41.879 into the shell, it will try to interpret it. The 243 00:11:41.919 --> 00:11:45.440 cookbook uses this fantastic example trying to get the shell 244 00:11:45.480 --> 00:11:49.360 to print the sentence a coffee is five dollars. 245 00:11:49.360 --> 00:11:51.679 Right with the dollar sign and the exclamation point. 246 00:11:51.799 --> 00:11:55.600 Yeah, if you wrap that sentence in double quotes, the 247 00:11:55.600 --> 00:11:58.559 shell tries to be entirely too helpful. It sees the 248 00:11:58.559 --> 00:12:01.639 dollar sign and thinks, aha, that is a variable. I 249 00:12:01.679 --> 00:12:04.360 need to swap five dollars out for whatever value is 250 00:12:04.399 --> 00:12:05.120 stored in memory. 251 00:12:05.159 --> 00:12:07.000 But there is no variable named five. 252 00:12:06.919 --> 00:12:10.200 Exactly, so it replaces five dollars with absolutely nothing. Then 253 00:12:10.240 --> 00:12:13.679 it scans further sees the exclamation point. Assumes you want 254 00:12:13.679 --> 00:12:16.480 to run a history substitution to pull up a previous command, 255 00:12:16.639 --> 00:12:18.759 and the whole thing just throws an error. 256 00:12:18.799 --> 00:12:22.440 It's chaos. The shell basically acts like an overeager assistant 257 00:12:22.440 --> 00:12:24.960 trying to interpret your words before passing them to the 258 00:12:25.000 --> 00:12:25.919 actual program. 259 00:12:26.000 --> 00:12:27.600 An over eager assistant. 260 00:12:27.639 --> 00:12:29.559 That's perfect, yees. So if you want the assistant to 261 00:12:29.600 --> 00:12:32.159 back off and take you literally, you use single quotes. 262 00:12:32.399 --> 00:12:33.159 Single quotes. 263 00:12:33.480 --> 00:12:37.919 Single quotes form an impenetrable barrier. They completely disable the 264 00:12:37.960 --> 00:12:41.480 shell's ability to interpret special characters, so the dollar sign 265 00:12:41.519 --> 00:12:44.440 and the exclamation point are treated merely as shapes on 266 00:12:44.480 --> 00:12:48.120 the screen. What you type inside single quotes is exactly 267 00:12:48.159 --> 00:12:49.120 what gets passed along. 268 00:12:49.279 --> 00:12:52.519 Okay, so here's where it gets really interesting. Once your 269 00:12:52.559 --> 00:12:55.559 text is protected and your commands are running, you can 270 00:12:55.559 --> 00:12:56.759 start building the plumbing. 271 00:12:56.879 --> 00:12:57.399 The plumbing. 272 00:12:57.480 --> 00:12:59.960 Yes, we use the greater than sign to read diirect 273 00:13:00.200 --> 00:13:03.360 output into a text file, a double grater than to 274 00:13:03.720 --> 00:13:06.200 append to the bottom of an existing file, and the 275 00:13:06.279 --> 00:13:09.159 less than sign to pull input into a program from a. 276 00:13:09.080 --> 00:13:10.519 File standard redirection. 277 00:13:10.759 --> 00:13:13.960 Right, But the show is actually performing some incredibly clever 278 00:13:14.080 --> 00:13:15.360 deception behind the scenes. 279 00:13:15.360 --> 00:13:15.559 Here. 280 00:13:16.039 --> 00:13:19.759 The cookbook highlights this bizarre anomaly with the l's command 281 00:13:19.840 --> 00:13:21.480 that completely caught me off guard. 282 00:13:21.639 --> 00:13:22.840 Oh, the column formatting. 283 00:13:22.960 --> 00:13:25.679 Yes, if you type l's rows directly into your terminal, 284 00:13:25.879 --> 00:13:28.559 it formats your files into neat wide columns so they 285 00:13:28.559 --> 00:13:31.399 fit perfectly on your monitor. But if you type l's 286 00:13:31.440 --> 00:13:34.399 and use the greater then sign to redirect that exact 287 00:13:34.480 --> 00:13:37.679 same output into a text file, the formatting changes. 288 00:13:37.799 --> 00:13:38.159 It does. 289 00:13:38.320 --> 00:13:40.759 You open the file and the columns are just gone. 290 00:13:41.200 --> 00:13:44.840 It's a single, massive, vertical list, one file per line. 291 00:13:45.799 --> 00:13:48.399 How does the program know we redirected it? I mean, 292 00:13:48.399 --> 00:13:50.840 how does it No, it's not looking at a screen anymore. 293 00:13:51.000 --> 00:13:53.279 So to understand the mechanism behind that, we have to 294 00:13:53.320 --> 00:13:54.600 talk about file descriptors. 295 00:13:54.639 --> 00:13:55.679 Okay, file descriptor. 296 00:13:55.759 --> 00:13:58.879 A file descriptor is basically just an ID tag, a 297 00:13:58.960 --> 00:14:02.759 simple number the operating system assigns to an open data stream, 298 00:14:03.120 --> 00:14:06.639 and by default, every single program gets three of these 299 00:14:06.679 --> 00:14:09.759 ID tags the moment it starts. Just three, just three. 300 00:14:10.000 --> 00:14:13.120 File descriptor zero is standard input, which is usually your keyboard, 301 00:14:13.279 --> 00:14:16.360 file scripture one is standard output, usually your screen, and 302 00:14:16.480 --> 00:14:19.360 file descriptor two is standard air, also usually your screen. 303 00:14:19.679 --> 00:14:22.000 Okay, so zero, one and two, right, So. 304 00:14:21.960 --> 00:14:24.600 When you use the greater than sign, you are telling 305 00:14:24.639 --> 00:14:27.759 the shell take file descriptor one, disconnect it from the 306 00:14:27.759 --> 00:14:29.919 monitor and plug it into this text file instead. 307 00:14:30.399 --> 00:14:32.960 Wait, the redirection is supposed to be invisible to the program, 308 00:14:33.039 --> 00:14:35.919 right the program just keeps pushing bytes to descriptor one. 309 00:14:36.240 --> 00:14:39.600 Generally, yes, that's the beauty of it. But programs are 310 00:14:39.639 --> 00:14:42.799 actually allowed to ask the operating system, hey, what exactly 311 00:14:42.840 --> 00:14:45.360 is my file descriptor one plugged into? Right now? O? 312 00:14:45.799 --> 00:14:46.039 Way? 313 00:14:46.440 --> 00:14:50.039 Yeah, The developers of L's added a tiny check. The 314 00:14:50.080 --> 00:14:53.279 program queries the OS. If the OS replies, you are 315 00:14:53.279 --> 00:14:57.039 plugged into a terminal display, then LS calculates the width 316 00:14:57.080 --> 00:15:00.679 of the screen and neatly arranges the columns for human eyes. 317 00:15:00.840 --> 00:15:02.879 But if it's plugged into a file exactly. 318 00:15:03.000 --> 00:15:05.159 If the ALWAYS replies you are plugged into a flat 319 00:15:05.200 --> 00:15:08.759 text file, L's knows the human isn't reading it right now. 320 00:15:08.840 --> 00:15:11.159 It knows another program is likely going to parse that 321 00:15:11.200 --> 00:15:15.360 file later, and parsing a single predictable vertical list is 322 00:15:15.559 --> 00:15:19.159 drastically easier for a computer than trying to untangle columns. 323 00:15:19.440 --> 00:15:22.360 Is that underlying logic is brilliant. It literally adapts to 324 00:15:22.399 --> 00:15:23.679 the medium it's being sent to. 325 00:15:23.840 --> 00:15:24.720 It's very smart. 326 00:15:24.960 --> 00:15:27.039 And speaking of plumbing, that brings us back to those 327 00:15:27.080 --> 00:15:30.600 file descriptors you mentioned specifically standard output which is one, 328 00:15:30.720 --> 00:15:34.720 and standard error which is two. This perfectly explains a headache. 329 00:15:34.720 --> 00:15:36.000 I think literally everyone. 330 00:15:35.759 --> 00:15:37.080 Encounters all the error bleed. 331 00:15:37.279 --> 00:15:40.879 Yes, you run a command that generates a massive wall 332 00:15:40.960 --> 00:15:44.120 of text, including a bunch of error messages. You decide 333 00:15:44.120 --> 00:15:46.639 to save it, so you use the greater than sign 334 00:15:46.879 --> 00:15:49.960 to redirect the output to a file. Right, you hit enter, 335 00:15:50.440 --> 00:15:52.919 and all the normal text goes quietly into the file, 336 00:15:52.960 --> 00:15:55.759 but the error message is still bleed all over your screen. 337 00:15:55.799 --> 00:15:57.559 It's so frustrating, it. 338 00:15:57.480 --> 00:16:00.840 Is, But that happens because the sta entered greater than 339 00:16:00.919 --> 00:16:05.840 sign only redirects file descriptor one. Oh yes, Stream one 340 00:16:05.919 --> 00:16:08.919 goes to the file, but stream two, the errors remains 341 00:16:08.960 --> 00:16:12.039 plugged directly into your monitor. To catch the errors, you 342 00:16:12.080 --> 00:16:15.360 have to explicitly redirect descriptor two by typing too. 343 00:16:15.639 --> 00:16:16.440 Ah okay. 344 00:16:16.519 --> 00:16:20.000 But the cookbook also highlights a vital, highly technical nuance 345 00:16:20.039 --> 00:16:24.759 regarding how these two streams deliver their payloads. And that's buffering, right, buffering. 346 00:16:24.840 --> 00:16:27.879 So standard output is buffered, meaning it acts like a bucket. 347 00:16:28.000 --> 00:16:30.360 The program doesn't send data to the screen one letter 348 00:16:30.399 --> 00:16:32.240 at a time. It fills up a memory bucket, and 349 00:16:32.279 --> 00:16:35.200 when the bucket is full, it dumps the whole chunk 350 00:16:35.240 --> 00:16:36.279 to the scream at once. 351 00:16:36.159 --> 00:16:37.879 Which is highly computationally efficient. 352 00:16:37.919 --> 00:16:43.039 Yeah, exactly, But Standard error Descriptor two is unbuffered. Every 353 00:16:43.080 --> 00:16:46.480 single character is transmitted the exact millisecond it is generated. 354 00:16:47.000 --> 00:16:48.480 Why treat the errors differently? 355 00:16:48.679 --> 00:16:50.240 It comes down to crash survival. 356 00:16:50.480 --> 00:16:51.480 Crash survival. 357 00:16:51.639 --> 00:16:56.480 Yeah, imagine a program encounters a catastrophic fatal error. It 358 00:16:56.559 --> 00:17:00.639 generates an error message detailing exactly what went wrong. But 359 00:17:00.720 --> 00:17:03.000 if that message goes into a buffer bucket and the 360 00:17:03.039 --> 00:17:05.920 program crashes before the bucket gets dumped, the bucket just 361 00:17:06.000 --> 00:17:10.599 vanishes exactly that error message is destroyed in memory. You're 362 00:17:10.720 --> 00:17:13.559 left staring at a dead program with zero clues as 363 00:17:13.559 --> 00:17:14.279 to why it died. 364 00:17:14.480 --> 00:17:15.440 That makes total sense. 365 00:17:15.680 --> 00:17:19.519 By making standard error entirely unbeffored, the system guarantees that 366 00:17:19.559 --> 00:17:22.880 the diagnostic text prints to your screen instantly. Even if 367 00:17:22.920 --> 00:17:25.880 the program violently terminates a fraction of a millisecond later, 368 00:17:26.359 --> 00:17:28.440 the error message has already made it out safely. 369 00:17:28.559 --> 00:17:32.519 It prioritizes survival of the forensic data over efficiency. I 370 00:17:32.559 --> 00:17:34.480 love that. And you know, if a program is just 371 00:17:34.519 --> 00:17:36.599 obnoxiously chatty and you don't want to see its output 372 00:17:36.640 --> 00:17:39.400 or save it. You can just redirect the stream to devnel. 373 00:17:39.599 --> 00:17:41.559 Ah. Yes, the bitbucket. 374 00:17:41.839 --> 00:17:44.400 It's a literal black hole where data goes to be 375 00:17:44.440 --> 00:17:48.920 permanently deleted. So okay, we've mastered basic plumbing. We can 376 00:17:49.039 --> 00:17:52.480 route output and errors to files or black holes. But 377 00:17:52.559 --> 00:17:56.319 a shell's real power isn't just shuffling data into files, right, 378 00:17:56.720 --> 00:17:58.000 it's chaining tools together. 379 00:17:58.240 --> 00:18:03.000 Yes, this is where we introduce pipes, represented by the vertical. 380 00:18:02.640 --> 00:18:03.960 Bar character the pipe. 381 00:18:04.000 --> 00:18:06.480 A pipe takes the standard output of one command and 382 00:18:06.480 --> 00:18:09.279 plugs it directly into the standard input of the next command. 383 00:18:09.559 --> 00:18:12.319 But the mechanism of how they execute is what makes 384 00:18:12.359 --> 00:18:15.079 them so powerful. A pipe does not act like a 385 00:18:15.119 --> 00:18:17.559 relay race. What do you mean by that, Well, it 386 00:18:17.599 --> 00:18:19.839 does not wait for the first program to finish its job, 387 00:18:20.079 --> 00:18:22.559 save the data, and hand a completed file over to 388 00:18:22.599 --> 00:18:25.359 the second program. A pipe is an assembly line. 389 00:18:25.440 --> 00:18:26.640 Oh, they run at the same time. 390 00:18:26.839 --> 00:18:30.759 Both programs launch and run simultaneously. The first program writes 391 00:18:30.799 --> 00:18:33.799 data into a shared memory buffer, and the second program 392 00:18:33.839 --> 00:18:36.880 continuously reads from that exact same buffer in real time. 393 00:18:37.079 --> 00:18:40.279 That parallel processing is why the command line can handle 394 00:18:40.359 --> 00:18:43.880 massive data sets. So quickly. You aren't constantly reading and 395 00:18:43.920 --> 00:18:45.759 writing temporary files. 396 00:18:45.359 --> 00:18:47.359 To the hard drive exactly. It's all in memory. 397 00:18:47.559 --> 00:18:51.200 But because data flows too easily, the cookbook mentions, users 398 00:18:51.240 --> 00:18:53.720 often fall into a trap called the useless use of 399 00:18:53.839 --> 00:18:54.440 cat award. 400 00:18:54.640 --> 00:18:55.680 Oh this is a classic. 401 00:18:55.839 --> 00:18:58.599