WEBVTT 1 00:00:00.080 --> 00:00:03.399 So if you walked into almost any tech company historically, 2 00:00:03.879 --> 00:00:07.639 there was basically a war going on, a literal epic 3 00:00:07.799 --> 00:00:08.839 turf war. Oh. 4 00:00:08.880 --> 00:00:11.480 Absolutely, a total battleground. 5 00:00:11.000 --> 00:00:13.679 Right, because on one side you had the software developers 6 00:00:14.160 --> 00:00:17.280 and their entire job, I mean, their whole performance metric 7 00:00:17.679 --> 00:00:21.239 was to just move fast, break things, and push continuous updates. 8 00:00:21.320 --> 00:00:22.920 Yeah, just ship it exactly. 9 00:00:23.280 --> 00:00:25.559 And then on the other side you had the operations team, 10 00:00:25.800 --> 00:00:29.000 the knops folks. Their job was stability. They were essentially 11 00:00:29.039 --> 00:00:29.920 the gatekeepers, and. 12 00:00:29.960 --> 00:00:31.440 They were the ones holding the pager. 13 00:00:31.359 --> 00:00:35.000 Right, So because they held that pager, their default answer 14 00:00:35.119 --> 00:00:38.079 to you almost any new developer idea was just an 15 00:00:38.119 --> 00:00:39.000 emphatic no. 16 00:00:39.359 --> 00:00:42.600 Well, I mean it really was a necessary survival tactic 17 00:00:42.640 --> 00:00:45.119 for them. Think about the stakes for an OPS engineer 18 00:00:45.200 --> 00:00:45.479 back then. 19 00:00:45.560 --> 00:00:46.560 Yeah, it was high stakes. 20 00:00:46.679 --> 00:00:49.960 Huge. If a developer wanted to try out, say a 21 00:00:49.960 --> 00:00:55.320 brand new, highly experimental library, and that code caused a 22 00:00:55.359 --> 00:00:58.679 memory leak, oh man, the dreaded memory leaks, right, and 23 00:00:58.759 --> 00:01:01.280 it brings down the entire server at two am. It 24 00:01:01.320 --> 00:01:03.320 wasn't the developer getting out of bed, it was the 25 00:01:03.359 --> 00:01:08.239 ops team waking up to all those alarms. So they 26 00:01:08.280 --> 00:01:11.359 had to be defensive. The infrastructure just simply couldn't handle 27 00:01:11.400 --> 00:01:13.040 that level of chaos. 28 00:01:12.680 --> 00:01:15.719 And that tension is exactly what we're unpacking today. We're 29 00:01:15.719 --> 00:01:18.799 doing a deep dive decoding the reality of running Docker 30 00:01:19.200 --> 00:01:22.480 when real customers and you know, real money are on. 31 00:01:22.439 --> 00:01:24.959 The line, which changes everything. 32 00:01:24.640 --> 00:01:27.480 It really does. We're pulling from this fascinating book called 33 00:01:27.480 --> 00:01:31.200 Docker in Production Lessons from the Trenches. And for you listening, 34 00:01:31.200 --> 00:01:34.120 this isn't your standard like Hello World tutorial, No, not 35 00:01:34.159 --> 00:01:36.920 at all. We're talking about the duct tape, the midnight alerts, 36 00:01:37.040 --> 00:01:40.079 the really gritty reality of what happens when you actually 37 00:01:40.120 --> 00:01:43.439 put this technology into production. Because Docker didn't just introduce 38 00:01:43.519 --> 00:01:45.040 a neat technical. 39 00:01:44.599 --> 00:01:46.840 Trick, No, it fundamentally shifted the culture. 40 00:01:46.959 --> 00:01:50.239 It essentially forced a peace treaty in that epic dev 41 00:01:50.319 --> 00:01:54.319 versus OPS war. Yeah, it completely changed the ops response 42 00:01:54.400 --> 00:01:57.159 from no to yes as long as it runs in Docker. 43 00:01:57.280 --> 00:02:00.159 Yeah. And that shift happened because Docker completely read to 44 00:02:00.159 --> 00:02:02.159 find the blast radius of bad code. 45 00:02:02.439 --> 00:02:03.560 The blast radius. 46 00:02:03.640 --> 00:02:06.319 I like that, right, Let's look at the before picture. 47 00:02:06.799 --> 00:02:11.400 In a traditional virtual machine setup, you might have say, 48 00:02:11.439 --> 00:02:14.159 five different applications running on one operating. 49 00:02:13.759 --> 00:02:16.240 System, right, all sharing the same space. 50 00:02:16.000 --> 00:02:19.759 Exactly, So if one of those apps has a catastrophic failure, 51 00:02:20.159 --> 00:02:23.080 it could crash the whole virtual machine. It takes the 52 00:02:23.080 --> 00:02:25.919 other four instant applications down with it, which is a nightmare, 53 00:02:26.039 --> 00:02:30.120 total nightmare. But with Docker, that bad code is tracked. 54 00:02:30.560 --> 00:02:34.639 It is entirely isolated inside its specific container. So if 55 00:02:34.680 --> 00:02:36.479 it crashes, only that container dies. 56 00:02:36.680 --> 00:02:37.199 Wow. 57 00:02:37.479 --> 00:02:39.759 The host server and all the other containers sharing that 58 00:02:39.800 --> 00:02:42.319 server they just keep humming along completely unaware. 59 00:02:42.439 --> 00:02:45.719 Okay, let's unpack this because getting to the point where 60 00:02:45.719 --> 00:02:50.400 ops actually trust that isolation, that requires a massive mental 61 00:02:50.439 --> 00:02:52.520 shift in how we even view our servers in the 62 00:02:52.560 --> 00:02:53.080 first place. 63 00:02:53.159 --> 00:02:54.879 Oh massive, It's a whole new paradigm. 64 00:02:55.000 --> 00:02:57.560 Yeah. And the book uses this brilliant analogy from a 65 00:02:57.599 --> 00:03:00.319 company called peer Space, detailing what they call the cattle 66 00:03:00.400 --> 00:03:01.599 versus pets philosophy. 67 00:03:01.879 --> 00:03:06.919 It is arguably the most critical mental model for modern infrastructure. Honestly, 68 00:03:07.560 --> 00:03:09.319 I mean, if you've ever managed servers, you know. 69 00:03:09.280 --> 00:03:11.039 What a pet is, right, you get attached. 70 00:03:11.240 --> 00:03:14.960 You really do think about your stateful services, like your 71 00:03:15.039 --> 00:03:18.719 massive relational databases or your search engines like elastic search. 72 00:03:19.240 --> 00:03:22.000 Those are pets. You give them specific names, like you 73 00:03:22.039 --> 00:03:24.159 know database primary era one. 74 00:03:24.360 --> 00:03:26.360 Right, they have a caller and a tag. 75 00:03:26.360 --> 00:03:31.719 Exactly because they hold precious, irreplaceable data. They require constant, 76 00:03:32.000 --> 00:03:35.039 meticulous upkeep. And if a pet gets sick, you. 77 00:03:35.039 --> 00:03:37.439 Call ve, you log in, you diagnose it. 78 00:03:37.560 --> 00:03:40.280 Yes, you carefully nurse that specific machine back to health. 79 00:03:40.599 --> 00:03:44.439 You cannot just delete your primary customer database and pretend 80 00:03:44.439 --> 00:03:45.000 it didn't happen. 81 00:03:45.080 --> 00:03:47.400 Yeah, that would be bad. But the other side of 82 00:03:47.400 --> 00:03:50.479 the infrastructure is entirely different. You have your stateless services, 83 00:03:50.919 --> 00:03:54.840 your web servers, the APR routers, the code that just 84 00:03:55.039 --> 00:03:58.479 processes requests without actually storing anything permanently on its own 85 00:03:58.919 --> 00:03:59.759 local hard. 86 00:03:59.680 --> 00:04:03.000 Drive, right, and those are your cattle. They are completely nameless. 87 00:04:03.000 --> 00:04:05.400 They don't store anything unique at all. So if they break, 88 00:04:05.719 --> 00:04:07.680 if one of them starts acting up or get sick, 89 00:04:08.199 --> 00:04:11.159 you absolutely do not nurse it back to health. You 90 00:04:11.159 --> 00:04:14.400 don't even investigate it. Wow, just brutal it is. You 91 00:04:14.439 --> 00:04:17.079 pull it out of rotation, destroy it, and immediately replace 92 00:04:17.120 --> 00:04:20.600 it with an identical, fresh clone. And peer space just 93 00:04:20.639 --> 00:04:24.360 embrace this entirely. They operate over twenty different micro services, 94 00:04:24.839 --> 00:04:27.959 and almost all of them are treated as disposable cattle. 95 00:04:27.800 --> 00:04:31.079 And the way they manage these disposable containers is shockingly 96 00:04:31.120 --> 00:04:34.439 bare bones. I mean they just use static environment variables, 97 00:04:34.759 --> 00:04:38.199 basically simple text strings telling the container like, hey, the 98 00:04:38.279 --> 00:04:41.160 database is over here. Yeah, it's incredibly simple, and they 99 00:04:41.240 --> 00:04:45.360 let standard Linux tools like systems manage them. And for 100 00:04:45.399 --> 00:04:48.839 anyone less familiar, systemed is just the built in Linux 101 00:04:48.839 --> 00:04:51.959 service manager that runs in the background to ensure programs 102 00:04:52.000 --> 00:04:53.240 start up and stay running. 103 00:04:53.439 --> 00:04:55.399 Right, it's just basic Linux plumbing. 104 00:04:55.519 --> 00:04:58.959 But hold on, if they are literally typing in environment 105 00:04:59.079 --> 00:05:03.360 variables and using basic Linux background commands to manage these cattle, 106 00:05:03.920 --> 00:05:08.279 doesn't that defeat the whole purpose? Well, that sounds incredibly tedious, 107 00:05:08.319 --> 00:05:11.079 like a massive recipe for human error. If someone has 108 00:05:11.120 --> 00:05:13.160 to restart these things manually all the time. 109 00:05:13.240 --> 00:05:16.160 Oh absolutely, If a human being were doing it by hand, 110 00:05:16.480 --> 00:05:17.839 it would be a complete disaster. 111 00:05:18.040 --> 00:05:19.519 Okay, good, I thought I was missing something. 112 00:05:19.560 --> 00:05:23.000 No. No. The key takeaway from pure Space's approach is 113 00:05:23.040 --> 00:05:27.879 that this simple static configuration is designed to be the 114 00:05:27.920 --> 00:05:32.560 foundation for automation. Ah automation. Right, You don't have an 115 00:05:32.560 --> 00:05:36.240 engineer typing those variables. You have a continuous integration and 116 00:05:36.279 --> 00:05:40.439 continuous delivery system a CICD pipeline doing all the typing 117 00:05:40.480 --> 00:05:40.800 for you. 118 00:05:40.879 --> 00:05:42.439 Okay, that makes sense, But to get to. 119 00:05:42.399 --> 00:05:45.480 A point where a machine can deploy these identical cattle 120 00:05:45.720 --> 00:05:49.360 incredibly fast and automatically, you first need to understand their DNA. 121 00:05:49.800 --> 00:05:51.519 You have to look under the hood at how Docker 122 00:05:51.519 --> 00:05:53.920 images are actually constructed. 123 00:05:53.399 --> 00:05:57.519 Which brings us to copy on write or COW. This 124 00:05:57.639 --> 00:06:00.800 is where the efficiency really shines, right. It's the secret sauce, 125 00:06:01.040 --> 00:06:04.199 because if your team was using traditional virtual machines and 126 00:06:04.240 --> 00:06:08.519 you needed, say ten web servers, you'd have ten massive 127 00:06:08.680 --> 00:06:12.040 duplicate copies of an entire operating system just eating up. 128 00:06:12.000 --> 00:06:14.199 Your hard drives, which is incredibly. 129 00:06:13.800 --> 00:06:17.600 Wasteful exactly, but Docker uses a layering system. Think of 130 00:06:17.639 --> 00:06:21.639 copy on write, like those old transparent acetate sheets teachers 131 00:06:21.720 --> 00:06:23.720 used to use on overhead projectors in school. 132 00:06:23.959 --> 00:06:26.759 I love this analogy. The overhead projector is a fantastic 133 00:06:26.800 --> 00:06:28.519 way to visualize the mechanics here. 134 00:06:28.680 --> 00:06:31.839 Right, So the base operating system, let's say it's a tiny, 135 00:06:32.160 --> 00:06:37.000 stripped down version of Linux. Is that bottom clear plastic sheet. Now, 136 00:06:37.000 --> 00:06:39.000 if you want to add a mustache to a face 137 00:06:39.079 --> 00:06:42.279 drawn on that bottom sheet, you don't take an eraser 138 00:06:42.399 --> 00:06:43.639 to the base layer. 139 00:06:43.519 --> 00:06:44.600 Right, You don't touch the base. 140 00:06:44.800 --> 00:06:47.759 You just draw your mustache on a brand new totally 141 00:06:47.839 --> 00:06:51.079 clear sheet of plastic and you drop it directly on 142 00:06:51.120 --> 00:06:54.120 top of the first one. When the projector light shines 143 00:06:54.199 --> 00:06:57.319 up through both sheets, you see the face with the mustache. 144 00:06:57.360 --> 00:06:59.560 That is exactly how a Docker layer works. 145 00:06:59.360 --> 00:07:02.199 And the storage savings are just astronomical. If you have 146 00:07:02.240 --> 00:07:06.360 twenty different containers all running that same base Linux operating system, 147 00:07:06.639 --> 00:07:09.920 Docker only stores that bottom transparent sheet once on the 148 00:07:09.920 --> 00:07:14.040 hard drive. Wow, just once, exactly. Every single container shares 149 00:07:14.079 --> 00:07:17.199 that exact same foundation, and they only add their own 150 00:07:17.199 --> 00:07:20.959 incredibly lightweight mustache layers on top. It saves time, it 151 00:07:21.040 --> 00:07:25.160 saves bandwidth, and it obviously saves disk space. Right. What's 152 00:07:25.199 --> 00:07:27.920 fascinating here is, however, there is a very dangerous sized 153 00:07:27.959 --> 00:07:30.879 trap hidden within these transparent layers. 154 00:07:30.600 --> 00:07:33.480 The permanent marker trap, because you can't just erase things 155 00:07:33.480 --> 00:07:34.279 once they're drawn. 156 00:07:34.480 --> 00:07:40.040 Exactly. That's the danger of immutability. Images never shrink, never never. 157 00:07:40.639 --> 00:07:43.959 Imagine on layer three your container needs to download a 158 00:07:44.079 --> 00:07:47.600 massive five hundred megabyte compressed archive just to compile some 159 00:07:47.720 --> 00:07:48.600 software from source. 160 00:07:48.720 --> 00:07:49.600 Okay, pretty common. 161 00:07:49.720 --> 00:07:52.120 Then on layer four, the compiling is done, so you 162 00:07:52.199 --> 00:07:55.680 tell Docker to delete that temporary five hundred megabyte archive 163 00:07:56.360 --> 00:07:58.879 from the top down. Looking through the sheets, the file 164 00:07:58.920 --> 00:07:59.839 appears to be gone. 165 00:07:59.720 --> 00:08:01.439 Because as the top sheet covers it up. 166 00:08:01.399 --> 00:08:04.600 Right, But because layers cannot be altered once they are created, 167 00:08:04.879 --> 00:08:07.920 that massive file is still permanently glued to layer three. 168 00:08:08.040 --> 00:08:10.639 It is completely invisible to the application, but it will 169 00:08:10.680 --> 00:08:14.480 be downloaded, transferred, and stored by every single server that 170 00:08:14.519 --> 00:08:15.800 ever uses your image. 171 00:08:15.920 --> 00:08:19.120 Oh man, that is brutal. So if the sheets are immutable, 172 00:08:19.160 --> 00:08:21.279 you can't just delete it later. You would have to 173 00:08:21.319 --> 00:08:25.160 somehow download the archive, extract it, compile a software, and 174 00:08:25.199 --> 00:08:28.839 clean up all the garbage, all before that transparent sheet dries. Right, 175 00:08:28.879 --> 00:08:31.439 you'd have to chain it all together into one single step. 176 00:08:31.759 --> 00:08:35.120 You just describe the exact solution from the trenches. You 177 00:08:35.159 --> 00:08:38.960 fix this by stringing your commands together using operators in 178 00:08:39.000 --> 00:08:42.840 your docro file. Oh instead of using three separate ru 179 00:08:42.879 --> 00:08:45.399 in commands you know, one to download, one to install, 180 00:08:45.440 --> 00:08:48.000 one to clean up, you combine them into a single 181 00:08:48.120 --> 00:08:51.799 long r ruin command. The file is downloaded, the software 182 00:08:51.879 --> 00:08:54.720 is built, and the garbage is deleted, all during the 183 00:08:54.759 --> 00:08:56.679 creation of a single transparent sheet. 184 00:08:56.759 --> 00:08:59.600 That was so smart, and the sources emphasize that this 185 00:08:59.639 --> 00:09:02.279 all star with choosing a tiny bottom sheet to begin with, 186 00:09:02.799 --> 00:09:06.360 you wouldn't use a massive, bloated desktop operating system as. 187 00:09:06.279 --> 00:09:08.320 Your base, No, that would defeat the purpose. 188 00:09:08.559 --> 00:09:11.720 You'd use something purpose built for containers like Alpine Linux 189 00:09:11.799 --> 00:09:14.879 or BusyBox, which are just phenomenally small. I mean, BusyBox 190 00:09:14.960 --> 00:09:16.919 is literally two point five megabytes. 191 00:09:16.960 --> 00:09:17.600 It's crazy. 192 00:09:17.799 --> 00:09:20.159 It's wild to think you can run a production web 193 00:09:20.200 --> 00:09:22.960 server on a foundation the size of a low quality 194 00:09:23.039 --> 00:09:25.240 MP three from nineteen ninety nine. 195 00:09:25.399 --> 00:09:28.000 The mineralism is really what makes the whole system viable. 196 00:09:28.360 --> 00:09:32.000 And once your team has mastered building these tiny, highly 197 00:09:32.000 --> 00:09:37.480 optimized images, you face the next operational hurdle orchestration. Exactly, 198 00:09:37.960 --> 00:09:39.679 how do you actually take the old version of your 199 00:09:39.679 --> 00:09:42.879 application down and put the new version up without dropping 200 00:09:42.919 --> 00:09:44.039 a single user's connection? 201 00:09:44.360 --> 00:09:49.440 Ah, the holy grail of zero downtime deployment. The book 202 00:09:49.519 --> 00:09:52.279 highlights an amazing real world set up from a company 203 00:09:52.279 --> 00:09:55.399 called relate Iq to achieve this, and honestly, given how 204 00:09:55.399 --> 00:09:58.519 complex this sounds, you'd think they're running some massive, enterprise 205 00:09:58.559 --> 00:10:01.480 grade cluster. Manager thinks, oh yeah, but their orchestration is 206 00:10:01.559 --> 00:10:04.960 shockingly simple. It's literally a bashscript. 207 00:10:04.320 --> 00:10:07.399 A simple basscript running on a kron job every five. 208 00:10:07.200 --> 00:10:08.720 Minutes wait, just a kron job. 209 00:10:08.919 --> 00:10:11.399 Yep, and a cron job is really just a basic 210 00:10:11.679 --> 00:10:15.200 time based task scheduler built into Linux. Every five minutes, 211 00:10:15.200 --> 00:10:17.440 it wakes up and asks is there a new version 212 00:10:17.440 --> 00:10:18.679 of our web server available? 213 00:10:18.919 --> 00:10:22.039 And if there is, it pulls down that tiny new image. 214 00:10:22.919 --> 00:10:25.039 But how does it swap the traffic without users seeing 215 00:10:25.039 --> 00:10:26.759 an error page? I mean, they don't want. 216 00:10:26.519 --> 00:10:29.159 A four or four error right, So they use a 217 00:10:29.159 --> 00:10:33.000 proxy called hippasche backed by a reddist database. Think of 218 00:10:33.000 --> 00:10:35.840 that hipache proxy as a train switch operator. 219 00:10:36.039 --> 00:10:37.279 Okay, I'm picturing it. 220 00:10:37.279 --> 00:10:39.440 It sits at the front of the station directing all 221 00:10:39.440 --> 00:10:43.120 the incoming user traffic. When that five minute kron job 222 00:10:43.200 --> 00:10:45.919 pulls down the new container image, it spins it up 223 00:10:45.960 --> 00:10:47.279 on the server alongside the. 224 00:10:47.240 --> 00:10:49.240 Old one, so they're both running at the same time. 225 00:10:49.600 --> 00:10:54.200 Yes, but crucially, the proxy doesn't send any user traffic 226 00:10:54.200 --> 00:10:55.519 to the new container immediately. 227 00:10:55.720 --> 00:10:58.639 Ah. It waits for a health check exactly. The script 228 00:10:58.679 --> 00:11:01.039 actually tests the new container to make sure it's fully 229 00:11:01.039 --> 00:11:03.399 booted up and breathing before putting. 230 00:11:03.080 --> 00:11:05.320 It on the front lines. Once the hell check passes, 231 00:11:05.679 --> 00:11:10.000 the bash script updates the reddest database. It pash monitors 232 00:11:10.000 --> 00:11:14.120 that database instantly sees the new destination and immediately flips 233 00:11:14.159 --> 00:11:17.080 the train tracks just like that. From that millisecond forward, 234 00:11:17.440 --> 00:11:21.320 all new user traffic flows seamlessly into the new container, and. 235 00:11:21.600 --> 00:11:23.759 They leave the old container running right next to it 236 00:11:23.799 --> 00:11:26.320 for a little while, which is brilliant. Just in case 237 00:11:26.360 --> 00:11:28.879 the new code has a weird bug that wasn't caught 238 00:11:28.879 --> 00:11:31.759 in testing, they can flip the switch back instantly. 239 00:11:31.840 --> 00:11:34.039 The rollback takes seconds instead of minutes. 240 00:11:34.200 --> 00:11:35.080 That's huge. 241 00:11:35.399 --> 00:11:38.840 It is in a traditional virtual machine environment, spinning up 242 00:11:38.879 --> 00:11:42.360 a second copy of your server, ensuring it's stable and 243 00:11:42.480 --> 00:11:46.600 manually switching dns or load balancers could take ten minutes 244 00:11:46.720 --> 00:11:47.320 or more. 245 00:11:47.279 --> 00:11:50.240 Which feels like an eternity during an outage. 246 00:11:50.440 --> 00:11:54.360 With Docker, because the containers are so lightweight, the infrastructure 247 00:11:54.360 --> 00:11:55.679 can pivot in milliseconds. 248 00:11:55.799 --> 00:11:59.720 Here's where it gets really interesting though. Related Iq didn't 249 00:11:59.759 --> 00:12:03.879 just use this architecture for zero downtime production deploys. They 250 00:12:03.960 --> 00:12:07.840 use this exact same concept to create parallel universes for 251 00:12:07.919 --> 00:12:10.120 their developers using awsban Stock. 252 00:12:10.360 --> 00:12:14.480 Yes, by completely decoupling their web tier from their database tier, 253 00:12:14.919 --> 00:12:17.519 they essentially created a playground. A playground. 254 00:12:17.559 --> 00:12:18.080 I like that. 255 00:12:18.240 --> 00:12:20.840 It's always one of the biggest bottlenecks in software development. 256 00:12:21.120 --> 00:12:23.159 The staging environment q oh. 257 00:12:23.159 --> 00:12:25.840 Right, waiting for your turn to test your code. So 258 00:12:26.039 --> 00:12:29.519 imagine you're a UI designer on their team. You just 259 00:12:29.679 --> 00:12:33.480 changed the CSS to make the checkout buttons blue instead 260 00:12:33.480 --> 00:12:36.919 of red. You commit your code to a specific branch, Right, 261 00:12:37.080 --> 00:12:40.840 the CICD system sees that automatically builds a brand new 262 00:12:40.960 --> 00:12:44.639 Docker image just for your specific branch and spins up 263 00:12:44.679 --> 00:12:46.120 a dedicated web container. 264 00:12:46.399 --> 00:12:50.559 And because that web tier is entirely stateless, it's cattle right. Yeah, 265 00:12:50.679 --> 00:12:53.080 it requires no special setup. It just boots up and 266 00:12:53.120 --> 00:12:54.960 points itself at a staging database. 267 00:12:55.320 --> 00:12:57.600 But the best part is with a quick change of 268 00:12:57.639 --> 00:13:01.639 a single environment variable, that designer can point their isolated 269 00:13:01.679 --> 00:13:04.759 test container at the actual production database. 270 00:13:04.360 --> 00:13:05.840 Which is incredibly powerful. 271 00:13:06.000 --> 00:13:08.519 Right, they can see exactly how their new blue buttons 272 00:13:08.559 --> 00:13:11.120 look and function with real customer data flowing through the app. 273 00:13:11.320 --> 00:13:14.159 But they aren't risking a single real user seeing their 274 00:13:14.200 --> 00:13:16.360 test environment if it completely breaks. 275 00:13:16.519 --> 00:13:19.879 It creates an incredibly tight feedback loop. I mean, a 276 00:13:19.919 --> 00:13:22.879 designer or a product manager can sketch an idea, write 277 00:13:22.919 --> 00:13:25.759 the code, and see it functioning in total isolation against 278 00:13:25.759 --> 00:13:27.559 real data in under fifteen minutes. 279 00:13:27.960 --> 00:13:31.039 That sounds like an absolute dream for a developer. But 280 00:13:31.720 --> 00:13:34.159 and I have to ask, if it's really this easy 281 00:13:34.200 --> 00:13:37.720 and perfectly isolated, why not just let every developer push 282 00:13:37.720 --> 00:13:40.799 their containers straight to production whenever they want? Oh boy, 283 00:13:40.960 --> 00:13:44.399 why do we still need OPS folks acting as gatekeepers? 284 00:13:44.639 --> 00:13:48.399 Because OPS still owns the underlying platform and they own 285 00:13:48.480 --> 00:13:51.639 the security of the host. Let's say the developer is 286 00:13:51.720 --> 00:13:55.279 rushing to me to deadline, they find a tutorial online 287 00:13:55.399 --> 00:13:58.440 and base their new container on an outdated Python image 288 00:13:58.480 --> 00:13:59.440 from three years ago. 289 00:13:59.559 --> 00:14:00.759 Oh where this is going? 290 00:14:00.919 --> 00:14:03.159 Yeah, if you let them push that directly to production, 291 00:14:03.639 --> 00:14:07.679 they've just introduced a container full of unpatched, highly exploitable 292 00:14:07.679 --> 00:14:12.240 security vulnerabilities into your network. That's terrifying. OPS completely loses 293 00:14:12.240 --> 00:14:15.320 control over what dependencies are running on their servers. 294 00:14:14.960 --> 00:14:17.639 Which leads us straight into the reality check of security 295 00:14:17.720 --> 00:14:21.000 in a containerized world. If you're giving developers the power 296 00:14:21.039 --> 00:14:24.399 to package their own environments, you need serious guardrails on 297 00:14:24.440 --> 00:14:25.200 the server side. 298 00:14:25.320 --> 00:14:29.159 Absolutely. The foundational concept that everyone needs to grasp here 299 00:14:29.279 --> 00:14:33.840 is that, unlike a virtual machine, containers share the host's kernel. 300 00:14:33.759 --> 00:14:36.320 Right, the kernel being the absolute core brain of the 301 00:14:36.320 --> 00:14:39.399 operating system, the part that actually talks to the physical 302 00:14:39.399 --> 00:14:40.360 hardware exactly. 303 00:14:40.399 --> 00:14:44.120 A virtual machine brings its own completely isolated kernel. It's 304 00:14:44.159 --> 00:14:48.679 a fortress. But a docker container is ultimately just a 305 00:14:48.799 --> 00:14:52.840 heavily restricted process running on the host server's brain. So 306 00:14:52.960 --> 00:14:56.360 if it's breached, if a hacker manages to compromise your 307 00:14:56.399 --> 00:15:00.000 application and they find a vulnerability in the shared Linux 308 00:15:00.039 --> 00:15:03.159 kernel itself, they can break out of the container entirely. 309 00:15:03.320 --> 00:15:05.879 Oh wow, they take over the entire host machine and 310 00:15:05.919 --> 00:15:08.919 by extension, every other container running on that metal. 311 00:15:08.960 --> 00:15:11.720 That is a massive threat. The sources actually mention a 312 00:15:11.759 --> 00:15:15.799 paper from Google about their internal container system BORG and 313 00:15:15.840 --> 00:15:18.480 it's very telling. Even Google, I mean, the pioneers of 314 00:15:18.480 --> 00:15:22.320 this technology. They use full heavy virtual machines when they 315 00:15:22.320 --> 00:15:25.039 are running untrusted external code. 316 00:15:25.200 --> 00:15:26.679 Yes that detail is crucial. 317 00:15:26.759 --> 00:15:30.639 They save their raw containers exclusively for their own highly trusted, 318 00:15:30.840 --> 00:15:32.320 internally written applications. 319 00:15:32.559 --> 00:15:35.759 When the engineers at Google don't trust a container boundary 320 00:15:36.000 --> 00:15:39.080 to hold back a dedicated attacker that tells you everything 321 00:15:39.120 --> 00:15:42.440 you need to know. The isolation is strong, but it 322 00:15:42.480 --> 00:15:43.679 is not impenetrable. 323 00:15:44.480 --> 00:15:47.080 To mitigate this, the book lists a bunch of security 324 00:15:47.120 --> 00:15:50.960 don'ts to lock these containers down in production One major 325 00:15:51.039 --> 00:15:53.639 rule is removing sued binaries from your images. 326 00:15:53.799 --> 00:15:53.960 Ah. 327 00:15:54.039 --> 00:15:57.679 Yes, SOUTH essentially stands for set owner User ID. It's 328 00:15:57.720 --> 00:16:00.600 a special permission in Linux that allows a normal user 329 00:16:00.639 --> 00:16:04.639 to execute a file with full root privileges temporarily, things 330 00:16:04.720 --> 00:16:07.039 like the ping command or the mount command. 331 00:16:07.159 --> 00:16:09.519 Use this right, which sounds harmless, but it's. 332 00:16:09.399 --> 00:16:12.759 Not, and the logic is bulletproof. Why on earth does 333 00:16:12.799 --> 00:16:15.320 a basic web server container need the ability to ping 334 00:16:15.360 --> 00:16:18.159 other servers on the network or mount physical hard drives. 335 00:16:18.159 --> 00:16:18.679 It doesn't. 336 00:16:18.840 --> 00:16:21.600 No, it really doesn't. It is the principle of least 337 00:16:21.600 --> 00:16:25.320 privilege applied to the extreme. You ruthlessly strip away any 338 00:16:25.360 --> 00:16:29.399 capability the application doesn't strictly need to function, vastly reducing 339 00:16:29.399 --> 00:16:30.279 the attack surface. 340 00:16:30.360 --> 00:16:33.440 They also talk about kernel security frameworks like a parmer 341 00:16:33.559 --> 00:16:36.159 or SELinux, though the author's joke that the most common 342 00:16:36.200 --> 00:16:39.360 Google search for cylinicx is how to disable Sylinics. 343 00:16:39.559 --> 00:16:41.600 Yeah, that's a very common joke in the ops. 344 00:16:41.320 --> 00:16:43.639 Community because it's so hard to use, because. 345 00:16:43.440 --> 00:16:48.120 Those frameworks are notoriously complex to configure, but they provide 346 00:16:48.120 --> 00:16:51.919 mandatory access control at the kernel level. If you can't 347 00:16:51.919 --> 00:16:56.080 implement Celinux, you should at least be utilizing Linux capabilities 348 00:16:56.240 --> 00:16:57.240 right capabilities. 349 00:16:57.279 --> 00:17:00.639 Instead of giving it container full root access, which is 350 00:17:00.679 --> 00:17:03.240 like giving a stranger the master skeleton key to your 351 00:17:03.360 --> 00:17:08.279 entire office building, you grant them one very specific capability, 352 00:17:08.480 --> 00:17:12.079 exactly like natadmin. They get the key to the networking closet, 353 00:17:12.119 --> 00:17:15.680 but they absolutely cannot open any file system doors. 354 00:17:15.599 --> 00:17:18.839 Which brings us to the most emphatic, non negotiable rule 355 00:17:19.119 --> 00:17:22.599 in the entire book. It is written in all caps 356 00:17:22.640 --> 00:17:23.359 for a reason. 357 00:17:23.599 --> 00:17:26.519 Right, do not run SSH in your containers. 358 00:17:26.640 --> 00:17:30.799 Yes, it completely violates the philosophy of immutable infrastructure. 359 00:17:30.920 --> 00:17:33.920 It really does, and for anyone listening, SSH is how 360 00:17:33.960 --> 00:17:36.200 you remotely log into a service command line to look 361 00:17:36.240 --> 00:17:39.759 around or fix things. Ssaging into a Docker container to 362 00:17:39.759 --> 00:17:42.319 troubleshoot is like popping the hood of your car while 363 00:17:42.359 --> 00:17:44.400 flying down the highway at seventy miles an hour to 364 00:17:44.480 --> 00:17:45.000 check the oil. 365 00:17:45.200 --> 00:17:46.400 That is exactly what it's like. 366 00:17:46.640 --> 00:17:48.400 You shouldn't be under the hood, you should just be 367 00:17:48.400 --> 00:17:50.000 looking at the dashboard telemetry. 368 00:17:50.279 --> 00:17:53.759 But this raises an important question. If you forbid engineers 369 00:17:53.759 --> 00:17:57.279 from popping the hood and a container crashes in production, 370 00:17:57.880 --> 00:18:01.279 how do you actually diagnose what went wrong? Right? 371 00:18:01.599 --> 00:18:04.920 You still need to know what happened telemetry and log mapping. 372 00:18:05.319 --> 00:18:08.920 You never store the application logs inside the container itself. 373 00:18:09.000 --> 00:18:09.599 No, never. 374 00:18:09.759 --> 00:18:12.079 You can figure Docker to map a volume, so the 375 00:18:12.079 --> 00:18:14.839 container is actually writing its logs directly to the host 376 00:18:14.880 --> 00:18:18.519 machine's hard drive. Then the host machine scoops up those 377 00:18:18.559 --> 00:18:21.119 logs and shicks them off to a central dashboard. Like 378 00:18:21.160 --> 00:18:23.000 Splunk or some logic. 379 00:18:22.920 --> 00:18:26.880 The container remains entirely disposable. If it crashes and burns, 380 00:18:27.160 --> 00:18:30.279 the crucial diagnostic logs survive safely on the host. It 381 00:18:30.480 --> 00:18:33.599 is brilliant an engineer can analyze what happened from their 382 00:18:33.640 --> 00:18:37.519 central dashboard without ever needing to SSH into a fragile, 383 00:18:37.640 --> 00:18:38.519 dying container. 384 00:18:39.319 --> 00:18:41.960 So what does this all mean? If we follow this 385 00:18:42.039 --> 00:18:45.119 logic to its conclusion. If the container is this perfectly 386 00:18:45.160 --> 00:18:48.559 sealed black box where we only extract logs from the outside, 387 00:18:48.880 --> 00:18:51.160 it completely changes how we think about the inside. 388 00:18:51.200 --> 00:18:51.839 It really does. 389 00:18:52.160 --> 00:18:54.680 What happens to all those configuration tools we used to 390 00:18:54.680 --> 00:18:58.279 rely on for a decade. Tools like chef, Puppet and 391 00:18:58.359 --> 00:19:03.079 antsible were the undisputed kings of configuration management. If the 392 00:19:03.240 --> 00:19:06.960 Docker image handles all the application dependencies, what do those 393 00:19:07.000 --> 00:19:08.079 tools even do now? 394 00:19:08.359 --> 00:19:12.279 Well, they suffer a major demotion. Really Yeah, the entropy, 395 00:19:12.440 --> 00:19:15.799 you know, the chaos of changing dependencies, library updates, and 396 00:19:15.839 --> 00:19:20.559 application code has been successfully contained inside the Docker image. 397 00:19:20.640 --> 00:19:22.920 So chef and Puppet are practically out of a job. 398 00:19:23.000 --> 00:19:25.839 I wouldn't say they are unemployed, but their scope is 399 00:19:25.960 --> 00:19:30.079 vastly reduced. Instead of intricately configuring every delicate detail the 400 00:19:30.119 --> 00:19:33.799 application environment, they are now mostly relegated to simply preparing 401 00:19:33.839 --> 00:19:36.359