WEBVTT 1 00:00:00.120 --> 00:00:04.480 Imagine you have just scraped this massive, maybe ten thousand 2 00:00:04.519 --> 00:00:08.320 page data set to train a new AI model, right, 3 00:00:08.480 --> 00:00:12.480 and your crawler did its job perfectly. But well, there 4 00:00:12.560 --> 00:00:14.240 is a catastrophic. 5 00:00:13.640 --> 00:00:15.560 Problem there usually is with raw data. 6 00:00:15.679 --> 00:00:19.519 Exactly, every single page in that data set is just poisoned. 7 00:00:19.519 --> 00:00:23.480 It's full of these invisible memory hogging script tags and 8 00:00:23.559 --> 00:00:24.679 tracking pixel. 9 00:00:24.480 --> 00:00:26.960 Oh yeah, that will absolutely kill your processing pipeline. 10 00:00:27.079 --> 00:00:29.920 Right, if you feed that raw HTML into your system, 11 00:00:30.239 --> 00:00:32.600 it's going to crash from memory overload. You don't even 12 00:00:32.640 --> 00:00:34.240 need to read the data at that point, you just 13 00:00:34.280 --> 00:00:36.880 need to like surgically destroy parts of it on the fly. 14 00:00:36.960 --> 00:00:38.880 Which is exactly what we're getting into today. 15 00:00:38.960 --> 00:00:42.520 Yes, welcome to the deep dive. Today we are learning 16 00:00:42.520 --> 00:00:46.479 how to actively hack and manipulate the HTML parse tree. 17 00:00:46.640 --> 00:00:49.079 And this is really a complete paradigm shift for anyone 18 00:00:49.119 --> 00:00:52.119 dealing with web data. I mean, when most developers start 19 00:00:52.119 --> 00:00:55.359 out using Python and beautiful Soup, they treat the pars 20 00:00:55.399 --> 00:00:58.439 tree like a static painting in a museum like look, 21 00:00:58.479 --> 00:01:01.640 but don't touch exactly. You can look at it, you 22 00:01:01.679 --> 00:01:04.879 can measure it, extract the colors, but you absolutely cannot 23 00:01:04.959 --> 00:01:08.760 touch the canvas. But today we are moving past that you, 24 00:01:09.120 --> 00:01:11.879 the listener, are stepping over the velvet rope. You weren't 25 00:01:11.920 --> 00:01:15.120 just reading a text file anymore. You are manipulating a live, 26 00:01:15.319 --> 00:01:18.480 intertrected database directly inside your computer's ram. 27 00:01:18.719 --> 00:01:21.000 And that is the crucial mental leap, right if you're 28 00:01:21.000 --> 00:01:25.239 building cybersecurity tools or you're just normalizing really messy data 29 00:01:25.280 --> 00:01:28.239 for machine learning, you have to transition from being just 30 00:01:28.280 --> 00:01:30.400 an observer to being an architect. 31 00:01:30.519 --> 00:01:33.640 Yes, every single tag in that document is a node, 32 00:01:34.359 --> 00:01:39.680 and beautiful soup gives you essentially root access to modify, delete, 33 00:01:39.840 --> 00:01:41.959 or even rebuild those nodes in real time. 34 00:01:42.519 --> 00:01:44.400 So to really unpack this and get a feel for 35 00:01:44.480 --> 00:01:47.359 how powerful it is, let's establish a sort of conceptual 36 00:01:47.439 --> 00:01:48.719 sandbox for you to picture. 37 00:01:48.840 --> 00:01:49.519 Okay, I like that. 38 00:01:49.680 --> 00:01:54.079 Imagine an incredibly simple HTML document loaded into your computer's memory. 39 00:01:54.280 --> 00:01:56.519 Let's call it the three Apples document. 40 00:01:56.560 --> 00:01:57.719 The three Apples document, got it? 41 00:01:57.840 --> 00:02:00.280 Yeah? It is a standard header a body, and inside 42 00:02:00.359 --> 00:02:03.359 that body there are just three very simple paragraphs. One 43 00:02:03.439 --> 00:02:06.159 is about Eve's Apple, the second is about Isaac Newton's Apple, 44 00:02:06.239 --> 00:02:08.080 and the third is about Steve jobs Apple. 45 00:02:08.360 --> 00:02:12.080 So no links, no images, just basic structural tags. 46 00:02:11.840 --> 00:02:15.719 Right now, exactly, just the bare bones, and I'm visualizing 47 00:02:15.840 --> 00:02:19.000 this HTML parstree like a living Bonzie tree. 48 00:02:19.039 --> 00:02:20.199 Oh that's a great analogy. 49 00:02:20.360 --> 00:02:22.479 Yeah, you have the root at the base, the trunk 50 00:02:22.560 --> 00:02:25.840 is the body tag, and the branches are your specific tags. 51 00:02:26.719 --> 00:02:29.199 Up until now in your scraping journey, you've just been 52 00:02:29.240 --> 00:02:32.479 observing the Bonzie. But today we are learning how to 53 00:02:32.520 --> 00:02:36.240 prune branches, graft entirely new ones on, and reshape the 54 00:02:36.360 --> 00:02:37.120 entire plant. 55 00:02:37.280 --> 00:02:40.000 And the cool thing is because that bon side tree 56 00:02:40.080 --> 00:02:44.400 exists entirely in your computer's active memory. Any modification you 57 00:02:44.479 --> 00:02:48.199 make instantly reorganizes the physical structure of the plant. 58 00:02:47.919 --> 00:02:50.280 Meaning if you change a branch, the leaves attached to 59 00:02:50.319 --> 00:02:51.919 it just move automatically exactly. 60 00:02:52.120 --> 00:02:55.000 So, if our scraper is working perfectly, but we've hit 61 00:02:55.039 --> 00:02:58.560 a roadblock with data inconsistency, we need to learn how 62 00:02:58.560 --> 00:03:01.280 to shape the branches we already have before we start 63 00:03:01.560 --> 00:03:02.360 planting new seeds. 64 00:03:02.400 --> 00:03:05.120 Okay, let's talk about data normalization then. Say I'm pulling 65 00:03:05.240 --> 00:03:08.719 data from thousands of older websites. Some developers use the 66 00:03:08.719 --> 00:03:11.319 bold tag the B tag, and others use the strong tag. 67 00:03:11.840 --> 00:03:15.759 But my AI model treats those as two completely different things. 68 00:03:15.840 --> 00:03:18.400 Which is a classic data science headache. I mean, we've 69 00:03:18.400 --> 00:03:21.199 all been there. Oh yeah, But solving it is incredibly 70 00:03:21.240 --> 00:03:24.840 straightforward once you understand that tag names are just mutable properties. 71 00:03:25.400 --> 00:03:28.400 Wait, so they're just variables you can change pretty much. 72 00:03:29.120 --> 00:03:32.759 If you locate a B tag in your soup object 73 00:03:32.919 --> 00:03:35.719 it has a name attribute, you can literally write a 74 00:03:35.759 --> 00:03:38.719 single line of Python code that assigns a new string 75 00:03:38.759 --> 00:03:39.479 to that property. 76 00:03:39.599 --> 00:03:40.719 Really, just like that. 77 00:03:40.919 --> 00:03:42.680 Yeah, you just tell the script to change the dot 78 00:03:42.759 --> 00:03:44.199 name property from b to strong. 79 00:03:44.639 --> 00:03:47.879 So I am not running some complex, error prone string 80 00:03:47.919 --> 00:03:51.039 replacement across the whole text document. I'm just modifying an 81 00:03:51.039 --> 00:03:53.759 object property and the tree instantly updates itself. 82 00:03:53.879 --> 00:03:58.439 That is the exact mechanism. The underlying HTML tag just transforms. 83 00:03:58.840 --> 00:04:02.360 If you use the Pretti five function, which outputs your 84 00:04:02.400 --> 00:04:05.599 live HTML structure in a clean indented format, you will 85 00:04:05.639 --> 00:04:08.960 see that your change just cascaded through the entire object. 86 00:04:09.080 --> 00:04:11.919 Wow. So your data set is instantly uniform, exactly, And 87 00:04:11.960 --> 00:04:14.039 it gets even more powerful when we look at the 88 00:04:14.080 --> 00:04:17.680 attributes buried inside those tags. Let's go back to our 89 00:04:17.720 --> 00:04:22.800 Bondsie tree, specifically the paragraph branch for Newton's apple. Okay, 90 00:04:23.079 --> 00:04:25.759 let's assume it currently has a class attribute called Apple, 91 00:04:26.399 --> 00:04:29.319 but my scraper needs that class to be more specific. 92 00:04:29.519 --> 00:04:31.839 I needed to say Newton. Can I just dive into 93 00:04:31.879 --> 00:04:32.839 the tag's dictionary? 94 00:04:32.959 --> 00:04:37.680 Yes you can, beautiful soup treats tag attributes identically to 95 00:04:37.800 --> 00:04:39.279 standard Python dictionaries. 96 00:04:39.319 --> 00:04:40.279 Oh that's so easy. 97 00:04:40.519 --> 00:04:44.079 It really is. You locate that specific paragraph tag, you 98 00:04:44.160 --> 00:04:46.879 look up the class key, and you just overwrite the 99 00:04:46.959 --> 00:04:48.160 value with Newton. 100 00:04:47.959 --> 00:04:50.680 And the moment that code executes. 101 00:04:50.279 --> 00:04:53.160 The metadata of that element is permanently altered in your 102 00:04:53.199 --> 00:04:54.120 computer's memory. 103 00:04:54.279 --> 00:04:58.319 That's amazing. But here's a pushback question. What if I'm 104 00:04:58.360 --> 00:05:01.879 working in a cybersecurity context. Say I find attributes that 105 00:05:01.920 --> 00:05:03.720 are actively malicious. 106 00:05:03.639 --> 00:05:07.040 Like an onload attribute hiding a nasty JavaScript. 107 00:05:06.600 --> 00:05:08.720 Paylost exactly, I definitely do not want to rename that. 108 00:05:08.759 --> 00:05:10.759 I want it completely eradicated from the document. 109 00:05:10.800 --> 00:05:13.079 Well, you just rely on standard Python syntax for that. 110 00:05:13.199 --> 00:05:13.399 Yeah. 111 00:05:13.439 --> 00:05:15.639 If you want an attribute gone, you use the Python 112 00:05:15.720 --> 00:05:17.360 dell do keyword. 113 00:05:17.000 --> 00:05:18.000 The delete keyword. 114 00:05:18.120 --> 00:05:21.680 Yeah, del you simply tell Python to delete that specific 115 00:05:21.720 --> 00:05:23.120 attribute key from the tag. 116 00:05:23.399 --> 00:05:27.600 That is just so elegant. Just dell a space and 117 00:05:27.879 --> 00:05:30.959 the attribute reference and the parstree doesn't complain that I'm 118 00:05:31.040 --> 00:05:33.319 just tearing metadata right out of it. 119 00:05:33.279 --> 00:05:36.319 Not at all. The attribute just vanishes. If you run 120 00:05:36.399 --> 00:05:39.759 prettify immediately after, you will see a bear tag. It's 121 00:05:39.800 --> 00:05:41.639 basically surgical data sanitization. 122 00:05:42.040 --> 00:05:45.920 You're stripping away the poison without damaging the surrounding text. Precisely, 123 00:05:46.079 --> 00:05:49.519 speaking of the surrounding text, I'm imagining we can manipulate 124 00:05:49.560 --> 00:05:53.319 the visible content just as easily. Right, Like, say, the 125 00:05:53.319 --> 00:05:56.600 text for Steve Jobs Apple has a glaring typo? Do 126 00:05:56.680 --> 00:05:59.279 I need to write a messy regular expression to find 127 00:05:59.319 --> 00:06:00.759 and fix that specific words? 128 00:06:00.839 --> 00:06:04.439 Oh? Please don't. Regular expressions would be massive overkill there. 129 00:06:04.480 --> 00:06:06.480 They're dangerous, right, very. 130 00:06:06.560 --> 00:06:09.720 They're highly prone to breaking the HTML structure. If you 131 00:06:09.800 --> 00:06:13.160 accidentally match a tag name instead of the visible text. 132 00:06:12.959 --> 00:06:15.040 Right, you could ruin the whole document exactly. 133 00:06:15.079 --> 00:06:17.879 So instead, you locate the tag containing the typo and 134 00:06:17.920 --> 00:06:21.240 you just assign a brand new, corrected string to its 135 00:06:21.279 --> 00:06:22.199 dot string property. 136 00:06:22.240 --> 00:06:24.800 Ah, so the old visible text is just overwritten. 137 00:06:25.120 --> 00:06:28.759 Yep. The new text drops right into place, and all 138 00:06:28.920 --> 00:06:32.519 those complex EACHTML brackets surrounding it are completely untouched. 139 00:06:32.879 --> 00:06:36.360 Okay, So we've reshaped the leaves, we've pruned the bad attributes, 140 00:06:36.399 --> 00:06:39.480 we've fixed the text. We've proven we can manipulate what's 141 00:06:39.519 --> 00:06:44.160 already there, but a digital architect needs to build. Let's 142 00:06:44.160 --> 00:06:47.319 talk about making this document actually grow. If I want 143 00:06:47.319 --> 00:06:50.920 to graft entirely new branches onto this Bonzi tree, how 144 00:06:50.920 --> 00:06:52.319 do I inject new data? 145 00:06:52.639 --> 00:06:56.160 Well, when it comes to adding content, the logic closely 146 00:06:56.160 --> 00:07:00.040 mirrors standard Python lists. Your main entry point is the 147 00:07:00.199 --> 00:07:02.639 pen function dot a pend. Okay, let's say you have 148 00:07:02.639 --> 00:07:05.920 a paragraph tag with the ID three apples, and the 149 00:07:05.959 --> 00:07:09.160 text inside says not really an insignificant fruit. If you 150 00:07:09.199 --> 00:07:10.959 want to expand on that thought, you called dot a 151 00:07:11.040 --> 00:07:12.639 pend on that tag and pass it a new strings 152 00:07:12.720 --> 00:07:14.240 something like quite interesting. 153 00:07:14.480 --> 00:07:17.600 Now, if I remember how Python lists work, dota pend 154 00:07:17.720 --> 00:07:20.319 always shoves the new item to the very back of 155 00:07:20.319 --> 00:07:22.839 the line, right, that's right, so that new text drops 156 00:07:22.839 --> 00:07:25.759 in at the absolute end of whatever is already inside 157 00:07:25.759 --> 00:07:26.720 that paragraph tag. 158 00:07:26.959 --> 00:07:29.160 It does. And if you have a massive list of 159 00:07:29.240 --> 00:07:31.920 new data points you want to add, maybe an array 160 00:07:31.959 --> 00:07:33.920 of a dozen new strings, you can graduate to the 161 00:07:33.959 --> 00:07:34.639 extend function. 162 00:07:34.759 --> 00:07:37.480 Oh, dot extend like lists again. 163 00:07:37.240 --> 00:07:40.360 Exactly, You pass in your list and it iterates through 164 00:07:40.639 --> 00:07:43.680 adding each element sequentially to the end of the tag's contents. 165 00:07:43.759 --> 00:07:45.519 But I'm guessing I don't always want to be at 166 00:07:45.560 --> 00:07:48.000 the back of the line. Usually not Like if I'm 167 00:07:48.040 --> 00:07:51.399 injecting a high priority warning label into a diff container, 168 00:07:52.040 --> 00:07:54.800 I need that warning at the very beginning before any 169 00:07:54.839 --> 00:07:56.560 of the existing texts even renders. 170 00:07:56.839 --> 00:08:00.279 For precision placement inside a tag, you use the inside. 171 00:08:00.240 --> 00:08:01.639 Function dot insert. 172 00:08:01.839 --> 00:08:05.680 Yes, it takes two arguments. The first is an index 173 00:08:05.759 --> 00:08:09.079 number representing this spatial position. So if you pass an 174 00:08:09.079 --> 00:08:12.279 index of zero, you are instructing the parse tree to 175 00:08:12.360 --> 00:08:15.839 propend your new content to the absolute beginning of the 176 00:08:15.879 --> 00:08:17.120 tag's internal structure. 177 00:08:17.240 --> 00:08:18.759 So you're basically cutting the line. 178 00:08:18.800 --> 00:08:19.480 You're cutting the line. 179 00:08:19.480 --> 00:08:22.199 Okay, picture the htmltree on your screen right now. Both 180 00:08:22.279 --> 00:08:25.079 dot a pend and dot insert are operating inside the 181 00:08:25.079 --> 00:08:27.920 boundaries of a specific tag. You are putting new leaves 182 00:08:27.920 --> 00:08:31.399 on an existing branch. But what if my scraper decides 183 00:08:31.439 --> 00:08:33.399 we need a brand new branch right next door. 184 00:08:33.519 --> 00:08:36.399 To step outside the tag, you have to use relative insertion, 185 00:08:37.000 --> 00:08:39.919 and the functions are beautifully explicit for this. They're called 186 00:08:39.960 --> 00:08:42.159 dot insert before and dot insert after. 187 00:08:42.399 --> 00:08:45.399 Oh, that is explicit. So if I locate the paragraph 188 00:08:45.559 --> 00:08:49.039 for Steve Jobs's apple, and I use dot in for tafter, 189 00:08:49.480 --> 00:08:51.559 I can place new data directly beneath it. 190 00:08:51.679 --> 00:08:54.919 Yep, it becomes a sibling in the overarching HTML hierarchy. 191 00:08:55.080 --> 00:08:58.159 That kind of spatial control is everything when you're dynamically 192 00:08:58.240 --> 00:09:02.519 generating reports or like rebuilding a web interface on the fly. 193 00:09:02.879 --> 00:09:05.639 It really is. However, this brings us to a really 194 00:09:05.720 --> 00:09:07.279 critical architectural rule. 195 00:09:07.440 --> 00:09:08.279 Okay, what is it? 196 00:09:08.440 --> 00:09:10.879 Well, so far we've just been pushing raw text strings 197 00:09:10.879 --> 00:09:14.039 into the tree. But HTML isn't just flat text. It's 198 00:09:14.080 --> 00:09:15.840 a strictly formatted ecosystem. 199 00:09:16.120 --> 00:09:19.240 Right. If I just shove raw Python strings everywhere, the 200 00:09:19.240 --> 00:09:21.639 tree won't recognize them as structural nodes, will it. 201 00:09:22.000 --> 00:09:22.720 No, it won't. 202 00:09:22.799 --> 00:09:25.519 I'm guessing that completely breaks the navigation logic if I 203 00:09:25.559 --> 00:09:27.080 try to run a search on the tree later. 204 00:09:27.440 --> 00:09:30.919 That is the exact reason why we need to introduce specialized, 205 00:09:31.440 --> 00:09:35.879 beautiful soup objects. If you inject a standard Python string 206 00:09:36.240 --> 00:09:39.840 into the tree, and later your scraper runs a command 207 00:09:40.000 --> 00:09:43.639 like dot find parent on that text, the program will crash. 208 00:09:43.879 --> 00:09:45.840 Oh wow, Why. 209 00:09:45.559 --> 00:09:47.720 Because the native Python string does not have a parent, 210 00:09:47.799 --> 00:09:51.360 it does not have siblings, it has zero spatial awareness. 211 00:09:51.639 --> 00:09:54.679 So it's just a dumb string floating in the voyage essentially, yes, 212 00:09:54.799 --> 00:09:56.679 So how do we give it spatial awareness. 213 00:09:56.759 --> 00:09:59.200 You have to import a tool called navigable. 214 00:09:58.720 --> 00:10:00.000 String navigable strength. 215 00:10:00.120 --> 00:10:05.080 It's beautiful soups special wrapper for text. When you instantiate 216 00:10:05.159 --> 00:10:08.039 a navigal string, it retains all the dictionary methods and 217 00:10:08.080 --> 00:10:11.000 traversal pointers of the whole HTML. 218 00:10:10.600 --> 00:10:13.279 Parstree ra I see. So it's a string that actually 219 00:10:13.320 --> 00:10:16.000 knows where it lives and knows who its parent tag is, 220 00:10:16.080 --> 00:10:17.759 and it knows what tags are sitting right next to 221 00:10:17.799 --> 00:10:18.480 it precisely. 222 00:10:18.519 --> 00:10:21.000 It becomes a fully integrated resident of the ecosystem. 223 00:10:21.080 --> 00:10:22.039 That makes total sense. 224 00:10:22.200 --> 00:10:25.000 Now, to be fair, the library is quite smart. If 225 00:10:25.039 --> 00:10:27.759 you pass a regular string into Dota pend, it will 226 00:10:27.799 --> 00:10:30.120 usually convert it to a navigable string under the hood 227 00:10:30.120 --> 00:10:30.360 for you. 228 00:10:30.440 --> 00:10:33.039 Oh so it handles it automatically sometimes. 229 00:10:32.879 --> 00:10:37.080 Sometimes, But when you are building complex structures from scratch, 230 00:10:37.679 --> 00:10:41.759 explicitly defining your navigable objects prevents a lot of really 231 00:10:41.799 --> 00:10:43.480 painful debugging down the line. 232 00:10:43.639 --> 00:10:47.679 Best practices, Right, What about other structural elements that aren't 233 00:10:47.759 --> 00:10:50.279 visible text like developer comments. 234 00:10:50.320 --> 00:10:51.320 Well that's a good one. 235 00:10:51.200 --> 00:10:54.080 Because sometimes if I'm sanitizing a page, I don't just 236 00:10:54.080 --> 00:10:56.200 want it to lead a bad tag. I want to 237 00:10:56.240 --> 00:10:58.879 comment it out and leave a breadcrumb for the auditing team. 238 00:10:59.120 --> 00:11:02.279 You handle that by by importing the comment object just 239 00:11:02.320 --> 00:11:05.759 called comment yep, capital C. You instantiate it with your 240 00:11:05.759 --> 00:11:08.480 auditing note and you inject it using dot insert for 241 00:11:08.720 --> 00:11:09.559 dot pend. 242 00:11:09.360 --> 00:11:10.799 And it formats it correctly. 243 00:11:10.919 --> 00:11:13.399 Yeah. When you export the HTML, it doesn't render as 244 00:11:13.519 --> 00:11:17.440 raw text. It formats perfectly as an official HTML comment, 245 00:11:17.879 --> 00:11:20.399 you know, complete with the angle brackets, the exclamation point, 246 00:11:20.440 --> 00:11:21.120 and the dashes. 247 00:11:21.200 --> 00:11:23.080 Okay, this is where we get to the real heavy lifting. 248 00:11:23.159 --> 00:11:26.200 We have moved texts, we've injected comments, but I want 249 00:11:26.200 --> 00:11:28.759 to build a totally new structural container. 250 00:11:29.000 --> 00:11:29.480 Let's do it. 251 00:11:29.639 --> 00:11:32.840 I want to build a brand new HTML tag from scratch, 252 00:11:32.960 --> 00:11:35.679 like an image tag or a hyperlink, and weave it 253 00:11:35.720 --> 00:11:36.600 into the document. 254 00:11:36.919 --> 00:11:40.240 Then you need the soup dot new tag function. Let's 255 00:11:40.240 --> 00:11:41.679 say you want to add a hyperlink, which is the 256 00:11:41.720 --> 00:11:44.279 tag tag. Right, you literally just call soup dot new 257 00:11:44.360 --> 00:11:47.840 tag and pass it the letter A. You now have 258 00:11:47.960 --> 00:11:50.360 a tag object floating in your system's RAM. 259 00:11:50.440 --> 00:11:53.480 But it's totally empty, right, It's just a blank node exactly. 260 00:11:53.480 --> 00:11:56.279 It's an empty show. Yeah, so you have to populate it. 261 00:11:56.720 --> 00:11:59.519 You set its string property to the word apple. Then 262 00:11:59.559 --> 00:12:02.200 you act it's href attribute, just like we did when 263 00:12:02.240 --> 00:12:05.679 we were modifying existing attributes earlier, and you assign it 264 00:12:05.919 --> 00:12:07.480 the URL for the Apple. 265 00:12:07.200 --> 00:12:09.879 Store and boom, now you have a fully formed, functional 266 00:12:09.960 --> 00:12:13.120 hyperlink in memory exactly. And from there I can just 267 00:12:13.279 --> 00:12:16.320 use dot insert after to graft this brand new branch 268 00:12:16.480 --> 00:12:18.879 directly on my Bonzi tree, like right next to the 269 00:12:18.919 --> 00:12:20.159 Steve jobs paragraph. 270 00:12:20.240 --> 00:12:23.159 You are literally engineering new architecture on the fly. It's 271 00:12:23.200 --> 00:12:26.799 pretty amazing, it is. But as any good gardener knows, 272 00:12:27.080 --> 00:12:30.200 creation is only half the job. Sometimes you really have 273 00:12:30.240 --> 00:12:31.399 to cut things away. 274 00:12:31.360 --> 00:12:33.799 Right, which brings us back to the massive problem we 275 00:12:33.840 --> 00:12:36.200 discussed at the very beginning of the deem dive, the 276 00:12:36.240 --> 00:12:39.440 ten thousand page data set that is just choking on 277 00:12:39.600 --> 00:12:41.039 memory heavy script tags. 278 00:12:41.440 --> 00:12:45.200 Yes, we need to prune the tree, and beautiful soup 279 00:12:45.440 --> 00:12:47.960 gives you a choice of how you destroy an element, 280 00:12:48.320 --> 00:12:50.720 which is absolutely vital for memory management. 281 00:12:50.960 --> 00:12:52.159 Okay, what are the choices? 282 00:12:52.399 --> 00:12:56.000 There are two main functions for removing elements dot extract 283 00:12:56.039 --> 00:12:57.000 and dot decomposed. 284 00:12:57.200 --> 00:12:59.000 Extract and decomposed, yes. 285 00:12:59.440 --> 00:13:03.399 And understanding the stark difference between them is probably the 286 00:13:03.480 --> 00:13:06.759 single most important takeaway for anyone dealing with large scale 287 00:13:06.799 --> 00:13:07.919 data processing today. 288 00:13:08.039 --> 00:13:10.559 All right, Well, let's start with dot extract. If my 289 00:13:10.720 --> 00:13:15.000 script finds a massive embedded video player tag and I 290 00:13:15.039 --> 00:13:18.159 called dot extract on it, what actually happens inside the. 291 00:13:18.200 --> 00:13:21.879 Ram beautiful soup immediately severs that tag and all of 292 00:13:21.919 --> 00:13:24.679 its nested contents from the HTML parstree. 293 00:13:24.720 --> 00:13:26.279 Okay, so it's out of the tree, right. 294 00:13:26.360 --> 00:13:29.360 If you look the tree afterwards, the video player is gone. 295 00:13:29.639 --> 00:13:33.000 If you check the extracted tags, carn't attribute, it returns 296 00:13:33.519 --> 00:13:35.120 none because it's been orphaned. 297 00:13:35.279 --> 00:13:36.919 But the tag isn't actually dead, is it. 298 00:13:37.039 --> 00:13:40.279 No, The dot extract function returns the data you just removed, 299 00:13:40.399 --> 00:13:43.559 So that massive video tag is still sitting there, completely intact, 300 00:13:43.840 --> 00:13:45.480 in a variable in your Python script. 301 00:13:45.679 --> 00:13:48.679 Ah. So, to use our gardening metaphor, dot extract is 302 00:13:48.720 --> 00:13:51.120 like carefully uprooting a weed and placing it into a 303 00:13:51.159 --> 00:13:54.200 temporary pot on your patio. Exactly, it's no longer in 304 00:13:54.240 --> 00:13:56.879 the garden soil, but it's still drinking water. It's still 305 00:13:56.919 --> 00:14:02.240 consuming your computer's memory. I could theoretically take that potted 306 00:14:02.279 --> 00:14:05.320 plant and graft it into a completely different HTML file 307 00:14:05.519 --> 00:14:07.879 if I wanted to, if there's an excellent. 308 00:14:07.559 --> 00:14:11.519 Way to visualize it. You preserve the data structure for 309 00:14:11.639 --> 00:14:14.960 future use. But think about our data set scenario. You 310 00:14:15.080 --> 00:14:18.720 have millions of malicious script tags. You do not want 311 00:14:18.720 --> 00:14:19.879 them in a pot on your patio. 312 00:14:19.960 --> 00:14:22.440 Oh definitely not. If I extract millions of script tags 313 00:14:22.480 --> 00:14:25.000 and hold them in variables, my RAM usage is going 314 00:14:25.080 --> 00:14:27.519 to spike until the whole system crashes. I don't want 315 00:14:27.519 --> 00:14:29.080 a pot, I want an incinerator. 316 00:14:29.080 --> 00:14:32.159 And that incinerator is the dot decomposed. 317 00:14:31.600 --> 00:14:33.080 Function dot decomposed Yep. 318 00:14:33.200 --> 00:14:36.240 When you locate a tag and call dot decompose on it, 319 00:14:36.240 --> 00:14:38.879 it doesn't just sever the node from the tree. It 320 00:14:38.919 --> 00:14:41.360 completely obliterates the tag and all of its. 321 00:14:41.240 --> 00:14:43.399 Contents, so it wipes it from memory entirely. 322 00:14:43.559 --> 00:14:47.120 Entirely, it reclaims that RAM the data is gone forever. 323 00:14:47.559 --> 00:14:51.200 Wow. That is a life saving distinction for AI developers. 324 00:14:51.480 --> 00:14:53.960 If you know you absolutely do not need the data, 325 00:14:54.159 --> 00:14:56.960 you should always use dot decomposed to keep your processing 326 00:14:56.960 --> 00:14:57.840 pipeline lightweight. 327 00:14:57.879 --> 00:15:02.080 Absolutely. But you know data cleaning is always binary. Sometimes 328 00:15:02.080 --> 00:15:03.639 you don't want to destroy a tag and you don't 329 00:15:03.679 --> 00:15:05.399 want to extract it either. You just need to give 330 00:15:05.399 --> 00:15:06.840 it a structural makeover. 331 00:15:06.639 --> 00:15:10.120 Like swapping an element out entirely exactly. Say I have 332 00:15:10.159 --> 00:15:13.720 an outdated H one header tag and my new design 333 00:15:13.759 --> 00:15:16.720 system requires it to be an italicized itag. 334 00:15:16.960 --> 00:15:19.120 For that, you use the dot replaced with function. 335 00:15:19.480 --> 00:15:20.639 Okay, how does that flow? 336 00:15:20.840 --> 00:15:23.320 Well? You use soup dot new tag to build your 337 00:15:23.360 --> 00:15:27.200 brand new italic tag in memory first, Then you locate 338 00:15:27.240 --> 00:15:29.600 the old header in your tree and you call dot 339 00:15:29.639 --> 00:15:32.639 replaced with passing in the new tag. It just swaps 340 00:15:32.679 --> 00:15:35.799 them instantly. The old branch is pulled out and the 341 00:15:35.840 --> 00:15:39.000 new branch is slided perfectly into its exact spatial position. 342 00:15:39.240 --> 00:15:42.279 That is incredibly clean. But what if the problem is 343 00:15:42.320 --> 00:15:45.759 a bit more insidious. I'm thinking of legacy websites, you know, 344 00:15:45.799 --> 00:15:48.320 the kind to built fifteen years ago, where every single 345 00:15:48.360 --> 00:15:51.759 piece of text is buried inside a nightmare of useless 346 00:15:51.879 --> 00:15:53.519 nested div and span tags. 347 00:15:53.840 --> 00:15:56.799 Ah yes, develvers call that div soup div soup. 348 00:15:56.879 --> 00:15:57.919 Yeah, it's awful. 349 00:15:58.120 --> 00:16:01.360 Dealing with div soup is notoriously difficult with regular expressions 350 00:16:01.600 --> 00:16:04.279 because you risk deleting the core text while you're trying 351 00:16:04.279 --> 00:16:05.559 to strip away all the brackets. 352 00:16:05.799 --> 00:16:08.159 Right, So how does beautiful soup handle it? 353 00:16:08.159 --> 00:16:10.559 It solves this beautifully with the dota and wrap function. 354 00:16:10.840 --> 00:16:14.039 Unwrap How does that work in the purse tree. If 355 00:16:14.080 --> 00:16:16.120 I have a core piece of text but it's wrapped 356 00:16:16.120 --> 00:16:20.120 in let's say three layers of useless span tags, how 357 00:16:20.159 --> 00:16:23.159 do I get to the center without destroying the whole branch. 358 00:16:23.240 --> 00:16:26.240 You locate the outermost span tag and you just called 359 00:16:26.320 --> 00:16:29.960 dota unwrap and what happens. Beautiful soup magically peels away 360 00:16:30.080 --> 00:16:33.240 that specific outer container, kind of like stripping bark from 361 00:16:33.240 --> 00:16:36.720 a branch. The core text and anything else nested inside 362 00:16:37.000 --> 00:16:39.919 is left completely untouched. Oh wow, Yeah, it occupies the 363 00:16:40.000 --> 00:16:42.759 exact same position in the document. And you could just 364 00:16:42.840 --> 00:16:45.840 loop this to strip away endless layers of garbage containers 365 00:16:46.200 --> 00:16:48.080 until your core data is pristine. 366 00:16:48.279 --> 00:16:52.200 That is brilliant. It's literally structural peeling. And I assume 367 00:16:52.200 --> 00:16:55.039 the inverse is also true. Like if I have raw 368 00:16:55.120 --> 00:16:57.639 texts that suddenly needs to be contained within a new 369 00:16:57.679 --> 00:16:59.240 formatting tag, I can wrap it up. 370 00:16:59.360 --> 00:17:02.679 Yes, the dot wrap function. You can struck a new tag, 371 00:17:02.799 --> 00:17:05.680 say a bold B tag, and you called dot wrap 372 00:17:05.720 --> 00:17:09.039 on your existing text node. The parse tree takes your 373 00:17:09.039 --> 00:17:12.799 text and meatly encapsulates it inside the new bold container. 374 00:17:13.119 --> 00:17:18.200 Man, we have sliced, diced, unwrapped, extracted, and decomposed our 375 00:17:18.240 --> 00:17:20.680 way through this entire document. We really have, But all 376 00:17:20.759 --> 00:17:25.160 of this intricate surgery has been happening invisibly suspended inside 377 00:17:25.200 --> 00:17:28.880 the Python object's memory. The Bondzi tree is perfect, but 378 00:17:28.920 --> 00:17:31.119 it's sort of trapped in the matrix. How do we 379 00:17:31.160 --> 00:17:33.000 pull it out into the real world. 380 00:17:33.240 --> 00:17:35.640 This is where you realize the ultimate payoff of everything 381 00:17:35.640 --> 00:17:38.720 we've discussed today. Now, once all your structural makeovers are complete, 382 00:17:39.039 --> 00:17:41.480 you just return to basic Python file operations. 383 00:17:41.519 --> 00:17:43.640 Okay, so you just open a brand new file, let's 384 00:17:43.640 --> 00:17:46.720 call it sanitized dataset dot HTML and write mode. 385 00:17:46.960 --> 00:17:50.440 Right. So, after all this complex object oriented memory manipulation, 386 00:17:51.079 --> 00:17:53.640 exporting it is really just saving a standard text file. 387 00:17:53.759 --> 00:17:55.759 Really, it's that simple exactly. 388 00:17:56.039 --> 00:17:58.240 You call the standard righte function and you pass it 389 00:17:58.319 --> 00:18:01.880 soup dot pritiphi. That de man takes your dynamically modified 390 00:18:01.920 --> 00:18:05.000 live in memory parse tree, flattens it back into beautify 391 00:18:05.039 --> 00:18:08.519 formatted HTML text, and writes it directly to your hard drive. 392 00:18:08.680 --> 00:18:12.759 Wow. You essentially freeze the Bonzai tree in time, perfectly pruned, 393 00:18:12.920 --> 00:18:15.079 and save it as a brand new reality. When you 394 00:18:15.119 --> 00:18:18.920 open that new file, every single change, the renamed tags, 395 00:18:18.960 --> 00:18:23.079 the eradicated attributes, the newly grafted hyperlinks. It's all prominently 396 00:18:23.119 --> 00:18:23.519 locked in. 397 00:18:23.880 --> 00:18:27.519 You started with a messy, potentially dangerous read only document, 398 00:18:27.839 --> 00:18:31.960 and you programmatically engineered a completely sanitized, normalized structure. 399 00:18:32.200 --> 00:18:35.359 You are now officially a digital architect. You have the 400 00:18:35.400 --> 00:18:39.599 toolkit to modify properties, precisely, navigate the spatial structure of 401 00:18:39.640 --> 00:18:42.759