WEBVTT 1 00:00:00.200 --> 00:00:02.600 Right now, as you're listening to this, there are just 2 00:00:03.160 --> 00:00:05.960 thousands of invisible bots crawling the way pages you visit 3 00:00:06.000 --> 00:00:09.480 every day, well constantly. They're ripping apart the structural skeleton 4 00:00:09.519 --> 00:00:14.599 of the Internet, extracting data to train massive AI models, 5 00:00:14.720 --> 00:00:18.000 scraping pricing information for hedge funds, and just executing these 6 00:00:18.039 --> 00:00:22.079 automated sweeps to find exposed cybersecurity vulnerabilities. 7 00:00:22.160 --> 00:00:23.800 Right because when you look at a web page, you 8 00:00:23.879 --> 00:00:26.280 see a finished, glossy product. 9 00:00:26.000 --> 00:00:30.399 Exactly, but undernith that visual layer, there was just this chaotic, 10 00:00:31.199 --> 00:00:33.119 sprawling matrix of code. 11 00:00:33.240 --> 00:00:35.960 It is the absolute definition of hidden complexity. 12 00:00:36.079 --> 00:00:36.280 Yeah. 13 00:00:36.320 --> 00:00:38.479 I mean to a normal user, a web browser is 14 00:00:38.520 --> 00:00:41.960 just rendering a beautiful magazine layout. But to a machine, 15 00:00:42.119 --> 00:00:45.640 that page is a highly structured database. Well, assuming you 16 00:00:45.679 --> 00:00:47.039 have the right tools to interrogate it. 17 00:00:47.119 --> 00:00:50.320 You know, welcome to this deep dive today. Our mission 18 00:00:50.479 --> 00:00:53.399 is to basically reverse engineer the mind of the web developer. 19 00:00:53.840 --> 00:00:57.759 We are going to explore how to mentally, visualize, navigate, 20 00:00:57.880 --> 00:01:01.000 and surgically extract the exact data you want from the 21 00:01:01.079 --> 00:01:02.600 raw HTML of a web page. 22 00:01:02.640 --> 00:01:04.359 And we're doing it using one of the most powerful 23 00:01:04.400 --> 00:01:07.120 Python libraries in existence, beautiful soup. 24 00:01:07.159 --> 00:01:10.079 Right, and we were taking this journey entirely through audio, 25 00:01:10.519 --> 00:01:13.040 so no screens required, no walls of text. We're going 26 00:01:13.120 --> 00:01:15.239 to build a really robust mental model together. 27 00:01:15.359 --> 00:01:17.719 By the end of this, you'll be able to turn dense, 28 00:01:17.959 --> 00:01:23.040 overwhelming code into a living, breathing landscape that you can 29 00:01:23.120 --> 00:01:24.319 navigate programmatically. 30 00:01:24.400 --> 00:01:27.040 Okay, let's unpack this, because if we're going to navigate 31 00:01:27.079 --> 00:01:29.959 this landscape, we really need to understand its architecture. A 32 00:01:30.000 --> 00:01:34.480 web page is fundamentally a massive, upside down family tree. 33 00:01:34.640 --> 00:01:35.840 That's a great way to look at it. 34 00:01:35.920 --> 00:01:39.200 At the very top, the ancient ancestor of the entire 35 00:01:39.239 --> 00:01:43.319 document is the HTML tag. Everything else, all the text, 36 00:01:43.480 --> 00:01:47.079 the images, the navigation bars, it all branches down from 37 00:01:47.079 --> 00:01:48.400 that single origin point. 38 00:01:48.519 --> 00:01:51.840 What's fascinating here is that the family tree concept is critical. 39 00:01:52.120 --> 00:01:55.519 I mean understanding it isn't just some theoretical computer science. 40 00:01:55.239 --> 00:01:56.680 Exercise, most highly practical. 41 00:01:57.040 --> 00:02:00.319 Exactly, if you're trying to automate data extractions and you 42 00:02:00.359 --> 00:02:02.799 have to realize that the data isn't just floating in 43 00:02:02.840 --> 00:02:06.599 a vacuum, it is anchored to specific branches of this tree. 44 00:02:07.159 --> 00:02:10.759 Your ability to scrape a site effectively it basically depends 45 00:02:10.919 --> 00:02:14.199 entirely on your ability to traverse those exact branches without 46 00:02:14.199 --> 00:02:15.319 getting lost in the noise. 47 00:02:15.479 --> 00:02:18.120 So let's start with vertical traversal. Before we get into 48 00:02:18.159 --> 00:02:21.280 any advanced filtering, we have to know how to climb 49 00:02:21.360 --> 00:02:25.120 up and down the bloodline. I know beautiful soup has 50 00:02:25.159 --> 00:02:26.400 a dot children property. 51 00:02:26.479 --> 00:02:26.840 Mm hmm. 52 00:02:27.479 --> 00:02:31.360 If I apply dot children to a tag, I'm guessing 53 00:02:31.400 --> 00:02:33.560 it doesn't give me everything inside it, right? 54 00:02:33.840 --> 00:02:36.960 Correct? So the dot children property returns an iterator containing 55 00:02:37.039 --> 00:02:40.159 only the direct immediate children of that specific tag. 56 00:02:40.199 --> 00:02:41.840 Okay, So if you are looking at the top level 57 00:02:41.960 --> 00:02:43.000 HTML tag. 58 00:02:42.840 --> 00:02:45.599 It's immediate children are generally just the head tag and 59 00:02:45.639 --> 00:02:46.319 the body tag. 60 00:02:46.400 --> 00:02:48.719 Wait, what about all the paragraphs and images inside the 61 00:02:48.719 --> 00:02:50.879 body Like? If I am at the HTML tag and 62 00:02:50.919 --> 00:02:53.639 I ask for its children, does it just ignore all 63 00:02:53.680 --> 00:02:54.319 of that data? 64 00:02:54.479 --> 00:02:55.400 Completely ignores it? 65 00:02:55.520 --> 00:02:55.759 Wow? 66 00:02:55.879 --> 00:02:58.400 Think about your own family tree, right? If I ask 67 00:02:58.479 --> 00:03:01.879 you who your children are, you name your sons and daughters. 68 00:03:02.280 --> 00:03:04.960 You don't start listing your grandchildren and great grandchildren, right? 69 00:03:04.960 --> 00:03:05.879 That would be overwhelming. 70 00:03:05.960 --> 00:03:09.000 And the parser works the exact same way. It's highly 71 00:03:09.080 --> 00:03:12.039 memory efficient because it only looks one level deep. So 72 00:03:12.199 --> 00:03:15.800 a paragraph nested inside a divtag which is nested inside 73 00:03:15.840 --> 00:03:19.439 the body tag is essentially a great grandchild of the 74 00:03:19.560 --> 00:03:20.360 HTML root. 75 00:03:20.520 --> 00:03:20.840 Got it. 76 00:03:21.159 --> 00:03:23.360 The dot children property won't even see it. 77 00:03:23.639 --> 00:03:26.800 Okay, but what if I actually need the whole sprawling 78 00:03:26.879 --> 00:03:30.360 ancestry chart, Like if I'm scraping an article, I don't 79 00:03:30.400 --> 00:03:33.080 just want the top level container. I want every single 80 00:03:33.120 --> 00:03:35.039 piece of text nested all the way down. 81 00:03:35.159 --> 00:03:37.240 Well, that is when you swap dot children for the 82 00:03:37.280 --> 00:03:40.960 dot descendant's properties descendence. Okay, Yeah, calling dot descendants forces 83 00:03:41.000 --> 00:03:44.319 the parser to recursively travel down every single branch until 84 00:03:44.360 --> 00:03:45.240 it hits a dead end. 85 00:03:45.479 --> 00:03:47.599 Oh so just keeps going exactly. 86 00:03:47.639 --> 00:03:49.560 It gives you the children, then it opens up those 87 00:03:49.639 --> 00:03:53.080 children and grabs the grandchildren. Just drilling all the way 88 00:03:53.080 --> 00:03:55.439 down to the deepest nested elements, which we call the 89 00:03:55.479 --> 00:03:56.080 leaf nodes. 90 00:03:56.400 --> 00:03:59.400 I can see how that would be compucationally heavier, but 91 00:03:59.479 --> 00:04:03.520 absolutely necessary if you're trying to flatten a really complex document, 92 00:04:04.159 --> 00:04:06.159 and I assume we can travel in. 93 00:04:06.159 --> 00:04:09.199 Reverse, you can. Yeah, every single tag in the parstree 94 00:04:09.240 --> 00:04:10.680 has a dot parent property. 95 00:04:11.360 --> 00:04:11.840 Makes sense. 96 00:04:11.919 --> 00:04:13.599 So if you find an image you want and you 97 00:04:13.639 --> 00:04:16.680 need to know what container it sits inside, calling dot 98 00:04:16.759 --> 00:04:20.319 parent steps you exactly one level up the hierarchy. 99 00:04:19.959 --> 00:04:22.759 And is there a plural version like with descendants. 100 00:04:23.439 --> 00:04:26.240 Yes, just like dot descendants, there is a dot parent's 101 00:04:26.360 --> 00:04:30.519 property plural that will yield a generator stepping all the 102 00:04:30.519 --> 00:04:33.199 way back up to the HTML ancestor. 103 00:04:33.319 --> 00:04:35.160 Now. I was looking into this earlier and I found 104 00:04:35.160 --> 00:04:38.199 a really strange technical quirk with the parent property. 105 00:04:38.240 --> 00:04:39.199 Oh, the root node thing. 106 00:04:39.519 --> 00:04:41.279 Yeah, if you step all the way up to the 107 00:04:41.279 --> 00:04:44.680 top HTML tag and you ask for its parent, you'd 108 00:04:44.680 --> 00:04:46.600 expect it to return an error, right because it's the 109 00:04:46.639 --> 00:04:48.079 top of the tree. But it doesn't. 110 00:04:48.240 --> 00:04:51.680 Yeah, that's a fascinating detail about how beautiful Soup constructs 111 00:04:51.720 --> 00:04:55.040 its internal memory model. The immediate parent of the HTML 112 00:04:55.160 --> 00:04:58.560 tag is actually the beautiful Soup document object itself. 113 00:04:58.600 --> 00:05:00.000 Wait, the document itself is a parent. 114 00:05:00.519 --> 00:05:03.800 Yeah, it treats the entire parse file as a master container. 115 00:05:04.000 --> 00:05:07.199 But if I ask for the parent of that Beautiful 116 00:05:07.199 --> 00:05:09.199 Soup document object. 117 00:05:08.920 --> 00:05:11.959 Then you finally get a return value of none. The 118 00:05:12.000 --> 00:05:14.879 document object is the absolute root of the universe for 119 00:05:14.920 --> 00:05:18.600 that specific scrape. It has no creator above it all. 120 00:05:18.680 --> 00:05:20.319 Right, So we can climb up and down the tags. 121 00:05:20.600 --> 00:05:23.079 But let's be real, we usually aren't scraping a site 122 00:05:23.120 --> 00:05:26.240 just to map out a bunch of EMPTYHTML tags. 123 00:05:26.279 --> 00:05:27.399 Now nobody does that. 124 00:05:27.519 --> 00:05:31.759 We're hunting for the actual human readable data trapped inside them, 125 00:05:32.199 --> 00:05:35.399 the text. I know. Beautiful Soup has a dot string 126 00:05:35.439 --> 00:05:38.000 property to extract text, but I've heard it can be 127 00:05:38.040 --> 00:05:40.759 incredibly fragile, Like, why does it break so easily? 128 00:05:40.879 --> 00:05:43.839 It breaks because it demands an absolute lack of ambiguity. 129 00:05:44.079 --> 00:05:46.199 What do you mean, Well, the dot string property will 130 00:05:46.240 --> 00:05:49.480 only extract text if the tag contains exactly one string 131 00:05:49.519 --> 00:05:52.439 of text and literally nothing else. Okay, let's say you 132 00:05:52.519 --> 00:05:54.920 have a paragraph tag and inside it is a solid 133 00:05:54.920 --> 00:05:57.759 block of text. Dot string works perfectly. But what if 134 00:05:57.759 --> 00:05:59.879 the developer bolted a single word in the middle of 135 00:05:59.879 --> 00:06:00.639 that paragraph? 136 00:06:00.959 --> 00:06:04.000 Oh, I see the problem. The moment they add a 137 00:06:04.000 --> 00:06:07.160 bold tag, the paragraph is no longer just holding a 138 00:06:07.199 --> 00:06:11.079 single string. It's holding a string of normal text than 139 00:06:11.120 --> 00:06:13.879 a nested bowld tag holding its own string and then 140 00:06:14.079 --> 00:06:14.920 more normal text. 141 00:06:15.160 --> 00:06:17.639 Exactly So, when you ask beautiful Soup for the dot 142 00:06:17.720 --> 00:06:20.959 string of that paragraph, it looks the nested chaos gets 143 00:06:20.959 --> 00:06:23.560 confused about which specific piece of text you want, and 144 00:06:23.639 --> 00:06:25.720 instead of guessing, it just gives up. 145 00:06:25.879 --> 00:06:27.160 It just returns none. 146 00:06:27.279 --> 00:06:29.680 Yeah, it completely refuses to assume your intent. 147 00:06:30.120 --> 00:06:33.079 That feels like a nightmare for real world scraping. Honestly, 148 00:06:33.839 --> 00:06:38.040 because web code is incredibly messy, developers are constantly throwing 149 00:06:38.079 --> 00:06:41.839 in spans, links, and formatting tags. So how do we 150 00:06:41.920 --> 00:06:44.480 force it to just give us all the text regardless 151 00:06:44.480 --> 00:06:45.160 of the formatting. 152 00:06:45.439 --> 00:06:49.120 You use the dot strings property plural okay. This tells 153 00:06:49.160 --> 00:06:52.399 the parser to recursively dive into the tag, ignore all 154 00:06:52.439 --> 00:06:56.279 the structural HTML boundaries, and yield every single piece of 155 00:06:56.319 --> 00:06:58.040 text it finds, piece by piece. 156 00:06:58.240 --> 00:07:00.959 Well wait, if it's blindly grabbing every string, isn't it 157 00:07:00.959 --> 00:07:04.160 going to pull in all the invisible junk Like developers 158 00:07:04.240 --> 00:07:07.199 use cabs, carriage returns, and empty spaces just to format 159 00:07:07.279 --> 00:07:10.319 their code so it looks pretty in their editor. Beautiful 160 00:07:10.399 --> 00:07:12.519 soup is going to see all that white space as 161 00:07:12.639 --> 00:07:14.000 valid text strings. 162 00:07:13.680 --> 00:07:16.439 Right, It absolutely does. If you just use dot strings, 163 00:07:16.519 --> 00:07:19.800 your data set will be polluted with thousands of useless 164 00:07:19.839 --> 00:07:20.399 new line. 165 00:07:20.279 --> 00:07:23.199 Characters, which is awful for data analysis. 166 00:07:22.560 --> 00:07:26.480 Which brings us to the ultimate cleanup tool, dot stripped 167 00:07:26.560 --> 00:07:30.639 underscore strings. This property does all the heavy lifting for you. 168 00:07:30.879 --> 00:07:33.800 It extracts all the text, but before it hands it back, 169 00:07:33.920 --> 00:07:36.279 it runs an algorithm to strip away all the leading 170 00:07:36.279 --> 00:07:37.639 and trailing white space. 171 00:07:37.439 --> 00:07:39.439 So it sanitizes it on the fly. 172 00:07:39.759 --> 00:07:42.399 Right, And if a string consists entirely of empty space, 173 00:07:42.439 --> 00:07:45.879 it just deletes it entirely, leaving you with raw, clean. 174 00:07:45.680 --> 00:07:49.519 Data that is really powerful. Okay, so moving vertically makes 175 00:07:49.560 --> 00:07:53.199 sense for hierarchical data. But let's pivot because vertical navigation 176 00:07:53.240 --> 00:07:55.560 has a massive blind spot. It does. What if I 177 00:07:55.600 --> 00:07:58.839 am scraping a massive financial table. All the stock tickers 178 00:07:58.879 --> 00:08:01.560 and prices are sitting side by side in individual cells. 179 00:08:02.000 --> 00:08:04.319 If I only move vertically, I hit a dead end 180 00:08:04.360 --> 00:08:05.079 at the first cell. 181 00:08:05.439 --> 00:08:08.399 Yeah, that's the limitation of the bloodline model. To master 182 00:08:08.519 --> 00:08:11.160 this landscape, you have to learn how to move horizontally. 183 00:08:11.600 --> 00:08:14.439 In the structural parse tree, elements that exist at the 184 00:08:14.480 --> 00:08:17.079 exact same level of nesting are called siblings. 185 00:08:17.399 --> 00:08:20.439 Okay, let's visualize this so we don't get lost. Imagine 186 00:08:20.480 --> 00:08:23.720 a developer writes a single anchor link. Sure, inside that 187 00:08:23.800 --> 00:08:27.879 link they put three distinct elements in a row. First, 188 00:08:28.439 --> 00:08:31.439 a bold tag with the word tom next to it, in, 189 00:08:31.519 --> 00:08:34.600 a talic tag with the word Jerry, and finally a 190 00:08:34.679 --> 00:08:37.519 standard text string that says iconic paar. 191 00:08:37.919 --> 00:08:40.039 Right. So, in the memory of the parser that bold 192 00:08:40.080 --> 00:08:42.879 tag for Tom, the italic tag for Jerry, and the 193 00:08:43.000 --> 00:08:46.679 raw text for iconic pair. They are all immediate children 194 00:08:46.720 --> 00:08:47.559 of the anchor. 195 00:08:47.320 --> 00:08:49.559 Link because they're side by side exactly. 196 00:08:49.600 --> 00:08:52.039 They share the same parent, which means they are siblings. 197 00:08:52.120 --> 00:08:54.120 So if my code is currently parked on the bold 198 00:08:54.120 --> 00:08:56.240 tag for Tom, I don't want to go up to 199 00:08:56.240 --> 00:08:58.360 the parent or down into the text. I just call 200 00:08:58.440 --> 00:09:01.519 the dot next sibling command and it hops perfectly sideways 201 00:09:01.559 --> 00:09:02.919 to the italic tag for Jerry. 202 00:09:03.120 --> 00:09:05.919 Yes, and you can shame these commands or use dot 203 00:09:05.960 --> 00:09:09.879 previous sibling to reverse course. But this is where the 204 00:09:09.919 --> 00:09:13.840 mental model gets really tricky, because beautiful Soup also gives 205 00:09:13.840 --> 00:09:17.639 you commands called dot next element and dot previous element. 206 00:09:17.720 --> 00:09:19.679 I'm going to push back on that right now. I'm 207 00:09:19.720 --> 00:09:22.320 looking at the conceptual model in my head, and isn't 208 00:09:22.320 --> 00:09:24.960 as sibling the exact same thing as the next element? 209 00:09:25.240 --> 00:09:25.759 Not quite? 210 00:09:26.200 --> 00:09:28.679 I mean, if Jerry is next to Tom, then Jerry's 211 00:09:28.720 --> 00:09:31.200 the next element. Why do we need a separate command 212 00:09:31.279 --> 00:09:33.320 for this? It feels completely redundant. 213 00:09:33.399 --> 00:09:36.320 It totally feels redundant until you understand how a parser 214 00:09:36.480 --> 00:09:39.960 actually reads a file into memory. This is the difference 215 00:09:39.960 --> 00:09:42.919 between structural geography and chronological sequence. 216 00:09:42.960 --> 00:09:44.279 Okay, explain that well. 217 00:09:44.120 --> 00:09:47.000 Dot next sibling respects the geography of the family tree. 218 00:09:47.080 --> 00:09:49.759 It stays on the exact same horizontal plane, hopping from 219 00:09:49.759 --> 00:09:50.759 container to container. 220 00:09:50.919 --> 00:09:51.120 Right. 221 00:09:51.399 --> 00:09:54.679 But Dot next element doesn't care about the family tree 222 00:09:54.679 --> 00:09:58.559 at all. It reads the raw HTML code like a book, 223 00:09:58.879 --> 00:10:01.279 line by line exactly in the sequence. 224 00:10:01.320 --> 00:10:04.120 The developer typed it, Oh, let me process that. So 225 00:10:04.240 --> 00:10:06.440 if I am on the bold tag for Tom, the 226 00:10:06.440 --> 00:10:10.200 next structural sibling is the italic tag for Jerry. But 227 00:10:10.240 --> 00:10:13.120 if I ask for the next element, you're saying, I 228 00:10:13.120 --> 00:10:14.759 don't go to Jerry. Where do I go? 229 00:10:15.279 --> 00:10:18.399 You go down? Remember, the bold tag itself isn't the text, 230 00:10:18.440 --> 00:10:21.960 It's just a container. When the parser reads the code chronologically, 231 00:10:22.440 --> 00:10:25.799 it encounters the opening boldtag first what is the very 232 00:10:25.840 --> 00:10:28.120 next thing, and encounters in the code the actual word 233 00:10:28.200 --> 00:10:32.080 Tom exactly. So Dot next element dives into the tag 234 00:10:32.120 --> 00:10:35.320 to grab the text itself. What beautiful Soup calls the 235 00:10:35.399 --> 00:10:36.600 navigable string node. 236 00:10:36.840 --> 00:10:37.440 Oh wow. 237 00:10:37.759 --> 00:10:40.440 The text inside the tag is treated as its own 238 00:10:40.519 --> 00:10:44.279 distinct entity in memory. So from the boldtag, Dot next 239 00:10:44.360 --> 00:10:45.960 element drops down to the text Tom. 240 00:10:46.039 --> 00:10:47.360 And if you call it again. 241 00:10:47.159 --> 00:10:49.480 If you call dot next element again. It moves up 242 00:10:49.519 --> 00:10:51.519 and out of the bold tag and finally hits the 243 00:10:51.559 --> 00:10:55.879 italic tag, then the text Jerry, it is a strict 244 00:10:56.000 --> 00:10:58.840 chronological breadcrumb trail of how the document was written. 245 00:10:59.080 --> 00:11:03.639 That is a phenomenal stinction siblings for horizontal structure, elements 246 00:11:03.639 --> 00:11:07.600 for strict chronological reading order. But as brilliant as all 247 00:11:07.600 --> 00:11:10.639 this manual navigation is, we really have to acknowledge a 248 00:11:10.639 --> 00:11:14.440 harsh reality here. Yes, scale, If I am scraping a 249 00:11:14.480 --> 00:11:19.240 fifty thousand line Amazon product page, writing a script that 250 00:11:19.320 --> 00:11:22.320 steps sibling by sibling through the tree is going to 251 00:11:22.320 --> 00:11:25.960 be incredibly slow and fragile. I mean, if Amazon changes 252 00:11:26.000 --> 00:11:28.159 a single divtag, my whole script breaks. 253 00:11:28.240 --> 00:11:31.879 Yeah, manual navigation is necessary for highly localized extraction, but 254 00:11:32.039 --> 00:11:34.679 it is an absolute nightmare at scale. You don't want 255 00:11:34.679 --> 00:11:37.000 to walk. You want to teleport directly to the data 256 00:11:37.039 --> 00:11:37.600 you care about. 257 00:11:37.679 --> 00:11:38.919 Teleporting sounds much. 258 00:11:38.799 --> 00:11:42.960 Better, and that introduces the core filtering engine of Beautiful Soup. 259 00:11:43.440 --> 00:11:45.360 Dot find and dot find all. 260 00:11:45.480 --> 00:11:47.879 These are the heavy hitters. If I use dot find, 261 00:11:47.960 --> 00:11:50.200 the parser scans the document and hands me the very 262 00:11:50.200 --> 00:11:53.639 first match it sees, then stops right. But dot find 263 00:11:53.720 --> 00:11:56.440 all is like a vacuum. It scans the entire tree, 264 00:11:56.559 --> 00:11:59.799 gathers every single instance that matches my criteria, and hands 265 00:11:59.799 --> 00:12:01.279 them back to me in a Python list. 266 00:12:01.480 --> 00:12:03.720 And the flexibility of these filters is really what makes 267 00:12:03.759 --> 00:12:06.399 the library and industry standard. I mean, the simplest filter 268 00:12:06.519 --> 00:12:08.600 is just by HTML tag. You pass the string p 269 00:12:09.039 --> 00:12:11.759 into dot find all and instantly you have a list 270 00:12:11.759 --> 00:12:13.440 of every single paragraph on the page. 271 00:12:13.519 --> 00:12:17.240 But paragraphs are generic. We usually want something highly specific. 272 00:12:17.559 --> 00:12:19.080 I can filter by attributes, right. 273 00:12:19.120 --> 00:12:19.759 You definitely can. 274 00:12:19.919 --> 00:12:21.679 Like if I only want profile pictures, I don't want 275 00:12:21.720 --> 00:12:24.320 all images, I can tell dot find all to look 276 00:12:24.360 --> 00:12:28.279 for an image tag where the src attributes specifically equals 277 00:12:28.320 --> 00:12:31.399 profile dot jpg. Or if I want to cast a 278 00:12:31.399 --> 00:12:34.120 wider net, I can say, hey, find me all anchor 279 00:12:34.159 --> 00:12:38.399 tags where an href attribute simply exists regardless of where 280 00:12:38.399 --> 00:12:41.720 it points. You just pass in href equals true. 281 00:12:41.759 --> 00:12:45.639 That boolean logic is incredibly useful for stripping out deadlinks. 282 00:12:46.159 --> 00:12:48.360 But we have to talk about the most notorious trap 283 00:12:48.399 --> 00:12:52.039 in beautiful soup filtering. It involves CSS classes. 284 00:12:52.159 --> 00:12:54.720 Oh man, here's where it gets really interesting. This is 285 00:12:54.759 --> 00:12:57.159 like a rite of passage for every developer. 286 00:12:57.279 --> 00:12:58.200 It really is, and. 287 00:12:58.159 --> 00:13:02.240 Webdev developers constantly use classes to style their tags. A 288 00:13:02.279 --> 00:13:05.519 warning banner might have a class called alert. But if 289 00:13:05.519 --> 00:13:09.240 you write your Python filter and literally type class equals alert, 290 00:13:09.639 --> 00:13:13.000 the entire program catastrophically crashes before it even runs. Yeah, 291 00:13:13.039 --> 00:13:15.240 why does Python lose its mind over the word. 292 00:13:15.039 --> 00:13:17.960 Class Because you are fighting the core architecture of the 293 00:13:17.960 --> 00:13:23.000 Python language itself. In Python, the word class is a reserved, 294 00:13:23.360 --> 00:13:25.399 deeply protected keywords. 295 00:13:25.080 --> 00:13:27.080 Used for object oriented programming. 296 00:13:26.720 --> 00:13:30.360 Right exactly. It's the command used to define object blueprints. 297 00:13:30.879 --> 00:13:34.159 So when you type class equals alert as an argument 298 00:13:34.200 --> 00:13:37.759 in your filter, the Python interpreter intercepts it before beautiful 299 00:13:37.799 --> 00:13:39.600 Soup even gets a chance to see it. 300 00:13:39.600 --> 00:13:40.919 It thinks you're writing a new class. 301 00:13:41.039 --> 00:13:44.000 Python thinks you are trying to invent a new object blueprint. 302 00:13:44.120 --> 00:13:47.039 Right there in the function call realizes the syntax makes 303 00:13:47.080 --> 00:13:48.960 no sense and throws a fatal error. 304 00:13:49.159 --> 00:13:52.240 It's a classic names based collision. So to get around it, 305 00:13:52.360 --> 00:13:56.159 Beautiful Soup implemented this tiny, invisible. 306 00:13:55.679 --> 00:13:57.080 Hack the underscore. 307 00:13:57.159 --> 00:13:59.399 Yeah, you have to use class underscore with an underscore 308 00:13:59.399 --> 00:14:02.480 at the end. That trailing underscore tells pipeon, Hey, this 309 00:14:02.519 --> 00:14:04.320 is just a normal argument, Ignore. 310 00:14:04.039 --> 00:14:06.480 It, and beautiful soup swips the underscore off in the 311 00:14:06.480 --> 00:14:09.879 background and searches the HTML correctly. If you don't know 312 00:14:09.919 --> 00:14:13.240 that quirk, you will spend hours staring at syntax errors. 313 00:14:13.480 --> 00:14:17.320 It is the ultimate invisible hurdle. But let's push the 314 00:14:17.360 --> 00:14:21.960 complexity further. What if you are dealing with dynamic, unpredictable data. 315 00:14:22.480 --> 00:14:24.879 You don't know the exact class name. You just know 316 00:14:24.919 --> 00:14:27.039 that all the data you want is trapped in tags 317 00:14:27.080 --> 00:14:29.679 where the ID attribute happens to contain a string of 318 00:14:29.759 --> 00:14:30.559 random numbers. 319 00:14:30.639 --> 00:14:32.519 You can't search for an exact match then. 320 00:14:32.480 --> 00:14:34.279 Right, So that is when you bring in the heavy 321 00:14:34.399 --> 00:14:38.639 artillery regular expressions or rejects. Beautiful Soup doesn't force you 322 00:14:38.679 --> 00:14:41.799 to use exact strings. You can actually pass a compiled 323 00:14:41.840 --> 00:14:44.240 rejects pattern directly into dot find all. 324 00:14:44.399 --> 00:14:47.279 The integration is seamless. You can write a reject using 325 00:14:47.320 --> 00:14:50.519 the carrot symbol and a P like carrot P, and 326 00:14:50.639 --> 00:14:53.840 the filter will dynamically gather any tag that starts with 327 00:14:53.840 --> 00:14:57.240 the letter P, paragraphs, pictures, whatever, wow. Or you can 328 00:14:57.360 --> 00:15:00.480 use the rejects backslash key pattern and it will instantly 329 00:15:00.480 --> 00:15:04.399 sweep the document for any element whose ID contains a digit. 330 00:15:04.720 --> 00:15:07.960 Okay, I'm going to push back again. Though, rejects is 331 00:15:08.080 --> 00:15:12.559 incredibly powerful for pattern matching. But sometimes data isn't hidden 332 00:15:12.600 --> 00:15:17.000 behind a pattern, it's hidden behind bizarre, hyper specific business logic. 333 00:15:17.120 --> 00:15:17.440 That's true. 334 00:15:17.559 --> 00:15:20.080 What if I want to find all image tags that 335 00:15:20.200 --> 00:15:23.840 have a source file, but for some inexplicable reason, they 336 00:15:23.879 --> 00:15:27.840 explicitly do not have an old text attribute. Rejects can't 337 00:15:27.879 --> 00:15:32.000 really express the presence of x and the absolute absence 338 00:15:32.039 --> 00:15:34.759 of why in an HTML parse note easily. 339 00:15:34.879 --> 00:15:38.039 This is where beautiful Soup transitions from just a simple 340 00:15:38.120 --> 00:15:42.480 library into a full scale data engineering platform. Also, when 341 00:15:42.519 --> 00:15:46.360 rejects fails, you can write a bespoke Python function with loops, 342 00:15:46.440 --> 00:15:50.000 conditionals any logic you want and pass that entire function 343 00:15:50.120 --> 00:15:51.799 directly into dot find all. 344 00:15:52.159 --> 00:15:54.799 Wait, I can pass a function as the Silter criteria. 345 00:15:54.919 --> 00:15:56.399 How does that even work? Under the hood? 346 00:15:56.559 --> 00:15:58.639 When you pass a custom function to dot find all, 347 00:15:58.799 --> 00:16:01.960 beautiful soup takes every single tag in the entire document 348 00:16:02.039 --> 00:16:04.240 and feeds it into your function one by one. 349 00:16:04.200 --> 00:16:05.279 Just iterates through them all. 350 00:16:05.399 --> 00:16:09.519 Yeah, your function inspects the tag. If your function returns true, 351 00:16:09.679 --> 00:16:12.840 beautiful sap keeps the tag. If it returns false, it 352 00:16:12.879 --> 00:16:13.519 throws it away. 353 00:16:13.720 --> 00:16:15.159 That sounds computationally heavy. 354 00:16:15.240 --> 00:16:18.279 Oh, it is highly computationally expensive because Python has to 355 00:16:18.320 --> 00:16:21.120 execute logic on every single node. But the tradeoff is 356 00:16:21.120 --> 00:16:22.039 surgical precision. 357 00:16:22.159 --> 00:16:23.799 Give me an example of when you'd use that. 358 00:16:24.159 --> 00:16:27.799 Let's say you are executing a cybersecurity audit on a 359 00:16:27.840 --> 00:16:32.000 massive corporate wiki. You need to find all outgoing links 360 00:16:32.000 --> 00:16:36.159 that specifically do not point to a secure internal domain. Okay, 361 00:16:36.279 --> 00:16:38.759 you write a function that looks at the h fref attribute, 362 00:16:39.080 --> 00:16:42.960 parses the URL string, checks against a list of approved domains, 363 00:16:43.159 --> 00:16:46.360 and only returns true if it's an unapproved external link. 364 00:16:46.799 --> 00:16:50.919 You're executing complex, multi stage logic across millions of lines 365 00:16:50.960 --> 00:16:51.519 of code. 366 00:16:51.799 --> 00:16:54.360 So we have built the mental model. Yeah, we can 367 00:16:54.480 --> 00:16:57.919 navigate vertically and horizontally, and we can teleport and filter 368 00:16:58.200 --> 00:17:01.679 using rajax and custom functions. Let's synthesize all of this 369 00:17:01.720 --> 00:17:04.759 into a high stakes practical scenario. We need to rip 370 00:17:04.839 --> 00:17:07.559 every single link off a live, massive web page. 371 00:17:07.680 --> 00:17:10.079 This is the exact scenario you face when building an 372 00:17:10.119 --> 00:17:13.559 automated web crawler. You pull down the raw HTML, parse 373 00:17:13.599 --> 00:17:16.000 it into the beautiful soup tree, and use find all 374 00:17:16.119 --> 00:17:17.880 to isolate every single anchor tag. 375 00:17:18.079 --> 00:17:20.599 But the filter hands me the entire HTML element. I 376 00:17:20.640 --> 00:17:22.880 don't want the bracket nonsense. I just want the raw 377 00:17:23.079 --> 00:17:25.720 URL string trapped inside it. So I know. I can 378 00:17:25.799 --> 00:17:28.799 treat the isolated tag almost exactly like a Python dictionary. 379 00:17:28.880 --> 00:17:31.799 I use square brackets. Open a quote type h ref, 380 00:17:31.880 --> 00:17:33.440 close the quote, and close the bracket. 381 00:17:33.680 --> 00:17:37.039 YEP. That instantly extracts the raw web a drafts. 382 00:17:36.759 --> 00:17:39.880 That extracts the value. But the moment you start extracting 383 00:17:39.880 --> 00:17:43.359 those URLs at scale, you run into the most dangerous 384 00:17:43.440 --> 00:17:45.480 threat to an automated crawler. 385 00:17:45.720 --> 00:17:48.400 The difference between absolute and relative links. 386 00:17:48.519 --> 00:17:51.839 Oh, relative links are absolute poison for a scraper. 387 00:17:51.880 --> 00:17:52.480 They really are. 388 00:17:52.640 --> 00:17:54.640 If I pull an absolute link, it has the whole 389 00:17:54.680 --> 00:17:58.960 package right, the HTTPS, the main domain, the exact filepath. 390 00:17:59.359 --> 00:18:01.880 My scraper just ping that URL and keep moving. 391 00:18:02.119 --> 00:18:05.480 But a relative link is just a lazy internal shortcut. 392 00:18:05.759 --> 00:18:09.039 The developer just writes slash log in or slash images, 393 00:18:09.079 --> 00:18:10.359 slash logo, dot. 394 00:18:10.240 --> 00:18:13.319 P and e right because the browser understands it exactly. 395 00:18:13.359 --> 00:18:15.880 The browser knows what website you are currently looking at. 396 00:18:16.200 --> 00:18:19.480 But your Python scraper is operating in a vacuum exactly. 397 00:18:19.519 --> 00:18:22.119 If my scraper isolates slash log in and tries to 398 00:18:22.160 --> 00:18:25.319 send a request to that literal string. It crashes instantly 399 00:18:25.359 --> 00:18:27.279 because it has no idea what server on the Internet 400 00:18:27.359 --> 00:18:30.519 to talk to. So how do we programmatically sanitize this 401 00:18:30.599 --> 00:18:31.480