WEBVTT 1 00:00:00.120 --> 00:00:05.639 Imagine having just a highly specific digital superpower, not like 2 00:00:06.280 --> 00:00:10.839 flying or invisibility, but the ability to instantly read, organize, 3 00:00:10.839 --> 00:00:14.720 and extract exactly what you want from any web page 4 00:00:14.720 --> 00:00:15.640 in the entire world. 5 00:00:15.679 --> 00:00:17.600 Oh yeah, that would be I mean that really is 6 00:00:17.600 --> 00:00:18.519 a superpower, right. 7 00:00:18.559 --> 00:00:21.280 You just snap your fingers and you perfectly filter out 8 00:00:21.399 --> 00:00:24.600 all the endless mountains of noise, you know, the pop ups, 9 00:00:24.600 --> 00:00:28.359 the autoplaying ads, the chaotic formatting, the tracking scripts, all 10 00:00:28.399 --> 00:00:31.480 of it, and you instantly harvest only the pure data 11 00:00:31.559 --> 00:00:32.560 you actually care about. 12 00:00:32.520 --> 00:00:35.640 It completely changes everything about how you interact with the Internet. 13 00:00:36.000 --> 00:00:37.719 I mean, think about it. We spend hours of our 14 00:00:37.719 --> 00:00:42.159 lives scrolling, manually searching for specific pieces of information across 15 00:00:42.240 --> 00:00:44.119 these chaotic digital environments. 16 00:00:44.159 --> 00:00:46.840 Oh absolutely, so much time just copying and pasting. 17 00:00:46.679 --> 00:00:49.399 Exactly, So having a computer do all the heavy lifting 18 00:00:49.399 --> 00:00:52.920 of reading, interpreting, and organizing a website for you it 19 00:00:52.960 --> 00:00:56.240 isn't just convenient. It's a fund mental shift in how 20 00:00:56.320 --> 00:00:59.840 you leverage the Web's underlying data. It basically turns the 21 00:01:00.119 --> 00:01:02.240 highre Internet into your own personal database. 22 00:01:02.600 --> 00:01:05.480 And that is exactly what we're getting into today on 23 00:01:05.560 --> 00:01:07.840 this deep dive. If you ever write clicked on a 24 00:01:07.840 --> 00:01:11.640 web page and hit a view source or inspect element. 25 00:01:12.040 --> 00:01:14.879 You know, it looks like an absolute nightmare of brackets and. 26 00:01:14.840 --> 00:01:16.079 Code, just a total mess. 27 00:01:16.159 --> 00:01:19.159 Yeah, okay, let's unpack this. Our mission today is to 28 00:01:19.239 --> 00:01:24.599 explore webscraping, specifically focusing on a famously powerful Python library 29 00:01:24.680 --> 00:01:29.519 called beautiful soup. It essentially acts as a translator between 30 00:01:29.519 --> 00:01:32.959 that messy reality of raw web code and the clean, 31 00:01:33.040 --> 00:01:34.680 structured data we want to extract. 32 00:01:34.920 --> 00:01:38.519 Yeah, and to really understand how beautiful soup accomplishes this translation, 33 00:01:38.640 --> 00:01:40.840 you first have to look at how we fetch that data, 34 00:01:40.920 --> 00:01:43.719 because you know, before we even touch the HTML, we 35 00:01:43.760 --> 00:01:45.359 have to pull it from the server, right, getting the 36 00:01:45.400 --> 00:01:50.120 actual files exactly. Typically, this process starts with the requests library. 37 00:01:49.680 --> 00:01:52.200 And Python, So we're talking about standard GIET requests here. 38 00:01:52.359 --> 00:01:55.640 Yes, exactly, When you type a URL into a web browser, 39 00:01:55.959 --> 00:01:59.439 your browser sends a gt request behind the scenes, downloads 40 00:01:59.439 --> 00:02:02.719 the code, and then renders it into a pretty visual 41 00:02:02.760 --> 00:02:04.599 page with colors and fonts. 42 00:02:04.280 --> 00:02:06.560 And layouts, all the stuff we're used to seeing, right. 43 00:02:07.079 --> 00:02:10.840 But when we scrape, we bypass that visual rendering entirely. 44 00:02:11.400 --> 00:02:14.039 You send a j ET request via Python, and you 45 00:02:14.080 --> 00:02:16.439 are basically knocking on the server's door and saying, hey, 46 00:02:16.800 --> 00:02:19.280 send me the raw text file that makes up this page. 47 00:02:19.360 --> 00:02:22.039 Skip the paint job, just give me the raw materials. 48 00:02:21.680 --> 00:02:25.280 Exactly, and the server responds with the raw HTML document. 49 00:02:25.879 --> 00:02:30.199 But at this stage, before Beautiful Soup even gets involved, 50 00:02:30.680 --> 00:02:33.400 what does that document actually look like? To the computer? 51 00:02:33.719 --> 00:02:37.879 It is just one massive, contiguous string of characters. 52 00:02:37.520 --> 00:02:39.719 Like just a giant block of text. 53 00:02:39.919 --> 00:02:43.039 Yeah, to the computer's memory, it holds zero structural meaning, 54 00:02:43.120 --> 00:02:45.960 it's just a wall of text millions of characters long. 55 00:02:46.000 --> 00:02:47.319 There is no hierarchy yet. 56 00:02:47.479 --> 00:02:50.360 Wow, which brings us to the blueprint of the web. 57 00:02:50.919 --> 00:02:53.080 HTML is supposed to be the language of the Internet. 58 00:02:53.280 --> 00:02:55.400 But Beautiful Soup doesn't just read it left or right 59 00:02:55.439 --> 00:02:56.479 like a normal book, does it. 60 00:02:56.919 --> 00:02:59.439 No, it can't. It builds what's known as a parse tree, 61 00:02:59.479 --> 00:03:03.919 a parse right. When you feed that massive string of 62 00:03:04.080 --> 00:03:07.759 raw HTML into Beautiful Soup, you have to specify a parser, 63 00:03:07.919 --> 00:03:10.800 And a parser is essentially an engine that reads the text, 64 00:03:11.039 --> 00:03:15.039 identifies the HTML tags, and attempts to organize it into 65 00:03:15.080 --> 00:03:17.919 a hierarchical multi dimensional map. 66 00:03:18.159 --> 00:03:19.919 So it's building the structure exactly. 67 00:03:20.199 --> 00:03:23.599 The parse tree gives the document a rigid, logical structure 68 00:03:23.599 --> 00:03:27.560 that a program can actually systematically navigate I see now. 69 00:03:27.639 --> 00:03:31.120 I know lexml's often the parser of choice here, but 70 00:03:31.879 --> 00:03:35.479 why XML specifically? I mean, there were standard HTML parsers 71 00:03:35.479 --> 00:03:38.280 build right into Python itself, right, Why go through the 72 00:03:38.280 --> 00:03:40.199 trouble to install a third party parser? 73 00:03:40.479 --> 00:03:42.680 Well, the built in parsers are completely fine if you 74 00:03:42.719 --> 00:03:45.400 are dealing with perfectly written, mathematically sound. 75 00:03:45.199 --> 00:03:47.719 Code, which, let's be honest, never happens exactly. 76 00:03:48.039 --> 00:03:50.360 The reality of the Internet is that most HTML is 77 00:03:50.400 --> 00:03:53.719 just heavily duct taped together. You'll find unclosed tags, missing 78 00:03:53.759 --> 00:03:58.319 body elements, stray brackets, typos from developers rushing to push updates. 79 00:03:58.360 --> 00:03:59.960 It's a wild last out there, it really is. 80 00:04:00.240 --> 00:04:02.360 And if you use a strict built in parser, it 81 00:04:02.400 --> 00:04:05.199 will hit a missing divtag and just boom, instantly throw 82 00:04:05.240 --> 00:04:07.199 a fatal error and crash your entire script. 83 00:04:07.479 --> 00:04:11.639 Oh brutal. So XML is favored because it's lenient, then 84 00:04:12.240 --> 00:04:15.560 how does it actually handle those errors without crashing? 85 00:04:15.840 --> 00:04:19.839 It actively infers intent. So if LexML is parsing a 86 00:04:19.879 --> 00:04:24.040 page and sees, say, a paragraph tag nested inside a 87 00:04:24.040 --> 00:04:27.040 table row, but the table row was never officially opened 88 00:04:27.079 --> 00:04:29.160 in the code, it doesn't crash. 89 00:04:29.600 --> 00:04:30.639 It just figures it out. 90 00:04:30.759 --> 00:04:34.199 Yeah, it retroactively creates the missing structural elements in the 91 00:04:34.199 --> 00:04:37.040 parse tree to make the hierarchy mathematically sound. 92 00:04:37.319 --> 00:04:38.040 That's amazing. 93 00:04:38.120 --> 00:04:40.720 It's a very forgiving translator. It kind of exposes just 94 00:04:40.759 --> 00:04:43.959 how messy the Internet that's infrastructure really is because it's 95 00:04:43.959 --> 00:04:47.120 building a fully functional tree even out of broken raw materials. 96 00:04:47.560 --> 00:04:50.360 It's like having an editor who fixes your grammar mistakes 97 00:04:50.360 --> 00:04:52.240 on the fly, so the book still makes perfect sense. 98 00:04:52.319 --> 00:04:53.240 That's a great way to put it. 99 00:04:53.319 --> 00:04:56.600 Yeah, And once LexML builds that tree, it's often really 100 00:04:56.639 --> 00:04:59.079 helpful to visualize it. I know. Beautiful Soup has a 101 00:04:59.079 --> 00:05:00.519 function called soup dot pretiphi. 102 00:05:00.639 --> 00:05:02.319 Right, Yes, Pretiphi is fantastic. 103 00:05:02.399 --> 00:05:05.399 It takes this newly structured, autocorrected tree and prints the 104 00:05:05.519 --> 00:05:09.560 HTML back out, but beautifully formatted with proper indentations in 105 00:05:09.639 --> 00:05:12.680 line breaks. It just makes the hierarchy readable for the 106 00:05:12.759 --> 00:05:13.319 human eye. 107 00:05:13.199 --> 00:05:17.160 Again, which is incredibly useful for debugging. You can physically 108 00:05:17.199 --> 00:05:20.720 see the nesting structure before you start writing your extraction code. 109 00:05:20.800 --> 00:05:24.120 Right, I usually visualize this HTML structure like a set 110 00:05:24.120 --> 00:05:25.279 of Russian nesting dolls. 111 00:05:25.360 --> 00:05:27.399 Oh interesting, how so, Well. 112 00:05:27.360 --> 00:05:29.360 Let's say we are looking at a fake e commerce 113 00:05:29.360 --> 00:05:33.639 site like a novelty gift shop. Okay, sure, the biggest 114 00:05:33.639 --> 00:05:37.199 wooden doll on the outside containing the whole visible page 115 00:05:37.480 --> 00:05:40.560 is typically an outer div element. You open that big 116 00:05:40.600 --> 00:05:44.360 dift doll, and inside you find smaller containers, maybe a logo, 117 00:05:44.879 --> 00:05:47.600 a header tag that says total normal gifts, and a 118 00:05:47.639 --> 00:05:49.759 massive table element containing all the products. 119 00:05:49.879 --> 00:05:52.560 Right. The analogy works pretty well for the concept of containment. 120 00:05:52.560 --> 00:05:55.160 But here's where it deviates from physical nesting doll. Okay, 121 00:05:55.240 --> 00:05:58.439 tell me a real nesting doll only ever holds one 122 00:05:58.480 --> 00:06:01.759 smaller doll, which holds one's smaller doll, and so on. True, 123 00:06:01.920 --> 00:06:04.920 But an HTML container like that outer dev you mentioned 124 00:06:05.399 --> 00:06:08.920 might hold five thousand differently shaped boxes at the exact 125 00:06:08.959 --> 00:06:09.560 same level. 126 00:06:09.720 --> 00:06:10.800 Oh wow, And some. 127 00:06:10.680 --> 00:06:13.959 Of those boxes hold text while others hold images, while 128 00:06:14.000 --> 00:06:17.279 others hold hidden scripts. It branches out exponentially. 129 00:06:17.600 --> 00:06:21.639 Fair point. Yeah, it's nesting dolls, but they're multiplying rapidly 130 00:06:21.680 --> 00:06:24.800 at every single level exactly. So if we look specifically 131 00:06:24.839 --> 00:06:27.800 at that table element on our fake gift shop page, 132 00:06:28.360 --> 00:06:31.720 the branching just continues. You open the table tag, find 133 00:06:31.720 --> 00:06:35.120 a t body tag, and inside that you have individual 134 00:06:35.240 --> 00:06:37.920 rows the price tags. Right, the roads and inside each 135 00:06:38.000 --> 00:06:40.879 row you finally hit the table cells the t tags 136 00:06:40.920 --> 00:06:43.160 holding the actual gift names and the prices. 137 00:06:43.360 --> 00:06:46.079 Yeah, and now that we have the entire document loaded 138 00:06:46.079 --> 00:06:50.560 into memory as a navigable, mathematically sound parse tree, the 139 00:06:50.680 --> 00:06:54.040 challenge shifts how So well, the computer sees the structure, 140 00:06:54.079 --> 00:06:56.120 but we need to tell it how to hunt down 141 00:06:56.240 --> 00:06:59.680 specific data points within that sprawling branching tree. 142 00:07:00.120 --> 00:07:03.639 Ah right, because extracting specific tags is great if you 143 00:07:03.680 --> 00:07:05.720 want everything. But if I want the price of a 144 00:07:05.759 --> 00:07:08.399 specific gift, I don't want to open every single branch 145 00:07:08.439 --> 00:07:10.439 on the entire website just to find it. I need 146 00:07:10.480 --> 00:07:11.839 to know how to jump straight to the data I 147 00:07:11.879 --> 00:07:12.439 actually care. 148 00:07:12.360 --> 00:07:15.879 About exactly, and Beautiful Soup provides a few targeted methods 149 00:07:15.879 --> 00:07:19.680 for this. The fastest, most intuitive way is called dot 150 00:07:19.680 --> 00:07:23.680 notation dot notation. Because beautiful Soup has turned the raw 151 00:07:23.839 --> 00:07:28.319 HTML into a Python object, you can access tags directly 152 00:07:28.360 --> 00:07:31.360 as properties of that object. So if you just type 153 00:07:31.399 --> 00:07:34.240 soup dot title, it instantly hands you the title tag. 154 00:07:34.319 --> 00:07:35.000 Oh that's easy. 155 00:07:35.120 --> 00:07:37.519 Yeah, if you type soup dot tr it hands you 156 00:07:37.560 --> 00:07:38.240 a table row. 157 00:07:38.319 --> 00:07:41.360 Wait. Wait, if I type soup dot img to get 158 00:07:41.360 --> 00:07:44.079 an image, which image does it actually give me? I 159 00:07:44.120 --> 00:07:48.079 mean a webpage could have fifty different product images on it. 160 00:07:48.079 --> 00:07:49.800 It gives you the very first one it encounters in 161 00:07:49.839 --> 00:07:51.920 the parstree, and then it immediately stops searching. 162 00:07:52.000 --> 00:07:55.680 Wait, really, that seems like a massive vulnerability. If dot 163 00:07:55.720 --> 00:07:59.519 notation only grabs the first image, isn't that practically useless 164 00:07:59.519 --> 00:08:02.560 for scrape an entire catalog? I mean, the first image 165 00:08:02.560 --> 00:08:04.839 on most sites is just the company logo in the 166 00:08:04.839 --> 00:08:05.839 top navigation bar. 167 00:08:06.160 --> 00:08:07.839 It is a vulnerability if you use it for the 168 00:08:07.879 --> 00:08:11.160 wrong task. Dot notation is really only designed for unique 169 00:08:11.319 --> 00:08:15.000 singleton elements like the page title or the main head tag. 170 00:08:15.480 --> 00:08:18.959 For bulk extraction, relying on it is a classic beginner mistake. 171 00:08:19.040 --> 00:08:20.560 I actually learned this the hard way a few years 172 00:08:20.560 --> 00:08:22.519 back when I first started building automated scrapers. 173 00:08:22.600 --> 00:08:23.959 Oh no, what happened? 174 00:08:23.959 --> 00:08:26.720 Well, I wrote a scraper for a financial data site. 175 00:08:27.240 --> 00:08:29.600 I was using a variation of a basic search to 176 00:08:29.639 --> 00:08:32.399 grab the first table row on the page, which always 177 00:08:32.399 --> 00:08:35.120 contained the daily ticker summary I needed. Makes sense, and 178 00:08:35.159 --> 00:08:39.360 it worked flawlessly for about three months. Then the site 179 00:08:39.399 --> 00:08:44.840 developers quietly updated their HTML template. Oh yeah, they added 180 00:08:44.879 --> 00:08:47.919 a tiny, invisible one by one pixel right at the 181 00:08:47.919 --> 00:08:50.960 top of the body tag for analytics tracking, and they 182 00:08:51.000 --> 00:08:53.440 wrapped it in its own hidden table structure. 183 00:08:53.559 --> 00:08:55.759 Oh no, I see exactly where this is going. 184 00:08:55.960 --> 00:08:59.600 My script ran every morning automatically at four am, so 185 00:08:59.639 --> 00:09:02.879 instead of grabbing the financial summary, the scraper hit that 186 00:09:02.960 --> 00:09:07.120 hidden analytics table row first, stop searching, and downloaded a 187 00:09:07.120 --> 00:09:11.399 bunch of useless metadata attached to an invisible pixel. Oh man, yeah, 188 00:09:11.399 --> 00:09:14.120 I collected absolute garbage data for a week before I 189 00:09:14.200 --> 00:09:15.120 even notice the error. 190 00:09:15.679 --> 00:09:19.039 That is painful, but it perfectly illustrates why we need 191 00:09:19.039 --> 00:09:21.799 something more robust. Yeah, so how did you fix it? 192 00:09:22.039 --> 00:09:24.000 How do you avoid the tracking pixel trap? 193 00:09:24.080 --> 00:09:27.720 By transitioning to explicit search methods For single items, you 194 00:09:27.840 --> 00:09:30.039 use the dot find function. It still returns only the 195 00:09:30.039 --> 00:09:33.039 first match, but it's highly flexible. Flexible hab Well, you 196 00:09:33.080 --> 00:09:35.799 aren't limited to just a generic tag name. You can 197 00:09:35.919 --> 00:09:38.679 use regular expressions to search for structural pattern. 198 00:09:38.759 --> 00:09:42.000 Meaning I can tell dot find to look for an 199 00:09:42.039 --> 00:09:45.480 element that starts with the letter H followed by a digit, 200 00:09:45.799 --> 00:09:47.840 and it will grab the first H one or each 201 00:09:47.879 --> 00:09:50.200 two it hits, just ignoring everything else. 202 00:09:50.320 --> 00:09:53.080 Correct. Or you can tell it to find the first 203 00:09:53.120 --> 00:09:57.600 table row that also possesses a specific CSS class bypassing 204 00:09:57.679 --> 00:09:59.559 those hidden tracking pixels entirely. 205 00:09:59.720 --> 00:10:01.200 That's but for the. 206 00:10:01.120 --> 00:10:04.840 Catalog scraping scenario you mentioned earlier, the true workhorse of 207 00:10:04.840 --> 00:10:06.639 beautiful Soup is dot findal. 208 00:10:06.840 --> 00:10:09.480 Okay, So if we use dot findal on our novelty 209 00:10:09.480 --> 00:10:13.159 gift shop, beautiful Soup scans the entire perse tree from 210 00:10:13.159 --> 00:10:16.480 top to bottom and gathers every single table cell on 211 00:10:16.519 --> 00:10:18.639 the page. Like if there are twenty cells in the 212 00:10:18.679 --> 00:10:20.039 product grid, it just hands you all. 213 00:10:19.919 --> 00:10:23.679 Twenty exactly, and crucially, it returns them as a standard 214 00:10:23.679 --> 00:10:24.480 Python list. 215 00:10:24.639 --> 00:10:25.279 Oh nice. 216 00:10:25.440 --> 00:10:29.440 This is where web scraping seamlessly merges with standard data manipulation. 217 00:10:29.919 --> 00:10:33.320 Because find all outputs a standard iterable object. You just 218 00:10:33.399 --> 00:10:35.600 loop through it using basic Python logic. 219 00:10:35.759 --> 00:10:38.279 Right, just simple for loop. And as we iterate over 220 00:10:38.320 --> 00:10:40.600 that list of table cells, we need to extract the 221 00:10:40.639 --> 00:10:43.600 actual text, right because right now the data is still 222 00:10:43.600 --> 00:10:45.919 wrapped in those HTML brackets. We want the name of 223 00:10:45.960 --> 00:10:48.240 the novelty gif, not the TD tags surrounding it. 224 00:10:48.440 --> 00:10:51.519 Yes, and this is handled by accessing the dot text 225 00:10:51.559 --> 00:10:53.480 property of each element in your loop. 226 00:10:53.639 --> 00:10:54.399 Just dot text. 227 00:10:54.679 --> 00:10:58.320 Yep. When you query dot text, beautiful Soup acts like 228 00:10:58.360 --> 00:11:02.679 a highly aggressive filter. It strips away all surrounding HTML tags. 229 00:11:02.919 --> 00:11:06.639 It removes the brackets, the markers, the inline scripts, everything. 230 00:11:06.720 --> 00:11:08.000 Wow, it really cleans it up. 231 00:11:08.080 --> 00:11:10.720 It does. It can even be configured to strip out 232 00:11:10.720 --> 00:11:13.759 extraneous white space and hidden newlines. It leaves you with 233 00:11:13.799 --> 00:11:15.879 nothing but pure usable data. 234 00:11:15.960 --> 00:11:18.679 So basically, dot find ol acts as the mining equipment 235 00:11:18.720 --> 00:11:21.480 gathering the raw ore, and the text property acts as 236 00:11:21.480 --> 00:11:24.600 the refinery, burning away the rock and handing us pure metal. 237 00:11:24.840 --> 00:11:26.200 That is a perfect analogy. 238 00:11:26.320 --> 00:11:29.279 But here's where it gets really interesting. Elements aren't just 239 00:11:29.360 --> 00:11:33.039 dumb containers holding text. They have metadata, they have hidden instructions. 240 00:11:33.440 --> 00:11:35.679 We need to look at the anatomy of the HTML 241 00:11:35.720 --> 00:11:39.279 tags themselves to understand how much data is actually available 242 00:11:39.360 --> 00:11:39.759 to us. 243 00:11:40.039 --> 00:11:43.720 Absolutely. Up to this point we've treated tags as basic 244 00:11:43.799 --> 00:11:48.879 structural boundaries. In Beautiful Soup, however, HTML elements are treated 245 00:11:48.960 --> 00:11:54.519 as sophisticated objects. Specifically, they are instantiated as tag objects. 246 00:11:54.639 --> 00:11:55.840 Okay, so they're objects. 247 00:11:56.000 --> 00:11:59.000 Yeah, they have properties of their own, completely independent of 248 00:11:59.039 --> 00:11:59.879 the texts they contain. 249 00:12:00.159 --> 00:12:03.240 Let me throw out another analogy to visualize this. If 250 00:12:03.240 --> 00:12:07.679 the tag itself is a physical cardboard box, the attributes 251 00:12:07.919 --> 00:12:10.879 are the shipping labels slapped onto the outside of that box. 252 00:12:10.960 --> 00:12:12.279 Oh I like that framing right. 253 00:12:12.480 --> 00:12:14.759 The shipping labels tell the browser where the box goes, 254 00:12:14.759 --> 00:12:17.159 what category belongs to, or how big it should be drawn, 255 00:12:17.519 --> 00:12:19.960 all without the browser ever having to actually open the 256 00:12:20.000 --> 00:12:21.240 box to see what's inside. 257 00:12:21.279 --> 00:12:24.120 That's exactly it. The box might be an image tag, 258 00:12:24.679 --> 00:12:28.240 but the shipping label the attribute is the CRC attribute. 259 00:12:28.279 --> 00:12:31.559 That attribute contains the actual URL pointing to the server 260 00:12:31.720 --> 00:12:34.080 where the image file is hosted right right. Or the 261 00:12:34.080 --> 00:12:36.879 attribute might be a CSS class that tells the website's 262 00:12:36.879 --> 00:12:39.799 stylesheet to render the box with a red border and 263 00:12:39.840 --> 00:12:40.720 a drop shadow. 264 00:12:40.960 --> 00:12:43.600 So how do we actually read those shipping labels in 265 00:12:43.679 --> 00:12:44.679 our Python script? 266 00:12:44.919 --> 00:12:47.720 Beautiful Soup stores all of a tag's attributes in a 267 00:12:47.759 --> 00:12:51.360 dictionary data structure, So if you've saved an image tag 268 00:12:51.440 --> 00:12:55.279 to a variable, you access its attributes exactly like you 269 00:12:55.320 --> 00:12:57.879 would retrieve a value by its key in a normal 270 00:12:57.919 --> 00:12:59.320 Python dictionary. 271 00:12:59.000 --> 00:13:01.159 Just standard brack notation exactly. 272 00:13:01.559 --> 00:13:05.080 You append square brackets to the variable, put the attribute 273 00:13:05.120 --> 00:13:09.840 name in quotes like src, and beautiful soup retrieves the 274 00:13:09.879 --> 00:13:11.039 destination URL. 275 00:13:11.559 --> 00:13:13.919 Okay, but what if a tag has multiple labels of 276 00:13:13.960 --> 00:13:17.399 the same type. It's pretty common for a single paragraph 277 00:13:17.480 --> 00:13:20.639 tag to have multiple CSS classes attached to it to 278 00:13:20.679 --> 00:13:24.399 handle complex styling, like say, it's class as highlight and 279 00:13:24.440 --> 00:13:27.679 also classed is important. How does a dictionary handle multiple 280 00:13:27.759 --> 00:13:28.879 values for a single key. 281 00:13:29.039 --> 00:13:33.240 It's clever, beautiful soup anticipates that HTML is highly flexible 282 00:13:33.240 --> 00:13:36.720 with multi valued attributes. When you query a multi valued 283 00:13:36.759 --> 00:13:39.440 attribute like a class, it doesn't just concatenate the words 284 00:13:39.440 --> 00:13:40.639 into one messy string. 285 00:13:40.840 --> 00:13:41.679 Oh good, right. 286 00:13:41.799 --> 00:13:44.159 It parses the values and returns them as a cleanly 287 00:13:44.240 --> 00:13:47.240 organized Python list. You get a list containing the string 288 00:13:47.279 --> 00:13:50.200 highlight and the string important, which is perfectly formatted for 289 00:13:50.279 --> 00:13:51.720 conditional logic in your script. 290 00:13:51.960 --> 00:13:54.120 Okay, so we have the box which is the tag object. 291 00:13:54.320 --> 00:13:57.120 We have the shipping labels, which are the attributes. The 292 00:13:57.240 --> 00:14:01.000 last piece of the element's anatomy is the actual item 293 00:14:01.080 --> 00:14:03.600 inside the box, the string of text. Right, earlier, we 294 00:14:03.600 --> 00:14:06.759 talked about extracting the pure data using the dot text property, 295 00:14:07.120 --> 00:14:10.360 but I also saw another property called dot string. What 296 00:14:10.440 --> 00:14:14.039 is the mechanical difference between extracting dot text and extracting 297 00:14:14.080 --> 00:14:14.600 dot string. 298 00:14:14.919 --> 00:14:17.600 The distinction is subtle, but it's critical for how your 299 00:14:17.600 --> 00:14:21.600 script manages memory and navigation. Okay, laid on me when 300 00:14:21.639 --> 00:14:26.600 you called dot text beautiful soup extracts the characters, copies 301 00:14:26.639 --> 00:14:29.200 them into a new block of memory, and hands you 302 00:14:29.360 --> 00:14:33.639 a standard Python string. It is dead text, dead text. Yeah, 303 00:14:33.679 --> 00:14:36.759 it has been entirely severed from the htmil document. You 304 00:14:36.759 --> 00:14:39.120 can analyze it, but you cannot use it to interact 305 00:14:39.159 --> 00:14:40.440 with the web page structure anymore. 306 00:14:40.440 --> 00:14:42.320 So it's isolated data doesn't know where it came from. 307 00:14:42.399 --> 00:14:45.919 Yes, but when you call string beautiful, soup returns a 308 00:14:46.000 --> 00:14:48.440 very specific object called a navigable string. 309 00:14:48.759 --> 00:14:51.840 A navigable string. Why does text need to be navigable? 310 00:14:52.080 --> 00:14:55.440 Because a navigable string operates as a memory pointer. It 311 00:14:55.480 --> 00:14:58.519 retains its reference back to the original parse tree. 312 00:14:58.679 --> 00:14:59.360 Oh icee. 313 00:14:59.440 --> 00:15:02.320 It inherents only knows its position in the hierarchy, knows 314 00:15:02.360 --> 00:15:05.039 which tag is its parent container and which tags are 315 00:15:05.080 --> 00:15:08.720 its siblings. If you possess a navigable string, you can 316 00:15:08.879 --> 00:15:11.840 pivot from the text itself right back into navigating the 317 00:15:11.960 --> 00:15:15.000 HTML structure, which a standard string just cannot do. 318 00:15:15.559 --> 00:15:18.960 That is fascinating. The text itself remains a tethered part 319 00:15:19.000 --> 00:15:22.519 of the ecosystem exactly. Speaking of hidden elements in that ecosystem, 320 00:15:22.759 --> 00:15:26.320 what about developer comments? Normal users never see them, but 321 00:15:26.399 --> 00:15:29.759 developers constantly leave notes for each other hidden directly in 322 00:15:29.799 --> 00:15:30.720 the HTML code. 323 00:15:30.840 --> 00:15:34.399 Oh yeah, beautiful Soup doesn't ignore those comments or blindly 324 00:15:34.440 --> 00:15:35.799 treat them as regular strings. 325 00:15:35.840 --> 00:15:36.080 Either. 326 00:15:36.559 --> 00:15:41.960 It identifies them as a distinct object type, a comment object, which. 327 00:15:41.759 --> 00:15:45.159 Seems like a massive advantage for cybersecurity research or auditing. Right. 328 00:15:45.240 --> 00:15:50.120 Absolutely, developers notoriously leave sensitive information in comments. They'll comment 329 00:15:50.159 --> 00:15:53.399 out old blocks of code that still contain valid API keys, 330 00:15:53.919 --> 00:15:57.159 or leave internal server IP addresses, or you wouldn't believe 331 00:15:57.200 --> 00:16:01.799 it even temporary passwords. That's wilds Beautiful soup isolates comments 332 00:16:01.840 --> 00:16:04.679 as their own object type. You can write a scraper 333 00:16:04.720 --> 00:16:08.440 to bypass all the visible text and purely aggregate every 334 00:16:08.519 --> 00:16:11.360 hidden comment across an entire domain in minutes. 335 00:16:11.600 --> 00:16:15.320 That's so powerful. So we've covered searching for tags directly 336 00:16:15.600 --> 00:16:19.000 using dot find and dot findal. We've extracted attributes and 337 00:16:19.000 --> 00:16:23.840 differentiated between dead text and navigable strings. But consider this scenario. 338 00:16:24.440 --> 00:16:27.159 What if the developers were just lazy. What if we're 339 00:16:27.200 --> 00:16:29.759 dealing with a web page that doesn't use unique classes 340 00:16:29.840 --> 00:16:33.240 or IDs. What if every single piece of data is 341 00:16:33.320 --> 00:16:37.000 just wrapped in generic paragraph tags. How do we locate 342 00:16:37.039 --> 00:16:40.360 our target data when direct searching is basically impossible. 343 00:16:40.480 --> 00:16:43.840 This is exactly where direct searching fails and relational navigation 344 00:16:43.960 --> 00:16:47.120 traversal becomes essential traversal. Right, you don't find the data 345 00:16:47.120 --> 00:16:48.960 by its name, you find it by its neighborhood. 346 00:16:49.159 --> 00:16:51.159 Oh, it's like knowing you on a specific house, but 347 00:16:51.240 --> 00:16:53.879 none of the houses on the street have numbers. However, 348 00:16:54.320 --> 00:16:56.360 you know the house you want is always exactly two 349 00:16:56.399 --> 00:16:58.000 doors down from the post office. 350 00:16:58.120 --> 00:17:02.720 That's the exact logic. Beautiful soup allows directional traversal through 351 00:17:02.759 --> 00:17:05.640 the parse tree. You can move up the hierarchy to 352 00:17:05.839 --> 00:17:08.920 parent and ancestor nodes, you can move down the tree 353 00:17:09.039 --> 00:17:13.599 to children and descendant nodes, and perhaps most powerfully, you 354 00:17:13.599 --> 00:17:16.319 can move sideways through sibling nodes. 355 00:17:16.559 --> 00:17:20.039 Moving sideways across siblings feels like a total game changer 356 00:17:20.039 --> 00:17:21.039 for generic code. 357 00:17:21.160 --> 00:17:24.960 It really is incredibly efficient. Sibling traversal allows you to 358 00:17:25.000 --> 00:17:27.759 move left and right across elements that share the exact 359 00:17:27.799 --> 00:17:30.519 same level of hierarchy and the same parent container. 360 00:17:31.000 --> 00:17:31.960 You give me an example. 361 00:17:32.039 --> 00:17:35.119 Sure, let's say you're scraping a wiki article. The data 362 00:17:35.119 --> 00:17:38.440 you want is in an unnamed generic paragraph, But you 363 00:17:38.480 --> 00:17:41.119 know that paragraph always appears immediately after an H two 364 00:17:41.160 --> 00:17:42.680 header that says early life. 365 00:17:42.799 --> 00:17:45.160 Okay, so the header is my post office. 366 00:17:44.799 --> 00:17:46.920 In this scenario, exactly, you don't need to search for 367 00:17:46.960 --> 00:17:49.119 the paragraph. You find the unique header and then simply 368 00:17:49.119 --> 00:17:52.400 command beautiful soup to select its stet next sibling. 369 00:17:52.640 --> 00:17:54.920 You just slide right over to the data by passing 370 00:17:55.000 --> 00:17:59.519 the generic tags entirely. That's brilliant. What about moving downwards into. 371 00:17:59.400 --> 00:18:03.079 The tags and to inspect a tag's immediate contents you 372 00:18:03.240 --> 00:18:06.440 use the dot content's property. If you select a top 373 00:18:06.519 --> 00:18:09.599 level tag like the head tag and query its dot contents, 374 00:18:09.799 --> 00:18:13.799 beautiful soup returns a list of everything nested directly inside. 375 00:18:13.480 --> 00:18:16.519 It, so like the title tag, meta tags. 376 00:18:16.319 --> 00:18:19.000 Yep, and even the literal line breaks that exist in 377 00:18:19.039 --> 00:18:20.680 the raw HTML formatting. 378 00:18:20.839 --> 00:18:24.079 Because it returns a list, I can use standard Python 379 00:18:24.160 --> 00:18:27.799 index numbers to grab a specific child right index zero 380 00:18:27.839 --> 00:18:30.119 gives me the first child, index one gives me the second, 381 00:18:30.119 --> 00:18:30.519 and so on. 382 00:18:30.680 --> 00:18:34.240 Precisely. Let's trace a real world extraction using that exact logic. 383 00:18:34.359 --> 00:18:34.759 Let's do it. 384 00:18:34.799 --> 00:18:37.559 Imagine you're scraping a blog. You use dot notation or 385 00:18:37.599 --> 00:18:41.559 dot find to locate a specific paragraph tag. You query 386 00:18:41.599 --> 00:18:44.119 dot contents to look inside it. You see the second 387 00:18:44.200 --> 00:18:46.759 child in the lists, an anchor link and a tag 388 00:18:46.799 --> 00:18:49.319 pointing to an external site. Right, you select that a 389 00:18:49.400 --> 00:18:52.519 tag using its index number, query its attributes as a dictionary, 390 00:18:52.640 --> 00:18:54.680 and extract the h ref url. 391 00:18:55.119 --> 00:18:57.200 That makes total sense, But I do have a mechanical 392 00:18:57.279 --> 00:19:00.920 question regarding traversing downwards. If I can use dot contents 393 00:19:01.359 --> 00:19:04.599 to open a tag and see its children, and then 394 00:19:04.720 --> 00:19:07.359 use dot contents on one of those children to see 395 00:19:07.359 --> 00:19:11.359 its children, can you just keep chaining dot contents indefinitely 396 00:19:11.480 --> 00:19:13.519 until I reach the very bottom of the text. 397 00:19:13.680 --> 00:19:16.720 You cannot, and attempting to do so is a trap 398 00:19:16.759 --> 00:19:20.279 that crashes a lot of scraper scripts. Really, why it 399 00:19:20.359 --> 00:19:23.359 loops back to our discussion on the navigable. 400 00:19:22.880 --> 00:19:26.039 String ah the text that remembers where it came from. 401 00:19:26.160 --> 00:19:29.559