WEBVTT 1 00:00:00.120 --> 00:00:03.759 Imagine the scenario. Okay, you are sitting at your computer, 2 00:00:04.120 --> 00:00:06.919 just staring at the countdown timer for tickets to see 3 00:00:06.919 --> 00:00:08.240 your absolute favorite band. 4 00:00:08.400 --> 00:00:10.919 Oh I know this feeling right, Yeah. 5 00:00:10.599 --> 00:00:14.039 The tension is real. The clock strike zero, the button 6 00:00:14.080 --> 00:00:17.679 turns green, and you click add to Kart in like 7 00:00:17.719 --> 00:00:20.359 a fraction of a second, yeah, only to see a 8 00:00:20.399 --> 00:00:24.239 message top up that the event is completely sold out. 9 00:00:24.359 --> 00:00:25.839 Heartbreaking it is. 10 00:00:26.160 --> 00:00:28.760 And you're sitting there thinking, how did that happen? I 11 00:00:28.839 --> 00:00:31.760 was lightning fast? Well, large truth is you were beaten 12 00:00:31.800 --> 00:00:35.000 by a machine. Yeah, a bot exactly, A bot that 13 00:00:35.079 --> 00:00:37.039 didn't have to wait for your web browser to render 14 00:00:37.039 --> 00:00:40.079 the colors or the images or the fancy layout. It 15 00:00:40.240 --> 00:00:43.799 just bypassed all that visual stuff and read the invisible 16 00:00:43.799 --> 00:00:44.560 scaffolding of. 17 00:00:44.479 --> 00:00:47.600 The internet, right, Because to a machine, it's a completely 18 00:00:47.640 --> 00:00:50.039 different way of seeing the web. We look at websites 19 00:00:50.039 --> 00:00:55.000 as these beautiful, glossy magazines, but underneath, I mean, every 20 00:00:55.119 --> 00:00:58.119 single piece of data, every image, every price tag, it's 21 00:00:58.159 --> 00:01:01.479 all trapped within this massive nest web of structural code. 22 00:01:01.520 --> 00:01:04.480 And it instantly found the exact line of code corresponding 23 00:01:04.519 --> 00:01:07.599 to that buy button, just triggered it exactly. 24 00:01:07.680 --> 00:01:10.239 And if you want to automate tasks or gather data 25 00:01:10.560 --> 00:01:13.519 or you know, build your own tools, you really have 26 00:01:13.599 --> 00:01:15.359 to learn how to read that matrix. 27 00:01:15.640 --> 00:01:18.439 And that is exactly the mission of our deep dive today. 28 00:01:18.680 --> 00:01:21.719 We are entirely upgrading your data parsing. 29 00:01:21.400 --> 00:01:23.079 Toolkit, which is so important. 30 00:01:23.280 --> 00:01:27.000 It really is whether you're say, a self taught programmer, 31 00:01:27.239 --> 00:01:31.120 a cybersecurity student, or just a professional trying to automate 32 00:01:31.159 --> 00:01:34.040 some tedious part of your workflow. We are going to 33 00:01:34.079 --> 00:01:39.400 explore three powerful techniques to search, filter, and extract specific 34 00:01:39.439 --> 00:01:43.400 information from those giant walls of HTML and XML text. 35 00:01:43.480 --> 00:01:46.719 Yeah, we're moving way beyond basic search functions. 36 00:01:46.359 --> 00:01:49.200 Here, Oh, way beyond. We're turning you into a data 37 00:01:49.239 --> 00:01:50.560 extraction power user. 38 00:01:50.359 --> 00:01:53.319 Which is an incredibly vital skill because you know, mastering 39 00:01:53.319 --> 00:01:56.200 these tools isn't just about memorizing commands. Is about developing 40 00:01:56.200 --> 00:02:00.159 a mental model of how information is logically or well, 41 00:02:00.319 --> 00:02:02.159 sometimes illogically organized on the. 42 00:02:02.120 --> 00:02:03.599 Web, sometimes very illogically. 43 00:02:03.920 --> 00:02:07.519 Oh absolutely, But once you understand the fundamental architecture, you 44 00:02:07.560 --> 00:02:11.120 can pull any data point you want with absolute surgical precision. 45 00:02:11.319 --> 00:02:14.919 Okay, so let's unpack this. Before you can extract data, 46 00:02:15.199 --> 00:02:19.400 you have to know how to move fluidly around its structure. Now, 47 00:02:19.599 --> 00:02:22.439 I know, if you've tinkered with Python libraries like beautiful Soup, 48 00:02:22.719 --> 00:02:25.400 you probably know how to do a basic search. You 49 00:02:25.439 --> 00:02:27.919 ask the code to find a link, and it finds 50 00:02:27.919 --> 00:02:31.120 a link. Simple enough, Simple enough. Yeah, But what happens 51 00:02:31.240 --> 00:02:34.520 when the thing you're looking for doesn't have a neat 52 00:02:34.560 --> 00:02:39.360 little label? You have to navigate relative to a different element. Yes, 53 00:02:39.479 --> 00:02:42.080 I like to frame this in my mind like navigating 54 00:02:42.080 --> 00:02:44.120 a massive, chaotic apartment building. 55 00:02:44.280 --> 00:02:46.919 I love that analogy. It perfectly maps to what we 56 00:02:46.960 --> 00:02:50.159 call the parse tree. The structure of an HTML document 57 00:02:50.199 --> 00:02:53.280 is hierarchical, just like a high rise building. You've got 58 00:02:53.280 --> 00:02:56.479 the main lobby, the floors, the hallways, and individual apartments. 59 00:02:56.719 --> 00:03:00.360 And sometimes you find yourself standing inside a specific apartment. 60 00:03:00.479 --> 00:03:03.199 Say you've located a specific image on the page, and 61 00:03:03.240 --> 00:03:06.120 you need to look around from that exact vantage. 62 00:03:05.759 --> 00:03:08.039 Point right to find what you actually want. So if 63 00:03:08.080 --> 00:03:10.599 I'm standing in that apartment, my first option is to 64 00:03:10.639 --> 00:03:13.560 look up in the code. These are called your parents. 65 00:03:14.360 --> 00:03:18.360 There's a concept here where you can ask your code 66 00:03:18.840 --> 00:03:21.520 to find the parent of the element you are currently 67 00:03:21.599 --> 00:03:21.960 looking at. 68 00:03:22.039 --> 00:03:23.199 Yep, the parent container. 69 00:03:23.680 --> 00:03:25.680 So if I'm looking at the link and I want 70 00:03:25.680 --> 00:03:28.080 to find the big container block that it lives inside. 71 00:03:28.319 --> 00:03:31.280 I tell the parser to go up. It searches up 72 00:03:31.319 --> 00:03:34.879 the parse tree and finds the very first parent container 73 00:03:34.879 --> 00:03:37.599 that matches my description exactly. It essentially walks up to 74 00:03:37.599 --> 00:03:38.719 the floor directly above you. 75 00:03:38.879 --> 00:03:40.919 Yes, and it's important to know you can also ask 76 00:03:41.000 --> 00:03:42.719 for all the parents plural. 77 00:03:42.840 --> 00:03:43.520 Oh right. 78 00:03:44.080 --> 00:03:46.759 If you do that, the parser doesn't stop at the 79 00:03:46.800 --> 00:03:49.719 first match. It will keep traveling all the way up 80 00:03:49.719 --> 00:03:53.840 the tree, returning every single matching parent container it finds 81 00:03:53.879 --> 00:03:56.719 along the way. It traces your steps all the way 82 00:03:56.800 --> 00:03:58.360 up to the roof of the building, which is the 83 00:03:58.439 --> 00:03:59.400 document route. 84 00:03:59.520 --> 00:04:03.039 Okay, so parents are the floors above you, then you 85 00:04:03.080 --> 00:04:03.439 have your. 86 00:04:03.360 --> 00:04:05.719 Siblings, right the siblings If parents are. 87 00:04:05.680 --> 00:04:09.319 Up, siblings are the apartments on your exact floor in 88 00:04:09.360 --> 00:04:12.319 the parse tree. Siblings refer to elements sitting at the 89 00:04:12.319 --> 00:04:15.120 exact same level of hierarchy. So let's say I'm looking 90 00:04:15.159 --> 00:04:17.079 at a piece of text and I want to find 91 00:04:17.079 --> 00:04:19.720 the image that sits right next to it. I can 92 00:04:19.759 --> 00:04:22.439 tell the code to find the next sibling. It just 93 00:04:22.480 --> 00:04:24.879 slides down the hall and grabs the next element on 94 00:04:24.920 --> 00:04:25.680 that exact. 95 00:04:25.399 --> 00:04:28.199 Same floor, and you can look backwards down the hall too, 96 00:04:28.240 --> 00:04:30.720 by asking for the previous siblings. Oh nice, This is 97 00:04:30.759 --> 00:04:35.079 incredibly useful for scraping. Imagine you are trying to pull 98 00:04:35.199 --> 00:04:38.519 data from a messy e commerce website. You want the 99 00:04:38.560 --> 00:04:41.759 price of a laptop, but the developer didn't give the 100 00:04:41.800 --> 00:04:45.800 price a unique identifier. It's literally just plain text. 101 00:04:46.240 --> 00:04:48.439 Wait if it's just plain text, how does the bot 102 00:04:48.560 --> 00:04:50.879 know it's a price and not like the weight of 103 00:04:50.920 --> 00:04:52.360 the laptop or the review score. 104 00:04:53.639 --> 00:04:56.480 What's fascinating here is how we use landmarks to solve 105 00:04:56.480 --> 00:04:57.480 that exact problem. 106 00:04:57.560 --> 00:04:58.120 Landmarks. 107 00:04:58.240 --> 00:05:00.920 Yeah, the actual price might just be a random number, 108 00:05:00.920 --> 00:05:04.240 but you notice a pattern. The price always sits exactly 109 00:05:04.279 --> 00:05:06.759 next to a bold label that says total price. 110 00:05:07.000 --> 00:05:08.959 Okay, I see where this is going. Right. 111 00:05:09.040 --> 00:05:12.879 You can't search for the fluctuating number directly because it changes, 112 00:05:13.639 --> 00:05:16.040 But you can tell your code to search for the 113 00:05:16.079 --> 00:05:21.240 text total price and then use the relational sibling function 114 00:05:21.399 --> 00:05:23.560 to just grab whatever is sitting right next door. 115 00:05:23.879 --> 00:05:26.959 That is incredibly clever. You use the reliable geography of 116 00:05:27.000 --> 00:05:30.199 the document to find the unreliable data exactly. But hold on, 117 00:05:30.240 --> 00:05:33.079 I have a question about siblings. Sure, what if the 118 00:05:33.319 --> 00:05:37.560 website updates its layout and the developer puts that price 119 00:05:37.680 --> 00:05:40.839 inside a little decorative box, so it's technically one level 120 00:05:40.879 --> 00:05:41.839 deeper in the code. 121 00:05:41.879 --> 00:05:44.560 Then your sibling search will return absolutely. 122 00:05:44.040 --> 00:05:46.120 Nothing, nothing at all, nothing. 123 00:05:46.079 --> 00:05:49.040 And that is the massive trap that catches beginners. The 124 00:05:49.079 --> 00:05:53.000 strictness of the sibling relationship is its biggest weakness. It 125 00:05:53.040 --> 00:05:56.439 only looks right next door on the exact same hierarchical level. 126 00:05:56.560 --> 00:05:56.879 Got it. 127 00:05:56.959 --> 00:05:59.600 If the target element is nested differently, maybe one floor 128 00:05:59.680 --> 00:06:02.639 up or tucked inside a closet on that same floor, 129 00:06:03.120 --> 00:06:05.240 the sibling function just won't see it. It will not 130 00:06:05.279 --> 00:06:07.439 look inside other rooms, and it will not look up 131 00:06:07.560 --> 00:06:08.079 or down the. 132 00:06:08.040 --> 00:06:10.759 Stairs, which naturally brings us to the stairs. 133 00:06:10.879 --> 00:06:11.759 Yes, exactly. 134 00:06:12.000 --> 00:06:14.879 If I want to search anywhere up or down the document, 135 00:06:15.519 --> 00:06:19.560 completely ignoring the level of hierarchy, I need a different tool. 136 00:06:20.160 --> 00:06:22.720 That's where we use the next and previous concepts. 137 00:06:22.279 --> 00:06:25.800 Right spot on. Instead of asking for a sibling, you 138 00:06:25.959 --> 00:06:28.600 just ask for the next element, period. 139 00:06:28.360 --> 00:06:30.319 So it's less restrictive, much less. 140 00:06:31.040 --> 00:06:33.560 When you ask for the next element, the parser starts 141 00:06:33.560 --> 00:06:37.000 at your current location and traverses down the entire rest 142 00:06:37.000 --> 00:06:39.879 of the parse tree. It walks down the hallway, opens 143 00:06:39.920 --> 00:06:43.319 every single door, walks down the stairs and checks every 144 00:06:43.360 --> 00:06:46.000 single room until it finds an element that matches what 145 00:06:46.040 --> 00:06:49.319 you're looking for. Wow, okay, And conversely, asking for the 146 00:06:49.399 --> 00:06:53.000 previous element just marches backward through the entire document flow. 147 00:06:53.120 --> 00:06:56.600 Okay. So relational navigation is wandering through the building using 148 00:06:56.720 --> 00:06:59.800 landmarks to find your way. It's powerful if things are messy. 149 00:07:00.279 --> 00:07:01.439 But what if I don't want to wander? 150 00:07:01.639 --> 00:07:02.160 What do you mean? 151 00:07:02.319 --> 00:07:04.199 Like, what if I'm looking at a modern, well built 152 00:07:04.240 --> 00:07:06.079 tech blog and know exactly where I want to go 153 00:07:06.480 --> 00:07:09.920 and I just want to hand over an exact mailing address. Nuh, 154 00:07:10.360 --> 00:07:12.959 that's where we shift our toolkit to these second method 155 00:07:13.040 --> 00:07:14.000 CSS selectors. 156 00:07:14.399 --> 00:07:18.199 Yes, this represents a major paradigm shift in how we 157 00:07:18.279 --> 00:07:22.319 query data. When you're using modern parsing libraries, they usually 158 00:07:22.319 --> 00:07:25.600 come with built in support for CSS selectors, which. 159 00:07:25.399 --> 00:07:27.920 Is what developers use to style things exactly. 160 00:07:28.439 --> 00:07:31.279 CSS is the language developers used to style web pages 161 00:07:31.319 --> 00:07:34.720 to make buttons read or fonts larger. But to apply 162 00:07:34.800 --> 00:07:38.879 those styles, developers have to label their HTML elements. Right, 163 00:07:39.160 --> 00:07:43.040 By using CSS selectors, we are piggybacking on the developer's 164 00:07:43.079 --> 00:07:45.319 own labeling system to extract data. 165 00:07:45.480 --> 00:07:48.160 So suddenly, instead of walking the halls. You can just 166 00:07:48.199 --> 00:07:51.160 shout out the exact address. Yes, the most basic way 167 00:07:51.199 --> 00:07:54.279 is selecting tags directly. I can just say give me 168 00:07:54.360 --> 00:07:55.959 every paragraph or give me every link. 169 00:07:56.079 --> 00:07:58.639 The address book gets much more granular than that. You 170 00:07:58.680 --> 00:08:01.839 can specify order. Let's say you don't want just any paragraph. 171 00:08:01.920 --> 00:08:04.160 You only want the very first paragraph of an article. 172 00:08:04.199 --> 00:08:06.279 Hold on, I've seen the documentation for this, and it 173 00:08:06.399 --> 00:08:10.000 uses a term that sounds super intimidating. Huh, a pseudo class. 174 00:08:10.279 --> 00:08:10.839 Ah. 175 00:08:11.000 --> 00:08:13.199 Yes, what does that actually mean in plain English? 176 00:08:13.360 --> 00:08:18.079 I know pseudoclass sounds like well academic jargon. Conceptually, it 177 00:08:18.240 --> 00:08:20.839 just means you are adding a condition to your search 178 00:08:21.160 --> 00:08:25.079 that isn't explicitly written in the HTMILT code. The ht 179 00:08:25.120 --> 00:08:27.759 MIL doesn't say I am the first paragraph. It's just 180 00:08:27.800 --> 00:08:31.279 a paragraph. But by using a pseudo class selector, you 181 00:08:31.360 --> 00:08:34.320 tell the parser count the paragraphs and only give me 182 00:08:34.360 --> 00:08:37.279 the one that is mathematically first or second or fifth. Oh, 183 00:08:37.320 --> 00:08:40.120 I see it allows for absolute numerical precision. 184 00:08:40.200 --> 00:08:43.879 That clarifies it perfectly. Now let's talk about hierarchy with CSS, 185 00:08:43.960 --> 00:08:46.440 because this is where the malely address really gets detailed. 186 00:08:46.639 --> 00:08:49.320 I can string these labels together, but I know there 187 00:08:49.399 --> 00:08:53.559 is a massive difference between asking for a descendant versus 188 00:08:53.639 --> 00:08:54.759 a direct child. 189 00:08:54.840 --> 00:08:57.679 Oh yes, this is a crucial distinction that will save 190 00:08:57.720 --> 00:08:58.720 you hours of debugging. 191 00:08:58.759 --> 00:08:59.120 Good to know. 192 00:08:59.320 --> 00:09:02.240 Let's stick with your house analogy. A descendant is anything 193 00:09:02.279 --> 00:09:05.639 contained inside a space, no matter how deeply buried. If 194 00:09:05.639 --> 00:09:07.519 you ask for a link that is a descendant of 195 00:09:07.519 --> 00:09:10.559 the living room, that link could be sitting on the sofa, 196 00:09:10.720 --> 00:09:12.960 or it could be inside a box, which is an 197 00:09:13.080 --> 00:09:15.759 inside a drawer, which is inside a cabinet in the living. 198 00:09:15.559 --> 00:09:17.200 Room, and it will still find it. 199 00:09:17.200 --> 00:09:18.559 It will absolutely find it. 200 00:09:18.960 --> 00:09:21.120 But a direct child is different. 201 00:09:21.440 --> 00:09:24.919 Very different. If you specify a direct child relationship, the 202 00:09:25.000 --> 00:09:28.960 element must be immediately inside the parent with nothing in between. Okay, 203 00:09:29.159 --> 00:09:31.159 If you ask for an image that is a direct 204 00:09:31.240 --> 00:09:33.240 child of the living room, it must be sitting right 205 00:09:33.279 --> 00:09:36.759 there on the floor. If the developer puts that image 206 00:09:36.799 --> 00:09:39.720 inside a structural container, like putting it inside a box, 207 00:09:40.360 --> 00:09:44.720 your direct child search will return completely empty because the 208 00:09:44.799 --> 00:09:47.360 image is a child of the box, not the living room. 209 00:09:47.559 --> 00:09:51.360 That exactness is demanding, But man, it's so powerful when 210 00:09:51.399 --> 00:09:54.000 you need it. And of course, see as selectors let 211 00:09:54.080 --> 00:09:57.559 you target the holy grail of web scraping classes and IDs. 212 00:09:57.799 --> 00:10:00.600 Yes, an idea is supposed to be completely unique on 213 00:10:00.639 --> 00:10:02.919 a page, like a social security number for a specific 214 00:10:02.919 --> 00:10:03.519 piece of data. 215 00:10:03.639 --> 00:10:03.960 Okay. 216 00:10:04.200 --> 00:10:06.679 A class is more of a category, like a label 217 00:10:06.720 --> 00:10:09.559 applied to all the author names on a page. CSS 218 00:10:09.600 --> 00:10:12.279 selectors allow you to target these instantly. You just drop 219 00:10:12.279 --> 00:10:14.759 in the ID or the class name and you teleport 220 00:10:14.840 --> 00:10:15.879 straight to the data. 221 00:10:15.960 --> 00:10:20.200 But the real power user moves involve fuzzy matching because 222 00:10:20.200 --> 00:10:23.320 sometimes the address book is well smudged. You don't have 223 00:10:23.360 --> 00:10:25.440 an exact match for a label, you just know part 224 00:10:25.480 --> 00:10:25.679 of it. 225 00:10:26.080 --> 00:10:29.240 This is where we bypass a lot of modern website defenses. 226 00:10:29.360 --> 00:10:30.840 Oh I love bypassing defenses. 227 00:10:31.000 --> 00:10:33.440 Right. Let's say a website is trying to track users, 228 00:10:33.559 --> 00:10:36.960 so every single image URL has a massive string of 229 00:10:37.039 --> 00:10:40.639 randomized gibberish numbers attached to the end of it. You 230 00:10:40.679 --> 00:10:43.639 can't search for an exact RL because it changes every 231 00:10:43.679 --> 00:10:45.080 single time you refresh the page. 232 00:10:45.200 --> 00:10:46.639 Right, so how do you find the image? 233 00:10:46.720 --> 00:10:50.200 You use fuzzy matching in CSS selectors. You can use 234 00:10:50.240 --> 00:10:54.080 special wildcard characters to perform partial matches. Okay, Instead of 235 00:10:54.120 --> 00:10:56.799 looking for the exact URL, you can tell the parts 236 00:10:56.799 --> 00:10:58.360 that look I don't care what the beginning of this 237 00:10:58.480 --> 00:11:00.840 link is just find me the ones that end with 238 00:11:00.879 --> 00:11:03.600 dot jpeg. That's amazing, Find me the links that start 239 00:11:03.639 --> 00:11:06.679 with https, or even find me a link that contains 240 00:11:06.720 --> 00:11:09.159 the word profile anywhere in the middle of the mess. 241 00:11:09.559 --> 00:11:12.799 It's an incredibly potent way to filter vast amounts of 242 00:11:12.840 --> 00:11:13.799 chaotic data. 243 00:11:13.879 --> 00:11:16.200 Okay, here's where it gets really interesting. I'll listening to 244 00:11:16.200 --> 00:11:20.360 you explain this. CSS selectors let you target unique IDs, 245 00:11:20.720 --> 00:11:24.080 specific classes, direct children, and even let you do this 246 00:11:24.200 --> 00:11:28.159 amazing fuzzy wild card matching. Yep, and it's all written 247 00:11:28.200 --> 00:11:31.600 in these tiny fast strings. So I have to ask, 248 00:11:31.679 --> 00:11:35.840 if CSS is this powerful, why would anyone ever bother 249 00:11:35.919 --> 00:11:38.279 with those relational parent sibling methods we talked about the 250 00:11:38.360 --> 00:11:41.759 very beginning. Doesn't CSS just make wandering around the apartment 251 00:11:41.799 --> 00:11:43.080 building totally obsolete? 252 00:11:43.200 --> 00:11:45.000 If we connect this to the bigger picture, you'll see 253 00:11:45.000 --> 00:11:48.120 why both are vital. Yes, CSS selectors are often faster 254 00:11:48.200 --> 00:11:50.639 to write and easier to read. But and this is 255 00:11:50.679 --> 00:11:54.080 a massive for term, CSS relies entirely on the structural 256 00:11:54.120 --> 00:11:56.360 logic provided by the original web developer. 257 00:11:56.480 --> 00:12:00.240 Oh, meaning the classes and IDs have to actually be 258 00:12:00.320 --> 00:12:02.000 there exactly, and they have to make sense. 259 00:12:02.159 --> 00:12:04.679 Right, You are relying on the assumption that the developer 260 00:12:04.720 --> 00:12:07.559 wrote semantic, well organized code. 261 00:12:07.240 --> 00:12:09.679 With useful hooks, which is a big assumption. 262 00:12:09.320 --> 00:12:12.159 A huge assumption. What if you are scraping a legacy 263 00:12:12.200 --> 00:12:15.000 website from nineteen ninety eight that was built using outdated 264 00:12:15.000 --> 00:12:18.679 table layouts there are no classes, there are no IDs, 265 00:12:19.360 --> 00:12:22.000 Or what if you are scraping a modern site built 266 00:12:22.039 --> 00:12:25.879 with frameworks that intentionally randomize their class names every single day, 267 00:12:26.320 --> 00:12:28.639 specifically to break your scraper. 268 00:12:28.840 --> 00:12:31.960 Then your exact mailing address is completely useless. The letters 269 00:12:31.960 --> 00:12:33.120 are written in a foreign. 270 00:12:32.879 --> 00:12:36.519 Language, precisely when the site lacks CSS hooks or is 271 00:12:36.559 --> 00:12:39.120 actively trying to hide them. You have to fall back 272 00:12:39.120 --> 00:12:42.879 on the relational document flow methods. The physical layout of 273 00:12:42.879 --> 00:12:45.720 the parse tree always exists, even if the CSS labels 274 00:12:45.720 --> 00:12:49.039 are a total mess. They are complementary tools in your toolkit, 275 00:12:49.159 --> 00:12:50.480 not replacements for each other. 276 00:12:50.720 --> 00:12:53.240 That makes so much sense. You need the map, but 277 00:12:53.279 --> 00:12:55.320 you also need to know how to walk the terrain 278 00:12:55.559 --> 00:12:57.799 if the map is suddenly written in invisible. 279 00:12:57.399 --> 00:12:58.360 Ink beautifully put. 280 00:12:58.480 --> 00:13:00.759 Okay, so we have the family tree for relative logic, 281 00:13:00.799 --> 00:13:03.559 and we have the direct address book with CSS. But 282 00:13:03.679 --> 00:13:06.159 what if we're dealing with a completely different kind of terrain. 283 00:13:06.399 --> 00:13:07.519 Okay, lay it on me. 284 00:13:07.720 --> 00:13:11.320 What if we are looking at strict machine readable XML, 285 00:13:11.960 --> 00:13:16.039 or what if the HTML is so incredibly broken that 286 00:13:16.080 --> 00:13:18.639 we need to mathematically force it into a rigid grid 287 00:13:18.679 --> 00:13:21.320 before we can even search it. That brings us to 288 00:13:21.360 --> 00:13:22.919 our third tool, XPath. 289 00:13:23.399 --> 00:13:26.720 XPath is a distinctly different animal. Up until now we've 290 00:13:26.759 --> 00:13:29.559 been talking about tools that are somewhat forgiving of bad 291 00:13:29.639 --> 00:13:33.440 web design. But sometimes, especially if you are building data 292 00:13:33.480 --> 00:13:36.759 pipelines for machine learning or dealing with strict data feeds 293 00:13:36.759 --> 00:13:41.600 from an API, you need absolute mathematical precision. Right to 294 00:13:41.720 --> 00:13:45.039 use XPath effectively, we usually shift our parsing engine to 295 00:13:45.080 --> 00:13:46.480 something called LXML. 296 00:13:46.639 --> 00:13:48.840 Wait why do we need a new engine? What's wrong 297 00:13:48.879 --> 00:13:51.360 with the default way browsers look at HTML? 298 00:13:51.519 --> 00:13:55.440 Well, web browsers are incredibly polite. Alite, yeah, polite. If 299 00:13:55.440 --> 00:13:58.159 a web developer forgets to close a tag or messes 300 00:13:58.240 --> 00:14:01.480 up the hierarchy, the browser quietly fixes it behind the scenes, 301 00:14:01.480 --> 00:14:03.879 so the human looking at the screen doesn't notice. 302 00:14:03.639 --> 00:14:04.240 Oh I see. 303 00:14:04.399 --> 00:14:07.320 But for a data extraction, the script that hidden chaos 304 00:14:07.399 --> 00:14:11.600 is a nightmare. The LXML engine takes that messy HTML 305 00:14:11.639 --> 00:14:16.279 and forces it into a highly rigid flawlessly structured XML tree. 306 00:14:16.559 --> 00:14:17.799 It cleans up the mess. 307 00:14:17.919 --> 00:14:21.039 It turns the chaotic apartment building into a flawless architectural 308 00:14:21.080 --> 00:14:22.240 blueprint exactly. 309 00:14:22.320 --> 00:14:24.360 And once you have that perfect blueprint, you can use 310 00:14:24.480 --> 00:14:26.039 XPath syntax to target. 311 00:14:25.720 --> 00:14:27.240 Element, which works a bit differently. 312 00:14:27.039 --> 00:14:31.200 Right it does. XPath stands for XML path language. It 313 00:14:31.240 --> 00:14:33.840 operates much like the file directory paths on your computer. 314 00:14:34.279 --> 00:14:38.440 You use forward slashes to define the route. Conceptually, if 315 00:14:38.440 --> 00:14:41.759 you use a single forward slash, you are telling the 316 00:14:41.799 --> 00:14:44.440 parser to start exactly at the root of the document 317 00:14:44.519 --> 00:14:47.360 and walk down step by step. If you miss a 318 00:14:47.399 --> 00:14:50.279 step where the element isn't directly underneath where you said 319 00:14:50.320 --> 00:14:51.919 it would be, the path breaks. 320 00:14:51.960 --> 00:14:53.799 So you don't always have to start at the route right. 321 00:14:53.879 --> 00:14:56.399 No, and that's the magic of XPath. If you use 322 00:14:56.440 --> 00:14:59.399 a double forward slash, you are telling XPath to search 323 00:14:59.399 --> 00:15:02.600 for the element anywhere in the entire documentary, no matter 324 00:15:02.639 --> 00:15:05.120 how deeply it is nested. You bypass the step by 325 00:15:05.120 --> 00:15:08.320 step strictness. You can just say, search the whole blueprint 326 00:15:08.360 --> 00:15:09.519 for any paragraph. 327 00:15:09.559 --> 00:15:11.600 And how do we handle filtering? Like if I only 328 00:15:11.600 --> 00:15:14.240 want a specific paragraph with a certain ID, how does 329 00:15:14.399 --> 00:15:17.080 XPath handle that? Compared to CSS In. 330 00:15:17.320 --> 00:15:19.799 XPath, you define the path first, and then you attach 331 00:15:19.799 --> 00:15:22.919 a filter bracket at the end. Inside that bracket you 332 00:15:23.080 --> 00:15:26.480 use an at bat symbol to target attributes. So conceptually 333 00:15:26.519 --> 00:15:29.759 you're saying, find me any element in the document, but 334 00:15:29.879 --> 00:15:32.440 only keep the ones where the eight ad id attribute 335 00:15:32.480 --> 00:15:36.240 perfectly matches the specific word. It drops you right onto 336 00:15:36.240 --> 00:15:37.240 that specific node. 337 00:15:37.320 --> 00:15:39.159 Wait, let me stop you there. If I drop onto 338 00:15:39.200 --> 00:15:41.600 that node, I just have the HTML code. How do 339 00:15:41.679 --> 00:15:43.799 I actually pull the text out so I can save 340 00:15:43.840 --> 00:15:45.559 it to a spreadsheet or a database. 341 00:15:45.799 --> 00:15:49.039 Ah, that's the beauty of XPath. The extraction is built 342 00:15:49.039 --> 00:15:53.000 into the pathing language itself. Really Yeah, Once you've navigated 343 00:15:53.039 --> 00:15:56.080 to the correct node, if you want the human readable 344 00:15:56.080 --> 00:15:59.919 text inside it, you literally just append a text extraction 345 00:16:00.039 --> 00:16:02.320 command to the end of your path. Just write text 346 00:16:02.360 --> 00:16:02.840 at the end. 347 00:16:03.000 --> 00:16:03.399 Oh wow. 348 00:16:03.519 --> 00:16:05.960 And if you want a specific attribute like the underlying 349 00:16:06.200 --> 00:16:08.360 URL of an image, you just append the at symbol 350 00:16:08.440 --> 00:16:10.320 and the name of that attribute to the end of 351 00:16:10.360 --> 00:16:13.559 the path. The parser doesn't return the HTML element. It 352 00:16:13.600 --> 00:16:15.879 returns the exact piece of data you asked for. 353 00:16:16.120 --> 00:16:19.200 And if your search finds, say fifty matches. 354 00:16:19.120 --> 00:16:21.879 You can just use brackets to index them directly. You 355 00:16:21.919 --> 00:16:24.240 can just add a number to the end of the 356 00:16:24.240 --> 00:16:26.159 path and say give me the fourth match or give 357 00:16:26.159 --> 00:16:27.000 me the first match. 358 00:16:27.480 --> 00:16:29.559 So what does this all mean. Let's bring this all 359 00:16:29.559 --> 00:16:33.639 together for you listening to solidify the mental model. We've 360 00:16:33.759 --> 00:16:36.840 updraded your toolkit with three distinct methods today. 361 00:16:36.960 --> 00:16:37.559 Yes, we have. 362 00:16:37.919 --> 00:16:42.440 First, we explored relational navigation. That is like physically walking 363 00:16:42.480 --> 00:16:45.480 through the apartment building. You are starting in one room 364 00:16:45.960 --> 00:16:48.639 and navigating based on what is next door, above you 365 00:16:48.840 --> 00:16:51.720 or down the hall. It is essential when the website 366 00:16:51.759 --> 00:16:53.399 is messy and lacks clear labels. 367 00:16:53.639 --> 00:16:57.159 The second tool was CSS selectors that is your mailing 368 00:16:57.159 --> 00:17:00.159 address book. You aren't walking the halls, you are relying 369 00:17:00.200 --> 00:17:03.600 on the developer's labels to teleport directly to the destination. 370 00:17:03.879 --> 00:17:07.440 It's fast, it's concise, and it supports brilliant wildcard matching. 371 00:17:07.559 --> 00:17:11.559 And the third tool, XPath is your raw GPS coordinates. 372 00:17:11.759 --> 00:17:15.599 It requires a perfectly clean architectural blueprint, but it provides 373 00:17:15.640 --> 00:17:20.920 a highly specific structural pathing language designed for absolute precision targeting. 374 00:17:21.079 --> 00:17:23.279 And it's worth noting that learning XPath gives you a 375 00:17:23.400 --> 00:17:27.559 universally transferable skill. While some libraries are specific to Python, 376 00:17:27.920 --> 00:17:31.359 XPath is a standardized language used across almost every programming 377 00:17:31.440 --> 00:17:32.880 environment and operating system. 378 00:17:32.960 --> 00:17:36.759 Exactly three tools, one massive upgrade to your ability to 379 00:17:36.799 --> 00:17:39.799 parse and extract data. Now, before we wrap up, I 380 00:17:39.880 --> 00:17:41.839 want to give you a quick exercise right now to 381 00:17:41.880 --> 00:17:43.079 reinforce what we just talked about. 382 00:17:43.160 --> 00:17:44.000 Oh, I like this, Yeah. 383 00:17:44.039 --> 00:17:47.279 I want you to imagine a web page. On this page, 384 00:17:47.400 --> 00:17:51.960 there is an image placed directly immediately inside a paragraph. 385 00:17:52.240 --> 00:17:55.279 Based on our discussion about CSS selectors, how would you 386 00:17:55.359 --> 00:17:58.880 target just that specific image. Not an image buried inside 387 00:17:58.920 --> 00:18:01.119 a box that happens be inside of paragraph, but an 388 00:18:01.119 --> 00:18:04.640 image that is a direct child of the paragraph. Think 389 00:18:04.640 --> 00:18:05.240 about for second. 390 00:18:05.279 --> 00:18:05.920 It's a good test. 391 00:18:06.160 --> 00:18:09.680 If you visualized asking for a paragraph followed by a 392 00:18:09.720 --> 00:18:13.119 greater than sign followed by the image, you absolutely nailed it. 393 00:18:13.720 --> 00:18:15.960 That is the direct child selector at work. 394 00:18:16.160 --> 00:18:19.519 This raises an important question though, for everyone listening. As 395 00:18:19.559 --> 00:18:22.200 you take these tools and start exploring, I encourage you 396 00:18:22.279 --> 00:18:25.400 to completely change how you view websites. The next time 397 00:18:25.400 --> 00:18:28.640 you load a page, look past the design, think about 398 00:18:28.640 --> 00:18:32.359 the invisible scaffolding. As you've learned today, how a developer 399 00:18:32.440 --> 00:18:36.079 structures their code completely dictates how easily that data can 400 00:18:36.119 --> 00:18:39.640 be accessed by machines, which leads to a massive friction 401 00:18:39.759 --> 00:18:43.039