WEBVTT 1 00:00:00.160 --> 00:00:04.839 So I want you to imagine this scenario. Your webscraper 2 00:00:05.040 --> 00:00:07.839 is well, it's perfect. The code is flawless, the logic 3 00:00:07.919 --> 00:00:10.679 is tight, your loops are like totally optimized. 4 00:00:10.720 --> 00:00:13.320 Oh yeah, I know exactly where this is going, right, So. 5 00:00:13.240 --> 00:00:16.640 You run the script on Monday and it flawlessly extracts 6 00:00:16.719 --> 00:00:20.440 I don't know, ten thousand data points. Then you run 7 00:00:20.480 --> 00:00:24.679 the exact same script on Tuesday, you haven't touched a 8 00:00:24.760 --> 00:00:25.800 single line. 9 00:00:25.559 --> 00:00:28.679 Of code, and it crashes instantly, instantly. 10 00:00:28.679 --> 00:00:32.359 You just get this massive wall of red text in 11 00:00:32.399 --> 00:00:36.799 your terminal, ending in this fatal no such element error. 12 00:00:37.560 --> 00:00:40.320 And you're sitting there like what happened? Because you open 13 00:00:40.359 --> 00:00:42.880 the website in your browser and it looks identical to yesterday. 14 00:00:42.920 --> 00:00:45.039 Yeah, the buttons are in the exact same place, text 15 00:00:45.039 --> 00:00:45.679 as the same. 16 00:00:45.520 --> 00:00:49.079 Color exactly, But underneath the hood, the entire architecture just 17 00:00:49.159 --> 00:00:50.920 like shifted while you were sleeping. 18 00:00:51.079 --> 00:00:53.880 Yeah. That is practically a rite of passage for anyone 19 00:00:53.920 --> 00:00:56.759 writing automation scripts. I mean, it's the realization that what 20 00:00:56.799 --> 00:00:58.880 we see on the screen, right and what the machine 21 00:00:58.920 --> 00:01:03.520 sees and document object model are just two entirely different realities. 22 00:01:04.120 --> 00:01:08.439 Because honestly, opening a web page with a script is trivial. 23 00:01:08.840 --> 00:01:12.200 You pass a URL, the page loads done. But the 24 00:01:12.239 --> 00:01:17.280 actual engineering challenge, whether you're building a data pipeline, running 25 00:01:17.280 --> 00:01:21.400 cybersecurity audits, or training in AI, the challenge is interacting 26 00:01:21.439 --> 00:01:24.120 with that environment. Once it's open. Your bot has to 27 00:01:24.200 --> 00:01:25.920 know how to actually see. 28 00:01:25.959 --> 00:01:29.519 Which is our mission for this deep dive, So welcome. 29 00:01:29.959 --> 00:01:33.640 Today we are exploring the hidden mechanics of web automation. 30 00:01:34.280 --> 00:01:37.159 We're going to unpack exactly how we use code to 31 00:01:37.280 --> 00:01:41.319 locate and interact with the building blocks of any web application, 32 00:01:41.719 --> 00:01:44.719 especially when those apps are actively mutating right under our feet. 33 00:01:44.719 --> 00:01:46.799 It's a moving target, for sure, it really is. 34 00:01:46.840 --> 00:01:49.719 And we are doing this completely screen free. Today. We're 35 00:01:49.760 --> 00:01:52.079 going to map out the logic and the architecture right 36 00:01:52.079 --> 00:01:54.680 here in your ears, so you can conceptualize the code 37 00:01:54.879 --> 00:01:57.079 without needing to squint at a terminal. 38 00:01:56.760 --> 00:01:59.560 Which is great because the logic is what actually matters anyway. 39 00:02:00.079 --> 00:02:01.959 I mean, before we can command a bot to grab 40 00:02:02.000 --> 00:02:04.239 an element, yeah, we have to understand the environment we're 41 00:02:04.280 --> 00:02:05.319 dropping it into. 42 00:02:05.200 --> 00:02:06.920 Right, the anatomy of the page. 43 00:02:06.599 --> 00:02:09.680 Exactly because modern web applications, you know, the ones built 44 00:02:09.719 --> 00:02:13.199 on frameworks like react or Angular, they are not static 45 00:02:13.240 --> 00:02:18.080 documents anymore. They are these highly complex session driven engines. 46 00:02:18.479 --> 00:02:21.039 Every single thing on the page, a text field and 47 00:02:21.120 --> 00:02:22.719 image of button is an element. 48 00:02:22.840 --> 00:02:25.879 Okay, So I always visualize a static web page kind 49 00:02:25.919 --> 00:02:28.680 of like a traditional room, right, and the dom elements 50 00:02:28.719 --> 00:02:29.479 are the furniture. 51 00:02:29.520 --> 00:02:30.280 Okay, I like that. 52 00:02:30.439 --> 00:02:32.960 So you have a chair, a desk, a rug. If 53 00:02:32.960 --> 00:02:35.360 I tell my script, hey, go interact with the chair, 54 00:02:35.639 --> 00:02:38.360 it just scans the room, finds the object designed for sitting, 55 00:02:38.479 --> 00:02:42.400 and takes action. Right, But modern web apps aren't a 56 00:02:42.439 --> 00:02:45.639 normal room. They're more like an Ikea showroom where a 57 00:02:45.680 --> 00:02:49.319 ghost is constantly rearranging the floor plan, like swapping the 58 00:02:49.360 --> 00:02:52.319 price tags and repainting the furniture every single time you blink. 59 00:02:52.560 --> 00:02:55.360 We're dealing with dynamic elements. 60 00:02:55.400 --> 00:02:57.240 Well, actually, it's worse than that. The ghost doesn't just 61 00:02:57.360 --> 00:03:01.159 repaint the chair. Wait really yeah, the server actually destroys 62 00:03:01.199 --> 00:03:03.919 the chair entirely and builds a brand new one in 63 00:03:03.919 --> 00:03:06.759 a microsecond. And when it does that, it assigns it 64 00:03:06.800 --> 00:03:10.520 a completely randomized internal identifier. Oh wow, Okay, so that's 65 00:03:10.560 --> 00:03:14.719 why your perfect script crashed on Tuesday. On Monday, that 66 00:03:14.800 --> 00:03:17.599 login button might have been internally labeled with an idea 67 00:03:17.599 --> 00:03:21.199 of like button underscore A seven B nine X, and 68 00:03:21.240 --> 00:03:22.400 you hardcode that into your. 69 00:03:22.360 --> 00:03:24.479 Script, right, because why wouldn't you? 70 00:03:24.639 --> 00:03:27.879 Exactly, But on Tuesday, a new user session generates a 71 00:03:27.879 --> 00:03:30.759 new hash, and the idea is now button underscore by 72 00:03:30.800 --> 00:03:34.560 two V four sets. Your script goes looking for Monday's button, 73 00:03:34.879 --> 00:03:37.120 finds nothing and just completely panics. 74 00:03:37.400 --> 00:03:39.560 So wait, if I write code to find a button, 75 00:03:39.759 --> 00:03:42.120 the button might just change its identity. It's a literal 76 00:03:42.199 --> 00:03:42.960 moving target. 77 00:03:43.120 --> 00:03:46.800 Precisely. The core identifiers, you know, the IDs, the names, 78 00:03:46.800 --> 00:03:49.199 they aren't fixed anymore. They're calculated on the fly by 79 00:03:49.240 --> 00:03:49.840 the database. 80 00:03:50.039 --> 00:03:52.520 So if that's the reality of the environment, how do 81 00:03:52.560 --> 00:03:55.039 we ever reliably find what we're looking for? I mean, 82 00:03:55.080 --> 00:03:57.400 we can't just cross our fingers and hope the identifiers 83 00:03:57.439 --> 00:03:58.039 stay the same. 84 00:03:58.280 --> 00:04:00.919 No, definitely not. What you have to do is rely 85 00:04:01.039 --> 00:04:04.439 on a hierarchy of locator strategies. You basically build this 86 00:04:04.520 --> 00:04:09.439 eight piece detective toolkit of fallback methods, and you escalate 87 00:04:09.479 --> 00:04:12.520 your approach based on how affuscated the element is. 88 00:04:12.639 --> 00:04:14.719 Okay, an eight piece toolkit. Let's break that down. 89 00:04:15.000 --> 00:04:18.319 So in an ideal world, method number one, the absolute 90 00:04:18.319 --> 00:04:21.519 gold standard is locating an element by its ID, right, 91 00:04:21.879 --> 00:04:25.519 When a developer builds a clean, well structured application, they 92 00:04:25.560 --> 00:04:29.600 assign a unique static ID to every critical web element. 93 00:04:29.959 --> 00:04:33.399 It's the fastest, most computationally efficient way to traverse the 94 00:04:33.480 --> 00:04:38.720 dom because by definition, an ID should only exist exactly 95 00:04:38.800 --> 00:04:39.920 once on a page. 96 00:04:40.079 --> 00:04:43.319 But wait, if ID is the absolute best, most efficient 97 00:04:43.360 --> 00:04:45.360 way to pinpoint an element, why do we have seven 98 00:04:45.399 --> 00:04:48.160 other methods? Like? Why not just use ID one hundred 99 00:04:48.160 --> 00:04:48.920 percent of the time? 100 00:04:49.079 --> 00:04:52.519 I wish we could, but the reality is well, developers 101 00:04:52.519 --> 00:04:55.360 get lazy fair enough, they don't always tag every single 102 00:04:55.360 --> 00:04:58.319 element with a unique ID, or, like we just talked 103 00:04:58.319 --> 00:05:02.199 about with the dynamic element, modern frameworks deliberately randomize them. 104 00:05:02.399 --> 00:05:05.399 A unique idea is entirely useless to us if it's 105 00:05:05.480 --> 00:05:06.439 mutating every. 106 00:05:06.199 --> 00:05:09.439 Session right ghosts in the IKEA showroom. So when the 107 00:05:09.480 --> 00:05:12.079 ID fails, or if it's missing entirely, what's next in 108 00:05:12.120 --> 00:05:12.600 the toolkit? 109 00:05:12.720 --> 00:05:14.600 We drop down to the next tier locator? So this 110 00:05:14.639 --> 00:05:17.240 Moivie methods two, six, and seven. We look at how 111 00:05:17.240 --> 00:05:20.279 the developer categorize the elements rather than how they uniquely 112 00:05:20.319 --> 00:05:23.240 identify them. We searched by the name attribute, the tag name, 113 00:05:23.439 --> 00:05:24.199 or the class name. 114 00:05:24.759 --> 00:05:30.240 But doesn't that introduce a massive new pro collateral damage. 115 00:05:30.279 --> 00:05:32.800 Oh absolutely, because if I search by name or class, 116 00:05:32.839 --> 00:05:36.120 I'm not looking for a unique identifier anymore. Let's say 117 00:05:36.120 --> 00:05:39.000 a developer styled all their primary action buttons with the 118 00:05:39.000 --> 00:05:41.360 CSS BTM primary. 119 00:05:41.560 --> 00:05:41.759 Right. 120 00:05:42.360 --> 00:05:44.480 If I tell my bot, hey, go click the element 121 00:05:44.519 --> 00:05:46.959 with the class BTM primary, expecting it to hit the 122 00:05:46.959 --> 00:05:50.240 submit button. What happens if the delete account button shares 123 00:05:50.279 --> 00:05:51.319 that exact same. 124 00:05:51.120 --> 00:05:54.399 Class, you trigger a catastrophic failure. You might delete the 125 00:05:54.480 --> 00:05:57.639 user's account instead of submitting the form. Yikes. Okay, and 126 00:05:57.680 --> 00:06:00.839 that is the inherent danger of using class, name or 127 00:06:00.879 --> 00:06:05.800 tag named locators. You are casting a very, very wide net. 128 00:06:05.959 --> 00:06:08.079 If you ask your script to find an input tag, 129 00:06:08.399 --> 00:06:11.040 it might find thirty of them on a complex form. 130 00:06:11.079 --> 00:06:13.839 Right, because every textbox is an input tag exactly. 131 00:06:14.160 --> 00:06:16.639 So using these categorical locators means you have to add 132 00:06:16.639 --> 00:06:19.920 secondary filtering logic to your script just to sift through 133 00:06:19.959 --> 00:06:23.240 the array of results you just pull down. They are clues, 134 00:06:23.560 --> 00:06:25.000 they aren't precise coordinates. 135 00:06:25.199 --> 00:06:28.800 Okay, so IDs are brittle if they're dynamic, and glasses 136 00:06:28.800 --> 00:06:31.920 and tags are dangerously broad. What about when we're just 137 00:06:32.000 --> 00:06:35.040 navigating between pages, like we're often just looking for a 138 00:06:35.079 --> 00:06:37.480 simple hyperlink right, just some text on the screen. 139 00:06:37.639 --> 00:06:39.680 Right, And that brings us to methods three and four 140 00:06:40.360 --> 00:06:43.800 link text and partial link text. For hyperlinks, we can 141 00:06:43.839 --> 00:06:47.800 actually bypass the structural attributes entirely. You simply command the 142 00:06:47.839 --> 00:06:50.759 code to scan the DOM for anchor tags that contain 143 00:06:50.800 --> 00:06:53.560 a specific string of user facing text. 144 00:06:53.680 --> 00:06:56.240 Oh, this one makes total sense. It's like it's like 145 00:06:56.360 --> 00:06:59.360 finding a book in a library. Using link text is 146 00:06:59.360 --> 00:07:02.480 like handing the librarian the exact, perfect full title of 147 00:07:02.480 --> 00:07:06.160 the manuscript, punctuation at all. But partial link text is 148 00:07:06.199 --> 00:07:08.160 the real life saver here, because if you can only 149 00:07:08.199 --> 00:07:10.680 remember one or two words of the title, it's still 150 00:07:10.800 --> 00:07:11.800 enough to find it. 151 00:07:11.800 --> 00:07:14.240 It builds incredible flexibility into your script. 152 00:07:14.399 --> 00:07:16.720 Yeah, because if a site has a link that says, 153 00:07:16.759 --> 00:07:19.360 you know, download the Q three financial report for twenty 154 00:07:19.399 --> 00:07:21.600 twenty six, I don't want to hard code the year 155 00:07:22.079 --> 00:07:25.360 because next year my scraper breaks. I just use partial 156 00:07:25.399 --> 00:07:28.439 link text to search for download the Q three financial 157 00:07:28.480 --> 00:07:31.360 report and let it totally ignore the dynamic date at 158 00:07:31.360 --> 00:07:31.720 the end. 159 00:07:32.120 --> 00:07:35.199 It's super smart. But again, this only works for anchor tags. 160 00:07:35.519 --> 00:07:38.399 If you need to interact with, say a complex drop 161 00:07:38.439 --> 00:07:41.399 down menu or a hidden form field, link text won't 162 00:07:41.399 --> 00:07:43.519 help you at all. Right, that's when you often have 163 00:07:43.560 --> 00:07:46.839 to resort to method five. You hijack the styling rules 164 00:07:46.839 --> 00:07:48.800 by leaning on CSS selectors. 165 00:07:48.959 --> 00:07:52.600 Ah, CSS selectors. This is actually one of my favorite workarounds. 166 00:07:52.920 --> 00:07:57.040 When IDs fail, I almost always end up hijacking the CSS. 167 00:07:57.480 --> 00:08:00.120 It's kind of funny, actually, how a syntax design and 168 00:08:00.160 --> 00:08:03.319 purely for front end aesthetics like making a button red 169 00:08:03.399 --> 00:08:06.720 or adding padding, becomes our best weapon for scraping. 170 00:08:06.600 --> 00:08:09.120 Is erotic, isn't it. But front end developers have to 171 00:08:09.120 --> 00:08:12.480 be incredibly precise when they apply styles. They write these 172 00:08:12.560 --> 00:08:16.959 highly specific CSS selector strings, chaining together classes and attributes 173 00:08:17.000 --> 00:08:19.680 just to ensure only one specific element gets that drop 174 00:08:19.680 --> 00:08:20.279 shadow right. 175 00:08:20.319 --> 00:08:21.920 They do the hard work for us. 176 00:08:21.800 --> 00:08:25.399 Exactly because their styling rules require pinpoint accuracy. We could 177 00:08:25.439 --> 00:08:27.800 just copy their selector strings and feed them directly into 178 00:08:27.879 --> 00:08:30.240 our automation bot to look at the exact same element. 179 00:08:30.399 --> 00:08:33.080 But wait, if I use a CSS selector, aren't I 180 00:08:33.200 --> 00:08:36.200 basically just relying on the UI designer not to change 181 00:08:36.240 --> 00:08:38.480 their mind. Like if they decide to rebrand the site 182 00:08:38.480 --> 00:08:42.120 tomorrow and change the styling framework, my CSS selector breaks instantly. 183 00:08:42.320 --> 00:08:45.200 You absolutely are yeah, and that is the perpetual Cold 184 00:08:45.240 --> 00:08:48.200 War of web automation. Every single time the front end 185 00:08:48.240 --> 00:08:51.000 team pushes a design update, the automation engineers have to 186 00:08:51.039 --> 00:08:52.559 rebuild their scrapers. 187 00:08:52.159 --> 00:08:54.440 Which brings us to the nuclear option. 188 00:08:54.639 --> 00:08:56.799 Ah yes, method number eight. 189 00:08:57.120 --> 00:09:00.360 What happens when the element is practically invisible to our code? 190 00:09:00.679 --> 00:09:03.039 It has no static idea, it has no name, It 191 00:09:03.120 --> 00:09:06.519 isn't a link. The classes are just completely randomized hashes. 192 00:09:06.600 --> 00:09:09.799 You are staring at a totally anonymous shifting button on 193 00:09:09.840 --> 00:09:12.200 the page. How do we force the bot to see it? 194 00:09:12.679 --> 00:09:15.159 That is when we abandon the simple attributes and we 195 00:09:15.159 --> 00:09:18.559 pull out XPath, the heavy hitter. Oh yeah. XPath or 196 00:09:18.679 --> 00:09:22.399 xmlpath language is a query language that treats the HTML 197 00:09:22.440 --> 00:09:26.399 document like a massive, interconnected database. It opens up entirely 198 00:09:26.480 --> 00:09:29.559 new possibilities because it allows you to map out complex 199 00:09:29.600 --> 00:09:31.480 geographical roots through the dom tree. 200 00:09:31.639 --> 00:09:35.440 So if the standard locators fail, XPath basically forces us 201 00:09:35.480 --> 00:09:39.039 to read the raw architectural blueprint. It's like, uh, we 202 00:09:39.120 --> 00:09:41.039 stop looking at the paint on the walls when we 203 00:09:41.080 --> 00:09:43.240 start looking at the structural beams underneath. 204 00:09:43.399 --> 00:09:46.039 That's a great way to put it. You're navigating the parent, 205 00:09:46.159 --> 00:09:49.799 child and sibling relationships of the code itself. Think of 206 00:09:49.840 --> 00:09:52.440 the web page as an upside down tree. Okay, the 207 00:09:52.519 --> 00:09:55.000 root is the start of the document, branching down into 208 00:09:55.000 --> 00:09:58.759 the head and body, splitting into individual containers, and finally 209 00:09:58.879 --> 00:10:01.399 ending into specific button and text fields at the very 210 00:10:01.440 --> 00:10:05.519 tips of the branches. XPath lets you trace that exact lineage. 211 00:10:05.679 --> 00:10:08.799 But and this is a massive butt. We really need 212 00:10:08.879 --> 00:10:13.279 to separate absolute XPath from relative XPath here, because using 213 00:10:13.320 --> 00:10:15.840 the wrong one is the number one reason beginners build 214 00:10:15.960 --> 00:10:20.159 terrible fragile bots. So true, absolute XPath maps the journey 215 00:10:20.159 --> 00:10:22.720 from the absolute root of the document, step by step, 216 00:10:22.759 --> 00:10:25.159 all the way down to your target element right, which. 217 00:10:25.000 --> 00:10:27.679 Means your locator looks like a terrifying string of slashes 218 00:10:27.720 --> 00:10:31.480 and node names. Oh, like HTML slash body, slash, div 219 00:10:31.519 --> 00:10:35.320 slash div slash, form slash, divslash button. It traces every 220 00:10:35.360 --> 00:10:36.720 single generational step. 221 00:10:36.919 --> 00:10:40.759 Yeah, using absolute XPath sounds like giving someone driving directions 222 00:10:40.840 --> 00:10:43.840 by saying, Okay, start at the very top border of 223 00:10:43.840 --> 00:10:46.759 the country. Drive exactly three thousand miles south on the 224 00:10:46.759 --> 00:10:51.200 main highway. Take the forty fifth exit, turn left, drive 225 00:10:51.240 --> 00:10:53.840 past two specific trees, and look for the third house 226 00:10:53.840 --> 00:10:54.320 on the right. 227 00:10:54.799 --> 00:10:56.720 That is painfully accurate right. 228 00:10:56.679 --> 00:10:59.799 And the mechanism of failure here is so obvious. If 229 00:11:00.120 --> 00:11:03.679 literally one detour happens, If the city plants a new tree, 230 00:11:03.840 --> 00:11:07.000 or the web developer throws a single new banner advertisement 231 00:11:07.039 --> 00:11:10.639 at the top of the page, your entire structural lineage shifts. 232 00:11:10.679 --> 00:11:12.720 Your directions are completely useless. 233 00:11:12.759 --> 00:11:16.519 Spot on, the bot gets hopelessly lost, and the script crashes. 234 00:11:16.799 --> 00:11:19.840 It is impossibly brittle. One minor update of the header 235 00:11:19.879 --> 00:11:22.519 breaks the scraper looking at the foot exact. And that 236 00:11:22.679 --> 00:11:25.960 is why no professional engineer uses absolute XPath unless they 237 00:11:25.960 --> 00:11:28.039 are absolutely forced to at gunpoint. 238 00:11:28.159 --> 00:11:29.240 So what's the alternative. 239 00:11:29.399 --> 00:11:34.039 The alternative, and the cornerstone of resilient automation is relative XPath. 240 00:11:34.480 --> 00:11:36.600 Instead of starting at the root of the document, we 241 00:11:36.679 --> 00:11:39.759 scan the chaotic dome for a nearby landmark that does 242 00:11:39.799 --> 00:11:41.080 have a stable identifier. 243 00:11:41.360 --> 00:11:44.240 Oh, I see we find an anchor. We basically say, 244 00:11:44.519 --> 00:11:46.960 I have no idea what the absolute coordinates of my 245 00:11:47.039 --> 00:11:49.440 target house are, but I know it is always sitting 246 00:11:49.480 --> 00:11:52.000 directly next to the giant permanent search bar. 247 00:11:52.159 --> 00:11:55.279 Yes, you locate that stable element and you calculate the 248 00:11:55.320 --> 00:11:58.200 path to your target relative to that anchor. So you say, 249 00:11:58.440 --> 00:12:00.759 find the search bar then traverse the dom tree to 250 00:12:00.799 --> 00:12:02.320 find its immediate sibling element. 251 00:12:02.480 --> 00:12:04.840 That makes the code highly robust. 252 00:12:04.679 --> 00:12:09.039 Incredibly robust. Because the broader page structure can be completely redesigned. 253 00:12:09.039 --> 00:12:11.799 They can add pop ups, change the headers, wrap the 254 00:12:11.799 --> 00:12:14.799 page in new containers. But as long as the relationship 255 00:12:14.799 --> 00:12:18.480 between your stable anchor and your target element remains intact, 256 00:12:19.080 --> 00:12:20.720 your relative x path holds strong. 257 00:12:21.240 --> 00:12:23.840 Okay, since we are visualizing this without screens today, let's 258 00:12:23.840 --> 00:12:26.679 decipher what a relative x path actually looks like when 259 00:12:26.720 --> 00:12:28.399 you type it out. If you look at a script, 260 00:12:28.399 --> 00:12:31.960 you'll usually see a basic format, something like a double 261 00:12:32.039 --> 00:12:36.360 slash tag named open bracket at attribute equals value, close bracket. 262 00:12:36.519 --> 00:12:36.639 Right. 263 00:12:36.879 --> 00:12:39.720 I don't want to bogg us down in basic HTML definitions, 264 00:12:39.759 --> 00:12:43.480 but can you translate that syntax into plain English for us? 265 00:12:43.919 --> 00:12:46.720 How does XPath bind those pieces together in a query? 266 00:12:47.080 --> 00:12:50.320 Sure? So it operates like a very strict database query. 267 00:12:50.840 --> 00:12:54.000 The double forward slash at the start is the command 268 00:12:54.159 --> 00:12:57.679 to select the current node bypassing the absolute route. Okay. 269 00:12:57.720 --> 00:13:00.399 It basically tells the engine scan any we're in the 270 00:13:00.440 --> 00:13:03.080 document for what comes next. Next is the tag name, 271 00:13:03.159 --> 00:13:06.799 which filters the search to only say input or button 272 00:13:06.840 --> 00:13:09.679 elements got it, and then the square brackets contain the 273 00:13:09.720 --> 00:13:12.600 specific condition. You use the ad symbol to target a 274 00:13:12.639 --> 00:13:15.879 specific attribute and check if it equals your defined value. 275 00:13:16.000 --> 00:13:20.399 Okay, So a query of double slash button open bracket 276 00:13:20.720 --> 00:13:25.080 at name equals submit close bracket. That's basically a machine 277 00:13:25.120 --> 00:13:29.159 readable sentence saying ignore the root structure. Scan the entire 278 00:13:29.159 --> 00:13:31.919 blueprint for any button, and filter for the one where 279 00:13:31.960 --> 00:13:34.080 the name attribute is exactly Submit. 280 00:13:34.360 --> 00:13:36.639 You nailed it. It's actually really elegant once you understand 281 00:13:36.639 --> 00:13:37.559 the syntax, it is. 282 00:13:37.759 --> 00:13:39.840 Okay, so we've mapped the chaotic showroom. We found our 283 00:13:39.879 --> 00:13:42.720 way around the ghost rearranging the furniture using our fallback 284 00:13:42.759 --> 00:13:46.759 locators and our relative XPath. We found the specific element. Now, 285 00:13:47.039 --> 00:13:49.360 how do we actually issue the command to our code 286 00:13:49.399 --> 00:13:50.120 to grab it? 287 00:13:50.399 --> 00:13:54.279 Right? Because pinpointing the element is only the setup, taking 288 00:13:54.320 --> 00:13:58.440 action requires you to choose between two fundamentally different execution methods. 289 00:13:59.240 --> 00:14:03.759 In most automateation frameworks, you have find element singular and 290 00:14:03.840 --> 00:14:08.200 find elements plural singular and plural exactly, and choosing the 291 00:14:08.240 --> 00:14:11.960 wrong one will dictate whether your script gracefully handles errors 292 00:14:12.320 --> 00:14:13.720 or just completely detonates. 293 00:14:14.000 --> 00:14:16.919 Let's start with the singular. The singular command find element 294 00:14:17.000 --> 00:14:21.080 is notorious for being incredibly literal and honestly incredibly lazy. 295 00:14:21.480 --> 00:14:24.840 You tell the code, go fetch the single unique element 296 00:14:24.960 --> 00:14:26.480 matching this locator. 297 00:14:26.159 --> 00:14:28.879 And it scans the DOM from top to bottom, and 298 00:14:28.919 --> 00:14:31.799 the millisecond it finds a single matching element, it grabs, 299 00:14:31.840 --> 00:14:34.480 It returns it to your script, completely stops looking. It 300 00:14:34.519 --> 00:14:35.840 just ignores the rest of the page. 301 00:14:36.080 --> 00:14:38.559 Right, which is fine if you use a unique ID. 302 00:14:38.799 --> 00:14:41.240 But wait, what if I accidentally use the singular command 303 00:14:41.600 --> 00:14:43.480 on something generic? Like what if I tell it to 304 00:14:43.519 --> 00:14:44.519 find an image tag? 305 00:14:44.600 --> 00:14:46.519 It just grabs the very first image it finds in 306 00:14:46.519 --> 00:14:49.039 the header and ignores the other fifty images on the page. 307 00:14:49.200 --> 00:14:49.639 Wow. 308 00:14:50.000 --> 00:14:53.159 Very But the real danger of the singular command isn't 309 00:14:53.159 --> 00:14:56.759 its laziness. It's it's temper tantrum when it fails. 310 00:14:57.080 --> 00:14:58.320 What happens when it fails? 311 00:14:58.480 --> 00:15:01.720 If the singular find element scans the DOM and finds 312 00:15:01.799 --> 00:15:05.919 absolutely nothing that matches your locator, it doesn't just return 313 00:15:05.960 --> 00:15:09.799 a blank value. It throws a no such element exception error. 314 00:15:09.919 --> 00:15:12.279 Ah, the wall of red test exactly. 315 00:15:12.399 --> 00:15:14.679 It triggers a hard stop and unless you've wrapped that 316 00:15:14.720 --> 00:15:19.279 command in complex error handling, logic. Your entire script halts 317 00:15:19.320 --> 00:15:21.080 execution and dies on the spot. 318 00:15:21.240 --> 00:15:24.080 It demands an object, or it throws a fit, which 319 00:15:24.080 --> 00:15:27.080 brings us to the plural command find elements. It operates 320 00:15:27.080 --> 00:15:28.519 on totally different logic, doesn't. 321 00:15:28.320 --> 00:15:31.519 It It does. The plural command is designed to identify 322 00:15:31.519 --> 00:15:34.720 a list or an array of web elements. It scans 323 00:15:34.759 --> 00:15:38.159 the entire document, gathers every single element that matches your locator, 324 00:15:38.320 --> 00:15:40.679 and returns them as a zero indexed list. 325 00:15:40.799 --> 00:15:42.919 So the first match is at index zero, the second 326 00:15:42.960 --> 00:15:44.879 at index one, and so forth exactly. 327 00:15:45.000 --> 00:15:47.279 But the safety met here is brilliant. If the plural 328 00:15:47.279 --> 00:15:50.559 command executes and the page is broken or the element 329 00:15:50.600 --> 00:15:51.559 simply doesn't. 330 00:15:51.279 --> 00:15:53.080 Exist today, it doesn't crash. 331 00:15:53.159 --> 00:15:56.279 It doesn't throw the exception. Nope, it just politely hands 332 00:15:56.279 --> 00:15:59.320 your script and empty list a list with zero items 333 00:15:59.360 --> 00:16:02.879 in it. Your code can easily evaluate that list, see 334 00:16:02.879 --> 00:16:05.960 that its length is zero, and smoothly pivot to a 335 00:16:05.960 --> 00:16:08.919 different set of instructions without throwing a fatal exception. 336 00:16:09.240 --> 00:16:12.399 Oh, I use this trick constantly for handling unpredictable pop ups. 337 00:16:12.519 --> 00:16:14.679 Yes, perfect use case, because you don't know if a 338 00:16:14.679 --> 00:16:17.360 promotional pop up will appear today. Right If you use 339 00:16:17.399 --> 00:16:19.519 the singular command to look for the close pop up 340 00:16:19.519 --> 00:16:22.799 button and there's no pop up, the script crashes, right. 341 00:16:23.080 --> 00:16:25.879 But if you use the plural command and it returns 342 00:16:25.919 --> 00:16:29.039 an empty list, your script says cool, no pop up 343 00:16:29.080 --> 00:16:32.399 today and keeps running. I mean understanding when to demand 344 00:16:32.399 --> 00:16:36.159 a guaranteed single object versus asking for a flexible list. 345 00:16:36.240 --> 00:16:40.399 That's really what separates junior script kitties from senior automation architects. 346 00:16:40.399 --> 00:16:45.159 Absolutely that one distinction dictates the stability of your entire pipeline. 347 00:16:45.480 --> 00:16:50.320 When you combine robust relative XPath mapping with intelligent plural fetching, 348 00:16:51.000 --> 00:16:53.519 you build scripts that can actually survive the chaos of 349 00:16:53.600 --> 00:16:57.759 dynamic web environments. What a deep dive today, Let's recap 350 00:16:57.759 --> 00:17:00.919 the journey here. We started by sting away the visual 351 00:17:00.960 --> 00:17:04.240 aesthetics of a web page to see the raw domn underneath. 352 00:17:04.720 --> 00:17:08.039 We explored the nightmare of dynamic mutating elements that break 353 00:17:08.119 --> 00:17:09.640 hard coded scripts. 354 00:17:09.319 --> 00:17:11.400 The ghost in the Ikia showroom right. 355 00:17:11.640 --> 00:17:14.559 And then we built an escalating strategy for finding things 356 00:17:14.640 --> 00:17:18.319 starting with IDs, managing the collateral damage of classes and tags, 357 00:17:18.519 --> 00:17:22.480 and utilizing CSS design roles. We learned why absolute XPath 358 00:17:22.559 --> 00:17:25.480 is a terrible set of driving directions and how relative 359 00:17:25.599 --> 00:17:28.279 XPath lets us anchor to permanent landmarks. 360 00:17:28.319 --> 00:17:31.440 And finally, we unpacked commanding the code with singular or 361 00:17:31.480 --> 00:17:33.279 plural fetch actions. 362 00:17:32.920 --> 00:17:37.440 Exactly the literal, unforgiving nature of singular fetch commands versus 363 00:17:37.480 --> 00:17:39.960 the robust safety of plural lists. 364 00:17:40.160 --> 00:17:42.720 And you know, the overarching takeaway here is really a 365 00:17:42.759 --> 00:17:45.240 fundamental shift in how you interact with the digital world. 366 00:17:45.799 --> 00:17:49.279 I mean, mastering these concepts transforms you from a passive 367 00:17:49.279 --> 00:17:53.200 consumer clicking through interfaces built by others into an active controller. 368 00:17:53.319 --> 00:17:55.880 You're learning to read the underlying matrix exactly. 369 00:17:55.960 --> 00:17:59.000 You're extracting data and automating actions at a scale that 370 00:17:59.079 --> 00:18:03.200 human hands simply cannot match. It's powerful stuff, it really is. 371 00:18:03.799 --> 00:18:07.119 So let's test that mastery with a quick mental exercise 372 00:18:07.119 --> 00:18:10.000 for you listening before we wrap up. Imagine you are 373 00:18:10.119 --> 00:18:13.279 building a scraper, right and you need your bot to 374 00:18:13.319 --> 00:18:16.640 click a specific submit button on a highly secure form. 375 00:18:16.759 --> 00:18:18.279 Okay, the same, but. 376 00:18:18.400 --> 00:18:22.039 Because of the security framework, the butt is ID, its class, 377 00:18:22.079 --> 00:18:25.559 and its name completely change every single time the page 378 00:18:25.640 --> 00:18:29.480 is refreshed. It is entirely dynamics. Sounds tricky, However, this 379 00:18:29.640 --> 00:18:33.880 mutating button is sitting directly beneath a standard username text field, 380 00:18:34.319 --> 00:18:37.079 and that username field happens to have a static, hard 381 00:18:37.119 --> 00:18:40.400 coded ID that never ever changes. Based on the logic 382 00:18:40.400 --> 00:18:43.839 we've discussed today, what is the only reliable locator method 383 00:18:43.880 --> 00:18:46.039 you should use to command your script to click that button? 384 00:18:46.079 --> 00:18:48.720 Well, the mechanics should be second nature by now. You 385 00:18:48.799 --> 00:18:51.640 deploy a relative X path. Ye, you curry the dam 386 00:18:51.759 --> 00:18:54.880 for the static username ID to establish your anchor, and 387 00:18:54.920 --> 00:18:58.039 then you map the relationship down one sibling node to 388 00:18:58.119 --> 00:18:59.720 strike the dynamic submit button. 389 00:19:00.079 --> 00:19:05.519 You use the permanent landmark to find the shifting target. Beautiful. Now, 390 00:19:05.960 --> 00:19:08.440 I want to leave you with the final provocative thought 391 00:19:08.440 --> 00:19:11.480 to mall over on your own. We've spent this entire 392 00:19:11.559 --> 00:19:15.680 time talking about mapping the DOM, deciphering tags, relying on 393 00:19:15.920 --> 00:19:19.319 X paths, and underlying code structure to guide. 394 00:19:19.160 --> 00:19:22.319 Our bots r forcing the machine to read the architectural 395 00:19:22.359 --> 00:19:23.559 blueprint exactly. 396 00:19:24.200 --> 00:19:27.759 But AI vision models are advancing at an incomprehensible rate. 397 00:19:27.839 --> 00:19:30.839 Right now, we are rapidly approaching a threshold where an 398 00:19:30.880 --> 00:19:33.319 AI agent doesn't need to read the DOM to scrape 399 00:19:33.319 --> 00:19:33.920 a website. 400 00:19:33.960 --> 00:19:36.119 Wait, really, it just looks at the rendered pixels on 401 00:19:36.160 --> 00:19:38.240