WEBVTT 1 00:00:00.040 --> 00:00:02.919 So imagine you've got this this digital assistant, and it 2 00:00:03.000 --> 00:00:06.639 literally just sits at your computer, opens up a web browser, 3 00:00:06.960 --> 00:00:09.359 and you does all your digital chores for you. 4 00:00:09.439 --> 00:00:11.439 Oh, that would be amazing. 5 00:00:11.000 --> 00:00:13.880 Right, I'm talking about like filling out those super tedious 6 00:00:13.960 --> 00:00:18.519 multi page onboarding forms, or clicking through just endless confirmation buttons, 7 00:00:18.640 --> 00:00:22.359 or even scouring databases for specific information, all at lightning 8 00:00:22.399 --> 00:00:25.000 speed while you just go grab a coffee. 9 00:00:25.079 --> 00:00:27.440 Yeah. I mean, it sounds like science fiction or at least, 10 00:00:27.440 --> 00:00:30.640 you know, a very expensive enterprise software solution that most 11 00:00:30.640 --> 00:00:32.000 people wouldn't have access to. 12 00:00:32.280 --> 00:00:35.399 Exactly, But it's actually just standard web automation, it is. 13 00:00:35.640 --> 00:00:37.520 And the crazy thing about it is to pull this off, 14 00:00:38.000 --> 00:00:40.799 you kind of have to completely unlearn how you view 15 00:00:40.799 --> 00:00:43.479 the Internet. What do you mean, Well, to a piece 16 00:00:43.520 --> 00:00:46.880 of automation code, the Internet has no colors, it has 17 00:00:46.920 --> 00:00:50.719 no shiny buttons, no pretty layouts. To automate the web, 18 00:00:50.880 --> 00:00:54.759 we have to look past the user interface. 19 00:00:54.520 --> 00:00:58.560 Entirely, right, You're interacting directly with the raw architecture underneath exactly. 20 00:00:58.600 --> 00:01:01.000 You're dealing with the skeleton of the site, not the 21 00:01:01.039 --> 00:01:01.600 paint job. 22 00:01:01.880 --> 00:01:05.159 So today, for this deep dive, we are basically going 23 00:01:05.200 --> 00:01:09.400 to act as the translators between human intention and machine execution. 24 00:01:10.200 --> 00:01:12.760 We're going to walk you through step by step how 25 00:01:12.840 --> 00:01:16.560 a script actually parses and manipulates the web page. 26 00:01:16.680 --> 00:01:20.040 Yeah, and we'll be talking specifically about using tools like 27 00:01:20.079 --> 00:01:23.760 Selenium and uh something called Chrome Driver. 28 00:01:24.120 --> 00:01:26.920 Right, So let's start with the ignition key. Basically, to 29 00:01:26.920 --> 00:01:29.400 have an invisible assistant click on anything, we need a 30 00:01:29.400 --> 00:01:31.760 way for our Python code to talk to the browser 31 00:01:31.799 --> 00:01:32.439 in the first place. 32 00:01:32.560 --> 00:01:35.040 Yeah, and that is exactly the role of the Chrome Driver. 33 00:01:35.280 --> 00:01:37.480 I mean, think of it as an executable application that 34 00:01:37.519 --> 00:01:38.120 acts as a. 35 00:01:38.040 --> 00:01:40.640 Bridge, okay, a bridge between what and what. 36 00:01:40.840 --> 00:01:44.079 Between your script and the browser. So your script sends 37 00:01:44.079 --> 00:01:47.040 commands to the driver, and then the driver translates those 38 00:01:47.040 --> 00:01:49.719 commands into native browser actions. 39 00:01:49.760 --> 00:01:50.079 Got it. 40 00:01:50.319 --> 00:01:53.200 When you instantiate this driver in your code, an actual 41 00:01:53.359 --> 00:01:56.959 instance of Chrome boots up on your machine, but it's 42 00:01:57.120 --> 00:01:59.359 entirely controlled by your script. You can literally sit there 43 00:01:59.359 --> 00:02:01.120 and watch it move on on its own, which. 44 00:02:00.959 --> 00:02:02.840 Is always a little spooky the first time you see it. 45 00:02:02.840 --> 00:02:03.519 It really is. 46 00:02:03.760 --> 00:02:06.439 Now, one of the first things a developer typically does 47 00:02:06.480 --> 00:02:09.400 after booting that instance is they pass an argument to 48 00:02:09.479 --> 00:02:11.479 maximize the browser window right away. 49 00:02:11.879 --> 00:02:14.840 Yes, that is a crucial step, but on. 50 00:02:14.759 --> 00:02:18.719 The surface that seems I don't know, purely cosmetic, like 51 00:02:19.080 --> 00:02:20.960 why would the script care how big the window is 52 00:02:21.000 --> 00:02:23.599 if it's just reading the underlying code anyway? 53 00:02:23.680 --> 00:02:27.120 Well, functionally, it's all about responsive design. I mean, modern 54 00:02:27.120 --> 00:02:30.199 web pages are built to change their structural layout based 55 00:02:30.199 --> 00:02:31.759 on the size of the screen you're using. 56 00:02:31.879 --> 00:02:34.360 Oh right, Like sites look different on a phone versus 57 00:02:34.360 --> 00:02:35.919 a desktop exactly. 58 00:02:36.159 --> 00:02:38.960 So if you load a complex web app in a tiny, 59 00:02:39.120 --> 00:02:42.719 minimized window, the site's code might dynamically hide at a 60 00:02:42.800 --> 00:02:45.759 navigation menu. It might shove it behind one of those 61 00:02:46.719 --> 00:02:48.599 mobile style Hamburger icons. 62 00:02:48.680 --> 00:02:50.479 Oh wow, I see where this is going. 63 00:02:50.800 --> 00:02:53.800 Yeah, So if your script is hunting for a specific 64 00:02:53.840 --> 00:02:56.639 settings button that has suddenly been hidden or removed from 65 00:02:56.639 --> 00:02:57.280 the active. 66 00:02:57.080 --> 00:02:59.560 Layout, the automation just crashes, right, It. 67 00:02:59.520 --> 00:03:01.719 Throws an error because the button isn't there anymore. So 68 00:03:02.120 --> 00:03:06.360 maximizing the window ensures the layout is predictable, It ensures 69 00:03:06.439 --> 00:03:10.599 all the standard desktop elements are rendered and you know, 70 00:03:10.759 --> 00:03:12.400 actually available to interact with. 71 00:03:12.599 --> 00:03:15.120 That makes total sense. So we've set the stage, we've 72 00:03:15.120 --> 00:03:18.240 maximized the window, and we've used to command like a driver, 73 00:03:18.319 --> 00:03:21.039 do get to navigate to a specific url. 74 00:03:21.159 --> 00:03:22.439 Yep, the page loads. 75 00:03:22.680 --> 00:03:25.280 Now to a human user, a search bar is obvious. 76 00:03:25.280 --> 00:03:27.599 It's a white rectangle with a little magnifine glass next 77 00:03:27.599 --> 00:03:27.800 to it. 78 00:03:28.000 --> 00:03:28.919 Right, you just click it. 79 00:03:28.960 --> 00:03:31.680 But since our script doesn't have eyes, how does it 80 00:03:31.719 --> 00:03:37.759 actually pinpoint that specific interactive element out of like thousands 81 00:03:37.800 --> 00:03:39.479 of lines of raw HTML code. 82 00:03:39.560 --> 00:03:44.240 It relies entirely on the Document object model, or the 83 00:03:44.319 --> 00:03:46.680 DOM as most people call it the DOM. Okay, Yeah, 84 00:03:46.719 --> 00:03:48.639 when a browser loads a web page, it takes the 85 00:03:49.000 --> 00:03:53.080 raw HTML text and turns it into this giant inverted tree. 86 00:03:52.879 --> 00:03:54.800 Structure, like a family tree sort of. 87 00:03:54.919 --> 00:03:58.159 Yeah, the page itself is the trunk, and every container, 88 00:03:58.199 --> 00:04:00.639 every paragraph, every button is a brand or a leaf 89 00:04:00.680 --> 00:04:01.240 on that tree. 90 00:04:01.319 --> 00:04:02.520 Okay, I'm visualizing that. 91 00:04:02.800 --> 00:04:05.759 So the script navigates this tree. If you want to 92 00:04:05.800 --> 00:04:09.319 find a specific element, you need a locator strategy, and 93 00:04:09.400 --> 00:04:12.240 the absolute most reliable strategy is looking for a unique 94 00:04:12.240 --> 00:04:15.360 identifier that the original developers left behind. 95 00:04:15.639 --> 00:04:18.160 It's kind of like giving our script the exact GPS 96 00:04:18.240 --> 00:04:20.720 coordinates of a specific house in a sprawling city. 97 00:04:20.879 --> 00:04:22.040 That is a perfect analogy. 98 00:04:22.040 --> 00:04:24.319 Actually, so if the developers built the site, well, they 99 00:04:24.360 --> 00:04:26.800 assign these unique attributes to elements. 100 00:04:27.040 --> 00:04:27.240 Yeah. 101 00:04:27.399 --> 00:04:30.959 Like if you use a browsers and spectool on a 102 00:04:31.000 --> 00:04:34.160 well designed site's search bar, you might see an attribute 103 00:04:34.199 --> 00:04:37.000 called ID, right, like i'd search field. 104 00:04:37.160 --> 00:04:40.439 Yes, and an ID is the gold standard for locators. 105 00:04:40.480 --> 00:04:43.839 Why is it the gold standard because by definition in 106 00:04:43.920 --> 00:04:47.199 standard HTML rules, an ID is supposed to be entirely 107 00:04:47.319 --> 00:04:49.759 unique on a single web page. It is literally like 108 00:04:49.800 --> 00:04:51.879 a social security number for a web element. 109 00:04:52.000 --> 00:04:53.879 Wow. Okay, so it's foolproof pretty much. 110 00:04:54.079 --> 00:04:58.240 If you command your script to findelement bide and pass 111 00:04:58.279 --> 00:05:01.480 it that specific string that driver searches, the dom tree 112 00:05:01.560 --> 00:05:04.759 instantly locks onto that specific node and completely ignores the 113 00:05:04.800 --> 00:05:05.439 rest of the page. 114 00:05:05.680 --> 00:05:09.319 That's super efficient. But what if the developer didn't use 115 00:05:09.360 --> 00:05:12.600 an ID, because we all know not every site is 116 00:05:12.639 --> 00:05:13.439 coded perfectly. 117 00:05:13.560 --> 00:05:15.000 Oh absolutely, it happens all the time. 118 00:05:15.160 --> 00:05:17.120 Like sometimes you inspect an element and it just has 119 00:05:17.160 --> 00:05:19.600 a name attribute, Like the main search box on a 120 00:05:19.639 --> 00:05:22.680 coding tutorial site might just have the name attribute assigned 121 00:05:22.680 --> 00:05:23.199 to the letter Q. 122 00:05:23.560 --> 00:05:28.040 Right. Q usually stands for a query and the fidelment 123 00:05:28.079 --> 00:05:32.319 by name command works very similarly to ID. It's incredibly fast. 124 00:05:32.439 --> 00:05:34.360 But I hear a butt coming. 125 00:05:34.800 --> 00:05:38.079 But name attributes aren't strictly required to be unique. 126 00:05:38.279 --> 00:05:38.959 Ah, there it is. 127 00:05:39.120 --> 00:05:42.319 Yeah. So if a developer got sloppy and named both 128 00:05:42.439 --> 00:05:46.839 the top search bar and say a bottom newsletter sign 129 00:05:46.879 --> 00:05:47.439 up form Q. 130 00:05:47.839 --> 00:05:49.040 Then your script might grab the. 131 00:05:49.040 --> 00:05:52.600 Wrong one exactly, because the script generally just defaults to 132 00:05:52.639 --> 00:05:54.680 the very first match and encounters as it scans the 133 00:05:54.720 --> 00:05:55.879 document from top to bottom. 134 00:05:55.920 --> 00:05:58.600 Oh right, so it hits the first que, says found it, 135 00:05:58.959 --> 00:06:02.160 and stops looking. Okay, so IDs and names are the 136 00:06:02.160 --> 00:06:05.360 easy street addresses, But we also have to navigate around 137 00:06:05.399 --> 00:06:07.600 the site. Say I want the script to click a 138 00:06:07.639 --> 00:06:10.120 specific text link that just says my courses. 139 00:06:10.480 --> 00:06:14.279 Well, Selenium provides a findelement by link text method for 140 00:06:14.360 --> 00:06:17.040 exactly this scenario. Oh nice, Yeah, you just passed the 141 00:06:17.040 --> 00:06:19.240 text of the link to the script, and it hunts 142 00:06:19.240 --> 00:06:21.800 for an anchor tag in the HTML that contains that 143 00:06:21.879 --> 00:06:22.639 exact string. 144 00:06:23.079 --> 00:06:25.480 I imagine that demand's absolute precision, though. 145 00:06:25.560 --> 00:06:26.439 Oh it really does. 146 00:06:26.560 --> 00:06:28.319 It's like telling a taxi driver to take you to 147 00:06:28.360 --> 00:06:31.959 the Grand Hotel and they flat out refuse to move 148 00:06:32.319 --> 00:06:35.040 because the physical sign on the building actually says the 149 00:06:35.079 --> 00:06:36.519 Grand Hotel Incorporated. 150 00:06:36.720 --> 00:06:41.360 Yes, the machine has zero intuition. It is executing a 151 00:06:41.560 --> 00:06:43.000 literal string comparison. 152 00:06:43.240 --> 00:06:45.839 So if I tell the script to look for my courses, 153 00:06:46.439 --> 00:06:49.879 but the web designer capitalized it to my courses, yeah, 154 00:06:49.879 --> 00:06:50.920 in all caps. 155 00:06:50.639 --> 00:06:52.720 The script will just crash. It'll throw a no such 156 00:06:52.720 --> 00:06:56.800 element exception, A single mismatch space, an invisible line break, 157 00:06:57.240 --> 00:06:58.879 or different capitalization will break it. 158 00:06:59.000 --> 00:07:00.600 That sounds incredibly fragile. 159 00:07:00.720 --> 00:07:03.319 It can be. But to mitigate that fragility, you can 160 00:07:03.399 --> 00:07:05.959 use a variation called partial link text. 161 00:07:06.040 --> 00:07:08.199 Oh, or you only need a piece of it exactly. 162 00:07:08.279 --> 00:07:10.279 You only match a substring. So if the link is 163 00:07:10.319 --> 00:07:14.600 a dynamically generated product title like blue Sneaker size ten, 164 00:07:14.759 --> 00:07:17.000 ID four four five nine, you can just target the 165 00:07:17.040 --> 00:07:18.360 blue Sneakers portion. 166 00:07:18.519 --> 00:07:21.199 Okay, So that ensures the script finds it even if 167 00:07:21.240 --> 00:07:23.879 the ID numbers or the sizes change tomorrow. 168 00:07:23.959 --> 00:07:25.439 Right. It gives you a lot more flexibility. 169 00:07:25.560 --> 00:07:29.040 Got it. So far we've relied on developers giving us 170 00:07:29.079 --> 00:07:33.079 clean IDs, names or exact text links. But what if 171 00:07:33.079 --> 00:07:37.040 we're automating some highly complex, deeply nested web app where 172 00:07:37.040 --> 00:07:39.360 the elements don't have any of those clean attributes. 173 00:07:39.519 --> 00:07:41.000 Yeah, the tricky stuff. 174 00:07:41.040 --> 00:07:42.639 How do we find the house if there's literally no 175 00:07:42.680 --> 00:07:43.839 address on the mailbox? 176 00:07:44.000 --> 00:07:47.680 This is where we shift from simple addresses to architectural blueprints. 177 00:07:48.120 --> 00:07:52.480 When basic locators fail, we rely on advanced locators, specifically 178 00:07:52.639 --> 00:07:55.199 CSS selectors and XPath Okay, let's. 179 00:07:55.000 --> 00:07:57.560 Start with CSS now. CSS is what dictates the visual 180 00:07:57.600 --> 00:08:01.519 styling of a page, right like colors, padding, Yes, exactly, 181 00:08:01.600 --> 00:08:04.680 and developers apply these styles by assigning elements to classes. 182 00:08:04.759 --> 00:08:06.519 So if a button is red and round, it might 183 00:08:06.519 --> 00:08:10.040 have a class called primary submit btn. We can actually 184 00:08:10.079 --> 00:08:11.920 hijack those styling hooks for automation. 185 00:08:12.360 --> 00:08:16.639 You absolutely can. Using a CSS selector locator, you can 186 00:08:16.680 --> 00:08:19.680 command the script to find an element based entirely on 187 00:08:19.720 --> 00:08:20.560 how it was styled. 188 00:08:20.680 --> 00:08:21.399 That's so clever. 189 00:08:21.680 --> 00:08:24.600 Yeah, in your code, you use standard CSS syntax like 190 00:08:24.639 --> 00:08:26.879 a dot to denote a class, so you'd tell the 191 00:08:26.959 --> 00:08:30.600 driver to find dot primary dash submit dash btn. 192 00:08:30.720 --> 00:08:31.480 And is that fast? 193 00:08:31.720 --> 00:08:34.240 It is incredibly fast. The beauty of this is that 194 00:08:34.320 --> 00:08:38.120 CSS selectors are heavily optimized by the browser itself. 195 00:08:37.960 --> 00:08:41.320 But CSS classes still rely on the developer leaving a 196 00:08:41.360 --> 00:08:45.159 recognizable hook. Sure, what if there was absolutely nothing unique 197 00:08:45.159 --> 00:08:48.440 about the element itself, no class, no ID. That's where 198 00:08:48.559 --> 00:08:51.080 XPath comes in, right, Yes, because I've seen complex XPath 199 00:08:51.159 --> 00:08:53.799 strings and honestly, they look like absolute gibberish. It's just 200 00:08:54.120 --> 00:08:57.399 like slashes, brackets and random HTML tags. 201 00:08:57.480 --> 00:09:00.759 They do look intimidating. But XPath is the heavy artillery 202 00:09:00.799 --> 00:09:04.320 of web automation. It stands for XML path language. Okay, 203 00:09:04.519 --> 00:09:06.759 remember how we describe the dom as an inverted tree. 204 00:09:07.039 --> 00:09:09.200 This path is a literal set of step by step 205 00:09:09.240 --> 00:09:12.159 directions for navigating the branches of that tree. You can 206 00:09:12.279 --> 00:09:14.879 use it to triangulate an element based entirely on its 207 00:09:14.919 --> 00:09:16.240 relationship to other elements. 208 00:09:16.440 --> 00:09:18.240 How does that actually map out in the code though? 209 00:09:18.480 --> 00:09:21.799 Okay, imagine you want to click a specific input box, 210 00:09:22.000 --> 00:09:25.559 but it has no ID, no name, and a generic class. 211 00:09:25.720 --> 00:09:26.840 Basically a ghost element. 212 00:09:27.039 --> 00:09:30.480 Right, But you notice it sits inside a larger container, 213 00:09:30.519 --> 00:09:33.279 a divtag that does have an idea of login panel. 214 00:09:33.360 --> 00:09:36.519 Okay, so the parent container has a clear address exactly. 215 00:09:37.120 --> 00:09:40.399 Using XPath, you can write a path that says, scan 216 00:09:40.519 --> 00:09:43.879 the document for a div with the ID log in panel. 217 00:09:44.159 --> 00:09:47.440 Once you find it, step inside that container, find the 218 00:09:47.480 --> 00:09:51.519 second paragraph, and give me the input box inside that paragraph. Wow, 219 00:09:51.679 --> 00:09:54.080 it doesn't matter if the input box has no identifying 220 00:09:54.120 --> 00:09:57.320 features of its own. You've located it purely based on 221 00:09:57.360 --> 00:09:58.200 its surroundings. 222 00:09:58.240 --> 00:10:00.320 It's literally like saying, go to the house house on 223 00:10:00.320 --> 00:10:02.559 the corner of Fifth and Main, walk to the kitchen 224 00:10:02.720 --> 00:10:05.200 and grab the second drawer on the left. Yes, you 225 00:10:05.240 --> 00:10:06.799 don't need a label in the drawer if you know 226 00:10:06.879 --> 00:10:08.080 the exact path to reach it. 227 00:10:08.200 --> 00:10:11.000 That is the power of x path. It can navigate 228 00:10:11.080 --> 00:10:13.320 down from parent to child, or even up from a 229 00:10:13.399 --> 00:10:16.679 child to a parent, or sideways across sibling elements. It's 230 00:10:16.679 --> 00:10:17.759 incredibly robust. 231 00:10:17.919 --> 00:10:21.000 All right, So we've mastered finding elements using the dom tree. 232 00:10:21.360 --> 00:10:24.159 But just pointing at a search bar doesn't execute a search. 233 00:10:24.320 --> 00:10:26.159 We have to interact with it, right. 234 00:10:26.240 --> 00:10:29.159 Once the driver locates the element, it stores it as 235 00:10:29.200 --> 00:10:31.840 an object in your Python code, okay, and you can 236 00:10:31.879 --> 00:10:35.120 then call methods on that object. So to type into 237 00:10:35.159 --> 00:10:37.519 a field, you use the sen keys command. Is a 238 00:10:37.559 --> 00:10:39.799 keys You pass it a string of texts, and the 239 00:10:39.879 --> 00:10:43.519 driver basically simulates the electrical signals of a human physically 240 00:10:43.600 --> 00:10:47.600 typing on a keyboard, sending those exact keystrokes directly into 241 00:10:47.600 --> 00:10:48.480 the input node. 242 00:10:48.679 --> 00:10:51.960 Wait if it just sends keystrokes blindly, what happens if 243 00:10:51.960 --> 00:10:55.840 the search bar already has pre populated text, Like sometimes 244 00:10:55.840 --> 00:10:57.879 you visit a portal and the search box already says 245 00:10:57.960 --> 00:11:01.559 search by employee name and light gray text. If I 246 00:11:01.639 --> 00:11:03.799 just fire off send keys with the name John, does 247 00:11:03.840 --> 00:11:07.000 it just appendit like? Does it search for search by 248 00:11:07.080 --> 00:11:08.480 employee name John. 249 00:11:08.840 --> 00:11:11.919 It absolutely will do exactly that. Oh no, yeah, send 250 00:11:12.000 --> 00:11:15.360 keys doesn't overwrite existing values, it appends to them. This 251 00:11:15.399 --> 00:11:17.759 is a classic architectural trap. 252 00:11:17.639 --> 00:11:22.639 Because humans instinctively recognize placeholder text right we backspace it 253 00:11:22.879 --> 00:11:24.639 or we highlight it before typing. 254 00:11:24.559 --> 00:11:28.759 Exactly, but the machine does not. Therefore, robust automation requires 255 00:11:28.799 --> 00:11:33.080 explicit instruction. Before you ever send new keys to a 256 00:11:33.120 --> 00:11:37.519 text input, you invoke the dot clear method yeap. This 257 00:11:37.600 --> 00:11:40.120 commands the driver to wipe the input node clean of 258 00:11:40.200 --> 00:11:43.919 any pre existing string values. You locate it, you clear it, 259 00:11:44.120 --> 00:11:45.840 and only then do you send your keys. 260 00:11:46.559 --> 00:11:50.039 Man, it really emphasizes how much micrologic goes into our 261 00:11:50.159 --> 00:11:53.000 daily browsing habits. Stuff we don't even think about. 262 00:11:53.080 --> 00:11:54.639 We take it totally for granted. 263 00:11:54.799 --> 00:11:57.240 So we've cleared the field and typed our query. To 264 00:11:57.279 --> 00:12:00.399 actually run the search, we need to get enter. How 265 00:12:00.399 --> 00:12:00.960 do we do that? 266 00:12:01.080 --> 00:12:03.919 You import a specific keys class in your script. This 267 00:12:03.960 --> 00:12:07.799 class contains representations of all the non alphanumeric keyboard. 268 00:12:07.519 --> 00:12:09.720 Keys, like shift, alt enter all that. 269 00:12:09.919 --> 00:12:13.080 Exactly, So you just deppend keys, dot enter or keys 270 00:12:13.200 --> 00:12:16.159 return to your sen keys command. The driver simulates that 271 00:12:16.200 --> 00:12:19.639 specific keypress, the web server receives the form submission, and boom, 272 00:12:19.639 --> 00:12:20.919 your automated search is running. 273 00:12:21.080 --> 00:12:23.679 That is so cool. Now, this all works beautifully in 274 00:12:23.720 --> 00:12:26.240 a vacuum. But let's look at the real world, where 275 00:12:26.240 --> 00:12:29.960 the Internet is messy, connections are slow, and web apps 276 00:12:30.000 --> 00:12:31.399 are incredibly complex. 277 00:12:31.559 --> 00:12:35.320 Oh yes, the real world is a nightmare for automation, right. 278 00:12:35.759 --> 00:12:39.679 What happens when our script encounters, say, a multi step 279 00:12:39.840 --> 00:12:42.320 sign up process with a really heavy loading screen. 280 00:12:42.639 --> 00:12:45.559 This is where we hit the fundamental friction of web automation. 281 00:12:46.039 --> 00:12:49.399 It's the speed gap between the CPU and the network. 282 00:12:49.519 --> 00:12:50.279 The speed gap. 283 00:12:50.480 --> 00:12:54.360 Yeah, your Python script executes commands in fractions of a millisecond. 284 00:12:54.720 --> 00:12:57.039 It tells the driver to click next. The browser sends 285 00:12:57.080 --> 00:13:00.000 a request to the server, and the server takes say two, 286 00:13:00.000 --> 00:13:02.480 two full seconds to respawn and draw the new page. 287 00:13:02.759 --> 00:13:05.159 And during those two seconds, our script is already trying 288 00:13:05.159 --> 00:13:07.320 to execute the next line of code right yep, which 289 00:13:07.399 --> 00:13:10.120 is filling out the first name field on a page 290 00:13:10.120 --> 00:13:12.120 that literally doesn't exist yet exactly. 291 00:13:12.360 --> 00:13:15.440 The script checks the dom doesn't see the first name 292 00:13:15.480 --> 00:13:19.519 element and instantly throws a fatal no such element exception. 293 00:13:19.720 --> 00:13:20.519 It just crashes. 294 00:13:20.799 --> 00:13:23.519 Okay, So the obvious brute force fix would be to 295 00:13:23.600 --> 00:13:26.759 just tell the script a pause, like use a sleep 296 00:13:26.759 --> 00:13:29.360 command to freeze the code for five seconds after every 297 00:13:29.399 --> 00:13:31.039 single click, just to be safe. 298 00:13:31.120 --> 00:13:33.679 You could do that, but static sleeps are a terrible 299 00:13:33.720 --> 00:13:34.799 engineering practice. 300 00:13:34.840 --> 00:13:36.799 Really why well, think about it. 301 00:13:36.960 --> 00:13:39.200 If you tell the code to sleep for ten seconds 302 00:13:39.240 --> 00:13:42.200 and the page loads in two seconds, you are wasting 303 00:13:42.279 --> 00:13:43.720 eight seconds of compute time. 304 00:13:43.919 --> 00:13:44.480 Oh true. 305 00:13:44.600 --> 00:13:47.399 If you scale that script up to process a ten 306 00:13:47.440 --> 00:13:50.320 thousand records, you've just added hours of dead air. 307 00:13:50.440 --> 00:13:51.480 That's highly inefficient. 308 00:13:51.840 --> 00:13:54.480 And conversely, if the server is having a slow day 309 00:13:54.480 --> 00:13:57.679 and takes eleven seconds to load, your script still crashes anyway. 310 00:13:57.759 --> 00:14:00.039 Oh man, Yeah, so we need a dynamic way to 311 00:14:00.080 --> 00:14:03.000 bridge the CPU speed and then network speed. 312 00:14:03.240 --> 00:14:05.759 Right, And the elegant solution here is something called an 313 00:14:05.799 --> 00:14:09.120 explicit weight. It's often implemented as web driver weight. 314 00:14:09.360 --> 00:14:11.279 How does that differ from just sleeping. 315 00:14:11.200 --> 00:14:14.480 Instead of freezing the code entirely? You define a maximum 316 00:14:14.480 --> 00:14:17.759 time out, say fifteen seconds, and you define a specific 317 00:14:17.799 --> 00:14:21.120 condition like wait until the first name element is clickable. 318 00:14:21.399 --> 00:14:22.960 So how does that look under the hood? Is it 319 00:14:23.039 --> 00:14:24.360 just checking constantly? 320 00:14:24.639 --> 00:14:28.399 It utilizes a polling mechanism. The driver hooks into the 321 00:14:28.399 --> 00:14:32.039 process and checks the DOM tree every five hundred milliseconds. 322 00:14:32.240 --> 00:14:34.840 Oh wow, it asks, is the element here yet? 323 00:14:34.960 --> 00:14:35.120 No? 324 00:14:35.440 --> 00:14:38.720 Half a second later, is it here yet? No? The 325 00:14:38.840 --> 00:14:43.120 absolute millisecond. The server finishes rendering the element and attaches 326 00:14:43.159 --> 00:14:45.200 it to the DOM. The driver detects it and then 327 00:14:45.200 --> 00:14:48.000 he just goes instantly. It breaks the weight sequence and 328 00:14:48.080 --> 00:14:52.080 executes the next command. It completely optimizes the script's efficiency. 329 00:14:52.159 --> 00:14:55.960 That is brilliant. It ties the execution pace strictly to 330 00:14:56.039 --> 00:15:00.279 the application's actual state, not just arbitrary guessing. Exactly all right, 331 00:15:00.360 --> 00:15:03.279 So our weight command worked. The page is loaded and 332 00:15:03.279 --> 00:15:06.679 we're looking at a registration form. Normal text fields are easy. 333 00:15:06.720 --> 00:15:09.879 Now we locate clear and send keys. But then we 334 00:15:09.960 --> 00:15:12.320 hit the country of residence field and it's a drop 335 00:15:12.360 --> 00:15:12.919 down menus. 336 00:15:13.000 --> 00:15:16.440 Oh, drop down menus. They reveal how deceptive graphical user 337 00:15:16.440 --> 00:15:17.600 interfaces really are. 338 00:15:17.720 --> 00:15:18.200 What do you mean? 339 00:15:18.240 --> 00:15:20.519 As users? We just see a single seamless box that 340 00:15:20.559 --> 00:15:23.240 we click and a nice list drops down. But the 341 00:15:23.320 --> 00:15:26.679 script bypasses the UI completely. It looks at the HTML 342 00:15:26.720 --> 00:15:28.639 document and sees a completely different structure. 343 00:15:28.960 --> 00:15:33.000 Right, Because HTML strictly defines how data is nested. A 344 00:15:33.080 --> 00:15:35.960 dropdown isn't just one element, is it. It's a parent 345 00:15:36.000 --> 00:15:38.399 container holding dozens of child elements. 346 00:15:38.519 --> 00:15:41.879 Precisely. It is typically built using a select tag for 347 00:15:42.039 --> 00:15:46.519 the main container, and inside that container are individual option 348 00:15:46.679 --> 00:15:50.080 tags for every single choice, one for Canada, one for 349 00:15:50.120 --> 00:15:51.879 the UK, one for the United States. 350 00:15:52.240 --> 00:15:53.519 So you can't just type into it. 351 00:15:53.919 --> 00:15:56.679 No, you cannot just use send keys on the select 352 00:15:56.720 --> 00:15:59.320 container because it's not a text input field. 353 00:15:59.440 --> 00:16:02.039 So we have to somehow tell the script to dig 354 00:16:02.080 --> 00:16:05.840 inside that container, sift through all the option tags and 355 00:16:05.919 --> 00:16:07.639 pick the right one. How do we do that? 356 00:16:08.159 --> 00:16:10.919 First you use a locator to isolate the main select 357 00:16:11.000 --> 00:16:13.440 dropdown element. Once you have that object, you run a 358 00:16:13.440 --> 00:16:17.000 command called finned elements plural with an s okay plural yes, 359 00:16:17.360 --> 00:16:20.080 and you tell it to find every single child element 360 00:16:20.120 --> 00:16:21.120 with the tag name option. 361 00:16:21.240 --> 00:16:23.559 Because it's plural, it doesn't just stop at the first 362 00:16:23.639 --> 00:16:26.240 match like before. It grabs every single one of them. 363 00:16:26.320 --> 00:16:26.720 Exactly. 364 00:16:26.759 --> 00:16:28.080 What does it do with all that data? 365 00:16:28.200 --> 00:16:30.919 It extracts them from the dom and stores them in 366 00:16:30.960 --> 00:16:33.559 your script's memory as a list like an array of 367 00:16:33.600 --> 00:16:36.080 web element objects. So if there are one hundred and 368 00:16:36.120 --> 00:16:38.759 ninety five countries in that dropdown, your script is now 369 00:16:38.759 --> 00:16:41.759 holding an array of one hundred and ninety five distinct objects. 370 00:16:41.840 --> 00:16:44.639 Okay, so the script has a massive bucket full of 371 00:16:44.799 --> 00:16:48.080 country options. How does it isolate just the United States 372 00:16:48.120 --> 00:16:49.240 option so it can click it. 373 00:16:49.799 --> 00:16:53.200 You employ a standard programming loop, afore loop. You instruct 374 00:16:53.200 --> 00:16:55.080 your code to iterate. 375 00:16:54.720 --> 00:16:56.960 Over that array, so it just checks some one by one. 376 00:16:57.120 --> 00:17:00.600 Basically, the script takes the first element, reads it's internal 377 00:17:00.639 --> 00:17:04.640 text attribute, and asks a logical question, does this text 378 00:17:04.680 --> 00:17:05.960 equal United States? 379 00:17:06.039 --> 00:17:06.799 And if it doesn't? 380 00:17:07.000 --> 00:17:09.799 If false, it discards it and checks the second element. 381 00:17:09.839 --> 00:17:11.960 It loops through the entire array at the speed of 382 00:17:12.000 --> 00:17:15.519 the processor, verifying the text of each option tag. 383 00:17:15.319 --> 00:17:17.839 In the moment it hits the true condition. The moment 384 00:17:17.880 --> 00:17:21.000 the text matches exactly, we trigger the doc click method 385 00:17:21.079 --> 00:17:22.839 on that specific object exactly. 386 00:17:22.839 --> 00:17:25.559 It scans the array, finds the match, executes the click, 387 00:17:25.559 --> 00:17:28.240 and breaks the loop. The browser receives the click event 388 00:17:28.279 --> 00:17:30.880 on that specific option note, and the drop down selection 389 00:17:31.000 --> 00:17:32.000 updates on the screen. 390 00:17:32.319 --> 00:17:36.440 It is a wildly methodical process to achieve what takes 391 00:17:36.440 --> 00:17:38.480 a human like a flick of the wrist. 392 00:17:38.839 --> 00:17:39.599 It really is. 393 00:17:39.759 --> 00:17:42.640 It really pulls back the curtain on how complex the 394 00:17:42.720 --> 00:17:43.640 underlying web is. 395 00:17:43.880 --> 00:17:44.000 Ye. 396 00:17:44.240 --> 00:17:48.599 I mean we've mastered finding elements, typing, waiting for a 397 00:17:48.680 --> 00:17:52.839 synchrodous network loads, and looping through complex data arrays. Our 398 00:17:52.880 --> 00:17:55.119 script must be practically unstoppable. 399 00:17:55.160 --> 00:17:58.319 At this point you'd think so, wouldn't you. But modern 400 00:17:58.359 --> 00:18:02.000 web architecture has a few security features and structural quarks 401 00:18:02.039 --> 00:18:05.240