WEBVTT 1 00:00:00.080 --> 00:00:04.080 Picture this. You've just spent the entire afternoon building a 2 00:00:04.080 --> 00:00:04.759 web scraper. 3 00:00:04.879 --> 00:00:06.879 Oh yeah, a very familiar feeling. 4 00:00:06.719 --> 00:00:09.080 Right, You have your logic dialed in, you're ready to 5 00:00:09.080 --> 00:00:12.480 gather this incredible data set. You hit run, and instantly 6 00:00:12.919 --> 00:00:16.760 your scraper just smacks face first into a massive digital 7 00:00:16.800 --> 00:00:17.359 brick wall. 8 00:00:17.519 --> 00:00:18.359 A log in page. 9 00:00:18.399 --> 00:00:19.839 Exactly, a log in page. 10 00:00:19.920 --> 00:00:23.039 It's basically the ultimate roadblock in data gathering. I mean, 11 00:00:23.399 --> 00:00:25.640 you can pull all the public information you want, but 12 00:00:25.719 --> 00:00:30.440 the second a specialized database or a social platform asks 13 00:00:30.480 --> 00:00:34.039 your script to prove who it is, a basic scraper. 14 00:00:34.439 --> 00:00:39.039 Just freezes because to the scraper, yeah, that user name 15 00:00:39.039 --> 00:00:41.880 and password field is well, it's an alien. 16 00:00:41.560 --> 00:00:43.679 Concept, right, it doesn't have hands to typewy. 17 00:00:43.719 --> 00:00:46.560 It's a total dead end. But today, our mission for 18 00:00:46.600 --> 00:00:49.479 this deep dive is to synthesize all the incredible documentation 19 00:00:49.600 --> 00:00:51.840 and coding tutorials we've been going through to help you 20 00:00:51.920 --> 00:00:52.759 tear down that wall. 21 00:00:52.840 --> 00:00:53.920 We're going to get you through it. 22 00:00:54.000 --> 00:00:56.920 We're going to give your code unfettered access to the 23 00:00:56.920 --> 00:01:01.520 web by teaching it how to navigate security checkpoints and 24 00:01:02.640 --> 00:01:04.840 handle complex user interfaces. 25 00:01:05.200 --> 00:01:07.560 Yeah, we're going to break down the actual architecture of 26 00:01:07.560 --> 00:01:10.120 how a log in functions on the server side, figure 27 00:01:10.120 --> 00:01:12.760 out how to bypass the traps developers set for bots, 28 00:01:12.840 --> 00:01:16.640 and then walk step by step through automating live browser interactions. 29 00:01:16.840 --> 00:01:19.680 And to do this, we're using a format we absolutely 30 00:01:19.719 --> 00:01:23.159 love for technical topics, which is the invisible screen. 31 00:01:23.280 --> 00:01:24.239 I love this approach. 32 00:01:24.359 --> 00:01:27.519 It's so helpful. We are going to describe the architecture 33 00:01:27.560 --> 00:01:31.040 and the code so vividly that you will be able 34 00:01:31.040 --> 00:01:34.519 to visualize the exact programming steps in your mind's eye. 35 00:01:34.760 --> 00:01:36.519 No monitor required exactly. 36 00:01:36.719 --> 00:01:39.920 So let's get into it. I imagine before we can write 37 00:01:39.920 --> 00:01:42.159 a script to bypass a log in, we really need 38 00:01:42.200 --> 00:01:44.400 to understand what a log in actually looks like to 39 00:01:44.439 --> 00:01:45.200 the server. Right. 40 00:01:45.239 --> 00:01:47.280 That is the foundation. I mean, when you sit at 41 00:01:47.319 --> 00:01:49.439 your computer and log into a site, the interface makes 42 00:01:49.439 --> 00:01:52.719 it feel like one continuous motion, very smooth, right, But 43 00:01:52.799 --> 00:01:55.920 underneath it's actually a multi step conversation. It begins with 44 00:01:55.959 --> 00:02:01.359 your browser initiating ant get request request. Yeah, it reaches 45 00:02:01.359 --> 00:02:03.519 out to the server and essentially just asks to view 46 00:02:03.560 --> 00:02:05.159 the logging page, and then. 47 00:02:05.040 --> 00:02:09.800 The server responds by sending back the HTML. You know, 48 00:02:09.840 --> 00:02:12.680 the structure of the page containing those mpy input fields 49 00:02:12.680 --> 00:02:14.039 for your credentials. 50 00:02:13.599 --> 00:02:16.080 Right, and then you type your details, you hit enter, 51 00:02:16.599 --> 00:02:18.960 and your browser Initially it's a PSD request. 52 00:02:19.080 --> 00:02:22.240 Okay, so get is asking for the page, post is 53 00:02:22.240 --> 00:02:23.520 sending the data exactly. 54 00:02:24.159 --> 00:02:26.759 It takes your username and password, bundles them up as 55 00:02:26.759 --> 00:02:29.840 a payload, and posts them back to the server. 56 00:02:30.080 --> 00:02:30.479 Makes sense. 57 00:02:30.479 --> 00:02:34.759 The server cross references that payload with its database, and well, 58 00:02:34.840 --> 00:02:36.439 if there's a match, it logs you in. 59 00:02:36.599 --> 00:02:37.719 Okay, boom, you're in. 60 00:02:37.840 --> 00:02:41.919 You're in. But this introduces a massive architectural problem. How 61 00:02:41.960 --> 00:02:44.199 does the server remember that you are logged in when 62 00:02:44.240 --> 00:02:45.719 you click to the next page. 63 00:02:46.039 --> 00:02:48.479 Well, I'm assuming it doesn't just naturally remember, like every 64 00:02:48.520 --> 00:02:50.360 time I click a new link, doesn't the server just 65 00:02:50.400 --> 00:02:51.919 see a brand new request coming in? 66 00:02:52.080 --> 00:02:54.560 It does. The web is built on HTTP, which is 67 00:02:54.560 --> 00:02:56.080 a stateless. 68 00:02:55.439 --> 00:02:58.639 Protocol, stateless meaning no memory. 69 00:02:58.520 --> 00:03:03.080 By designers, zero memory of previous interactions. And there's a 70 00:03:03.120 --> 00:03:04.800 really good reason for this, which is scale. 71 00:03:05.000 --> 00:03:06.000 Oh because of traffic. 72 00:03:06.159 --> 00:03:09.080 Yeah, if a server had to actively memorize the continuous 73 00:03:09.120 --> 00:03:13.400 state of millions of concurrent users, the memory overhead would 74 00:03:13.439 --> 00:03:14.319 just crash the system. 75 00:03:14.400 --> 00:03:16.879 It would completely melt down. Yeah, so it offloads that 76 00:03:16.919 --> 00:03:18.680 memory burden back to the user. 77 00:03:18.919 --> 00:03:21.639 Exactly, it hands your browser a session. 78 00:03:21.360 --> 00:03:23.919 Cookie, ah, the famous cookie. Right. 79 00:03:24.319 --> 00:03:27.199 In web terms, a cookie is just a tiny text file. 80 00:03:27.879 --> 00:03:31.120 When that initial PST request with your password is approved, 81 00:03:31.479 --> 00:03:35.520 the server generates this unique encrypted strain of text, the 82 00:03:35.599 --> 00:03:38.840 session cookie, and tells your browser to hold on to it. Okay, 83 00:03:38.960 --> 00:03:41.360 And then on every subsequent request you make, your browser 84 00:03:41.400 --> 00:03:44.000 silently attaches that cookie to the outgoing message. 85 00:03:44.039 --> 00:03:49.000 Okay, I'm picturing like a VIP handstamp at a nightclub. 86 00:03:49.080 --> 00:03:50.719 Oh, that's a perfect analogy, right. 87 00:03:51.039 --> 00:03:53.439 You wait in line, you show your physical ID to 88 00:03:53.479 --> 00:03:55.560 the bouncer at the front door, and that is your 89 00:03:55.599 --> 00:04:00.000 initial post request with your passwords. Yes, the bouncer verifies you, 90 00:04:00.080 --> 00:04:01.960 but instead of making you show your idea at every 91 00:04:02.000 --> 00:04:04.360 single room inside the club, he just stamps. 92 00:04:04.080 --> 00:04:05.680 Your hand, which is the session cookie. 93 00:04:05.759 --> 00:04:07.919 Exactly. That stamp is the cookie. Once you have it, 94 00:04:07.960 --> 00:04:10.639 you just flash the stamp to get into the VIP areas. 95 00:04:11.000 --> 00:04:14.960 And when the server sees that vallid stamp, it dynamically 96 00:04:15.039 --> 00:04:17.160 changes the content it serves you. It routes you to 97 00:04:17.240 --> 00:04:20.639 your private dashboard instead of the public landing page. 98 00:04:20.800 --> 00:04:22.920 Well wait, let me push back on this a little. Sure, 99 00:04:23.000 --> 00:04:26.720 if this highly secure log in is ultimately just relying 100 00:04:26.759 --> 00:04:30.680 on a tiny text file. Couldn't a malicious program forge 101 00:04:30.680 --> 00:04:34.519 one or hijacket absolutely? Like, if my browser is just 102 00:04:34.560 --> 00:04:38.759 automatically flashing this stamp at the door, what is stopping 103 00:04:38.800 --> 00:04:41.399 a bad actor from exploiting that automation? 104 00:04:41.680 --> 00:04:43.759 You just hit on a vulnerability that was a massive 105 00:04:43.800 --> 00:04:47.360 headache for security engineers for years. Really, Oh yeah, it's 106 00:04:47.399 --> 00:04:49.959 an exploit known as a cross site request forgery or 107 00:04:50.079 --> 00:04:51.399 CSRF cross. 108 00:04:51.160 --> 00:04:54.079 Site request forgery. Okay, break down the mechanics of that attack. 109 00:04:54.160 --> 00:04:54.839 How does it work? 110 00:04:54.920 --> 00:04:57.160 So imagine you're logged into your bank's website. In one 111 00:04:57.199 --> 00:05:01.279 browser tab, your VIP handstamp, the session cookie is active. 112 00:05:01.439 --> 00:05:02.879 Okay, I'm logged in. 113 00:05:02.560 --> 00:05:04.680 In another tab, you click a sketchy link. 114 00:05:04.839 --> 00:05:05.680 Classic mistake. 115 00:05:05.839 --> 00:05:08.759 Right, That malicious website has code hidden in it that 116 00:05:08.839 --> 00:05:13.319 silently sends an HTTP post request to your banks server, 117 00:05:13.560 --> 00:05:16.199 perhaps instructing it to transfer funds. 118 00:05:16.519 --> 00:05:20.639 Whoa, And because my browser is programmed to automatically attach 119 00:05:20.720 --> 00:05:24.360 the bank's cookie to any request going to the bank's domain. 120 00:05:24.240 --> 00:05:27.600 The browser helpfully attaches the stamp to the malicious requests. 121 00:05:27.680 --> 00:05:30.600 Oh my god, it just volunteers the VIP pass. 122 00:05:30.720 --> 00:05:34.199 It acts entirely on your behalf. The bank's server receives 123 00:05:34.199 --> 00:05:37.480 the transfer command, sees your valid session cookie attached to it, 124 00:05:37.519 --> 00:05:40.399 and processes the transaction just assuming you click the button. 125 00:05:40.600 --> 00:05:44.279 The browser being helpful is the actual vulnerability. 126 00:05:43.560 --> 00:05:44.399 Here here, it really is. 127 00:05:44.560 --> 00:05:48.600 Wow. So the malicious site is basically grabbing your stamped 128 00:05:48.639 --> 00:05:50.560 hand and just waving it at the bouncer. 129 00:05:50.680 --> 00:05:51.839 That's exactly what it's doing. 130 00:05:52.399 --> 00:05:55.800 So how do modern websites defend against this? Because obviously 131 00:05:55.800 --> 00:05:57.000 we still use banks. 132 00:05:56.680 --> 00:06:00.399 Online, they implement CSORF tokens. They need it a way 133 00:06:00.439 --> 00:06:03.680 to verify that the request was intentionally submitted from their 134 00:06:03.720 --> 00:06:08.000 actual forms and not from some hidden script on another tab. Okay, So, 135 00:06:08.079 --> 00:06:10.639 whenever the server sends you the initial login page, it 136 00:06:10.759 --> 00:06:15.600 generates a random, cryptographically secure string of characters. That's the token. 137 00:06:15.800 --> 00:06:16.959 Yeah, where does it put it? 138 00:06:16.959 --> 00:06:20.000 It embeds this token directly into the HTML of the 139 00:06:20.079 --> 00:06:22.040 form as a hidden input field. 140 00:06:22.360 --> 00:06:25.560 Oh so the user never even sees it, but it's 141 00:06:25.560 --> 00:06:26.560 sitting there in the code. 142 00:06:26.839 --> 00:06:30.319 Yes. So when you submit your password, your browser sends 143 00:06:30.360 --> 00:06:32.639 the password payload and the hidden token back to the server. 144 00:06:32.839 --> 00:06:33.160 Got it. 145 00:06:33.560 --> 00:06:36.600 The server checks the password, but it also verifies that 146 00:06:36.680 --> 00:06:39.079 the token perfectly matches the one it generated a few 147 00:06:39.079 --> 00:06:39.600 minutes ago. 148 00:06:39.759 --> 00:06:42.480 I see. So the malicious site in the other tab 149 00:06:43.079 --> 00:06:45.399 might be able to trigger a request, and it might 150 00:06:45.399 --> 00:06:47.839 even have the cookie, but it has absolutely no way 151 00:06:47.879 --> 00:06:51.279 of guessing that randomly generated token exactly, so the request 152 00:06:51.360 --> 00:06:54.399 just gets rejected. It's like a secret handshake that rotates constantly. 153 00:06:54.560 --> 00:06:55.920 That is a great way to put. 154 00:06:55.720 --> 00:06:59.000 It, which means for our automated scraper, we can't just 155 00:06:59.040 --> 00:07:02.319 blast a post request out of nowhere. We have to 156 00:07:02.399 --> 00:07:03.439 learn the handshake first. 157 00:07:03.480 --> 00:07:06.720 That is the core takeaway a scraper. Attempting a log 158 00:07:06.720 --> 00:07:10.439 in requires a specific sequence. First, it must send a 159 00:07:10.519 --> 00:07:13.959 geet request to the log in form to get the token. Second, 160 00:07:14.360 --> 00:07:17.560 it must parse the return to HTML and extract that 161 00:07:17.680 --> 00:07:22.079 hidden CSRF token. Only then can it bundle the username, password, 162 00:07:22.120 --> 00:07:25.120 and token together and posted it all back to the server. 163 00:07:25.480 --> 00:07:29.240 Okay, let's step into our invisible code editor and visualize 164 00:07:29.279 --> 00:07:33.480 writing this sequence in Python. Our sources rely heavily on 165 00:07:33.519 --> 00:07:36.480 the Python requests library for this back end approach. 166 00:07:36.639 --> 00:07:40.360 Yeah, it's the industry standard for handling HTTP protocols. But 167 00:07:40.560 --> 00:07:43.000 right away, if we use the request library, we have 168 00:07:43.040 --> 00:07:44.160 to handle the cookie. 169 00:07:43.920 --> 00:07:46.879 Problem right because we need our script to catch that 170 00:07:47.040 --> 00:07:49.920 VIP handstamp and remember to use it on future. 171 00:07:49.600 --> 00:07:54.199 Requests, and doing that manually is extremely tedious, I can imagine. Fortunately, 172 00:07:54.360 --> 00:07:57.920 the library has a built in feature called a session object. 173 00:07:58.240 --> 00:07:59.439 Okay, how does that work? 174 00:07:59.600 --> 00:08:03.160 You and stantiate a variable let's call it session, and 175 00:08:03.240 --> 00:08:06.399 assign it to request session. From that point on, you 176 00:08:06.519 --> 00:08:10.360 route all your gettee and post requests through that specific object. 177 00:08:10.399 --> 00:08:12.319 So it's basically a container for the requests. 178 00:08:12.399 --> 00:08:15.319 Yeah, it acts as an automatic internal cookie jar. It 179 00:08:15.399 --> 00:08:18.399 persists your log in state across the entire script without 180 00:08:18.439 --> 00:08:21.360 you writing another single line of cookie management code. 181 00:08:21.439 --> 00:08:24.839 That completely abstracts the difficulty. I love that. So, using 182 00:08:24.879 --> 00:08:28.920 that session object, your code sends a JT request to 183 00:08:29.120 --> 00:08:32.480 the log in URL. You now have the raw HTML 184 00:08:32.600 --> 00:08:35.919 of the login page stored in your script's memory. Now 185 00:08:35.960 --> 00:08:39.399 comes the extraction you need to find that hidden token. 186 00:08:39.519 --> 00:08:41.559 This requires a bit of reconnaissance on your part. 187 00:08:41.720 --> 00:08:44.039 Right before writing the script, you would open the target 188 00:08:44.080 --> 00:08:47.240 website in a normal browser, write, click the page, and 189 00:08:47.279 --> 00:08:50.200 inspect the elements to view the raw EAHTML tree. 190 00:08:50.399 --> 00:08:52.799 You are hunting for an input tag, usually with a 191 00:08:52.840 --> 00:08:56.480 type of hidden and a name like Sarahso middleware token 192 00:08:56.759 --> 00:08:58.120 or authenticity token. 193 00:08:58.240 --> 00:09:00.759 Okay, so once you know its name, how do we 194 00:09:00.799 --> 00:09:02.960 actually tell our Python script to find it? 195 00:09:03.279 --> 00:09:08.519 You utilize an XPath expression XPath. Yeah. The document object 196 00:09:08.600 --> 00:09:11.639 model or dom of a web page is structured like 197 00:09:11.679 --> 00:09:14.639 an upside down family tree. You have parent nodes, child nodes, 198 00:09:14.679 --> 00:09:15.440 and sibling nodes. 199 00:09:15.519 --> 00:09:17.080 Right, it's all nested exactly. 200 00:09:17.360 --> 00:09:20.039 XPath is a query language that traces the branches of 201 00:09:20.039 --> 00:09:22.559 that tree. So you write a short expression that commands 202 00:09:22.559 --> 00:09:24.720 the code to search the document tree for an input 203 00:09:24.759 --> 00:09:27.960 node where the name attribute equals say, zeros token, and 204 00:09:28.000 --> 00:09:29.919 then extract its assigned value. 205 00:09:30.039 --> 00:09:32.720 Awesome, So you now have the password, the username, and 206 00:09:32.799 --> 00:09:36.759 the secret handshake. Now we construct the payload in Python. 207 00:09:37.039 --> 00:09:39.279 The most efficient way to package this for an h 208 00:09:40.080 --> 00:09:42.039 post request is using a. 209 00:09:42.039 --> 00:09:46.279 Dictionary, because a Python dictionary utilizes key value pairs, which 210 00:09:46.399 --> 00:09:49.679 maps perfectly to how HTML forms transmit data. 211 00:09:49.919 --> 00:09:51.840 It's a one to one translation exactly. 212 00:09:52.120 --> 00:09:54.639 You create a dictionary where the keys are the names 213 00:09:54.639 --> 00:09:57.080 of the form fields and the values are your credentials 214 00:09:57.080 --> 00:10:00.159 and the token. But you really have to be meticulous here. 215 00:10:00.240 --> 00:10:02.679 Yeah, this is a massive pitfall. I might label my 216 00:10:02.720 --> 00:10:05.799 dictionary key username, but if the developer who built the 217 00:10:05.840 --> 00:10:09.480 website named their HTML input field loge and mail address. 218 00:10:10.080 --> 00:10:11.679 The post request will fail. 219 00:10:11.840 --> 00:10:12.960 It will completely fail. 220 00:10:13.039 --> 00:10:15.399 The server is looking for a specific key. If you 221 00:10:15.440 --> 00:10:18.399 send it username, it simply ignores the data. You have 222 00:10:18.480 --> 00:10:21.679 to inspect the site and mirror their naming conventions exactly 223 00:10:21.919 --> 00:10:22.399 spot on. 224 00:10:22.879 --> 00:10:25.480 And once your dictionary is perfectly structured, you use your 225 00:10:25.480 --> 00:10:28.879 session object to fire off the post request, passing that 226 00:10:28.960 --> 00:10:30.600 dictionary into the data parameter. 227 00:10:30.919 --> 00:10:34.399 So the script executes, the data flies across the web, 228 00:10:35.320 --> 00:10:38.879 but our terminal doesn't throw confetti. How do we programmatically 229 00:10:39.000 --> 00:10:41.000 verify that the bouncer actually let us in? 230 00:10:42.000 --> 00:10:44.759 There is a remarkably elegant trick for this. How to 231 00:10:44.879 --> 00:10:48.279 tell you command your script to search the raw HTML 232 00:10:48.360 --> 00:10:50.600 of the server's response for the word logout. 233 00:10:50.799 --> 00:10:52.240 Oh that's brilliant, right. 234 00:10:52.440 --> 00:10:55.840 If your code failed and you're still unauthenticated, the page 235 00:10:55.879 --> 00:10:59.000 will likely display links for log in or sign up. 236 00:10:59.240 --> 00:11:02.440 But if the word out appears in the HTML, you. 237 00:11:02.480 --> 00:11:05.159 Know the server has routed you to an authenticated state. 238 00:11:05.240 --> 00:11:05.799 You are in. 239 00:11:06.080 --> 00:11:09.440 It's so simple, but it completely validates the logic. We've 240 00:11:09.480 --> 00:11:12.759 bypassed the wall. But we need to talk about the 241 00:11:12.759 --> 00:11:13.960 limitations of this approach. 242 00:11:14.039 --> 00:11:14.639 Yeah we do. 243 00:11:15.120 --> 00:11:18.000 Using the request library is fantastic for sites that use 244 00:11:18.080 --> 00:11:21.840 traditional HTML routing, but modern web development is shifted heavily 245 00:11:21.879 --> 00:11:22.919 towards single page. 246 00:11:22.720 --> 00:11:25.919 Applications, sites built with frameworks like react or view. 247 00:11:26.080 --> 00:11:28.799 Right, if you send a giddyt request to a modern 248 00:11:28.879 --> 00:11:31.519 dynamic site, the server doesn't send you a fully rendered 249 00:11:31.639 --> 00:11:34.279 HTML page with all the forms neatly laid out. 250 00:11:34.480 --> 00:11:37.679 No, it sends you a mostly empty HTML, a shell, 251 00:11:38.080 --> 00:11:40.679 and a massive bundle of JavaScript, and. 252 00:11:40.679 --> 00:11:43.720 The browser is supposed to execute that javascripts to build 253 00:11:43.759 --> 00:11:44.720 the page locally. 254 00:11:44.919 --> 00:11:49.919 And here's the kicker. The Python requests library cannot execute JavaScript. 255 00:11:49.960 --> 00:11:52.039 It just reads the initial raw response. 256 00:11:52.639 --> 00:11:55.679 It is a data career. It doesn't have a rendering engine. 257 00:11:55.720 --> 00:11:58.000 So if the log in form doesn't exist until the 258 00:11:58.080 --> 00:12:01.240 JavaScript builds it, or or if the log in requires 259 00:12:01.279 --> 00:12:04.799 interacting with a complex dynamic pop up, your script is blind, 260 00:12:05.120 --> 00:12:05.960 totally blind. 261 00:12:06.080 --> 00:12:09.279 So to handle that level of complexity we need to 262 00:12:09.399 --> 00:12:12.759 completely shift our paradigm. We need a puppet master. 263 00:12:13.120 --> 00:12:13.799 Yes we do. 264 00:12:14.200 --> 00:12:17.159 This is where we introduce Selenium. If using the request 265 00:12:17.200 --> 00:12:19.360 library is like sending letters back and forth through the mail, 266 00:12:19.840 --> 00:12:22.200 Selenium is like building a robot that sits in your 267 00:12:22.200 --> 00:12:25.720 office chair, opens Google Chrome, looks at the monitor, and 268 00:12:25.720 --> 00:12:27.480 physically presses keys on the keyboard. 269 00:12:27.519 --> 00:12:29.720 That's a great visual. Selenium is a suite of tools 270 00:12:29.759 --> 00:12:32.080 engineered specifically for browser automation. 271 00:12:32.320 --> 00:12:34.120 So it's not just doing each GPP stuff. 272 00:12:34.279 --> 00:12:38.159 No, it doesn't just send HTTP packets. It interfaces directly 273 00:12:38.200 --> 00:12:39.159 with a webdriver. 274 00:12:39.279 --> 00:12:39.799 Oh webdrive. 275 00:12:39.919 --> 00:12:42.639 The webdriver acts as a bridge between your code and 276 00:12:42.679 --> 00:12:46.360 the native internal API of the web browser. It commands 277 00:12:46.360 --> 00:12:49.519 the browser to render the JavaScript, scroll down pages, and 278 00:12:49.559 --> 00:12:52.360 trigger click events exactly as a human user would. 279 00:12:52.519 --> 00:12:56.559 It is incredibly powerful. You can write your instructions in Python, Java, 280 00:12:56.679 --> 00:13:01.000 c Sharp, and drive browsers across Windows, Mac, or Linux. 281 00:13:01.080 --> 00:13:02.679 The flexibility is amazing. 282 00:13:02.799 --> 00:13:05.039 So let's visualize this. Let's put the robot to work. 283 00:13:05.840 --> 00:13:09.879 Imagine we are automating a highly complex user registration form 284 00:13:10.320 --> 00:13:11.639 filled with dynamic elements. 285 00:13:11.720 --> 00:13:15.240 Okay. In our script, we initialize the webdriver. This physically 286 00:13:15.320 --> 00:13:18.159 launches a new browser window on your machine. 287 00:13:18.200 --> 00:13:19.720 You actually see it pop up. Yeah. 288 00:13:19.879 --> 00:13:23.120 We use thecommand driver dot get and pass it the 289 00:13:23.120 --> 00:13:26.399 registration url. The browser navigates to the page and begins 290 00:13:26.399 --> 00:13:27.200 rendering the layout. 291 00:13:27.279 --> 00:13:29.279 Okay, so we want to type an email address into 292 00:13:29.279 --> 00:13:31.440 the very first textbox. So we just tell the robot 293 00:13:31.480 --> 00:13:32.039 to type right. 294 00:13:32.240 --> 00:13:34.399 Well, if you do that immediately, your scriptill crash. 295 00:13:34.480 --> 00:13:34.679 Wait? 296 00:13:34.720 --> 00:13:38.600 Really, why we have to account for execution speed. Your 297 00:13:38.639 --> 00:13:43.039 Python script executes commands in milliseconds, but the web browser 298 00:13:43.120 --> 00:13:45.679 might take a full second to download an image or 299 00:13:45.720 --> 00:13:48.120 render a form field via JavaScript. 300 00:13:48.200 --> 00:13:50.879 Oh, I see the robot is too fast. It tries 301 00:13:50.919 --> 00:13:53.360 to type on a keyboard that the webpage hasn't finished 302 00:13:53.360 --> 00:13:54.039 building yet. 303 00:13:54.159 --> 00:13:56.879 Exactly, we have to implement a webdriver. 304 00:13:57.279 --> 00:13:58.519 Wait, a webdriver. Wait. 305 00:13:58.600 --> 00:14:01.679 We explicitly pause the code, but we don't just use 306 00:14:01.679 --> 00:14:05.000 a hard sleep command because network speeds vary. Sometimes it 307 00:14:05.039 --> 00:14:06.720 takes one second, sometimes five. 308 00:14:06.879 --> 00:14:08.240 Right, hard sleep is too brittle. 309 00:14:08.320 --> 00:14:11.720 Instead, we program the webdriver to continuously pull the DOM, 310 00:14:11.840 --> 00:14:16.559 checking the underlying HTML structure every five hundred milliseconds until 311 00:14:16.600 --> 00:14:19.799 the specific element we want is physically present and clickable. 312 00:14:20.240 --> 00:14:22.879 That is so smart. It ensures the script only precedes 313 00:14:22.919 --> 00:14:24.360 when the UI is actually ready. 314 00:14:24.480 --> 00:14:24.960 Exactly. 315 00:14:25.159 --> 00:14:27.080 Okay, so the UI is ready, we need to locate 316 00:14:27.120 --> 00:14:30.559 that email input box. You inspect the page beforehand, and 317 00:14:30.600 --> 00:14:33.279 you notice the input tag has an ID attribute. Let's 318 00:14:33.279 --> 00:14:36.399 say use a email in Selenium. You use a locator 319 00:14:36.480 --> 00:14:39.080 function findelment, passing it the ID. 320 00:14:39.360 --> 00:14:42.279 Now Selenium has locked its focus onto that specific node 321 00:14:42.279 --> 00:14:42.879 in the dom. 322 00:14:42.799 --> 00:14:44.080 Tree's targeting it. 323 00:14:44.120 --> 00:14:47.120 We command it to clear any pre existing text, and 324 00:14:47.159 --> 00:14:51.000 then we utilize the sen keys function enkeys. This sends 325 00:14:51.240 --> 00:14:54.360 native keyboard strokes to the browser, typing out the email 326 00:14:54.399 --> 00:14:58.840 address and triggering any JavaScript event listeners attached to that box. 327 00:14:59.120 --> 00:15:02.480 Incredible. Next we hit the main form and we encounter 328 00:15:02.759 --> 00:15:09.399 radio buttons. Say, selecting a user tier basic, pro or enterprise, 329 00:15:09.440 --> 00:15:10.360 you can only pick one. 330 00:15:10.759 --> 00:15:15.240 Radio buttons present a really interesting locator challenge. How so, Well, 331 00:15:15.279 --> 00:15:18.000 if you inspect the HTML for all three tiers, you 332 00:15:18.039 --> 00:15:21.559 will notice they all share the exact same name attribute. 333 00:15:21.600 --> 00:15:23.399 Oh so they aren't uniquely named, right. 334 00:15:23.480 --> 00:15:26.000 The developer groups them under one name, so the browser 335 00:15:26.080 --> 00:15:28.120 knows they're mutually exclusive. 336 00:15:27.720 --> 00:15:30.799 Which means if you tell Selenium to find element by name, 337 00:15:31.080 --> 00:15:33.919 it gets confused because it finds three identical matches. It 338 00:15:33.960 --> 00:15:35.639 doesn't know which one you want precisely. 339 00:15:35.840 --> 00:15:39.000 Therefore, you must use the ID attribute. The idea unlike 340 00:15:39.120 --> 00:15:42.879 names or complex XPath expressions, which can break if the 341 00:15:42.919 --> 00:15:45.759 developers slightly alter the layout of the page. An ID 342 00:15:45.960 --> 00:15:49.120 is designed via completely unique anchor for that specific element. 343 00:15:49.639 --> 00:15:52.000 You locate the unique id for the pro tear button 344 00:15:52.279 --> 00:15:54.600 and simply a pend dot click to your code. 345 00:15:54.720 --> 00:15:59.559 Bam, the robot clicks the button. Next hurdle drop down menus. 346 00:16:00.120 --> 00:16:05.320 Imagine a complex date selector with three separate dropdowns for day, month, 347 00:16:05.480 --> 00:16:05.919 and year. 348 00:16:06.399 --> 00:16:09.600 Selenium provides a specialized module just for this, called the 349 00:16:09.679 --> 00:16:13.120 select class. You import it and you wrap your targeted 350 00:16:13.240 --> 00:16:16.000 HTML element inside it. Okay, but there is a vital 351 00:16:16.000 --> 00:16:18.960 distinction you must make when manipulating drop downs. You can 352 00:16:18.960 --> 00:16:21.559 either select by value or select by visible text. 353 00:16:21.799 --> 00:16:24.039 Explain the difference there. I assume visible text is just 354 00:16:24.080 --> 00:16:25.840 what the human sees on the screen. 355 00:16:25.679 --> 00:16:29.200 Exactly Selective visible text interacts with the exact string displayed 356 00:16:29.240 --> 00:16:31.759 on the screen. So, for the month's dropdown, the user 357 00:16:31.799 --> 00:16:34.840 sees the word october, You pass the string october into 358 00:16:34.840 --> 00:16:36.320 your code and Selenium clicks it. 359 00:16:36.559 --> 00:16:37.919 But what is happening underneath that? 360 00:16:38.159 --> 00:16:42.000 Underneath? The HTML uses option tags, and each option tag 361 00:16:42.080 --> 00:16:44.960 has a hidden back end value attribute. I see, So 362 00:16:45.200 --> 00:16:48.799 while the user sees october, the value attribute might just 363 00:16:48.840 --> 00:16:52.120 be the integer ten. If you select by value, you 364 00:16:52.200 --> 00:16:54.000 pass the number ten to your script. 365 00:16:54.320 --> 00:16:57.080 Both achieve the same result, but you really have to 366 00:16:57.120 --> 00:17:00.120 know whether your script's interacting with the front end display 367 00:17:00.279 --> 00:17:01.759 or the back end data structure. 368 00:17:01.879 --> 00:17:04.920 It's a crucial distinction for building robust scripts totally. 369 00:17:05.319 --> 00:17:09.359 Finally, we have checkboxes, say agreeing to terms of service. 370 00:17:09.559 --> 00:17:12.960 These are honestly the simplest elements. Checkboxes are just independent 371 00:17:13.000 --> 00:17:13.920 binary toggles. 372 00:17:13.920 --> 00:17:15.319 So you just find it and click it. 373 00:17:15.400 --> 00:17:19.480 You locate its unique ID and fire a dot click event. 374 00:17:19.880 --> 00:17:22.279 Once all the fields are populated, the drop down selected, 375 00:17:22.319 --> 00:17:24.839 and the box is checked, you locate the final submit 376 00:17:24.880 --> 00:17:25.759 button and click it. 377 00:17:25.799 --> 00:17:27.319 And since it's a real browser, the. 378 00:17:27.200 --> 00:17:31.559 Browser packages the payload, handles the CSRF tokens natively, and 379 00:17:31.599 --> 00:17:32.240 logs you in. 380 00:17:32.680 --> 00:17:36.079 So the robot has successfully navigated the dynamic form. We 381 00:17:36.160 --> 00:17:39.359 are inside the database. But here is the catch. 382 00:17:39.480 --> 00:17:40.400 There's always a catch. 383 00:17:40.640 --> 00:17:44.480 Selenium is a browser automator. It is relatively heavy and 384 00:17:44.559 --> 00:17:47.920 slow compared to a raw HTTP parser. How do we 385 00:17:47.960 --> 00:17:50.599 pivot back to actually scraping the data efficiently? 386 00:17:50.680 --> 00:17:54.400 You execute a handoff a handoff. Once Selenium has successfully 387 00:17:54.440 --> 00:17:56.640 logged you in and the target data is rendered on 388 00:17:56.680 --> 00:17:59.680 the screen, you call drider dot. 389 00:17:59.519 --> 00:18:01.920 Page so drop at page source. What does that do? 390 00:18:02.359 --> 00:18:06.039 This command grabs the entire fully rendered HTML structure of 391 00:18:06.079 --> 00:18:09.039 the current page as a massive tech string. You then 392 00:18:09.119 --> 00:18:13.400 feed that string directly into a dedicated parsing library. 393 00:18:13.319 --> 00:18:16.559 Like beautiful Soup, which is built entirely for slicing up 394 00:18:16.680 --> 00:18:18.640 HTML and extracting. 395 00:18:18.160 --> 00:18:21.160 The data points precisely. Selenium does the heavy lifting of 396 00:18:21.200 --> 00:18:25.160 dealing with the JavaScript, the dynamic menus, and the log insecurity. 397 00:18:25.319 --> 00:18:26.839 It breaks through the wall. 398 00:18:26.519 --> 00:18:28.440 Right and once the vault is open and the data 399 00:18:28.480 --> 00:18:31.400 is visible, it passes the baton to the parser and 400 00:18:31.440 --> 00:18:34.400 your scraper resumes its lightning fast operations. 401 00:18:34.640 --> 00:18:37.319