WEBVTT 1 00:00:00.120 --> 00:00:03.560 You know, we walk around all day with this massive, 2 00:00:03.640 --> 00:00:06.480 invisible engine running our modern lives, and most of the 3 00:00:06.480 --> 00:00:07.960 time we just don't even think about it. 4 00:00:08.080 --> 00:00:10.240 Oh, for sure, we're completely oblivious to it. 5 00:00:10.400 --> 00:00:13.160 Right, I'm talking about data. Just think about it for 6 00:00:13.199 --> 00:00:16.600 a second. For you listening, your smart watch tracks your 7 00:00:16.600 --> 00:00:19.120 heart rate right, and your sleep cycles. 8 00:00:19.199 --> 00:00:21.519 And it triggers all those personalized. 9 00:00:20.960 --> 00:00:24.559 Health alerts exactly. And your browsing habits are being quietly 10 00:00:24.640 --> 00:00:28.679 collected to you know, serve up those hyper targeted ads 11 00:00:28.679 --> 00:00:30.480 that sometimes feel it still a little too. 12 00:00:30.440 --> 00:00:32.240 Accurate, like they're reading your mind. 13 00:00:32.439 --> 00:00:35.960 Yeah, even your Friday night on the couch is being analyzed, 14 00:00:36.359 --> 00:00:39.880 like you're viewing habits magically predicting the exact next TV 15 00:00:39.960 --> 00:00:41.000 show you want to binge. 16 00:00:41.079 --> 00:00:45.359 Well, the modern digital experience is entirely fueled by data collection. 17 00:00:45.520 --> 00:00:49.799 I mean, every click, every pause, every service, Yeah, it's 18 00:00:49.840 --> 00:00:53.200 a data point. It's being processed and utilized to build 19 00:00:53.280 --> 00:00:55.520 a customized reality just for you. 20 00:00:55.920 --> 00:00:58.640 But here's the thing that always gets me. If you're 21 00:00:58.679 --> 00:01:02.759 a data analyst or a data scientist, or just someone 22 00:01:02.759 --> 00:01:06.359 who likes building cool tech projects, that data isn't just 23 00:01:06.400 --> 00:01:08.359 magically handed to you on a silver platter. 24 00:01:08.599 --> 00:01:09.439 No, definitely not. 25 00:01:09.560 --> 00:01:11.560 You don't just ask the Internet nicely and get a 26 00:01:11.560 --> 00:01:14.560 perfectly formatted spreadsheet in return. You have to go get 27 00:01:14.599 --> 00:01:18.959 it right. And one of the biggest, most powerful methods 28 00:01:19.000 --> 00:01:21.079 forgetting that data is webscraping. 29 00:01:21.480 --> 00:01:24.040 It is an absolutely foundational skill. I mean, if the 30 00:01:24.120 --> 00:01:26.719 data is out there on a web page, webscraping is 31 00:01:26.760 --> 00:01:30.519 how you programmatically extract it, structure it, and you know, 32 00:01:30.599 --> 00:01:32.719 actually make it useful for your own analysis. 33 00:01:32.760 --> 00:01:34.879 But the Internet has changed, hasn't it. Like the easy 34 00:01:34.959 --> 00:01:35.719 days are kind of. 35 00:01:35.640 --> 00:01:38.239 Over, oh completely, The landscape is entirely different now. 36 00:01:38.400 --> 00:01:41.319 So the mission of today's deep dive is exactly that 37 00:01:41.719 --> 00:01:45.200 we are leveling up. We're taking your Python skills from 38 00:01:45.439 --> 00:01:49.680 basic data extraction and stepping into the gritty, real world 39 00:01:49.799 --> 00:01:51.680 tactics of advanced webscraping. 40 00:01:51.840 --> 00:01:52.519 This can be fun. 41 00:01:52.680 --> 00:01:55.280 We're going to map out exactly how to bypass the 42 00:01:55.319 --> 00:01:59.000 common roadblocks that modern websites throw at automated scripts. 43 00:01:59.359 --> 00:02:02.920 And just to set the stage for you listening right now, 44 00:02:03.560 --> 00:02:06.000 we are operating on the assumption that you already have 45 00:02:06.200 --> 00:02:09.280 a basic to intermediate grasp of Python programming. 46 00:02:09.479 --> 00:02:12.240 Right. We're not doing a beginner tutorial here exactly. 47 00:02:12.280 --> 00:02:14.360 We are going to be covering basic syntax or how 48 00:02:14.400 --> 00:02:18.159 to install libraries. Today's goal is to understand the architecture 49 00:02:18.199 --> 00:02:20.840 of resistance on the modern web and how you can 50 00:02:20.879 --> 00:02:24.319 script your way through it using advanced architectural concepts. 51 00:02:24.400 --> 00:02:27.080 Okay, let's unpack this because before we start writing scripts, 52 00:02:27.120 --> 00:02:29.159 we need to define the battlefield. 53 00:02:29.240 --> 00:02:30.439 Good call you hear. 54 00:02:30.280 --> 00:02:34.280 The term advanced web scraping a lot, But like, why 55 00:02:34.319 --> 00:02:37.680 isn't the basic stuff enough anymore? Where exactly is the 56 00:02:37.719 --> 00:02:41.039 line between a basic scrape and an advanced scrape? 57 00:02:41.240 --> 00:02:44.120 Well, in our context today, we define the difference by 58 00:02:44.120 --> 00:02:46.439 looking at the level of resistance. Okay, let's look at 59 00:02:46.439 --> 00:02:49.639 basic webscraping. In a basic scenario, you target a static 60 00:02:49.639 --> 00:02:52.719 web page, you send a request, you apply some simple 61 00:02:52.759 --> 00:02:55.840 scraping process to parse the HTML, and you get your 62 00:02:55.840 --> 00:02:56.639 required data. 63 00:02:56.680 --> 00:02:58.639 The server just hands you an HTML document and you 64 00:02:58.639 --> 00:02:59.000 read it. 65 00:02:59.000 --> 00:03:01.039 Right exactly minimal resistance. 66 00:03:01.319 --> 00:03:04.840 It's like reading a public billboard from the highway. Anyone 67 00:03:04.919 --> 00:03:07.280 driving by can see it, take a picture of it, 68 00:03:07.360 --> 00:03:08.400 and write down what it says. 69 00:03:08.479 --> 00:03:12.319 I like that. Yeah, that's basic scraping. But advanced scraping, however, 70 00:03:12.680 --> 00:03:16.479 is entirely defined by resistance. The roadblocks, right, It's what 71 00:03:16.560 --> 00:03:19.919 happens when a web page adds layers of difficulty between 72 00:03:19.960 --> 00:03:22.840 you and the data. Because much of the modern web 73 00:03:22.879 --> 00:03:26.439 is built on single page applicationss the spas, they use 74 00:03:26.479 --> 00:03:29.560 frameworks like reactor view, And when you send a basic 75 00:03:29.599 --> 00:03:32.400 request to one of these sites, the server doesn't hand 76 00:03:32.439 --> 00:03:36.120 you a nicely formatted HTMIL document with all the data. 77 00:03:36.159 --> 00:03:37.039 So what does it hand you. 78 00:03:37.159 --> 00:03:39.879 It hands you a virtually anti skeleton page and a 79 00:03:39.919 --> 00:03:41.360 massive bundle of JavaScript. 80 00:03:41.520 --> 00:03:43.639 Oh wow. So if you just run a standard scrape 81 00:03:43.639 --> 00:03:45.400 on that, you get a blank page. 82 00:03:45.240 --> 00:03:46.120 Back pretty much. 83 00:03:46.240 --> 00:03:48.479 You pull down the billboard and it's just a blank canvas. 84 00:03:48.520 --> 00:03:50.759 You're hitting the nail on the head. The data only 85 00:03:50.800 --> 00:03:55.080 populates after a browser executes that JavaScript. Ah, and that 86 00:03:55.159 --> 00:03:59.680 fundamentally breaks basic scraping tools. So advanced scraping is like 87 00:03:59.719 --> 00:04:03.960 trying to get information from inside an exclusive club, not 88 00:04:04.000 --> 00:04:07.599 a billboard exactly. Not a billboard. You can't just walk 89 00:04:07.639 --> 00:04:10.120 in and read the walls. You have to talk to 90 00:04:10.159 --> 00:04:13.680 a bouncer, show an ID, prove you've been there before, 91 00:04:14.240 --> 00:04:17.879 and actively participate in the mechanics of the building just 92 00:04:17.920 --> 00:04:19.160 to find the room you're looking for. 93 00:04:19.560 --> 00:04:21.959 That makes so much sense. So to get past this 94 00:04:22.079 --> 00:04:25.000 bouncer and into the club, we need the right tools 95 00:04:25.079 --> 00:04:28.040 on our tool belts we do. We need an environment 96 00:04:28.079 --> 00:04:30.920 that lets us test our strategy without starting from scratch 97 00:04:30.959 --> 00:04:33.680 every time we hit a wall. And for this, Jupiter 98 00:04:33.759 --> 00:04:38.240 Notebook is still the absolute gold standard for the scraping workflow. 99 00:04:38.399 --> 00:04:42.480 Oh, without a doubt, Jupiter's fantastic for this because statefulness 100 00:04:42.560 --> 00:04:44.519 is everything in advanced scraping. 101 00:04:44.639 --> 00:04:46.040 What do you mean my statefulness? 102 00:04:46.079 --> 00:04:48.680 Well, when you're testing how a website responds to your 103 00:04:48.720 --> 00:04:50.759 log in attempts, you don't want to have to run 104 00:04:50.800 --> 00:04:53.879 a massive script from the very beginning, right like triggering 105 00:04:53.879 --> 00:04:56.079 a fresh log in navigating three pages deep. 106 00:04:56.120 --> 00:04:58.600 Oh yeah, every single time you tweak one line of code, 107 00:04:58.720 --> 00:05:01.120 that would be bru It's. 108 00:05:01.000 --> 00:05:03.920 A massive waste of time. With Jupiter, you can test 109 00:05:03.959 --> 00:05:07.199 your connection in one cell, establish your session, and then 110 00:05:07.279 --> 00:05:10.160 parse the data in another cell while keeping your active 111 00:05:10.160 --> 00:05:11.639 connection totally intact. 112 00:05:11.759 --> 00:05:16.079 That's super efficient. And alongside Jupiter you have your foundational tools. 113 00:05:16.199 --> 00:05:19.079 The Requests library is your scout going out to fetch the. 114 00:05:19.160 --> 00:05:21.759 Raw data yep, your HTTP scout. 115 00:05:21.560 --> 00:05:25.000 And Beautiful Soup is your translator turning that raw HTML 116 00:05:25.040 --> 00:05:28.160 string into a navigable tree structure exactly. But those are 117 00:05:28.199 --> 00:05:30.040 the tools for the billboard. If we want to get 118 00:05:30.040 --> 00:05:31.720 into the club, and if we want to deal with 119 00:05:31.759 --> 00:05:35.199 those react and view websites that require JavaScript to run, 120 00:05:35.720 --> 00:05:37.279 we need the heavy lifter. 121 00:05:37.439 --> 00:05:38.120 We a Selenium. 122 00:05:38.240 --> 00:05:42.959 Selenium which lets Python actually automate a real web browser. 123 00:05:42.680 --> 00:05:45.800 Which is a complete paradigm shift. You know, you are 124 00:05:46.079 --> 00:05:50.879 moving from simply downloading a file to puppeteering a complex 125 00:05:50.920 --> 00:05:51.720 piece of software. 126 00:05:51.759 --> 00:05:53.160 But wait, I have to push back on this for 127 00:05:53.160 --> 00:05:54.920 a second, because this always trips people up. 128 00:05:54.959 --> 00:05:55.319 Go for it. 129 00:05:55.439 --> 00:05:58.480 If I have Python and I import the Selenium library, 130 00:05:58.639 --> 00:05:59.800 why do I also need to go out to the 131 00:05:59.800 --> 00:06:03.439 inn and download a separate, standalone executable called a web 132 00:06:03.560 --> 00:06:05.600 driver for my specific browser. 133 00:06:05.839 --> 00:06:08.000 It's a common question like if I use. 134 00:06:07.959 --> 00:06:10.639 Chrome, I need Chrome driver. Why can't Python just talk 135 00:06:10.680 --> 00:06:12.000 to my Chrome browser directly. 136 00:06:12.360 --> 00:06:15.800 What's fascinating here is the architecture of how these programs 137 00:06:15.839 --> 00:06:21.199 actually communicate. You have to remember Google, Chrome, Firefox, Safari. 138 00:06:21.480 --> 00:06:25.040 They weren't built with a native backdoor for Python scripts 139 00:06:25.079 --> 00:06:28.360 to just waltz in and take over their rendering engines. Right. 140 00:06:28.399 --> 00:06:30.040 They're built for humans exactly. 141 00:06:30.319 --> 00:06:32.920 They're built to listen to the operating system for human 142 00:06:32.959 --> 00:06:36.040 inputs like physical mouse clicks and keyboard strokes. 143 00:06:36.240 --> 00:06:39.879 So Python is basically speaking a completely different language than 144 00:06:39.920 --> 00:06:41.319 the browser's core engine. 145 00:06:41.399 --> 00:06:46.560 It is Selenium speaks a generalized automation language. Your specific 146 00:06:46.639 --> 00:06:50.839 version of Google Chrome speaks its own, highly specific, constantly 147 00:06:50.920 --> 00:06:54.319 updating language based on its underlying Chromium engine. 148 00:06:54.360 --> 00:06:55.120 Ah okay. 149 00:06:55.199 --> 00:06:58.120 So the webdriver acts as a dedicated translator between your 150 00:06:58.160 --> 00:07:00.959 Python code and the actual bread howser's debugging port. 151 00:07:01.040 --> 00:07:01.959 So it sits in the middle. 152 00:07:02.480 --> 00:07:04.800 Right. It takes a Python command like click this button 153 00:07:05.279 --> 00:07:08.040 translates it into the specific def duels protocol command that 154 00:07:08.120 --> 00:07:11.279 Chrome version one fourteen understands and executes it. 155 00:07:11.480 --> 00:07:15.000 Which explains why matching the versions is so critical. Like 156 00:07:15.120 --> 00:07:18.600 if your Chrome browser updates overnight to version one fifteen, 157 00:07:19.000 --> 00:07:21.279 but your web driver is still speaking the language of 158 00:07:21.360 --> 00:07:22.639 version one fourteen. 159 00:07:22.360 --> 00:07:24.680 The translation fails and your script crashes. 160 00:07:24.839 --> 00:07:27.439 Wow okay, So setting this up requires you to check 161 00:07:27.480 --> 00:07:31.759 your exact browser build version, download the matching web driver executable, 162 00:07:32.040 --> 00:07:34.759 and put it in your system's path variable. 163 00:07:34.399 --> 00:07:37.959 Yep, And once Selenium knows where that translator lives, it 164 00:07:38.000 --> 00:07:42.319 can initialize a real visible browser window that starts navigating 165 00:07:42.399 --> 00:07:43.319 the web on its own. 166 00:07:43.399 --> 00:07:45.399 It's a little spooky the first time you see it happen, 167 00:07:45.920 --> 00:07:47.839 just a ghost driving or browser. 168 00:07:47.519 --> 00:07:48.040 It really is. 169 00:07:48.120 --> 00:07:50.800 Okay, so our environment is set up. We have Jupiter 170 00:07:50.879 --> 00:07:54.839 for iterative testing, our scout requests, our translator beautiful Soup, 171 00:07:55.160 --> 00:07:59.000 and our robotic arm Selenium. Successfully connected to the browser 172 00:07:59.120 --> 00:07:59.920 via the web driver. 173 00:08:00.079 --> 00:08:01.199 The squad is ready, the. 174 00:08:01.120 --> 00:08:03.839 Bouncer is looking at us, and from what I understand, 175 00:08:03.959 --> 00:08:06.279 the first two major hurdles we have to clear are 176 00:08:06.519 --> 00:08:07.600 forms and cookies. 177 00:08:07.680 --> 00:08:11.040 That's right, and a web form is essentially a gateway 178 00:08:11.120 --> 00:08:13.759 that allows a user to send data back to the 179 00:08:13.800 --> 00:08:16.920 serfer for processing. In the context of scraping, the most 180 00:08:16.920 --> 00:08:19.519 common form we care about is the search bar. You 181 00:08:19.560 --> 00:08:22.959 want to retrieve specific results from a database, Okay. To 182 00:08:23.000 --> 00:08:26.120 do this, your script needs to know the architectural difference 183 00:08:26.160 --> 00:08:29.000 between a get to request and a post request. 184 00:08:29.199 --> 00:08:31.600 Well, I can usually spot a giddy request just by 185 00:08:31.639 --> 00:08:34.480 looking at my browser right. It basically staples your search 186 00:08:34.559 --> 00:08:36.279 query right onto the end of the URL. 187 00:08:36.440 --> 00:08:36.799 Exactly. 188 00:08:36.960 --> 00:08:39.519 If I search Wikipedia for Python, I can see my 189 00:08:39.559 --> 00:08:42.320 search terms right there in the address bar at the top, 190 00:08:42.519 --> 00:08:44.039 appended after a question. 191 00:08:43.799 --> 00:08:48.000 Mark that visibility makes GIDDY requests incredibly straightforward to scrape. 192 00:08:48.320 --> 00:08:51.120 You don't even really need to simulate filling out the form. 193 00:08:51.399 --> 00:08:54.919 You can often just programmatically construct the final URL string 194 00:08:54.919 --> 00:08:57.320 and Python and send a request directly to it. 195 00:08:57.360 --> 00:09:00.360 Oh that's easy, But a PS two request doesn't do that. 196 00:09:00.759 --> 00:09:01.639 No, it doesn't. 197 00:09:01.679 --> 00:09:04.000 When I log into my bank account, my password doesn't 198 00:09:04.000 --> 00:09:06.159 suddenly appear in the URL bar for the whole world 199 00:09:06.200 --> 00:09:09.679 to see. A POSD request packages that data up securely 200 00:09:09.720 --> 00:09:10.519 behind the scenes. 201 00:09:11.159 --> 00:09:14.720 Right. The payload of a p post request is embedded 202 00:09:14.720 --> 00:09:18.799 in the body of the HTTP request itself, alongside specific 203 00:09:18.799 --> 00:09:21.919 headers that tell the server what kind of data is incoming. 204 00:09:22.039 --> 00:09:23.000 So how do we script that? 205 00:09:23.480 --> 00:09:27.200 Well, if you want to scrape data hidden behind a 206 00:09:27.200 --> 00:09:30.840 PST form, your script has to be tailored to construct 207 00:09:30.840 --> 00:09:34.559 that exact payload dictionary. Okay, you're mapping the specific input 208 00:09:34.600 --> 00:09:38.279 field names to your desired values, and then you submit 209 00:09:38.320 --> 00:09:39.159 it programmatically. 210 00:09:39.279 --> 00:09:41.320 And if you mess it up, if you format. 211 00:09:41.000 --> 00:09:44.240 The payload wrong or miss a required hidden field, the 212 00:09:44.279 --> 00:09:47.200 server will just reject it or return a generic air page. 213 00:09:47.360 --> 00:09:50.639 Brutal. But here's where it gets really interesting. Though. Submitting 214 00:09:50.639 --> 00:09:52.759 a form is one thing you get past the bouncer. 215 00:09:53.120 --> 00:09:55.320 But what if you need to browse five different pages 216 00:09:55.320 --> 00:09:55.879 behind that. 217 00:09:55.840 --> 00:09:58.159 Login screen is where a lot of people get stuck. 218 00:09:58.279 --> 00:10:00.399 My instinct tells me I could just use a standard 219 00:10:01.240 --> 00:10:03.799 requests dot get command for the next page, But I 220 00:10:03.799 --> 00:10:04.440 know that fails. 221 00:10:04.559 --> 00:10:07.159 Why it fails because of a fundamental flaw on the 222 00:10:07.279 --> 00:10:11.360 HTTP protocol for our purposes, it is inherently stateless, meaning 223 00:10:11.399 --> 00:10:12.360 it is zero memory. 224 00:10:12.519 --> 00:10:14.879 Wait, so every time I click a link, the server 225 00:10:15.000 --> 00:10:16.879 basically treats me like a brand new visitor who just 226 00:10:16.919 --> 00:10:17.759 walked at the front. 227 00:10:17.519 --> 00:10:21.399 Door without a mechanism to maintain state. Yes, if you 228 00:10:21.519 --> 00:10:25.519 log in to a site via post request and then 229 00:10:25.559 --> 00:10:29.080 immediately send a new request dot get command to access 230 00:10:29.080 --> 00:10:32.399 your profile page, the server has already forgotten your successful login. 231 00:10:32.559 --> 00:10:34.039 Oh wow, it will just bounce you back to the 232 00:10:34.080 --> 00:10:37.600 login screen. And this is why cookies were invented. Cookies 233 00:10:37.639 --> 00:10:40.559 are just tiny text files the server slips into your browser. 234 00:10:40.919 --> 00:10:43.399 They contain a unique session identifier. 235 00:10:43.600 --> 00:10:45.360 Okay, I have an analogy for this. It's like the 236 00:10:45.440 --> 00:10:48.320 bouncer checking your ID, letting you into the club, and 237 00:10:48.360 --> 00:10:49.240 stamping your hand. 238 00:10:49.440 --> 00:10:49.960 I love that. 239 00:10:50.159 --> 00:10:53.159 Yes, that stamp is the cookie. When you walk from 240 00:10:53.200 --> 00:10:56.200 the bar, to the VIP lounge. The security guard inside 241 00:10:56.240 --> 00:10:58.440 doesn't need to see your ID again, he just checks 242 00:10:58.480 --> 00:10:59.039 for the snamp. 243 00:10:59.159 --> 00:11:01.840 That is the perfect analogy, and this dictates how we 244 00:11:01.840 --> 00:11:04.080 write our Python scripts. If we just use the standard 245 00:11:04.120 --> 00:11:07.720 request dot get method, we are walking into the club, 246 00:11:08.279 --> 00:11:11.360 refusing the handstamp and wondering why we keep getting thrown out? 247 00:11:11.440 --> 00:11:13.600 Right, So how do we take the stamp to solve this? 248 00:11:14.039 --> 00:11:18.679 Instead of standard requests, we initialize a request's session object. 249 00:11:18.919 --> 00:11:21.600 Ah. So the session object acts as. 250 00:11:21.399 --> 00:11:25.480 Our hand exactly. It automatically catches the cookies the server sends, 251 00:11:25.559 --> 00:11:29.480 stores them, and meticulously attaches them to every subsequent request, 252 00:11:29.559 --> 00:11:31.120 so the server knows it's still us. 253 00:11:31.480 --> 00:11:32.399 That is so clever. 254 00:11:32.600 --> 00:11:36.320 Managing that session state programmatically is what elevates a script 255 00:11:36.399 --> 00:11:39.960 from a basic pull to an advanced operation. It allows 256 00:11:39.960 --> 00:11:43.840 you to maintain the illusion of a continuous, authenticated user journey. 257 00:11:43.919 --> 00:11:46.879 So cookies solve our amnesia problem behind the scenes. We 258 00:11:46.919 --> 00:11:48.919 can log in and we can remember who. 259 00:11:48.720 --> 00:11:49.679 We are yep. 260 00:11:49.799 --> 00:11:52.600 But what happens when the data is literally locked behind 261 00:11:52.600 --> 00:11:54.840 a visual pop up on the screen that requires a 262 00:11:54.840 --> 00:11:55.840 physical mouse click. 263 00:11:56.080 --> 00:11:57.559 Ah. The interactive elements. 264 00:11:57.639 --> 00:12:00.480 Yeah, I'm thinking about those terms and conditions, check boxes 265 00:12:00.639 --> 00:12:03.159 or radio buttons where you have to select a gender 266 00:12:03.279 --> 00:12:06.320 or an industry before the next button becomes clickable. 267 00:12:06.639 --> 00:12:10.360 This where we fully transition into using Selenium to mimic 268 00:12:10.440 --> 00:12:14.559 human interaction. From a human perspective, a checkbox is just 269 00:12:14.600 --> 00:12:17.240 a minor annoyance. You see the box, you move your mouse, 270 00:12:17.399 --> 00:12:20.360 you click. But a standard Python script doesn't have eyes. 271 00:12:20.720 --> 00:12:22.720 So if I'm blocked by a checkbox, can I just 272 00:12:22.759 --> 00:12:25.240 tell Python to search the misible text of the page 273 00:12:25.279 --> 00:12:27.519 for the word agree and click on it. 274 00:12:27.799 --> 00:12:30.399 You'd think so, But the text you see rendered on 275 00:12:30.440 --> 00:12:32.840 the screen and the structural code the website reads are 276 00:12:32.840 --> 00:12:36.799 two different things. Often the text I agree is just 277 00:12:36.879 --> 00:12:40.120 a visual label floating next to the actual functional element. 278 00:12:40.200 --> 00:12:41.759 So where is the functional element? 279 00:12:42.080 --> 00:12:45.759 We have to dive into the underlying HTML document object model, 280 00:12:45.840 --> 00:12:46.240 the DOM. 281 00:12:46.360 --> 00:12:49.440 Okay, so imagine the DOM like a massive upside down tree, 282 00:12:49.519 --> 00:12:52.840 right with the main HTML tag at the route, branching 283 00:12:52.879 --> 00:12:56.519 out into body tags, div containers, and eventually the tiny 284 00:12:56.559 --> 00:12:58.399 little leaf that represents our checkbox. 285 00:12:58.440 --> 00:13:00.600 That's a great visual and to click that leaf we 286 00:13:00.639 --> 00:13:03.000 need its exact address in the tree. We use a 287 00:13:03.039 --> 00:13:06.759 locator strategy, most commonly an XPath an XPath, Yeah, an 288 00:13:06.840 --> 00:13:10.080 XPath is a syntax for navigating through elements and attributes 289 00:13:10.440 --> 00:13:12.600 in an XML or HTML document. 290 00:13:12.720 --> 00:13:15.279 But earlier you mentioned React and View making things difficult 291 00:13:15.320 --> 00:13:19.440 by dynamically generating the page. Don't those frameworks also randomize 292 00:13:19.480 --> 00:13:22.279 the class names of those elements every time the page loads. 293 00:13:22.639 --> 00:13:25.679 They absolutely do. It's a nightmare for basic scrapers. 294 00:13:25.240 --> 00:13:28.720 Right, because if the class name of the checkbox changes 295 00:13:28.720 --> 00:13:31.360 from box one twenty three to box ninety eighty seven 296 00:13:31.399 --> 00:13:34.679 on my next visit, a basic XPath looking for box 297 00:13:34.720 --> 00:13:37.440 one twenty three is going to break immediately. 298 00:13:37.000 --> 00:13:41.759 Which is why advance scraping requires writing robust relative. 299 00:13:41.559 --> 00:13:44.279 Xpaths relative xpaths. Okay, how did those work? 300 00:13:44.600 --> 00:13:47.600 Instead of looking for a hard coded randomized class name, 301 00:13:48.000 --> 00:13:51.279 you program your XPath to look for structural relationships, like 302 00:13:51.360 --> 00:13:55.080 what you might tell Selenium. Find the text terms and conditions, 303 00:13:55.320 --> 00:13:58.399 then find the div container immediately preceding it, and click 304 00:13:58.440 --> 00:14:00.559 the input tag inside that container. 305 00:14:00.759 --> 00:14:03.000 Oh, I see, you're triangulating it exactly. 306 00:14:03.480 --> 00:14:06.559 Once you establish that robust locator, you issue the dot 307 00:14:06.639 --> 00:14:09.879 click command and Selenium uses the webdriver to simulate a 308 00:14:09.919 --> 00:14:13.279 fiscal OS level mouse click on those precise coordinates in 309 00:14:13.320 --> 00:14:13.799 the browser. 310 00:14:14.000 --> 00:14:16.720 It's like giving our script a tiny robotic finger that 311 00:14:16.759 --> 00:14:19.639 knows exactly how to navigate a shifting maze. Well said, 312 00:14:19.960 --> 00:14:22.399 but clicking a box is just one action. There's a 313 00:14:22.440 --> 00:14:25.759 final boss to this whole operation that requires passing actual 314 00:14:25.840 --> 00:14:27.840 files through the browser. Uploads. 315 00:14:28.200 --> 00:14:32.559 Oh, file uploads. They are a significant hurdle, but mastering 316 00:14:32.559 --> 00:14:34.679 them unlocks a massive amount of potential. 317 00:14:34.720 --> 00:14:35.559 Give me an example. 318 00:14:35.720 --> 00:14:39.519 Think about trying to scrape data from a professional networking site. 319 00:14:39.600 --> 00:14:42.360 You often can't even see other user's data until you 320 00:14:42.559 --> 00:14:45.080 fully filled out your own profile, which means you have 321 00:14:45.120 --> 00:14:47.679 to programmatically upload a profile picture. 322 00:14:47.840 --> 00:14:48.559 Oh raw. 323 00:14:48.879 --> 00:14:51.200 Maybe you need to submit a PDF resume to a 324 00:14:51.320 --> 00:14:54.480 job board to access premium employer data. 325 00:14:54.960 --> 00:14:57.240 Now my brain immediately goes to how I do this? 326 00:14:57.279 --> 00:14:59.879 As a human? I click the upload button on the 327 00:15:00.120 --> 00:15:03.200 web page. Then a smaller window pops up on my 328 00:15:03.240 --> 00:15:06.600 screen like my Macfinder or Windows file explorer, and I 329 00:15:06.639 --> 00:15:08.639 click through my folders to find the PDF. 330 00:15:08.720 --> 00:15:10.200 That's the human way. 331 00:15:10.240 --> 00:15:12.960 But Selenium is controlling the web browser. It doesn't have 332 00:15:13.039 --> 00:15:15.960 access to my max operating system Windows. So when that 333 00:15:15.960 --> 00:15:18.720 file explorer pops up, isn't Selenium just totally blind? 334 00:15:18.720 --> 00:15:21.480 To it. It is entirely blind to it. That operating 335 00:15:21.519 --> 00:15:24.879 system dialogue box is a walled garden that browser automation 336 00:15:25.000 --> 00:15:28.919 tools cannot touch. Wait, really, really, if your script clicks 337 00:15:28.960 --> 00:15:31.639 the upload button and triggers the pop up, your automation 338 00:15:31.720 --> 00:15:32.720 effectively freezes. 339 00:15:32.960 --> 00:15:34.720 So how on earth do we get the file from 340 00:15:34.759 --> 00:15:36.960 our hard drive to the server if we can't use 341 00:15:36.960 --> 00:15:37.799 the upload window. 342 00:15:38.320 --> 00:15:42.320 We bypass the visual interface entirely and exploit the underlying 343 00:15:42.480 --> 00:15:43.519 HTML mechanics. 344 00:15:43.600 --> 00:15:44.519 Okay, I'm listening. 345 00:15:44.759 --> 00:15:47.399 When a developer builds an upload feature, they use a 346 00:15:47.399 --> 00:15:51.600 specific HTML tag. It's an input tag with the attribute 347 00:15:51.600 --> 00:15:52.159 type file. 348 00:15:52.360 --> 00:15:55.159 That's the actual engine of the upload, hiding behind the 349 00:15:55.200 --> 00:15:58.440 pretty styled click here to upload button exactly. 350 00:15:58.519 --> 00:16:00.919 The visual button is just a trigger. The hidden input 351 00:16:00.960 --> 00:16:04.559 type filetag is what actually handles the file transfer. 352 00:16:04.200 --> 00:16:05.480 So we don't click the button at all. 353 00:16:05.600 --> 00:16:08.639 Nope. Instead of using Selenium to click the visual button 354 00:16:08.799 --> 00:16:12.080 and trigger the OS window, we use our robust XPath 355 00:16:12.279 --> 00:16:14.320 to locate that hidden input. 356 00:16:13.919 --> 00:16:15.960 Tag in the dom, and once we find it. 357 00:16:15.879 --> 00:16:18.360 We use the dot send keys yeah command in Selenium. 358 00:16:18.399 --> 00:16:21.120 Wait, usually you use send keys to type text into 359 00:16:21.120 --> 00:16:21.720 a search bar. 360 00:16:21.840 --> 00:16:24.600 Right, Yes, normally, but when you use it on an 361 00:16:24.600 --> 00:16:28.159 input type filetag, and you pass it the absolute filepath 362 00:16:28.159 --> 00:16:31.159 of your PDF on your hard drive. The browser's engine 363 00:16:31.200 --> 00:16:33.120 interprets that as a direct file selection. 364 00:16:33.519 --> 00:16:37.000 Oh, that is brilliant. You just inject the filepath straight 365 00:16:37.039 --> 00:16:40.440 into the DOM and the browser attaches the file exactly 366 00:16:40.519 --> 00:16:42.679 as if you would click through the OS window. 367 00:16:42.840 --> 00:16:45.919 You bypass the pop up window entirely. And the beauty 368 00:16:45.919 --> 00:16:48.840 of this mechanism is its universality. What do you mean. 369 00:16:49.000 --> 00:16:51.919 Once you understand the DOM interaction for pushing a jpeg 370 00:16:51.960 --> 00:16:55.080 to a web server, you understand the process for pushing 371 00:16:55.120 --> 00:16:59.279 any file type, a PDF, a CSV, a text document. 372 00:16:59.360 --> 00:16:59.960 It's all the same. 373 00:17:00.159 --> 00:17:04.279 Yeah, the underlying mechanics of the HTML input tag remain identical. 374 00:17:04.559 --> 00:17:06.559 So what does this all mean When you pull back 375 00:17:06.599 --> 00:17:08.839 and look at everything we've unpacked today, it's a lot. 376 00:17:09.000 --> 00:17:13.200 It is managing post payloads, utilizing session objects to retain 377 00:17:13.200 --> 00:17:16.119 our cookies so we don't look like an amnesiac, writing 378 00:17:16.200 --> 00:17:20.200 relative X paths to navigate randomized down trees, and injecting 379 00:17:20.200 --> 00:17:23.200 filepaths to bypass OS level upload windows. 380 00:17:23.279 --> 00:17:24.240 It's a full toolkit. 381 00:17:24.480 --> 00:17:27.240 Advanced webscraping is really just the art of perfectly reverse 382 00:17:27.319 --> 00:17:31.279 engineering human browser interaction. You are teaching a Python script 383 00:17:31.319 --> 00:17:34.200 to navigate the complex architecture of the modern web at 384 00:17:34.240 --> 00:17:35.400 superhuman speeds. 385 00:17:35.799 --> 00:17:39.559 You are building a digital proxy of yourself, capable of 386 00:17:39.559 --> 00:17:43.839 handling the same stateful, complex interactions that you do every day. 387 00:17:44.079 --> 00:17:46.559 To really lock this in, let's do a quick mental 388 00:17:46.599 --> 00:17:50.240 exercise for you listening. Imagine a real world scenario. Let's 389 00:17:50.279 --> 00:17:52.240 say you want to build a tool that scrapes a 390 00:17:52.359 --> 00:17:56.079 major job board to analyze salary trends. Okay, To do this, 391 00:17:56.240 --> 00:17:59.599 your script needs to execute a sequence. First, it needs 392 00:17:59.599 --> 00:18:02.400 to log in to your account. Then it needs to 393 00:18:02.400 --> 00:18:05.759 click a checkbox that says I am seeking remote work right. Next, 394 00:18:05.799 --> 00:18:09.119 it not's to search the term Python developer, and finally 395 00:18:09.200 --> 00:18:11.680 it needs to upload your resume to access the premium 396 00:18:11.720 --> 00:18:15.480 employer's salary data. Mentally, map out which tools you would 397 00:18:15.559 --> 00:18:18.319 use for each step of that process. Where does a 398 00:18:18.359 --> 00:18:21.359 tool like beautiful Soup end and where does Selenium begin. 399 00:18:21.599 --> 00:18:25.480 It's a brilliant exercise in script architecture because the site 400 00:18:25.599 --> 00:18:28.839 likely relies heavily on JavaScript. To render those premium data 401 00:18:28.920 --> 00:18:32.519