WEBVTT 1 00:00:00.160 --> 00:00:06.240 So imagine you're architecting this really groundbreaking machine learning pipeline, right, oh, 2 00:00:06.360 --> 00:00:09.480 the dream scenario exactly. You have your models perfectly tuned, 3 00:00:09.480 --> 00:00:11.919 and your compute clusters are all spun up and ready. 4 00:00:12.199 --> 00:00:15.160 You really just need the raw material, which is, you know, the. 5 00:00:15.160 --> 00:00:18.359 Data, right, because without data, the whole thing is useless. Yeah. 6 00:00:18.440 --> 00:00:23.160 So you find this modern data rich dashboard online and 7 00:00:23.199 --> 00:00:25.839 you do what like any developer would normally do. You 8 00:00:25.879 --> 00:00:29.079 write a quick Python script using the request labrary or 9 00:00:29.120 --> 00:00:30.120 beautiful soup. 10 00:00:30.320 --> 00:00:33.679 Just a standard http gt request. 11 00:00:33.359 --> 00:00:35.679 Right exactly. You just fire it off to the target 12 00:00:35.920 --> 00:00:38.920 URL and print the response to your terminal. But instead 13 00:00:38.960 --> 00:00:41.479 of that rich structured data you saw in your browser, 14 00:00:41.880 --> 00:00:46.280 your terminal just spits out this hollow, empty HTML shell, yeah. 15 00:00:46.079 --> 00:00:49.719 With like a single script tag and literally nothing else. Right. 16 00:00:49.759 --> 00:00:51.479 The data is just completely missing. 17 00:00:51.560 --> 00:00:53.119 It's honestly a brutal realization. 18 00:00:53.240 --> 00:00:53.439 Yeah. 19 00:00:53.439 --> 00:00:55.240 I mean you are staring right at the data on 20 00:00:55.280 --> 00:00:59.119 your monitor. Yeah, but programmatically you are just entirely locked out. 21 00:00:59.200 --> 00:01:00.920 We've all hit that one at some point, and that 22 00:01:01.079 --> 00:01:04.079 is exactly what we're dismantling in today's deep dive. 23 00:01:04.239 --> 00:01:05.200 I love this topic. 24 00:01:05.439 --> 00:01:08.719 It's so relevant, right, our mission today is to decode 25 00:01:08.719 --> 00:01:12.359 the mechanics of advanced web scraping using Python and Selenium. 26 00:01:12.640 --> 00:01:15.599 We're going to figure out how to bypass those MPHTML 27 00:01:15.640 --> 00:01:20.000 shells and really explore how modern front end architectures load 28 00:01:20.079 --> 00:01:21.000 data and. 29 00:01:21.120 --> 00:01:24.000 More importantly, look at the actual programmatic tools you need 30 00:01:24.040 --> 00:01:25.359 to extract it exactly. 31 00:01:25.840 --> 00:01:29.840 Let's unpack this before we unleash the automation frameworks. We 32 00:01:29.879 --> 00:01:34.159 need to understand why standard HTTP requests fail so spectacularly 33 00:01:34.200 --> 00:01:35.640 on modern web applications. 34 00:01:35.799 --> 00:01:38.359 What it really comes down to this fundamental architectural shift 35 00:01:38.400 --> 00:01:40.400 and how the web is built these days, we have 36 00:01:40.480 --> 00:01:43.680 basically moved from server side rendering to client side rendering 37 00:01:44.079 --> 00:01:45.239 or CSR. 38 00:01:45.439 --> 00:01:46.799 Okay, so break that down for us. 39 00:01:46.840 --> 00:01:49.680 Sure. So when you scrape a classic static web page, 40 00:01:49.959 --> 00:01:52.799 say like a standard Wikipedia article, the server does all 41 00:01:52.799 --> 00:01:56.000 the heavy lifting. It constructs the final HTML document with 42 00:01:56.079 --> 00:01:58.000 all the text already in it, and sends it over 43 00:01:58.040 --> 00:01:58.400 the wire. 44 00:01:58.519 --> 00:02:01.079 So your basic requests liber he just grabs that file 45 00:02:01.120 --> 00:02:03.159 and boom you have your data exactly. 46 00:02:03.760 --> 00:02:06.680 But with client side rendering, which by the way powers 47 00:02:06.719 --> 00:02:10.319 almost every modern web app built on React or Angular 48 00:02:10.439 --> 00:02:13.879 or View, the server doesn't send the data initially. 49 00:02:13.919 --> 00:02:15.000 Wait, so what does it send. 50 00:02:15.319 --> 00:02:18.439 It just sends a skeletal HTML file and this massive 51 00:02:18.479 --> 00:02:22.240 bundle of JavaScript. The browser then has to actually execute 52 00:02:22.240 --> 00:02:25.560 that JavaScript to fetch the data via background. 53 00:02:25.080 --> 00:02:28.280 APIs Oh, I see, So it dynamically injects it into 54 00:02:28.360 --> 00:02:29.159 the page. 55 00:02:28.919 --> 00:02:32.080 After the fat right, which means if you're scraping cool 56 00:02:32.120 --> 00:02:34.879 doesn't have a built in JavaScript engine to execute that code. 57 00:02:35.240 --> 00:02:38.240 The data literally never renders. The page just stays blank. 58 00:02:38.400 --> 00:02:41.280 That makes so much sense. Standard scrapers are basically acting 59 00:02:41.360 --> 00:02:44.199 like a camera taking a snapshot the exact millisecond the 60 00:02:44.199 --> 00:02:44.960 front door opens. 61 00:02:45.039 --> 00:02:46.039 It's a perfect way to put it. 62 00:02:46.199 --> 00:02:48.840 Yeah, Like, if the data requires an action to appear, 63 00:02:49.000 --> 00:02:52.159 like the JavaScript executing or a user scrolling down to 64 00:02:52.159 --> 00:02:55.080 trigger a lazy load function, that snapshot is just going 65 00:02:55.159 --> 00:02:57.599 to capture an empty room exactly. I actually like to 66 00:02:57.639 --> 00:03:01.599 think of the document object model as an interactive stage play. 67 00:03:01.879 --> 00:03:02.919 I like this analogy. 68 00:03:03.159 --> 00:03:06.520 Yeah. So in the old days of static sites, every 69 00:03:06.560 --> 00:03:09.159 actor was on stage the moment the curtain went up, 70 00:03:09.879 --> 00:03:12.560 but in a modern dynamic web app the stage is 71 00:03:12.680 --> 00:03:14.560 mostly empty at the start. 72 00:03:14.360 --> 00:03:16.960 Right, The actors are all just wading in the wings exactly. 73 00:03:17.560 --> 00:03:20.719 The actors which represent your data in this case, they 74 00:03:20.800 --> 00:03:24.759 only enter the stage when specific cues are called. Sometimes 75 00:03:24.759 --> 00:03:27.879 the queue is a timer, sometimes it's a user scrolling 76 00:03:27.919 --> 00:03:29.840 to the bottom of the page or hitting a load 77 00:03:29.919 --> 00:03:30.520 more button. 78 00:03:30.759 --> 00:03:33.360 And standard tools like Beautiful Soup are just sitting in 79 00:03:33.400 --> 00:03:36.719 the audience reading the initial script. They have absolutely no 80 00:03:37.000 --> 00:03:40.000 way to shout those cues to get the actors. 81 00:03:39.719 --> 00:03:42.120 On stage, right, They can't interact to get. 82 00:03:41.919 --> 00:03:45.039 That dynamically rendered data. You need a tool that can 83 00:03:45.080 --> 00:03:47.960 interact with the dom exactly like a human user would. 84 00:03:48.199 --> 00:03:51.039 You need a way to physically trigger those events programmatically. 85 00:03:51.400 --> 00:03:53.479 Before we get into the exact tool for that, though, 86 00:03:53.719 --> 00:03:55.400 we really need to talk about prerequisite. 87 00:03:55.439 --> 00:03:57.719 Oh definitely. You can't just jump into this blind right. 88 00:03:57.560 --> 00:04:01.120 Because webscripting is essentially extracting information from a web page 89 00:04:01.120 --> 00:04:04.639 into a structured format for analysis, and to do that 90 00:04:04.680 --> 00:04:06.280 you need some foundational knowledge. 91 00:04:06.479 --> 00:04:11.120 Yet you really need basic to intermediate Python programming skills, 92 00:04:11.800 --> 00:04:15.680 and honestly, a solid understanding of HTML and CSS is 93 00:04:15.840 --> 00:04:16.680 non negotiable. 94 00:04:16.759 --> 00:04:19.720 It's like going back with the stage play analogy, or actually, 95 00:04:19.800 --> 00:04:22.360 let's say it's like sending a digital vacuum cleaner into 96 00:04:22.399 --> 00:04:27.319 a massive library to suck up specific sentences from specific books. 97 00:04:27.519 --> 00:04:27.680 OK. 98 00:04:28.240 --> 00:04:31.600 Knowing HTML and CSS is like having the architectural floor 99 00:04:31.639 --> 00:04:33.600 plan of that library. If you don't know how the 100 00:04:33.680 --> 00:04:35.480 rooms are laid out, your vacuum is just going to 101 00:04:35.519 --> 00:04:36.360 bump into walls. 102 00:04:36.600 --> 00:04:38.560 If we connect this to the bigger picture, though, this 103 00:04:38.639 --> 00:04:41.759 isn't just about coding, It's about data analytics. 104 00:04:41.800 --> 00:04:42.519 Oh. Absolutely. 105 00:04:42.639 --> 00:04:46.000 Having a fundamental understanding of analytics gives you the why 106 00:04:46.560 --> 00:04:50.279 behind your scraping. It dictates exactly what structured data you're 107 00:04:50.279 --> 00:04:51.279 trying to pull in the first place. 108 00:04:51.319 --> 00:04:53.439 You need a gold exactly. You aren't just scraping for 109 00:04:53.480 --> 00:04:55.439 the fun of it. Well maybe some people are, but 110 00:04:55.519 --> 00:04:56.879 usually you have a pipeline to feed. 111 00:04:57.079 --> 00:05:00.319 Right, So we know why we are scraping and what 112 00:05:00.399 --> 00:05:03.560 background we need, and we know that static tools are 113 00:05:03.639 --> 00:05:07.199 totally useless against infinitive scrolls and clickable menus. 114 00:05:07.079 --> 00:05:11.079 Which brings us to Selenium. Here's where it gets really interesting. 115 00:05:11.680 --> 00:05:16.079 Selenium is basically this set of tools and libraries that 116 00:05:16.199 --> 00:05:18.120 automate web browser actions. 117 00:05:18.319 --> 00:05:20.720 It is essentially a digital puppet master. 118 00:05:20.680 --> 00:05:23.879 Yes, or like a robot butler. It's not just a 119 00:05:23.959 --> 00:05:27.439 scraping script. It is literally opening a real browser and 120 00:05:27.480 --> 00:05:29.360 clicking around on your behalf and it. 121 00:05:29.319 --> 00:05:32.800 Is incredibly flexible. It's free, it's open source, and it 122 00:05:32.839 --> 00:05:34.959 works across Windows, mac Os, and Linux. 123 00:05:35.040 --> 00:05:38.160 Plus, it supports a massive variety of programming languages right. 124 00:05:38.040 --> 00:05:41.600 Now, Oh yeah, Python, Java, Ruby, Scala, see, you name it. 125 00:05:41.720 --> 00:05:43.759 You can basically use whatever you are most. 126 00:05:43.600 --> 00:05:46.600 Comfortable with, though Python really seems to be the crowd 127 00:05:46.759 --> 00:05:48.920 favorite for data extraction definitely. 128 00:05:49.439 --> 00:05:52.759 The brilliance of Selenium's flexibility is that it essentially acts 129 00:05:52.800 --> 00:05:55.959 as a universal translator between your Python code and the 130 00:05:56.000 --> 00:05:57.639 web browser's native actions. 131 00:05:58.040 --> 00:06:00.600 Wait, looking at the history here, I realized something crazy. 132 00:06:00.680 --> 00:06:04.040 Selenium was never actually designed to be a data scraper. 133 00:06:03.680 --> 00:06:06.120 Not at all. What's fascinating here is that Selenium was 134 00:06:06.120 --> 00:06:08.720 born out of sheer developer frustration back in the early 135 00:06:08.759 --> 00:06:11.639 two thousands. It was strictly a quality assurance tool. 136 00:06:11.680 --> 00:06:13.240 Wait really just for QA. 137 00:06:13.319 --> 00:06:17.199 Yeah, it was built entirely for automated web application user 138 00:06:17.240 --> 00:06:18.160 interface testing. 139 00:06:18.399 --> 00:06:20.439 Okay, well that actually makes perfect sense when you think 140 00:06:20.480 --> 00:06:22.839 about it. If you're a front end developer and you 141 00:06:22.959 --> 00:06:26.879 just pushed a massive update to a complex e commerce 142 00:06:27.040 --> 00:06:30.240 checkout flow, you don't want to test that manually, exactly. 143 00:06:30.279 --> 00:06:32.920 You don't want to physically open Chrome type in a 144 00:06:32.959 --> 00:06:35.639 fake email, click add to KRT, fill out a fake 145 00:06:35.680 --> 00:06:37.879 credit card, and click submit one hundred. 146 00:06:37.639 --> 00:06:40.040 Times just to make sure it doesn't crash on edge cases. 147 00:06:40.160 --> 00:06:41.199 That would be a nightmare. 148 00:06:41.319 --> 00:06:45.240 So they built Selenium Remote Control, which is historically referred 149 00:06:45.240 --> 00:06:48.240 to as Selenium one point zero. It allowed developers to 150 00:06:48.240 --> 00:06:51.480 write scripts that would automatically drive the browser through those 151 00:06:51.519 --> 00:06:53.279 exact UI workflows, so it. 152 00:06:53.240 --> 00:06:55.759 Would just mimic the key strokes and mouse clicks to 153 00:06:55.800 --> 00:06:56.920 test the infrastructure. 154 00:06:57.040 --> 00:06:59.920 Right. But then data scientists and analysts looked at that 155 00:07:00.120 --> 00:07:05.000 capability and essentially hijacked the testing tool for their own purposes. 156 00:07:05.120 --> 00:07:08.000 I love that data scientists are basically hackers in their 157 00:07:08.000 --> 00:07:08.439 own right. 158 00:07:08.600 --> 00:07:12.639 Pretty much. They realized that if a framework is sophisticated 159 00:07:12.800 --> 00:07:16.399 enough to mimic a human clicking a log in button 160 00:07:16.439 --> 00:07:20.279 to test a database, it is definitely sophisticated enough to 161 00:07:20.439 --> 00:07:23.759 mimic a human clicking a load more button to expose 162 00:07:23.839 --> 00:07:24.639 hidden data. 163 00:07:24.720 --> 00:07:28.360 It was the ultimate workaround. It solved the exact problem 164 00:07:28.439 --> 00:07:31.639 that static screepers couldn't handle because it forced the browser 165 00:07:31.720 --> 00:07:35.040 to execute the JavaScript and render the dom fully exactly. 166 00:07:35.079 --> 00:07:38.000 It gets those actors onto the stage before taking the picture. 167 00:07:38.360 --> 00:07:40.879 Let's dig into the actual mechanics of how that works, 168 00:07:40.920 --> 00:07:43.959 because this is where the engineering gets really impressive. When 169 00:07:44.000 --> 00:07:46.720 we talk about Selenium today, we're really talking about its 170 00:07:46.879 --> 00:07:49.360 core component, which is the Selenium webdriver. 171 00:07:49.600 --> 00:07:51.680 Right. The webdriver is the heart of the operation. 172 00:07:51.959 --> 00:07:54.839 And it isn't just a basic macro recorder. Right, It's 173 00:07:54.839 --> 00:07:58.480 not just moving the mouse to specific screen coordinates. How 174 00:07:58.480 --> 00:08:02.079 does the web driver actually translate a Python script into 175 00:08:02.120 --> 00:08:05.000 a physical action inside something like Google Chrome? 176 00:08:05.399 --> 00:08:08.920 So it operates through a highly structured translation layer. When 177 00:08:08.959 --> 00:08:11.959 you write a Python script using the Selenium bindings, you 178 00:08:11.959 --> 00:08:14.279 aren't actually talking directly to Chrome. 179 00:08:14.240 --> 00:08:17.079 Because Chrome doesn't natively understand Python. 180 00:08:16.839 --> 00:08:21.279 Right exactly. Instead, your Python code sends commands via an 181 00:08:21.319 --> 00:08:25.040 API using a standardized protocol. It's called a W three 182 00:08:25.120 --> 00:08:26.360 C webdriver protocol. 183 00:08:26.560 --> 00:08:29.079 Okay, And how does that protocol transmit the data? 184 00:08:29.199 --> 00:08:33.320 It transmits everything using JSON over each tdp Oh wow. 185 00:08:33.440 --> 00:08:35.879 So if I write a Python command that says, uh, 186 00:08:36.159 --> 00:08:39.639 driver dot find element by ID, submit button dot click, 187 00:08:40.159 --> 00:08:44.600 Python serializes that exact command into a Jason payload. Yes, 188 00:08:44.679 --> 00:08:46.440 exactly where does that payload actually go? 189 00:08:46.600 --> 00:08:49.559 It goes to a middleman known as the browser driver executable. 190 00:08:49.879 --> 00:08:53.639 Every major browser has its own specific driver, usually maintained 191 00:08:53.639 --> 00:08:55.120 by the browser creators themselves. 192 00:08:55.120 --> 00:08:57.840 I like Chrome driver for Chrome right, or get go driver. 193 00:08:57.759 --> 00:09:01.639 For Firefox, edge driver for Edge. This executable x as 194 00:09:01.679 --> 00:09:04.840 the bridge. It receives that Jason payload, decodes it and 195 00:09:04.879 --> 00:09:07.320 translates it into the low level C plus plus or 196 00:09:07.679 --> 00:09:11.720 native commands that physically control the browser's internal engine. 197 00:09:11.799 --> 00:09:14.399 That is just fascinating. So the web driver doesn't bypass 198 00:09:14.440 --> 00:09:16.440 the browser at all. It drives the browser from the 199 00:09:16.480 --> 00:09:19.480 inside out using its own native hooks exactly. 200 00:09:19.600 --> 00:09:23.200 The website registers the interaction not as an external script, 201 00:09:23.440 --> 00:09:26.320 but as a genuine system level mouse click or. 202 00:09:26.360 --> 00:09:29.240 Keystroke, and that is why it is so effective against 203 00:09:29.240 --> 00:09:33.360 complex doms. The browser is genuinely rendering the page, executing 204 00:09:33.399 --> 00:09:35.919 the JavaScript, and responding to the events, just as it 205 00:09:35.919 --> 00:09:37.200 would for you or me. Right. 206 00:09:37.399 --> 00:09:40.799 Once the native click happens, the react or Angular app 207 00:09:40.879 --> 00:09:45.000 running on the page catches that event, fires its internal logic, 208 00:09:45.039 --> 00:09:46.240 and fetches the new data. 209 00:09:46.720 --> 00:09:50.120 But getting to that data requires knowing exactly where to click. 210 00:09:50.159 --> 00:09:52.320 You can't just pass a command to Selenium saying hey, 211 00:09:52.399 --> 00:09:53.919 click the big blue button on the left. 212 00:09:54.000 --> 00:09:55.200 Yeah, I wish it were that easy. 213 00:09:55.399 --> 00:09:58.159 Right, you have to give it the exact architectural coordinate 214 00:09:58.240 --> 00:10:00.879 of that element within the HT plan. 215 00:10:01.279 --> 00:10:04.159 Finding the elements is often, honestly the most challenging part 216 00:10:04.159 --> 00:10:08.000 of writing a robust scraper. You're constantly navigating the dom tree. 217 00:10:08.039 --> 00:10:10.559 And there are two primary methods for this, right, CSS 218 00:10:10.559 --> 00:10:11.879 selectors and XPath. 219 00:10:12.120 --> 00:10:15.480 Yes, well, CSS selectors are great for finding elements based 220 00:10:15.480 --> 00:10:18.840 on their styling classes. XPath is the real powerhouse for 221 00:10:18.879 --> 00:10:19.759 complex scraping. 222 00:10:20.000 --> 00:10:22.320 Why is XPath so much better for the complex stuff? 223 00:10:22.480 --> 00:10:26.240 Because XPath allows you to traverse the dom relationally. You 224 00:10:26.279 --> 00:10:28.279 don't just have to look for an element with a 225 00:10:28.320 --> 00:10:29.200 specific ID. 226 00:10:29.279 --> 00:10:32.200 Oh right, You can write an XPath query that basically says, 227 00:10:33.639 --> 00:10:36.039 find the third table on the page, go to the 228 00:10:36.080 --> 00:10:39.919 second row, find the cell containing the word revenue, and 229 00:10:40.039 --> 00:10:42.559 then grab the text from the cell immediately next to 230 00:10:42.600 --> 00:10:43.320 it exactly. 231 00:10:43.639 --> 00:10:47.279 That relational traversal is absolutely essential because modern web apps 232 00:10:47.600 --> 00:10:50.159 frequently use dynamically generated class names. 233 00:10:50.759 --> 00:10:54.200 Oh like if they use tailwind CSS or styled components. 234 00:10:54.240 --> 00:10:56.320 Right, the class name of a button might be a 235 00:10:56.399 --> 00:10:59.360 random string of characters like btn XJA seven two P 236 00:10:59.559 --> 00:11:02.759 today and then it's completely different tomorrow after a new deployment. 237 00:11:02.919 --> 00:11:05.919 And if your script relies on the static class name, 238 00:11:05.960 --> 00:11:07.240 it just breaks instantly. 239 00:11:07.360 --> 00:11:10.039 Ye XPath allows you to navigate based on the structure 240 00:11:10.080 --> 00:11:13.000 and content of the page rather than volatile styling hooks. 241 00:11:13.080 --> 00:11:15.519 Okay, so we have our Python script. It's sending Jason 242 00:11:15.559 --> 00:11:17.960 Pailott's to Chrome driver. It's driving the browser, and we're 243 00:11:18.000 --> 00:11:20.639 using XPath to pinpoint the exact dynamic. 244 00:11:20.279 --> 00:11:22.320 Elements we want sell like a solid pipeline. 245 00:11:22.440 --> 00:11:25.879 It is, But there's a massive trap here that trips 246 00:11:25.960 --> 00:11:29.039 up almost everyone when they first start building these automated systems. 247 00:11:29.279 --> 00:11:30.200 Has to do with timing. 248 00:11:30.320 --> 00:11:33.159 Oh yeah, the asynchronous nature of the modern web. 249 00:11:33.320 --> 00:11:37.320 Yes. Let's say you tell Selenium to click that load 250 00:11:37.320 --> 00:11:39.879 more button and then immediately tell it to scrape the 251 00:11:39.879 --> 00:11:43.559 new data that appears. If you execute those commands back 252 00:11:43.600 --> 00:11:47.080 to back, your script is going to crash. 253 00:11:46.360 --> 00:11:49.639 Every single time. It'll throw a no silk element exception. 254 00:11:49.879 --> 00:11:50.720 Why does that happen? 255 00:11:51.159 --> 00:11:55.320 Because your Python code executes in milliseconds, but the browser 256 00:11:55.360 --> 00:11:57.279 takes actual time to respond. 257 00:11:57.000 --> 00:11:58.480 Right, It has to do the network round. 258 00:11:58.279 --> 00:12:02.799 Trip Exactly when Selenium clicks load more, the website has 259 00:12:02.840 --> 00:12:05.639 to send a network request to its server, wait for 260 00:12:05.679 --> 00:12:07.960 the database to query the new items, send them back 261 00:12:07.960 --> 00:12:11.159 over the wire, and then their browser's JavaScript engine has 262 00:12:11.200 --> 00:12:14.000 to parse that data and inject it into the DOM. 263 00:12:13.960 --> 00:12:16.200 Which could take half a second or it could take 264 00:12:16.240 --> 00:12:18.960 three seconds, depending on your network latency. 265 00:12:19.080 --> 00:12:21.600 Exactly. If your script tries to grab the data before 266 00:12:21.639 --> 00:12:24.559 that entire process finishes, it's looking for an actor that 267 00:12:24.559 --> 00:12:26.080 hasn't stepped onto the stage yet. 268 00:12:26.240 --> 00:12:28.080 So how do we handle that? Because I know the 269 00:12:28.080 --> 00:12:31.639 amateur approach is to just use Python's time dot sleep module. 270 00:12:32.000 --> 00:12:34.240 You click the button, tell the script to freeze completely 271 00:12:34.240 --> 00:12:36.159 for like five seconds, and then look for the data. 272 00:12:36.279 --> 00:12:38.759 Yeah, you really want to avoid that. Hard Coded sleeps 273 00:12:38.799 --> 00:12:41.039 are highly inefficient and super. 274 00:12:40.720 --> 00:12:43.320 Brittle because the timing is always unpredictable. 275 00:12:43.559 --> 00:12:46.799 Right, if the network is fast and the data loads 276 00:12:46.840 --> 00:12:49.919 in one second, you are wasting four seconds of compute 277 00:12:49.919 --> 00:12:53.919 time doing absolutely nothing. Multiply that by ten thousand pages 278 00:12:54.120 --> 00:12:57.360 and you've just added hours of unnecessary delay to or pipeline. 279 00:12:57.399 --> 00:12:59.279 Oh wow, Yeah, that scales terribly. 280 00:12:59.519 --> 00:13:02.879 And if the server's underload and take six seconds to respond, 281 00:13:03.279 --> 00:13:05.759 your five second sleep isn't long enough anyway, and your 282 00:13:05.799 --> 00:13:07.120 script crashes. 283 00:13:07.000 --> 00:13:10.039 Which introduces one of the most powerful concepts in the 284 00:13:10.080 --> 00:13:12.799 Selenium Webdriver explicit weights. 285 00:13:12.960 --> 00:13:15.120 Yes, explicit weights are a game changer. 286 00:13:15.240 --> 00:13:18.279 Instead of freezing the entire script for an arbitrary amount 287 00:13:18.320 --> 00:13:22.080 of time, you leverage Selenium's webdriver weight class combined with 288 00:13:22.200 --> 00:13:23.279 expected conditions. 289 00:13:23.600 --> 00:13:26.440 This is where you basically tell the webdriver, Hey, I 290 00:13:26.519 --> 00:13:29.799 want you to pause, but keep actively pulling the DOM 291 00:13:30.000 --> 00:13:32.120 every five hundred milliseconds. 292 00:13:31.840 --> 00:13:33.519 Just constantly checking in right. 293 00:13:33.600 --> 00:13:35.919 You tell it to do this until a specific element 294 00:13:36.120 --> 00:13:39.759 physically becomes present on the page, or until a maximum 295 00:13:39.799 --> 00:13:40.960 timeout is reached. 296 00:13:40.960 --> 00:13:44.200 So it's reactive rather than passive. You are tying the 297 00:13:44.240 --> 00:13:47.240 execution of your code directly to the state of the 298 00:13:47.279 --> 00:13:48.320 application itself. 299 00:13:48.440 --> 00:13:51.240 Exactly the moment the new data renders in the DOM, 300 00:13:51.279 --> 00:13:54.399 Selenium detects it, stops waiting and instantly grabs it. 301 00:13:54.399 --> 00:13:57.320 It optimizes the speed of your scraper perfectly to the 302 00:13:57.320 --> 00:13:58.559 response time of the server. 303 00:13:58.799 --> 00:14:02.360 That is so elegant it makes your automation incredibly resilient. 304 00:14:02.639 --> 00:14:06.879 Building in these intelligent weights, combined with robust XPath locators, 305 00:14:07.200 --> 00:14:10.440 that's what separates a fragile script from an enterprise grade 306 00:14:10.519 --> 00:14:11.879 data pipeline totally. 307 00:14:12.000 --> 00:14:14.519 Now let's look at the broader development life cycle. Because 308 00:14:14.519 --> 00:14:17.879 you don't just jump straight into writing complex Python scripts 309 00:14:17.919 --> 00:14:19.320 with explicit weights right away. 310 00:14:19.639 --> 00:14:22.919 No, you really shouldn't. You need a structured approach to 311 00:14:22.960 --> 00:14:26.720 building these systems utilizing the other components of the Selenium suite. 312 00:14:26.919 --> 00:14:29.879 Right before you spend hours writing Python code to navigate 313 00:14:29.919 --> 00:14:33.360 some crazy twenty step authentication process, you probably want to 314 00:14:33.480 --> 00:14:35.799 verify that the workflow is even viable. 315 00:14:35.679 --> 00:14:38.720 And that is exactly where the Selenium IDE comes into play. 316 00:14:38.919 --> 00:14:40.919 Okay, what is the IDE exactly? 317 00:14:41.000 --> 00:14:44.200 It is a really lightweight browser extension that serves as 318 00:14:44.200 --> 00:14:48.200 an excellent prototyping environment. It basically provides a visual interface 319 00:14:48.279 --> 00:14:49.440 to record your actions. 320 00:14:49.480 --> 00:14:52.200 Oh so you just hit record and navigate the site yourself. 321 00:14:52.279 --> 00:14:56.240 Yep, you click around and the IDE automatically logs your clicks, inputs, 322 00:14:56.240 --> 00:14:57.000 and navigations. 323 00:14:57.080 --> 00:14:59.960 That's amazing. So it generates like a rough draft of the. 324 00:15:00.120 --> 00:15:02.480 Logic exactly, and you can immediately play it back to 325 00:15:02.480 --> 00:15:04.519 see if there are hidden eye frames blocking your path 326 00:15:04.679 --> 00:15:06.759 or dynamic pop ups that interrupt the flow. 327 00:15:07.080 --> 00:15:10.159 It's a low friction way to basically map out your 328 00:15:10.200 --> 00:15:14.639 don traversal strategy before you commit to engineering the actual 329 00:15:14.679 --> 00:15:16.240 Python webdriver script. 330 00:15:16.320 --> 00:15:18.879 It saves so much time. Yeah, but once you do 331 00:15:19.000 --> 00:15:22.360 engineer that Python script, you face the final hurdle of 332 00:15:22.480 --> 00:15:26.200 data extraction just a scale. Yep, Running a Selenium script 333 00:15:26.240 --> 00:15:29.240 locally on your laptop is fantastic for a few hundred pages, 334 00:15:29.480 --> 00:15:32.440 but what if your analytics project requires scraping one hundred 335 00:15:32.480 --> 00:15:34.360 thousand product pages every single night. 336 00:15:34.720 --> 00:15:38.919 I mean Selenium is inherently resource intensive, right because you 337 00:15:38.960 --> 00:15:42.200 are opening real instances of Google Chrome. It eats up 338 00:15:42.279 --> 00:15:43.600 RAM and CPU cycle. 339 00:15:43.720 --> 00:15:46.360 Oh yeah, it'll destroy your local resources. If you try 340 00:15:46.360 --> 00:15:49.720 to run fifty browsers simultaneously on a standard machine, it 341 00:15:49.799 --> 00:15:51.279 is just going to grind to a halt. 342 00:15:51.360 --> 00:15:54.120 Which brings us to another major component of the suite, 343 00:15:54.399 --> 00:15:55.360 Selenium grid. 344 00:15:55.879 --> 00:15:59.759 Yes, remember how we discuss Selenium's origins in UI testing. 345 00:16:00.360 --> 00:16:03.320 The grid was originally developed to solve the problem of 346 00:16:03.440 --> 00:16:06.720 testing a web app across dozens of different browser and 347 00:16:06.799 --> 00:16:09.279 operating system combinations simultaneously. 348 00:16:09.360 --> 00:16:12.320 But just like the web driver, data engineers repurpose the 349 00:16:12.360 --> 00:16:14.600 grid for distributed scraping exactly. 350 00:16:14.759 --> 00:16:16.799 Seleaning grid allows you to set up a central hub 351 00:16:17.159 --> 00:16:20.240 that distributes commands to multiple remote nodes. 352 00:16:20.360 --> 00:16:22.360 Oh wow, so you can spin up a cluster of 353 00:16:22.360 --> 00:16:25.559 servers in the cloud, register them as nodes to your grid. 354 00:16:25.639 --> 00:16:29.399 Hub and suddenly your single Python script can orchestrate hundreds 355 00:16:29.399 --> 00:16:31.919 of browser instances running in parallel. 356 00:16:31.559 --> 00:16:35.559 All across different machines. It's parallel processing for browser automation. 357 00:16:35.799 --> 00:16:38.159 You could tear through massive data sets in a fraction 358 00:16:38.240 --> 00:16:38.600 of the time. 359 00:16:38.639 --> 00:16:42.120 Doing that definitely, and to optimize it even further, you 360 00:16:42.120 --> 00:16:44.720 would typically run those remote browsers in headless mode. 361 00:16:44.840 --> 00:16:47.399 Headless mode, I love that term. What does it actually do? 362 00:16:47.960 --> 00:16:52.360 Headless browsing is absolutely essential for production deployments. It basically 363 00:16:52.360 --> 00:16:55.919 instructs the browser executable to run all of its internal 364 00:16:55.960 --> 00:17:01.080 rendering logic, processing the HTML, executing the Java script, constructing 365 00:17:01.080 --> 00:17:04.759 the DOM tree, but without actually generating a graphical user interface. 366 00:17:04.920 --> 00:17:06.960 Oh so it doesn't draw the pixels on a screen. 367 00:17:07.160 --> 00:17:10.640 Right. By dropping that requirement, you significantly reduce the memory 368 00:17:10.640 --> 00:17:14.240 footprint and CP overhead of each browser instance. 369 00:17:14.480 --> 00:17:18.599 So the browser exists purely in the machine's memory, completely invisible, 370 00:17:18.640 --> 00:17:21.960 but operating just as effectively. Exactly so, we look at 371 00:17:21.960 --> 00:17:24.519 the complete architecture we built here. We use the ID 372 00:17:24.880 --> 00:17:28.680 to prototype our traversal strategy. We write our production logic 373 00:17:28.720 --> 00:17:32.200 and Python leveraging XPath to navigate complex DOM structures and 374 00:17:32.279 --> 00:17:36.480 explicit weights to handle asynchronous rendering. Then that Python code 375 00:17:36.519 --> 00:17:40.079 sends JSON commands to a headless Chrome driver, and finally 376 00:17:40.079 --> 00:17:44.039 we scale the entire operation across multiple servers. Using Selenium grid. 377 00:17:44.400 --> 00:17:48.160 You have literally transitioned from simply downloading a static file 378 00:17:48.640 --> 00:17:53.680 to engineering a massive fleet of automated, invisible browsers, browsers. 379 00:17:53.279 --> 00:17:56.319 That are capable of interacting with the most sophisticated web 380 00:17:56.359 --> 00:18:00.160 applications on the Internet. It completely changes how you approach 381 00:18:00.359 --> 00:18:02.960 data gathering for machine learning and analytics. 382 00:18:03.079 --> 00:18:05.279 It really does. You are no longer limited to what 383 00:18:05.319 --> 00:18:07.759 an API provides or what is readily available in the 384 00:18:07.759 --> 00:18:13.720 CSV file. The entire interactive web basically becomes an accessible database. 385 00:18:13.400 --> 00:18:16.759 That is so empowering. So to reinforce the architecture we've 386 00:18:16.839 --> 00:18:19.559 unpacked today for you listening at home, let's do a 387 00:18:19.599 --> 00:18:22.240 quick mental exercise. Oh I like these, Think about the 388 00:18:22.240 --> 00:18:24.559 matching of your tool to your target. If you are 389 00:18:24.680 --> 00:18:28.160 tasked with pulling a table of historical environmental regulations from 390 00:18:28.200 --> 00:18:31.880 a plain text nineteen nineties government database, do you need 391 00:18:31.920 --> 00:18:33.799 to spin up a Selenium webdriver. 392 00:18:34.319 --> 00:18:35.480 Take a second to think about it. 393 00:18:35.599 --> 00:18:38.160 The answer is absolutely not. That is a static, server 394 00:18:38.240 --> 00:18:40.759 side rendered architecture all the data is delivered in the 395 00:18:40.839 --> 00:18:42.440 initial HTML payload. 396 00:18:42.640 --> 00:18:44.839 Firing up a headless browser for that would be a 397 00:18:44.880 --> 00:18:46.319 massive waste of compute. 398 00:18:46.480 --> 00:18:50.440 Right, You use a lightweight HTTP request for the nineties database, 399 00:18:50.720 --> 00:18:53.279 and you save your heavy duty browser automation for the 400 00:18:53.319 --> 00:18:56.720 modern dynamic sites relying on React and infinite scrolling. 401 00:18:57.640 --> 00:18:59.880