WEBVTT 1 00:00:00.040 --> 00:00:01.639 You know, when you open up a web browser and 2 00:00:01.720 --> 00:00:05.960 navigate to your favorite site, it feels, well, it feels 3 00:00:06.000 --> 00:00:10.720 a bit like walking into a well organized, peaceful public library, right. 4 00:00:11.000 --> 00:00:13.480 Yeah, everything is just sort of there, exactly. 5 00:00:13.599 --> 00:00:16.320 The information is just sitting there, neatly cataloged on the shelves, 6 00:00:16.359 --> 00:00:17.480 waiting for you to read it. 7 00:00:17.480 --> 00:00:21.519 It definitely feels very calm from the user's perspective. I mean, 8 00:00:21.920 --> 00:00:24.679 you type in a URL, hit enter, and a beautiful 9 00:00:24.800 --> 00:00:26.320 page renders on your screen. 10 00:00:26.399 --> 00:00:28.640 But if you peak behind the curtain, you know, under 11 00:00:28.679 --> 00:00:31.679 the surface of the web, it is not a peaceful 12 00:00:31.719 --> 00:00:34.600 library at all. It is basically an invisible war. 13 00:00:34.920 --> 00:00:37.280 Oh absolutely, an invisible high speed war. 14 00:00:37.520 --> 00:00:41.000 Every single second of every day there is this massive, 15 00:00:41.600 --> 00:00:45.479 relentless tug of war happening. You've got these automated webscrapers, 16 00:00:45.520 --> 00:00:49.679 these bots tirelessly gathering data, and then the websites that 17 00:00:49.719 --> 00:00:53.079 are actively trying to control throttle or just outright block them. 18 00:00:53.119 --> 00:00:54.600 So welcome to the deep dive. 19 00:00:54.759 --> 00:00:57.719 Thanks for having me. It's a fascinating battlefield, really, and 20 00:00:57.759 --> 00:01:00.600 the stakes are incredibly high. We're talking about the core 21 00:01:00.679 --> 00:01:03.640 mechanisms that power the modern Internet economy. 22 00:01:03.840 --> 00:01:06.280 Yeah, and for anyone listening who is stepping into the 23 00:01:06.280 --> 00:01:12.760 world of programming, cybersecurity, or AI mastering. This webscraping process 24 00:01:12.879 --> 00:01:14.680 is like an absolute superpower. 25 00:01:14.760 --> 00:01:17.400 Oh for sure, you have the ability to gather massive 26 00:01:17.439 --> 00:01:20.000 data sets automatically, but it. 27 00:01:19.959 --> 00:01:22.599 Comes with a lot of hurdles. So our mission for 28 00:01:22.680 --> 00:01:25.400 this deep dive is to map out these common challenges 29 00:01:25.439 --> 00:01:28.560 of extracting web data. We're going to uncover the technical 30 00:01:28.560 --> 00:01:31.920 mechanics to make your scraping scripts run at lightning. 31 00:01:31.640 --> 00:01:32.680 Speed, which is crucial. 32 00:01:32.840 --> 00:01:35.760 Yeah, and we'll also outline the unwritten rules of being 33 00:01:35.799 --> 00:01:39.280 a good Internet citizen why you do it? Because you 34 00:01:39.319 --> 00:01:41.280 can't just go wild, No, you really can. 35 00:01:41.480 --> 00:01:44.799 That context is critical because webscraping is, as you said, 36 00:01:44.799 --> 00:01:47.439 incredibly powerful. I mean, you can pull vast amounts of 37 00:01:47.439 --> 00:01:51.319 information to feed machine learning algorithms, or track real time 38 00:01:51.319 --> 00:01:56.319 competitor pricing, or just aggregate research. But from the website's perspective, 39 00:01:56.319 --> 00:01:59.840 there are very real, very technical reasons they deploy countermeasures 40 00:01:59.840 --> 00:02:01.400 to keep these automated scripts out. 41 00:02:01.640 --> 00:02:05.040 It's not just companies being stingy with their public data, right, 42 00:02:05.319 --> 00:02:08.000 I imagine A big part of it is server preservation. 43 00:02:08.400 --> 00:02:11.759 That is a massive factor. A well programmed bot can 44 00:02:11.800 --> 00:02:14.319 send thousands of requests per second. 45 00:02:14.479 --> 00:02:14.919 Wow. 46 00:02:15.080 --> 00:02:18.639 Yeah. If a site doesn't throttle that, their server resources 47 00:02:18.680 --> 00:02:22.120 are completely drained, and that adversely affects the site's performance 48 00:02:22.159 --> 00:02:25.080 for real human users who are just trying to load 49 00:02:25.120 --> 00:02:25.560 the page. 50 00:02:25.719 --> 00:02:26.159 Makes sense. 51 00:02:26.439 --> 00:02:29.800 Beyond that, there's the intellectual property angle. We've actually seen 52 00:02:30.000 --> 00:02:33.879 massive legal battles over this, like the famous case between 53 00:02:33.879 --> 00:02:36.960 LinkedIn and a data analytics company called high q Labs. 54 00:02:37.120 --> 00:02:38.639 Oh I remember hearing about that. Yeah. 55 00:02:38.680 --> 00:02:41.439 It's centered entirely on whether a company is allowed to 56 00:02:41.520 --> 00:02:45.680 scrape public profiles or if the host platform has the 57 00:02:45.759 --> 00:02:49.159 legal right to build technical walls to protect that user data, 58 00:02:49.479 --> 00:02:50.240 which brings. 59 00:02:50.080 --> 00:02:52.840 Us perfectly to the first line of defense, because before 60 00:02:52.879 --> 00:02:54.879 your Python script can even look at the data on 61 00:02:54.879 --> 00:02:56.719 a web page, it has to get through the front 62 00:02:56.719 --> 00:02:57.759 door exactly. 63 00:02:57.919 --> 00:03:00.360 Websites are not just sitting ducks out there put up 64 00:03:00.360 --> 00:03:01.800 deliberate barriers. 65 00:03:01.400 --> 00:03:03.039 Right, So what's the first barrier? 66 00:03:03.400 --> 00:03:06.319 Well, the very first checkpoint you encounter when mapping out 67 00:03:06.319 --> 00:03:09.800 a scraping project is a file called robots dot txt. 68 00:03:10.120 --> 00:03:13.319 I always think of robots dot txt like checking a 69 00:03:13.400 --> 00:03:15.639 restaurant's stress code before we try to walk in the 70 00:03:15.639 --> 00:03:17.800 front door. Yeah, yeah, you have to know if you're 71 00:03:17.800 --> 00:03:20.360 even allowed inside and what you're allowed to wear. 72 00:03:20.680 --> 00:03:23.719 That's a great analogy. The mechanism behind it is surprisingly 73 00:03:23.800 --> 00:03:27.439 simple too. It's literally just a plain text file hosted 74 00:03:27.479 --> 00:03:29.319 at the root directory of a website. 75 00:03:29.400 --> 00:03:32.240 So just typing slash robots dot txt at the end 76 00:03:32.280 --> 00:03:33.240 of the UURL yep. 77 00:03:33.599 --> 00:03:35.719 If you type that into your browser, you will see 78 00:03:35.719 --> 00:03:39.560 a list of instructions created by the webmaster. It explicitly 79 00:03:39.639 --> 00:03:44.439 specifies which user agents, meaning which specific bots or browsers 80 00:03:44.479 --> 00:03:47.280 are allowed Oh I see, and which are disallowed. It 81 00:03:47.319 --> 00:03:50.599 also gives the exact directory paths those rules apply to. 82 00:03:51.479 --> 00:03:53.960 So checking this file along with the site's terms of 83 00:03:54.000 --> 00:03:57.439 service is you know your mandatory first step to understand 84 00:03:57.479 --> 00:03:59.000 what you are technically permitted to do. 85 00:03:59.360 --> 00:04:02.439 Okay, out of scenario the dress code says my bot 86 00:04:02.520 --> 00:04:04.719 is allowed in. That doesn't mean I have free reign 87 00:04:04.840 --> 00:04:07.919 to just start grabbing data, right, because the server might 88 00:04:08.000 --> 00:04:10.280 suddenly throw ATCCHA on my screen. 89 00:04:10.439 --> 00:04:15.879 Ah. Yes, the classic cap TTCCHA completely automated public turing 90 00:04:15.879 --> 00:04:17.600 test to tell computers and humans. 91 00:04:17.360 --> 00:04:21.319 Apart right, which is designed specifically to stop basic automated 92 00:04:21.360 --> 00:04:24.920 scripts in their tracks. But I kind of want to 93 00:04:24.959 --> 00:04:28.480 challenge the effectiveness of this. Haven't bots gotten smart enough 94 00:04:28.480 --> 00:04:29.480 to bypass these? Now? 95 00:04:29.680 --> 00:04:30.160 What do you mean? 96 00:04:30.439 --> 00:04:33.480 Well, I mean we have AI models that can pass 97 00:04:33.560 --> 00:04:36.959 the bar exam. Surely a Python script can be trained 98 00:04:37.000 --> 00:04:39.120 to click on the squares that contain a crosswalk or 99 00:04:39.160 --> 00:04:39.839 a traffic light. 100 00:04:39.959 --> 00:04:43.759 Yeah, advancements have absolutely been made, and the technology to 101 00:04:43.800 --> 00:04:48.439 solve kiptchas using computer vision can be integrated directly into 102 00:04:48.480 --> 00:04:49.560 your scraping scripts. 103 00:04:49.600 --> 00:04:51.639 Now, right, So kpcchas are dead. 104 00:04:51.920 --> 00:04:55.279 Not quite, because the modern KPCCHA doesn't just evaluate whether 105 00:04:55.319 --> 00:04:58.079 you click the right image. It evaluates the how. 106 00:04:58.120 --> 00:05:00.920 Way, meaning it's watching my mouse move precisely. 107 00:05:01.000 --> 00:05:03.199 If you are writing a script that tells the cursor 108 00:05:03.240 --> 00:05:06.079 to move to coordinates X and y, the computer moves 109 00:05:06.079 --> 00:05:08.920 that cursor in a perfectly straight line at a constant velocity. 110 00:05:09.199 --> 00:05:10.720 Because it's a machine. 111 00:05:10.360 --> 00:05:15.959 Exactly a human hand. Jittering on a physical mouse creates microfluctuations, 112 00:05:16.319 --> 00:05:21.319 and the KPTCHA algorithm analyzes that trajectory so overcoming it 113 00:05:21.360 --> 00:05:26.079 requires programming complex randomize Bizieri curs into your mouse automation 114 00:05:26.560 --> 00:05:28.800 just to simulate human imperfection. 115 00:05:29.040 --> 00:05:32.560 That sounds incredibly resource intensive to program and execute. It 116 00:05:32.600 --> 00:05:34.920 is which I'm guessing is the website's ultimate goal. Like, 117 00:05:34.959 --> 00:05:37.000 even if my bot can solve it, the site still 118 00:05:37.000 --> 00:05:37.720 wins Dingo. 119 00:05:38.319 --> 00:05:42.839 The GKPCHA succeeds in its primary secondary goal severely slowing 120 00:05:42.879 --> 00:05:46.839 down the extraction process. Differentiating between a computer and a 121 00:05:46.920 --> 00:05:49.680 human takes processing power, and sometimes you even have to 122 00:05:49.720 --> 00:05:53.000 pay for external solver services. When your script needs to 123 00:05:53.000 --> 00:05:56.839 scrape one hundred thousand product pages, hitting a KPCHA that 124 00:05:56.879 --> 00:06:00.720 takes even five seconds to bypass completely tanks your efficiency. 125 00:06:01.120 --> 00:06:03.680 The entire operation becomes sluggish. 126 00:06:03.319 --> 00:06:05.560 So the website wins just by making you late to 127 00:06:05.600 --> 00:06:08.720 the party. Okay, let's say your bot is sophisticated. You 128 00:06:08.800 --> 00:06:12.639 check robots dot txt, you simulated the human mouse movements 129 00:06:12.680 --> 00:06:15.519 to pass the kptcchas, and you are finally through the 130 00:06:15.519 --> 00:06:18.839 front door pulling data. You're in, but the environment inside 131 00:06:18.879 --> 00:06:22.439 still isn't stable. Websites actively hunt for bots, where sometimes 132 00:06:22.480 --> 00:06:23.759 they just accidentally break them. 133 00:06:23.959 --> 00:06:27.480 Yeah, through structural changes. This brings us to the shifting 134 00:06:27.519 --> 00:06:30.439 foundations of the web. To keep up with advancements in 135 00:06:30.600 --> 00:06:35.720 UI and UX user interface and user experience, websites undergo regular. 136 00:06:35.399 --> 00:06:37.480 Updates, and that misses with the bots. 137 00:06:37.319 --> 00:06:41.040 It's the most common non malicious hurdle a developer faces. 138 00:06:41.439 --> 00:06:44.399 Because they redesigned the layout right, they change the underlying 139 00:06:44.519 --> 00:06:46.000 HTML tags exactly. 140 00:06:46.759 --> 00:06:51.480 The mechanism here relies on the dom the document object model. 141 00:06:52.240 --> 00:06:56.399 Webscrapers are specifically written to target exact CSS selectors or 142 00:06:56.519 --> 00:06:59.160 HTML tags present at the exact moment you. 143 00:06:59.079 --> 00:07:00.600 Write the code. 144 00:07:00.680 --> 00:07:02.959 If you instruct your script to find the price inside 145 00:07:02.959 --> 00:07:06.240 a div class equals product price, and the web developer 146 00:07:06.319 --> 00:07:09.360 updates the site overnight to use a span ed equals cost, 147 00:07:09.759 --> 00:07:12.560 your scraper suddenly looks at the code and sees nothing. 148 00:07:12.800 --> 00:07:15.040 It's like going to the grocery store with a very 149 00:07:15.079 --> 00:07:18.279 specific hard coded map. Your instructions say the milk is 150 00:07:18.279 --> 00:07:21.560 an aisle four, shelf two, but overnight the store manager 151 00:07:21.600 --> 00:07:25.120 decided to redesign the layout and moved all the dairy 152 00:07:25.240 --> 00:07:29.079 to aisle nine. If you're a robot blindly following the 153 00:07:29.079 --> 00:07:31.279 old map, you're going to come back with a bottle 154 00:07:31.319 --> 00:07:34.279 of shampoo instead of milk hmm, or you know, you 155 00:07:34.360 --> 00:07:35.639 just come back empty handed. 156 00:07:35.519 --> 00:07:39.720 Which results in total data loss. This forces developers into 157 00:07:39.759 --> 00:07:43.879 a cycle of constant maintenance. For large scale operations, teams 158 00:07:43.920 --> 00:07:46.959 often have to write robust error handling into their scripts 159 00:07:47.439 --> 00:07:50.000 just to alert them when a structural change occurs. That 160 00:07:50.120 --> 00:07:53.680 sounds exhausting, it is, or they just outsource the scraping 161 00:07:53.759 --> 00:07:57.240 architecture entirely to dedicated service providers who monitor those damn 162 00:07:57.319 --> 00:07:58.279 changes in real time. 163 00:07:58.360 --> 00:08:01.759 Okay, so that's an accidental break. Let's escalate to deliberate 164 00:08:01.800 --> 00:08:05.319 targeting like IP blocking. This is when the site server 165 00:08:05.439 --> 00:08:07.000 just says, I know who you are and you're not 166 00:08:07.040 --> 00:08:09.720 getting in right, But how does the server actually identify 167 00:08:09.759 --> 00:08:10.120 the bot? 168 00:08:10.199 --> 00:08:13.120 Well, every time your script requests a webpage, it initiates 169 00:08:13.160 --> 00:08:17.319 a connection via the TCP IP protocol. That request inherently 170 00:08:17.360 --> 00:08:19.720 carries your IP address. Think of it as the return 171 00:08:19.759 --> 00:08:21.000 address on a piece of mail. 172 00:08:21.120 --> 00:08:21.680 Okay, sure. 173 00:08:21.920 --> 00:08:25.319 IP blocking typically triggers when the server's firewall detects an 174 00:08:25.399 --> 00:08:29.680 unnaturally high volume of requests originating from the exact same 175 00:08:29.720 --> 00:08:32.720 return address in a really short amount of time. Ah, gotcha, 176 00:08:33.279 --> 00:08:36.919 Or if your crawler makes multiple parallel requests asking for 177 00:08:37.000 --> 00:08:41.000 twenty different pages at the exact same millisecond, because. 178 00:08:40.639 --> 00:08:45.240 No human clicks twenty lengths simultaneously exactly. But wait, if 179 00:08:45.240 --> 00:08:47.679 the firewall is just looking at the return address on 180 00:08:47.720 --> 00:08:50.279 the envelope. Couldn't my script just use a bunch of 181 00:08:50.320 --> 00:08:55.360 fake return addresses, like routing my requests through a massive 182 00:08:55.399 --> 00:08:56.759 network of different computers. 183 00:08:56.960 --> 00:08:59.679 You've just described rotating proxy pools. 184 00:08:59.799 --> 00:09:00.519 That's that's what that is. 185 00:09:00.639 --> 00:09:03.759 Yeah, that is the exact countermeasure developers use. They can 186 00:09:03.759 --> 00:09:06.559 figure their scripts to route each HTTP requests through a 187 00:09:06.559 --> 00:09:10.759 different proxy server, essentially masking the scraper's true IP and 188 00:09:10.759 --> 00:09:13.639 making the traffic look like it's coming from thousands of 189 00:09:13.639 --> 00:09:15.519 different human users across the globe. 190 00:09:15.519 --> 00:09:16.519 Problem solve then. 191 00:09:16.639 --> 00:09:20.559 Well not quite even then. Some server firewalls use aggressive 192 00:09:20.600 --> 00:09:23.799 behavioral analysis that can flag a quote unquote good bot 193 00:09:24.399 --> 00:09:26.120 just for acting too systematically. 194 00:09:26.440 --> 00:09:28.679 Wow, they really don't want you in there. And if 195 00:09:28.679 --> 00:09:31.399 they aren't just aggressively blocking your IP, they might be 196 00:09:31.440 --> 00:09:35.200 actively trying to trick you, which brings us to honeypot traps. 197 00:09:35.399 --> 00:09:41.159 Honeypots are fascinating. They are specific, deliberate traps meant solely 198 00:09:41.200 --> 00:09:43.200 to detect automated crawlers. 199 00:09:43.559 --> 00:09:46.240 I picture this like those invisible laser trip wires in 200 00:09:46.279 --> 00:09:47.720 a museum heist movie. 201 00:09:47.840 --> 00:09:49.159 Yes, perfect analogy. 202 00:09:49.360 --> 00:09:52.600 A human user is just browsing the site. They walk 203 00:09:52.679 --> 00:09:54.919 right past the lasers because they can't even see them. 204 00:09:55.200 --> 00:09:58.600 But a scraper is blindly sweeping the room, touching every 205 00:09:58.679 --> 00:10:01.360 piece of code it can fined, and bam, it triggers 206 00:10:01.399 --> 00:10:01.919 the alarm. 207 00:10:02.159 --> 00:10:06.039 The implementation of that trap is incredibly clever. Developers embed 208 00:10:06.159 --> 00:10:10.240 hidden links into the site's HTML. They use CSS attributes 209 00:10:10.279 --> 00:10:12.759 like display cole in Nune, or they make the link 210 00:10:12.799 --> 00:10:15.919 the exact same hex color as the background maky very. 211 00:10:16.399 --> 00:10:18.799 To a human user looking at the rendered page on 212 00:10:18.840 --> 00:10:21.759 a monitor, that link literally does not exist. It is 213 00:10:21.799 --> 00:10:24.399 invisible and a human physically cannot click it. 214 00:10:24.600 --> 00:10:27.279 But a bot isn't looking at a monitor. It's parsing 215 00:10:27.279 --> 00:10:30.240 the raw text of the HTML document exactly. 216 00:10:30.480 --> 00:10:33.120 The crawler's logic is usually programmed to find every a 217 00:10:33.320 --> 00:10:36.399 raft tag, every link and follow it to gather more data. 218 00:10:36.799 --> 00:10:39.399 When it reads the raw code, the honeypot just looks 219 00:10:39.440 --> 00:10:42.879 like another standard link, and then the second your crawler 220 00:10:42.960 --> 00:10:47.240 sends an httpay request to that specific hidden URL, the 221 00:10:47.360 --> 00:10:51.159 trap springs. The server instantly knows with one hundred percent 222 00:10:51.200 --> 00:10:54.120 certainty that you are a bot, and your proxy's IP 223 00:10:54.279 --> 00:10:56.240 address gets permanently blacklisted. 224 00:10:56.919 --> 00:11:00.440 That is brilliant. So we have these deliberate milicitiaus traps. 225 00:11:00.480 --> 00:11:03.399 And we have structural layout changes. But sometimes in modern 226 00:11:03.399 --> 00:11:08.279 web design itself is just accidentally a scraper's worst enemy. Definitely, 227 00:11:08.320 --> 00:11:11.799 because websites today aren't static pages anymore. They are heavily 228 00:11:11.840 --> 00:11:13.799 interactive applications. 229 00:11:13.279 --> 00:11:17.519 Which brings us to dynamic content. New websites utilize frameworks 230 00:11:17.559 --> 00:11:21.240 like React or Angular, which rely heavily on ajax calls 231 00:11:21.279 --> 00:11:25.080 to load data dynamically. A prime example is infinite scrolling. 232 00:11:25.159 --> 00:11:27.600 Oh infinite scrolling. You're on a social media feed, you 233 00:11:27.639 --> 00:11:29.799 reach the bottom of the page, and instead of having 234 00:11:29.799 --> 00:11:32.480 to click a next page button, more content just magically 235 00:11:32.559 --> 00:11:33.879 populates as you scroll down. 236 00:11:34.279 --> 00:11:38.240 Right. It creates a seamless human experience. But traditional web 237 00:11:38.240 --> 00:11:43.279 crawling libraries, like Python's requests module don't actually quote unquote 238 00:11:43.360 --> 00:11:46.879 brows more they do. They send a single request, download 239 00:11:46.879 --> 00:11:50.679 the initial static HTML response from the server, and disconnect. 240 00:11:51.080 --> 00:11:53.879 If the product, prices, or comments you need are waiting 241 00:11:53.919 --> 00:11:57.559 to be dynamically generated by JavaScript only after a user 242 00:11:57.600 --> 00:12:02.200 scrolls down, the traditional crawler misses that data entirely because 243 00:12:02.200 --> 00:12:03.960 it never executed the JavaScript. 244 00:12:05.360 --> 00:12:08.360 If the script can't scroll and it can't run JavaScript, 245 00:12:08.399 --> 00:12:10.360 I imagine the only way to fix that is to 246 00:12:10.399 --> 00:12:13.240 program the bot to actually open a real web browser 247 00:12:13.440 --> 00:12:15.720 and simulate the scrolling action itself exactly. 248 00:12:15.919 --> 00:12:19.759 This is where automation frameworks like Selenium or Playwright become essential. 249 00:12:20.120 --> 00:12:22.480 They allow your code to literally take control of a 250 00:12:22.519 --> 00:12:23.039 web browser. 251 00:12:23.080 --> 00:12:24.679 It's basically clicking and scrolling for you. 252 00:12:24.840 --> 00:12:27.759 Yep, you can write commands instructing the browser to scroll 253 00:12:27.799 --> 00:12:30.360 down five hundred pixels, wait for the dynamic elements to 254 00:12:30.399 --> 00:12:32.840 populate in the dom and then extract the data. 255 00:12:32.879 --> 00:12:36.679 But opening a full version of Chrome for every single 256 00:12:36.720 --> 00:12:40.639 page you want to scrape sounds incredibly slow. I mean, 257 00:12:40.879 --> 00:12:44.600 your computer's processor would be begging for mercy rendering all 258 00:12:44.639 --> 00:12:45.840 those visuals. 259 00:12:45.440 --> 00:12:48.360 Which is why developers utilize headless browsers. 260 00:12:48.639 --> 00:12:51.320 Headless browsers, let's break that down. What exactly makes a 261 00:12:51.320 --> 00:12:52.240 browser headless? 262 00:12:52.279 --> 00:12:54.200 Think of it like a ghost driving a car. 263 00:12:54.559 --> 00:12:55.759 A ghost driving a car. 264 00:12:55.960 --> 00:12:59.720 Yeah, all the underlying mechanics of the vehicle are operating 265 00:12:59.720 --> 00:13:03.480 per per The engine is running, the steering wheel is turning, 266 00:13:03.559 --> 00:13:07.600 the JavaScript is executing, and the network requests are firing. Okay, 267 00:13:08.000 --> 00:13:10.360 but there is no driver looking out the window, and 268 00:13:10.399 --> 00:13:14.200 more importantly, the car has no exterior shell. A headless 269 00:13:14.240 --> 00:13:17.679 browser performs all the complex operations of Chrome or Firefox 270 00:13:17.919 --> 00:13:20.639 entirely in the system's background memory without ever rendering the 271 00:13:20.720 --> 00:13:22.519 visual pixels on a graphical screen. 272 00:13:22.639 --> 00:13:23.679 Wow, that's wild. 273 00:13:23.759 --> 00:13:27.360 It strips away the heavy visual interface, making it incredibly fast. 274 00:13:27.799 --> 00:13:32.039 Is a phenomenal tool. But while we're using ghost cars 275 00:13:32.080 --> 00:13:36.399 to scroll through dynamic content, there is another massive issue. 276 00:13:36.399 --> 00:13:40.120 We have to watch out for data quality. Because scraping 277 00:13:40.159 --> 00:13:43.159 isn't just about successfully getting data, it is about getting good, 278 00:13:43.399 --> 00:13:44.360 reliable data. 279 00:13:44.440 --> 00:13:48.000 Oh. Absolutely, Faulty data is disastrous, especially if you are 280 00:13:48.000 --> 00:13:51.799 feeding your scraped outputs into advanced systems like machine learning models. 281 00:13:52.399 --> 00:13:55.440 If your scraper is running on autopilot extracting tens of 282 00:13:55.480 --> 00:13:59.080 thousands of rosen hour and a website's formatting changes slightly, 283 00:13:59.519 --> 00:14:02.399 you might start pulling corrupted values without even realizing it. 284 00:14:02.639 --> 00:14:05.840 Because if the mL training data is garbage, the AI 285 00:14:05.919 --> 00:14:09.159 output will be garbage. You can't manually read ten thousand 286 00:14:09.159 --> 00:14:12.360 spreadsheet rows after the fact to check for errors. So 287 00:14:12.399 --> 00:14:15.000 how do we prevent bad data from poising the database 288 00:14:15.039 --> 00:14:15.759 in the first place. 289 00:14:16.039 --> 00:14:19.120 The mechanism for this is writing real time assertions directly 290 00:14:19.120 --> 00:14:21.759 into your scraping script. A certain yeah, you build test 291 00:14:21.759 --> 00:14:24.799 cases that evaluate the data type the exact moment it 292 00:14:24.840 --> 00:14:28.120 is extracted from the page. For instance, if your script 293 00:14:28.159 --> 00:14:32.080 is targeting a price field, you write an assertion verifying 294 00:14:32.080 --> 00:14:34.679 that the extracted value is a floating point number. 295 00:14:34.759 --> 00:14:36.240 Okay, so a number with a decimal. 296 00:14:36.440 --> 00:14:38.720 Right now, if the item goes out of stock and 297 00:14:38.759 --> 00:14:41.519 the website replaces the price tag with the text string 298 00:14:41.559 --> 00:14:45.000 out of stock, your scraper grabs a straying instead of 299 00:14:45.000 --> 00:14:45.480 a number. 300 00:14:45.559 --> 00:14:49.440 And because you wrote that assertion, the script instantly flags 301 00:14:49.480 --> 00:14:52.159 the error and drops the bad data before it ever 302 00:14:52.200 --> 00:14:53.399 gets saved to your database. 303 00:14:53.519 --> 00:14:56.840 Exactly. It acts as an automated quality assurance gatekeeper. 304 00:14:56.919 --> 00:15:00.600 That's super smart. Let's take stock of our scripts workload. 305 00:15:00.679 --> 00:15:06.399 Right now, we are parsing robots dot txt, simulating mouse curves, 306 00:15:06.919 --> 00:15:12.399 rotating proxyips, dodging invisible honeypots, driving headless browsers to scroll 307 00:15:12.480 --> 00:15:16.080 dynamic pages, and running real time type assertions a lot. 308 00:15:16.159 --> 00:15:18.879 That is a massive computational load. If we want this 309 00:15:18.919 --> 00:15:21.360 scraper to be viable at scale, we have to claw 310 00:15:21.440 --> 00:15:23.639 back some efficiency and speed wherever we can. 311 00:15:23.919 --> 00:15:27.840 Optimization is key here, and the first and often most 312 00:15:27.879 --> 00:15:31.080 impactful line of code you can add to increase efficiency 313 00:15:31.639 --> 00:15:33.559 is disabling image loading. 314 00:15:33.519 --> 00:15:37.039 Oh, sure, because modern websites are incredibly heavy with high 315 00:15:37.080 --> 00:15:38.320 resolution media files. 316 00:15:38.480 --> 00:15:42.200 Exactly, when a headless browser navigates to a URL, it 317 00:15:42.320 --> 00:15:45.679 naturally requests all the elements required to build that page, 318 00:15:45.919 --> 00:15:48.039 including massive image and video files. 319 00:15:48.120 --> 00:15:48.320 Right. 320 00:15:48.720 --> 00:15:52.240 But if your Python script is only programmed to scrape text, 321 00:15:52.519 --> 00:15:55.440 say a table of financial data or a news article, 322 00:15:55.519 --> 00:15:59.080 it doesn't need the images. Requesting them consumes bandwidth and 323 00:15:59.200 --> 00:16:03.159 drastically down your page load times. This becomes glaringly apparent 324 00:16:03.240 --> 00:16:05.279 when your loop runs ten thousand times. 325 00:16:05.440 --> 00:16:08.080 It's like downloading a massive four K movie file just 326 00:16:08.120 --> 00:16:10.360 to read the text in the subtitle track. It's a 327 00:16:10.480 --> 00:16:11.200 huge waste of. 328 00:16:11.120 --> 00:16:14.480 Bandwidth, perfect way to put it. The implementation is straightforward, though. 329 00:16:14.559 --> 00:16:17.360 You just pass specific argument flags to your webdriver, like 330 00:16:17.440 --> 00:16:20.879 cooptions instructing the browser engine to block all requests for 331 00:16:20.919 --> 00:16:21.840 image file types. 332 00:16:22.039 --> 00:16:24.200 What else can we configure in those options? To speed 333 00:16:24.240 --> 00:16:24.639 things up? 334 00:16:24.960 --> 00:16:28.840 You should configure the driver to utilize disc caching. This 335 00:16:28.919 --> 00:16:32.440 reduces subsequent page loading times by a massive margin. 336 00:16:32.600 --> 00:16:34.399 How does the cash actually save time? 337 00:16:35.159 --> 00:16:38.120 When you visit a website, there are foundational assets required 338 00:16:38.159 --> 00:16:42.000 to build the page, the CSS files that dictate the styling, 339 00:16:42.440 --> 00:16:44.840 the JavaScript libraries, the custom. 340 00:16:44.519 --> 00:16:46.279 Fonts, right the building blocks. 341 00:16:46.360 --> 00:16:49.879 Caching saves a blueprint of those foundational files directly in 342 00:16:49.960 --> 00:16:51.519 your local disk storage. 343 00:16:51.600 --> 00:16:53.919 So instead of asking the website server to send you 344 00:16:53.960 --> 00:16:56.799 the exact same two megabyte font file every single time 345 00:16:56.879 --> 00:16:59.720 you click a new product link, your browser just loads 346 00:16:59.720 --> 00:17:01.360 the locally from your. 347 00:17:01.240 --> 00:17:05.640 Hard drive exactly. It eliminates redundant network requests. When your 348 00:17:05.680 --> 00:17:08.160 script needs to navigate through hundreds of pages on the 349 00:17:08.200 --> 00:17:11.559 same domain, cashing ensures you are only downloading the unique 350 00:17:11.680 --> 00:17:14.920 HTML data for each new page, rather than re downloading 351 00:17:14.920 --> 00:17:16.839 the entire site architecture every time. 352 00:17:17.000 --> 00:17:21.119 It's amazing, we've stripped the images, we've cached the site assets. 353 00:17:21.559 --> 00:17:24.599 But there is a crucial optimization tip regarding how we 354 00:17:24.640 --> 00:17:26.799 actually shut the scraper down at the end of the run. 355 00:17:27.440 --> 00:17:29.839 It comes down to the difference between two specific commands. Right, 356 00:17:30.400 --> 00:17:33.240 driver dot close and driver dot quit. Oh. 357 00:17:33.480 --> 00:17:38.079 Yes, this distinction is vital from memory resource management. When 358 00:17:38.079 --> 00:17:42.519 your Python automation script initializes a Selenium webdriver, it spins 359 00:17:42.640 --> 00:17:46.559 up additional background processes on your operating system. Okay, after 360 00:17:46.680 --> 00:17:49.559 your script finishes its extraction loop, the operating system does 361 00:17:49.599 --> 00:17:54.039 not automatically kill those background processes unless you explicitly command 362 00:17:54.079 --> 00:17:55.200 it to wait. 363 00:17:55.039 --> 00:17:57.079 So the script might be done running, but the headless 364 00:17:57.079 --> 00:17:59.880 browser is basically still sitting there silently eating up your 365 00:18:00.079 --> 00:18:01.119 systems ram yep. 366 00:18:01.240 --> 00:18:04.400 A common pitfall is developers using the command driver dot close, 367 00:18:04.920 --> 00:18:08.559 but driver dot close only closes the current active window 368 00:18:08.720 --> 00:18:10.200 or tab within the browser. 369 00:18:10.359 --> 00:18:12.480 It's like clicking the little X on a single tab. 370 00:18:13.200 --> 00:18:15.799 The specific web page goes away, but the browser application 371 00:18:15.839 --> 00:18:18.640 itself is still open, running in the background, consuming memory. 372 00:18:18.880 --> 00:18:22.400 But using driver dot quit is like forcing the entire 373 00:18:22.480 --> 00:18:23.599 application to terminate. 374 00:18:24.200 --> 00:18:28.000 Right, driver dot quit safely closes all active tabs, disconnects 375 00:18:28.000 --> 00:18:31.720 the communication port, and fully terminates the web driver executable, 376 00:18:32.039 --> 00:18:35.960 releasing those memory resources back to your operating system. Because 377 00:18:36.000 --> 00:18:38.519 if you have a scraper scheduled to run every hour 378 00:18:38.799 --> 00:18:41.799 and you use driver dot close, you will spawn a 379 00:18:41.839 --> 00:18:45.960 new zombie browser process every hour. Eventually your system will 380 00:18:45.960 --> 00:18:47.799 completely crash from memory. 381 00:18:47.519 --> 00:18:52.559 Exhaustion zombie browsers. That sounds terrifying. Okay, we now have 382 00:18:52.640 --> 00:18:57.039 a blazing fast optimized scraper. It doesn't leak memory, it 383 00:18:57.079 --> 00:19:00.839 ignores heavy images, and it seamlessly dotes. Honey, we are 384 00:19:00.880 --> 00:19:05.079 officially a webscraping powerhouse we are, But just because you 385 00:19:05.119 --> 00:19:06.960 can scrape aggressively, it doesn't mean you should. 386 00:19:07.039 --> 00:19:10.079 Absolutely not. The technical capability must be paired with the 387 00:19:10.160 --> 00:19:12.960 unwritten rules of the road. The field relies heavily on 388 00:19:13.000 --> 00:19:16.720 a set of widely accepted best practices for bought etiquette. 389 00:19:16.160 --> 00:19:19.200 The golden rule being do not crash the target server. 390 00:19:19.680 --> 00:19:23.160 Yes, we touched on server load earlier, but it cannot 391 00:19:23.160 --> 00:19:28.319 be overstated. Sending multiple asynchronous requests too frequently mimics a 392 00:19:28.319 --> 00:19:31.279 distributed denial of service or DIDO. 393 00:19:31.240 --> 00:19:32.920 Attack, which is illegal. 394 00:19:33.119 --> 00:19:36.000 Yes, and the server gets overwhelmed trying to answer your 395 00:19:36.000 --> 00:19:40.720 bot and drops connections. For legitimate human users, you must 396 00:19:40.720 --> 00:19:44.480 program reasonable time delays into your script's execution loop. 397 00:19:44.759 --> 00:19:47.839 And speaking of those time delays, there is a brilliant 398 00:19:47.839 --> 00:19:52.880 strategy involving geolocation to ensure you aren't fighting human traffic. 399 00:19:53.039 --> 00:19:55.839 Oh, I love this one. To minimize your footprint, you 400 00:19:55.839 --> 00:19:58.839 should schedule your crawling tasks to run during the target 401 00:19:58.880 --> 00:19:59.960