WEBVTT 1 00:00:01.280 --> 00:00:04.320 Consider the sheer architecture of the modern web for a second. 2 00:00:04.919 --> 00:00:08.560 We are really dealing with simple, static HTML documents. 3 00:00:08.240 --> 00:00:11.400 Anymore, Absolutely not. Those days are long on, right. 4 00:00:11.679 --> 00:00:17.160 We are dealing with massive, sprawling, dynamically hydrated applications. You 5 00:00:17.239 --> 00:00:20.000 inspect a page looking for a specific data set, say 6 00:00:20.399 --> 00:00:22.519 I don't know real estate prices, and you find out 7 00:00:22.519 --> 00:00:23.760 it's not even in the source code. 8 00:00:23.839 --> 00:00:25.879 Yeah, it's completely hidden exactly. 9 00:00:26.160 --> 00:00:30.719 It's loading via some undocumented back end API which is 10 00:00:30.719 --> 00:00:34.119 buried behind a category link which requires a session token, 11 00:00:34.119 --> 00:00:37.640 which leads to a subcategory, and then that triggers an 12 00:00:37.719 --> 00:00:39.560 infinite scroll pagination script. 13 00:00:39.679 --> 00:00:41.439 It's a nightmare, it really is. 14 00:00:41.679 --> 00:00:45.960 Attempting to extract that data with a standard linear Python script, 15 00:00:46.320 --> 00:00:49.520 like using something like the requests library. It's basically like 16 00:00:49.560 --> 00:00:52.320 trying to map a sprawling underground labyrinth with a single 17 00:00:52.359 --> 00:00:53.920 flashlight and a spool of thread. 18 00:00:54.119 --> 00:00:55.159 That's a great way to put it. 19 00:00:55.159 --> 00:00:59.320 The complexity just scales exponentially, The script gets trapped waiting 20 00:00:59.359 --> 00:01:03.240 for servers, and the whole operation in heavily collapses under 21 00:01:03.280 --> 00:01:03.840 its own weight. 22 00:01:04.079 --> 00:01:06.680 Right, because it's a classic scaling problem. I mean, when 23 00:01:06.719 --> 00:01:10.239 you move from scraping a dozen pages to targeting thousands 24 00:01:10.319 --> 00:01:13.920 or hundreds of thousands of heavily nested asynchronous endpoints, a 25 00:01:14.000 --> 00:01:17.040 linear approach just becomes computationally non viable. 26 00:01:17.239 --> 00:01:18.920 You just hit a wall, you do. 27 00:01:19.040 --> 00:01:23.480 You run into memory leaks, IP bands, network latency bottlenecks, 28 00:01:23.480 --> 00:01:26.439 and just you know, absolute spaghetti code trying to manage 29 00:01:26.439 --> 00:01:29.519 the state of a thousand different HDTP requests. 30 00:01:29.120 --> 00:01:31.680 Which brings us to the core of today's deep dive. 31 00:01:32.280 --> 00:01:35.799 We are looking at a highly specialized, industrial grade solution 32 00:01:36.239 --> 00:01:40.680 designed to actually navigate and extract data from these massive 33 00:01:40.680 --> 00:01:43.439 web structures. And it is a powerhouse it is. It's 34 00:01:43.439 --> 00:01:46.120 called Scrapie. Yeah, And our mission today is to move 35 00:01:46.159 --> 00:01:49.879 past the superficial tutorials and understand exactly how Scrapey works 36 00:01:49.959 --> 00:01:54.159 under the hood. We really want to unpack its asynchronous architecture, 37 00:01:54.439 --> 00:01:57.519 understand its underlying engine, and figure out how it provides 38 00:01:57.560 --> 00:02:01.000 developers with such fine grain control of a vast information 39 00:02:01.079 --> 00:02:01.480 on the web. 40 00:02:01.519 --> 00:02:02.799 It's going to be a fun one. 41 00:02:02.879 --> 00:02:06.200 Okay, let's unpack this. Yeah, because to really get how 42 00:02:06.200 --> 00:02:10.520 Scrapey operates, we need to talk about design patterns first. Specifically, 43 00:02:10.879 --> 00:02:13.159 we need to clear up the distinction between a library 44 00:02:13.199 --> 00:02:17.680 and a framework. Yes, because that distinction dictates how this 45 00:02:17.960 --> 00:02:19.240 entire tool is utilized. 46 00:02:19.599 --> 00:02:21.479 It is the crucial starting point. I see this all 47 00:02:21.520 --> 00:02:25.199 the time. A lot of developers approach scrapeye expecting it 48 00:02:25.240 --> 00:02:27.439 to behave like beautiful soup or earlib. 49 00:02:27.599 --> 00:02:29.479 Right. They just want to plug it in exactly. 50 00:02:29.560 --> 00:02:32.719 They try to import it into their existing procedural code 51 00:02:32.759 --> 00:02:36.159 and call it whenever they need it. But screepy isn't 52 00:02:36.199 --> 00:02:40.000 a library you invoke. It is a framework that invokes you. 53 00:02:40.479 --> 00:02:43.719 Okay, that concept right there inversion of control is something 54 00:02:43.759 --> 00:02:46.159 that developers encounter all the time, but I think it's 55 00:02:46.199 --> 00:02:49.159 worth isolating here definitely. When you use a library, you 56 00:02:49.240 --> 00:02:52.199 own the main loop. You write your script, and line 57 00:02:52.199 --> 00:02:55.400 by line you execute your logic. If you need to 58 00:02:55.400 --> 00:02:59.280 make an HGTP request, you call your networking library. If 59 00:02:59.319 --> 00:03:02.400 you need to bars Ason, you call your Jason library. 60 00:03:02.919 --> 00:03:04.680 So I like to think of a library as being 61 00:03:04.680 --> 00:03:05.400 like a power drill. 62 00:03:05.479 --> 00:03:08.039 A power drill, okay, yeah, yeah, you know, you. 63 00:03:08.039 --> 00:03:09.719 Pick it up, you drill a hole when your project 64 00:03:09.759 --> 00:03:12.280 needs it, and then you put it back down. You 65 00:03:12.319 --> 00:03:13.080 are the conductor. 66 00:03:18.400 --> 00:03:21.520 A framework, on the other hand, operates on the Hollywood principle. 67 00:03:21.599 --> 00:03:22.599 The Hollywood principles. 68 00:03:22.680 --> 00:03:24.840 Yeah, you know, don't call us, We'll call you. 69 00:03:25.599 --> 00:03:26.719 Oh I love that. Right. 70 00:03:27.240 --> 00:03:30.120 When you build a Scrapy project, you are not writing 71 00:03:30.159 --> 00:03:34.199 the main executable script. Scrape already wrote the main loop. 72 00:03:34.400 --> 00:03:38.400 Scrapey already built the HTTP client, the air handlers, the 73 00:03:38.439 --> 00:03:40.719 memory managers, and the threading models. 74 00:03:41.039 --> 00:03:44.400 So extending the analogy, if a library is a power drill, 75 00:03:45.240 --> 00:03:48.199 a framework is like the architectural blueprint and foundation of 76 00:03:48.240 --> 00:03:50.919 a house. Precisely it dictates where the walls and the 77 00:03:50.960 --> 00:03:54.319 plumbing go. And your custom code is really just like 78 00:03:54.439 --> 00:03:56.199 picking the paint colors for the rooms. 79 00:03:56.319 --> 00:03:59.120 That is spot on. It's a pre built chassis. It 80 00:03:59.159 --> 00:04:02.800 has specific slots left open for your custom logic. So 81 00:04:02.919 --> 00:04:05.159 you write what's called a spider class and you plug 82 00:04:05.199 --> 00:04:07.439 it into the Scrapey chassis and then you just let 83 00:04:07.520 --> 00:04:11.400 it run. Exactly when you hit run, scrapee takes over. 84 00:04:11.800 --> 00:04:14.199 It spins up its own environment, and it calls your 85 00:04:14.199 --> 00:04:17.040 custom parsing code only when it has a web response 86 00:04:17.160 --> 00:04:18.560 ready for you to actually look at. 87 00:04:18.959 --> 00:04:21.839 And this design isn't just, you know, an arbitrary choice 88 00:04:21.839 --> 00:04:23.800 to make your life difficult, right, I mean sometimes it 89 00:04:23.839 --> 00:04:25.079 feels that way when you're learning it. 90 00:04:25.160 --> 00:04:27.759 Oh, sure, the learning curve is real, but the restriction 91 00:04:28.079 --> 00:04:32.199 is actually the feature. By forcing your code to run 92 00:04:32.240 --> 00:04:36.519 inside its highly controlled environment, Scrape can guarantee that everything 93 00:04:36.560 --> 00:04:40.800 operates asynchronously, because if it lets you control the main loop, 94 00:04:40.879 --> 00:04:44.120 you might accidentally write a blocking command that just halts 95 00:04:44.120 --> 00:04:48.639 the entire crawling operation. So by inverting the control, scrapeye 96 00:04:48.639 --> 00:04:51.120 protects the integrity of its own event loop. 97 00:04:51.439 --> 00:04:53.480 Let's drill into that event loop actually, because I think 98 00:04:53.480 --> 00:04:56.240 this is where the real power lies. Any modern developer 99 00:04:56.319 --> 00:04:59.279 knows the term asynchronous, but Scrape handles this in a 100 00:04:59.360 --> 00:05:03.079 very specific It does it doesn't rely on modern Python asensio. 101 00:05:03.319 --> 00:05:06.279 It is built on top of a much older, incredibly 102 00:05:06.360 --> 00:05:09.079 battle tested networking framework called Twisted. 103 00:05:09.240 --> 00:05:12.959 Twisted is absolutely legendary in the Python ecosystem. It's an 104 00:05:12.959 --> 00:05:17.040 event driven networking engine, and to understand why Scrape uses it, 105 00:05:17.439 --> 00:05:21.000 you really have to look at the fundamental bottleneck of webscraping, 106 00:05:21.319 --> 00:05:22.680 which is network latency. 107 00:05:22.920 --> 00:05:27.040 Right, because it takes basically milliseconds for your CPU to 108 00:05:27.199 --> 00:05:28.040 parse a block of. 109 00:05:28.079 --> 00:05:30.680 HTML barely anytime at all, but it can. 110 00:05:30.639 --> 00:05:34.160 Take hundreds of milliseconds or honestly even full seconds for 111 00:05:34.439 --> 00:05:37.480 a server across the globe to send that HTML back 112 00:05:37.480 --> 00:05:38.240 to you in the first place. 113 00:05:38.319 --> 00:05:38.800 Exactly. 114 00:05:38.839 --> 00:05:41.040 So if your code is synchronous, your CP is just 115 00:05:41.040 --> 00:05:44.199 sitting there idling, waiting for packets to arrive over the wire. 116 00:05:44.399 --> 00:05:47.839 It's just dead time, completely dead time. It is iobound, 117 00:05:48.319 --> 00:05:51.879 and Twisted solves this using what's called the reactor pattern. 118 00:05:52.120 --> 00:05:53.639 Okay, break that down for us. 119 00:05:53.720 --> 00:05:55.920 So instead of waiting for a network request to finish, 120 00:05:56.160 --> 00:05:59.600 Twisted uses a central event loop the reactor. When you 121 00:05:59.639 --> 00:06:02.079 ask Twisted to fetch a web page, it sends the 122 00:06:02.240 --> 00:06:06.360 HTTP request out, registers a callback, and immediately moves on 123 00:06:06.399 --> 00:06:07.959 to the next task. It doesn't block. 124 00:06:08.040 --> 00:06:09.839 I always think of it like a restaurant. Yeah, so, 125 00:06:09.879 --> 00:06:13.439 synchronous code is basically a chef cooking one meal entirely 126 00:06:14.079 --> 00:06:16.879 before starting the next. They put water onto boil and 127 00:06:16.920 --> 00:06:19.160 they just stand there staring at the pot until. 128 00:06:18.920 --> 00:06:21.000 It boils, which would be a terrible way to run a. 129 00:06:21.000 --> 00:06:25.480 Kitchen, right, But asynchronous code, specifically Twisted, is like a 130 00:06:25.519 --> 00:06:27.959 master chef. They put the water onto boil and then 131 00:06:28.000 --> 00:06:31.040 immediately turn around and start chopping vegetables, and then they 132 00:06:31.120 --> 00:06:34.560 sear the meat. They're never standing still. They manage multiple 133 00:06:34.560 --> 00:06:35.279 tasks at once. 134 00:06:35.519 --> 00:06:38.519 That is a perfect analogy. And what's fascinating here is 135 00:06:38.560 --> 00:06:43.959 how this architecture inherently provides extreme fault tolerance. How so, well, 136 00:06:44.000 --> 00:06:46.879 going back to your master chef, if the chef burns 137 00:06:46.879 --> 00:06:50.839 the vegetables, the boiling water doesn't suddenly disappear, right, the 138 00:06:50.959 --> 00:06:54.759 system survives the localized failure. Oh that makes sense because 139 00:06:54.879 --> 00:06:58.639 every single network request is isolated as an asynchronous event. 140 00:06:58.920 --> 00:07:02.560 Failures are highly localized. If server A times out, or 141 00:07:02.639 --> 00:07:06.160 server B returns a five hundred internal error, or the 142 00:07:06.360 --> 00:07:10.000 HTML on server C is totally malformed and causes your 143 00:07:10.040 --> 00:07:12.560 parsing logic to throw an exception. 144 00:07:12.319 --> 00:07:14.600 It doesn't crash the whole program exactly. 145 00:07:14.600 --> 00:07:17.439 It doesn't crash the main thread. The reactor loop just 146 00:07:17.519 --> 00:07:20.879 catches the error logs, it fires an error callback, and 147 00:07:20.920 --> 00:07:23.759 continues processing the thousands of other open connections. 148 00:07:23.920 --> 00:07:25.720 Oh wait, wait, I have a question about that. Sure 149 00:07:25.959 --> 00:07:29.399 Python has the global interpreter lock, the GIL. 150 00:07:29.560 --> 00:07:30.920 Yeah, the infamous GIO. 151 00:07:31.120 --> 00:07:34.160 Right, it is inherently single threaded. So if scrape is 152 00:07:34.240 --> 00:07:37.879 running on Twisted, and Twisted is single threaded, how are 153 00:07:37.920 --> 00:07:42.000 we actually achieving the massive concurrency that Scrapey's famous for. 154 00:07:42.519 --> 00:07:46.480 Doesn't the single thread eventually become a bottleneck if we 155 00:07:46.519 --> 00:07:48.519 are parsing really complex data. 156 00:07:49.079 --> 00:07:51.600 That's a fantastic point, and it's a very common source 157 00:07:51.639 --> 00:07:56.040 of confusion. You're right, The GIL prevents multiple Python threads 158 00:07:56.040 --> 00:07:58.199 from executing pikeon bytecodes at the same time. 159 00:07:58.319 --> 00:07:58.680 Okay. 160 00:07:58.959 --> 00:08:02.319 But twisted leverage, which is IO multiplexing at the operating 161 00:08:02.319 --> 00:08:05.360 system level. It uses system calls like a PALL on 162 00:08:05.439 --> 00:08:08.399 Linux or QQ on macOS, and these allow the operating 163 00:08:08.439 --> 00:08:12.759 system to monitor thousands of open network sockets simultaneously outside 164 00:08:12.759 --> 00:08:13.680 of Python's GIO. 165 00:08:13.959 --> 00:08:17.040 Okay, I see, so it's essentially an automated switchboard operator exactly. 166 00:08:17.040 --> 00:08:19.879 The switchboard operator, which is the reactor loop, plugs a 167 00:08:19.920 --> 00:08:23.480 cable into a server to request a page, but instead 168 00:08:23.480 --> 00:08:25.160 of holding the phone to their ear waiting for the 169 00:08:25.160 --> 00:08:27.959 server to pickup, they instantly plug in one hundred other 170 00:08:28.040 --> 00:08:31.240 cables yep, and the operating system is the one monitoring 171 00:08:31.319 --> 00:08:35.080 all those lines. When a serfer finally responds and sends 172 00:08:35.200 --> 00:08:38.639 data down a specific line, the OS flags it and 173 00:08:38.679 --> 00:08:42.120 the switchboard operator routes that data to the appropriate callback function. 174 00:08:42.320 --> 00:08:43.759 That is exactly how it works. 175 00:08:43.840 --> 00:08:47.080 Okay, So twisted gives us this non blocking highly concurrent, 176 00:08:47.200 --> 00:08:50.559 fault tolerant foundation. But twist it is just a general 177 00:08:50.600 --> 00:08:55.840 networking tool. Scrab is the specific web crawling implementation built 178 00:08:55.879 --> 00:08:58.480 on top of it. Let's look at how scrape actually 179 00:08:58.480 --> 00:09:01.960 structures this work. If Twisted is the raw electricity, what 180 00:09:02.000 --> 00:09:04.120 does the scrapey factory floor actually look like? 181 00:09:04.159 --> 00:09:04.840 The assembly line? 182 00:09:04.879 --> 00:09:07.320 Yeah, the assembly line. I want to trace the actual 183 00:09:07.440 --> 00:09:10.480 journey of a data request through Scrapey's architecture. 184 00:09:10.559 --> 00:09:13.480 It's highly modular, and the best way to understand it 185 00:09:13.559 --> 00:09:16.440 is to follow the life cycle of what scrapeye calls 186 00:09:16.480 --> 00:09:20.919 a request object. So the entire factory floor is overseen 187 00:09:20.960 --> 00:09:24.879 by the core engine. The engine is the centralized nervous system. 188 00:09:25.320 --> 00:09:28.320 It orchestrates the flow of data between all the other components. 189 00:09:28.360 --> 00:09:31.360 Okay, but if the engine touches every single request and 190 00:09:31.440 --> 00:09:36.519 response in a system handling thousands of concurrent connections, how 191 00:09:36.559 --> 00:09:40.000 does the engine itself not become the ultimate bottleneck? 192 00:09:40.159 --> 00:09:43.919 Because the engine isn't actually doing any of the heavy lifting, really, right, 193 00:09:44.039 --> 00:09:46.720 It isn't parsing HTML, and it isn't downloading bytes. It 194 00:09:46.759 --> 00:09:49.440 relies entirely on Twisted's. 195 00:09:48.960 --> 00:09:51.279 Deferred object, which I like promises. 196 00:09:51.039 --> 00:09:54.279 Exactly a deferred is essentially a promise. When the engine 197 00:09:54.360 --> 00:09:57.200 routes a task, it immediately gets a promise back saying 198 00:09:57.559 --> 00:09:59.759 I'll let you know when this is done. The engine 199 00:09:59.799 --> 00:10:03.720 just manages those promises, which computationally is incredibly cheap. 200 00:10:04.039 --> 00:10:07.240 Got it, So let's start the assembly line. Our custom 201 00:10:07.320 --> 00:10:11.120 spider code yields a request for a specific URL. It 202 00:10:11.159 --> 00:10:13.159 hands that to the engine. What does the engine do 203 00:10:13.240 --> 00:10:13.480 with it? 204 00:10:13.559 --> 00:10:16.360 The engine takes that and wrotes that request straight to 205 00:10:16.399 --> 00:10:20.159 the scheduler. The scheduler is a highly optimized priority queue. 206 00:10:20.879 --> 00:10:24.440 Its job is to take all the requested URLs, organize them, 207 00:10:24.679 --> 00:10:30.840 and decide what gets processed next. And crucially, it handles dduplications. 208 00:10:30.080 --> 00:10:33.320 Right the RFPDUP filter. Yeah, because in a massive crawl, 209 00:10:33.799 --> 00:10:36.639 you might encounter a link to the contact does page 210 00:10:36.639 --> 00:10:40.000 on five thousand different product pages. 211 00:10:39.720 --> 00:10:42.000 And you definitely don't want to scrape the same contact 212 00:10:42.039 --> 00:10:43.480 page five thousand times. 213 00:10:43.559 --> 00:10:46.559 No, that would ruin your efficiency. So the scheduler hashes 214 00:10:46.600 --> 00:10:50.000 the request fingerprints and quietly drops the duplicates before they 215 00:10:50.039 --> 00:10:51.639 ever take up network bandwidth. 216 00:10:51.679 --> 00:10:54.440 Exactly, so, when the engine is ready for more work, 217 00:10:54.519 --> 00:10:57.879 it pulls the next request from the scheduler now it 218 00:10:57.919 --> 00:11:00.080 actually needs to fetch the data over the Internet, so 219 00:11:00.159 --> 00:11:02.919 the engine hands the request to the downloader. 220 00:11:03.000 --> 00:11:07.000 Okay, so the downloader is doing the actual HTTP heavy lifting. Yes, 221 00:11:07.080 --> 00:11:09.960 but before the request hits the raw Internet, it passes 222 00:11:10.000 --> 00:11:12.639 through something vital, right, the downloader middleware. 223 00:11:12.799 --> 00:11:16.639 Yes, the middleware middlewaars are hooks injected between the components. 224 00:11:16.799 --> 00:11:19.960 Downloader middlewaars sit between the engine and the downloader. They 225 00:11:19.960 --> 00:11:23.039 act as interceptors. This is where you manipulate the request 226 00:11:23.200 --> 00:11:24.639 right before it leaves your machine. 227 00:11:24.679 --> 00:11:27.039 So this is where you'd rotate your user agents to 228 00:11:27.120 --> 00:11:31.759 mimic different web browsers or inject proxy IP addresses so 229 00:11:31.799 --> 00:11:35.080 the target server doesn't realize all these requests are coming 230 00:11:35.080 --> 00:11:36.039 from a single machine. 231 00:11:36.039 --> 00:11:41.279 Precisely. You modify the headers, attach cookies, or handle authentication tokens, 232 00:11:41.360 --> 00:11:43.639 whatever you need to do to prepare that request. 233 00:11:43.840 --> 00:11:44.320 Got it. 234 00:11:44.480 --> 00:11:47.720 Once the downloader middleware is done, the downloader reaches out 235 00:11:47.759 --> 00:11:51.279 to the target server, grabs the raw HTML, and constructs 236 00:11:51.320 --> 00:11:55.200 a response object. The downloader then hands that response back. 237 00:11:55.080 --> 00:11:58.080 To the engine, and the engine, keeping the assembly line moving, 238 00:11:58.480 --> 00:12:01.320 takes that raw HTML response and writes it back to 239 00:12:01.360 --> 00:12:02.679 our custom code the spider. 240 00:12:02.840 --> 00:12:04.480 We are finally back at the spider. 241 00:12:04.679 --> 00:12:07.000 Right. So wait, if the engine is the traffic cop 242 00:12:07.080 --> 00:12:10.519 routing requests and the downloader is the delivery driver bringing 243 00:12:10.519 --> 00:12:13.639 the HTML, who actually decides what data we care about keeping? 244 00:12:13.759 --> 00:12:17.200 Ah? Okay, The spider receives the response and applies the 245 00:12:17.200 --> 00:12:21.360 extraction logic you wrote. Usually this involves using XPath expressions 246 00:12:21.440 --> 00:12:23.960 or CSS selectors to traverse the dom tree of the 247 00:12:24.120 --> 00:12:26.679 HTML and pluck out the specific text strings you. 248 00:12:26.639 --> 00:12:30.360 Want, like the product title, the price, the stock availability. 249 00:12:30.679 --> 00:12:34.039 Right, the spiders extract the raw goods. But the spider 250 00:12:34.039 --> 00:12:36.399 doesn't just save the data to a file and call 251 00:12:36.440 --> 00:12:40.559 it a day. It yields an item object An item. Yeah, 252 00:12:40.600 --> 00:12:43.960 an item is essentially a dictionary, a structured container for 253 00:12:44.000 --> 00:12:46.960 the scraped data. And this brings us to the final 254 00:12:46.960 --> 00:12:49.879 stage of the architecture, which I think is arguably the 255 00:12:49.960 --> 00:12:54.279 most important for data integrity. The item pipeline. 256 00:12:54.320 --> 00:12:57.360 The item pipeline. Okay, so if the spiders extract the 257 00:12:57.480 --> 00:13:00.720 raw goods, the item pipeline is the crew vitual final 258 00:13:00.840 --> 00:13:03.799 quality control inspector that cleans and stores the data. 259 00:13:04.120 --> 00:13:07.639 I agree completely. In you know, amateur scraping scripts, extraction 260 00:13:07.720 --> 00:13:10.320 and storage are usually just mashed together. In the same function. 261 00:13:10.360 --> 00:13:11.679 Oh yeah, I've written those scripts. 262 00:13:11.720 --> 00:13:15.759 We all have, but Scrapeye forces you to decouple them. 263 00:13:16.080 --> 00:13:18.679 The spider's only job is to extract raw strings and 264 00:13:18.759 --> 00:13:21.799 yield an item. The engine takes that item and sends 265 00:13:21.799 --> 00:13:23.039 it down the item pipeline. 266 00:13:23.039 --> 00:13:24.919 I always think of the item pipeline as a series 267 00:13:24.960 --> 00:13:28.159 of quality assurance inspectors standing along a conveyor belt. 268 00:13:28.279 --> 00:13:29.080 That's a good visual. 269 00:13:29.360 --> 00:13:32.600 Each pipeline class you write is a separate inspector that 270 00:13:32.720 --> 00:13:36.960 receives the item, performs one specific check or transformation, and 271 00:13:37.039 --> 00:13:38.600 passes it to the next inspector. 272 00:13:38.840 --> 00:13:43.639 Exactly so, you might have say three pipeline stages. Stage 273 00:13:43.679 --> 00:13:46.600 one checks if the price field is empty. If it is, 274 00:13:46.720 --> 00:13:49.639 it drops the item entirely, because incomplete data is useless 275 00:13:49.639 --> 00:13:52.679 to you, makes sense. Stage two takes a price string 276 00:13:52.840 --> 00:13:55.320 like one thousand dollars and converts it into a clean 277 00:13:55.399 --> 00:13:59.000 float integer for your database. And then stage three takes 278 00:13:59.039 --> 00:14:02.480 that cleaned, valveolidated item and writes it to a post 279 00:14:02.519 --> 00:14:05.120 rescul database or maybe an Amazon S three bucket. 280 00:14:05.159 --> 00:14:08.559 This modular separation of concerns is just brilliant. If your 281 00:14:08.600 --> 00:14:11.440 database schema changes, you don't even touch your parsing code. 282 00:14:11.480 --> 00:14:14.360 You just update your pipeline exactly, and if the website 283 00:14:14.480 --> 00:14:17.559 changes its HTML structure, you update the spider. But your 284 00:14:17.600 --> 00:14:19.759 database pipeline remains perfectly intact. 285 00:14:19.840 --> 00:14:23.039 And because it's all asynchronous, while one item is waiting 286 00:14:23.080 --> 00:14:26.200 for the database to acknowledge the insert query in the pipeline, 287 00:14:26.279 --> 00:14:28.600 the spider is already parsing the next response and the 288 00:14:28.639 --> 00:14:30.519 downloader is catching fifty more pages. 289 00:14:30.879 --> 00:14:35.279 Okay, so we have this incredibly powerful, highly concurrent asynchronous 290 00:14:35.279 --> 00:14:39.360 engine pulling down data, parsing it, cleaning it, and storing 291 00:14:39.399 --> 00:14:40.480 it all simultaneously. 292 00:14:40.559 --> 00:14:41.639 It's a beast, it is. 293 00:14:42.200 --> 00:14:45.519 But having a massive, high speed automated assembly line is 294 00:14:45.559 --> 00:14:49.759 incredibly powerful. But without proper breaks, it's a disaster waiting 295 00:14:49.799 --> 00:14:50.200 to happen. 296 00:14:50.240 --> 00:14:51.039 Oh. Absolutely. 297 00:14:51.120 --> 00:14:54.320 If we point this crapy engine at a small, poorly 298 00:14:54.360 --> 00:14:57.559 optimized web server and twist it opens up a thousand 299 00:14:57.600 --> 00:15:01.399 concurrent connections, we are going to crash that server. 300 00:15:01.639 --> 00:15:05.000 Yes, you would essentially launch a distributed denial of service 301 00:15:05.000 --> 00:15:06.720 ADIDAS attack entirely. 302 00:15:06.399 --> 00:15:08.440 By accident, which is obviously very bad. 303 00:15:08.279 --> 00:15:10.600 Which is considered crawl abuse. Yes, and it is a 304 00:15:10.679 --> 00:15:14.039 major issue in the web data extraction industry. This is 305 00:15:14.039 --> 00:15:17.360 why Scrapeye provides such fine grain control to the developer. 306 00:15:17.600 --> 00:15:19.840 It is designed to be a polite and responsible tool, 307 00:15:20.080 --> 00:15:21.720 provided you can figure it correctly. 308 00:15:22.200 --> 00:15:24.559 Right, let's talk about the steering in the brakes. Then 309 00:15:24.919 --> 00:15:28.679 how does scrapee prevent chaos? How do we control this beast? 310 00:15:29.080 --> 00:15:32.720 It starts with hard limits. The framework provides fine grain 311 00:15:32.799 --> 00:15:38.960 control through three specific features. The first is limiting download requests. 312 00:15:39.000 --> 00:15:43.519 In your Scrapeia configuration settings, you can explicitly define concurrency limits. 313 00:15:43.919 --> 00:15:47.200 You can set a parameter like concurrent request burtamin to 314 00:15:47.360 --> 00:15:49.879 restrict how many open connections you have with a single 315 00:15:49.919 --> 00:15:51.720 website at any given millisecond. 316 00:15:51.840 --> 00:15:54.840 So I can say, never hit this specific real estate 317 00:15:54.879 --> 00:15:57.799 site with more than five simultaneous requests. 318 00:15:58.000 --> 00:15:58.159 Right. 319 00:15:58.440 --> 00:16:01.200 That protects the target server, But I also need to 320 00:16:01.240 --> 00:16:05.480 protect my own system's memory. Right, How does the framework 321 00:16:05.519 --> 00:16:07.519 handle tasks in parallel on my end? 322 00:16:07.759 --> 00:16:11.279 That's the second feature, managing concurrent connections globally. You have 323 00:16:11.360 --> 00:16:15.159 a global concurrent requests. Setting this limits the total number 324 00:16:15.159 --> 00:16:18.639 of connections across all domains you might be crawling at once. Oh, okay, 325 00:16:18.720 --> 00:16:20.399 if you set that to one hundred, the engine will 326 00:16:20.399 --> 00:16:22.759 never process more than one hundred active downloads at a 327 00:16:22.799 --> 00:16:25.639 time across your whole project. That keeps your RAM and 328 00:16:25.679 --> 00:16:27.879 CPU usage entirely predictable. 329 00:16:28.240 --> 00:16:31.159 But hard coding limits can be tricky, right. I mean, 330 00:16:31.200 --> 00:16:34.320 if a server's fast, maybe you can handle twenty requests 331 00:16:34.320 --> 00:16:37.200 a second. If it's under heavy load, maybe it struggles 332 00:16:37.240 --> 00:16:39.919 with two. If you hardcode your limit to twenty, you 333 00:16:40.039 --> 00:16:43.120 might still overwhelm it during its peak hours. And here's 334 00:16:43.159 --> 00:16:46.440 where it gets really interesting. Scrapee has a built in 335 00:16:46.519 --> 00:16:51.039 feature called the autothrottle Extension. Yes, it doesn't just apply 336 00:16:51.080 --> 00:16:53.919 a static limit. The software is actually smart enough to 337 00:16:54.000 --> 00:16:56.600 regulate its own speed using a feedback loop. 338 00:16:57.159 --> 00:17:00.840 Auto throttle is a phenomenal piece of engineering. It's crucial 339 00:17:00.919 --> 00:17:04.359 for preventing the system from becoming overloaded. Instead of guessing 340 00:17:04.440 --> 00:17:07.799 how fast you should crawl, auto throttle continuously measures the 341 00:17:07.880 --> 00:17:09.319 latency of the target server. 342 00:17:09.519 --> 00:17:10.279 How does it do that? 343 00:17:10.759 --> 00:17:14.359 It calculates the exact time between when the downloader sends 344 00:17:14.400 --> 00:17:17.720 a request and when the response heeaders actually begin to arrive. 345 00:17:17.880 --> 00:17:19.400 Oh wow. So if you think about it as a 346 00:17:19.400 --> 00:17:22.839 PID controller, it's like cruise control in a car reacting 347 00:17:22.880 --> 00:17:26.079 to a hill. If the target server starts responding slower 348 00:17:26.119 --> 00:17:30.400 than usual, meaning its latency goes up. Auto throttle detects 349 00:17:30.400 --> 00:17:34.720 the struggle and automatically increases the delay for your next requests. 350 00:17:35.559 --> 00:17:37.319 It taps the brakes precisely. 351 00:17:37.519 --> 00:17:40.680 It mathematically adjusts the delay to match the server's capacity 352 00:17:40.720 --> 00:17:43.599 in real time. The goal of autothrottle is to ensure 353 00:17:43.599 --> 00:17:47.160 that you always maintain a target concurrency, say an average 354 00:17:47.160 --> 00:17:50.960 of one request per second, without ever overwhelming the server's queue. Right, 355 00:17:51.039 --> 00:17:53.880 and then as the server speeds back up, auto throttle 356 00:17:54.039 --> 00:17:58.440 slowly releases the brakes, optimizing your crawl speed without crossing 357 00:17:58.480 --> 00:17:59.559 the line into abuse. 358 00:18:00.039 --> 00:18:02.640 I will say I push back on auto throttle sometimes. 359 00:18:02.319 --> 00:18:03.920 Though, oh really, why is that? 360 00:18:04.400 --> 00:18:06.920 Well? I've seen developers turn it on and suddenly their 361 00:18:06.960 --> 00:18:08.839 crawl that you used to take ten minutes now takes 362 00:18:08.880 --> 00:18:11.839 three hours. It can be overly conservative if you don't 363 00:18:11.880 --> 00:18:12.880 tune its parameters. 364 00:18:12.920 --> 00:18:16.440 Well, that's a valid critique, but auto throttle is conservative 365 00:18:16.480 --> 00:18:19.880 by design. And if we connect this to the bigger picture, 366 00:18:20.279 --> 00:18:23.640 throttling isn't just about protecting your own system. It's about 367 00:18:23.640 --> 00:18:27.960 being a responsible digital citizen. You don't want to accidentally 368 00:18:28.039 --> 00:18:30.279 launch a denial of service attack on the website you 369 00:18:30.359 --> 00:18:32.440 are trying to learn from Yeah, that's fair. Plus you 370 00:18:32.440 --> 00:18:34.960 can tune it. You can change the initial download delay, 371 00:18:35.039 --> 00:18:38.799 the maximum delay, and the target concurrency. The key is 372 00:18:38.920 --> 00:18:43.119 understanding that you are trading raw aggressive speed for stealth 373 00:18:43.200 --> 00:18:43.960 and stability. 374 00:18:44.319 --> 00:18:44.559 Right. 375 00:18:45.039 --> 00:18:48.839 An overly aggressive crawl without throttling is the fastest way 376 00:18:48.839 --> 00:18:51.759 to trigger a website's anti bought protection and get your 377 00:18:51.799 --> 00:18:56.200 IP address permanently banned. A throttled crawl might take longer, 378 00:18:56.519 --> 00:18:58.759 but it actually finishes successfully. 379 00:18:58.319 --> 00:19:02.880 Stealth, stability, and data integrity. That is really the true 380 00:19:02.960 --> 00:19:04.400 value proposition of the framework. 381 00:19:04.480 --> 00:19:05.440 It absolutely is. 382 00:19:05.720 --> 00:19:08.400 Well, we've covered the entire factory floor today. We started 383 00:19:08.400 --> 00:19:12.559 by understanding frameworks versus libraries, establishing that SCRAPEE dictates the 384 00:19:12.640 --> 00:19:14.480 architecture through inversion of control. 385 00:19:14.599 --> 00:19:15.039 We did. 386 00:19:15.240 --> 00:19:18.279 We explored how the twisted reactor loop allows for massive 387 00:19:18.400 --> 00:19:22.119 asynchronous concurrency without blocking the main thread, and then. 388 00:19:22.079 --> 00:19:25.680 We trace the entire data flow through the assembly line. 389 00:19:25.480 --> 00:19:28.079 Right the engine orchestrating the process, the scheduler queuing and 390 00:19:28.160 --> 00:19:32.960 d duplicating requests, the downloader fetching the actual HTML. 391 00:19:32.480 --> 00:19:36.920 Web pages, the spiders parsing the dom and yielding items, and. 392 00:19:36.960 --> 00:19:40.960 Finally the item pipeline cleaning and persisting the data all 393 00:19:41.000 --> 00:19:43.599 locked down with the safety of concurrency limits and the 394 00:19:43.680 --> 00:19:46.200 dynamic feedback loop of the autothrottle extension. 395 00:19:46.279 --> 00:19:49.319 It's an elegant system. Once you see all the pieces together. 396 00:19:49.160 --> 00:19:52.880 It really is. Now this architecture solves the networking problem. 397 00:19:53.079 --> 00:19:55.720 But once you have that raw HTML and your spider, 398 00:19:56.279 --> 00:19:58.359 you still have to parse the dom tree to extract 399 00:19:58.359 --> 00:19:58.759 the data. 400 00:19:58.920 --> 00:19:59.480 Yes, you do. 401 00:20:00.039 --> 00:20:03.079