WEBVTT 1 00:00:00.040 --> 00:00:01.919 I want you to imagine a scenario for a second. 2 00:00:02.080 --> 00:00:04.639 Let's say you have a new task at work or 3 00:00:05.120 --> 00:00:07.759 maybe a personal project, and you need to collect the 4 00:00:07.839 --> 00:00:11.880 daily stock prices of every single major technology company. 5 00:00:11.960 --> 00:00:12.439 Oh wow. 6 00:00:12.679 --> 00:00:15.199 Right, So you open your browser, you navigate to a 7 00:00:15.240 --> 00:00:19.519 finance site, you highlight a company name, hit copy, switch 8 00:00:19.559 --> 00:00:21.480 to your spreadsheet, hit paste. 9 00:00:21.879 --> 00:00:24.120 Just hearing that is making me tired, I know. Right. 10 00:00:24.519 --> 00:00:29.239 Then you go back, find the price, highlight copy switch paste. 11 00:00:29.280 --> 00:00:32.320 Now imagine doing that for fifty companies. Yeah, no, thank you, 12 00:00:32.520 --> 00:00:35.000 And now imagine doing that every single day. 13 00:00:35.439 --> 00:00:39.520 I mean it sounds completely exhausting. That is the literal 14 00:00:39.560 --> 00:00:41.920 definition of manual labor in the digital age. 15 00:00:41.960 --> 00:00:42.560 It really is. 16 00:00:42.719 --> 00:00:46.039 It's error prone, it's tedious, and honestly, it's just a 17 00:00:46.119 --> 00:00:48.280 terrible use of human intelligence. Like when we spend our 18 00:00:48.280 --> 00:00:51.679 time acting like human copy paste machines, we aren't analyzing 19 00:00:51.719 --> 00:00:55.079 the data. We're just, you know, plindly transporting it exactly. 20 00:00:55.119 --> 00:00:58.719 But what if instead of manually migrating numbers for hours 21 00:00:58.799 --> 00:01:02.039 until your eyes literally really cross, you could dispatch a 22 00:01:02.119 --> 00:01:04.599 custom built automated digital assistant. 23 00:01:04.640 --> 00:01:06.000 Now that sounds much better. 24 00:01:06.000 --> 00:01:08.319 Right, and assistant to go out into the internet and 25 00:01:08.319 --> 00:01:12.799 gather everything for you in seconds. So today's mission in 26 00:01:12.840 --> 00:01:16.120 this deep dive is all about moving past that prototyping phase. 27 00:01:16.439 --> 00:01:19.760 We are talking about productionizing a data gathering operation. 28 00:01:19.920 --> 00:01:21.840 Yeah, taking it to the next level exactly. 29 00:01:21.959 --> 00:01:24.680 Okay, let's unpack this. Our goal today is to take 30 00:01:24.840 --> 00:01:29.840 abstract webscraping concepts and turn them into practical, deployable code 31 00:01:30.040 --> 00:01:33.359 that you can actually use to build custom automated bots 32 00:01:33.920 --> 00:01:35.920 or what we call spiders. 33 00:01:36.120 --> 00:01:38.719 It's a fantastic transition to make. I mean, we're fundamentally 34 00:01:39.000 --> 00:01:42.120 shifting from being passive consumers of the web right where 35 00:01:42.159 --> 00:01:45.159 we just you know, read what a browser renders for us. 36 00:01:45.359 --> 00:01:49.719 We're shifting to becoming active, programmatic gatherers of information. 37 00:01:49.840 --> 00:01:52.239 We're treating the Internet like a database exactly. 38 00:01:52.239 --> 00:01:54.599 We're treating the Internet not as a magazine to flip through, 39 00:01:54.599 --> 00:01:58.239 it as a massive, queriable database. But before we get 40 00:01:58.239 --> 00:02:00.319 into the nuts and bolts of building a spy, we 41 00:02:00.359 --> 00:02:02.719 have to understand the machinery we're plugging into. 42 00:02:02.640 --> 00:02:03.680 The framework itself. 43 00:02:03.799 --> 00:02:07.280 Right, scrapy isn't just a simple library, it's a comprehensive 44 00:02:07.280 --> 00:02:10.199 application framework. And if we don't have a blueprint of 45 00:02:10.240 --> 00:02:12.960 how that framework operates, under the hood, we're really just 46 00:02:13.039 --> 00:02:15.800 blindly guessing at how to structure our code. 47 00:02:16.039 --> 00:02:20.240 Yeah, and understanding the why behind the architecture definitely makes 48 00:02:20.280 --> 00:02:24.800 writing the actual code like infinitely easier. Oh absolutely, Because 49 00:02:24.919 --> 00:02:28.360 when you first look at the documentation diagrams of how 50 00:02:28.400 --> 00:02:31.960 Screepy handles data, it can look incredibly overwhelming. 51 00:02:32.080 --> 00:02:33.039 It's a lot to take in. 52 00:02:33.120 --> 00:02:36.840 There are arrows pointing in a dozen different directions mapping 53 00:02:36.879 --> 00:02:40.000 out all these asynchronous data flows. It's wild. 54 00:02:40.159 --> 00:02:42.439 Yeah, at first glance, it absolutely looks like a complex 55 00:02:42.520 --> 00:02:46.280 subway map, but the core philosophy is actually quite elegant. 56 00:02:46.719 --> 00:02:49.319 So at the absolute heart of this Scrapy framework is 57 00:02:49.360 --> 00:02:51.800 a component called the Scrapy Engine, the brain of the 58 00:02:51.840 --> 00:02:54.879 operation exactly. Think of the engine as the central nervous 59 00:02:54.879 --> 00:02:57.639 system of the entire setup. It controls the flow of 60 00:02:57.719 --> 00:03:01.439 data between all the other components and and it triggers 61 00:03:01.520 --> 00:03:03.159 the events that everything else responds to. 62 00:03:03.599 --> 00:03:06.840 So if the engine is the central nervous system, all 63 00:03:06.879 --> 00:03:11.000 the custom logic we write to parse and extract information 64 00:03:11.560 --> 00:03:15.879 is really just a small peripheral attachment to this much 65 00:03:15.960 --> 00:03:16.719 larger puzzle. 66 00:03:16.879 --> 00:03:18.080 Right, that's a great way to put it. 67 00:03:18.120 --> 00:03:20.560 We write our custom instructions, but we don't actually trigger 68 00:03:20.599 --> 00:03:24.360 them ourselves. The Scrapy engine is the entity calling the shots, 69 00:03:24.599 --> 00:03:24.960 and that. 70 00:03:24.960 --> 00:03:27.960 Is the defining characteristic of a framework versus a library. 71 00:03:28.120 --> 00:03:29.159 Oh interesting, how so? 72 00:03:29.639 --> 00:03:31.919 Well, with a library, your code is in control, right, 73 00:03:32.000 --> 00:03:34.479 you call the library when you need help. But with 74 00:03:34.520 --> 00:03:37.520 a framework like Scrapye, the framework is in control and 75 00:03:37.560 --> 00:03:40.479 it calls your code when it needs your specific logic. 76 00:03:40.639 --> 00:03:40.960 Got it? 77 00:03:41.360 --> 00:03:41.639 Okay? 78 00:03:41.680 --> 00:03:44.879 To make this architecture super concrete for everyone listening, and 79 00:03:44.960 --> 00:03:48.000 to avoid getting bogged down in networking jargon right out 80 00:03:48.000 --> 00:03:51.319 of the gate, consider this analogy, kayl it's here. Imagine 81 00:03:51.319 --> 00:03:55.080 Scrapeye is a highly efficient, high end restaurant kitchen. 82 00:03:55.280 --> 00:03:56.439 I love food analogies. 83 00:03:56.520 --> 00:03:59.520 Right, So the Scrapy engine is the head chef. They're 84 00:03:59.639 --> 00:04:02.960 standing at the pass orchestrating the entire room, making sure 85 00:04:03.000 --> 00:04:06.199 everyone is doing their highly specialized job at the exact 86 00:04:06.280 --> 00:04:06.759 right time. 87 00:04:07.000 --> 00:04:07.479 Makes sense. 88 00:04:07.719 --> 00:04:09.840 Now, the head chef needs to know what meal to 89 00:04:09.879 --> 00:04:12.360 cook next, so the engine turns to a component called 90 00:04:12.360 --> 00:04:16.600 the scheduler. In our kitchen, the scheduler is the ticket rail. 91 00:04:17.279 --> 00:04:17.759 Oh okay. 92 00:04:17.839 --> 00:04:21.399 It holds all the incoming orders, prioritizes them, and tells 93 00:04:21.399 --> 00:04:24.959 the head chef, Hey, here's the very next URL that 94 00:04:25.040 --> 00:04:26.040 needs to be processed. 95 00:04:26.319 --> 00:04:29.639 What's fascinating here is how that separates the concerns, Like 96 00:04:30.519 --> 00:04:33.759 the engine doesn't have to waste memory or processing power 97 00:04:33.800 --> 00:04:35.040 deciding what to do next. 98 00:04:35.199 --> 00:04:37.199 It just relies on the ticket rail exactly. 99 00:04:37.279 --> 00:04:40.839 The scheduler handles all that complex queuing logic, you know, 100 00:04:41.240 --> 00:04:44.800 maybe prioritizing certain domains over others, and simply hands the 101 00:04:44.839 --> 00:04:46.920 next request over when the engine is ready for it. 102 00:04:47.000 --> 00:04:49.480 And once the head chef gets that ticket, they don't 103 00:04:49.560 --> 00:04:52.319 leave the kitchen to go fetch the ingredients. The engine 104 00:04:52.360 --> 00:04:54.759 takes that URL request and hands it to the downloader 105 00:04:54.800 --> 00:04:57.560 the runner exactly. The downloader is our kitchen runner. This 106 00:04:57.600 --> 00:05:01.600 component actually leaves the restaurant, navigates the chaos of the market, 107 00:05:01.879 --> 00:05:05.199 which in this case is the internet, very chaotic, highly chaotic. 108 00:05:05.319 --> 00:05:07.959 It fetches the raw web page content and brings it 109 00:05:08.000 --> 00:05:09.279 back to the head chef. 110 00:05:09.240 --> 00:05:13.240 And the downloader. Is arguably where scrape provides the most 111 00:05:13.319 --> 00:05:14.639 value straight out of the. 112 00:05:14.560 --> 00:05:16.279 Box, because networking is hard. 113 00:05:16.439 --> 00:05:20.680 Oh. As a developer, dealing with network requests is notoriously frustrating. 114 00:05:20.959 --> 00:05:26.399 You have to handle timeouts, retries, SSL certificates, connection pooling. 115 00:05:27.160 --> 00:05:29.079 It's a nightmare, and Scrapey just does all that. 116 00:05:29.279 --> 00:05:32.959 Yep, the Scrapy downloader handles all of that complex Internet 117 00:05:33.000 --> 00:05:37.600 handshake logic asynchronously. It can manage hundreds of these requests 118 00:05:37.680 --> 00:05:40.120 concurrently without blocking the rest of the system. 119 00:05:40.199 --> 00:05:41.000 That's incredible. 120 00:05:41.040 --> 00:05:43.160 It just goes out, does the heavy lifting and returns 121 00:05:43.199 --> 00:05:46.439 a nice, neat response object back to the engine. 122 00:05:46.120 --> 00:05:48.839 Which brings us to our role in this whole operation. 123 00:05:49.279 --> 00:05:51.560 If the engine is the head chef and the downloader 124 00:05:51.600 --> 00:05:54.639 is the runner, the spider, which is the actual code 125 00:05:54.680 --> 00:05:57.360 you write, is the specialized prep cook. 126 00:05:57.199 --> 00:05:58.240 The prip cook. I like that. 127 00:05:58.519 --> 00:06:01.720 Yeah. The runner brings back a massive, messy create of 128 00:06:01.839 --> 00:06:05.040 raw HTML. The head chef drops it on your station. 129 00:06:05.360 --> 00:06:07.399 You don't care how it got there. You don't care 130 00:06:07.399 --> 00:06:08.639 about the network protocols. 131 00:06:08.759 --> 00:06:09.879 Now you're problem right. 132 00:06:09.920 --> 00:06:12.399 You already have your precise instructions on how to slice, 133 00:06:12.480 --> 00:06:15.199 dice and extract just the prime cuts of data we want. 134 00:06:15.439 --> 00:06:19.839 And once you, the prep cook isolate those prime cuts, 135 00:06:20.279 --> 00:06:23.600 say the stock names and prices. You hand that finished 136 00:06:23.639 --> 00:06:27.040 item back. Scrapey then passes it to the item pipelines 137 00:06:27.199 --> 00:06:30.959 the packaging department. Precisely the pipelines box the meal up 138 00:06:30.959 --> 00:06:34.600 for delivery, They validate the data, check for missing fields, 139 00:06:35.720 --> 00:06:38.639 maybe convert a tech string into a proper decimal number, 140 00:06:38.879 --> 00:06:40.399 and finally save it to a database. 141 00:06:40.480 --> 00:06:43.600 So by enforcing this strict division of labor, Scrapey allows 142 00:06:43.600 --> 00:06:45.879 you to focus purely on the logic of data extraction. 143 00:06:46.120 --> 00:06:47.920 Exactly, you just write the prep cook constructions. 144 00:06:48.360 --> 00:06:51.160 Okay, so we have our restaurant kitchen fully staffed, but 145 00:06:51.279 --> 00:06:53.720 before we can dispatch our runner to the market, we 146 00:06:53.920 --> 00:06:55.800 have to know what we are shopping for. 147 00:06:56.040 --> 00:06:56.800 We need to target. 148 00:06:56.879 --> 00:06:59.360 We need to scout our target. So let's use the 149 00:06:59.439 --> 00:07:03.079 Yahoo Finans tech sector page as our hypothetical target today. 150 00:07:03.560 --> 00:07:06.160 Our specific mission is to grab the symbol name and 151 00:07:06.199 --> 00:07:09.120 the inter day price for every stock listed on that page. 152 00:07:09.240 --> 00:07:11.759 To do that, programmatically, we really have to understand how 153 00:07:11.759 --> 00:07:15.439 that data is visually presented versus how it's structurally embedded 154 00:07:15.439 --> 00:07:19.199 in the code the HTML. Right, Every website you interact 155 00:07:19.199 --> 00:07:22.800 with is built on a Document Object model or DOM. 156 00:07:23.360 --> 00:07:27.199 It's essentially a giant inverted tree of nested HTML tags. 157 00:07:27.319 --> 00:07:29.879 So it's not just text, No, not at all. 158 00:07:30.199 --> 00:07:33.199 If you open your browsers developer tools and inspect the page, 159 00:07:33.319 --> 00:07:35.519 you aren't just looking at text, you are looking at 160 00:07:35.519 --> 00:07:37.319 the structural skeleton of the site. 161 00:07:37.360 --> 00:07:38.839 And I know it can look like a waterfall of 162 00:07:38.920 --> 00:07:42.319 chaotic text at first, but it's actually a highly organized hierarchy. 163 00:07:42.680 --> 00:07:45.519 On a site like Yahoo Finance, tabular data like stock 164 00:07:45.560 --> 00:07:50.800 prices usually lives inside, unsurprisingly an HTML table element. 165 00:07:50.600 --> 00:07:52.160 Right, the good old table tag. 166 00:07:52.040 --> 00:07:55.759 Yeah, and web developers typically attach unique identifiers to these 167 00:07:55.759 --> 00:07:58.600 structural containers so they can style them later. So in 168 00:07:58.639 --> 00:08:01.639 our specific target, the main data table might have a 169 00:08:01.759 --> 00:08:04.959 unique class name attached to it, like a label yeah, 170 00:08:05.000 --> 00:08:08.519 something like a strash table. That class name is our 171 00:08:08.560 --> 00:08:09.240 anchor point. 172 00:08:09.399 --> 00:08:12.040 And once you find that anchor, you navigate the nesting 173 00:08:12.079 --> 00:08:15.560 dolls of the HQL structure. Inside that main table, there's 174 00:08:15.639 --> 00:08:17.759 a T head section for the headers and a T 175 00:08:17.920 --> 00:08:20.800 body section for the content. Inside the T body, the 176 00:08:20.920 --> 00:08:24.360 data is organized row by row using tr tag stable 177 00:08:24.439 --> 00:08:29.000 row tags, and finally, inside those rows, the individual pieces 178 00:08:29.000 --> 00:08:31.319 of data like the actual company name or the stock 179 00:08:31.360 --> 00:08:36.039 price are isolated inside individual TD cells or table data cells. 180 00:08:36.240 --> 00:08:39.120 So to reach into that specific cell and grab the data, 181 00:08:39.240 --> 00:08:42.120 we use a tool called XPath. Instead of writing a 182 00:08:42.120 --> 00:08:46.080 complex regular expression to search through the raw text, XPath 183 00:08:46.240 --> 00:08:49.879 acts as a dedicated query language designed specifically for navigating 184 00:08:50.039 --> 00:08:51.679 XML and HTML documents. 185 00:08:51.879 --> 00:08:54.399 You think of XPath not as a set of direct coordinates, 186 00:08:54.399 --> 00:08:55.919 but as a set of descriptive. 187 00:08:55.440 --> 00:08:57.759 Instructions, like giving someone directions exactly. 188 00:08:58.080 --> 00:09:00.960 You can use your browser to generate an absolute XPath, 189 00:09:01.000 --> 00:09:03.840 which is like saying walk exactly three steps forward, turn left, 190 00:09:03.879 --> 00:09:06.360 go down two steps, and open the box. The problem 191 00:09:06.440 --> 00:09:09.360 is that absolute path only points to one single cell 192 00:09:09.399 --> 00:09:11.679 on the entire page. We don't want just one stock, 193 00:09:11.720 --> 00:09:13.080 we want the entire list. 194 00:09:13.120 --> 00:09:15.679 So we have to alter the XPath to be generic. 195 00:09:16.480 --> 00:09:18.879 We change the query from a rigid set of coordinates 196 00:09:18.879 --> 00:09:22.080 into a pattern matching instruction. We basically tell the spider 197 00:09:22.519 --> 00:09:26.840 find every troro that exists inside this specific table container 198 00:09:27.200 --> 00:09:29.159 and extract the contents of the name cell and the 199 00:09:29.200 --> 00:09:29.879 price cell. 200 00:09:29.759 --> 00:09:32.240 For all of them. And that allows a single elegant 201 00:09:32.320 --> 00:09:35.919 line of code to extract fifty different data points simultaneously. 202 00:09:36.080 --> 00:09:39.639 Wait if Yahoo Finance updates their website tomorrow and changes 203 00:09:39.679 --> 00:09:43.600 that table structure, won't our automated spider completely break. Is 204 00:09:43.799 --> 00:09:46.360 XPath really the most reliable way to do this? 205 00:09:46.960 --> 00:09:49.279 That is a great question, and yes, that is the 206 00:09:49.320 --> 00:09:53.320 inherent fragility of all web scriping. You are building software 207 00:09:53.360 --> 00:09:55.600 that relies on the structural integrity of a system you 208 00:09:55.639 --> 00:09:56.639 do not control. 209 00:09:56.399 --> 00:09:58.320 Which is terrifying as a developer. 210 00:09:58.399 --> 00:10:02.320 It is when the architecture of scrapers break. But there 211 00:10:02.320 --> 00:10:05.120 are ways to build resilience into your spiders, which is 212 00:10:05.120 --> 00:10:07.840 where a more advanced scrapy feature called item loaders comes 213 00:10:07.879 --> 00:10:08.320 into play. 214 00:10:08.399 --> 00:10:11.039 Okay, how do item loaders solve a shifting layout? 215 00:10:11.200 --> 00:10:14.000 Well? Instead of relying heavily on the visual layout tags 216 00:10:14.240 --> 00:10:17.240 like rows and columns which change all the time, item 217 00:10:17.240 --> 00:10:20.039 loaders allow you to target custom data attributes. 218 00:10:20.159 --> 00:10:21.279 Custom attributes. 219 00:10:21.440 --> 00:10:26.519 Yeah. Modern web developers often embed invisible meta tags directly 220 00:10:26.679 --> 00:10:31.000 into the HTML elements, specifically for their internal JavaScript or 221 00:10:31.039 --> 00:10:34.039 analytics to read. Think of it like an invisible name tag. 222 00:10:34.159 --> 00:10:35.399 Oh that's clever, right. 223 00:10:35.600 --> 00:10:38.200 Instead of telling your spider extract the data from the 224 00:10:38.200 --> 00:10:40.720 third chair in the second row, you tell the item 225 00:10:40.759 --> 00:10:44.120 loader scan the room and extract the data from whoever 226 00:10:44.200 --> 00:10:47.000 is wearing the name tag labeled current price. 227 00:10:47.480 --> 00:10:50.559 That makes total sense. It completely decouples the data extraction 228 00:10:50.639 --> 00:10:53.840 from the visual presentation exactly. The developers can move that 229 00:10:53.879 --> 00:10:55.960 price data anywhere on the page. They can put it 230 00:10:56.000 --> 00:10:58.080 in a new table or changes color, but as long 231 00:10:58.080 --> 00:11:00.919 as that invisible name tag remains, the item loader will 232 00:11:00.919 --> 00:11:01.200 find it. 233 00:11:01.320 --> 00:11:05.279 Yep. It makes your spiders significantly more resilient to routine 234 00:11:05.360 --> 00:11:09.200 front end updates. But you know, for a prototype, understanding 235 00:11:09.240 --> 00:11:12.519 the foundational XPath targeting is still crucial because you always 236 00:11:12.519 --> 00:11:15.000 need a fallback method to navigate the documentary. 237 00:11:15.360 --> 00:11:18.320 So we have our target, we've mapped the HTML structure, 238 00:11:18.519 --> 00:11:21.519 and we understand our extraction strategy. It's time to build 239 00:11:21.519 --> 00:11:24.879 the infrastructure. We are moving from inspecting elements in a 240 00:11:24.919 --> 00:11:27.720 browser to setting up our workspace and the command line. 241 00:11:28.120 --> 00:11:30.799 But we aren't just like creating a blank text file 242 00:11:30.879 --> 00:11:32.000 and starting from scratch. 243 00:11:32.240 --> 00:11:36.559 No, we use scaffolding. Scrapeye comes with built in command 244 00:11:36.639 --> 00:11:40.039 line tools that generate the foundational architecture for you. 245 00:11:40.120 --> 00:11:41.840 Which is such a time saver. 246 00:11:41.879 --> 00:11:45.600 Huge time saver. By running a simple startup command like 247 00:11:45.720 --> 00:11:49.799 scrapey start Projects, simple Spider, scrapey instantly generates an entire 248 00:11:49.840 --> 00:11:53.519 directory structure. Just boom. All there, Yeah, the configuration files, 249 00:11:53.840 --> 00:11:57.320 the mogular folders for pipelines, the settings files, and a 250 00:11:57.360 --> 00:11:59.120 dedicated folder just for your spiders. 251 00:11:59.320 --> 00:12:02.159 And modern de development relies so heavily on this kind 252 00:12:02.200 --> 00:12:05.960 of automated staffolding because it enforces best practices right out 253 00:12:06.000 --> 00:12:07.639 of the gate. You don't have to wonder where to 254 00:12:07.639 --> 00:12:10.159 put your settings or how to name your configuration file. 255 00:12:10.600 --> 00:12:14.000 The framework dictates the architecture, which ensures everything plugs into 256 00:12:14.080 --> 00:12:17.200 that central engine seamlessly, and once. 257 00:12:16.960 --> 00:12:22.240 The workspace is generated, you use another Scrapey command, scrapeygen spider, 258 00:12:22.559 --> 00:12:26.200 to generate the actual Python script for your spider, pointing 259 00:12:26.240 --> 00:12:29.320 it at the Yahoo finance to me. This generates a 260 00:12:29.320 --> 00:12:33.799 boilerplate Python file that inherits from the core scrapy Spider module. 261 00:12:33.960 --> 00:12:37.519 That inheritance is the magic key. Right by simply telling 262 00:12:37.600 --> 00:12:40.000 Python that our custom class is a child of the 263 00:12:40.039 --> 00:12:44.159 main Scrapy spider, our little script automatically inherits all the 264 00:12:44.360 --> 00:12:48.759 asynchronous downloading capabilities, the scheduling hooks, and the massive underlying 265 00:12:48.799 --> 00:12:50.759 power of the framework. We just have to fill in 266 00:12:50.799 --> 00:12:52.639 a few custom variables exactly. 267 00:12:52.919 --> 00:12:55.639 The framework provides three vital variables you need to configure 268 00:12:55.679 --> 00:12:57.879 immediately in your new Python file. The first is the 269 00:12:57.960 --> 00:13:00.559 name of the spider, just a string right, unique string 270 00:13:00.600 --> 00:13:02.919 you'll use to trigger it later from the terminal. The 271 00:13:02.960 --> 00:13:05.919 second is the starterlls array. This is the very first 272 00:13:05.960 --> 00:13:08.360 web address the spider should visit to kick off the 273 00:13:08.399 --> 00:13:11.399 crawling process, which we update to point to our specific 274 00:13:11.440 --> 00:13:12.480 tech sector page. 275 00:13:12.600 --> 00:13:15.960 And the third variable is incredibly important for responsible screeping 276 00:13:16.519 --> 00:13:19.240 allowed domains. This is basically a safety leash. 277 00:13:19.519 --> 00:13:23.320 It is a critical safeguard. Without restricting the allowed domains, 278 00:13:23.759 --> 00:13:26.600 a spider designed to follow links can quickly spiral out 279 00:13:26.639 --> 00:13:27.440 of control. 280 00:13:27.120 --> 00:13:28.039 Take a runaway train. 281 00:13:28.679 --> 00:13:31.720 Literally. It might follow a social media icon on a 282 00:13:31.759 --> 00:13:34.720 finance page, jump over to a completely different network, and 283 00:13:34.759 --> 00:13:37.600 suddenly your spider is trying to download the entire Internet, 284 00:13:37.960 --> 00:13:41.279 just recursively, crowling millions of irrelevant pages, which. 285 00:13:41.080 --> 00:13:44.000 You'll get your IP banned very quickly, very quickly. 286 00:13:44.240 --> 00:13:47.320 So by setting the allowed domain strictly to the finance site, 287 00:13:47.600 --> 00:13:50.559 the engine will automatically drop any requests that try to 288 00:13:50.600 --> 00:13:51.399 wander off site. 289 00:13:51.440 --> 00:13:54.440 Get to know Okay. With the safety leash secure, we 290 00:13:54.559 --> 00:13:57.399 get to the core logic the parse method. This is 291 00:13:57.440 --> 00:14:00.200 the function inside our class that the engine triggers. The 292 00:14:00.320 --> 00:14:03.720 very moment the downloader returns with the raw HTML, the 293 00:14:03.799 --> 00:14:06.480 runner drops the creative ingredients on the table and the 294 00:14:06.519 --> 00:14:08.159 parse method takes over right. 295 00:14:08.360 --> 00:14:12.200 The parse method receives the HTTP response object. It contains 296 00:14:12.279 --> 00:14:14.440 the raw source code of the page. And this is 297 00:14:14.440 --> 00:14:17.120 where you apply those generic xpad queries. 298 00:14:16.720 --> 00:14:19.200 We discussed using that extract function right. 299 00:14:19.600 --> 00:14:23.000 Yes, you instruct the script to look at the response, 300 00:14:23.639 --> 00:14:26.960 apply the xpaths to isolate all the company names, and 301 00:14:27.080 --> 00:14:29.720 use extract to save them into a synchronized list in 302 00:14:29.759 --> 00:14:32.399 the computer's memory. And then you do the exact same 303 00:14:32.440 --> 00:14:35.279 thing for the stock prices, generating a parallel list. 304 00:14:35.360 --> 00:14:38.360 Because scrape handled all the complex network fetching and the 305 00:14:38.399 --> 00:14:41.600 asynchronous waiting, the actual logic we write in the parse 306 00:14:41.639 --> 00:14:45.159 method is incredibly minimal. We just take those two synchronized 307 00:14:45.200 --> 00:14:48.360 lists residing in memories, zip them together, and process them. 308 00:14:48.559 --> 00:14:50.480 Yeah, in a simple prototype, you might just write a 309 00:14:50.480 --> 00:14:54.000 basic four loop that iterates through those lists and tells 310 00:14:54.039 --> 00:14:56.840 Python to put the names and prices directly to your screen. 311 00:14:56.960 --> 00:14:59.720 Here's where it gets really interesting. When you execute the 312 00:14:59.720 --> 00:15:03.919 spy from your terminal using screepy crawl company details, the 313 00:15:03.960 --> 00:15:06.159 beauty of the framework just reveals itself. 314 00:15:06.200 --> 00:15:07.000 It's a great feeling. 315 00:15:07.200 --> 00:15:09.720 You see the engine fire up, the scheduler, queue the request, 316 00:15:09.840 --> 00:15:12.600 the downloader, fetch the page, and suddenly your terminal is 317 00:15:12.639 --> 00:15:17.320 flooded with successful two hundred status responses and perfectly organized data. 318 00:15:17.600 --> 00:15:20.840 You'll see the system logs flying by confirming the successful 319 00:15:20.879 --> 00:15:24.159 network handshakes that all just happened invisibly in the background. 320 00:15:24.279 --> 00:15:27.960 It's undeniably satisfying to see that structured data cascade down 321 00:15:27.960 --> 00:15:30.879 the screen. You've successfully forced the Internet to hand over 322 00:15:30.919 --> 00:15:34.399 its raw data on your terms. But as we establish 323 00:15:34.440 --> 00:15:37.840 in our mission today, terminal output is ephemeral. It's a 324 00:15:37.879 --> 00:15:40.360 great party trick to prove the concept works, but the 325 00:15:40.399 --> 00:15:44.000 moment you close that window, the data vanishes into the ether. 326 00:15:44.279 --> 00:15:44.879 It's gone. 327 00:15:45.039 --> 00:15:48.159 Right. For a true production environment, that data needs to 328 00:15:48.200 --> 00:15:49.440 be permanently persisted. 329 00:15:49.840 --> 00:15:52.320 Data extraction is really only half the battle. If you 330 00:15:52.360 --> 00:15:54.919 can't persist it reliably, you can't analyze it. You can't 331 00:15:54.960 --> 00:15:59.480 track historical trends or feed it into downstream applications. Which 332 00:15:59.519 --> 00:16:02.799 brings us to updating our spider's logic to actually interact 333 00:16:02.840 --> 00:16:03.679 with the filesystem. 334 00:16:03.879 --> 00:16:06.559 So to move this into production, we modify our spider. 335 00:16:06.720 --> 00:16:09.320 Let's say we create a second spider in the same project, 336 00:16:09.360 --> 00:16:13.240 maybe call it Actualcompanydetails dot PI. Instead of a command 337 00:16:13.320 --> 00:16:15.279 to print to the screen, we define a file name 338 00:16:15.360 --> 00:16:18.080 variable like a simple text file. We instruct Python to 339 00:16:18.120 --> 00:16:20.679 open that text file on our local hard drive, but 340 00:16:20.759 --> 00:16:23.120 we don't just open it normally. We open it in 341 00:16:23.120 --> 00:16:23.919 a pen mode. 342 00:16:24.200 --> 00:16:27.399 And that distinction is vital when automating data. 343 00:16:27.120 --> 00:16:29.200 Collection because of overwriting. 344 00:16:29.440 --> 00:16:32.559 Exactly, if you open a file in standard write mode, 345 00:16:32.840 --> 00:16:37.639 the system completely overwrites whatever was there previously. Every single 346 00:16:37.639 --> 00:16:40.399 time the script runs, you'd only ever have the most 347 00:16:40.399 --> 00:16:41.960 recent data point. 348 00:16:41.679 --> 00:16:44.360 Which defeats the purpose of daily tracking right. 349 00:16:44.480 --> 00:16:47.200 By forcing a pen mode, you ensure that every new 350 00:16:47.240 --> 00:16:50.480 scrape safely adds the latest companies and prices to the 351 00:16:50.600 --> 00:16:53.600 very bottom of the file. You are building a continuous 352 00:16:53.840 --> 00:16:57.519 historical ledger rather than just erasing and rewriting the same 353 00:16:57.559 --> 00:16:58.279 single page. 354 00:16:58.360 --> 00:17:01.679 So inside the parsing logic, as our for loop processes 355 00:17:01.759 --> 00:17:05.279 the synchronized lists of data, we write each extracted pair 356 00:17:05.440 --> 00:17:08.960 directly into our persistent text document. We save the Python file, 357 00:17:09.119 --> 00:17:11.960 go back to the terminal and run the crawl command again, and. 358 00:17:11.960 --> 00:17:14.839 This time the terminal output is much quieter. You won't 359 00:17:14.839 --> 00:17:18.279 see the stock prices flying across the screen, which can 360 00:17:18.319 --> 00:17:20.200 momentarily make you think this spider failed. 361 00:17:20.480 --> 00:17:22.039 I've definitely had that panic moment. 362 00:17:22.160 --> 00:17:23.400 Oh, we all have, but. 363 00:17:23.359 --> 00:17:25.960 It's actually just working silently in the background. When the 364 00:17:25.960 --> 00:17:28.440 crawl finishes, you can use the ills command in your 365 00:17:28.559 --> 00:17:31.720 terminal to check your project directory, and sitting right there 366 00:17:31.799 --> 00:17:34.519 is a brand new file and if you use the 367 00:17:34.599 --> 00:17:37.599 less command to open it up, you'll see your perfectly 368 00:17:37.640 --> 00:17:40.119 formative stock data inside. 369 00:17:40.240 --> 00:17:44.559 We have successfully automated the collection and persistence of the data, 370 00:17:44.599 --> 00:17:47.119 which is amazing if we connect this to the bigger picture. 371 00:17:48.000 --> 00:17:51.079 Writing to a text file is just the absolute simplest 372 00:17:51.119 --> 00:17:54.960 form of what scrapee calls an item pipeline. When you 373 00:17:55.000 --> 00:17:58.039 scale this operation up, you aren't storing data in flat 374 00:17:58.079 --> 00:18:01.240 text files on a laptop. Those scraped items are passed 375 00:18:01.240 --> 00:18:04.279 from the spider into a gauntlet of complex processors. 376 00:18:04.519 --> 00:18:07.440 Because raw data pulled from the Internet is rarely clean 377 00:18:07.559 --> 00:18:10.119 enough to use immediately right seldom. 378 00:18:09.759 --> 00:18:12.119 If ever, I mean, the Internet is messy. You might 379 00:18:12.160 --> 00:18:14.599 pull a stock price, but the text string includes a 380 00:18:14.599 --> 00:18:17.319 currency symbol, a comma, and maybe some trailing white spaces. 381 00:18:17.680 --> 00:18:20.720 If you dump that rawstring into a database, it corrupts 382 00:18:20.720 --> 00:18:24.240 your downstream analytics because you can't perform mathematical operations on text. 383 00:18:24.799 --> 00:18:27.839 The item pipeline is where you deploy data sanitization. 384 00:18:28.279 --> 00:18:31.119 Ah. So, going back to our kitchen analogy, this is 385 00:18:31.119 --> 00:18:33.480 the packaging department, ensuring the meal is perfect before it 386 00:18:33.559 --> 00:18:34.759 leaves the building exactly. 387 00:18:34.839 --> 00:18:38.519 The processors strip out the weird formatting, cast the text 388 00:18:38.559 --> 00:18:42.839 strings into actual floating point numbers, check from missing critical values, 389 00:18:43.079 --> 00:18:46.480 and flag anomalies. Once the data is scrubbed clean, the 390 00:18:46.519 --> 00:18:51.559 pipeline automatically pushes it directly into massive SQL databases, data lakes, 391 00:18:51.920 --> 00:18:53.160 or cloud storage erase. 392 00:18:53.400 --> 00:18:57.079 It makes the data instantly available and reliable for analytics 393 00:18:57.160 --> 00:18:58.960 teams or machine learning models. 394 00:18:59.079 --> 00:19:00.000 That's the enterprise. 395 00:19:00.319 --> 00:19:03.319 That is the true power of automation. We took a 396 00:19:03.359 --> 00:19:07.480 process that would take hours of agonizing, mind numbing manual 397 00:19:07.559 --> 00:19:11.000 labor and reduced it to a highly structured framework that 398 00:19:11.119 --> 00:19:14.720 runs in seconds, cleans the data, and stores it permanently. 399 00:19:14.920 --> 00:19:16.519 It's a game changer, it really is. 400 00:19:16.720 --> 00:19:18.559 So what does this all mean for you as you 401 00:19:18.680 --> 00:19:21.759