WEBVTT 1 00:00:00.120 --> 00:00:05.280 You've shared some incredibly relatable notes with us lately about 2 00:00:05.280 --> 00:00:09.720 the absolute headache that is raw webscraping, Like you know, 3 00:00:09.800 --> 00:00:11.839 the exact feeling we were talking about here. 4 00:00:11.880 --> 00:00:14.279 Oh yeah, completely. It's a universal pain point. 5 00:00:14.400 --> 00:00:14.560 Right. 6 00:00:14.759 --> 00:00:16.879 You write a script, You ask your code to go 7 00:00:16.920 --> 00:00:20.120 out and fetch data maybe I don't know prices, book titles, 8 00:00:20.160 --> 00:00:23.039 contact information, and instead of giving you a neat spreadsheet, 9 00:00:23.399 --> 00:00:26.879 your script basically kicks open your front door and dumps 10 00:00:27.000 --> 00:00:32.920 a massive, unorganized, completely unstructured pile of raw HTML tags 11 00:00:33.439 --> 00:00:34.159 right in the middle of. 12 00:00:34.159 --> 00:00:38.079 Your floor, which is exactly the painful reality of data 13 00:00:38.119 --> 00:00:40.320 extraction when you first start out as a developer. I mean, 14 00:00:40.320 --> 00:00:42.320 you successfully get the data, which is great, but it 15 00:00:42.399 --> 00:00:46.039 is just buried under weird formatting, Unicode errors and broken 16 00:00:46.079 --> 00:00:46.719 relative lengths. 17 00:00:46.799 --> 00:00:48.039 It's a total mess, exactly. 18 00:00:48.039 --> 00:00:50.240 It's practically unusable in that raw state. 19 00:00:50.079 --> 00:00:54.159 Precisely, and that is the mission for our deep dive today. 20 00:00:54.399 --> 00:00:58.399 We are going fundamentally beyond just fetching pages. We are 21 00:00:58.479 --> 00:01:03.359 architecting a clean, highly efficient, basically enterprise grade data extraction machine. 22 00:01:03.439 --> 00:01:04.359 I love the sound of that. 23 00:01:04.799 --> 00:01:06.599 We are going to walk through this step by step 24 00:01:06.680 --> 00:01:10.920 logic of taking that raw chaotic web data and transforming 25 00:01:11.000 --> 00:01:15.359 it into polished, validated information using built in framework tools. 26 00:01:15.719 --> 00:01:17.239 Because let's be real. 27 00:01:17.319 --> 00:01:20.760 Nobody wants a pile of digital garbage crashing their database. 28 00:01:20.959 --> 00:01:24.680 What's fascinating here is the necessary shift in a developer's 29 00:01:24.719 --> 00:01:29.719 mindset because scraping isn't just about grabbing data. Frankly, anyone 30 00:01:29.760 --> 00:01:31.599 can write a script to grab data, right. 31 00:01:31.640 --> 00:01:33.400 The grabbing part is easy, exactly. 32 00:01:33.719 --> 00:01:36.840 Professional level scraping is entirely about how you structure that 33 00:01:36.920 --> 00:01:40.200 data efficiently while it is in transit. So today we 34 00:01:40.280 --> 00:01:43.280 are really analyzing the journey that data takes. Okay, we 35 00:01:43.319 --> 00:01:45.920 will explore the mechanisms to collect it cleanly at the 36 00:01:46.000 --> 00:01:48.959 source and then crucially, how to route it through an 37 00:01:49.000 --> 00:01:52.239 automated sequential assembly line for processing and validation. 38 00:01:52.799 --> 00:01:55.439 Let's dive right into that source collection part, because before 39 00:01:55.439 --> 00:01:58.799 we can process any information at all, we need a reliable, 40 00:01:58.959 --> 00:02:01.879 scalable way to pick the actual items out of that 41 00:02:01.959 --> 00:02:04.760 missy pile of markup and the materials you've shared with us. 42 00:02:04.760 --> 00:02:07.280 Introduce a well a highly specific tool for this called 43 00:02:07.280 --> 00:02:08.120 an item loader. 44 00:02:08.479 --> 00:02:13.080 Yes, item loaders. In software engineering, we love encapsulation, right, 45 00:02:13.120 --> 00:02:18.120 and item loaders are these delightfully convenient mechanisms designed specifically 46 00:02:18.199 --> 00:02:20.120 to populate data objects. 47 00:02:20.360 --> 00:02:22.199 Okay, can you give me an example of how that looks. 48 00:02:22.240 --> 00:02:25.960 Sure, Let's imagine you are scraping a bookstore. You have 49 00:02:26.000 --> 00:02:28.639 a digital blueprint defined in your code for what a 50 00:02:28.639 --> 00:02:31.159 book should look like. That's your book item class, right, 51 00:02:31.199 --> 00:02:34.319 the template exactly. The item loader is the specialized tool 52 00:02:34.360 --> 00:02:37.879 you use to safely inject the actual extracted data into 53 00:02:37.879 --> 00:02:38.960 that book object. 54 00:02:39.240 --> 00:02:43.039 Tracking with you so far, But how does that logic 55 00:02:43.080 --> 00:02:45.919 actually execute in the code like step by step? 56 00:02:46.000 --> 00:02:48.879 Well, it begins with initialization at the top of your script. 57 00:02:48.960 --> 00:02:52.319 You import the item loader class. Then, when your spider 58 00:02:52.360 --> 00:02:55.919 hits a web page, you instantiate an item loader object okay, 59 00:02:55.960 --> 00:02:59.159 and you pass it to critical arguments. First the specific 60 00:02:59.199 --> 00:03:01.800 class you want to build, which is your book item. Second, 61 00:03:01.919 --> 00:03:05.080 you hand it the raw response data that your spider. 62 00:03:04.759 --> 00:03:07.199 Just downloaded, which is that messy pile of code on 63 00:03:07.240 --> 00:03:07.759 our floor. 64 00:03:08.000 --> 00:03:10.039 Precisely, that response. 65 00:03:09.719 --> 00:03:12.400 Is the pile. Okay, let's unpack this. So the loader 66 00:03:12.479 --> 00:03:15.800 is currently holding an empty box the book object, and 67 00:03:15.879 --> 00:03:19.280 it is looking at the pile of raw web data. 68 00:03:19.319 --> 00:03:21.719 But it needs a mechanism to actually move the data 69 00:03:21.759 --> 00:03:22.759 into the box. 70 00:03:22.599 --> 00:03:26.319 Right, Yes, exactly, And for every single field you want 71 00:03:26.360 --> 00:03:29.639 to populate, say the title, the author, or the price, 72 00:03:30.080 --> 00:03:33.240 you use specific extraction methods built right into the loader. 73 00:03:33.400 --> 00:03:35.719 Like what the most common ones you will use are 74 00:03:35.960 --> 00:03:38.759 ad XPath, add serial tip, or add value. 75 00:03:39.000 --> 00:03:39.479 Oh okay. 76 00:03:39.639 --> 00:03:43.439 Think of XPath and CSS selectors as like highly specific 77 00:03:43.520 --> 00:03:48.039 GPS coordinates on a web page's document object model or down. 78 00:03:48.199 --> 00:03:49.919 I like the GPS coordinates. 79 00:03:50.039 --> 00:03:53.400 Yeah, and these loader methods act as targeted grabbers. They 80 00:03:53.439 --> 00:03:56.280 reach into the raw HTML, pull out the exact string 81 00:03:56.360 --> 00:03:58.919 value at those coordinates, and stage it for the book object. 82 00:03:59.080 --> 00:04:00.680 I think I have a good announology for this that 83 00:04:00.719 --> 00:04:03.479 builds on the problem of the messy pile. Is an 84 00:04:03.479 --> 00:04:07.439 item loader essentially like sending a very literal, highly trained 85 00:04:07.439 --> 00:04:09.879 personal shopper into a massive grocery store. 86 00:04:09.639 --> 00:04:10.280 A game listening. 87 00:04:10.439 --> 00:04:12.840 You have them a specific list that is your item 88 00:04:12.879 --> 00:04:16.959 class the book object, and they know exactly which aisles 89 00:04:16.959 --> 00:04:19.360 and shelves to visit. Those are the XPath or CSS 90 00:04:19.360 --> 00:04:22.800 selectors to grab the exact raw goods you asked for, 91 00:04:23.319 --> 00:04:25.720 completely ignoring all the other noise in the store. 92 00:04:25.839 --> 00:04:28.800 That is a brilliant way to visualize it. Actually, the 93 00:04:28.879 --> 00:04:32.839 personal shopper doesn't get distracted by the promotional displays or 94 00:04:32.879 --> 00:04:35.639 the layout of the store. They just navigate directly to 95 00:04:35.680 --> 00:04:38.800 the precise coordinates you provided, grab the target item, and 96 00:04:38.839 --> 00:04:39.839 place it in their cart. 97 00:04:39.959 --> 00:04:41.920 But you know, as a developer, I have to push 98 00:04:42.000 --> 00:04:44.319 back a little here. Sure, why do we need to 99 00:04:44.360 --> 00:04:49.480 instantiate this specific loader object to act as the personal shopper? 100 00:04:49.800 --> 00:04:52.240 I mean, if I already know the xpass coordinates, why 101 00:04:52.279 --> 00:04:55.519 can't I just write a basic Python dictionary, grab the 102 00:04:55.600 --> 00:04:57.639 data manually in my code, and assign. 103 00:04:57.360 --> 00:04:59.600 It to a variable like a standard key value pair. 104 00:04:59.560 --> 00:05:02.800 Exactly, something like item title equals response dot x path 105 00:05:02.879 --> 00:05:04.800 and so on. It just feels like we are adding 106 00:05:04.839 --> 00:05:07.759 an extra layer of abstraction for absolutely no reason. 107 00:05:07.839 --> 00:05:10.360 It is a totally fair question, and honestly, it is 108 00:05:10.399 --> 00:05:12.879 a trap a lot of self taught developers fall into. 109 00:05:13.079 --> 00:05:16.079 Oh really, yeah, The reason you use an item loader 110 00:05:16.199 --> 00:05:19.920 is separation of concerns. When you grab data manually and 111 00:05:19.959 --> 00:05:23.560 stuff it into a dictionary, your core spider code gets 112 00:05:23.680 --> 00:05:25.519 extremely messy very. 113 00:05:25.360 --> 00:05:27.079 Quickly, because it's doing too much at once. 114 00:05:27.279 --> 00:05:31.040 Exactly, you are mixing the physical act of extracting data 115 00:05:31.040 --> 00:05:34.040 from the dom with the logic of parsing and structuring it. 116 00:05:34.639 --> 00:05:38.240 If a website changes its layout, your extraction logic and 117 00:05:38.279 --> 00:05:41.040 your structuring logic are completely tangled together. 118 00:05:41.319 --> 00:05:44.680 Ah, I see the problem. If I have fifty fields 119 00:05:44.720 --> 00:05:49.319 to scrape, a manual dictionary approach turns my spider into 120 00:05:49.360 --> 00:05:51.680 a massive, unreadable. 121 00:05:51.079 --> 00:05:51.920 Wall of code. 122 00:05:52.040 --> 00:05:56.279 Precisely, and it's a nightmare to maintain. The loader neatly 123 00:05:56.399 --> 00:06:00.000 encapsulates the extraction process. It holds the raw data secure 124 00:06:00.079 --> 00:06:02.120 surely so that it can be cleanly handed off to 125 00:06:02.160 --> 00:06:02.879 the next stage. 126 00:06:02.959 --> 00:06:03.879 That makes a lot of sense. 127 00:06:03.920 --> 00:06:06.959 And here is the mechanical kicker. The whole loading process 128 00:06:07.040 --> 00:06:09.720 officially concludes when you call a built in method called 129 00:06:09.800 --> 00:06:10.319 load item. 130 00:06:10.439 --> 00:06:12.519 Okay, so, if the loader is the shopper holding the 131 00:06:12.560 --> 00:06:14.879 box and we've mapped out the coordinates and grab the goods, 132 00:06:15.040 --> 00:06:16.959 there has to be a trigger that actually seals the 133 00:06:16.959 --> 00:06:19.480 box and says, hey, this item is finished. Because I'm 134 00:06:19.519 --> 00:06:22.560 assuming we can't just leave the data floating in the 135 00:06:22.560 --> 00:06:26.079 loader's memory forever. Load item must be that trigger. 136 00:06:26.160 --> 00:06:28.839 You hit the nail on the head. Load itom packages 137 00:06:28.879 --> 00:06:31.600 the populated object and hands it back to the framework's 138 00:06:31.639 --> 00:06:35.399 core engine. It essentially takes the finished box out of 139 00:06:35.439 --> 00:06:38.720 the shopper's hands and places it onto a conveyor belt. 140 00:06:38.759 --> 00:06:40.079 A conveyor belt, okay. 141 00:06:39.839 --> 00:06:43.439 Yes, and it also serves as the catalyst that triggers 142 00:06:43.480 --> 00:06:47.759 any pre processing or post processing logic you might have configured. 143 00:06:48.319 --> 00:06:51.519 If you were building this manually with basic Python dictionaries, 144 00:06:51.560 --> 00:06:54.519 you'd have to write all that handoff and processing logic 145 00:06:54.560 --> 00:06:57.360 from scratch every single time. 146 00:06:57.519 --> 00:07:00.439 Which brings us perfectly to the next major problem. Our 147 00:07:00.519 --> 00:07:02.800 personal shopper sealed the box and put it on the 148 00:07:02.839 --> 00:07:06.639 cavera belt, But if we look inside that box, the 149 00:07:06.720 --> 00:07:10.240 raw materials they gathered are probably still flawed. Oh all, 150 00:07:10.639 --> 00:07:12.920 Like a price might have a pound symbol or dollar 151 00:07:13.000 --> 00:07:14.720 sign stuck to it, or a string of text might 152 00:07:14.759 --> 00:07:17.720 have a bunch of weird whitespace characters from the HTML formatting. 153 00:07:18.000 --> 00:07:21.399 If I'm building a database for bargain hunters, saving a 154 00:07:21.439 --> 00:07:26.199 string like pound fifty five point zero zero is completely useless. 155 00:07:26.240 --> 00:07:29.680 Because a computer program cannot execute math on a currency symbol. 156 00:07:30.399 --> 00:07:33.120 We need a way to intercept and mutate that data 157 00:07:33.240 --> 00:07:34.759 in transit exactly. 158 00:07:34.839 --> 00:07:39.040 Structure does not equal cleanliness. The raw materials have been gathered, 159 00:07:39.160 --> 00:07:42.399 but they are absolutely not a finished product ready for 160 00:07:42.439 --> 00:07:46.319 a database. They need to be refined, checked, and validated 161 00:07:46.399 --> 00:07:48.800 in a very specific automated sequence. 162 00:07:48.879 --> 00:07:49.519 So how do we do that? 163 00:07:49.720 --> 00:07:51.000 Enter the item pipeline. 164 00:07:51.040 --> 00:07:54.720 Item pipelines. That implies a literal factory assembly line. 165 00:07:54.519 --> 00:07:58.240 And that is exactly how they function mechanically. Item pipelines 166 00:07:58.279 --> 00:08:01.879 are Python classes that the framework uses to chain together 167 00:08:02.040 --> 00:08:06.279 various transformations, operations, and checks on your raw scraped data. 168 00:08:06.319 --> 00:08:07.639 So it's an automated sequence. 169 00:08:07.759 --> 00:08:11.600 Exactly. Once LOADEDUM yields the data, the framework's engine automatically 170 00:08:11.680 --> 00:08:14.959 routes that item through these pipeline classes, executing them one 171 00:08:14.959 --> 00:08:15.439 after the other. 172 00:08:15.680 --> 00:08:17.439 Let's dig into the mechanics of that. What are we 173 00:08:17.439 --> 00:08:19.759 actually doing to the data inside these pipelines? Give me 174 00:08:19.839 --> 00:08:21.079 some real world use cases. 175 00:08:21.319 --> 00:08:25.720 They are essential for almost every professional scraping scenario. First 176 00:08:25.839 --> 00:08:29.000 is simply cleaning data. You might need a pipeline to 177 00:08:29.040 --> 00:08:32.919 format results, strip out unneeded HTML characters, or remove that 178 00:08:32.960 --> 00:08:35.159 currency symbol you mentioned, so you just have a pure 179 00:08:35.360 --> 00:08:37.200 integer or float to do mathon. 180 00:08:37.440 --> 00:08:40.240 That makes total sense. We need clean data types. 181 00:08:40.519 --> 00:08:40.960 What else? 182 00:08:41.279 --> 00:08:44.919 Validating data is massive. You need a pipeline step to 183 00:08:45.000 --> 00:08:48.000 ensure that the prices you've scraped are actually numerical values 184 00:08:48.080 --> 00:08:49.759 and not just a string of text that says out 185 00:08:49.799 --> 00:08:50.159 of stock. 186 00:08:50.200 --> 00:08:51.600 Well, I hate when that happens, right. 187 00:08:52.120 --> 00:08:56.080 You also use pipelines to check that URLs are structurally valid, 188 00:08:56.559 --> 00:08:59.440 or to verify with a regular expression that something claiming 189 00:08:59.440 --> 00:09:02.399 to be an email address actually contains an at symbol 190 00:09:02.480 --> 00:09:03.159 and a domain. 191 00:09:03.679 --> 00:09:05.759 So it's strict quality control it is. 192 00:09:06.080 --> 00:09:08.840 You also use pipelines to ensure that crucial items actually 193 00:09:08.840 --> 00:09:11.799 contain values and didn't just come back as empty null values. 194 00:09:12.360 --> 00:09:14.919 You can write logic to cross reference data and remove 195 00:09:15.000 --> 00:09:15.919 duplicate entries. 196 00:09:16.000 --> 00:09:16.320 Okay. 197 00:09:16.360 --> 00:09:20.799 Well, and finally, storing the final polished items into external files, 198 00:09:20.840 --> 00:09:24.639 sql databases or cloud storage is the classic final step 199 00:09:24.679 --> 00:09:25.440 of a pipeline. 200 00:09:25.559 --> 00:09:28.840 Let's ground this with a highly specific example. Let's say 201 00:09:28.840 --> 00:09:31.519 we are scraping products. We have our item loader, and 202 00:09:31.559 --> 00:09:33.159 we need to deal with pricing anomalies. 203 00:09:33.240 --> 00:09:36.120 Okay, Let's imagine we build a class called price pipeline. 204 00:09:36.559 --> 00:09:40.519 Every pipeline class requires a specific method called process item. 205 00:09:40.919 --> 00:09:44.399 As the framework routes your data, it passes the item 206 00:09:44.480 --> 00:09:45.399 through this method. 207 00:09:45.519 --> 00:09:47.039 Okay, process item. Got it. 208 00:09:47.440 --> 00:09:50.720 Let's say the website we are scraping lists prices before 209 00:09:50.759 --> 00:09:54.080 a mandatory sales tax inside process item, you can write 210 00:09:54.080 --> 00:09:57.240 a simple rule look at the items price field and 211 00:09:57.320 --> 00:09:59.919 if it doesn't include the sales tax, multiply the float 212 00:10:00.080 --> 00:10:02.600 value by the tax rate to calculate the true total, 213 00:10:02.919 --> 00:10:05.480 and then return the updated item to the conveyor belt. 214 00:10:05.720 --> 00:10:08.360 But wait, what if the personal shop or grabbed a 215 00:10:08.399 --> 00:10:12.399 product and because the target website had a temporary rendering glitch, 216 00:10:12.879 --> 00:10:16.399 the price field is just well empty. It happens a lot, right, 217 00:10:16.679 --> 00:10:18.759 And if you try to multiply a null value or 218 00:10:18.759 --> 00:10:21.279 an empty string by a tax rate, Python is going 219 00:10:21.360 --> 00:10:23.840 to throw a type AAR and your entire SCRAPEG program 220 00:10:23.960 --> 00:10:24.559 is going to crash. 221 00:10:24.600 --> 00:10:27.080 Right, It absolutely would crash, which is why the pipeline 222 00:10:27.159 --> 00:10:30.039 architecture has a built in error handling mechanism called the 223 00:10:30.120 --> 00:10:31.120 drop item exception. 224 00:10:31.559 --> 00:10:34.919 Drop item. Yes, inside your price pipeline, you write a 225 00:10:34.960 --> 00:10:38.840 defensive condition. If an item arrives completely missing a mandatory 226 00:10:38.879 --> 00:10:41.720 price field, you use the command raised drop item. 227 00:10:42.000 --> 00:10:45.360 What does raising that exception actually do mechanically? Does the 228 00:10:45.399 --> 00:10:47.080 program just stop entirely? 229 00:10:47.399 --> 00:10:50.240 No, And that is the beauty of it. The framework 230 00:10:50.360 --> 00:10:55.039 catches that specific exception, it completely halts that single item's 231 00:10:55.120 --> 00:10:56.559 progress down the pipeline. 232 00:10:56.600 --> 00:10:58.320 Oh, just the one item exactly. 233 00:10:58.759 --> 00:11:01.240 It pulls that defective item right off the conveyor belt, 234 00:11:01.440 --> 00:11:04.159 logs it, and discards it so it doesn't crash the 235 00:11:04.200 --> 00:11:07.200 next worker down the line. Meanwhile, all the other healthy 236 00:11:07.240 --> 00:11:08.960 items keep slowing without interruption. 237 00:11:09.440 --> 00:11:13.039 Here's where it gets really interesting. If we extend our metaphor, 238 00:11:13.480 --> 00:11:16.639 the pipeline is the factory floor. Yes, the shopper dumps 239 00:11:16.679 --> 00:11:19.279 the raw materials at the start of the belt as 240 00:11:19.320 --> 00:11:22.720 the item moves down the line. Each worker, meaning each 241 00:11:23.039 --> 00:11:26.320 process item method, has one hyper specific job. 242 00:11:26.720 --> 00:11:30.000 One worker cleans the text, the next worker verifies it's 243 00:11:30.000 --> 00:11:33.519 a real number. The next worker applies the sales tax calculation. 244 00:11:34.120 --> 00:11:37.080 And if any worker detects a critically defective item, they 245 00:11:37.159 --> 00:11:40.159 hit a button that raises the drop item exception, basically 246 00:11:40.279 --> 00:11:42.600 dropping it through a trapdoor before it can ruin the 247 00:11:42.639 --> 00:11:43.279 final shipment. 248 00:11:43.480 --> 00:11:46.799 That is the exact architectural flow you should be visualizing. 249 00:11:46.919 --> 00:11:50.240 It is a sequence of highly specialized, isolated workers. 250 00:11:50.480 --> 00:11:54.080 To truly understand how this machine is constructed, we need 251 00:11:54.120 --> 00:11:56.480 to see it run end to end. Let's trace a 252 00:11:56.559 --> 00:11:59.240 practical example, step by step. Let's talk about scraping a 253 00:11:59.279 --> 00:12:00.080 sandbox site. 254 00:12:00.240 --> 00:12:04.519 A great exercise. Let's use books dot to scrape dot com. Okay, 255 00:12:04.559 --> 00:12:08.720 it's a classic sandbox environment structured into different catalog categories. 256 00:12:09.080 --> 00:12:12.120 It is essentially a mock online bookstore, which makes it 257 00:12:12.240 --> 00:12:15.480 the perfect legal playground to test these extraction tools. 258 00:12:15.639 --> 00:12:16.840 Let's look at the before picture. 259 00:12:17.440 --> 00:12:20.799 Before developer learns about loaders and pipelines, how do they 260 00:12:20.840 --> 00:12:24.120 traditionally try to scrape this bookstore. Let's talk about the 261 00:12:24.159 --> 00:12:25.399 pain of the manual approach. 262 00:12:25.720 --> 00:12:29.080 Oh, the manual approach, Well, you write your basic spider logic, 263 00:12:29.360 --> 00:12:31.360 you set it up to crawl through the catalog pages. 264 00:12:31.799 --> 00:12:34.759 You write your expath selectors to target the book's title, 265 00:12:34.879 --> 00:12:38.279 the price, the cover image URL, and the actual product 266 00:12:38.279 --> 00:12:39.000 page URL. 267 00:12:39.200 --> 00:12:41.799 That sounds like standard targeting, but the pain starts where 268 00:12:41.799 --> 00:12:42.159 you try to. 269 00:12:42.120 --> 00:12:46.480 Save it exactly. To save that data manually, developers typically 270 00:12:46.519 --> 00:12:49.840 set up a four loop to iterate over the extracted elements, 271 00:12:50.240 --> 00:12:53.039 and inside this loop, if they aren't using a framework's 272 00:12:53.080 --> 00:12:56.279 built in export tools, they will often import a Python 273 00:12:56.360 --> 00:12:58.960 library called codex codex. 274 00:12:59.120 --> 00:13:01.879 If you're doing this mannually, you're probably wrestling with Codex 275 00:13:01.879 --> 00:13:02.799 to handle file writing. 276 00:13:03.000 --> 00:13:07.480 Yes, Codex is used for encoding and decoding data, specifically 277 00:13:07.480 --> 00:13:10.840 to handle things like UTF eight character encoding when writing 278 00:13:10.879 --> 00:13:14.000 text files safely. Right, you would manually use codex dot 279 00:13:14.039 --> 00:13:17.639 open to create a file called books dot txt or 280 00:13:17.679 --> 00:13:21.759 books dot csv. Then, line by line inside your loop, 281 00:13:21.799 --> 00:13:24.440 you command your script to write the extracted title, price, 282 00:13:24.480 --> 00:13:27.600 and URLs directly into that file, usually combining them into 283 00:13:27.600 --> 00:13:28.879 a comma separated string. 284 00:13:29.120 --> 00:13:32.360 And the second a book title actually contains a comma 285 00:13:32.399 --> 00:13:35.799 in its text, your entire CSV structure breaks because you 286 00:13:35.879 --> 00:13:37.360 just added an accidental column. 287 00:13:37.480 --> 00:13:39.279 Yes, so frustrating. 288 00:13:39.440 --> 00:13:43.399 Plus you are manually hard coding filepaths, opening file loocks, 289 00:13:43.679 --> 00:13:45.879 and just hoping the script doesn't crash half way through 290 00:13:45.879 --> 00:13:46.840 and corrupt the whole file. 291 00:13:47.000 --> 00:13:49.360 It is incredibly brittle. I mean it works for a 292 00:13:49.399 --> 00:13:52.159 ten item test, sure, but it does not scale to 293 00:13:52.200 --> 00:13:54.639 ten thousand items. If you want to change how the 294 00:13:54.679 --> 00:13:57.679 data is saved or add a step to clean the data, 295 00:13:58.039 --> 00:14:00.159 you have to rewrite your core spider code. 296 00:14:00.399 --> 00:14:02.399 Okay, so let's look at the after picture. How do 297 00:14:02.440 --> 00:14:06.559 we upgrade this manual, brittle clunker into a high performance 298 00:14:06.600 --> 00:14:08.679 machine using the framework tools. 299 00:14:09.080 --> 00:14:12.320 First, we completely delete the manual text file writing. We 300 00:14:12.440 --> 00:14:15.039 throw out the codex logic and the four loops entirely. 301 00:14:15.279 --> 00:14:16.200 Goodbye codex. 302 00:14:16.279 --> 00:14:19.399 Instead, we bring in our personal shopper. We instantiate the 303 00:14:19.399 --> 00:14:21.759 item loader and pass it the selector for the current 304 00:14:21.759 --> 00:14:22.399 book element. 305 00:14:22.559 --> 00:14:23.840 Okay, so the shopper's ready. 306 00:14:24.039 --> 00:14:28.600 Now we introduce a powerful new concept input processors, specifically 307 00:14:28.759 --> 00:14:31.240 a built in processor called map compose. 308 00:14:31.440 --> 00:14:34.200 The map compose what exactly is map composed doing under 309 00:14:34.200 --> 00:14:34.519 the hood. 310 00:14:34.679 --> 00:14:37.360 So, map compose is a processor that allows you to 311 00:14:37.440 --> 00:14:40.480 apply a sequence of functions to your extracted data immediately, 312 00:14:40.559 --> 00:14:43.279 like before it even fully settles into the item object. 313 00:14:43.399 --> 00:14:45.000 Wait immediately, Yes. 314 00:14:45.600 --> 00:14:47.960 Let's look at the URLs we scraped from the sandbox 315 00:14:48.000 --> 00:14:50.879 site on books dot to scrape dot com. The image 316 00:14:51.080 --> 00:14:55.200 URLs and the book URLs are incomplete. They are relative links. Okay, 317 00:14:55.279 --> 00:14:59.399 they look like catalog slash book one dot HTML instead 318 00:14:59.399 --> 00:15:00.960 of a full clickable web address. 319 00:15:01.080 --> 00:15:02.000 Right, they're just fragments. 320 00:15:02.000 --> 00:15:03.639 If you try to open them in a browser or 321 00:15:03.720 --> 00:15:05.440 save them to a database, they lead. 322 00:15:05.320 --> 00:15:09.120 Nowhere, precisely. So outside of our spider, we create a 323 00:15:09.159 --> 00:15:13.120 simple custom Python function called adlink. All this function does 324 00:15:13.200 --> 00:15:16.960 is take a string and concatenate or join the base 325 00:15:17.320 --> 00:15:20.200 URL of the website with the fragmented url. We just 326 00:15:20.279 --> 00:15:24.399 passed it returning a fully qualified functioning weblink. 327 00:15:23.919 --> 00:15:26.200 And we plug this custom function into map compost. 328 00:15:26.320 --> 00:15:29.039 Yes. When we configure our item loader, we tell it 329 00:15:29.080 --> 00:15:31.840 that whenever it grabs a raw URL fragment via XPath, 330 00:15:32.159 --> 00:15:35.279 it must immediately pass that fragment through the map composed processor, 331 00:15:35.440 --> 00:15:37.759 and we pass ad link as the argument. Oh wow, 332 00:15:37.919 --> 00:15:41.480 the processor applies the function, meaning the URL is fixed 333 00:15:41.559 --> 00:15:44.480 and properly formatted before it is ever officially assigned to 334 00:15:44.480 --> 00:15:47.679 our book item. You can even chain them. You could 335 00:15:47.720 --> 00:15:50.840 have map composed, strip white space, then apply ad link 336 00:15:51.200 --> 00:15:52.679 all in one line of code. 337 00:15:52.720 --> 00:15:55.519 That is incredibly slick. But wait, I have a mechanical 338 00:15:55.600 --> 00:15:56.320 question about that. 339 00:15:56.440 --> 00:15:56.919 Go for it. 340 00:15:57.039 --> 00:15:59.960 Why is map composed designed this way? If I target 341 00:16:00.120 --> 00:16:03.440 one single price or one URL on a page, shouldn't 342 00:16:03.480 --> 00:16:05.879 it just return a single string? Why do we need 343 00:16:05.879 --> 00:16:08.080 a special processor to map functions over it? 344 00:16:08.200 --> 00:16:10.840 That is a fantastic catch. It has to do with 345 00:16:10.879 --> 00:16:14.639 how extraction languages work. XPath and CSS selectors are designed 346 00:16:14.679 --> 00:16:17.039 to query the dom and find all matches that fit. 347 00:16:16.879 --> 00:16:18.360 Your coordinates, right, all of them. 348 00:16:18.679 --> 00:16:20.360 Even if you know there is only one price on 349 00:16:20.399 --> 00:16:23.440 the page. The underlying framework engine doesn't know that so 350 00:16:23.559 --> 00:16:26.519 to be safe, it always returns extracted data as a 351 00:16:26.559 --> 00:16:30.200 Python list an iterable array. It returns a list containing 352 00:16:30.240 --> 00:16:31.000 your one item. 353 00:16:31.240 --> 00:16:34.159 Ah, so the data is trapped inside a list format 354 00:16:34.559 --> 00:16:38.039 like bracket quote fifty pounds end quote bracket instead of 355 00:16:38.080 --> 00:16:39.200 just the string fifty. 356 00:16:38.919 --> 00:16:42.559 Pounds exactly, and mac composed is specifically built to iterate 357 00:16:42.639 --> 00:16:45.759 over lists. It applies your function to every element inside 358 00:16:45.759 --> 00:16:47.360 that list. Okay, that makes sense, so you still have 359 00:16:47.360 --> 00:16:48.919 a list at the end of it. This is why 360 00:16:48.919 --> 00:16:51.879 we must also assign an output processor called take first 361 00:16:51.919 --> 00:16:52.600 to our loader. 362 00:16:52.879 --> 00:16:56.200 Let me guess takes first, looks at the list, grabs 363 00:16:56.240 --> 00:16:59.240 the very first element out of it, and discards the 364 00:16:59.279 --> 00:17:00.679 list's structure entirely. 365 00:17:00.759 --> 00:17:03.559 You've got it. It extracts the clean string value right 366 00:17:03.600 --> 00:17:04.279 out of the array. 367 00:17:04.640 --> 00:17:08.000 So to summarize the immense benefit for you listening, by 368 00:17:08.039 --> 00:17:11.200 deleting our manual codex loop and plugging into map compose 369 00:17:11.279 --> 00:17:14.079 and take first via the item loader, we are leveraging 370 00:17:14.119 --> 00:17:18.279 the frameworks Optimize internal engine exactly. We aren't reinventing the wheel. 371 00:17:18.519 --> 00:17:21.680 It makes your code modular, highly scalable, and the resulting 372 00:17:21.680 --> 00:17:23.759 items are so well structured that you can just tell 373 00:17:23.799 --> 00:17:27.519 the framework to output them directly to ADJSON or CSV 374 00:17:27.640 --> 00:17:31.200 file with a single command line flag, without writing a 375 00:17:31.240 --> 00:17:33.400 single line of file saving code. 376 00:17:33.599 --> 00:17:35.759 It lets the framework handle all the heavy lifting of 377 00:17:35.759 --> 00:17:38.640 state management and file locking, freeing you up to focus 378 00:17:38.799 --> 00:17:40.480 entirely on the data logic. 379 00:17:40.359 --> 00:17:43.279 Which means we can finally move to the advanced logic. 380 00:17:44.079 --> 00:17:46.920 Our URLs are fixed, our strings are pulled out of 381 00:17:46.920 --> 00:17:50.640 their lists, our data is structured. Let's use the pipelines 382 00:17:50.680 --> 00:17:54.400 to execute some actual business rules. Let's return to our 383 00:17:54.440 --> 00:17:58.519 scenario where we're building a tool for bargain hunters. We 384 00:17:58.559 --> 00:18:01.160 only want to find and process affordable books. 385 00:18:01.319 --> 00:18:04.599 A perfect real world application, Let's say we want to 386 00:18:04.599 --> 00:18:07.599 analyze the prices and anything that costs over fifty we 387 00:18:07.640 --> 00:18:10.640 want to flag it as expensive, so our downstream systems 388 00:18:10.720 --> 00:18:12.359 know exactly how to categorize it. 389 00:18:12.480 --> 00:18:14.079 How do we architect that sequence? 390 00:18:14.599 --> 00:18:17.079 First, we rely on the cleaning task we mentioned earlier. 391 00:18:17.400 --> 00:18:19.960 The price we scraped definitely has a pound symbol attached 392 00:18:20.000 --> 00:18:22.759 to it. A computer cannot evaluate if a symbol is 393 00:18:22.799 --> 00:18:25.960 greater than fifty, so we already have an input processor 394 00:18:25.960 --> 00:18:28.200 stripping out the parent symbol and converting the string to 395 00:18:28.240 --> 00:18:29.839 a float. So we have a pure number. 396 00:18:29.960 --> 00:18:31.319 Okay, so the data is prepped. 397 00:18:31.559 --> 00:18:34.400 Next, we set up two entirely separate pipeline classes in 398 00:18:34.440 --> 00:18:37.559 our project to handle our business logics sequentially, and this 399 00:18:37.640 --> 00:18:41.000 sequence is strictly managed in a central configuration file called 400 00:18:41.119 --> 00:18:42.000 settings dot pi. 401 00:18:42.400 --> 00:18:45.720