WEBVTT 1 00:00:00.160 --> 00:00:02.640 Imagine it is, I don't know, like four point five 2 00:00:02.720 --> 00:00:04.559 to five pm on a Friday. 3 00:00:04.240 --> 00:00:06.440 The worst possible time, right. 4 00:00:06.360 --> 00:00:10.119 At the absolute worst, and your boss drops by your 5 00:00:10.119 --> 00:00:14.359 desk and hands you what sounds like an incredibly simple assignment. 6 00:00:14.359 --> 00:00:16.440 They're like, hey, I need you to go to the 7 00:00:16.519 --> 00:00:20.640 Yahoo Finance technology page and just pull the data for 8 00:00:20.879 --> 00:00:23.719 every single tech stock listed in their main table. 9 00:00:23.879 --> 00:00:26.559 It is a quick favor before the weekend, exactly. 10 00:00:26.640 --> 00:00:30.160 And for every single company you need to grab four 11 00:00:30.199 --> 00:00:32.840 specific things. You need the company name, the intra day 12 00:00:32.880 --> 00:00:36.039 stock price, the stock symbol, and then the actual URL 13 00:00:36.079 --> 00:00:37.960 that links to the company's full profile. 14 00:00:38.679 --> 00:00:40.600 I mean, on the surface, that sounds like a five 15 00:00:40.600 --> 00:00:43.479 minute task. You know, you open a spreadsheet, you open 16 00:00:43.520 --> 00:00:45.000 your web browser, and you just get to. 17 00:00:44.960 --> 00:00:47.960 Work, right, But then reality actually sets in because you're 18 00:00:48.039 --> 00:00:50.920 highlighting the first company's name, you're hitting copy, tabbing over 19 00:00:50.920 --> 00:00:53.920 to your spreadsheet, hitting paste. Yeah, then you're tabbing back 20 00:00:54.240 --> 00:00:56.799 highlighting the price, copy tab paste, And you know, doing 21 00:00:56.880 --> 00:00:59.320 that for one company is completely fine, but doing it 22 00:00:59.359 --> 00:01:03.479 for a company, well, it's brutal. You are making hundreds 23 00:01:03.560 --> 00:01:07.719 of highly repetitive, like precision clicks, Your risk starts aching, 24 00:01:07.799 --> 00:01:10.840 your eyes are completely glazing over, and inevitably you start 25 00:01:10.840 --> 00:01:13.959 making mistakes. You paste a price into the symbol column 26 00:01:14.000 --> 00:01:15.000 by accident. 27 00:01:14.719 --> 00:01:16.640 Which throws off the whole sheet exactly. 28 00:01:16.680 --> 00:01:20.120 It is the kind of mind numbing, soul crushing manual 29 00:01:20.280 --> 00:01:23.640 data entry that really makes you question your career choices. 30 00:01:23.799 --> 00:01:27.439 It really does. But you know, humans are just fundamentally 31 00:01:28.040 --> 00:01:33.040 unsuited for repetitive, high precision tasks over long durations. We 32 00:01:33.200 --> 00:01:36.560 just lose focus. Computers however, I mean they thrive on 33 00:01:36.640 --> 00:01:37.680 that exact type of work. 34 00:01:37.719 --> 00:01:39.480 You don't get bored, No, not at all. 35 00:01:39.799 --> 00:01:43.439 And this is the precise friction point where programming becomes 36 00:01:43.640 --> 00:01:45.120 frankly practically. 37 00:01:44.640 --> 00:01:47.400 Magic, which perfectly brings us to our mission for this 38 00:01:47.480 --> 00:01:50.599 deep dive today. We're going to build a mental blueprint 39 00:01:50.719 --> 00:01:54.040 of an automated data extraction pipeline, and we're looking really 40 00:01:54.040 --> 00:01:56.159 closely at a Python framework called screepy. 41 00:01:56.319 --> 00:01:58.480 Oh yeah, Scrapy is amazing for this, right. 42 00:01:58.519 --> 00:02:01.120 It basically serves as the ultimate engine for this kind 43 00:02:01.159 --> 00:02:01.920 of web scraping. 44 00:02:02.280 --> 00:02:03.599 It really is a powerhouse. 45 00:02:03.920 --> 00:02:06.319 And you know, our goal here for everyone listening isn't 46 00:02:06.359 --> 00:02:08.400 just to talk about writing a quick duty script that 47 00:02:08.479 --> 00:02:11.719 haphazardly yanks text off a page and calls. 48 00:02:11.479 --> 00:02:11.879 It a day. 49 00:02:12.000 --> 00:02:13.080 Right, we want to go deeper. 50 00:02:13.520 --> 00:02:14.919 Exactly, we are. 51 00:02:14.840 --> 00:02:20.280 Going to architect a robust, professional grade data pipeline. So 52 00:02:20.319 --> 00:02:23.560 we will start by examining how to test raw code 53 00:02:23.639 --> 00:02:26.960 in an isolated sandbox, figure out how to scale that 54 00:02:27.159 --> 00:02:30.599 up to crawl entire pages, and eventually build a system 55 00:02:30.680 --> 00:02:35.000 that automatically cleans and formats the data before saving it 56 00:02:35.039 --> 00:02:36.439 into a pristine spreadsheet. 57 00:02:36.599 --> 00:02:40.439 Okay, let's unpack this. Because whether you are a developer 58 00:02:40.520 --> 00:02:44.000 prepping for a major data project, or you know, maybe 59 00:02:44.039 --> 00:02:46.400 a self taught code or trying to automate your own life, 60 00:02:46.560 --> 00:02:49.400 or just someone curious about how data actually moves around 61 00:02:49.439 --> 00:02:53.599 the Internet, understanding this architecture gives you a whole new superpower. 62 00:02:53.840 --> 00:02:55.919 So where do we even begin? Because if I'm staring 63 00:02:55.960 --> 00:02:58.960 at that Yahoo Finance page, my first instinct is to just, 64 00:02:59.080 --> 00:03:00.960 I don't know, write a man subscript and hope it 65 00:03:01.000 --> 00:03:01.840 grabs the right number. 66 00:03:01.960 --> 00:03:04.479 Yeah, and plunging straight into writing a massive script is 67 00:03:04.479 --> 00:03:06.120 the fastest way to get completely. 68 00:03:05.840 --> 00:03:07.199 Overwhelmed, I can imagine. 69 00:03:07.319 --> 00:03:10.879 Because before building a factory, you need to test if 70 00:03:10.919 --> 00:03:13.800 the raw materials are even usable. You need a testing 71 00:03:13.800 --> 00:03:15.960 ground to see if the data you want is accessible 72 00:03:15.960 --> 00:03:18.000 in the way you think it is, and in the 73 00:03:18.039 --> 00:03:22.800 scrape ecosystem, this isolated environment is called the scrape shell. 74 00:03:22.919 --> 00:03:26.560 The screepy shell. Okay, so I'm picturing like a digital sandbox. 75 00:03:26.840 --> 00:03:28.000 That's a great way to look at it. 76 00:03:28.080 --> 00:03:30.520 So you boot up this shell in your computer's terminal, 77 00:03:30.879 --> 00:03:33.680 and you feed it the specific web address, like our 78 00:03:33.759 --> 00:03:37.240 Yaku finance page, and it basically goes out, grabs the 79 00:03:37.319 --> 00:03:41.080 underlying HTML of that page and downloads it right into 80 00:03:41.120 --> 00:03:42.199 your sandbox exactly. 81 00:03:42.280 --> 00:03:43.479 It pulls the raw code. 82 00:03:43.560 --> 00:03:46.800 So now you can experiment with extracting the data without 83 00:03:46.840 --> 00:03:50.000 the pressure of running a full automated program that might 84 00:03:50.039 --> 00:03:51.039 break at any second. 85 00:03:51.120 --> 00:03:53.000 Right, the sandbox isolates the problem. 86 00:03:53.039 --> 00:03:53.840 You can just tinker. 87 00:03:54.360 --> 00:03:56.680 But once you have the page data in there, you 88 00:03:56.759 --> 00:03:59.800 need a designated place to actually put the specific fragments 89 00:03:59.840 --> 00:04:04.039 of information you plan to extract. Okay, So the source 90 00:04:04.080 --> 00:04:07.599 material introduces a really foundational concept here called a. 91 00:04:07.520 --> 00:04:11.879 Scrapy item, Okay, an item. The way the documentation describes 92 00:04:11.919 --> 00:04:14.639 this item, it actually reminds me of a bento box. 93 00:04:14.919 --> 00:04:15.560 Oh I like that. 94 00:04:15.719 --> 00:04:18.399 Yeah, because you have this container, but you don't just 95 00:04:18.519 --> 00:04:21.600 dump all your food into one big, messy pile. You 96 00:04:21.680 --> 00:04:26.079 have very specific pre labeled compartments. A compartment for the rice, 97 00:04:26.360 --> 00:04:27.480 a compartment for the fish. 98 00:04:27.600 --> 00:04:28.800 Right, it's structured. 99 00:04:28.600 --> 00:04:32.079 Exactly in Python terms. It's like a highly structured dictionary. 100 00:04:32.639 --> 00:04:35.959 We define our compartments or feels basically up front. So 101 00:04:36.000 --> 00:04:39.040 we create a label for company name, one for inter 102 00:04:39.160 --> 00:04:41.879 day price, one for symbol, and one for link. 103 00:04:42.040 --> 00:04:46.000 The Bento box analogy perfectly captures the strict structural nature 104 00:04:46.040 --> 00:04:49.240 of the item based class. When you first instantiate this 105 00:04:49.319 --> 00:04:52.519 item in scrapy, it exists in memory as a completely 106 00:04:52.560 --> 00:04:56.480 empty shell. The compartments are there, but they contain absolutely nothing. 107 00:04:56.519 --> 00:04:57.879 You're just waiting for data, right. 108 00:04:58.360 --> 00:05:01.079 Filling those compartments requires it to look at the underlined 109 00:05:01.120 --> 00:05:04.160 structure of the website itself and extract the actual data. 110 00:05:04.240 --> 00:05:06.600 Okay, and this is where it gets a little abstract, 111 00:05:06.639 --> 00:05:10.079 I think, but it really is the core mechanism of webscreeping. 112 00:05:10.680 --> 00:05:13.560 How do we actually tell our computer, hey, go look 113 00:05:13.560 --> 00:05:16.319 at this massive wall of code and grab the words 114 00:05:16.519 --> 00:05:17.680 Microsoft Corporation. 115 00:05:18.120 --> 00:05:19.800 Yeah, it's not like you can just point at the screen. 116 00:05:20.160 --> 00:05:23.000 So we use a mechanism called an XPath selector. 117 00:05:22.680 --> 00:05:24.399 An XPath selector right. 118 00:05:24.639 --> 00:05:27.360 To understand why XPath is necessary, we have to look 119 00:05:27.360 --> 00:05:30.439 at how websites are actually built. HTML is not just 120 00:05:30.480 --> 00:05:34.519 a flat text document. It is a complex, nested hierarchy 121 00:05:34.639 --> 00:05:35.879 like a tree exactly. 122 00:05:35.920 --> 00:05:36.839 It's a tree of elements. 123 00:05:37.000 --> 00:05:40.360 You have this massive HTML trunk which branches off into 124 00:05:40.360 --> 00:05:44.399 body tags, which branch off into dip containers, down into tables, rows, 125 00:05:44.600 --> 00:05:48.279 and finally individual data cells. So to a computer, a 126 00:05:48.279 --> 00:05:51.199 webpage looks like a massive set of Russian mesting dolls. 127 00:05:51.319 --> 00:05:54.160 Okay, so an XPath is essentially like a set of 128 00:05:54.240 --> 00:05:57.680 navigation directions through that specific family tree. Yes, you are 129 00:05:57.680 --> 00:05:59.800 telling the scraper, okay, go into the table container of 130 00:06:00.199 --> 00:06:03.759 the rows and look inside the cell at index two, right, and. 131 00:06:03.720 --> 00:06:06.240 You've hit on a really vital detail there with index two. 132 00:06:06.319 --> 00:06:10.000 Oh really, yeah, because in programming we generally start counting from. 133 00:06:10.000 --> 00:06:13.560 Zero ah right, zero based indexing exactly. 134 00:06:13.639 --> 00:06:15.720 So if the company name is in the third column 135 00:06:15.759 --> 00:06:18.279 of the visual tail you see on the screen, it's 136 00:06:18.399 --> 00:06:20.399 index in the actual code is two. 137 00:06:20.680 --> 00:06:21.160 Wow. 138 00:06:21.560 --> 00:06:22.480 Okay, yeah, if. 139 00:06:22.399 --> 00:06:24.920 You tell your scraper to look at index three, you 140 00:06:24.959 --> 00:06:27.600 are going to pull the entirely wrong column of data, 141 00:06:27.920 --> 00:06:30.160 which is a very common trap when people first start 142 00:06:30.199 --> 00:06:31.319 testing in the scrapey shell. 143 00:06:31.360 --> 00:06:34.240 That would be so frustrating. Okay, so I write my 144 00:06:34.279 --> 00:06:37.959 expath directions, I target index two. Does the shell just 145 00:06:38.160 --> 00:06:42.160 like hand me the text Microsoft Corporation right then and there? 146 00:06:42.399 --> 00:06:43.319 Uh No, it does not. 147 00:06:43.480 --> 00:06:44.480 Oh it does. No. 148 00:06:44.680 --> 00:06:48.399 The XPath selector merely returns a match. It basically hands 149 00:06:48.399 --> 00:06:51.399 you a selector object, essentially a pointer that says, hey, 150 00:06:51.439 --> 00:06:52.360 I found the digital. 151 00:06:52.160 --> 00:06:54.920 Nesting doll you asked for, but it hasn't opened it yet. 152 00:06:54.959 --> 00:06:57.800 Exactly the text you actually want is still sealed inside 153 00:06:57.879 --> 00:07:00.480 that doll. So to reach in and extra the human 154 00:07:00.519 --> 00:07:04.000 readable data, you have to execute a specific function on 155 00:07:04.079 --> 00:07:05.040 that select or object. 156 00:07:05.079 --> 00:07:07.480 Okay, looking at the material, it seems we use functions 157 00:07:07.519 --> 00:07:08.439 like get for this. 158 00:07:08.600 --> 00:07:09.959 Yes, get is the one. 159 00:07:10.120 --> 00:07:13.439 So the workflow is basically, use x path to point 160 00:07:13.439 --> 00:07:16.199 at the cell, then used get to physically pull the 161 00:07:16.199 --> 00:07:18.879 string of text out, and once we have that text, 162 00:07:18.879 --> 00:07:21.279 we place it neatly into the company name compartment of 163 00:07:21.319 --> 00:07:22.120 our Bento box. 164 00:07:22.199 --> 00:07:22.680 You got it. 165 00:07:23.519 --> 00:07:27.079 That completes the entire cycle for a single data point. 166 00:07:27.800 --> 00:07:31.759 We've successfully navigated the HTML tree, extracted the raw text, 167 00:07:31.800 --> 00:07:34.839 and structured it. But we did this inside an isolated 168 00:07:34.879 --> 00:07:37.399 sandbox for one single company. 169 00:07:37.600 --> 00:07:40.959 Right, And here's the fatal flaw with staying in the sandbox. 170 00:07:41.279 --> 00:07:43.879 We are still manually pointing it at one company at 171 00:07:43.920 --> 00:07:47.240 a time. If we stop here, we haven't actually automated anything, 172 00:07:47.279 --> 00:07:49.800 have we. We have just changed the interface from copy 173 00:07:49.879 --> 00:07:52.519 pasting in a web browser to typing manual commands in 174 00:07:52.560 --> 00:07:53.040 a terminal. 175 00:07:53.160 --> 00:07:54.720 Exactly. It's the same amount of work. 176 00:07:54.759 --> 00:07:56.839 So to build a factory floor that can do this 177 00:07:56.879 --> 00:07:58.759 one hundred times a second, we need a way to 178 00:07:58.800 --> 00:07:59.600 scale the logic. 179 00:07:59.680 --> 00:08:03.240 Yeah, moving from testing to actual automation requires building a spider. 180 00:08:03.360 --> 00:08:04.399 A spider right. 181 00:08:04.560 --> 00:08:06.839 When you are ready to scale, you create a full 182 00:08:06.879 --> 00:08:11.120 scrapy project. Running a simple setup command generates an entire 183 00:08:11.240 --> 00:08:15.040 architecture of folders and files. It essentially lays the concrete 184 00:08:15.079 --> 00:08:16.079 for your factory floor. 185 00:08:16.399 --> 00:08:19.480 Okay, so inside that new folder structure, we have a 186 00:08:19.519 --> 00:08:23.000 file called items dot pie. I assume this is where 187 00:08:23.000 --> 00:08:26.360 we officially encode the blueprint for our Bento box. Right, 188 00:08:26.439 --> 00:08:29.040 listing out those four specific fields we want to fill. 189 00:08:29.160 --> 00:08:31.399 Yes, you define your item there, but the actual end 190 00:08:31.480 --> 00:08:33.799 of the thing doing the crawling lives in a separate file, 191 00:08:33.919 --> 00:08:37.080 maybe something we call a companydetailseem dot. 192 00:08:36.919 --> 00:08:37.679 Pie got it. 193 00:08:37.759 --> 00:08:39.840 And that is the spider file exactly. 194 00:08:39.879 --> 00:08:43.240 And the logic you encode inside the spider is incredibly 195 00:08:43.279 --> 00:08:45.919 elegant because it relies on loops ah loops. 196 00:08:46.279 --> 00:08:46.559 Yeah. 197 00:08:47.240 --> 00:08:50.519 Instead of writing separate XPath directions for every single row 198 00:08:50.559 --> 00:08:53.600 on the Yahoo finance page, you write a generalized set 199 00:08:53.600 --> 00:08:56.879 of instructions. You instruct the spider to target the entire 200 00:08:56.919 --> 00:09:00.279 table first, then you tell it to iterate. So row 201 00:09:00.320 --> 00:09:03.039 by row right, you say, look at row one, perform 202 00:09:03.080 --> 00:09:05.320 these actions, then move to row two and repeat. 203 00:09:05.440 --> 00:09:08.480 Okay, so for row one, it instantiates a brand new 204 00:09:08.559 --> 00:09:11.360 empty Bento box. It runs the x path directions to 205 00:09:11.360 --> 00:09:14.480 grab the name, price, symbol, and link. It packs them 206 00:09:14.519 --> 00:09:16.679 all into the box. But then it uses a command 207 00:09:16.720 --> 00:09:19.279 called yield. Yes, yield, And here is where I need 208 00:09:19.279 --> 00:09:22.960 to clarify something, because yield sounds a bit mysterious. If 209 00:09:22.960 --> 00:09:27.080 I tell the program to yield my carefully packed box, 210 00:09:27.799 --> 00:09:30.360 am I just throwing it out the window into the void? 211 00:09:30.480 --> 00:09:33.000 Like how are we actually capturing this data. 212 00:09:33.080 --> 00:09:36.399 It's a very common point of confusion, actually, because in 213 00:09:36.559 --> 00:09:40.600 standard Python functions, people are really used to the return command. Right. 214 00:09:40.679 --> 00:09:42.440 Return just gives you the answer and stops. 215 00:09:42.600 --> 00:09:45.720 Exactly, it hands back a value and immediately kills the function. 216 00:09:46.519 --> 00:09:50.120 But yield is fundamentally different. It creates what's called a generator. 217 00:09:50.240 --> 00:09:50.960 A generator. 218 00:09:51.080 --> 00:09:54.759 Okay, so when the spider yields the item, it hands 219 00:09:54.799 --> 00:09:58.559 the Bento box up to the Scrapy engine, basically pauses 220 00:09:58.600 --> 00:10:01.240 its current state, and then immediately turns back to the 221 00:10:01.279 --> 00:10:02.639 loop to process road too. 222 00:10:02.759 --> 00:10:04.840 Oh I see, so it operates almost like a worker 223 00:10:04.879 --> 00:10:05.759 on an assembly line. 224 00:10:05.840 --> 00:10:06.919 Yeah, perfect analogy. 225 00:10:07.279 --> 00:10:09.240 They finish packing a box, they hand it to the 226 00:10:09.279 --> 00:10:12.159 supervisor standing directly behind them, and then instantly turn back 227 00:10:12.200 --> 00:10:14.200 to the conveyor belt to pack the next one. They 228 00:10:14.200 --> 00:10:16.240 don't clock out and go home just because one single 229 00:10:16.279 --> 00:10:17.120 box is done. 230 00:10:17.480 --> 00:10:18.039 Exactly. 231 00:10:18.480 --> 00:10:22.000 The supervisor in your analogy is Scrapy itself. If you 232 00:10:22.039 --> 00:10:25.080 go to your terminal and simply tope screaby crawl company 233 00:10:25.120 --> 00:10:28.960 details item, Scrapey spins up the spider, The spider starts 234 00:10:29.000 --> 00:10:32.799 iterating yielding those boxes, and Scrapey just prints the resulting 235 00:10:32.879 --> 00:10:36.360 Jason objects right there on your screen, cascading down the terminal. 236 00:10:36.480 --> 00:10:38.679 I mean, seeing the matrix code fall down the screen 237 00:10:38.759 --> 00:10:41.879 is really fun. But data printed in a terminal basically 238 00:10:41.919 --> 00:10:44.240 disappears the second you close the window. We need a 239 00:10:44.240 --> 00:10:46.399 physical spadsheet we can actually send to our boss. 240 00:10:46.399 --> 00:10:51.080 Oh absolutely, And scrapey makes saving the data almost shockingly simple. 241 00:10:51.279 --> 00:10:53.679 Really yeah, you do not need to write complex file 242 00:10:53.720 --> 00:10:57.000 handling code inside your spider at all. You simply modify 243 00:10:57.120 --> 00:10:59.559 the terminal command you just ran. Oh nice, By adding 244 00:10:59.600 --> 00:11:04.279 an out put flag, specifically appending augiocompanydetails dot csv to 245 00:11:04.360 --> 00:11:07.399 the end of your command, scrape basically catches every single 246 00:11:07.440 --> 00:11:10.720 item the spider yields and dynamically appends into a neatly 247 00:11:10.759 --> 00:11:12.039 formatted spreadsheet file. 248 00:11:12.279 --> 00:11:16.000 Wait, really, so one simple terminal flag replaces dozens of 249 00:11:16.039 --> 00:11:18.360 lines of file saving logic just one flag. 250 00:11:18.440 --> 00:11:18.840 That's it. 251 00:11:19.080 --> 00:11:21.799 That is amazing. We run the command, the spider crawls 252 00:11:21.840 --> 00:11:24.240 the page, and boom, we have a CSV file sitting 253 00:11:24.279 --> 00:11:26.679 on our hard drive. But you know, if we open 254 00:11:26.720 --> 00:11:28.639 that spreadsheet right now, I have a feeling it is 255 00:11:28.679 --> 00:11:32.080 going to be a disaster. The web is inherently messy, 256 00:11:32.159 --> 00:11:36.519 right and raw scraped data is rarely usable right out 257 00:11:36.519 --> 00:11:37.080 of the box. 258 00:11:37.159 --> 00:11:40.960 Oh, it's almost always unusable. Raw data is rough. The 259 00:11:41.000 --> 00:11:44.480 source material points out two glaring formatting issues. You will 260 00:11:44.559 --> 00:11:48.000 immediately encounter when scraping that Yahoo Finance page. 261 00:11:48.080 --> 00:11:49.120 Okay, what's the first one. 262 00:11:49.279 --> 00:11:51.360 The first problem has to do with the URLs we 263 00:11:51.399 --> 00:11:54.360 are extracting for the company profiles. They are what developers 264 00:11:54.360 --> 00:11:55.759 call relative. 265 00:11:55.320 --> 00:11:59.120 Links, a relative links, meaning they don't start with HTP 266 00:11:59.399 --> 00:12:02.879 or wa ww. The HTML might just say something like 267 00:12:03.039 --> 00:12:07.159 quote MSFT exactly because websites do this to save space, right, yeah, 268 00:12:07.200 --> 00:12:09.799 they assume the browser already knows you are sitting on 269 00:12:09.799 --> 00:12:13.039 the Yahoo Finance domain. But if you drop that relative 270 00:12:13.080 --> 00:12:15.799 link into an offline Excel file and click it, your 271 00:12:15.840 --> 00:12:18.320 computer has literally no idea where to go. It's a 272 00:12:18.320 --> 00:12:19.799 completely broken link. 273 00:12:19.639 --> 00:12:22.720 Right, The spreadsheet has completely lost the context of the website. 274 00:12:23.000 --> 00:12:26.519 So that is the first major hurdle, appending the base 275 00:12:26.639 --> 00:12:28.200 domain to every single link on. 276 00:12:28.200 --> 00:12:29.000 The second issue. 277 00:12:29.039 --> 00:12:32.320 The second issue is a weird quirk of how Scrape's 278 00:12:32.360 --> 00:12:36.320 extraction functions work. By default, when you tell scrape to 279 00:12:36.360 --> 00:12:39.399 extract text, it doesn't just hand you a clean string. 280 00:12:39.759 --> 00:12:41.720 It returns a Python list object. 281 00:12:41.759 --> 00:12:44.120 Oh I see. It assumes there might be a multiple 282 00:12:44.120 --> 00:12:46.440 matches for your expath, so it wraps everything in a 283 00:12:46.480 --> 00:12:47.240 list just to be. 284 00:12:47.320 --> 00:12:49.240 Safe, exactly, just in case. 285 00:12:49.039 --> 00:12:52.440 Which means our spreadsheet doesn't say Microsoft Corporation. It says 286 00:12:52.600 --> 00:12:57.960 bracket quote Microsoft Corporation quote bracket like it's literally trapped 287 00:12:57.960 --> 00:12:59.480 inside Microsoft Corporation. 288 00:12:59.639 --> 00:13:01.720 Yeah, and if you try to hand a spreadsheet full 289 00:13:01.720 --> 00:13:04.399 of bracketed list objects to a data analyst, they will 290 00:13:04.399 --> 00:13:05.000 send it right. 291 00:13:04.919 --> 00:13:05.320 Back to you. 292 00:13:05.480 --> 00:13:06.000 I would too. 293 00:13:06.240 --> 00:13:09.240 You cannot perform basic database functions on text that's trapped 294 00:13:09.240 --> 00:13:11.320 in a list format. We need a way to refine 295 00:13:11.320 --> 00:13:15.320 this data systematically, and this introduces a vital scrapy component 296 00:13:15.399 --> 00:13:16.399 called the item loader. 297 00:13:16.720 --> 00:13:19.360 The item loader. Okay, so instead of the spider just 298 00:13:19.360 --> 00:13:22.799 stuffing raw, dirty data directly into our bento box fields, 299 00:13:23.200 --> 00:13:26.519 an item loader acts as an intelligent populator. Yes, we 300 00:13:26.600 --> 00:13:30.200 feed the raw XPath data into the loader and it processes, it, 301 00:13:30.240 --> 00:13:33.279 cleans it, and formats it before permanently sealing the box. 302 00:13:33.840 --> 00:13:36.960 And based on the documentation, it achieves this using two 303 00:13:37.200 --> 00:13:40.840 distinct mechanisms, input processors and output processor. 304 00:13:41.000 --> 00:13:41.440 You've got it. 305 00:13:41.519 --> 00:13:45.480 Let's examine the input processor first. Okay, This mechanism intercepts 306 00:13:45.480 --> 00:13:48.200 the data the exact millisecond it is scraped off the 307 00:13:48.240 --> 00:13:51.320 page before it even touches the item. Think about our 308 00:13:51.399 --> 00:13:53.120 prokin relative URL. 309 00:13:52.879 --> 00:13:55.320 Problem right, the quote MSFT problem. 310 00:13:55.399 --> 00:13:58.840 Right In our blueprint file items dot piy, we can 311 00:13:58.840 --> 00:14:01.679 attach an input process a specifically to the link field. 312 00:14:02.120 --> 00:14:05.120 And that documentation highlights a really powerful object for this 313 00:14:05.200 --> 00:14:07.080 called map compose map composed. 314 00:14:07.120 --> 00:14:08.840 I mean it sounds a bit complicated, but the way 315 00:14:08.840 --> 00:14:11.279 I understand it, it is essentially a midair interceptor. 316 00:14:11.440 --> 00:14:11.919 I like that. 317 00:14:12.080 --> 00:14:15.080 Yes, So you write a tiny helper function let's just 318 00:14:15.159 --> 00:14:17.960 name it pullings ah. And the only job of pullinks 319 00:14:18.080 --> 00:14:20.360 is to take a broken piece of text like quote 320 00:14:20.480 --> 00:14:24.080 MSFT and just gluefinance dot yahoo dot com to the 321 00:14:24.080 --> 00:14:25.240 front of it exactly. 322 00:14:25.600 --> 00:14:28.720 And map compose allows you to seamlessly map that custom 323 00:14:28.759 --> 00:14:32.240 pullings function to your incoming data stream. So the raw 324 00:14:32.279 --> 00:14:34.879 relative link comes off the web page, map composed catches 325 00:14:34.879 --> 00:14:37.759 it in midair, feeds it through your helper function, and 326 00:14:37.799 --> 00:14:41.600 the fully repaired absolute URL is what actually gets placed 327 00:14:41.600 --> 00:14:42.480 into the Bento box. 328 00:14:42.559 --> 00:14:45.039 That is so clean, So that handles the data on 329 00:14:45.080 --> 00:14:48.279 the way in. But what about the bracket problem that 330 00:14:48.440 --> 00:14:51.279 messy list format the Microsoft Corporation issue? 331 00:14:51.360 --> 00:14:51.480 Right? 332 00:14:51.519 --> 00:14:53.639 I assume that is where output processors step in. If 333 00:14:53.679 --> 00:14:56.080 the input processor is washing the ingredients before they go 334 00:14:56.120 --> 00:14:58.679 into the box. The output processor operates at the very 335 00:14:58.759 --> 00:15:01.679 end of the line, right before the spider yields the 336 00:15:01.720 --> 00:15:02.440 final product. 337 00:15:02.480 --> 00:15:06.919 Precisely, output processors handle the final data transformation, and Scrapie 338 00:15:06.960 --> 00:15:11.240 actually provides a built in output processor explicitly designed to 339 00:15:11.320 --> 00:15:13.000 solve this exact list problem. 340 00:15:13.000 --> 00:15:14.720 Oh really, yeah, it'll call take first. 341 00:15:14.960 --> 00:15:16.759 Take first. I was actually trying to think of an 342 00:15:16.759 --> 00:15:18.960 analogy for this earlier. It's almost like shucking an oyster. 343 00:15:19.120 --> 00:15:21.440 Shucking an oyster. Okay, how so, Well. 344 00:15:21.279 --> 00:15:24.120 The data you want is the pearl, but it's trapped 345 00:15:24.159 --> 00:15:29.240 inside this totally useless hardshell the list brackets. The take 346 00:15:29.320 --> 00:15:33.000 first processor prize open, the list reaches in, takes the 347 00:15:33.120 --> 00:15:35.799 very first item it finds, and just hands you the clean, 348 00:15:36.240 --> 00:15:38.240 unencumbered pearl of text. 349 00:15:38.639 --> 00:15:40.360 That is a perfect analogy. Yes. 350 00:15:40.639 --> 00:15:44.320 By setting take first as the default output processor for 351 00:15:44.360 --> 00:15:48.720 your item loader, every single field, the company, name, the price, 352 00:15:49.080 --> 00:15:52.799 the symbol is automatically shucked from those annoying brackets. Nice. 353 00:15:52.840 --> 00:15:55.240 They all become clean, plain strengths. So now when you 354 00:15:55.279 --> 00:15:58.440 save to your CSV, the spreadsheet looks completely professional. 355 00:15:58.559 --> 00:16:01.559 Okay, I am really loving this architecture. Item loaders give 356 00:16:01.639 --> 00:16:04.200 us structural cleaning on the fly. But let's push this 357 00:16:04.279 --> 00:16:04.840 even further. 358 00:16:05.000 --> 00:16:05.559 Let's do it. 359 00:16:05.759 --> 00:16:08.759 What if we want to apply actual business logic, Like, 360 00:16:08.799 --> 00:16:11.279 what if we aren't just formatting text, but we want 361 00:16:11.279 --> 00:16:14.440 to make active decisions based on the data itself after 362 00:16:14.480 --> 00:16:16.120 the whole extraction process is complete. 363 00:16:16.200 --> 00:16:19.720 Well, structural cleaning is one thing, but applying business logic 364 00:16:19.840 --> 00:16:23.679 requires the final and perhaps most powerful piece of this 365 00:16:23.720 --> 00:16:26.200 scrapy architecture. Data pipelines. 366 00:16:26.720 --> 00:16:30.799 Data pipelines. I visualize a pipeline as a quality assurance 367 00:16:30.840 --> 00:16:32.759 conveyor belt on a factory floor. 368 00:16:32.840 --> 00:16:33.879 Oh absolutely so. 369 00:16:33.919 --> 00:16:37.879 Our item loaders have packed these perfectly formatted bento boxes. 370 00:16:38.480 --> 00:16:42.080 Now those completed boxes are rolling down a long conveyor belt, 371 00:16:42.519 --> 00:16:45.879 and positioned along this belt are different QA stations. As 372 00:16:45.919 --> 00:16:49.360 a box rolls up to a station, an inspector opens it, 373 00:16:49.399 --> 00:16:53.000 looks at the data inside based on very strict mathematical rules, 374 00:16:53.240 --> 00:16:55.799 and decides what to do. They can modify the data, 375 00:16:55.879 --> 00:16:58.279 they can drop the box entirely into the trash, or 376 00:16:58.320 --> 00:17:00.320 they can simply wave it through to the next station. 377 00:17:00.559 --> 00:17:04.640 The conveyor belt analogy is incredibly accurate in your scraping project. 378 00:17:04.680 --> 00:17:07.599 These pipelines live in their own dedicated file simply named 379 00:17:07.640 --> 00:17:10.559 pipelines dot pi makes sense, and inside this file you 380 00:17:10.559 --> 00:17:13.200 can construct as many of these QA stations as you need. 381 00:17:13.440 --> 00:17:16.319 Each station is written as a standard Python class, and 382 00:17:16.359 --> 00:17:20.640 each class must contain a specific mandatory function called process item. 383 00:17:20.839 --> 00:17:25.119 Because its entire existence is dedicated to processing the item currently. 384 00:17:24.960 --> 00:17:26.559 Rolling down the belt exactly. 385 00:17:26.880 --> 00:17:30.359 The source material actually provides a highly practical example of 386 00:17:30.400 --> 00:17:33.960 this business logic involving the stock prices. Let's walk through 387 00:17:33.960 --> 00:17:34.640 how we build that. 388 00:17:35.000 --> 00:17:35.319 Sure. 389 00:17:35.400 --> 00:17:38.680 So imagine you are building a tool for a financial 390 00:17:38.720 --> 00:17:42.960 analyst who only cares about extreme stock price movements. They 391 00:17:43.000 --> 00:17:45.759 want to flag any stock over one hundred dollars or 392 00:17:45.799 --> 00:17:48.880 under fifty dollars. Okay, we can build two separate stations 393 00:17:48.920 --> 00:17:51.559 on our conveyor belt. We define our first class and 394 00:17:51.640 --> 00:17:53.359 name it above one hundred price check. 395 00:17:53.519 --> 00:17:56.920 Station one is operational the item rolls up. What actually 396 00:17:56.920 --> 00:17:58.960 happens inside that process item function. 397 00:17:59.160 --> 00:18:01.319 First, it looks at the inter day price field. Now, 398 00:18:01.359 --> 00:18:03.799 remember the price is currently stored as a string of text. 399 00:18:04.240 --> 00:18:06.799 You cannot perform mathematical operations on text. 400 00:18:06.559 --> 00:18:08.720 Right, Python would just throw an error exactly. 401 00:18:09.000 --> 00:18:10.920