WEBVTT 1 00:00:00.160 --> 00:00:02.839 Welcome to this deep dive. Today we're taking a look 2 00:00:02.919 --> 00:00:07.120 at well a pretty massive digital superpower, and more importantly, 3 00:00:07.360 --> 00:00:10.439 how to actually use it without you know, totally blowing 4 00:00:10.519 --> 00:00:10.880 up your. 5 00:00:10.759 --> 00:00:14.320 Own systems, right, which is surprisingly easy to do if 6 00:00:14.320 --> 00:00:15.080 you aren't careful. 7 00:00:15.279 --> 00:00:18.920 Oh absolutely so. Imagine you have this digital scout, a 8 00:00:18.960 --> 00:00:23.440 really tireless, lightning fast web scraper that you can just 9 00:00:23.480 --> 00:00:27.399 send out onto the chaotic landscape of the Internet to 10 00:00:27.480 --> 00:00:29.719 gather massive amounts of information for you. 11 00:00:29.960 --> 00:00:32.200 It sounds incredible, right, You just pointed at a website 12 00:00:32.200 --> 00:00:33.479 and say, go get me the data. 13 00:00:33.960 --> 00:00:37.000 Yeah, but that's where almost everyone makes their first like 14 00:00:37.600 --> 00:00:38.799 catastrophic mistake. 15 00:00:38.880 --> 00:00:41.520 Oh yeah, you think your main problem is just finding 16 00:00:41.600 --> 00:00:45.399 the data, but within about five minutes of launching that scout, 17 00:00:45.600 --> 00:00:48.359 your actual problem becomes drowning in an absolute ocean of 18 00:00:48.399 --> 00:00:49.600 garbage exactly. 19 00:00:49.840 --> 00:00:51.600 And that is the core of our mission today. We 20 00:00:51.640 --> 00:00:54.280 are mastering the mechanics of how to really control that 21 00:00:54.359 --> 00:00:55.479 digital scout. 22 00:00:55.359 --> 00:00:58.320 Because if data is the new oil, mean crude oil 23 00:00:58.479 --> 00:01:00.799 straight out of the ground, will just completely destroy a 24 00:01:00.840 --> 00:01:01.679 modern engine. 25 00:01:01.840 --> 00:01:05.519 Right If a multi million dollar e commerce algorithm scrapes 26 00:01:05.599 --> 00:01:07.560 like a messy string of web code instead of a 27 00:01:07.599 --> 00:01:08.879 clean numerical price. 28 00:01:09.319 --> 00:01:12.319 The whole system just crashes, it really does. So today 29 00:01:12.319 --> 00:01:14.079 we're going to look at how you build an automated 30 00:01:14.120 --> 00:01:18.519 refinery something to turn that raw, messy web code into polished, 31 00:01:19.040 --> 00:01:20.799 organized business intelligence. 32 00:01:21.000 --> 00:01:25.640 We'll be unpacking the underlying architecture of web crawling, and 33 00:01:25.719 --> 00:01:28.000 the beauty of this architecture is that it all relies 34 00:01:28.079 --> 00:01:29.519 on logical flow. 35 00:01:29.760 --> 00:01:32.000 Yeah, if you can understand the flow of the information, 36 00:01:32.359 --> 00:01:35.719 the actual mechanics of how you gather it become well 37 00:01:36.079 --> 00:01:37.120 incredibly intuitive. 38 00:01:37.400 --> 00:01:40.040 So our roadmap for this deep dive, we're going to 39 00:01:40.120 --> 00:01:42.840 break this down into three main phases. First, how to 40 00:01:42.879 --> 00:01:45.159 steer the crawler using allow and deny. 41 00:01:44.959 --> 00:01:47.439 Rules right so it doesn't get completely lost exactly. 42 00:01:47.680 --> 00:01:50.480 Then we'll explore this invisible assembly line of input and 43 00:01:50.519 --> 00:01:52.640 output processors that clean up the data. 44 00:01:52.680 --> 00:01:54.680 And finally we'll open up the toolkit and look at 45 00:01:54.680 --> 00:01:59.040 how to handle escalating data problems using specific functions. 46 00:01:59.319 --> 00:02:01.319 And for those of you listening, even though this is 47 00:02:01.359 --> 00:02:04.719 technical programming architecture, we're breaking it down step by step 48 00:02:04.760 --> 00:02:07.480 you'll be able to completely visualize the logic in your head. 49 00:02:07.599 --> 00:02:09.400 So let's start at the very beginning of that flow. 50 00:02:10.000 --> 00:02:13.680 The scout itself, which in the web scraping world is 51 00:02:13.719 --> 00:02:17.280 almost universally called a spider. A spider, right, Yeah, and 52 00:02:17.319 --> 00:02:20.759 a spider is essentially just an automated script, a bot really, 53 00:02:20.800 --> 00:02:24.479 that navigates from link to link across the web, fetching 54 00:02:24.479 --> 00:02:26.479 the contents of whatever pages it visits. 55 00:02:26.560 --> 00:02:29.560 But if you just release a spider onto a major 56 00:02:29.719 --> 00:02:32.479 retail website without giving it any instructions, I mean, it 57 00:02:32.520 --> 00:02:35.000 has no concept of what's useful then at all. 58 00:02:35.039 --> 00:02:37.240 It'll just click every single link it finds. 59 00:02:37.039 --> 00:02:40.400 Like, it'll download the about us page, the terms of service, 60 00:02:40.560 --> 00:02:42.159 the user forums. 61 00:02:41.800 --> 00:02:45.360 The shopping card, password reset pages, all of it, which 62 00:02:45.400 --> 00:02:49.360 burns massive amounts of your processing power and bandwidth to 63 00:02:49.439 --> 00:02:51.560 retrieve data that you're just going to throw right into 64 00:02:51.560 --> 00:02:52.000 the trash. 65 00:02:52.120 --> 00:02:55.080 Yeah, if your goal is to build like a competitive 66 00:02:55.159 --> 00:02:59.039 database of their product catalog, a download of their privacy 67 00:02:59.039 --> 00:03:00.919 policy is totally. 68 00:03:00.639 --> 00:03:03.960 Useless, exactly. This is where we have to establish our 69 00:03:03.960 --> 00:03:06.759 guard rails. Before we can even think about refining the data. 70 00:03:06.759 --> 00:03:09.080 We have to make sure our crawler is fetching the 71 00:03:09.080 --> 00:03:10.360 correct pages to begin. 72 00:03:10.199 --> 00:03:11.800 With, and we do that by setting up allow and 73 00:03:11.840 --> 00:03:12.520 deny rules. 74 00:03:12.599 --> 00:03:15.639 Right, you got it. The spider really needs to know 75 00:03:15.639 --> 00:03:16.319 what it's looking for. 76 00:03:16.680 --> 00:03:20.680 So how does the allow rule actually communicate with the spider. 77 00:03:20.960 --> 00:03:23.599 Well, the allow option basically acts as a strict filter 78 00:03:24.120 --> 00:03:27.759 for the URLs the spiders permitted to follow. It relies 79 00:03:27.800 --> 00:03:31.560 on regular expressions, which you'll usually hear abbreviated as redx. 80 00:03:32.199 --> 00:03:35.560 Right. And for anyone newer to programming, a regular expression 81 00:03:35.680 --> 00:03:38.800 isn't just looking for an exact word match. It's more 82 00:03:38.960 --> 00:03:42.840 like a sequence of characters that specifies a search pattern. 83 00:03:42.879 --> 00:03:45.520 Yeah, it's like a super powered find function that looks 84 00:03:45.520 --> 00:03:47.319 at the structural DNA of the web address. 85 00:03:47.439 --> 00:03:49.639 So for URL to be extracted, it has to match 86 00:03:49.680 --> 00:03:51.319 that redx pattern exactly. 87 00:03:51.680 --> 00:03:54.360 And if you leave the allow rule completely empty, the 88 00:03:54.400 --> 00:03:57.719 default behavior is that the spider matches all links. It 89 00:03:57.840 --> 00:04:01.159 just goes everywhere, It just runs wild. Yeah. But let's 90 00:04:01.159 --> 00:04:03.960 say we're targeting a specific online bookstore and we only 91 00:04:04.000 --> 00:04:06.000 want the actual book listings. 92 00:04:06.080 --> 00:04:08.479 Okay, so we'd set an allow rule to look for 93 00:04:08.520 --> 00:04:11.400 a specific pattern in the URL path, like we might 94 00:04:11.439 --> 00:04:15.199 configure it to only follow links that contain the string 95 00:04:15.639 --> 00:04:18.240 catalog slash category, slash books. 96 00:04:18.600 --> 00:04:20.879 Right. So, if the spider is crawling the main page 97 00:04:21.079 --> 00:04:24.600 and sees a link pointing to catalog slash category slash 98 00:04:24.639 --> 00:04:28.439 books slash science fiction. It analyzes that URL, sees the 99 00:04:28.480 --> 00:04:30.560 matching pattern and proceeds to crawl it. 100 00:04:30.959 --> 00:04:34.199 And if it sees a link to customer support slash contact, 101 00:04:35.319 --> 00:04:36.160 the pattern fails. 102 00:04:36.279 --> 00:04:39.680 Yep, the spider ignores the link completely. It never even 103 00:04:39.720 --> 00:04:42.199 loads the page, which is really the key to scaling 104 00:04:42.240 --> 00:04:43.519 your data gathering efficiently. 105 00:04:43.639 --> 00:04:45.399 I definitely follow the logic there, but you know, it 106 00:04:45.399 --> 00:04:47.480 brings up an interesting question. If the allow rule is 107 00:04:47.519 --> 00:04:50.519 already acting as the strict bouncer, restricting the spider to 108 00:04:50.600 --> 00:04:52.600 just the book categories, why do we even need a 109 00:04:52.639 --> 00:04:53.319 deny a rule. 110 00:04:53.439 --> 00:04:57.240 Oh, because website structures are notoriously chaotic. I mean, they 111 00:04:57.240 --> 00:04:59.800 are rarely built with perfect clean. 112 00:04:59.600 --> 00:05:04.399 Category ah right, things get messy, It's very messy. 113 00:05:04.720 --> 00:05:08.839 Categories overlap, legacy pages get left behind. Or sometimes you 114 00:05:08.879 --> 00:05:10.839 just want to cast a slightly wider net with your 115 00:05:10.879 --> 00:05:12.959 allow rule so you don't miss anything, but. 116 00:05:12.959 --> 00:05:15.959 You still have those highly specific areas you absolutely know 117 00:05:16.120 --> 00:05:19.120 will ruin your data set exactly. It kind of reminds 118 00:05:19.120 --> 00:05:23.160 me of sending an eager, maybe a slightly hyperactive intern 119 00:05:23.319 --> 00:05:26.240 into an enormous physical library to do research. 120 00:05:26.600 --> 00:05:27.600 I like that analogy. 121 00:05:27.759 --> 00:05:30.439 Yeah, the allow rule is you telling the intern, hey, 122 00:05:30.680 --> 00:05:33.360 only bring me books from the nonfiction wing. Do not 123 00:05:33.439 --> 00:05:35.720 go to fiction, do not go to the cafeteria. 124 00:05:36.319 --> 00:05:38.240 But that intern is still going to bring you everything 125 00:05:38.319 --> 00:05:40.399 in nonfiction, including things you really don't. 126 00:05:40.160 --> 00:05:43.360 Want, right, which is where the deny rule acts as 127 00:05:43.399 --> 00:05:46.120 that vital safety net. You tell them you are allowed 128 00:05:46.160 --> 00:05:48.639 in the nonfiction wing, but whatever you do, stay out 129 00:05:48.680 --> 00:05:50.639 of the true crime section. I do not want to 130 00:05:50.680 --> 00:05:51.360 see those books. 131 00:05:51.480 --> 00:05:55.000 Translating that back to our webscraping scenario, you might have 132 00:05:55.040 --> 00:05:58.560 an allow rule capturing all books, but you could configure 133 00:05:58.600 --> 00:06:03.759 deny rules to block specific subcategory paths, like say erotica 134 00:06:03.800 --> 00:06:05.160 fifty or crime fifty one. 135 00:06:05.319 --> 00:06:08.399 So even if those paths technically sit inside the broader 136 00:06:08.439 --> 00:06:11.360 book catalog that's allowed by the first rule, the spider 137 00:06:11.399 --> 00:06:15.680 sees those specific strings in the denialist and just stops instantly. 138 00:06:16.040 --> 00:06:17.560 It acts as an absolute override. 139 00:06:17.600 --> 00:06:20.040 Okay, so we have the guardrails up. The spider goes out, 140 00:06:20.160 --> 00:06:23.399 navigates using the allow patterns, avoids the traps with the 141 00:06:23.439 --> 00:06:25.879 denial patterns, and brings back the right pages. 142 00:06:26.040 --> 00:06:27.639 The initial gathering phase. 143 00:06:27.439 --> 00:06:31.000 Is complete, but the data it brings back is well, 144 00:06:31.399 --> 00:06:33.160 it's far from ready for analysis, isn't it. 145 00:06:33.240 --> 00:06:36.240 Oh, it's terrible. By default, the data the spider hands 146 00:06:36.279 --> 00:06:40.240 you is just raw, messy HTML markup directly from the 147 00:06:40.279 --> 00:06:40.920 source code. 148 00:06:41.000 --> 00:06:44.279 Right, It's not just handing you a clean, nice spreadsheet, 149 00:06:44.480 --> 00:06:46.439 Like if we're trying to scrape the title of a book. 150 00:06:46.560 --> 00:06:48.959 It doesn't just hand over the words the Great Gatsby. 151 00:06:49.160 --> 00:06:52.399 No, it brings back the invisible infrastructure that tells the 152 00:06:52.439 --> 00:06:54.680 browser how to actually display those words. 153 00:06:54.759 --> 00:06:57.639 So you get like an opening heading tag, maybe a 154 00:06:57.639 --> 00:07:01.639 class attribute defining the font size, then some invisible white 155 00:07:01.639 --> 00:07:05.319 space line breaks the actual text the Great Gatsby, and 156 00:07:05.360 --> 00:07:06.759 then a closing heading tag. 157 00:07:06.959 --> 00:07:09.600 It's a lot of junk, which means we need to 158 00:07:09.600 --> 00:07:12.319 build the refinery. We need a system to strip all 159 00:07:12.319 --> 00:07:14.720 that away and leave just the clean data. 160 00:07:14.759 --> 00:07:16.800 And this brings us to processors. 161 00:07:16.959 --> 00:07:20.920 Yes, processors handle the pre processing and post processing of 162 00:07:20.959 --> 00:07:22.480 all this raw scrape data. 163 00:07:22.600 --> 00:07:27.720 They basically take that messy HTML string, format it, clean it, 164 00:07:27.759 --> 00:07:30.360 and turn it into a pristine data point. 165 00:07:30.480 --> 00:07:33.920 And this entire architecture relies on an invisible assembly line 166 00:07:34.160 --> 00:07:38.639 made of two really distinct phases, input processors and output processors. 167 00:07:38.759 --> 00:07:42.160 So an input processor acts immediately on the extracted data, 168 00:07:42.279 --> 00:07:44.680 like the very second the spider receives it right. 169 00:07:44.560 --> 00:07:47.160 As soon as that raw piece of HTML is plucked 170 00:07:47.199 --> 00:07:50.120 from the page, the input processor grabs it and does 171 00:07:50.160 --> 00:07:51.000 the initial cleaning. 172 00:07:51.199 --> 00:07:53.959 Then we have the output processors. So after all the 173 00:07:54.040 --> 00:07:56.839 raw data fragments for a specific item are collected and 174 00:07:56.879 --> 00:08:00.439 push through those input processors, the output processor takes over exactly. 175 00:08:00.480 --> 00:08:03.600 It looks at the pre clean data, applies the final formatting, 176 00:08:03.800 --> 00:08:06.199 and locks it in as the permanent value for that item. 177 00:08:06.600 --> 00:08:09.199 I kind of have to challenge this two step architecture 178 00:08:09.199 --> 00:08:13.000 though it feels like we're adding unnecessary complexity. Well, why 179 00:08:13.040 --> 00:08:15.879 not just build a massive bucket, Let the spider dump 180 00:08:15.879 --> 00:08:18.839 all the raw, messy HTML into the bucket for an hour, 181 00:08:19.120 --> 00:08:21.720 and then we just run one massive cleaning script at 182 00:08:21.720 --> 00:08:23.319 the very end in a single batch. 183 00:08:23.720 --> 00:08:26.360 It really comes down to system memory and the sheer 184 00:08:26.519 --> 00:08:29.879 scale of web data. Think about the efficiency of a 185 00:08:29.879 --> 00:08:33.559 physical assembly line. Okay, if you're scraping one hundred thousand 186 00:08:33.559 --> 00:08:36.759 pages and you wait until the very end to clean everything, 187 00:08:37.159 --> 00:08:40.840 you are forcing your system to hold massive amounts of heavy, 188 00:08:40.879 --> 00:08:43.879 redundant HTML tags in its active memory. 189 00:08:44.000 --> 00:08:46.840 Oh, so it just consumes a tremendous amount of ram huge. 190 00:08:46.679 --> 00:08:50.039 Amounts all right. By filtering individual pieces the moment they arrive, 191 00:08:50.320 --> 00:08:54.000 which is the input processor's job, you're constantly stripping away 192 00:08:54.039 --> 00:08:56.240 the excess weight before it even enters your database. 193 00:08:56.320 --> 00:08:59.159 Oh I see, So the input processor is essentially knocking 194 00:08:59.240 --> 00:09:01.240 the heavy mud off the boots on the porch so 195 00:09:01.279 --> 00:09:03.879 you don't have to spend hours cleaning the carpets inside 196 00:09:03.919 --> 00:09:04.519 the house later. 197 00:09:04.840 --> 00:09:06.519 That is a perfect way to look at it. The 198 00:09:06.600 --> 00:09:11.000 other crucial factor is that web data frequently arrives completely 199 00:09:11.080 --> 00:09:14.720 shattered into lists, shattered like an HTML page might have 200 00:09:14.799 --> 00:09:17.360 the title of a book broken up into multiple different 201 00:09:17.440 --> 00:09:21.080 text nodes scattered across the code. The input processor cleans 202 00:09:21.120 --> 00:09:23.600 each of those tiny shards individually as they arrive. 203 00:09:23.879 --> 00:09:26.559 Okay, and then the output processor acts as the final 204 00:09:26.639 --> 00:09:30.159 packaging step. It takes those clean little shards, assembles them 205 00:09:30.159 --> 00:09:34.279 into one beautiful string, and just seals the box exactly. 206 00:09:34.639 --> 00:09:37.720 But to run an assembly line that's sophisticated, we obviously 207 00:09:37.759 --> 00:09:41.120 need specific tools to solve specific problems, right. 208 00:09:40.960 --> 00:09:43.759 And as we look at the data arriving, the framework 209 00:09:43.840 --> 00:09:48.840 provides built in processors to handle escalating levels of messiness. 210 00:09:49.039 --> 00:09:51.279 Let's actually start with a scenario where the data isn't 211 00:09:51.320 --> 00:09:51.840 messy at. 212 00:09:51.799 --> 00:09:53.480 All, okay, the ideal scenario. 213 00:09:53.559 --> 00:09:55.960 In that case, we use the most basic tool available, 214 00:09:56.279 --> 00:10:00.399 the identity processor. It's incredibly simple because it's only function 215 00:10:00.519 --> 00:10:01.799 is to do absolutely nothing. 216 00:10:01.919 --> 00:10:05.200 Wait, it returns the original values completely unchanged. 217 00:10:05.320 --> 00:10:05.559 Yep. 218 00:10:05.919 --> 00:10:08.080 If it does nothing, why not just leave it blank? 219 00:10:08.279 --> 00:10:10.120 Why even declare a processor at all? 220 00:10:10.399 --> 00:10:14.799 It maintains the structural integrity of your pipeline. Sometimes a 221 00:10:14.840 --> 00:10:17.039 field genuinely doesn't need cleaning. 222 00:10:17.000 --> 00:10:21.080 Like an internal product ID number that's already perfectly formatted exactly. 223 00:10:21.480 --> 00:10:24.000 But the framework still expects a processor to be declared 224 00:10:24.000 --> 00:10:28.159 for consistency. So calling the identity processor tells the framework, Hey, 225 00:10:28.600 --> 00:10:31.360 I acknowledge this data is here, and I am deliberately 226 00:10:31.399 --> 00:10:35.000 passing it straight through to the next phase without modifications. 227 00:10:35.120 --> 00:10:38.120 Got it the deliberate pass through? But let's escalate the problem. 228 00:10:38.120 --> 00:10:40.720 Because the web is chaotic. What happens if the spider 229 00:10:40.879 --> 00:10:44.720 accidentally scrapes empty data structures alongside our actual target. 230 00:10:44.960 --> 00:10:47.879 Then we use the take first processor. It analyzes the 231 00:10:47.919 --> 00:10:50.960 list of values it receives and simply returns the first 232 00:10:51.240 --> 00:10:54.720 non null or non empty value and discards everything else. 233 00:10:55.159 --> 00:10:57.159 So if the input comes in as a list that 234 00:10:57.200 --> 00:11:00.360 reads empty space, empty space at the Great Gatsby space, 235 00:11:00.440 --> 00:11:02.720 it just grabs the Great Gatsby and throws the rest 236 00:11:02.720 --> 00:11:03.720 into the incinerator. 237 00:11:03.799 --> 00:11:07.879 Right. And because web scraping constantly pulls an accidental empty data, 238 00:11:08.279 --> 00:11:11.240 take first is typically deployed as an output processor for 239 00:11:11.279 --> 00:11:14.759 single valued fields. When you only want one definitive answer, 240 00:11:15.039 --> 00:11:16.320 you just tell it to take first. 241 00:11:16.639 --> 00:11:21.320 Let's escalate again. We mentioned earlier that sometimes a single 242 00:11:21.320 --> 00:11:24.279 piece of data is shattered across different nodes in the HTML. 243 00:11:24.440 --> 00:11:26.440 Yes, the shattered chards. 244 00:11:26.360 --> 00:11:29.679 Right, so the title gets scraped as three separate fragments, 245 00:11:29.720 --> 00:11:33.480 the Great and Gatsby. If we use take first on that, 246 00:11:33.519 --> 00:11:36.960 we just get the which is obviously wrong, Which. 247 00:11:36.720 --> 00:11:40.120 Is when we turn to the join processor. Just as 248 00:11:40.120 --> 00:11:44.039 the name implies, it takes multiple fragmented values and stitches 249 00:11:44.080 --> 00:11:46.039 them back together into a single strength. 250 00:11:46.120 --> 00:11:48.320 And you define a separator, right, so you can tell 251 00:11:48.320 --> 00:11:51.559 it to insert a space or comma between each fragment 252 00:11:51.639 --> 00:11:52.399 as it joins them. 253 00:11:52.639 --> 00:11:55.200 Exactly, so we have basic pass throughs. We can grab 254 00:11:55.240 --> 00:11:58.600 the first clean item, and we can stytch fragments together. 255 00:11:58.879 --> 00:12:02.600 But what if the data requires a multi step decontamination process. 256 00:12:02.840 --> 00:12:05.120 Then we move into the advanced tools, starting with a 257 00:12:05.159 --> 00:12:08.519 composed processor. This allows you to chain mathematical or logical 258 00:12:08.519 --> 00:12:12.879 functions together, basically creating a mini pipeline within your main pipeline. 259 00:12:12.960 --> 00:12:14.279 How does that work in practice? 260 00:12:14.399 --> 00:12:17.320 Well, the composed processor takes entire chunk of input data 261 00:12:17.759 --> 00:12:20.519 and passes it wholly into the first function you define. 262 00:12:21.080 --> 00:12:23.960 That function does its job and outputs. 263 00:12:23.519 --> 00:12:25.960 A result, and then that result is passed directly into 264 00:12:25.960 --> 00:12:29.600 the second function. In summer, value A goes into the machine, 265 00:12:29.720 --> 00:12:32.440 gets transformed into value B, which drops into the next 266 00:12:32.440 --> 00:12:33.600 machine and becomes value C. 267 00:12:33.879 --> 00:12:37.679 Which works beautifully for single, solid blocks of data. But 268 00:12:37.840 --> 00:12:40.200 as we've said, web data almost always arise as a 269 00:12:40.200 --> 00:12:43.679 list of multiple items. This brings us to the most common, 270 00:12:43.840 --> 00:12:47.399 probably the most powerful processor in the entire framework, the 271 00:12:47.440 --> 00:12:48.679 map composed processor. 272 00:12:49.080 --> 00:12:52.679 Okay, so how does map compose handle those lists differently 273 00:12:52.759 --> 00:12:53.919 than regular. 274 00:12:53.559 --> 00:12:57.679 Composed The fundamental difference is iteration. Instead of passing the 275 00:12:57.720 --> 00:13:00.360 whole massive chunk of data into a function at one uns, 276 00:13:00.799 --> 00:13:03.600 map compose isolates the elements in the list. 277 00:13:03.720 --> 00:13:04.480 It separates them. 278 00:13:04.600 --> 00:13:07.159 Yes, it takes the first function and applies it to 279 00:13:07.200 --> 00:13:11.559 each individual element separately. It processes element one, then element two, 280 00:13:11.639 --> 00:13:13.080 then element three. 281 00:13:12.960 --> 00:13:16.000 And the results of those individual cleanings are concatenated to 282 00:13:16.039 --> 00:13:18.000 construct a new list exactly. 283 00:13:18.879 --> 00:13:22.679 Then that entire new list is subjected to the second function, 284 00:13:23.320 --> 00:13:25.519 again iterating element by element. 285 00:13:25.759 --> 00:13:28.279 To make sure we're crystal clear on the mechanics here, 286 00:13:28.360 --> 00:13:31.600 let's use a physical analogy. Let's say our raw data 287 00:13:31.720 --> 00:13:35.600 is a large basket of freshly dug potatoes completely covered 288 00:13:35.639 --> 00:13:36.360 in thick mud. 289 00:13:36.600 --> 00:13:38.320 Okay, I'm following, and our goal. 290 00:13:38.159 --> 00:13:41.759 Is a basket of perfectly washed, peeled potatoes. 291 00:13:41.159 --> 00:13:43.960 A very standard data cleaning requirement, right. 292 00:13:43.840 --> 00:13:46.840 So using the regular composed processor is like taking the 293 00:13:47.000 --> 00:13:49.720 entire basket of potatoes and dumping it into a large 294 00:13:49.759 --> 00:13:52.080 tub of water and trying to wash the whole mass 295 00:13:52.080 --> 00:13:52.480 at once. 296 00:13:52.639 --> 00:13:55.159 Yeah, the potatoes on the outside might get clean, but 297 00:13:55.240 --> 00:13:56.840 the ones trapped in the middle are still going to 298 00:13:56.879 --> 00:13:58.679 have mud on them when you pull the basket out 299 00:13:58.720 --> 00:13:59.720 to try and peel them. 300 00:14:00.000 --> 00:14:03.360 Exactly. It acts on the entire mass simultaneously, which leaves 301 00:14:03.440 --> 00:14:04.679 huge room for error. 302 00:14:04.919 --> 00:14:08.559 Especially when dealing with complex nested HTML structures. 303 00:14:08.600 --> 00:14:11.799 But map composed is different. Because map compose iterates, it 304 00:14:11.879 --> 00:14:14.200 means You take one single potato out of the basket, 305 00:14:14.399 --> 00:14:17.559 you wash it, You place it in a new clean basket. 306 00:14:17.720 --> 00:14:20.279 You take the next potato, wash it, move it right, 307 00:14:20.639 --> 00:14:23.519 and once the entire basket is washed, you iterate again 308 00:14:23.559 --> 00:14:26.279 with the peeler. You take one clean potato, peel it 309 00:14:26.320 --> 00:14:29.799 and move it. You process each piece individually, which guarantees 310 00:14:29.919 --> 00:14:31.840 every single element is perfectly handled. 311 00:14:32.440 --> 00:14:35.840 The logic of that analogy maps perfectly to messy web data. 312 00:14:36.399 --> 00:14:38.440 We do not want to try and clean a massive 313 00:14:38.480 --> 00:14:41.679 block of HTML code at once. Too messy, way too messy. 314 00:14:41.919 --> 00:14:45.080 We use map composed to isolate every single HTML element, 315 00:14:45.480 --> 00:14:48.519 strip the code tags off each one individually, and collect 316 00:14:48.519 --> 00:14:50.039 the clean text brilliant. 317 00:14:50.480 --> 00:14:52.600 Now there's one more tool we need to cover for 318 00:14:52.639 --> 00:14:55.000 a very specific type of modern data. 319 00:14:54.720 --> 00:14:58.440 Ah the select chains processor. Yes, this is a specialized 320 00:14:58.480 --> 00:15:02.159 tool for querying values using a Jason path because modern 321 00:15:02.200 --> 00:15:05.720 web applications and APIs frequently send data back in JSON 322 00:15:05.840 --> 00:15:07.519 format instead of traditional htemail. 323 00:15:07.759 --> 00:15:10.240 Let's visualize Jason format for a second. It's essentially a 324 00:15:10.279 --> 00:15:13.320 series of nested folders, or like Russian nesting dolls. 325 00:15:13.399 --> 00:15:14.519 That's a great way to picture it. 326 00:15:14.720 --> 00:15:18.039 You open the biggest doll labeled store and inside is 327 00:15:18.080 --> 00:15:21.120 a doll labeled books. You open that to find science fiction, 328 00:15:21.399 --> 00:15:24.120 and inside that is the final piece of data the title. 329 00:15:24.399 --> 00:15:27.120 And the select James processor allows you to write a 330 00:15:27.159 --> 00:15:30.559 specific path that acts like a laser pointer. It pierces 331 00:15:30.600 --> 00:15:33.919 straight through those nested dolls to extract exactly the data 332 00:15:33.960 --> 00:15:37.639 point you need without having to manually open every single layer. 333 00:15:37.799 --> 00:15:40.639 Okay, so we have a full toolkit. Now we understand 334 00:15:40.679 --> 00:15:43.320 how to set guardrails. We understand the input and output 335 00:15:43.360 --> 00:15:46.720 assembly line, and we have tools to handle empty spaces, 336 00:15:47.200 --> 00:15:52.159 shattered data, complex iterative cleaning, and nested Jason structures. 337 00:15:52.279 --> 00:15:52.960 We are ready. 338 00:15:53.519 --> 00:15:57.320 Let's mentally walk through a practical scenario. Let's look at 339 00:15:57.360 --> 00:15:59.600 the data blueprint for our catalog of books and actually 340 00:15:59.639 --> 00:16:01.159 set up the assembly line to clean it. 341 00:16:01.159 --> 00:16:04.759 All right, let's establish our workbench. We import the specific 342 00:16:04.799 --> 00:16:07.720 processors we know we'll need from the framework Join, map, 343 00:16:07.759 --> 00:16:10.399 compose and take first GOTTAM. We also bring in a 344 00:16:10.440 --> 00:16:14.039 specialized function called removed tags, which is exactly what it 345 00:16:14.039 --> 00:16:16.960 sounds like. It targets HTML markup and deletes it, leaving 346 00:16:17.039 --> 00:16:17.799 only the text. 347 00:16:18.080 --> 00:16:20.480 And let's also say we wrote our own custom validation 348 00:16:20.600 --> 00:16:23.799 function called filter price. It analyzes text to ensure the 349 00:16:23.840 --> 00:16:25.679 value is actually a mathematical number. 350 00:16:25.879 --> 00:16:29.879 Right, Because the spider accidentally scrapes the word free or 351 00:16:29.960 --> 00:16:32.679 out of stock, we don't want those words corrupting a 352 00:16:32.720 --> 00:16:36.480 database column designed specifically for numerical pricing data. 353 00:16:36.600 --> 00:16:38.799 So our tools are ready. The raw data for the 354 00:16:38.840 --> 00:16:41.759 book title field comes in from the spider. The input 355 00:16:41.799 --> 00:16:43.639 processor takes over immediately, and. 356 00:16:43.600 --> 00:16:46.279 We instructed to use map composed to run our removed 357 00:16:46.320 --> 00:16:47.000 tags function. 358 00:16:47.159 --> 00:16:49.679 So the raw data, which is just a messy list 359 00:16:49.720 --> 00:16:53.080 of heading tags and text, gets fed into map compose. 360 00:16:53.600 --> 00:16:57.440 It isolates every individual piece, strips out all the HTML, 361 00:16:57.799 --> 00:17:00.960 and creates a new list containing just the clean words 362 00:17:00.960 --> 00:17:01.559 of the title. 363 00:17:01.639 --> 00:17:04.880 The input phase is complete. Now the output processor kicks 364 00:17:04.880 --> 00:17:05.680 in to finalize the. 365 00:17:05.680 --> 00:17:08.319 Field for the title. We call the join function. It 366 00:17:08.359 --> 00:17:11.039 takes that list of clean text fragments and joins them 367 00:17:11.039 --> 00:17:14.880 together into a single cohesive string. The title is officially 368 00:17:14.920 --> 00:17:16.480 clean and stored perfect. 369 00:17:16.519 --> 00:17:18.400 Now we move to the price field. The raw price 370 00:17:18.519 --> 00:17:21.640 data comes in. The input processor again uses map composed 371 00:17:21.720 --> 00:17:24.200 running removed tags to strip the HTML. 372 00:17:24.359 --> 00:17:27.759 But remember map compose allows us to chain functions iteratively. 373 00:17:28.119 --> 00:17:30.799 Yes, so after it strips the tags from an element 374 00:17:31.039 --> 00:17:34.519 we instructed to immediately run that custom filter price function. 375 00:17:34.759 --> 00:17:37.640 It strips the HTML and then instantly verifies hey is 376 00:17:37.680 --> 00:17:41.160 the remaining text a valid number? It validates the data 377 00:17:41.240 --> 00:17:42.000 on the fly. 378 00:17:42.319 --> 00:17:44.519 And by the time the input processing is done, we 379 00:17:44.599 --> 00:17:48.160 have a list containing only clean, verified numerical prices. 380 00:17:48.359 --> 00:17:51.000 But for the output processor on the price, we face 381 00:17:51.039 --> 00:17:54.480 a different logical problem. We can't use join. No, definitely not, 382 00:17:54.680 --> 00:17:57.839 because if we use join and the list had two numbers, 383 00:17:57.839 --> 00:18:00.400 our final price would literally be nine time teaen point 384 00:18:01.039 --> 00:18:04.599 nine point ninety nine. We just want the definitive price. 385 00:18:04.440 --> 00:18:07.640 Which is why we invoke take first. It grabs the 386 00:18:07.640 --> 00:18:10.599 first clean valid number from our filtered list, assigns it 387 00:18:10.640 --> 00:18:12.960 as the final value for the books price, and just 388 00:18:13.119 --> 00:18:14.039 discards the rest. 389 00:18:14.160 --> 00:18:16.880 The elegance of this architecture is just remarkable. I mean, 390 00:18:17.200 --> 00:18:19.920 the raw, messy code comes in from the chaotic Internet, 391 00:18:20.279 --> 00:18:23.480 the input processor isolates it, strips it down, and verifies 392 00:18:23.480 --> 00:18:26.440 it piece by piece like a relentless assembly line, and. 393 00:18:26.400 --> 00:18:30.000 Then the output processor neatly packages it up into a 394 00:18:30.079 --> 00:18:33.359 single perfect data point. Because you know, if your input 395 00:18:33.400 --> 00:18:36.440 is garbage, your output is garbage. This system ensures your 396 00:18:36.480 --> 00:18:37.519 input is pristine. 397 00:18:37.680 --> 00:18:40.559 It really does well. Before we wrap up, let's pose 398 00:18:40.599 --> 00:18:43.839 a quick review exercise for you listening to test your 399 00:18:43.880 --> 00:18:45.240 grasp of this architecture. 400 00:18:45.319 --> 00:18:48.599 Good idea. Think back to our toolkit. If you have 401 00:18:48.680 --> 00:18:51.359