WEBVTT

1
00:00:00.200 --> 00:00:05.280
<v Speaker 1>Right at this exact millisecond, there are millions of invisible

2
00:00:05.400 --> 00:00:09.320
<v Speaker 1>digital agents just silently traversing fiber optic cables all around

3
00:00:09.359 --> 00:00:09.880
<v Speaker 1>the globe. Oh.

4
00:00:09.960 --> 00:00:14.160
<v Speaker 2>Absolutely, they're constantly mapping network infrastructure or you know, indexing

5
00:00:14.199 --> 00:00:16.120
<v Speaker 2>the sum total of human knowledgy, yeah.

6
00:00:15.919 --> 00:00:19.199
<v Speaker 1>And screenping these massive data sets to feed the next

7
00:00:19.239 --> 00:00:20.960
<v Speaker 1>generation of AI models.

8
00:00:21.120 --> 00:00:25.160
<v Speaker 2>Right. The scale of this automated extraction is honestly staggering.

9
00:00:26.120 --> 00:00:29.160
<v Speaker 2>When we navigate the modern information landscape today, we aren't

10
00:00:29.160 --> 00:00:31.800
<v Speaker 2>talking about like human hands on keyboards anymore.

11
00:00:31.839 --> 00:00:35.479
<v Speaker 1>Definitely not the whole manual process of browsing and copying

12
00:00:35.520 --> 00:00:37.920
<v Speaker 1>and pasting. That's completely obsolete.

13
00:00:38.039 --> 00:00:41.280
<v Speaker 2>Exactly. We are dealing with the deployment of autonomous fleets

14
00:00:41.280 --> 00:00:43.399
<v Speaker 2>that operate at computational speeds.

15
00:00:43.320 --> 00:00:47.359
<v Speaker 1>Which brings us to today's topic. Welcome to the deep dive. Today,

16
00:00:47.399 --> 00:00:50.719
<v Speaker 1>we are going to visualize and master the underlying architecture

17
00:00:50.719 --> 00:00:55.119
<v Speaker 1>of web automation using Python's screepy library. Specifically, we're focusing

18
00:00:55.119 --> 00:00:58.200
<v Speaker 1>on spiders, right, and not the creepy Crawley kind. We

19
00:00:58.320 --> 00:01:01.840
<v Speaker 1>mean the highly specialized digital tools engineered to fetch massive

20
00:01:01.840 --> 00:01:03.280
<v Speaker 1>amounts of data from the Internet.

21
00:01:03.359 --> 00:01:05.799
<v Speaker 2>And I think it's important to note that understanding what

22
00:01:05.920 --> 00:01:09.439
<v Speaker 2>tools are available is just as critical as knowing why

23
00:01:09.480 --> 00:01:12.000
<v Speaker 2>we use them to navigate all this information.

24
00:01:11.560 --> 00:01:15.319
<v Speaker 1>Overload, for sure, and since we are building mental models today,

25
00:01:15.519 --> 00:01:19.480
<v Speaker 1>you don't need a screen. We are doing a purely audio,

26
00:01:19.879 --> 00:01:23.640
<v Speaker 1>step by step walkthrough of how these tools operate under

27
00:01:23.640 --> 00:01:24.640
<v Speaker 1>the hood, which.

28
00:01:24.480 --> 00:01:26.879
<v Speaker 2>Is a great way to learn it, honestly, because understanding

29
00:01:26.879 --> 00:01:29.719
<v Speaker 2>the structural framework of these tools gives you a distinct

30
00:01:29.719 --> 00:01:34.120
<v Speaker 2>advantage whether you're, say, building pipelines for cybersecurity auditing, or

31
00:01:34.239 --> 00:01:36.000
<v Speaker 2>training large language models.

32
00:01:35.719 --> 00:01:38.719
<v Speaker 1>Or engineering massive data aggregation platforms.

33
00:01:38.799 --> 00:01:42.920
<v Speaker 2>Right exactly, Mastering the scrapey architecture allows you to transform

34
00:01:42.959 --> 00:01:46.560
<v Speaker 2>the chaotic unstructured web into normalized data sets.

35
00:01:46.879 --> 00:01:51.000
<v Speaker 1>So before we instantiate a single Python class or pick

36
00:01:51.000 --> 00:01:53.959
<v Speaker 1>a specific tool, we have to define what a spider

37
00:01:54.000 --> 00:01:55.640
<v Speaker 1>actually is in this context.

38
00:01:55.719 --> 00:01:58.920
<v Speaker 2>Right, in this scrapey ecosystem, a spider isn't the entire

39
00:01:58.959 --> 00:02:02.480
<v Speaker 2>scraping operation. It's actually a very specific Python class where

40
00:02:02.519 --> 00:02:03.959
<v Speaker 2>custom behaviors are defined.

41
00:02:04.120 --> 00:02:07.519
<v Speaker 1>Because the underlying scrapy engine handles all the heavy lifting,

42
00:02:07.599 --> 00:02:12.319
<v Speaker 1>right like the asynchronous network requests the concurrency the scheduling.

43
00:02:12.479 --> 00:02:15.599
<v Speaker 2>That's right, the spider is simply the behavioral blueprint that

44
00:02:15.639 --> 00:02:18.360
<v Speaker 2>you hand over to the engine. It just dictates the

45
00:02:18.400 --> 00:02:21.520
<v Speaker 2>logic for crawling a domain and parsing its pages. They

46
00:02:21.520 --> 00:02:24.080
<v Speaker 2>are organized structures that make it easy for the engine

47
00:02:24.120 --> 00:02:25.120
<v Speaker 2>to retrieve content.

48
00:02:25.280 --> 00:02:28.039
<v Speaker 1>Okay, let's unpack this. If I'm trying to visualize the

49
00:02:28.080 --> 00:02:30.639
<v Speaker 1>scrapy engine's relationship with the spider, I kind of think

50
00:02:30.639 --> 00:02:33.240
<v Speaker 1>of it like sending an assistant to a massive library.

51
00:02:33.360 --> 00:02:36.360
<v Speaker 1>Oh I like that, right, So the engine needs three

52
00:02:36.479 --> 00:02:39.879
<v Speaker 1>fundamental instructions from the spider to function effectively. The first

53
00:02:39.919 --> 00:02:43.039
<v Speaker 1>is the scope. So that's like telling your assistant exactly

54
00:02:43.159 --> 00:02:44.840
<v Speaker 1>which building and floor to go to.

55
00:02:45.080 --> 00:02:48.800
<v Speaker 2>Yes, exactly. In the code, this scope is defined by

56
00:02:48.879 --> 00:02:52.360
<v Speaker 2>understanding what can be crawled. Usually you specify the exact

57
00:02:52.599 --> 00:02:54.840
<v Speaker 2>URLs in a method called start requests.

58
00:02:55.159 --> 00:02:57.840
<v Speaker 1>Got it. So without that starting line, the engine has

59
00:02:57.919 --> 00:02:59.879
<v Speaker 1>no coordinates to initiate.

60
00:02:59.479 --> 00:03:03.599
<v Speaker 2>Anything precisely, the engine needs those initial coordinates to kick

61
00:03:03.639 --> 00:03:06.159
<v Speaker 2>off the asynchronous HTTP requests.

62
00:03:06.439 --> 00:03:09.280
<v Speaker 1>Okay, So once the scope is defined, the engine needs

63
00:03:09.319 --> 00:03:11.319
<v Speaker 1>to know the crawling mechanism right.

64
00:03:11.240 --> 00:03:14.280
<v Speaker 2>Right, which is understanding how it will be crawled.

65
00:03:14.560 --> 00:03:17.439
<v Speaker 1>So, going back to my assistant analogy, the mechanism is

66
00:03:17.479 --> 00:03:20.159
<v Speaker 1>telling them whether to just read the book they find,

67
00:03:20.639 --> 00:03:23.240
<v Speaker 1>or to actually follow the footnotes to go find other

68
00:03:23.319 --> 00:03:24.360
<v Speaker 1>books in the library.

69
00:03:24.639 --> 00:03:27.159
<v Speaker 2>That's a perfect way to put it. You are programming

70
00:03:27.199 --> 00:03:31.960
<v Speaker 2>the logic for discovering new URLs. The spider tells the

71
00:03:32.000 --> 00:03:35.039
<v Speaker 2>engine whether it should follow links on the page or

72
00:03:35.199 --> 00:03:37.560
<v Speaker 2>just look at one specific portion makes sense.

73
00:03:37.639 --> 00:03:40.360
<v Speaker 1>And then the third fundamental aspect is the parsing.

74
00:03:40.879 --> 00:03:43.520
<v Speaker 2>Yes, understanding how the data is extracted.

75
00:03:44.039 --> 00:03:46.919
<v Speaker 1>So telling the library assistant to only write down the

76
00:03:46.960 --> 00:03:50.000
<v Speaker 1>text that's in bold, for example, does that accurately describe

77
00:03:50.000 --> 00:03:51.000
<v Speaker 1>the relationship there?

78
00:03:51.120 --> 00:03:55.000
<v Speaker 2>It absolutely does. The web is inherently unstructured, so parsing

79
00:03:55.080 --> 00:03:58.319
<v Speaker 2>is where you apply specific rules, like telling it exactly

80
00:03:58.319 --> 00:04:00.520
<v Speaker 2>what parts of the page are processed and next distracted

81
00:04:00.759 --> 00:04:01.800
<v Speaker 2>for post processing.

82
00:04:01.960 --> 00:04:05.599
<v Speaker 1>Okay, so we have our three instructions scope, crawling, mechanism,

83
00:04:05.759 --> 00:04:09.000
<v Speaker 1>and parsing. Now let's talk about the most foundational tool

84
00:04:09.080 --> 00:04:11.199
<v Speaker 1>Scripey offers to execute them right.

85
00:04:11.159 --> 00:04:14.520
<v Speaker 2>The base class. It's literally just called screepy dot spider, and.

86
00:04:14.439 --> 00:04:16.439
<v Speaker 1>It's basically a blank canvas, isn't.

87
00:04:16.199 --> 00:04:18.800
<v Speaker 2>It It is it's the simplest category, and because it's

88
00:04:18.800 --> 00:04:21.680
<v Speaker 2>the base class, every other spider must inherit from it.

89
00:04:21.800 --> 00:04:25.000
<v Speaker 1>Whether it's one bundled with scrapey or a custom one

90
00:04:25.000 --> 00:04:26.240
<v Speaker 1>you write yourself exactly.

91
00:04:26.319 --> 00:04:29.279
<v Speaker 2>It provides a default start request method that just sends

92
00:04:29.319 --> 00:04:32.360
<v Speaker 2>requests from a start earl's attribute, and then it calls

93
00:04:32.360 --> 00:04:33.920
<v Speaker 2>a parse method for the responses.

94
00:04:34.519 --> 00:04:38.439
<v Speaker 1>Let's walk through a mental visualization of building this. So first,

95
00:04:38.560 --> 00:04:41.040
<v Speaker 1>obviously you import the scrapee library right.

96
00:04:41.319 --> 00:04:44.120
<v Speaker 2>Then you define the name, which is just the spider's

97
00:04:44.199 --> 00:04:45.519
<v Speaker 2>unique identifier, and.

98
00:04:45.480 --> 00:04:48.800
<v Speaker 1>Then critically, you establish your allowed domains.

99
00:04:49.000 --> 00:04:52.000
<v Speaker 2>Yes, that is so important. It's a list that keeps

100
00:04:52.000 --> 00:04:53.959
<v Speaker 2>the spider from wandering off site.

101
00:04:53.720 --> 00:04:57.199
<v Speaker 1>Because setting aloud domains isn't just about focus, right. The

102
00:04:57.240 --> 00:04:59.079
<v Speaker 1>web is full of spider traps.

103
00:04:58.759 --> 00:05:03.439
<v Speaker 2>Oh, completely infinitely loops of dynamically generated URLs right.

104
00:05:03.480 --> 00:05:07.800
<v Speaker 1>Or interconnected domains. If you forget to configure this hard boundary,

105
00:05:08.240 --> 00:05:11.600
<v Speaker 1>a single outbound link on your target site could send

106
00:05:11.639 --> 00:05:14.319
<v Speaker 1>your scraper careening into external servers and.

107
00:05:14.360 --> 00:05:17.639
<v Speaker 2>Suddenly your local machine is trying to index the entire Internet.

108
00:05:17.839 --> 00:05:21.120
<v Speaker 2>You would saturate your bandwidth and trigger automated IP bands

109
00:05:21.160 --> 00:05:22.160
<v Speaker 2>almost immediately.

110
00:05:22.319 --> 00:05:25.560
<v Speaker 1>Wow. Yeah, so the engine absolutely needs that whitelist. And

111
00:05:25.600 --> 00:05:28.040
<v Speaker 1>then after that you define your start rolls, which is

112
00:05:28.040 --> 00:05:29.879
<v Speaker 1>where the crawling actually begins right.

113
00:05:29.920 --> 00:05:32.839
<v Speaker 2>And finally you define the parse method which processes the

114
00:05:32.920 --> 00:05:34.000
<v Speaker 2>downloaded responses.

115
00:05:34.120 --> 00:05:37.560
<v Speaker 1>So inside that parse method, you could say, iterate through

116
00:05:37.560 --> 00:05:40.399
<v Speaker 1>all the H one tags on a page to get

117
00:05:40.639 --> 00:05:41.920
<v Speaker 1>values using.

118
00:05:41.680 --> 00:05:44.519
<v Speaker 2>The XPath method exactly, and then you could iterate through

119
00:05:44.560 --> 00:05:47.160
<v Speaker 2>the A tags to get all the available links. And

120
00:05:47.240 --> 00:05:50.720
<v Speaker 2>what's cool is it can ultimately return multiple requests and

121
00:05:50.800 --> 00:05:53.839
<v Speaker 2>items from just a single callback.

122
00:05:53.399 --> 00:05:55.839
<v Speaker 1>Which brings up a question for me. If this base

123
00:05:55.920 --> 00:05:59.399
<v Speaker 1>spider doesn't provide any special functionality on its own, like

124
00:05:59.560 --> 00:06:02.079
<v Speaker 1>as it's just a blank canvas, why is it the

125
00:06:02.079 --> 00:06:03.079
<v Speaker 1>most commonly.

126
00:06:02.759 --> 00:06:04.680
<v Speaker 2>Used Well comes down to control.

127
00:06:04.959 --> 00:06:07.920
<v Speaker 1>Is it simply because developers prefer a blank canvas where

128
00:06:07.959 --> 00:06:11.519
<v Speaker 1>they can manually define the exact xpaths and tags rather

129
00:06:11.519 --> 00:06:12.399
<v Speaker 1>than being boxed in.

130
00:06:12.839 --> 00:06:16.920
<v Speaker 2>That's exactly it. The value of flexibility is huge. Writing

131
00:06:17.000 --> 00:06:20.839
<v Speaker 2>customized logic inside the base spider makes perfect sense for

132
00:06:21.000 --> 00:06:27.399
<v Speaker 2>navigating standard chaotic HTML documents. You have total granular control

133
00:06:27.480 --> 00:06:28.319
<v Speaker 2>over the extraction.

134
00:06:28.600 --> 00:06:31.959
<v Speaker 1>Here's where it gets really interesting though, because while a

135
00:06:32.000 --> 00:06:34.600
<v Speaker 1>blank canvas is great, sometimes you don't want to reinvent

136
00:06:34.639 --> 00:06:35.040
<v Speaker 1>the wheel.

137
00:06:35.240 --> 00:06:39.319
<v Speaker 2>Right. If you have a highly specific, standardized task, building

138
00:06:39.319 --> 00:06:40.959
<v Speaker 2>from scratch is just inefficient.

139
00:06:41.079 --> 00:06:43.399
<v Speaker 1>You need a pre built tool. So let's pivot to

140
00:06:43.519 --> 00:06:46.720
<v Speaker 1>data structures that break the standard web paradigm. Like, what

141
00:06:46.839 --> 00:06:49.680
<v Speaker 1>if you're targeting a massive online data set that is

142
00:06:49.839 --> 00:06:51.399
<v Speaker 1>just a giant spreadsheet.

143
00:06:51.519 --> 00:06:54.720
<v Speaker 2>Yeah, like a monolithic XML feed or massive CSV file.

144
00:06:55.240 --> 00:06:57.639
<v Speaker 2>In those cases, you use Crapy's generic spiders.

145
00:06:57.680 --> 00:07:01.040
<v Speaker 1>These are designed for those common scraping use cases exactly.

146
00:07:01.160 --> 00:07:04.360
<v Speaker 2>For example, the XML feed spider, it's specifically designed to

147
00:07:04.399 --> 00:07:07.600
<v Speaker 2>parse XML feeds by iterating through node names.

148
00:07:07.519 --> 00:07:10.000
<v Speaker 1>Rather than trying to load the entire document into memory

149
00:07:10.000 --> 00:07:10.399
<v Speaker 1>at once.

150
00:07:10.600 --> 00:07:12.360
<v Speaker 2>Right, because if you apply a base bider to a

151
00:07:12.360 --> 00:07:15.000
<v Speaker 2>fifty gigabyte file, the engine will try to build a

152
00:07:15.040 --> 00:07:16.720
<v Speaker 2>complete dom tree in RAM.

153
00:07:16.480 --> 00:07:19.079
<v Speaker 1>Which would immediately trigger an out of memory exception.

154
00:07:19.480 --> 00:07:23.839
<v Speaker 2>Boom, your pipeline crashes. So the XML feed spider reads

155
00:07:23.879 --> 00:07:26.839
<v Speaker 2>the stream sequentially and discards nodes as it goes.

156
00:07:26.959 --> 00:07:29.600
<v Speaker 1>That is so smart, and I'm guessing the CSF feed

157
00:07:29.639 --> 00:07:31.720
<v Speaker 1>spider works similarly yep.

158
00:07:32.000 --> 00:07:34.759
<v Speaker 2>It's designed for iterating through rows rather than nodes. You

159
00:07:34.879 --> 00:07:37.279
<v Speaker 2>just give it the delimiter and it executes a callback

160
00:07:37.319 --> 00:07:39.480
<v Speaker 2>for every single row without overloading the system.

161
00:07:39.800 --> 00:07:43.079
<v Speaker 1>So if I'm targeting that massive spreadsheet data set, I

162
00:07:43.079 --> 00:07:46.199
<v Speaker 1>shouldn't waste time programming a base spider to look for

163
00:07:46.360 --> 00:07:49.759
<v Speaker 1>HTM tags, I should just deploy the CSD feed spider

164
00:07:49.800 --> 00:07:51.439
<v Speaker 1>to immediately start reading rows.

165
00:07:51.720 --> 00:07:52.160
<v Speaker 2>Exactly.

166
00:07:52.240 --> 00:07:55.399
<v Speaker 1>It's about picking the right heavy machinery for the specific.

167
00:07:54.959 --> 00:07:58.240
<v Speaker 2>Terrain, perfectly said. And there's one more generic spider worth

168
00:07:58.279 --> 00:08:00.480
<v Speaker 2>mentioning here, which is the site map spider.

169
00:08:00.720 --> 00:08:03.839
<v Speaker 1>Oh right, because sometimes the site is standard HTML, but

170
00:08:03.879 --> 00:08:05.759
<v Speaker 1>it's just an absolute labyrinth.

171
00:08:05.360 --> 00:08:08.319
<v Speaker 2>Yes, sprawling e commerce domains with millions of pages and

172
00:08:08.560 --> 00:08:11.800
<v Speaker 2>terrible internal linking. The sitemap spider allows you to crawl

173
00:08:11.800 --> 00:08:14.439
<v Speaker 2>sites by discovering URLs via their site.

174
00:08:14.160 --> 00:08:15.839
<v Speaker 1>Maps, including nested site maps.

175
00:08:15.959 --> 00:08:19.160
<v Speaker 2>Right. It can discover sitemap URLs directly from the robots

176
00:08:19.199 --> 00:08:23.079
<v Speaker 2>dot txt file, so you bypass the messy page level

177
00:08:23.199 --> 00:08:24.879
<v Speaker 2>HTML navigation entirely.

178
00:08:25.160 --> 00:08:29.399
<v Speaker 1>Okay, so those are amazing for specific file types XML, CSV,

179
00:08:29.879 --> 00:08:34.919
<v Speaker 1>site maps. But what if we face a sprawling, interconnected

180
00:08:34.960 --> 00:08:38.519
<v Speaker 1>network of standard HTML that doesn't have a reliable map.

181
00:08:38.600 --> 00:08:39.919
<v Speaker 2>Ah, then you need the heavy lifter.

182
00:08:40.120 --> 00:08:43.519
<v Speaker 1>Right, what is the best pre built tool for crawling

183
00:08:43.679 --> 00:08:45.559
<v Speaker 1>regular complex websites?

184
00:08:45.879 --> 00:08:48.000
<v Speaker 2>That would be the crawl spider. It's the go to

185
00:08:48.159 --> 00:08:51.720
<v Speaker 2>subclass for regular websites because it provides a highly convenient

186
00:08:51.759 --> 00:08:55.080
<v Speaker 2>mechanism for following links based on a set of defined rules.

187
00:08:55.240 --> 00:08:57.720
<v Speaker 1>Okay, let's break down the properties of this cross spider.

188
00:08:57.799 --> 00:08:58.559
<v Speaker 1>You mentioned rules.

189
00:08:58.639 --> 00:09:01.360
<v Speaker 2>Yes, rules are a list of one or more rule

190
00:09:01.440 --> 00:09:03.480
<v Speaker 2>objects that define the crawling behavior.

191
00:09:03.559 --> 00:09:06.720
<v Speaker 1>And it also has a parse start earl method right.

192
00:09:06.559 --> 00:09:09.480
<v Speaker 2>Exactly, which is called for the start url responses yeah

193
00:09:09.559 --> 00:09:12.240
<v Speaker 2>and returns an item, a request or an interval. But

194
00:09:12.320 --> 00:09:15.240
<v Speaker 2>the real magic is the link extractor right. The link extractor,

195
00:09:15.240 --> 00:09:17.879
<v Speaker 2>it's an object that defines exactly how links are extracted

196
00:09:17.879 --> 00:09:21.200
<v Speaker 2>from each crawl page. It generates a request containing the

197
00:09:21.240 --> 00:09:21.840
<v Speaker 2>links text.

198
00:09:22.039 --> 00:09:24.159
<v Speaker 1>Let's do a mental walkthrough of how you'd implement this

199
00:09:24.240 --> 00:09:27.639
<v Speaker 1>in the code. So you import scrapeye, you import crawl spider,

200
00:09:27.759 --> 00:09:29.000
<v Speaker 1>rule and link extractor.

201
00:09:29.240 --> 00:09:33.000
<v Speaker 2>Right. You define your standard name, allowed domains, and start earls,

202
00:09:33.279 --> 00:09:33.879
<v Speaker 2>and then.

203
00:09:33.840 --> 00:09:36.759
<v Speaker 1>You set a rule. So, for example, you could use

204
00:09:36.799 --> 00:09:43.240
<v Speaker 1>a link extractor looking specifically for URL's containing product dot php, yep, and.

205
00:09:43.240 --> 00:09:46.159
<v Speaker 2>When that URL is found, it triggers a parse item.

206
00:09:45.919 --> 00:09:50.240
<v Speaker 1>Method and inside that method you create a new scrapey

207
00:09:50.360 --> 00:09:53.519
<v Speaker 1>item object. You assign the product name to the name

208
00:09:53.679 --> 00:09:56.360
<v Speaker 1>property and maybe assign the meta value of the link

209
00:09:56.399 --> 00:09:59.519
<v Speaker 1>text to the link text property, and then return the object.

210
00:09:59.600 --> 00:10:01.840
<v Speaker 2>That's the exact implementation structure. Yes.

211
00:10:01.879 --> 00:10:04.320
<v Speaker 1>Wait, you mentioned something earlier about the rules property that

212
00:10:04.360 --> 00:10:06.120
<v Speaker 1>I want to push back on. I'm sure you said

213
00:10:06.200 --> 00:10:09.639
<v Speaker 1>that if multiple rules match the same link, only the

214
00:10:09.639 --> 00:10:12.120
<v Speaker 1>first one is used according to the order they are defined.

215
00:10:12.279 --> 00:10:14.600
<v Speaker 2>Yes, that is correct. It evaluates sequentially.

216
00:10:14.840 --> 00:10:17.639
<v Speaker 1>That sounds like rule ordering is incredibly critical. I mean,

217
00:10:17.679 --> 00:10:19.559
<v Speaker 1>if you put a broad rule at the top of

218
00:10:19.559 --> 00:10:22.440
<v Speaker 1>your list, you might accidentally trigger the wrong extraction and

219
00:10:22.480 --> 00:10:26.559
<v Speaker 1>skip your highly specific rules entirely. Yes, So what does

220
00:10:26.600 --> 00:10:29.159
<v Speaker 1>this all mean for how carefully we have to plan

221
00:10:29.240 --> 00:10:31.720
<v Speaker 1>our logic before we even write the code.

222
00:10:31.919 --> 00:10:34.399
<v Speaker 2>What's fascinating here is that you've hit on this single

223
00:10:34.480 --> 00:10:38.279
<v Speaker 2>most common point of failure for developers using this tool.

224
00:10:38.759 --> 00:10:41.399
<v Speaker 1>Really, just the order of the rule, just the order.

225
00:10:41.840 --> 00:10:45.039
<v Speaker 2>Designing a cross spider is essentially designing a strict filtering

226
00:10:45.080 --> 00:10:50.960
<v Speaker 2>funnel because of that, sequential evaluation. Logical sequencing is absolutely essential.

227
00:10:51.200 --> 00:10:53.360
<v Speaker 1>So if I have a broad catchal rule at index

228
00:10:53.480 --> 00:10:55.919
<v Speaker 1>zero just to map the site, it swallows.

229
00:10:55.600 --> 00:10:59.080
<v Speaker 2>The url the engine applies that broad rule stops evaluating

230
00:10:59.120 --> 00:11:02.519
<v Speaker 2>the rest of the list and your highly specific product

231
00:11:02.639 --> 00:11:06.240
<v Speaker 2>extraction rules sitting below. It never even sees the link.

232
00:11:06.360 --> 00:11:08.840
<v Speaker 1>Oh wow, So you could end up downloading thousands of

233
00:11:08.840 --> 00:11:11.759
<v Speaker 1>pages and returning a completely empty data set just because

234
00:11:11.759 --> 00:11:12.759
<v Speaker 1>of the order of your list.

235
00:11:12.960 --> 00:11:17.279
<v Speaker 2>Exactly, you must place your narrowest, most specific edge cases

236
00:11:17.279 --> 00:11:20.480
<v Speaker 2>at the very top and your broadcatch all rules at

237
00:11:20.480 --> 00:11:21.559
<v Speaker 2>the absolute bottom.

238
00:11:21.679 --> 00:11:24.000
<v Speaker 1>That is such a critical detail. So we've covered the

239
00:11:24.159 --> 00:11:26.600
<v Speaker 1>entire operational spectrum of web automation today.

240
00:11:26.679 --> 00:11:27.200
<v Speaker 2>We really have.

241
00:11:27.360 --> 00:11:30.279
<v Speaker 1>We started by understanding the three core aspects of a

242
00:11:30.279 --> 00:11:33.960
<v Speaker 1>spider its scope, its mechanism, and its parsing.

243
00:11:34.200 --> 00:11:37.120
<v Speaker 2>We built from the base scrapey dot spider and looked

244
00:11:37.120 --> 00:11:38.159
<v Speaker 2>at its flexibility.

245
00:11:38.399 --> 00:11:43.039
<v Speaker 1>Then we utilize the generic tools. The generic spiders for XML,

246
00:11:43.240 --> 00:11:44.559
<v Speaker 1>CSV and site.

247
00:11:44.320 --> 00:11:46.679
<v Speaker 2>Maps save massive amounts of compute and memory.

248
00:11:46.799 --> 00:11:50.159
<v Speaker 1>Yeah, and finally we unleashed the rule based crawl spider

249
00:11:50.240 --> 00:11:51.799
<v Speaker 1>to automate heavy reb navigation.

250
00:11:52.360 --> 00:11:55.399
<v Speaker 2>These tools really do save massive amounts of time and

251
00:11:55.480 --> 00:11:58.200
<v Speaker 2>manual effort for programmers and data enthusiasts.

252
00:11:58.320 --> 00:12:01.679
<v Speaker 1>They are literal superpowers. But before we wrap up, I

253
00:12:01.720 --> 00:12:04.399
<v Speaker 1>want to leave you the listener with a broader technical

254
00:12:04.519 --> 00:12:05.600
<v Speaker 1>question to ponder.

255
00:12:05.799 --> 00:12:06.519
<v Speaker 2>Yeah, let's hear it.

256
00:12:07.000 --> 00:12:10.279
<v Speaker 1>If a crawl spider can be perfectly programmed to blindly

257
00:12:10.360 --> 00:12:13.840
<v Speaker 1>follow rules and extract text, what happens to our data

258
00:12:13.879 --> 00:12:18.399
<v Speaker 1>integrity when the underlying architecture of a website secretly changes

259
00:12:18.440 --> 00:12:20.440
<v Speaker 1>its HTML structure overnight.

260
00:12:20.519 --> 00:12:22.720
<v Speaker 2>Oh that is the nightmare scenario, right.

261
00:12:22.960 --> 00:12:25.720
<v Speaker 1>It really highlights the fragility of web scraping and the

262
00:12:25.759 --> 00:12:28.320
<v Speaker 1>need for constant, ongoing maintenance.

263
00:12:27.879 --> 00:12:30.960
<v Speaker 2>Of your code. Absolutely, you can build the perfect pipeline,

264
00:12:31.000 --> 00:12:34.080
<v Speaker 2>but if the target changes its CSS classes, your perfectly

265
00:12:34.120 --> 00:12:36.360
<v Speaker 2>extracted data sets suddenly turns to garbage.

266
00:12:36.399 --> 00:12:39.320
<v Speaker 1>Exactly something to think about now. To reinforce everything we've

267
00:12:39.320 --> 00:12:41.519
<v Speaker 1>talked about before we go, let's test your memory.

268
00:12:41.600 --> 00:12:43.120
<v Speaker 2>Well, pop quiz yep.

269
00:12:43.840 --> 00:12:45.879
<v Speaker 1>If you are sitting down right now to write a

270
00:12:45.919 --> 00:12:49.759
<v Speaker 1>brand new scrapey spider, can you name the three fundamental

271
00:12:49.799 --> 00:12:52.159
<v Speaker 1>aspects you must define before you start?

272
00:12:52.639 --> 00:12:54.000
<v Speaker 2>Give a second, got your answer?

273
00:12:54.200 --> 00:12:57.759
<v Speaker 1>Remember you need to define its scope, its crawling mechanism,

274
00:12:57.919 --> 00:12:59.320
<v Speaker 1>and how it handles parsing.

275
00:13:00.159 --> 00:13:01.879
<v Speaker 2>Those three and you control the pipeline.

276
00:13:01.919 --> 00:13:04.320
<v Speaker 1>Well said. Thanks for joining us on this deep dive,

277
00:13:04.440 --> 00:13:07.120
<v Speaker 1>keep building, keep exploring, and we'll see you next time.