WEBVTT

1
00:00:00.120 --> 00:00:04.599
<v Speaker 1>Imagine for a second that you've just engineered this really

2
00:00:04.679 --> 00:00:05.759
<v Speaker 1>highly efficient web.

3
00:00:05.559 --> 00:00:07.639
<v Speaker 2>Scraper, all the dream scenario.

4
00:00:07.360 --> 00:00:10.279
<v Speaker 1>Right, You've deployed it, and it is just tearing through

5
00:00:10.359 --> 00:00:14.279
<v Speaker 1>a target website, executing perfectly. I mean, it's crawling, pagination,

6
00:00:14.359 --> 00:00:18.120
<v Speaker 1>it's parsing the HTML, yielding thousands of data points a minute.

7
00:00:18.199 --> 00:00:22.239
<v Speaker 2>Yeah, you're getting titles, pricing matrices, inventory status.

8
00:00:21.800 --> 00:00:26.719
<v Speaker 1>Exactly, just an absolute flood of structured data. But then

9
00:00:26.960 --> 00:00:30.440
<v Speaker 1>you hit this architectural bottleneck. Your scraper is generating all

10
00:00:30.440 --> 00:00:34.159
<v Speaker 1>this beautiful data in memory, but you haven't actually implemented

11
00:00:34.200 --> 00:00:37.840
<v Speaker 1>a robust, scalable way to store it. So the scraper finishes,

12
00:00:37.880 --> 00:00:41.079
<v Speaker 1>the process terminates in poof all that data just vanishes

13
00:00:41.119 --> 00:00:41.719
<v Speaker 1>into the void.

14
00:00:41.920 --> 00:00:44.719
<v Speaker 2>It's heartbreaking, and you know it happens way more often

15
00:00:44.719 --> 00:00:47.759
<v Speaker 2>than you might think, even with really experienced developers. Really,

16
00:00:47.920 --> 00:00:50.759
<v Speaker 2>oh totally, because you can get so hyper focused on

17
00:00:50.799 --> 00:00:54.880
<v Speaker 2>the extraction logic, right, like navigating the dom, handling proxies,

18
00:00:54.920 --> 00:00:58.600
<v Speaker 2>writing those flawless X path selectors. Yeah, the fun stuff, exactly,

19
00:00:58.640 --> 00:01:01.280
<v Speaker 2>the fun stuff. And then when the actual data destination

20
00:01:01.399 --> 00:01:02.159
<v Speaker 2>just becomes well an.

21
00:01:02.119 --> 00:01:05.840
<v Speaker 1>Afterthought, which brings us to today's deep dive. We are

22
00:01:05.840 --> 00:01:10.640
<v Speaker 1>tackling this specific architectural challenge by mastering the native data

23
00:01:10.680 --> 00:01:15.480
<v Speaker 1>export mechanisms within the Scrapey framework, because I mean, extracting

24
00:01:15.560 --> 00:01:19.239
<v Speaker 1>data asynchronously is a great skill, obviously, absolutely, but safely

25
00:01:19.280 --> 00:01:22.879
<v Speaker 1>and efficiently storing that data at scale that is what

26
00:01:22.959 --> 00:01:25.760
<v Speaker 1>makes the system production ready. So we're going to break

27
00:01:25.760 --> 00:01:27.840
<v Speaker 1>down the mechanics the Scrapy feedexporter.

28
00:01:28.120 --> 00:01:31.560
<v Speaker 2>Yeah, and the term feedexporter might sound like just a

29
00:01:31.640 --> 00:01:35.599
<v Speaker 2>simple utility, but it's actually a pretty sophisticated subsystem. Right

30
00:01:35.879 --> 00:01:39.120
<v Speaker 2>when you're scraping at scale, you need a way to

31
00:01:39.200 --> 00:01:42.799
<v Speaker 2>serialize your Python dictionaries into standard formats and transport them

32
00:01:42.840 --> 00:01:44.920
<v Speaker 2>to a storage back end. And you have to do

33
00:01:44.959 --> 00:01:48.000
<v Speaker 2>all of that without slowing down the actual scraping process.

34
00:01:48.200 --> 00:01:50.959
<v Speaker 1>Okay, so let's frame the engineering problem here for the listener.

35
00:01:51.159 --> 00:01:53.000
<v Speaker 1>I think of it like this. It's like going to

36
00:01:53.040 --> 00:01:54.359
<v Speaker 1>a massive grocery.

37
00:01:53.959 --> 00:01:55.719
<v Speaker 2>Store, right, I like where this is going.

38
00:01:56.280 --> 00:01:58.719
<v Speaker 1>You fill your cart to the absolute brim with all

39
00:01:58.760 --> 00:02:02.280
<v Speaker 1>these great groceries, but you brought no bags, and you

40
00:02:02.280 --> 00:02:03.719
<v Speaker 1>didn't bring a car to take it home.

41
00:02:03.879 --> 00:02:05.359
<v Speaker 2>You're just stranded in the parking lot.

42
00:02:05.760 --> 00:02:09.639
<v Speaker 1>Exactly. The data is useless if you can't transport it properly. Yeah,

43
00:02:09.680 --> 00:02:11.719
<v Speaker 1>but let me play Devil's advocate for a second. If

44
00:02:11.759 --> 00:02:13.960
<v Speaker 1>I'm building a simple Python script and I want to

45
00:02:13.960 --> 00:02:16.199
<v Speaker 1>save some text, yeah, I normally just use you know,

46
00:02:16.199 --> 00:02:18.120
<v Speaker 1>a standard file context manager.

47
00:02:17.879 --> 00:02:20.199
<v Speaker 2>Right, the classic with OpenData dot txt.

48
00:02:20.560 --> 00:02:22.400
<v Speaker 1>Yeah, I just depend my string and move on.

49
00:02:22.560 --> 00:02:23.000
<v Speaker 2>Yeah.

50
00:02:23.039 --> 00:02:27.039
<v Speaker 1>So is relying on that standard manual file write operation

51
00:02:27.439 --> 00:02:31.439
<v Speaker 1>actually bad when using a framework like scrape Or is

52
00:02:31.479 --> 00:02:34.520
<v Speaker 1>this just like programming snobbery? Why do I need to

53
00:02:34.599 --> 00:02:37.800
<v Speaker 1>learn this whole new built in feedexporter mechanism.

54
00:02:37.840 --> 00:02:41.319
<v Speaker 2>That is a completely fair question. It's not snobbery at all. Actually,

55
00:02:41.400 --> 00:02:43.159
<v Speaker 2>the core issue here is concurrency.

56
00:02:43.240 --> 00:02:44.159
<v Speaker 1>Okay, break that down.

57
00:02:44.360 --> 00:02:47.840
<v Speaker 2>So scrapey doesn't run sequentially. It is built on top

58
00:02:47.879 --> 00:02:50.759
<v Speaker 2>of Twisted, which, if you recall, is this highly asynchronous,

59
00:02:50.800 --> 00:02:54.360
<v Speaker 2>event driven networking engine. Right, So your scraper isn't just

60
00:02:54.439 --> 00:02:56.800
<v Speaker 2>processing one web page, waiting for it to finish and

61
00:02:56.840 --> 00:02:59.560
<v Speaker 2>then calmly moving to the next one. It has dozens,

62
00:02:59.599 --> 00:03:04.120
<v Speaker 2>potential hundreds of concurrent requests in flight at any given millisecond.

63
00:03:04.520 --> 00:03:07.599
<v Speaker 1>Ah, I see, So if I have one hundred different

64
00:03:07.639 --> 00:03:11.759
<v Speaker 1>spider callbacks, all yielding data simultaneously, and I try to

65
00:03:11.800 --> 00:03:15.080
<v Speaker 1>manually appen all that data to a single text file

66
00:03:15.360 --> 00:03:16.960
<v Speaker 1>using standard Python file.

67
00:03:16.759 --> 00:03:21.680
<v Speaker 2>Handling, you introduce a massive race condition. Yeah, multiple asynchronous

68
00:03:21.759 --> 00:03:24.879
<v Speaker 2>routines are trying to open, write to, and close the

69
00:03:24.960 --> 00:03:27.719
<v Speaker 2>exact same file pointer at the exact same time.

70
00:03:27.840 --> 00:03:28.960
<v Speaker 1>That sounds like a disaster.

71
00:03:29.159 --> 00:03:32.000
<v Speaker 2>It is. The best case scenario is that your data

72
00:03:32.039 --> 00:03:34.479
<v Speaker 2>gets totally garbled and rows override each.

73
00:03:34.360 --> 00:03:35.719
<v Speaker 1>Other, and the worst case.

74
00:03:35.599 --> 00:03:39.240
<v Speaker 2>The worst case scenario is hard crash due to threadlocking conflicts.

75
00:03:39.520 --> 00:03:41.039
<v Speaker 2>Your whole scraper just dies.

76
00:03:41.280 --> 00:03:43.400
<v Speaker 1>Well. Wait, if I'm a developer, I might say, fine,

77
00:03:43.479 --> 00:03:46.159
<v Speaker 1>I'll just import a standard threading lock. Yeah, lock the file,

78
00:03:46.280 --> 00:03:49.159
<v Speaker 1>right the row and release the lock. I mean, problem solved.

79
00:03:49.400 --> 00:03:51.680
<v Speaker 2>You could implement a lock. Sure. Yeah, Now you've just

80
00:03:51.719 --> 00:03:53.400
<v Speaker 2>created a huge io.

81
00:03:53.319 --> 00:03:55.680
<v Speaker 1>Bottleneck because everyone's waiting in line.

82
00:03:55.599 --> 00:03:58.759
<v Speaker 2>Exactly every time a callback wants to save data, it

83
00:03:58.800 --> 00:04:01.759
<v Speaker 2>has to wait for the discoperation to finish. You've taken

84
00:04:01.840 --> 00:04:05.599
<v Speaker 2>this lightning fast asynchronous event loop and forced it to

85
00:04:05.719 --> 00:04:09.520
<v Speaker 2>just pause for standard disc io. Your scraping speed will

86
00:04:09.560 --> 00:04:10.800
<v Speaker 2>absolutely plumme it.

87
00:04:11.080 --> 00:04:13.639
<v Speaker 1>Wow. Okay, so you're basically putting a single lane toll

88
00:04:13.680 --> 00:04:15.840
<v Speaker 1>booth on a massive data highway.

89
00:04:15.960 --> 00:04:17.399
<v Speaker 2>That is a perfect analogy.

90
00:04:17.560 --> 00:04:20.879
<v Speaker 1>Yes, So, how does the native feedexporter solve this without

91
00:04:20.920 --> 00:04:22.040
<v Speaker 1>blocking the event loop.

92
00:04:22.160 --> 00:04:25.639
<v Speaker 2>Well, the native mechanism utilizes a queuing system and a

93
00:04:25.680 --> 00:04:31.439
<v Speaker 2>synchronous streams. So when your spider yields an item, feedexporter catches, it,

94
00:04:31.560 --> 00:04:35.600
<v Speaker 2>serializes it in the background, and batches the IO operations.

95
00:04:36.000 --> 00:04:36.600
<v Speaker 1>Oh nice.

96
00:04:36.800 --> 00:04:40.360
<v Speaker 2>It manages the concurrency for you, ensuring that the file

97
00:04:40.360 --> 00:04:43.720
<v Speaker 2>pointer is handled safely, and most importantly, the event loop

98
00:04:43.879 --> 00:04:44.879
<v Speaker 2>just keeps spinning.

99
00:04:45.160 --> 00:04:48.360
<v Speaker 1>So we are offloading the IO management entirely to the framework.

100
00:04:48.399 --> 00:04:49.879
<v Speaker 1>That is a massive advantage.

101
00:04:50.040 --> 00:04:52.920
<v Speaker 2>Huge It means less brittle code and way fewer bugs.

102
00:04:53.199 --> 00:04:56.199
<v Speaker 1>Let's dig into the actual architecture of this subsystem then,

103
00:04:56.680 --> 00:05:00.800
<v Speaker 1>because the feedexporter operates using two distinct layers, right, serialization

104
00:05:00.879 --> 00:05:02.160
<v Speaker 1>formats and storage.

105
00:05:01.800 --> 00:05:04.399
<v Speaker 2>Back ends yep, and separating those two layers is what

106
00:05:04.480 --> 00:05:08.680
<v Speaker 2>gives the system all its flexibility. The serialization format dictates

107
00:05:08.680 --> 00:05:11.680
<v Speaker 2>how the Python dictionary is translated into a string of bytes,

108
00:05:12.199 --> 00:05:14.879
<v Speaker 2>and then the storage back end dictates the destination of

109
00:05:14.879 --> 00:05:15.399
<v Speaker 2>those bites.

110
00:05:15.600 --> 00:05:18.439
<v Speaker 1>It's kind of like a shipping department. The serialization carmat

111
00:05:18.560 --> 00:05:22.199
<v Speaker 1>is how you box the item upright, So JSON is

112
00:05:22.240 --> 00:05:26.480
<v Speaker 1>this neatly organized segmented box. CSV is maybe like a

113
00:05:26.519 --> 00:05:29.279
<v Speaker 1>flat envelope. I like that, and the storage back end

114
00:05:29.399 --> 00:05:31.959
<v Speaker 1>is the delivery truck you put it on. Local FS

115
00:05:32.040 --> 00:05:35.000
<v Speaker 1>is a local courier. S three is an international freight plane.

116
00:05:35.120 --> 00:05:37.920
<v Speaker 2>Exactly. They are totally separate concepts in the architecture.

117
00:05:37.959 --> 00:05:41.680
<v Speaker 1>So let's start with the serialization formats. The boxes natively

118
00:05:41.839 --> 00:05:46.360
<v Speaker 1>Scrapey supports Jason, CSV and XML out of the box,

119
00:05:46.759 --> 00:05:49.199
<v Speaker 1>but there's also Jason lines, which I really want to

120
00:05:49.279 --> 00:05:52.720
<v Speaker 1>highlight because it feels incredibly relevant to large scale scraping.

121
00:05:52.879 --> 00:05:56.600
<v Speaker 2>Jason lines is absolutely vital here, So think about what

122
00:05:56.639 --> 00:05:59.720
<v Speaker 2>happens if you can figure your feedexporder to use standard Jason.

123
00:06:00.079 --> 00:06:02.680
<v Speaker 2>The structural requirement of a valid Jason file is that

124
00:06:02.720 --> 00:06:05.519
<v Speaker 2>the entire thing has to be one giant array enclosed

125
00:06:05.519 --> 00:06:06.319
<v Speaker 2>in square.

126
00:06:06.000 --> 00:06:09.639
<v Speaker 1>Brackets, right, and valid Jason expects commas between every single

127
00:06:09.680 --> 00:06:12.399
<v Speaker 1>object yep, but no comma after the very last object,

128
00:06:12.560 --> 00:06:14.319
<v Speaker 1>which is always annoying to code manually.

129
00:06:14.480 --> 00:06:17.959
<v Speaker 2>It is, which means the system either has to really

130
00:06:18.040 --> 00:06:21.560
<v Speaker 2>carefully manipulate the file pointer to inject commas and brackets

131
00:06:21.560 --> 00:06:24.480
<v Speaker 2>on the fly, or and this is worse, it has

132
00:06:24.560 --> 00:06:27.680
<v Speaker 2>to load the entire scraped data set into memory before

133
00:06:27.759 --> 00:06:28.759
<v Speaker 2>dumping it. To the file.

134
00:06:28.959 --> 00:06:32.920
<v Speaker 1>Oh wow. So if you are scraping, say five.

135
00:06:32.759 --> 00:06:35.240
<v Speaker 2>Million records, holding all of that in RAM, just to

136
00:06:35.279 --> 00:06:37.759
<v Speaker 2>write a valid Jason array will trigger an out of

137
00:06:37.759 --> 00:06:40.519
<v Speaker 2>memory air. The kernel will literally just kill your process.

138
00:06:40.720 --> 00:06:42.519
<v Speaker 1>Enter Jason lines exactly.

139
00:06:42.720 --> 00:06:45.600
<v Speaker 2>Json lines, which uses the dad j al extension, solves

140
00:06:45.639 --> 00:06:48.839
<v Speaker 2>this beautifully. Every single line in the file is its

141
00:06:48.920 --> 00:06:52.639
<v Speaker 2>own valid Jason objects, separated by a standard new line character.

142
00:06:52.800 --> 00:06:54.480
<v Speaker 1>So there's no root array at all.

143
00:06:54.399 --> 00:06:56.600
<v Speaker 2>No root array, there are no trailing comma issues. The

144
00:06:56.600 --> 00:06:59.439
<v Speaker 2>feedexporter can instantly stream each item to the disc the

145
00:06:59.519 --> 00:07:02.639
<v Speaker 2>millisec it's screened. It just flushes the memory immediately.

146
00:07:02.759 --> 00:07:05.920
<v Speaker 1>That is such a brilliant design choice for streaming data,

147
00:07:06.000 --> 00:07:07.959
<v Speaker 1>so that covers how the data is boxed out. Let's

148
00:07:08.000 --> 00:07:10.920
<v Speaker 1>look at the storage back ends, the delivery trucks. We

149
00:07:10.959 --> 00:07:13.000
<v Speaker 1>can write to the local FS, which is just your

150
00:07:13.040 --> 00:07:16.319
<v Speaker 1>local hard drive. We can stream it to standard output

151
00:07:16.399 --> 00:07:19.879
<v Speaker 1>in the console, we can send it via FPP, and natively,

152
00:07:19.959 --> 00:07:22.519
<v Speaker 1>we can pipe it directly into Amazon S three, which

153
00:07:22.560 --> 00:07:23.240
<v Speaker 1>is huge.

154
00:07:23.480 --> 00:07:27.279
<v Speaker 2>The S three integration is incredibly powerful for cloud native applications,

155
00:07:27.600 --> 00:07:31.160
<v Speaker 2>but you do have to ensure your environment is configured correctly.

156
00:07:32.120 --> 00:07:35.759
<v Speaker 2>Scrape doesn't reinvent the wheel for ABUS communication, it relies

157
00:07:35.800 --> 00:07:38.319
<v Speaker 2>on standard Python interfaces.

158
00:07:38.040 --> 00:07:41.279
<v Speaker 1>Meaning the AWSSDK, photo three or photocore.

159
00:07:41.399 --> 00:07:44.279
<v Speaker 2>Right, that is the hard requirement. If you attempt to

160
00:07:44.319 --> 00:07:46.879
<v Speaker 2>set an S three destination but you haven't installed those

161
00:07:46.920 --> 00:07:50.800
<v Speaker 2>specific libraries in your virtual environment, the feedex border will

162
00:07:50.800 --> 00:07:51.839
<v Speaker 2>just throw an error.

163
00:07:51.639 --> 00:07:53.240
<v Speaker 1>Because it doesn't know how to talk to Amazon.

164
00:07:53.360 --> 00:07:56.439
<v Speaker 2>Right. It lacks the underlying bridge to authenticate and communicate

165
00:07:56.519 --> 00:07:57.879
<v Speaker 2>with the AWSAPI.

166
00:07:58.199 --> 00:08:00.800
<v Speaker 1>Makes sense? Okay, I have a question about this decoupling

167
00:08:00.839 --> 00:08:02.560
<v Speaker 1>of the two layers because this is where I think

168
00:08:02.560 --> 00:08:05.439
<v Speaker 1>the architecture really shines. Let's say I build a scraper today,

169
00:08:05.800 --> 00:08:08.480
<v Speaker 1>I set it up to serialize as a CSV and

170
00:08:08.639 --> 00:08:10.560
<v Speaker 1>save to my local hard drive for testing.

171
00:08:10.759 --> 00:08:12.720
<v Speaker 2>Okay, standard workflow, right.

172
00:08:13.000 --> 00:08:16.360
<v Speaker 1>But tomorrow we move to production and my data engineering

173
00:08:16.360 --> 00:08:19.720
<v Speaker 1>team says we actually need this as Jason lines and

174
00:08:19.759 --> 00:08:22.519
<v Speaker 1>it needs to land directly in an S three bucket. Ah.

175
00:08:22.639 --> 00:08:23.759
<v Speaker 2>The classic bait.

176
00:08:23.600 --> 00:08:26.560
<v Speaker 1>And switch always So if I'm setting this up for

177
00:08:26.600 --> 00:08:29.360
<v Speaker 1>the first time, does changing the delivery truck from my

178
00:08:29.399 --> 00:08:32.879
<v Speaker 1>local drive to an Amazon server require me to completely

179
00:08:32.919 --> 00:08:36.000
<v Speaker 1>rewrite how I box the data? How much of my

180
00:08:36.200 --> 00:08:40.200
<v Speaker 1>actual spider code, the parsing, the logic the yielding do

181
00:08:40.240 --> 00:08:41.039
<v Speaker 1>I have to rewrite?

182
00:08:41.159 --> 00:08:43.519
<v Speaker 2>You don't touch a single line of your spider code. Wait?

183
00:08:43.559 --> 00:08:44.639
<v Speaker 1>Really, nothing, nothing.

184
00:08:45.240 --> 00:08:48.279
<v Speaker 2>Your Spider's only job is to yield a Python dictionary.

185
00:08:48.639 --> 00:08:52.320
<v Speaker 2>It is completely agnostic about what happens next. They are

186
00:08:52.360 --> 00:08:55.279
<v Speaker 2>completely decoupled. That's the beauty of the framework. You handle

187
00:08:55.320 --> 00:08:58.440
<v Speaker 2>the shift from local CSV to s three JSON lines

188
00:08:58.879 --> 00:09:00.720
<v Speaker 2>entirely through your settings.

189
00:09:00.720 --> 00:09:01.480
<v Speaker 1>Oh that's awesome.

190
00:09:01.559 --> 00:09:05.080
<v Speaker 2>Yeah, it dynamically instantiates the new exporter classes at runtime.

191
00:09:05.159 --> 00:09:07.879
<v Speaker 1>Okay, let's actually walk through what that implementation looks like

192
00:09:07.919 --> 00:09:10.039
<v Speaker 1>in practice. I want to build out a scenario and

193
00:09:10.080 --> 00:09:12.320
<v Speaker 1>trace the data flow step by step for the listener.

194
00:09:12.399 --> 00:09:12.879
<v Speaker 2>Let's do it.

195
00:09:13.120 --> 00:09:15.960
<v Speaker 1>Let's imagine you are building a scraper for a massive

196
00:09:16.120 --> 00:09:20.799
<v Speaker 1>online technical bookstore. Your spider is working perfectly. It's yielding

197
00:09:20.840 --> 00:09:24.720
<v Speaker 1>thousands of dictionaries containing the book title, author, and price. Okay,

198
00:09:25.120 --> 00:09:28.480
<v Speaker 1>but we have a strict budget rule from the business side.

199
00:09:28.559 --> 00:09:31.120
<v Speaker 1>We do not want any expensive books in our final

200
00:09:31.200 --> 00:09:34.559
<v Speaker 1>data set. Anything priced at fifty dollars or higher needs

201
00:09:34.559 --> 00:09:37.399
<v Speaker 1>to be discarded before it ever reaches the export file.

202
00:09:37.480 --> 00:09:40.559
<v Speaker 2>Gotcha. So this introduces a really critical component of the

203
00:09:40.639 --> 00:09:42.799
<v Speaker 2>data flow, the item pipeline.

204
00:09:42.879 --> 00:09:43.679
<v Speaker 1>Right, the pipeline.

205
00:09:43.679 --> 00:09:46.559
<v Speaker 2>The pipeline sits exactly between the spider yielding the data

206
00:09:46.600 --> 00:09:49.480
<v Speaker 2>and the feed exporter saving the data. It acts as

207
00:09:49.519 --> 00:09:51.480
<v Speaker 2>a series of filters and processors.

208
00:09:51.639 --> 00:09:54.320
<v Speaker 1>Okay, step one. If I open up the pipelines dot

209
00:09:54.360 --> 00:09:57.679
<v Speaker 1>pifile in my scrapey project, how do I structure the

210
00:09:57.759 --> 00:10:01.080
<v Speaker 1>logic to act as a gatekeeper for these book prices.

211
00:10:01.240 --> 00:10:04.240
<v Speaker 2>So inside pipelines dot pie you define a custom class,

212
00:10:04.399 --> 00:10:07.879
<v Speaker 2>and every pipeline class requires a specific method called process item,

213
00:10:07.919 --> 00:10:10.919
<v Speaker 2>which literally takes this scraped item as an argument. Okay,

214
00:10:11.080 --> 00:10:13.679
<v Speaker 2>inside this method you write your conditional logic. You just

215
00:10:13.759 --> 00:10:16.120
<v Speaker 2>check the price field of the item, right.

216
00:10:15.960 --> 00:10:17.600
<v Speaker 1>So if the price is less than fifty, I just

217
00:10:17.919 --> 00:10:20.000
<v Speaker 1>return the item, letting it continue down.

218
00:10:19.840 --> 00:10:20.759
<v Speaker 2>The chain exactly.

219
00:10:21.120 --> 00:10:24.080
<v Speaker 1>But if it's fifty or higher, what's the mechanism to

220
00:10:24.159 --> 00:10:27.000
<v Speaker 1>kill it? I mean, I can't just return.

221
00:10:26.759 --> 00:10:30.759
<v Speaker 2>None, right, good instinct. Returning none would actually pass a

222
00:10:30.919 --> 00:10:32.799
<v Speaker 2>null object down the chain.

223
00:10:32.559 --> 00:10:33.519
<v Speaker 1>Which would break things.

224
00:10:33.679 --> 00:10:37.399
<v Speaker 2>Yeah, it could cause errors in the exporter. Instead, scrape

225
00:10:37.559 --> 00:10:40.559
<v Speaker 2>provides a specific exception class called drop item.

226
00:10:40.799 --> 00:10:41.399
<v Speaker 1>Drop item.

227
00:10:41.720 --> 00:10:44.440
<v Speaker 2>Yeah, so you write a statement to raise drop item,

228
00:10:44.480 --> 00:10:46.840
<v Speaker 2>and you can even include a message like you know,

229
00:10:47.000 --> 00:10:48.480
<v Speaker 2>prize succeeds threshold.

230
00:10:48.720 --> 00:10:52.200
<v Speaker 1>What actually happens under the hood when that exception is raised.

231
00:10:52.360 --> 00:10:56.240
<v Speaker 2>The scrapy engine catches that specific exception and it instantly

232
00:10:56.320 --> 00:10:59.679
<v Speaker 2>halts any further processing of that item in the pipeline chain.

233
00:11:00.000 --> 00:11:01.360
<v Speaker 1>So it just surgically removes it.

234
00:11:01.440 --> 00:11:04.519
<v Speaker 2>Exactly, and it logs the item as dropped in your

235
00:11:04.559 --> 00:11:08.519
<v Speaker 2>console analytics, which is fantastic for debugging, and most importantly,

236
00:11:08.559 --> 00:11:10.799
<v Speaker 2>it prevents the item from ever being passed to the

237
00:11:10.840 --> 00:11:11.639
<v Speaker 2>feed exporter.

238
00:11:12.039 --> 00:11:13.399
<v Speaker 1>Just to make sure I have the sequence right in

239
00:11:13.440 --> 00:11:16.120
<v Speaker 1>my head. If I drop an item in the pipeline

240
00:11:16.600 --> 00:11:19.440
<v Speaker 1>using drop item, the feed exporter never even sees it, right,

241
00:11:19.759 --> 00:11:22.440
<v Speaker 1>It's basically bouncing the expensive books at the door before

242
00:11:22.440 --> 00:11:23.559
<v Speaker 1>they even get into the club.

243
00:11:23.759 --> 00:11:26.679
<v Speaker 2>That is exactly what it's doing. The feed exporter is

244
00:11:26.720 --> 00:11:30.080
<v Speaker 2>completely insulated from the dirty data. It only receives the

245
00:11:30.159 --> 00:11:32.240
<v Speaker 2>surviving affordable books.

246
00:11:32.559 --> 00:11:35.720
<v Speaker 1>Okay, so step two we need to actually configure the

247
00:11:35.759 --> 00:11:39.639
<v Speaker 1>exporter to save these surviving books to adjason file. So

248
00:11:39.679 --> 00:11:42.519
<v Speaker 1>we move over to the settings dot PI file. I

249
00:11:42.559 --> 00:11:45.200
<v Speaker 1>know I don't need to write any custom file handling logic,

250
00:11:45.559 --> 00:11:47.879
<v Speaker 1>but how exactly do I tell scrape you what to do?

251
00:11:48.080 --> 00:11:51.320
<v Speaker 2>It's surprisingly simple. You can figure the exporter by declaring

252
00:11:51.360 --> 00:11:54.919
<v Speaker 2>global variables right there in settings dot pi I. Crapie's

253
00:11:54.919 --> 00:11:56.600
<v Speaker 2>engine reads this file at startup.

254
00:11:56.679 --> 00:11:57.000
<v Speaker 1>Okay.

255
00:11:57.240 --> 00:11:59.879
<v Speaker 2>To define your serialization format, you just set a very

256
00:12:00.080 --> 00:12:02.600
<v Speaker 2>well named feed format and assign it a string value

257
00:12:02.759 --> 00:12:04.200
<v Speaker 2>like json or CSV.

258
00:12:04.480 --> 00:12:06.000
<v Speaker 1>That's it. Just feed format Jason.

259
00:12:06.200 --> 00:12:07.120
<v Speaker 2>That's it, okay.

260
00:12:07.120 --> 00:12:08.399
<v Speaker 1>And to tell it where to save.

261
00:12:08.399 --> 00:12:11.440
<v Speaker 2>You set feederi. If you assign fiddery a simple string

262
00:12:11.519 --> 00:12:15.000
<v Speaker 2>like books dot json, the engine parses that string. And

263
00:12:15.039 --> 00:12:18.080
<v Speaker 2>because there is no protocol prefix like you know S

264
00:12:18.120 --> 00:12:22.080
<v Speaker 2>three or FTP, the underlying opponent manager automatically infers that

265
00:12:22.120 --> 00:12:25.240
<v Speaker 2>you want to use the local fs storage back end class.

266
00:12:25.519 --> 00:12:30.159
<v Speaker 1>That dynamic instantiation is fascinating to me. By literally just

267
00:12:30.240 --> 00:12:34.960
<v Speaker 1>typing feederiiebooks dot Jason, I'm essentially triggering Scrapie to import

268
00:12:34.960 --> 00:12:38.120
<v Speaker 1>the local filesystem back end instantiated, open a file pointer,

269
00:12:38.440 --> 00:12:41.600
<v Speaker 1>attach the Jason serialization class to it, and map it

270
00:12:41.639 --> 00:12:43.320
<v Speaker 1>to the end of the item pipeline.

271
00:12:43.480 --> 00:12:46.879
<v Speaker 2>Yeah. All of that complexity is completely abstracted away into

272
00:12:47.039 --> 00:12:48.799
<v Speaker 2>just those two settings variables.

273
00:12:48.799 --> 00:12:52.080
<v Speaker 1>Step three the magic switch. I run the spider.

274
00:12:52.320 --> 00:12:53.919
<v Speaker 2>Yeah, and when you run it, you will see the

275
00:12:53.960 --> 00:12:57.240
<v Speaker 2>asynchronous engines spinning up in your console. You'll watch your

276
00:12:57.279 --> 00:12:59.919
<v Speaker 2>log output and you'll see the items dropped count tick

277
00:13:00.159 --> 00:13:02.159
<v Speaker 2>up as they hit your pipeline exception, which.

278
00:13:02.000 --> 00:13:05.240
<v Speaker 1>Shows me exactly how many over fifty dollars books were tossed.

279
00:13:05.000 --> 00:13:08.399
<v Speaker 2>Out exactly, and boom, you'll see a perfectly structured books

280
00:13:08.399 --> 00:13:12.200
<v Speaker 2>dot json file materialize in your director, safely streamed without

281
00:13:12.240 --> 00:13:13.279
<v Speaker 2>blocking the event loop.

282
00:13:13.360 --> 00:13:15.960
<v Speaker 1>But what if my boss suddenly wants XML instead.

283
00:13:16.200 --> 00:13:18.720
<v Speaker 2>Like we said earlier, you don't rewrite the scraper. No

284
00:13:18.879 --> 00:13:23.200
<v Speaker 2>manual string concatenation, no building XML trees in memory, no

285
00:13:23.360 --> 00:13:25.320
<v Speaker 2>worrying about escaping special characters.

286
00:13:25.440 --> 00:13:28.399
<v Speaker 1>I just go back to settings dot pi, change feed

287
00:13:28.440 --> 00:13:31.759
<v Speaker 1>format to XML, update the URI to that XML, and

288
00:13:31.840 --> 00:13:32.559
<v Speaker 1>run it again.

289
00:13:32.639 --> 00:13:37.240
<v Speaker 2>And you have books dot xml. The exporter classes handle

290
00:13:37.279 --> 00:13:40.879
<v Speaker 2>the translation seamlessly. You are letting the framework do the

291
00:13:40.919 --> 00:13:44.639
<v Speaker 2>heavy lifting here. By combining the filtering power of pipelines

292
00:13:44.679 --> 00:13:47.840
<v Speaker 2>dot PUI with the native routing of the feedexporter via

293
00:13:47.879 --> 00:13:52.639
<v Speaker 2>settings dot pi, you create a highly scalable, robust data

294
00:13:52.679 --> 00:13:54.039
<v Speaker 2>extraction system.

295
00:13:54.080 --> 00:13:58.360
<v Speaker 1>This architecture completely changes how you approach scraping projects. Yeah,

296
00:13:58.399 --> 00:14:00.960
<v Speaker 1>you really stop thinking about how to opening closed files.

297
00:14:01.120 --> 00:14:04.360
<v Speaker 1>You start thinking purely about dataflow and component routing exactly.

298
00:14:04.399 --> 00:14:07.399
<v Speaker 2>It elevates you from just writing scripts to designing real systems.

299
00:14:07.440 --> 00:14:09.840
<v Speaker 1>All right, Before we wrap up, we promise the listener

300
00:14:09.840 --> 00:14:13.559
<v Speaker 1>a quick interactive review exercise to reinforce everything we've covered today.

301
00:14:13.679 --> 00:14:17.159
<v Speaker 2>Yes, let's map out a new architecture. Mentally, imagine you

302
00:14:17.200 --> 00:14:20.279
<v Speaker 2>are tasked with scraping a real estate website. Your spider

303
00:14:20.399 --> 00:14:23.600
<v Speaker 2>is successfully extracting property listings. Okay, got it, But the

304
00:14:23.639 --> 00:14:26.639
<v Speaker 2>business requirement dictates two things. First, we only want to

305
00:14:26.679 --> 00:14:29.919
<v Speaker 2>save houses that cost under three hundred thousand dollars. Second,

306
00:14:29.960 --> 00:14:33.000
<v Speaker 2>the final data set must be delivered as a CSV file. Okay,

307
00:14:33.279 --> 00:14:36.080
<v Speaker 2>Based on our deep dive today, which two specific files

308
00:14:36.080 --> 00:14:37.879
<v Speaker 2>would you need to modify and what would you do

309
00:14:37.960 --> 00:14:38.320
<v Speaker 2>in each?

310
00:14:38.519 --> 00:14:41.519
<v Speaker 1>All right, give the listener a beat to think about it. Okay, So,

311
00:14:41.879 --> 00:14:45.320
<v Speaker 1>tracing the data flow, the spider parses the real estate

312
00:14:45.360 --> 00:14:49.039
<v Speaker 1>site yields a dictionary with the property data. That dictionary

313
00:14:49.080 --> 00:14:52.519
<v Speaker 1>immediately enters the item pipeline. So file number one pipelines

314
00:14:52.559 --> 00:14:55.240
<v Speaker 1>dot PI correct. I'd use a drop item class in

315
00:14:55.320 --> 00:14:58.200
<v Speaker 1>pipelines dot pi to filter the price if it's three

316
00:14:58.279 --> 00:15:01.799
<v Speaker 1>hundred thousand dollars or more, raise drop item kill it

317
00:15:01.799 --> 00:15:04.200
<v Speaker 1>from memory if it's under pass.

318
00:15:04.000 --> 00:15:06.240
<v Speaker 2>Along spot on and file number two.

319
00:15:06.320 --> 00:15:09.759
<v Speaker 1>File number two is settings dot PI to handle the

320
00:15:09.799 --> 00:15:12.200
<v Speaker 1>CSV requirement, I'd go in there and change the feed

321
00:15:12.240 --> 00:15:14.960
<v Speaker 1>format to CSV, and I had change the feed dry

322
00:15:15.039 --> 00:15:17.320
<v Speaker 1>to whatever my destination file name is perfect.

323
00:15:17.320 --> 00:15:20.279
<v Speaker 2>The logic holds completely. You've isolated the validation logic in

324
00:15:20.320 --> 00:15:23.320
<v Speaker 2>the pipeline and you've abstracted the formatting into the settings.

325
00:15:23.559 --> 00:15:26.159
<v Speaker 2>That is how professional scraping architecture is designed.

326
00:15:26.279 --> 00:15:28.639
<v Speaker 1>It is incredibly satisfying when you see all these decoupled

327
00:15:28.679 --> 00:15:32.639
<v Speaker 1>components working together in an asynchronous stream. But this actually

328
00:15:32.679 --> 00:15:34.639
<v Speaker 1>brings up an interesting edge case that I want to

329
00:15:34.679 --> 00:15:37.320
<v Speaker 1>pose as a provocative final thought for everyone listening.

330
00:15:37.320 --> 00:15:38.480
<v Speaker 2>Oh, I love a good edge case.

331
00:15:38.679 --> 00:15:42.360
<v Speaker 1>Because everything we've exported today, book titles, prices, real estate

332
00:15:42.440 --> 00:15:44.159
<v Speaker 1>data has been text based. Right.

333
00:15:44.320 --> 00:15:48.039
<v Speaker 2>Yeah, standard string data easily mapped to CSVS or JSON.

334
00:15:48.320 --> 00:15:51.279
<v Speaker 1>But web scraping isn't just about text. What happens when

335
00:15:51.320 --> 00:15:53.639
<v Speaker 1>the web data you want to export isn't text at all?

336
00:15:53.759 --> 00:15:57.600
<v Speaker 2>Oh man? That changes the architectural requirements entirely exactly.

337
00:15:58.000 --> 00:16:01.799
<v Speaker 1>What if you are scraping massive binary files like high

338
00:16:01.840 --> 00:16:06.240
<v Speaker 1>resolution TIF images or heavy PDF documents if your spider

339
00:16:06.320 --> 00:16:09.799
<v Speaker 1>yields an item dictionary containing an image URL, the standard

340
00:16:09.799 --> 00:16:13.559
<v Speaker 1>feedexport can easily write that text URL to a CSV file. Sure,

341
00:16:13.720 --> 00:16:16.919
<v Speaker 1>but how would a feedexporter handle actually saving the picture?

342
00:16:17.399 --> 00:16:21.200
<v Speaker 1>How do you configure scrape to resolve that URL asynchronously

343
00:16:21.240 --> 00:16:24.120
<v Speaker 1>download a ten megaby binary payload and save the raw

344
00:16:24.159 --> 00:16:26.919
<v Speaker 1>image to your local drive or S three without blocking

345
00:16:26.960 --> 00:16:27.480
<v Speaker 1>the reactor.

346
00:16:27.600 --> 00:16:30.679
<v Speaker 2>Yeah, standard text serialization formats are definitely not equipped to

347
00:16:30.720 --> 00:16:34.080
<v Speaker 2>handle streaming binary blobs. You have to utilize a completely

348
00:16:34.080 --> 00:16:37.080
<v Speaker 2>different subsystem designed specifically for media.

349
00:16:36.639 --> 00:16:40.440
<v Speaker 1>Right, and that introduces the world of media pipelines, specifically

350
00:16:40.919 --> 00:16:44.720
<v Speaker 1>the images pipeline and the files pipeline. It's a whole

351
00:16:44.720 --> 00:16:47.360
<v Speaker 1>different level of data routing, and we will leave you

352
00:16:47.399 --> 00:16:49.080
<v Speaker 1>to explore that architecture on your own.

353
00:16:49.159 --> 00:16:51.159
<v Speaker 2>It's a fun rabbit hole to go down, for sure.

354
00:16:51.320 --> 00:16:54.000
<v Speaker 1>Definitely, just remember you do not want to be the

355
00:16:54.039 --> 00:16:57.200
<v Speaker 1>developer manually writing text strings to a disc in an

356
00:16:57.240 --> 00:17:01.399
<v Speaker 1>asynchronous framework. Leverage the item pipeline, master the FedEx porter,

357
00:17:01.480 --> 00:17:04.720
<v Speaker 1>and let the architecture handle the heavy lifting until next time.

358
00:17:05.039 --> 00:17:06.559
<v Speaker 1>Keep optimizing your data flows