WEBVTT

1
00:00:00.120 --> 00:00:03.279
<v Speaker 1>Imagine for a second turning the entire Internet into your

2
00:00:03.319 --> 00:00:06.599
<v Speaker 1>own personal, highly structural database.

3
00:00:06.679 --> 00:00:08.320
<v Speaker 2>Oh that is the dream, right right.

4
00:00:08.800 --> 00:00:13.359
<v Speaker 1>I mean, no manual copying, no pasting, just pristine data

5
00:00:13.480 --> 00:00:17.280
<v Speaker 1>running quietly in the background, bring you exactly what you need, which.

6
00:00:17.120 --> 00:00:19.640
<v Speaker 2>Is exactly what we're getting into today, building the digital

7
00:00:19.760 --> 00:00:22.079
<v Speaker 2>robots that actually make that happen exactly.

8
00:00:22.480 --> 00:00:25.839
<v Speaker 1>Welcome to this deep dive. Today we are unpacking web scraping.

9
00:00:25.879 --> 00:00:30.600
<v Speaker 1>We're going step by step from raw Python requests, surviving

10
00:00:30.640 --> 00:00:35.920
<v Speaker 1>the absolute chaos of broken HTML all the way to

11
00:00:36.039 --> 00:00:39.479
<v Speaker 1>computer vision and massive asynchronous crawling.

12
00:00:39.560 --> 00:00:42.320
<v Speaker 2>Yeah. It's a huge topic and it's an incredibly powerful

13
00:00:42.359 --> 00:00:43.960
<v Speaker 2>capability once you wrap your head around it.

14
00:00:44.000 --> 00:00:46.200
<v Speaker 1>Okay, let's unpack this. I like to think of webscraping

15
00:00:46.359 --> 00:00:50.840
<v Speaker 1>like sending a digital robot into a massive, sprawling library.

16
00:00:51.039 --> 00:00:52.000
<v Speaker 2>I like that analogy.

17
00:00:52.079 --> 00:00:53.799
<v Speaker 1>Yeah, because you don't want the robot to just grab

18
00:00:53.840 --> 00:00:55.840
<v Speaker 1>millions of heavy books and dump them all over your desk,

19
00:00:55.960 --> 00:00:58.159
<v Speaker 1>right right. You want to actually read those books. Yeah,

20
00:00:58.359 --> 00:01:01.600
<v Speaker 1>locate the specific paragraphs and bring back only the exact

21
00:01:01.719 --> 00:01:03.840
<v Speaker 1>data points you ask for exactly.

22
00:01:03.920 --> 00:01:05.959
<v Speaker 2>And to get your robot to do that, you have

23
00:01:06.000 --> 00:01:09.640
<v Speaker 2>to realize that web browsers are at their core just

24
00:01:09.840 --> 00:01:13.879
<v Speaker 2>really complex HTTP clients. They're just interpreting.

25
00:01:13.359 --> 00:01:16.680
<v Speaker 1>Markup right, So when we scrape, we're basically stripping away

26
00:01:16.680 --> 00:01:18.200
<v Speaker 1>all the flashy stuff completely.

27
00:01:18.239 --> 00:01:22.040
<v Speaker 2>We strip away the JavaScript engines, the complex rendering pipelines

28
00:01:22.040 --> 00:01:26.040
<v Speaker 2>of Chrome or Safari. We bypass the visual stuff entirely

29
00:01:26.159 --> 00:01:29.120
<v Speaker 2>and just ask the server for the RAM material.

30
00:01:29.280 --> 00:01:31.519
<v Speaker 1>Okay, so how do we actually tell our robot to

31
00:01:31.519 --> 00:01:31.799
<v Speaker 1>do that?

32
00:01:31.879 --> 00:01:33.760
<v Speaker 2>Well, you have to break the whole process down into

33
00:01:33.760 --> 00:01:38.000
<v Speaker 2>two distinct phases. Phase one is the fetch the fetch okay, yeah,

34
00:01:38.079 --> 00:01:40.840
<v Speaker 2>your program literally has to travel to the website server

35
00:01:41.200 --> 00:01:42.640
<v Speaker 2>and request the document.

36
00:01:42.920 --> 00:01:45.920
<v Speaker 1>And if we're doing this programmatically, you know, in Python,

37
00:01:46.319 --> 00:01:48.040
<v Speaker 1>we have a few distinct paths we can take.

38
00:01:48.159 --> 00:01:51.040
<v Speaker 2>We do. If you're writing a script, the heavy favorite,

39
00:01:51.680 --> 00:01:54.359
<v Speaker 2>the incredibly intuitive API that should probably just be your

40
00:01:54.439 --> 00:01:58.359
<v Speaker 2>default choice is a third party library simply called requests.

41
00:01:58.519 --> 00:02:01.200
<v Speaker 1>Oh yeah, request is beloved. I mean it abstracts away

42
00:02:01.200 --> 00:02:02.519
<v Speaker 1>all the complex socket programming.

43
00:02:02.760 --> 00:02:06.040
<v Speaker 2>Yeah right, It handles all the connection pooling and session

44
00:02:06.079 --> 00:02:09.919
<v Speaker 2>management under the hood. It makes your code highly human readable.

45
00:02:10.080 --> 00:02:11.520
<v Speaker 1>But it's not the only way to do it.

46
00:02:11.439 --> 00:02:13.719
<v Speaker 2>No, definitely, not like if you're in an environment where

47
00:02:13.759 --> 00:02:17.520
<v Speaker 2>you absolutely cannot install external packages, you know, for security

48
00:02:17.560 --> 00:02:20.520
<v Speaker 2>reasons or whatever, you have erlib, which.

49
00:02:20.319 --> 00:02:22.479
<v Speaker 1>Comes standard within the base Python distribution.

50
00:02:22.719 --> 00:02:27.599
<v Speaker 2>Exactly, it's pure dependency free Python. Now it is significantly

51
00:02:27.639 --> 00:02:29.840
<v Speaker 2>more verbose. You need a lot more boilerplate code to

52
00:02:29.879 --> 00:02:31.919
<v Speaker 2>get the exact same job done.

53
00:02:32.000 --> 00:02:33.960
<v Speaker 1>Right, It's a bit clunky, yeah it is.

54
00:02:34.560 --> 00:02:38.039
<v Speaker 2>And then if you need incredibly granular control over the

55
00:02:38.120 --> 00:02:41.919
<v Speaker 2>caching of your HTDP requests, there's hgclib.

56
00:02:41.400 --> 00:02:44.240
<v Speaker 1>Too, gotcha. Oh, and I should mention if you aren't

57
00:02:44.240 --> 00:02:47.439
<v Speaker 1>writing a full Python script yet, Like, if you're just

58
00:02:47.479 --> 00:02:50.000
<v Speaker 1>testing an endpoint directly in your terminal, you can always

59
00:02:50.039 --> 00:02:51.759
<v Speaker 1>rely on the classic utility CURL.

60
00:02:52.039 --> 00:02:54.039
<v Speaker 2>Oh. Curl's a classic for a reason. You just pass

61
00:02:54.080 --> 00:02:57.360
<v Speaker 2>it a URL and it dumps the payload right into

62
00:02:57.400 --> 00:02:58.479
<v Speaker 2>your standard output.

63
00:02:58.680 --> 00:03:01.520
<v Speaker 1>So all of these tools, whether it's requests or a CURL,

64
00:03:01.759 --> 00:03:03.960
<v Speaker 1>they all accomplish the same fundamental task for.

65
00:03:03.960 --> 00:03:08.759
<v Speaker 2>Phase one, Right, They format a standard HTTP get request,

66
00:03:09.479 --> 00:03:12.840
<v Speaker 2>manage the network handshake, and basically say to the server, hey,

67
00:03:13.080 --> 00:03:15.719
<v Speaker 2>i'd like the document at this end point please.

68
00:03:15.520 --> 00:03:18.360
<v Speaker 1>And then they download the raw content directly into your

69
00:03:18.400 --> 00:03:19.759
<v Speaker 1>program's allocated memory.

70
00:03:19.879 --> 00:03:22.439
<v Speaker 2>Spot on. But this is where people usually hit a wall.

71
00:03:22.520 --> 00:03:25.080
<v Speaker 1>Yeah, the immediate bottleneck. So let's say I've run my

72
00:03:25.159 --> 00:03:29.240
<v Speaker 1>Python script, the gt request succeeds, and now my terminal

73
00:03:29.319 --> 00:03:32.479
<v Speaker 1>is just flooded with a massive, like ten thousand line

74
00:03:32.520 --> 00:03:33.759
<v Speaker 1>string of raw text.

75
00:03:33.960 --> 00:03:36.400
<v Speaker 2>It's completely chaotic, totally chaotic.

76
00:03:36.479 --> 00:03:39.080
<v Speaker 1>I mean, I can't just read this to find the data.

77
00:03:39.199 --> 00:03:40.840
<v Speaker 1>How does the machine actually make sense of it?

78
00:03:40.919 --> 00:03:43.360
<v Speaker 2>Well, getting the string into your terminal is really only

79
00:03:43.439 --> 00:03:46.280
<v Speaker 2>half the battle. If you stop there, you don't have

80
00:03:46.360 --> 00:03:47.159
<v Speaker 2>structured data.

81
00:03:47.280 --> 00:03:50.639
<v Speaker 1>You haven't extracted the specific sentences from the library book yet.

82
00:03:50.599 --> 00:03:52.360
<v Speaker 2>Exactly, you've just thrown the whole book on the desk.

83
00:03:52.639 --> 00:03:55.639
<v Speaker 2>That massive string is useless until you process it. And

84
00:03:55.680 --> 00:03:58.360
<v Speaker 2>that brings us to phase two, which is parsing.

85
00:03:58.599 --> 00:03:58.919
<v Speaker 1>Parsing.

86
00:03:59.039 --> 00:04:02.599
<v Speaker 2>Okay, yeah, distract the nuggets of information we actually care about.

87
00:04:02.840 --> 00:04:05.319
<v Speaker 2>We have to understand the inherent structure of what we

88
00:04:05.439 --> 00:04:06.159
<v Speaker 2>just downloaded.

89
00:04:06.400 --> 00:04:08.560
<v Speaker 1>But there's a fork in the road right here, isn't there?

90
00:04:08.919 --> 00:04:12.199
<v Speaker 1>Because parsing the modern web is definitely not a one

91
00:04:12.240 --> 00:04:13.719
<v Speaker 1>size fits all operation.

92
00:04:13.759 --> 00:04:17.879
<v Speaker 2>Not at all. We have to clearly separate standard HTML

93
00:04:17.959 --> 00:04:19.439
<v Speaker 2>parsing from DOM parsing.

94
00:04:19.879 --> 00:04:22.240
<v Speaker 1>Let's break that distinction down because it's so critical.

95
00:04:22.480 --> 00:04:24.920
<v Speaker 2>So HTML as it comes over the wire from the server.

96
00:04:25.240 --> 00:04:28.720
<v Speaker 2>It's not just flat linear text. It has this inbuilt

97
00:04:28.800 --> 00:04:31.279
<v Speaker 2>hierarchical tree like structure.

98
00:04:30.920 --> 00:04:32.920
<v Speaker 1>Right, Like you have a root element which branches out

99
00:04:32.959 --> 00:04:33.720
<v Speaker 1>into a head and a.

100
00:04:33.639 --> 00:04:36.839
<v Speaker 2>Body exactly, and then the body branches into structural containers

101
00:04:36.839 --> 00:04:40.319
<v Speaker 2>which branch into paragraphs and so on. Standard HTML parsing

102
00:04:40.399 --> 00:04:43.399
<v Speaker 2>is simply the act of reading that static downloaded text

103
00:04:43.399 --> 00:04:45.360
<v Speaker 2>file and mapping out its family tree.

104
00:04:45.480 --> 00:04:48.639
<v Speaker 1>But the web isn't just static text files anymore, which

105
00:04:48.680 --> 00:04:52.040
<v Speaker 1>brings us to the DOM, the document object model. Right.

106
00:04:52.399 --> 00:04:55.959
<v Speaker 2>When a real web browser receives that static HTML text,

107
00:04:56.199 --> 00:04:59.240
<v Speaker 2>it doesn't just read it. It tokenizes it and translates

108
00:04:59.240 --> 00:05:02.199
<v Speaker 2>it into a live object representation in its active memory.

109
00:05:02.240 --> 00:05:04.240
<v Speaker 1>So it builds an interactive API for that tree.

110
00:05:04.360 --> 00:05:07.800
<v Speaker 2>Yes, that live in memory tree is the DOM. And

111
00:05:07.879 --> 00:05:09.720
<v Speaker 2>the only reason we have to make this distinction is

112
00:05:09.879 --> 00:05:11.360
<v Speaker 2>entirely because of JavaScript.

113
00:05:11.600 --> 00:05:14.959
<v Speaker 1>I always like to think of standard static HTML parsing

114
00:05:15.079 --> 00:05:17.079
<v Speaker 1>like reading a printed menu at a restaurant.

115
00:05:17.120 --> 00:05:18.319
<v Speaker 2>Oh, that's a great way to look at it.

116
00:05:18.399 --> 00:05:21.120
<v Speaker 1>Yeah, because what is printed on the paper in front

117
00:05:21.160 --> 00:05:24.120
<v Speaker 1>of you is exactly what you get. But DOM parsing

118
00:05:24.240 --> 00:05:26.839
<v Speaker 1>is like watching one of those digital menu boards above

119
00:05:26.879 --> 00:05:29.800
<v Speaker 1>the counter that constantly changes as daily specials.

120
00:05:29.879 --> 00:05:32.879
<v Speaker 2>Right, they flip an update constantly exactly.

121
00:05:32.600 --> 00:05:35.079
<v Speaker 1>So if you only read the printed paper menu, you

122
00:05:35.120 --> 00:05:37.959
<v Speaker 1>are completely blind to the dynamic updates happening on the

123
00:05:37.959 --> 00:05:38.600
<v Speaker 1>digital board.

124
00:05:38.720 --> 00:05:43.360
<v Speaker 2>That perfectly isolates the mechanism at play here. Because JavaScript

125
00:05:43.519 --> 00:05:47.680
<v Speaker 2>executes within the browser environment and modifies that live dom.

126
00:05:47.560 --> 00:05:50.040
<v Speaker 1>On the fly, it can just change things whenever it wants.

127
00:05:50.120 --> 00:05:53.600
<v Speaker 2>Yeah, it can inject new elements, delete entire sections, or

128
00:05:53.759 --> 00:05:57.560
<v Speaker 2>asynchronously fetch and load new data long after the initial

129
00:05:57.680 --> 00:05:58.959
<v Speaker 2>HTML string was downloaded.

130
00:05:59.120 --> 00:06:02.120
<v Speaker 1>So if my Python's just uses the requests library to

131
00:06:02.160 --> 00:06:05.480
<v Speaker 1>grab the HTML, it is only looking at the printed.

132
00:06:05.199 --> 00:06:08.720
<v Speaker 2>Menu precisely, it has no JavaScript engine. It literally cannot

133
00:06:08.759 --> 00:06:09.519
<v Speaker 2>see the live dom.

134
00:06:09.720 --> 00:06:11.879
<v Speaker 1>Okay, wait, let's stick to the printed menu for a second.

135
00:06:12.279 --> 00:06:15.199
<v Speaker 1>Let's say I am dealing with the static site. The

136
00:06:15.399 --> 00:06:19.160
<v Speaker 1>HTML is ultimately just text coming over the wire. Why

137
00:06:19.199 --> 00:06:21.560
<v Speaker 1>am I bothering with specialized parsers at all?

138
00:06:22.079 --> 00:06:23.000
<v Speaker 2>Where are you going with this?

139
00:06:23.199 --> 00:06:26.279
<v Speaker 1>Well, I already know regular expressions. REGX is basically the

140
00:06:26.360 --> 00:06:30.240
<v Speaker 1>universal Swiss army knife for finding text patterns. Right, Oh no,

141
00:06:30.720 --> 00:06:32.240
<v Speaker 1>hear me out. If I want to find a price

142
00:06:32.279 --> 00:06:34.360
<v Speaker 1>on a page, I can just write a quick script

143
00:06:34.399 --> 00:06:37.920
<v Speaker 1>to find the specific price tag and extract the number inside.

144
00:06:38.399 --> 00:06:39.439
<v Speaker 1>Why wouldn't that work?

145
00:06:39.680 --> 00:06:42.560
<v Speaker 2>It is the most common trap developers fall into. Really,

146
00:06:42.680 --> 00:06:45.680
<v Speaker 2>oh absolutely, your first instinct is to treat the document

147
00:06:45.720 --> 00:06:48.920
<v Speaker 2>as a string and just use string searching tools. But

148
00:06:49.240 --> 00:06:52.800
<v Speaker 2>using regular expressions to parse HTML is just a nightmare

149
00:06:52.839 --> 00:06:53.600
<v Speaker 2>waiting to happen.

150
00:06:53.839 --> 00:06:54.879
<v Speaker 1>Why is this so bad?

151
00:06:55.279 --> 00:06:58.959
<v Speaker 2>The structural reason is that regx is strictly linear. Think

152
00:06:59.000 --> 00:07:02.079
<v Speaker 2>of REGX less like a smart search tool and more

153
00:07:02.199 --> 00:07:05.800
<v Speaker 2>like a guy on a factory conveyor belt looking through

154
00:07:05.800 --> 00:07:07.079
<v Speaker 2>a tiny magnifying glass.

155
00:07:07.120 --> 00:07:08.240
<v Speaker 1>Okay, I'm picturing it.

156
00:07:08.279 --> 00:07:11.120
<v Speaker 2>He only sees the exact sequence of letters passing directly

157
00:07:11.240 --> 00:07:14.120
<v Speaker 2>under his lens. He has absolutely no awareness of the

158
00:07:14.199 --> 00:07:16.199
<v Speaker 2>larger structure of the factory around him.

159
00:07:16.360 --> 00:07:18.079
<v Speaker 1>Oh I see. So he doesn't know if the text

160
00:07:18.079 --> 00:07:20.279
<v Speaker 1>he's looking at is inside a paragraph, or if it's

161
00:07:20.319 --> 00:07:22.199
<v Speaker 1>hidden in a comment or a part of the script block.

162
00:07:22.439 --> 00:07:26.519
<v Speaker 2>Exactly. He is just blindly pattern matching. He is completely

163
00:07:26.560 --> 00:07:30.120
<v Speaker 2>blind to the hierarchical tree context. That makes sense, And

164
00:07:30.160 --> 00:07:32.160
<v Speaker 2>beyond that, I mean, if you've ever spent time in

165
00:07:32.199 --> 00:07:36.040
<v Speaker 2>developer forums, you know that trying to match nested HTML

166
00:07:36.160 --> 00:07:42.120
<v Speaker 2>structures with regx quickly leaves to infinitely complex, just unreadable syntax.

167
00:07:42.240 --> 00:07:45.000
<v Speaker 1>Oh yeah, a single misplaced character like a missing slash

168
00:07:45.079 --> 00:07:48.319
<v Speaker 1>or an extra parenthesis, and it destroys the entire pattern. Yep.

169
00:07:48.600 --> 00:07:51.439
<v Speaker 2>If you've ever seen those infamous stack overflow threads where

170
00:07:51.439 --> 00:07:55.279
<v Speaker 2>a developer asks how to parse HTML with regx.

171
00:07:55.120 --> 00:07:57.800
<v Speaker 1>Oh, the community response is usually a joke about how

172
00:07:57.920 --> 00:08:00.959
<v Speaker 1>doing so will summon ancient demons or corrupt your soul

173
00:08:01.079 --> 00:08:02.040
<v Speaker 1>with zalgo text.

174
00:08:02.160 --> 00:08:05.839
<v Speaker 2>Ah, exactly. It's funny, but is rooted in a very

175
00:08:06.040 --> 00:08:09.600
<v Speaker 2>real developer pain point, which is the fragility factor.

176
00:08:09.720 --> 00:08:12.959
<v Speaker 1>Right, because the web is essentially a graveyard of malformed markup.

177
00:08:13.240 --> 00:08:17.120
<v Speaker 2>It really is. In a perfect theoretical world, every HTML

178
00:08:17.240 --> 00:08:20.040
<v Speaker 2>tag that is open is neatly closed. But out in

179
00:08:20.040 --> 00:08:22.240
<v Speaker 2>the wild developers make mistakes constantly.

180
00:08:22.519 --> 00:08:26.639
<v Speaker 1>They leave divtags open, they nest lists, improperly, use illegal

181
00:08:26.720 --> 00:08:27.480
<v Speaker 1>characters all.

182
00:08:27.399 --> 00:08:30.680
<v Speaker 2>The time, And because rej x is entirely rigid, it

183
00:08:30.759 --> 00:08:34.559
<v Speaker 2>fails the exact moment. Reality doesn't match its stripped pattern.

184
00:08:35.000 --> 00:08:38.639
<v Speaker 1>So if a web admin accidentally adds an unexpected space

185
00:08:38.799 --> 00:08:42.080
<v Speaker 1>inside a class attribute, your conveyor belt worker with a

186
00:08:42.080 --> 00:08:44.519
<v Speaker 1>magnifying glass completely misses it.

187
00:08:44.399 --> 00:08:45.559
<v Speaker 2>And the whole script breaks.

188
00:08:45.840 --> 00:08:48.559
<v Speaker 1>Okay, so if rejex is a fragile nightmare, how do

189
00:08:48.600 --> 00:08:51.000
<v Speaker 1>we handle the fact that the web is essentially chaos?

190
00:08:51.840 --> 00:08:54.480
<v Speaker 1>We clearly need a tool that doesn't just read the string,

191
00:08:54.559 --> 00:08:58.320
<v Speaker 1>but actually understands the intent of the HTML hierarchy.

192
00:08:57.919 --> 00:09:01.039
<v Speaker 2>And forgives those human mistakes. Yes, this is where beautiful

193
00:09:01.080 --> 00:09:04.159
<v Speaker 2>soup comes in. Beautiful Soup it is a brilliantly designed

194
00:09:04.159 --> 00:09:09.240
<v Speaker 2>Python library specifically engineered to mitigate all of those exact weaknesses.

195
00:09:09.320 --> 00:09:11.960
<v Speaker 1>So how does it process the text differently than rej ex?

196
00:09:12.159 --> 00:09:15.080
<v Speaker 2>When you pass your fetched HTML string into beautiful soup,

197
00:09:15.279 --> 00:09:18.399
<v Speaker 2>it doesn't read it line by line. It actively digests

198
00:09:18.440 --> 00:09:21.879
<v Speaker 2>the markup and constructs a complete traversible parse tree right

199
00:09:21.960 --> 00:09:23.080
<v Speaker 2>in your system's memory.

200
00:09:23.360 --> 00:09:25.879
<v Speaker 1>And it's crucial to understand how it does this right,

201
00:09:26.240 --> 00:09:29.000
<v Speaker 1>because beautiful soup itself isn't doing the heavy lifting of

202
00:09:29.039 --> 00:09:32.320
<v Speaker 1>interpreting the text. It's more of an orchestration layer, right.

203
00:09:32.360 --> 00:09:36.360
<v Speaker 2>It sits on top of incredibly fast underlying CEA based parsers.

204
00:09:36.919 --> 00:09:40.200
<v Speaker 2>You can configure it to use pythons built in EAHTML

205
00:09:40.240 --> 00:09:43.360
<v Speaker 2>dot parser, but more often you'll pair it with something

206
00:09:43.519 --> 00:09:45.519
<v Speaker 2>like lxml for pure.

207
00:09:45.320 --> 00:09:47.639
<v Speaker 1>Speed, or HTML fivelib Yes.

208
00:09:47.840 --> 00:09:52.080
<v Speaker 2>HTML FIVELB is a particularly fascinating choice because it parses

209
00:09:52.080 --> 00:09:55.120
<v Speaker 2>the text exactly the same way a modern web browser does.

210
00:09:55.159 --> 00:09:58.080
<v Speaker 2>It uses the same heuristic algorithms.

211
00:09:57.519 --> 00:10:01.879
<v Speaker 1>Which leads to Beautiful Soup's most beloved feature. It's extreme

212
00:10:02.000 --> 00:10:03.600
<v Speaker 1>tolerance for garbage data.

213
00:10:03.639 --> 00:10:04.840
<v Speaker 2>It is so forgiving.

214
00:10:05.000 --> 00:10:08.039
<v Speaker 1>If you've ever spent like three hours debugging a scraper

215
00:10:08.120 --> 00:10:11.320
<v Speaker 1>because a site admin forgot to close a single starmatting tag,

216
00:10:11.679 --> 00:10:13.679
<v Speaker 1>Beautiful Soup is going to feel like actual magic.

217
00:10:13.840 --> 00:10:16.480
<v Speaker 2>It really does. You can feed it an absolute disaster

218
00:10:16.600 --> 00:10:18.679
<v Speaker 2>of incomplete HTML and it won't crash.

219
00:10:18.799 --> 00:10:22.159
<v Speaker 1>It uses those browser like heuristics to make educated guesses right.

220
00:10:22.679 --> 00:10:25.720
<v Speaker 1>It will infer where the missing closing tags belong, inject

221
00:10:25.759 --> 00:10:29.919
<v Speaker 1>them for you, and output perfectly well formed, navigable HTML.

222
00:10:30.279 --> 00:10:33.440
<v Speaker 2>It acts as a resilient translator between the messy reality

223
00:10:33.480 --> 00:10:36.519
<v Speaker 2>of the web and the strict logical requirements of your script.

224
00:10:36.639 --> 00:10:37.360
<v Speaker 1>That's so cool.

225
00:10:37.440 --> 00:10:40.840
<v Speaker 2>And once that tree is successfully built, you can navigate

226
00:10:40.879 --> 00:10:44.039
<v Speaker 2>it structurally. You can tell your program to find a

227
00:10:44.080 --> 00:10:47.679
<v Speaker 2>specific element and then traverse up to its parent, down

228
00:10:47.720 --> 00:10:49.879
<v Speaker 2>to its children, or sideways to its siblings.

229
00:10:49.919 --> 00:10:52.480
<v Speaker 1>So you just ask the library for say, all the

230
00:10:52.519 --> 00:10:55.240
<v Speaker 1>anchor tags inside the article body and it just hands

231
00:10:55.279 --> 00:10:56.879
<v Speaker 1>you a clean list exactly.

232
00:10:56.960 --> 00:10:59.519
<v Speaker 2>It makes data extraction feel highly intuitive.

233
00:10:59.639 --> 00:11:03.039
<v Speaker 1>It does, But we have to address the elephant in

234
00:11:03.039 --> 00:11:06.159
<v Speaker 1>the room here, which is Beautiful Soup is incredible for

235
00:11:06.279 --> 00:11:10.440
<v Speaker 1>parsing a single static document. But what if my project

236
00:11:10.799 --> 00:11:13.480
<v Speaker 1>isn't just one document. What if I need to fetch

237
00:11:13.519 --> 00:11:17.759
<v Speaker 1>the data, parse the HTML and aggressively crawl across tens

238
00:11:17.799 --> 00:11:20.840
<v Speaker 1>of thousands of different pages scaling up right. Or what

239
00:11:20.840 --> 00:11:24.080
<v Speaker 1>if I'm hitting a modern React application and I absolutely

240
00:11:24.120 --> 00:11:26.759
<v Speaker 1>need to interact with that live dynamic dom we talked

241
00:11:26.759 --> 00:11:31.080
<v Speaker 1>about earlier. A simple Python script running beautiful Soup sequentially

242
00:11:31.320 --> 00:11:33.360
<v Speaker 1>just isn't going to cut it for a massive, industrial

243
00:11:33.360 --> 00:11:34.120
<v Speaker 1>strength project.

244
00:11:34.480 --> 00:11:36.799
<v Speaker 2>No, it won't, not at all. When you need to

245
00:11:36.840 --> 00:11:39.159
<v Speaker 2>scale up your programmatic extraction like that, you have to

246
00:11:39.200 --> 00:11:40.240
<v Speaker 2>graduate to the heavyweights.

247
00:11:40.240 --> 00:11:41.279
<v Speaker 1>Okay, let's talk heavyweights.

248
00:11:41.360 --> 00:11:43.919
<v Speaker 2>If your challenge is scale and speed, you bring in

249
00:11:43.960 --> 00:11:45.320
<v Speaker 2>a framework like Scrapeye.

250
00:11:45.519 --> 00:11:45.960
<v Speaker 1>Scrapey.

251
00:11:46.159 --> 00:11:48.919
<v Speaker 2>Yeah, Scrapy is not just a library. It is a

252
00:11:49.080 --> 00:11:53.879
<v Speaker 2>full production grade web crawling framework. It represents a massive

253
00:11:53.919 --> 00:11:55.080
<v Speaker 2>shift in architecture.

254
00:11:55.360 --> 00:11:59.840
<v Speaker 1>Let's drill into that architecture because with requests, everything is synchronous. Right,

255
00:12:00.039 --> 00:12:03.080
<v Speaker 1>you ask for a page, your script halts, it waits

256
00:12:03.080 --> 00:12:05.799
<v Speaker 1>for the server to reply, downloads the page, and then

257
00:12:05.840 --> 00:12:07.720
<v Speaker 1>moves to the next line of code exactly.

258
00:12:08.039 --> 00:12:10.639
<v Speaker 2>And if you have ten thousand pages to scrape, doing

259
00:12:10.679 --> 00:12:13.240
<v Speaker 2>it one by one is agonizingly slow.

260
00:12:13.600 --> 00:12:14.840
<v Speaker 1>So how does scrape fix that?

261
00:12:15.240 --> 00:12:20.440
<v Speaker 2>Scraepy fundamentally changes that by using asynchronous networking under the hood. Specifically,

262
00:12:20.600 --> 00:12:23.440
<v Speaker 2>it uses an event driven networking engine called twisted.

263
00:12:23.600 --> 00:12:25.240
<v Speaker 1>Oh so scrapey doesn't wait.

264
00:12:25.120 --> 00:12:29.519
<v Speaker 2>Around, Nope, it fires off hundreds of HTTP requests concurrently.

265
00:12:29.879 --> 00:12:32.759
<v Speaker 2>While it's waiting for server A to respond, it's already

266
00:12:32.799 --> 00:12:37.600
<v Speaker 2>fetching from server BCND. It manages the concurrency, the request

267
00:12:37.759 --> 00:12:41.799
<v Speaker 2>throttling to avoid overloading servers, and all the retries for

268
00:12:41.840 --> 00:12:44.440
<v Speaker 2>failed connections completely automatically.

269
00:12:44.600 --> 00:12:47.600
<v Speaker 1>It's essentially a spider factory. You just define the rules

270
00:12:47.600 --> 00:12:49.840
<v Speaker 1>of how the spider should navigate the links it finds,

271
00:12:50.279 --> 00:12:53.720
<v Speaker 1>and scrape handles the immense logistical overhead.

272
00:12:53.320 --> 00:12:55.879
<v Speaker 2>Of the crawl right, and it passes the raw data

273
00:12:55.919 --> 00:12:59.000
<v Speaker 2>it finds through integrated item pipelines, where you can clean

274
00:12:59.039 --> 00:13:01.240
<v Speaker 2>it and dump it directly into a database.

275
00:13:01.440 --> 00:13:05.559
<v Speaker 1>But scraping by default is still essentially just downloading static

276
00:13:05.720 --> 00:13:09.399
<v Speaker 1>HTML really really fast, isn't it. It is so if

277
00:13:09.399 --> 00:13:13.360
<v Speaker 1>the data you need requires rendering that live dom, like

278
00:13:13.399 --> 00:13:15.759
<v Speaker 1>if the text literally does not exist in the HTML

279
00:13:15.840 --> 00:13:20.279
<v Speaker 1>until a complex JavaScript bundle executes, then scrape alone won't

280
00:13:20.320 --> 00:13:21.080
<v Speaker 1>solve it. Right.

281
00:13:21.120 --> 00:13:22.960
<v Speaker 2>In that case, you have to shift your strategy to

282
00:13:23.000 --> 00:13:24.399
<v Speaker 2>a tool like Selenium. No.

283
00:13:24.559 --> 00:13:27.360
<v Speaker 1>Selenium is fascinating because it wasn't even built for web scraping.

284
00:13:27.480 --> 00:13:29.919
<v Speaker 2>No, it was originally built for automated software testing.

285
00:13:29.840 --> 00:13:31.919
<v Speaker 1>Right, but it has become the gold standard for scraping

286
00:13:31.960 --> 00:13:36.480
<v Speaker 1>heavily obfuscated or dynamic web applications. How does it actually

287
00:13:36.519 --> 00:13:38.080
<v Speaker 1>solve the JavaScript problem?

288
00:13:38.279 --> 00:13:40.840
<v Speaker 2>It solves it by not trying to emulate a browser,

289
00:13:40.919 --> 00:13:42.679
<v Speaker 2>but by actually puppeting a real one.

290
00:13:42.720 --> 00:13:43.759
<v Speaker 1>Puppeting yeah.

291
00:13:44.360 --> 00:13:48.720
<v Speaker 2>Selenium utilizes a protocol called webdriver. Your Python script talks

292
00:13:48.759 --> 00:13:52.120
<v Speaker 2>to the webdriver, and the webdriver reaches directly into a

293
00:13:52.159 --> 00:13:55.519
<v Speaker 2>real instance of Chrome or Firefox installed on your machine.

294
00:13:55.559 --> 00:13:55.879
<v Speaker 1>Okay.

295
00:13:56.240 --> 00:13:59.159
<v Speaker 2>It spins the browser up, often in headless mode.

296
00:13:59.039 --> 00:14:01.440
<v Speaker 1>Meaning it runs into highly in the background without drawing

297
00:14:01.480 --> 00:14:03.600
<v Speaker 1>a physical window on your screen exactly.

298
00:14:03.720 --> 00:14:05.879
<v Speaker 2>Yeah, and then it navigates to the URL.

299
00:14:05.679 --> 00:14:09.240
<v Speaker 1>Which means the actual physical browsers v eight engine executes

300
00:14:09.240 --> 00:14:13.279
<v Speaker 1>the JavaScript, makes the asynchronous ATI calls, and fully builds

301
00:14:13.279 --> 00:14:14.679
<v Speaker 1>the live dom in memory.

302
00:14:14.879 --> 00:14:18.639
<v Speaker 2>Exactly, It's a real browser doing real browser things. And

303
00:14:18.720 --> 00:14:22.279
<v Speaker 2>once that live dom is fully realized, your Selenium script

304
00:14:22.360 --> 00:14:24.960
<v Speaker 2>can interact with it programmatically.

305
00:14:24.440 --> 00:14:27.279
<v Speaker 1>So you can command the automated browser to like click

306
00:14:27.320 --> 00:14:30.080
<v Speaker 1>specific buttons or type text into log in.

307
00:14:30.039 --> 00:14:33.200
<v Speaker 2>Fields yep, or scroll down the page to trigger infinite

308
00:14:33.200 --> 00:14:36.720
<v Speaker 2>scroll data loading. You literally force the web application to

309
00:14:36.759 --> 00:14:39.519
<v Speaker 2>reveal the data and then you extract it directly from

310
00:14:39.559 --> 00:14:40.480
<v Speaker 2>the live dom state.

311
00:14:40.840 --> 00:14:44.000
<v Speaker 1>That is incredibly powerful, but the trade off, of course

312
00:14:44.080 --> 00:14:44.759
<v Speaker 1>is overhead.

313
00:14:45.039 --> 00:14:46.000
<v Speaker 2>Massive overhead.

314
00:14:46.120 --> 00:14:48.840
<v Speaker 1>Yeah, spinning up an actual browser instance and executing full

315
00:14:48.919 --> 00:14:53.360
<v Speaker 1>JavaScript bundles has to be incredibly resource intensive and magnitude

316
00:14:53.440 --> 00:14:57.600
<v Speaker 1>slower than a raw HTTP get request.

317
00:14:57.720 --> 00:14:59.919
<v Speaker 2>Oh, without a doubt. You only pull out Selenium, wh

318
00:15:00.000 --> 00:15:03.159
<v Speaker 2>when the static HTML approaches simply cannot see the data.

319
00:15:03.559 --> 00:15:07.399
<v Speaker 1>Makes sense. So SCRAPI and Selenium represent the absolute pinnacle

320
00:15:07.440 --> 00:15:11.120
<v Speaker 1>of code based extraction. You're either scaling the network requests

321
00:15:11.159 --> 00:15:15.720
<v Speaker 1>asynchronously or you are deeply manipulating the browser's rendering engine. Right.

322
00:15:16.080 --> 00:15:19.320
<v Speaker 2>But this all assumes that the underlying code structure is

323
00:15:19.360 --> 00:15:20.720
<v Speaker 2>something we can parse.

324
00:15:20.759 --> 00:15:22.840
<v Speaker 1>Wait what do you mean, What if we approach the

325
00:15:22.879 --> 00:15:24.720
<v Speaker 1>problem from a totally alien angle?

326
00:15:24.799 --> 00:15:25.559
<v Speaker 2>Okay, layd on me.

327
00:15:25.720 --> 00:15:27.600
<v Speaker 1>What if we don't parse the code at all?

328
00:15:27.879 --> 00:15:30.240
<v Speaker 2>You're talking about the wildcard approach exactly.

329
00:15:30.600 --> 00:15:33.399
<v Speaker 1>This completely flips the script. We've spent this entire deep

330
00:15:33.440 --> 00:15:37.120
<v Speaker 1>dive obsessed with code, you know, HTML tags, DOM trees,

331
00:15:37.559 --> 00:15:41.360
<v Speaker 1>giddy requests. But there is a completely different approach based

332
00:15:41.399 --> 00:15:42.519
<v Speaker 1>heavily on computer vision.

333
00:15:42.840 --> 00:15:46.200
<v Speaker 2>Right Because what if the underlying code is so aggressively

334
00:15:46.240 --> 00:15:50.639
<v Speaker 2>obfuscated or trapped inside an inaccessible canvas element that traditional

335
00:15:50.679 --> 00:15:52.360
<v Speaker 2>parsing is just impossible.

336
00:15:52.559 --> 00:15:54.639
<v Speaker 1>Then you step outside the browser entirely and look at

337
00:15:54.639 --> 00:15:55.000
<v Speaker 1>the screen.

338
00:15:55.799 --> 00:15:59.519
<v Speaker 2>Consider a visual automation technology like sickly Tickle with yes

339
00:16:00.000 --> 00:16:04.799
<v Speaker 2>poll is designed to automate graphical user interfaces GUIs. It

340
00:16:04.840 --> 00:16:07.039
<v Speaker 2>does not care about your HTML structure. It does not

341
00:16:07.120 --> 00:16:09.120
<v Speaker 2>care if the DOM is dynamic or static.

342
00:16:09.320 --> 00:16:13.559
<v Speaker 1>It literally analyzes the screen visually using computer vision algorithms

343
00:16:13.639 --> 00:16:15.600
<v Speaker 1>exactly the way a human user does.

344
00:16:15.799 --> 00:16:18.759
<v Speaker 2>It's doing template matching on the screen buffer itself.

345
00:16:18.960 --> 00:16:22.279
<v Speaker 1>That is wild. So instead of telling your Python script, hey,

346
00:16:22.519 --> 00:16:26.240
<v Speaker 1>find the HTML button tag with the specific CSS class

347
00:16:26.480 --> 00:16:27.960
<v Speaker 1>of submit btn, you.

348
00:16:27.919 --> 00:16:30.399
<v Speaker 2>Literally crop an image of what the button looks like,

349
00:16:30.799 --> 00:16:33.200
<v Speaker 2>feed that image file into siculey and say, scan the

350
00:16:33.200 --> 00:16:36.559
<v Speaker 2>pixels on the screen, find the area that visually matches

351
00:16:36.600 --> 00:16:39.559
<v Speaker 2>this picture, and click your virtual mouse right there.

352
00:16:39.679 --> 00:16:43.360
<v Speaker 1>It's an incredibly robust fallback mechanism. By interacting with the

353
00:16:43.440 --> 00:16:47.120
<v Speaker 1>visual layer, you completely sidestep the entire arms race of

354
00:16:47.159 --> 00:16:48.120
<v Speaker 1>code obfuscation.

355
00:16:48.399 --> 00:16:51.200
<v Speaker 2>Exactly if a human can see the data on the screen,

356
00:16:51.399 --> 00:16:54.840
<v Speaker 2>a computer vision tool can theoretically identify its location and

357
00:16:54.919 --> 00:16:57.600
<v Speaker 2>extract it, or at least interact with the interface to

358
00:16:57.679 --> 00:16:58.639
<v Speaker 2>reveal what you need.

359
00:16:58.919 --> 00:17:01.759
<v Speaker 1>It opens up a completeletely different paradigm for interacting with

360
00:17:01.799 --> 00:17:02.679
<v Speaker 1>digital systems.

361
00:17:02.840 --> 00:17:03.480
<v Speaker 2>It really does.

362
00:17:03.600 --> 00:17:06.640
<v Speaker 1>Man, we have covered a massive amount of technical ground today.

363
00:17:06.720 --> 00:17:07.319
<v Speaker 2>We really have.

364
00:17:07.680 --> 00:17:11.839
<v Speaker 1>The fundamental takeaway here is that extracting data programmatically requires

365
00:17:11.880 --> 00:17:16.039
<v Speaker 1>two distinct actions, fetching and parsing. You fetch the raw

366
00:17:16.079 --> 00:17:19.039
<v Speaker 1>material using robust HTTP clients.

367
00:17:18.680 --> 00:17:22.759
<v Speaker 2>Like requests right, and then you must parse that chaotic text.

368
00:17:23.400 --> 00:17:25.920
<v Speaker 2>And the cardinal rule of parsing is to respect the

369
00:17:26.000 --> 00:17:30.559
<v Speaker 2>document structure, avoid the linear fragility of regular expressions at all.

370
00:17:30.400 --> 00:17:34.720
<v Speaker 1>Costs, rely on tolerant tree building parsers like beautiful Soup

371
00:17:35.160 --> 00:17:37.319
<v Speaker 1>to normalize MESSYHTML, and.

372
00:17:37.319 --> 00:17:40.440
<v Speaker 2>When the project demands it step up to the asynchronous

373
00:17:40.480 --> 00:17:44.160
<v Speaker 2>power of scrapey for scale, or the webrooky capabilities of

374
00:17:44.200 --> 00:17:49.400
<v Speaker 2>Selenium to manipulate live JavaScript rendered doms perfect summary.

375
00:17:49.640 --> 00:17:51.119
<v Speaker 1>Now, before we sign off, I want to leave you

376
00:17:51.160 --> 00:17:53.559
<v Speaker 1>with a quick mental exercise to apply what we've covered today.

377
00:17:53.599 --> 00:17:54.279
<v Speaker 2>Oh I love these.

378
00:17:54.559 --> 00:17:58.640
<v Speaker 1>Think about a modern, highly interactive web application you use daily.

379
00:17:59.200 --> 00:18:03.200
<v Speaker 1>Maybe it's a complex financial dashboard or a continuously updating

380
00:18:03.200 --> 00:18:07.440
<v Speaker 1>social media feed. If you were tasked with programmatically extracting

381
00:18:07.480 --> 00:18:10.720
<v Speaker 1>the live, minute by minute updates from that feed, would

382
00:18:10.759 --> 00:18:14.599
<v Speaker 1>you use static HTML parsing with a tool like beautiful

383
00:18:14.640 --> 00:18:18.359
<v Speaker 1>Soup or would you need to implement DOM parsing to

384
00:18:18.480 --> 00:18:19.359
<v Speaker 1>capture the data.

385
00:18:19.440 --> 00:18:20.359
<v Speaker 2>That's a great question.

386
00:18:20.640 --> 00:18:22.680
<v Speaker 1>Think through the architecture of that site next time you

387
00:18:22.720 --> 00:18:23.160
<v Speaker 1>log in.

388
00:18:23.319 --> 00:18:26.240
<v Speaker 2>And to push that thought even further, consider the trajectory

389
00:18:26.279 --> 00:18:29.240
<v Speaker 2>of the visual tools we discussed. We dedicate so much

390
00:18:29.279 --> 00:18:33.200
<v Speaker 2>engineering effort to meticulously reverse engineering and parsing the underlying

391
00:18:33.240 --> 00:18:36.160
<v Speaker 2>code of the web. Right, But as computer vision models

392
00:18:36.160 --> 00:18:39.559
<v Speaker 2>become vastly more sophisticated at simply seeing and understanding a

393
00:18:39.599 --> 00:18:44.000
<v Speaker 2>screen exactly as humans do, will digging through chaotic HTML

394
00:18:44.079 --> 00:18:48.359
<v Speaker 2>parse trees. Eventually become an antiquated practice. Will the digital

395
00:18:48.440 --> 00:18:51.680
<v Speaker 2>robots of the future stop reading the underlying text entirely

396
00:18:51.799 --> 00:18:53.559
<v Speaker 2>and just start comprehending the pixels.

397
00:18:53.839 --> 00:18:56.759
<v Speaker 1>That is a wild architectural thought to end on the

398
00:18:56.880 --> 00:19:00.200
<v Speaker 1>entire paradigm of how software interacts with other software could

399
00:19:00.200 --> 00:19:03.839
<v Speaker 1>shift from code reading to visual comprehension. Well, thank you

400
00:19:03.880 --> 00:19:06.799
<v Speaker 1>for joining us on this exploration of automated extraction. Yeah,

401
00:19:06.880 --> 00:19:09.480
<v Speaker 1>keep questioning how the digital systems around you are constructed

402
00:19:09.599 --> 00:19:10.480
<v Speaker 1>and happy building