WEBVTT

1
00:00:00.080 --> 00:00:05.759
<v Speaker 1>So picture this. You've written a scraping script to pull

2
00:00:05.799 --> 00:00:09.480
<v Speaker 1>some highly specific data from a modern web app, right.

3
00:00:09.359 --> 00:00:11.880
<v Speaker 2>Like maybe you're analyzing market trends or I don't know,

4
00:00:11.960 --> 00:00:13.759
<v Speaker 2>tracking inventory exactly.

5
00:00:14.439 --> 00:00:17.079
<v Speaker 1>But the data isn't just sitting there in some nice

6
00:00:17.160 --> 00:00:20.039
<v Speaker 1>static HTML file waiting to be grabbed.

7
00:00:19.800 --> 00:00:22.440
<v Speaker 2>Oh definitely not. Usually it's buried behind a client side

8
00:00:22.480 --> 00:00:24.079
<v Speaker 2>JavaScript rendering engine.

9
00:00:23.879 --> 00:00:27.559
<v Speaker 1>Yeah, or like hidden behind drop down menu, search queries,

10
00:00:27.719 --> 00:00:29.280
<v Speaker 1>authentication walls, all that stuff.

11
00:00:29.440 --> 00:00:31.800
<v Speaker 2>Right. And you know that data is physically there in

12
00:00:31.839 --> 00:00:33.799
<v Speaker 2>your browser's memory, right, you can see it on.

13
00:00:33.759 --> 00:00:36.280
<v Speaker 1>The screen, But if you try to extract it using

14
00:00:36.280 --> 00:00:40.479
<v Speaker 1>a standard automated script, your code just suddenly bam, grinds

15
00:00:40.520 --> 00:00:41.520
<v Speaker 1>to an absolute halt.

16
00:00:41.640 --> 00:00:43.359
<v Speaker 2>It's incredibly frustrating.

17
00:00:42.880 --> 00:00:45.640
<v Speaker 1>It really is. So for today's deep dive, we're looking

18
00:00:45.640 --> 00:00:48.600
<v Speaker 1>at why bypassing your browser's native language is kind of

19
00:00:48.640 --> 00:00:50.560
<v Speaker 1>the only way to get that data out efficiently.

20
00:00:50.719 --> 00:00:53.600
<v Speaker 2>Yeah, and we'll look at how combining Python, Selenium and

21
00:00:53.640 --> 00:00:59.600
<v Speaker 2>beautiful Soup builds this architecture that solves this exact bottleneck.

22
00:00:59.119 --> 00:01:02.280
<v Speaker 1>Which is so u because I mean, most developers run

23
00:01:02.280 --> 00:01:05.200
<v Speaker 1>into this wall the exact moment they graduate from scraping

24
00:01:05.239 --> 00:01:06.760
<v Speaker 1>simple static pages oh.

25
00:01:06.840 --> 00:01:11.400
<v Speaker 2>Absolutely. The minute you try interacting with real world dynamic applications,

26
00:01:11.879 --> 00:01:13.319
<v Speaker 2>things change.

27
00:01:13.319 --> 00:01:16.840
<v Speaker 1>Because a basic HTTP request to a modern site often

28
00:01:16.920 --> 00:01:19.879
<v Speaker 1>just returns like a skeleton HTML file, right.

29
00:01:19.799 --> 00:01:22.480
<v Speaker 2>Exactly, just a skeleton with a bunch of JavaScript tags.

30
00:01:22.480 --> 00:01:23.599
<v Speaker 2>The data isn't even there yet.

31
00:01:23.640 --> 00:01:26.239
<v Speaker 1>You actually need a real browser environment to execute that

32
00:01:26.319 --> 00:01:32.079
<v Speaker 1>JavaScript and render the actual document object model or the DOM.

33
00:01:32.439 --> 00:01:35.959
<v Speaker 2>Right. So naturally the instinct is to just bring in Selenium.

34
00:01:36.200 --> 00:01:39.280
<v Speaker 1>Because Selenium controls a real web browser, it can click buttons,

35
00:01:39.319 --> 00:01:40.879
<v Speaker 1>bypass pop ups, type.

36
00:01:40.719 --> 00:01:43.040
<v Speaker 2>Into fields yeah, and it can wait for the JavaScript

37
00:01:43.079 --> 00:01:45.560
<v Speaker 2>to execute it as functions build right into it to

38
00:01:45.680 --> 00:01:47.799
<v Speaker 2>locate elements and extract their text.

39
00:01:48.040 --> 00:01:50.840
<v Speaker 1>Right. So, if Selenium can do all of that, a

40
00:01:50.840 --> 00:01:54.560
<v Speaker 1>lot of developers learning web automation naturally wonder why don't

41
00:01:54.560 --> 00:01:57.159
<v Speaker 1>we just use it for the entire data extraction process.

42
00:01:57.200 --> 00:02:00.640
<v Speaker 2>I mean, it seems totally logical at first glance, because

43
00:02:00.959 --> 00:02:04.439
<v Speaker 2>the capability to read the DOM is absolutely built into Selenium.

44
00:02:04.680 --> 00:02:08.319
<v Speaker 2>You can target a specific paragraph tag and extract the

45
00:02:08.360 --> 00:02:09.680
<v Speaker 2>text string right out of.

46
00:02:09.639 --> 00:02:11.400
<v Speaker 1>It, So why not just do that for everything?

47
00:02:11.639 --> 00:02:15.439
<v Speaker 2>Well, relying on Selenium to extract large data sets is

48
00:02:15.439 --> 00:02:16.919
<v Speaker 2>actually an architectural trap.

49
00:02:17.000 --> 00:02:18.319
<v Speaker 1>Okay, an architectural trap.

50
00:02:18.319 --> 00:02:22.000
<v Speaker 2>How so the issue isn't capability, you know, it's the

51
00:02:22.159 --> 00:02:27.240
<v Speaker 2>immense serialization overhead HATCHA. To really understand the bottleneck, we

52
00:02:27.360 --> 00:02:30.240
<v Speaker 2>have to look at the W three C webdriver protocol

53
00:02:30.560 --> 00:02:32.199
<v Speaker 2>that powers it under the hood.

54
00:02:32.360 --> 00:02:34.919
<v Speaker 1>Right, let's get into the mechanics of that. Because your

55
00:02:35.120 --> 00:02:39.280
<v Speaker 1>Python script and say the Chrome or Firefox browser it's controlling,

56
00:02:39.879 --> 00:02:42.840
<v Speaker 1>they aren't actually running in the same memory space exactly.

57
00:02:42.919 --> 00:02:47.280
<v Speaker 2>They are completely separate entities. Your Python environment cannot directly

58
00:02:47.319 --> 00:02:49.479
<v Speaker 2>read the RAM allocated to the web browser.

59
00:02:49.599 --> 00:02:51.080
<v Speaker 1>Okay, so how do they talk?

60
00:02:51.360 --> 00:02:54.639
<v Speaker 2>They have to communicate over a network protocol. Historically this

61
00:02:54.759 --> 00:02:57.960
<v Speaker 2>was known as the JSON wire protocol. Okay, So when

62
00:02:57.960 --> 00:03:00.680
<v Speaker 2>you tell Selenium to find an element and get it's text,

63
00:03:00.960 --> 00:03:03.680
<v Speaker 2>it doesn't just like read a local variable.

64
00:03:03.759 --> 00:03:05.159
<v Speaker 1>It's essentially texting a middleman.

65
00:03:05.560 --> 00:03:06.639
<v Speaker 2>That's a great way to put it.

66
00:03:06.759 --> 00:03:09.680
<v Speaker 1>Like it's like needing to read a massive encyclopedia, but

67
00:03:09.759 --> 00:03:12.919
<v Speaker 1>the book is locked in a library completely across town,

68
00:03:13.159 --> 00:03:16.199
<v Speaker 1>and instead of just going there yourself, you hire someone

69
00:03:16.240 --> 00:03:18.520
<v Speaker 1>to stand in the library and you text them, like,

70
00:03:18.639 --> 00:03:21.639
<v Speaker 1>turn to page one, read line one, text it back

71
00:03:21.639 --> 00:03:21.840
<v Speaker 1>to me.

72
00:03:22.080 --> 00:03:25.400
<v Speaker 2>Yeah, that is a highly accurate way to visualize the latency.

73
00:03:25.919 --> 00:03:29.800
<v Speaker 2>Every single interaction involves packaging your command into a JSON

74
00:03:29.879 --> 00:03:35.039
<v Speaker 2>payload wow, then opening an HTTP connection to the browser

75
00:03:35.240 --> 00:03:39.360
<v Speaker 2>driver like Chrome driver or Gecko driver, which then executes

76
00:03:39.400 --> 00:03:42.080
<v Speaker 2>the command inside the browser's JavaScript engine.

77
00:03:42.120 --> 00:03:44.520
<v Speaker 1>So it's literally sending it over the network every single

78
00:03:44.520 --> 00:03:45.319
<v Speaker 1>time exactly.

79
00:03:45.520 --> 00:03:48.439
<v Speaker 2>The browser finds the text, passes it back to the driver,

80
00:03:48.560 --> 00:03:51.759
<v Speaker 2>which serializes it back into a JSON response.

81
00:03:51.360 --> 00:03:54.199
<v Speaker 1>And sends it back over HTTP to your Python script.

82
00:03:54.280 --> 00:03:56.439
<v Speaker 2>Right, and then your script finally has to de serialize it.

83
00:03:56.400 --> 00:03:58.719
<v Speaker 1>Which I mean sounds fast enough for a single query.

84
00:03:58.759 --> 00:04:00.680
<v Speaker 2>You're sure, for one query, it's fine, But let's do.

85
00:04:00.639 --> 00:04:03.319
<v Speaker 1>The math on a real world's grading task. Say you're

86
00:04:03.360 --> 00:04:06.120
<v Speaker 1>pulling a data table with one thousand rows and five columns.

87
00:04:06.280 --> 00:04:09.120
<v Speaker 1>Oh boy, Yeah, if you use Selenium to iterate through

88
00:04:09.159 --> 00:04:12.319
<v Speaker 1>every single cell to extract the text, you are making

89
00:04:12.400 --> 00:04:15.800
<v Speaker 1>five thousand separate HTTP network request.

90
00:04:15.919 --> 00:04:18.600
<v Speaker 2>Just to read data that is already sitting right there

91
00:04:18.600 --> 00:04:20.879
<v Speaker 2>on your local machine. That's insane, and that is the

92
00:04:21.000 --> 00:04:26.160
<v Speaker 2>hidden text that communication overhead snowballs so incredibly fast. Yeah,

93
00:04:26.199 --> 00:04:29.360
<v Speaker 2>a script that should take milliseconds ends up taking three

94
00:04:29.439 --> 00:04:33.000
<v Speaker 2>or four minutes to execute. You're just bottlenecking your CPU

95
00:04:33.160 --> 00:04:37.240
<v Speaker 2>with constant network serialization and de serialization delays.

96
00:04:37.439 --> 00:04:41.000
<v Speaker 1>Okay, so if the whole texting the middleman approach fails

97
00:04:41.079 --> 00:04:45.120
<v Speaker 1>at scale, we need a way to just grab the

98
00:04:45.279 --> 00:04:48.240
<v Speaker 1>entire encyclopedia, bring it back to our own workspace, and

99
00:04:48.279 --> 00:04:49.360
<v Speaker 1>flip through the pages.

100
00:04:49.120 --> 00:04:51.040
<v Speaker 2>Ourselves, right at top speed.

101
00:04:50.839 --> 00:04:53.920
<v Speaker 1>Exactly, which brings us to the parsing phase and our

102
00:04:53.959 --> 00:04:56.000
<v Speaker 1>local speed reader. Beautiful Soup.

103
00:04:56.199 --> 00:04:59.920
<v Speaker 2>Yes, Beautiful Soup is designed specifically for this local extra

104
00:05:00.079 --> 00:05:00.720
<v Speaker 2>action phase.

105
00:05:00.959 --> 00:05:03.279
<v Speaker 1>Just to define it simply, it's a Python library that

106
00:05:03.319 --> 00:05:06.360
<v Speaker 1>pulls data out of HTML and XML files right exactly.

107
00:05:07.079 --> 00:05:11.079
<v Speaker 2>Rather than communicating across a network protocol, it takes the

108
00:05:11.199 --> 00:05:14.839
<v Speaker 2>raw HTML text and structures it locally in your Python

109
00:05:14.959 --> 00:05:16.279
<v Speaker 2>environment's memory, so.

110
00:05:16.199 --> 00:05:19.079
<v Speaker 1>You could just traverse it instantly. Chris, But you know,

111
00:05:19.160 --> 00:05:21.800
<v Speaker 1>before we even talk about how beautiful soupstructure is the memory,

112
00:05:21.920 --> 00:05:23.839
<v Speaker 1>I kind of want to play Devil's Advocate for a second.

113
00:05:23.879 --> 00:05:24.519
<v Speaker 2>Sure, go for it.

114
00:05:24.680 --> 00:05:29.399
<v Speaker 1>If parsing EACHTML is technically just string manipulation and we

115
00:05:29.439 --> 00:05:33.000
<v Speaker 1>have the whole HTML file locally, why pull in a

116
00:05:33.040 --> 00:05:34.839
<v Speaker 1>dedicated library at all?

117
00:05:35.120 --> 00:05:37.439
<v Speaker 2>Ah? The classic developer question.

118
00:05:37.399 --> 00:05:39.839
<v Speaker 1>Right, like, why couldn't I just write a few clever

119
00:05:40.040 --> 00:05:42.720
<v Speaker 1>regular expressions in Python to find the data I need.

120
00:05:42.959 --> 00:05:45.759
<v Speaker 2>It is a temptation every developer faces, and I'll tell

121
00:05:45.800 --> 00:05:47.439
<v Speaker 2>you it usually ends in.

122
00:05:47.480 --> 00:05:49.959
<v Speaker 1>Total disaster, really total disaster.

123
00:05:50.079 --> 00:05:53.480
<v Speaker 2>Yeah, because HTML is not a regular language. It features

124
00:05:53.639 --> 00:05:55.959
<v Speaker 2>nested recursive structures. Ah.

125
00:05:56.079 --> 00:05:56.240
<v Speaker 1>Right.

126
00:05:56.360 --> 00:05:59.319
<v Speaker 2>If you write a rejex to find a specific divtag

127
00:05:59.639 --> 00:06:02.639
<v Speaker 2>and the way websites developer and nested another div inside of.

128
00:06:02.560 --> 00:06:04.360
<v Speaker 1>It, well they just forgot to close the tag entirely,

129
00:06:04.360 --> 00:06:05.120
<v Speaker 1>which happens all.

130
00:06:05.040 --> 00:06:08.360
<v Speaker 2>The time exactly. Your rejects will break instantly or just

131
00:06:08.439 --> 00:06:12.040
<v Speaker 2>return garbage data. Beautiful soup doesn't just read the string.

132
00:06:12.199 --> 00:06:14.199
<v Speaker 2>It actually analyzes the syntax and.

133
00:06:14.240 --> 00:06:16.959
<v Speaker 1>Builds a structural representation of the document, right, which is

134
00:06:16.959 --> 00:06:18.040
<v Speaker 1>known as a parse tree.

135
00:06:18.120 --> 00:06:22.399
<v Speaker 2>Exactly. It turns this chaotic wall of text into a

136
00:06:22.519 --> 00:06:24.360
<v Speaker 2>navigable hierarchy.

137
00:06:24.120 --> 00:06:26.879
<v Speaker 1>Which is so much safer. Yeah, and speaking of chaos,

138
00:06:27.040 --> 00:06:30.240
<v Speaker 1>let's talk about the encoding nightmare. Oh yes, because before

139
00:06:30.279 --> 00:06:33.759
<v Speaker 1>you even worry about parsing syntax, just the sheer act

140
00:06:33.800 --> 00:06:37.160
<v Speaker 1>of importing web data can completely mangle your text.

141
00:06:37.560 --> 00:06:38.199
<v Speaker 2>Absolutely.

142
00:06:38.240 --> 00:06:41.600
<v Speaker 1>Anyone who has scraped international sites knows the pain of

143
00:06:42.120 --> 00:06:45.360
<v Speaker 1>seeing like currency symbols turn into hollow boxes.

144
00:06:45.480 --> 00:06:51.000
<v Speaker 2>Or apostrophes turning into weird question marks. It's character encoding mismatches. Yeah,

145
00:06:51.040 --> 00:06:53.720
<v Speaker 2>a web server might send you data encoded in Windows

146
00:06:53.759 --> 00:06:56.759
<v Speaker 2>twelve five to two or ISO eight eight five nine

147
00:06:56.839 --> 00:06:57.600
<v Speaker 2>one right.

148
00:06:57.439 --> 00:07:00.160
<v Speaker 1>And if your Python environment is strictly expecting utfa E.

149
00:07:00.240 --> 00:07:02.720
<v Speaker 2>It totally fails to map the raw bites to the

150
00:07:02.759 --> 00:07:04.120
<v Speaker 2>correct characters.

151
00:07:03.680 --> 00:07:06.759
<v Speaker 1>Which is a huge headache. But this brings up one

152
00:07:06.759 --> 00:07:10.920
<v Speaker 1>of the most powerful, yet kind of invisible features of

153
00:07:10.959 --> 00:07:11.680
<v Speaker 1>Beautiful Soup.

154
00:07:11.839 --> 00:07:15.279
<v Speaker 2>Yes, the sublibrary it uses and it's called unicode dam it.

155
00:07:15.480 --> 00:07:18.040
<v Speaker 1>Yeah. Wait, is that genuinely the name of the library?

156
00:07:18.120 --> 00:07:20.759
<v Speaker 2>It is, Yeah, and honestly it completely earns the name.

157
00:07:21.000 --> 00:07:22.720
<v Speaker 1>That's hilarious. How does it work?

158
00:07:22.920 --> 00:07:26.519
<v Speaker 2>When you pass raw HTML into Beautiful Soup, Unicode dam

159
00:07:26.519 --> 00:07:29.879
<v Speaker 2>it actively sniffs the document from meta tags to detect

160
00:07:29.879 --> 00:07:34.120
<v Speaker 2>the original encoding. It then automatically converts those incoming bytes

161
00:07:34.160 --> 00:07:36.680
<v Speaker 2>into standard Unicode for internal processing.

162
00:07:36.759 --> 00:07:37.680
<v Speaker 1>So it does it all for you.

163
00:07:37.879 --> 00:07:41.000
<v Speaker 2>Yep, And it ensures that any output you generate is

164
00:07:41.120 --> 00:07:42.759
<v Speaker 2>cleanly encoded in UTF eight.

165
00:07:42.920 --> 00:07:45.399
<v Speaker 1>So it just normalizes the byte data automatically. You don't

166
00:07:45.399 --> 00:07:48.519
<v Speaker 1>have to manually write like trixcept blocks to guess if

167
00:07:48.720 --> 00:07:49.959
<v Speaker 1>a page is in Latin one.

168
00:07:49.920 --> 00:07:52.279
<v Speaker 2>Or whatever exactly. It just hands you clean text. You

169
00:07:52.399 --> 00:07:54.680
<v Speaker 2>rarely have to think about encodings at all.

170
00:07:54.879 --> 00:07:57.560
<v Speaker 1>That is such a lifesaver. Is there ever a time

171
00:07:57.560 --> 00:07:58.040
<v Speaker 1>it fails?

172
00:07:58.399 --> 00:08:01.160
<v Speaker 2>The only real exception is if the ver provides literally

173
00:08:01.240 --> 00:08:05.240
<v Speaker 2>no encoding metadata and the byte sequence is entirely ambiguous.

174
00:08:05.279 --> 00:08:06.680
<v Speaker 1>Okay, so that's pretty rare.

175
00:08:06.839 --> 00:08:10.040
<v Speaker 2>Yeah, only then might you have to manually specify the format.

176
00:08:10.199 --> 00:08:12.759
<v Speaker 2>But for the vast majority of projects, it functions as

177
00:08:12.800 --> 00:08:14.680
<v Speaker 2>a seamless universal translator.

178
00:08:14.839 --> 00:08:18.360
<v Speaker 1>Awesome. Okay, so it translates device to Unicode. But beautiful

179
00:08:18.399 --> 00:08:21.879
<v Speaker 1>Soup itself doesn't actually do the heavy lifting of breaking

180
00:08:21.879 --> 00:08:23.800
<v Speaker 1>down the HTML tags right correct.

181
00:08:23.800 --> 00:08:26.800
<v Speaker 2>It relies on an underlying parser engine. Beautiful Soup is

182
00:08:26.879 --> 00:08:30.319
<v Speaker 2>essentially an interface. It sits on top of different parser engines,

183
00:08:30.399 --> 00:08:32.519
<v Speaker 2>allowing you to swap them out depending on whether your

184
00:08:32.600 --> 00:08:35.360
<v Speaker 2>priority is you know, speed or flexibility.

185
00:08:35.519 --> 00:08:38.000
<v Speaker 1>I know Python comes with the built in HTML parser.

186
00:08:38.039 --> 00:08:38.759
<v Speaker 1>Is that one any good?

187
00:08:39.600 --> 00:08:42.480
<v Speaker 2>It's decent for simple scripts, but if we are building

188
00:08:42.519 --> 00:08:44.720
<v Speaker 2>a professional pipeline, you need something better.

189
00:08:45.120 --> 00:08:46.960
<v Speaker 1>What are the serious alternatives? Then?

190
00:08:47.559 --> 00:08:50.120
<v Speaker 2>You generally look at two main options. The first is

191
00:08:50.320 --> 00:08:51.240
<v Speaker 2>HTML five lib.

192
00:08:51.279 --> 00:08:52.600
<v Speaker 1>Okay, what's special about that one.

193
00:08:52.679 --> 00:08:56.360
<v Speaker 2>This engine parses HTML exactly the same way a modern

194
00:08:56.399 --> 00:08:58.960
<v Speaker 2>web browser would. It is incredibly.

195
00:08:58.440 --> 00:09:01.240
<v Speaker 1>Forgiving, meaning if the HTML is.

196
00:09:01.159 --> 00:09:05.440
<v Speaker 2>Broken exactly, if a webpage has terribly broken HTML, like

197
00:09:05.679 --> 00:09:10.200
<v Speaker 2>missing closing tags, overlapping elements HTML five, lib will actually

198
00:09:10.240 --> 00:09:12.399
<v Speaker 2>syntactically repair it and build a valid tree.

199
00:09:12.440 --> 00:09:13.279
<v Speaker 1>Oh that's really cool.

200
00:09:13.279 --> 00:09:16.080
<v Speaker 2>But there's a downside, right Yeah, it's written in pure Python,

201
00:09:16.159 --> 00:09:17.240
<v Speaker 2>making it quite.

202
00:09:16.960 --> 00:09:20.080
<v Speaker 1>Slow, which completely defeats the purpose of moving away from

203
00:09:20.159 --> 00:09:22.879
<v Speaker 1>seeing eme slowness in the first place. We want raw

204
00:09:22.960 --> 00:09:24.279
<v Speaker 1>speed here, right.

205
00:09:24.360 --> 00:09:27.919
<v Speaker 2>Which points us to the absolute industry standard for this workflow,

206
00:09:28.639 --> 00:09:31.600
<v Speaker 2>the LexML parser ah XML.

207
00:09:31.600 --> 00:09:32.919
<v Speaker 1>Why is it so much faster.

208
00:09:32.919 --> 00:09:36.200
<v Speaker 2>Because it relies on highly optimized SEA libraries under the hood.

209
00:09:36.279 --> 00:09:39.159
<v Speaker 2>Oh nice, Because it executes at sea level speed rather

210
00:09:39.200 --> 00:09:42.320
<v Speaker 2>than being interpreted step by step in Python. It can

211
00:09:42.399 --> 00:09:45.799
<v Speaker 2>parse massive documents and traverse millions of nodes in a

212
00:09:45.799 --> 00:09:47.080
<v Speaker 2>fraction of a second.

213
00:09:47.159 --> 00:09:50.000
<v Speaker 1>The Sea extension advantage. I love that. So we take

214
00:09:50.039 --> 00:09:52.879
<v Speaker 1>our raw HTML, we feed it to beautiful soup, and

215
00:09:52.919 --> 00:09:54.960
<v Speaker 1>the LexML engine just rips through.

216
00:09:54.759 --> 00:09:58.039
<v Speaker 2>It exactly, building this parse tree in our local memory.

217
00:09:58.279 --> 00:10:01.200
<v Speaker 1>Let's dig into the anatomy of that tree for a

218
00:10:01.240 --> 00:10:05.000
<v Speaker 1>developer writing the actual extraction logic. How is this dom

219
00:10:05.159 --> 00:10:07.399
<v Speaker 1>data mapped into Python objects?

220
00:10:07.919 --> 00:10:11.080
<v Speaker 2>Well, the parser maps the entire HTML document into a

221
00:10:11.120 --> 00:10:15.559
<v Speaker 2>tree consisting of four primary Python objects. Okay, four objects, yeah,

222
00:10:15.600 --> 00:10:18.600
<v Speaker 2>and understanding how these objects interact is really the key

223
00:10:18.679 --> 00:10:21.720
<v Speaker 2>to navigating the structure without getting totally lost in the code.

224
00:10:21.960 --> 00:10:23.080
<v Speaker 1>Right, So what's the first one?

225
00:10:23.200 --> 00:10:25.840
<v Speaker 2>The root of this entire structure is the beautiful soup

226
00:10:25.960 --> 00:10:26.799
<v Speaker 2>object itself.

227
00:10:27.039 --> 00:10:30.120
<v Speaker 1>So if we think about tree data structures and computer science,

228
00:10:30.840 --> 00:10:34.080
<v Speaker 1>the beautiful soup object is basically the root node exactly.

229
00:10:34.120 --> 00:10:37.039
<v Speaker 1>It's like the entire filing cabinet containing all the folders.

230
00:10:37.360 --> 00:10:39.879
<v Speaker 2>That's a perfect analogy. It contains the entire state of

231
00:10:39.919 --> 00:10:43.039
<v Speaker 2>the document. And because it is the root representation of

232
00:10:43.080 --> 00:10:46.240
<v Speaker 2>the document as a whole, it doesn't represent any specific

233
00:10:46.440 --> 00:10:47.360
<v Speaker 2>HTML tag.

234
00:10:47.679 --> 00:10:50.440
<v Speaker 1>Oh interesting, So I couldn't query the beautiful soup object

235
00:10:50.440 --> 00:10:53.240
<v Speaker 1>directly for like a class attribute.

236
00:10:53.320 --> 00:10:55.879
<v Speaker 2>No you can't, or in reflink. It has no attributes

237
00:10:55.919 --> 00:10:58.519
<v Speaker 2>of its own. It serves strictly as your starting point

238
00:10:58.519 --> 00:11:01.120
<v Speaker 2>to initiate searches down into the branches.

239
00:11:00.919 --> 00:11:04.600
<v Speaker 1>Gotcha, which leads us to the internal nodes of the tree.

240
00:11:05.519 --> 00:11:08.360
<v Speaker 1>The folders in our filing cabinet, which are the tag objects.

241
00:11:08.600 --> 00:11:12.840
<v Speaker 2>Right When the parser encounters an HTML element, say an

242
00:11:12.960 --> 00:11:15.519
<v Speaker 2>H one header, a DIV container, or a P paragraph,

243
00:11:15.840 --> 00:11:18.639
<v Speaker 2>it wraps that structural element in a tag object.

244
00:11:18.720 --> 00:11:21.080
<v Speaker 1>And this is where the pythonic nature of the library

245
00:11:21.120 --> 00:11:23.559
<v Speaker 1>really shines. I think, oh definitely, because if I have

246
00:11:23.600 --> 00:11:26.039
<v Speaker 1>a tag object assigned to a variable, I get to

247
00:11:26.080 --> 00:11:28.799
<v Speaker 1>type tag dot name to find out what kind of

248
00:11:28.919 --> 00:11:29.840
<v Speaker 1>HTML element it.

249
00:11:29.840 --> 00:11:33.720
<v Speaker 2>Is, or access its attributes dynamically like a Python dictionary

250
00:11:33.799 --> 00:11:36.000
<v Speaker 2>to check its eyed or class.

251
00:11:36.000 --> 00:11:36.799
<v Speaker 1>That's so clean.

252
00:11:37.039 --> 00:11:39.960
<v Speaker 2>You can even modify those tags dynamically, altering the true

253
00:11:39.960 --> 00:11:42.159
<v Speaker 2>structure on the fly if you need to. But the

254
00:11:42.240 --> 00:11:45.440
<v Speaker 2>tag object is just the structural boundary. It tells you

255
00:11:45.480 --> 00:11:49.320
<v Speaker 2>where the data lives, not what the data actually is, which.

256
00:11:49.080 --> 00:11:52.200
<v Speaker 1>Brings us to the actual documents inside the folders.

257
00:11:51.759 --> 00:11:55.600
<v Speaker 2>Exactly when you want the actual human readable text inside

258
00:11:55.639 --> 00:11:59.080
<v Speaker 2>that element, you are looking for the third object type

259
00:12:00.000 --> 00:12:00.799
<v Speaker 2>favigable string.

260
00:12:01.120 --> 00:12:04.639
<v Speaker 1>This is a really crucial distinction for people writing extraction scripts.

261
00:12:05.320 --> 00:12:08.480
<v Speaker 1>The tag and the text inside it are two completely

262
00:12:08.519 --> 00:12:09.720
<v Speaker 1>different objects in memory.

263
00:12:09.879 --> 00:12:13.360
<v Speaker 2>They have to be Think about a complex HTML structure.

264
00:12:14.000 --> 00:12:16.600
<v Speaker 2>You might have a paragraph tag that contains some text,

265
00:12:16.840 --> 00:12:20.039
<v Speaker 2>but it also contains a bold tag which contains more text.

266
00:12:20.240 --> 00:12:23.000
<v Speaker 1>Right, so the tag object defines the nesting exactly.

267
00:12:23.039 --> 00:12:25.279
<v Speaker 2>The navigal string is the leaf node of our tree.

268
00:12:25.440 --> 00:12:27.720
<v Speaker 2>It contains nothing but the pure string literal.

269
00:12:28.080 --> 00:12:30.919
<v Speaker 1>So if I query tagged for its text, beautiful soup

270
00:12:31.039 --> 00:12:33.480
<v Speaker 1>is actually like traversing down to the leaf nose, stripping

271
00:12:33.480 --> 00:12:36.519
<v Speaker 1>away all the structural boundaries and returning just the combined

272
00:12:36.559 --> 00:12:37.600
<v Speaker 1>navigable string data.

273
00:12:37.799 --> 00:12:40.840
<v Speaker 2>Yes, which is exactly why it's so much more reliable

274
00:12:40.879 --> 00:12:45.200
<v Speaker 2>than regular expressions. You navigate parent, child, and sibling relationships natively.

275
00:12:45.519 --> 00:12:48.279
<v Speaker 1>That makes total sense, and that brings us to the

276
00:12:48.320 --> 00:12:50.679
<v Speaker 1>fourth and final object, which you mentioned earlier. Is a

277
00:12:50.679 --> 00:12:54.120
<v Speaker 1>bit of an edge case, yeah, but highly relevant for dynamic.

278
00:12:53.759 --> 00:12:55.399
<v Speaker 2>Sites, right, the common object.

279
00:12:55.840 --> 00:12:58.679
<v Speaker 1>Now, why do we care about developer comments in the

280
00:12:58.720 --> 00:13:02.159
<v Speaker 1>source code? Isn't that just leftover notes like update the

281
00:13:02.200 --> 00:13:03.799
<v Speaker 1>CSS later or something?

282
00:13:04.159 --> 00:13:08.720
<v Speaker 2>Sometimes? Yes, But in modern web development, frontend frameworks often

283
00:13:08.840 --> 00:13:13.080
<v Speaker 2>use HTML comments to temporarily hide dynamic data or raw

284
00:13:13.159 --> 00:13:16.360
<v Speaker 2>Jason payloads. Oh wait really yeah, they hide it there

285
00:13:16.360 --> 00:13:18.600
<v Speaker 2>before it's officially injected into the active dom.

286
00:13:18.799 --> 00:13:19.679
<v Speaker 1>That's fascinating.

287
00:13:20.000 --> 00:13:23.759
<v Speaker 2>So by treating comments as a specialized type of navigable string,

288
00:13:24.200 --> 00:13:27.279
<v Speaker 2>beautiful Soup allows you to easily search for and extract

289
00:13:27.639 --> 00:13:29.279
<v Speaker 2>that hidden application state.

290
00:13:29.440 --> 00:13:32.559
<v Speaker 1>It's incredible how much structure is hiding behind what looks

291
00:13:32.600 --> 00:13:34.440
<v Speaker 1>like a totally chaotic wall of code.

292
00:13:34.600 --> 00:13:35.200
<v Speaker 2>It really is.

293
00:13:35.440 --> 00:13:37.919
<v Speaker 1>So we have the beautiful Soup root, the tag branches,

294
00:13:38.159 --> 00:13:41.120
<v Speaker 1>the naviglobal string leaves, and the comment hidden leaves.

295
00:13:41.320 --> 00:13:44.679
<v Speaker 2>And once you understand that memory structure, writing the extraction

296
00:13:44.799 --> 00:13:48.919
<v Speaker 2>logic becomes almost trivial. You're just traversing a well organized

297
00:13:48.960 --> 00:13:50.000
<v Speaker 2>Python data structure.

298
00:13:50.159 --> 00:13:53.000
<v Speaker 1>Okay, let's bring this all together. We establish the bottleneck

299
00:13:53.039 --> 00:13:56.919
<v Speaker 1>of Selenium, right. Yeah, it's brilliant at rendering JavaScript and

300
00:13:57.000 --> 00:14:02.840
<v Speaker 1>navigating dynamic state, but agonizingly slow at extracting data because

301
00:14:02.840 --> 00:14:04.720
<v Speaker 1>of that HTTP serialization tax.

302
00:14:04.960 --> 00:14:05.399
<v Speaker 2>Correct.

303
00:14:05.960 --> 00:14:09.720
<v Speaker 1>Then we explored beautiful Soup backed by a C level

304
00:14:09.840 --> 00:14:13.960
<v Speaker 1>LXML parser. It can traverse millions of local nodes in milliseconds,

305
00:14:14.879 --> 00:14:17.840
<v Speaker 1>but it can't run JavaScript exactly. So how do we

306
00:14:17.840 --> 00:14:19.519
<v Speaker 1>build a bridge between them? What's the workflow?

307
00:14:19.600 --> 00:14:22.879
<v Speaker 2>The architectural solution is to strictly separate the state generation

308
00:14:22.960 --> 00:14:25.960
<v Speaker 2>phase from the data extraction phase. You do not force

309
00:14:26.039 --> 00:14:27.120
<v Speaker 2>one tool to do both.

310
00:14:27.279 --> 00:14:29.679
<v Speaker 1>Okay, so step one of this transition strategy is state

311
00:14:29.720 --> 00:14:33.000
<v Speaker 1>generation we use Selenium as our navigator. We boot up

312
00:14:33.000 --> 00:14:35.559
<v Speaker 1>the webdriver, navigate to the target URL, and let the

313
00:14:35.600 --> 00:14:38.320
<v Speaker 1>real browser engine execute the initial JavaScript.

314
00:14:38.360 --> 00:14:40.559
<v Speaker 2>You let the browser do what it does best. You

315
00:14:40.600 --> 00:14:44.679
<v Speaker 2>write Selenium commands to interact with the page, injecting authentication tokens,

316
00:14:44.960 --> 00:14:48.840
<v Speaker 2>clicking through drop down venues, scrolling to trigger lazy loaded images.

317
00:14:48.559 --> 00:14:52.559
<v Speaker 1>Basically just manipulating the application until the specific dynamic data

318
00:14:52.600 --> 00:14:54.840
<v Speaker 1>you want is visually rendered on the screen.

319
00:14:55.240 --> 00:14:59.039
<v Speaker 2>Yes, the DOM is now fully updated in the browser's memory.

320
00:14:59.159 --> 00:15:01.919
<v Speaker 2>The data is there, and here is the pivot.

321
00:15:02.000 --> 00:15:02.840
<v Speaker 1>Okay, the pivot.

322
00:15:02.919 --> 00:15:06.879
<v Speaker 2>We completely stop using Selenium's element location methods. We don't

323
00:15:06.919 --> 00:15:08.720
<v Speaker 2>ask it to read the data row by row.

324
00:15:08.840 --> 00:15:12.039
<v Speaker 1>Because that would trigger all those HTTT requests exactly.

325
00:15:12.559 --> 00:15:14.600
<v Speaker 2>Instead, we ask it to dump the memory.

326
00:15:14.720 --> 00:15:15.480
<v Speaker 1>How do we do that?

327
00:15:15.679 --> 00:15:19.440
<v Speaker 2>You call the page source property on your Selenium driver. Okay,

328
00:15:19.480 --> 00:15:22.600
<v Speaker 2>this is the master stroke of this architecture. Instead of

329
00:15:22.639 --> 00:15:26.320
<v Speaker 2>making thousands of network requests to read individual tags, this

330
00:15:26.440 --> 00:15:29.600
<v Speaker 2>executes a single request over the webdriver protocol.

331
00:15:29.720 --> 00:15:30.279
<v Speaker 1>Oh wow.

332
00:15:30.679 --> 00:15:34.480
<v Speaker 2>It asks the browser to serialize the entire fully rendered

333
00:15:34.519 --> 00:15:38.240
<v Speaker 2>DOM state as one massive HTML string and send it

334
00:15:38.279 --> 00:15:39.519
<v Speaker 2>back to your Python script.

335
00:15:39.799 --> 00:15:43.440
<v Speaker 1>So we pay that heavy network communication tax exactly. Once

336
00:15:43.480 --> 00:15:46.559
<v Speaker 1>we literally just grab the entire encyclopedia and bring it out.

337
00:15:46.720 --> 00:15:49.519
<v Speaker 2>Yes, once that raw HTML string is in your local

338
00:15:49.600 --> 00:15:53.000
<v Speaker 2>Python memory, Selenium's job is done. You can minimize the

339
00:15:53.000 --> 00:15:54.159
<v Speaker 2>browser or close it.

340
00:15:54.159 --> 00:15:56.080
<v Speaker 1>Entirely amend we hand it off.

341
00:15:56.159 --> 00:15:58.960
<v Speaker 2>Yep. You take that massive string and pass it directly

342
00:15:59.000 --> 00:16:02.240
<v Speaker 2>into the beautiful soup constructor, making sure to specify the

343
00:16:02.480 --> 00:16:04.120
<v Speaker 2>LXML parser.

344
00:16:04.039 --> 00:16:08.399
<v Speaker 1>And instantly we shift from network bound execution to CPU

345
00:16:08.480 --> 00:16:12.200
<v Speaker 1>bound execution. The C engine rips through the HTML builds,

346
00:16:12.240 --> 00:16:15.559
<v Speaker 1>our root object, our tags, and our navigable strings.

347
00:16:15.799 --> 00:16:18.639
<v Speaker 2>From that point forward, all your iteration, all your searching

348
00:16:18.679 --> 00:16:22.759
<v Speaker 2>for specific classes IDs where text patterns happens locally, so.

349
00:16:22.759 --> 00:16:27.720
<v Speaker 1>You extract your data points instantly bypassing the HTTP overhead entirely.

350
00:16:27.720 --> 00:16:31.000
<v Speaker 1>That's a surprisingly clean workaround. You know. You let the

351
00:16:31.039 --> 00:16:35.279
<v Speaker 1>heavy armoured tool handle the complex environment navigation and you

352
00:16:35.360 --> 00:16:38.679
<v Speaker 1>let the highly optimized parser handle the data traversal.

353
00:16:38.879 --> 00:16:41.559
<v Speaker 2>It really highlights the difference between just knowing how a

354
00:16:41.600 --> 00:16:45.080
<v Speaker 2>library works and knowing how to architect a scalable system totally.

355
00:16:45.159 --> 00:16:48.840
<v Speaker 2>It is the defining design pattern for professional webscraping pipelines.

356
00:16:49.360 --> 00:16:52.159
<v Speaker 2>The next time you're building an extraction tool. Remember to

357
00:16:52.200 --> 00:16:56.559
<v Speaker 2>separate your concerns. State generation belongs to the browser automation tool,

358
00:16:57.120 --> 00:16:59.480
<v Speaker 2>data extraction belongs to the local parser.

359
00:16:59.559 --> 00:17:02.440
<v Speaker 1>That is fantastic advice. Well, before we wrap up, we

360
00:17:02.440 --> 00:17:04.319
<v Speaker 1>want to leave you with a quick review exercise to

361
00:17:04.359 --> 00:17:06.359
<v Speaker 1>test your own understanding of this architecture.

362
00:17:06.640 --> 00:17:09.240
<v Speaker 2>Yeah, so, think back to the bottleneck we diagnosed at

363
00:17:09.279 --> 00:17:12.279
<v Speaker 2>the start of our deep dive. Why do we extract

364
00:17:12.319 --> 00:17:15.599
<v Speaker 2>the full page source to pass to beautiful soup instead

365
00:17:15.640 --> 00:17:19.839
<v Speaker 2>of just querying individual tags through Selenium? Good question, And

366
00:17:20.000 --> 00:17:25.359
<v Speaker 2>what specific protocol mechanism is responsible for Selenium's data extraction bottleneck.

367
00:17:25.519 --> 00:17:27.599
<v Speaker 1>I'll give you a second to think about that. If

368
00:17:27.680 --> 00:17:31.160
<v Speaker 1>you immediately thought about avoiding the thousands of serialization trips

369
00:17:31.160 --> 00:17:34.160
<v Speaker 1>over the jason Wire protocol by grabbing the memory in

370
00:17:34.200 --> 00:17:37.319
<v Speaker 1>one go, well your head is in the exact right place.

371
00:17:37.480 --> 00:17:39.920
<v Speaker 1>Spot on. Here's a final thought to mull over. As

372
00:17:39.960 --> 00:17:42.279
<v Speaker 1>you close out this deep dive, will you realize that

373
00:17:42.319 --> 00:17:44.680
<v Speaker 1>you can write a few lines of Python to programmatically

374
00:17:44.680 --> 00:17:49.000
<v Speaker 1>control a browser trigger any dynamic state, instantly dump the

375
00:17:49.079 --> 00:17:52.680
<v Speaker 1>raw HTML, and traverse it locally at sea level speeds?

376
00:17:52.839 --> 00:17:54.400
<v Speaker 2>It really changes your perspective.

377
00:17:54.519 --> 00:17:57.240
<v Speaker 1>It does you realize the Web isn't just a collection

378
00:17:57.319 --> 00:18:00.480
<v Speaker 1>of pages to visit. It is a massive, globally distributed,

379
00:18:00.599 --> 00:18:04.319
<v Speaker 1>queriable database just waiting for your instructions. The only question

380
00:18:04.400 --> 00:18:06.279
<v Speaker 1>left is what will you extract first