WEBVTT

1
00:00:00.040 --> 00:00:02.359
<v Speaker 1>I want you to picture something for a second. You're

2
00:00:02.359 --> 00:00:04.440
<v Speaker 1>sitting at your computer and you're doing something you do

3
00:00:04.559 --> 00:00:07.080
<v Speaker 1>probably a dozen times a day. You type a search

4
00:00:07.200 --> 00:00:11.400
<v Speaker 1>term into a massive e commerce site like Amazon or Walmart,

5
00:00:11.439 --> 00:00:14.400
<v Speaker 1>right right, yeah, and you hit enter and then boom,

6
00:00:14.400 --> 00:00:18.239
<v Speaker 1>magic happens. A customized list of products just pops onto

7
00:00:18.239 --> 00:00:21.559
<v Speaker 1>your screen. But what is actually happening behind the scenes

8
00:00:21.719 --> 00:00:24.559
<v Speaker 1>when we submit that data to a server, Like, how

9
00:00:24.559 --> 00:00:26.039
<v Speaker 1>does a process our intent?

10
00:00:26.600 --> 00:00:29.239
<v Speaker 2>Well, it's funny you call it magic, because you know,

11
00:00:29.239 --> 00:00:31.920
<v Speaker 2>when you're first learning about programming or data science, it

12
00:00:31.960 --> 00:00:34.200
<v Speaker 2>really does feel that way totally. But it's not magic.

13
00:00:34.240 --> 00:00:38.119
<v Speaker 2>It's essentially a digital contract. Basic static websites, they don't

14
00:00:38.119 --> 00:00:40.759
<v Speaker 2>really require you to know much. Pulling text off a

15
00:00:40.759 --> 00:00:43.880
<v Speaker 2>flat article is one thing, But interacting with search bars

16
00:00:43.880 --> 00:00:47.560
<v Speaker 2>and massive databases that is essential for any aspiring tech.

17
00:00:47.520 --> 00:00:50.759
<v Speaker 1>Enthusiast exactly, which is the whole mission of our Deep

18
00:00:50.799 --> 00:00:53.000
<v Speaker 1>Die today. We are going to learn how to master

19
00:00:53.119 --> 00:00:57.840
<v Speaker 1>web scraping by understanding how to programmatically handle these web

20
00:00:57.840 --> 00:00:58.759
<v Speaker 1>form submissions.

21
00:00:59.039 --> 00:01:01.359
<v Speaker 2>Yeah, because to get that dynamic data, you can't just

22
00:01:01.399 --> 00:01:04.680
<v Speaker 2>passively read a page. You have to actively converse with

23
00:01:04.719 --> 00:01:07.959
<v Speaker 2>the server. And to do that you need to know

24
00:01:08.000 --> 00:01:10.719
<v Speaker 2>the map, the map. I like that, right, So we're

25
00:01:10.719 --> 00:01:12.640
<v Speaker 2>going to break down the two dominant types of web

26
00:01:12.680 --> 00:01:15.920
<v Speaker 2>forms you'll encounter out in the wild. That's get forms

27
00:01:16.319 --> 00:01:20.239
<v Speaker 2>and post forms, and you know how to actually write

28
00:01:20.280 --> 00:01:21.920
<v Speaker 2>Python code to handle them.

29
00:01:21.920 --> 00:01:23.879
<v Speaker 1>Okay, let's unpack this. We'll start with the first type,

30
00:01:23.879 --> 00:01:25.519
<v Speaker 1>which I think you said is the most common, right,

31
00:01:25.719 --> 00:01:26.879
<v Speaker 1>the get form.

32
00:01:27.000 --> 00:01:30.159
<v Speaker 2>Yeah, exactly. A geet form is basically how the vast

33
00:01:30.200 --> 00:01:35.159
<v Speaker 2>majority of simple web pages work, and it's defining characteristic

34
00:01:35.400 --> 00:01:37.280
<v Speaker 2>is transparency.

35
00:01:37.439 --> 00:01:38.920
<v Speaker 1>Transparency. What do you mean by that?

36
00:01:39.319 --> 00:01:41.599
<v Speaker 2>Well, the parameters of what you're searching for are placed

37
00:01:41.599 --> 00:01:44.439
<v Speaker 2>directly out in the open, right in the URL. So

38
00:01:44.480 --> 00:01:46.680
<v Speaker 2>if you go to a huge database like the Internet

39
00:01:46.680 --> 00:01:50.000
<v Speaker 2>movie database IMDb, and you search for an exact name

40
00:01:50.079 --> 00:01:52.239
<v Speaker 2>match let's say, an actor name mark.

41
00:01:52.359 --> 00:01:55.120
<v Speaker 1>Right, I just type mark into the search bar.

42
00:01:55.319 --> 00:01:57.879
<v Speaker 2>You do, and when you hit enter, you arrive at

43
00:01:57.920 --> 00:02:00.159
<v Speaker 2>a new page. But if you look up at your

44
00:02:00.159 --> 00:02:03.519
<v Speaker 2>browser's address bar, the URL has completely changed.

45
00:02:03.599 --> 00:02:06.760
<v Speaker 1>Right, It's not just MDB dot dot com anymore, exactly.

46
00:02:06.920 --> 00:02:09.120
<v Speaker 2>It's a long string of text and right at the

47
00:02:09.199 --> 00:02:12.280
<v Speaker 2>end you'll see a question mark followed by something like

48
00:02:13.280 --> 00:02:14.159
<v Speaker 2>qequals mark.

49
00:02:14.360 --> 00:02:16.039
<v Speaker 1>Oh, Wow. I see that all the time and I

50
00:02:16.080 --> 00:02:18.439
<v Speaker 1>just sort of ignore it. But it's like a postcard.

51
00:02:18.560 --> 00:02:19.960
<v Speaker 2>A postcard, Yeah, think about it.

52
00:02:20.000 --> 00:02:24.240
<v Speaker 1>A get form is literally like a postcard. Anyone handling

53
00:02:24.240 --> 00:02:26.800
<v Speaker 1>it can read the message right on the back. The

54
00:02:26.919 --> 00:02:30.960
<v Speaker 1>que in that URL literally denotes the query, the term

55
00:02:31.039 --> 00:02:33.759
<v Speaker 1>you searched for. It's just sitting there in plaintext.

56
00:02:33.960 --> 00:02:37.840
<v Speaker 2>That's a perfect analogy. Actually, the question mark is the

57
00:02:37.840 --> 00:02:41.879
<v Speaker 2>dividing line between the server address and your specific instructions.

58
00:02:42.159 --> 00:02:45.639
<v Speaker 2>And because it's a postcard, anyone, your local router, the

59
00:02:45.680 --> 00:02:48.800
<v Speaker 2>internet service provider can read it without opening an envelope.

60
00:02:48.919 --> 00:02:50.800
<v Speaker 1>So if I want to scrape this data using Python,

61
00:02:50.840 --> 00:02:53.199
<v Speaker 1>I basically just have to forge that postcard. Right.

62
00:02:53.280 --> 00:02:54.800
<v Speaker 2>You hit the nail on the head. You don't have

63
00:02:54.879 --> 00:02:57.400
<v Speaker 2>to trick a browser into clicking a button. You just

64
00:02:57.520 --> 00:02:58.639
<v Speaker 2>generate the right URL.

65
00:02:59.120 --> 00:03:01.800
<v Speaker 1>Walk me through the actual code for that, like step

66
00:03:01.800 --> 00:03:02.199
<v Speaker 1>by step.

67
00:03:02.240 --> 00:03:05.479
<v Speaker 2>Okay, so first you import your necessary packages for this.

68
00:03:05.520 --> 00:03:09.479
<v Speaker 2>You really just need two foundational Python libraries, Requests and

69
00:03:09.719 --> 00:03:10.479
<v Speaker 2>beautiful Soup.

70
00:03:10.719 --> 00:03:12.639
<v Speaker 1>Okay, requests and beautiful Soup.

71
00:03:12.759 --> 00:03:15.439
<v Speaker 2>Got it. So you use the get mechod from the

72
00:03:15.479 --> 00:03:18.680
<v Speaker 2>request library. You literally type request stut get and you

73
00:03:18.759 --> 00:03:22.319
<v Speaker 2>feed it that full URL string with your q equals

74
00:03:22.360 --> 00:03:23.879
<v Speaker 2>mark parameter.

75
00:03:23.560 --> 00:03:26.159
<v Speaker 1>And Python basically acts as my browser firing off that

76
00:03:26.240 --> 00:03:27.319
<v Speaker 1>postcard exactly.

77
00:03:27.400 --> 00:03:30.400
<v Speaker 2>It retrieves the raw HTML data from the URL. But

78
00:03:30.639 --> 00:03:33.599
<v Speaker 2>you know, eahml is just a massive wall of text

79
00:03:33.680 --> 00:03:35.759
<v Speaker 2>and angled brackets. It's really messy, right.

80
00:03:35.680 --> 00:03:37.919
<v Speaker 1>So how do I pull just the actors names.

81
00:03:37.680 --> 00:03:40.240
<v Speaker 2>Out of that mess That's where you pass the content

82
00:03:40.319 --> 00:03:43.000
<v Speaker 2>through the built in parser in beautiful Soup. You don't

83
00:03:43.000 --> 00:03:45.919
<v Speaker 2>just search the text. You go to the actual web

84
00:03:45.960 --> 00:03:49.159
<v Speaker 2>page in your regular browser first write click and hit

85
00:03:49.240 --> 00:03:50.439
<v Speaker 2>inspect ah.

86
00:03:50.360 --> 00:03:52.479
<v Speaker 1>The inspect element tool. I love that thing.

87
00:03:52.639 --> 00:03:55.280
<v Speaker 2>It's the best. You use it to find these specific

88
00:03:55.400 --> 00:03:58.759
<v Speaker 2>HTML tags or CSS classes that hold the actors' names.

89
00:03:59.080 --> 00:04:02.319
<v Speaker 2>Then you tell beautiful Soup to find those exact elements

90
00:04:02.360 --> 00:04:04.240
<v Speaker 2>in the raw data it downloaded, and it just.

91
00:04:04.280 --> 00:04:07.680
<v Speaker 1>Plucks them right out. That is incredibly powerful, And because

92
00:04:07.680 --> 00:04:10.000
<v Speaker 1>all the data is right there in the URL, gettt

93
00:04:10.159 --> 00:04:12.840
<v Speaker 1>forms are incredibly easy to share too, Like I can

94
00:04:12.879 --> 00:04:14.719
<v Speaker 1>copy and paste that link to a friend and they'll

95
00:04:14.719 --> 00:04:16.439
<v Speaker 1>see the exact same list of actors.

96
00:04:16.759 --> 00:04:20.600
<v Speaker 2>Yes, because the URL contains the entire state of the search.

97
00:04:21.000 --> 00:04:21.720
<v Speaker 2>It's brilliant.

98
00:04:22.000 --> 00:04:24.759
<v Speaker 1>But wait a minute, if geet forms are so easy

99
00:04:24.800 --> 00:04:27.680
<v Speaker 1>and so shareable, why do we need anything else? Like

100
00:04:27.759 --> 00:04:29.680
<v Speaker 1>what happens when a website doesn't want to put all

101
00:04:29.720 --> 00:04:31.519
<v Speaker 1>its cards on the table in the URL?

102
00:04:31.879 --> 00:04:34.759
<v Speaker 2>Well, think about your postcard analogy. What if you're sending

103
00:04:34.759 --> 00:04:37.120
<v Speaker 2>your bank account number or a password?

104
00:04:37.319 --> 00:04:39.839
<v Speaker 1>Oh right, I definitely don't want that written on the

105
00:04:39.839 --> 00:04:42.120
<v Speaker 1>back of a postcard for my WiFi router to log.

106
00:04:42.040 --> 00:04:45.399
<v Speaker 2>Exactly when you're dealing with sensitive data or when a

107
00:04:45.519 --> 00:04:48.680
<v Speaker 2>request actually changes something on the server, like updating a profile,

108
00:04:48.879 --> 00:04:52.199
<v Speaker 2>you need a sealed envelope. And that introduces us to

109
00:04:52.759 --> 00:04:54.839
<v Speaker 2>the hidden reality of PST.

110
00:04:54.600 --> 00:04:56.680
<v Speaker 1>Forms the post form okay.

111
00:04:56.519 --> 00:04:59.079
<v Speaker 2>And I have to warn you these are fundamentally not

112
00:04:59.160 --> 00:05:03.720
<v Speaker 2>scrape friendly. Unlike geet forms, PSTR forms do not change

113
00:05:03.720 --> 00:05:06.639
<v Speaker 2>the URL to reflect the search parameters, so the data

114
00:05:06.720 --> 00:05:09.240
<v Speaker 2>is just hidden completely. Let me give you a real

115
00:05:09.279 --> 00:05:12.600
<v Speaker 2>world example. There's this database run by the California Department

116
00:05:12.600 --> 00:05:17.680
<v Speaker 2>of Consumer Affairs the DCA. They license practitioners across like

117
00:05:17.680 --> 00:05:19.480
<v Speaker 2>two hundred and fifty five different professions.

118
00:05:19.480 --> 00:05:21.240
<v Speaker 1>Okay, So let's say I want to find a specific

119
00:05:21.360 --> 00:05:22.759
<v Speaker 1>dentist in California.

120
00:05:22.920 --> 00:05:25.319
<v Speaker 2>Right, you go to the site. It's like the Dental

121
00:05:25.360 --> 00:05:28.199
<v Speaker 2>Board of California. You're from a dropdown. You choose dentist license,

122
00:05:28.959 --> 00:05:31.680
<v Speaker 2>and let's say you leave the license number blank, but

123
00:05:31.759 --> 00:05:34.240
<v Speaker 2>you type in the first name John and the last.

124
00:05:34.120 --> 00:05:36.680
<v Speaker 1>Name Kelly standard search form staff standard stuff.

125
00:05:36.879 --> 00:05:40.160
<v Speaker 2>You hit search, the database churns, and it spits back

126
00:05:40.160 --> 00:05:43.720
<v Speaker 2>a result showing say, eight people holding that specific license.

127
00:05:43.800 --> 00:05:46.639
<v Speaker 1>Here's where it gets really interesting, though. If you check

128
00:05:46.720 --> 00:05:51.480
<v Speaker 1>the URL on this results page, it gives absolutely no indication.

129
00:05:51.079 --> 00:05:52.120
<v Speaker 2>Of your search none at all.

130
00:05:52.199 --> 00:05:55.319
<v Speaker 1>It's just a generic web address. There's no query string,

131
00:05:55.399 --> 00:05:58.879
<v Speaker 1>no question mark, no John Kelly anywhere. The parameters are

132
00:05:59.160 --> 00:06:00.000
<v Speaker 1>entirely hidden.

133
00:06:00.160 --> 00:06:03.120
<v Speaker 2>Right. Building on what we said earlier, a POSTD form

134
00:06:03.279 --> 00:06:06.439
<v Speaker 2>is like a sealed envelope. You securely hand the data

135
00:06:06.480 --> 00:06:09.839
<v Speaker 2>to the server, but it's completely hidden from the outside observer.

136
00:06:09.600 --> 00:06:12.240
<v Speaker 1>And it's hidden from a simple URL scrape too.

137
00:06:12.360 --> 00:06:15.959
<v Speaker 2>Exactly. If you open Python and try that request dot

138
00:06:16.120 --> 00:06:19.040
<v Speaker 2>get trick on the generic URL of the results page,

139
00:06:19.160 --> 00:06:19.560
<v Speaker 2>you will.

140
00:06:19.399 --> 00:06:21.680
<v Speaker 1>Get nothing back, really just an error.

141
00:06:21.879 --> 00:06:25.360
<v Speaker 2>Basically, you can even check the response text in Python

142
00:06:25.399 --> 00:06:27.920
<v Speaker 2>and it'll confirm it's empty or just a blank form

143
00:06:28.319 --> 00:06:31.000
<v Speaker 2>because the server is looking at your request saying where's

144
00:06:31.000 --> 00:06:31.680
<v Speaker 2>the envelope?

145
00:06:31.879 --> 00:06:35.040
<v Speaker 1>That makes total sense. So to succeed, my Python code

146
00:06:35.079 --> 00:06:37.560
<v Speaker 1>essentially has to pretend to be a human being physically

147
00:06:37.560 --> 00:06:40.160
<v Speaker 1>filling out the drop downs and clicking the button exactly.

148
00:06:40.480 --> 00:06:42.480
<v Speaker 2>But how does Python know what to put in the

149
00:06:42.600 --> 00:06:45.560
<v Speaker 2>envelope if we can't see the URL changing?

150
00:06:45.920 --> 00:06:49.600
<v Speaker 1>Yeah, that's the million dollar question. How do we crack

151
00:06:49.759 --> 00:06:51.680
<v Speaker 1>the post form code?

152
00:06:51.800 --> 00:06:55.639
<v Speaker 2>What's fascinating here is we actually have to spy on ourselves.

153
00:06:55.839 --> 00:06:58.399
<v Speaker 1>Spy on ourselves, I love it. Tell me more.

154
00:06:58.560 --> 00:07:03.199
<v Speaker 2>We use a clever workaround leveraging Chrome's developer tools, specifically

155
00:07:03.279 --> 00:07:05.759
<v Speaker 2>the network tab. Who basically set up a wire tab

156
00:07:05.800 --> 00:07:08.639
<v Speaker 2>on our own browser to see the hidden request right

157
00:07:08.639 --> 00:07:09.399
<v Speaker 2>as it's being said.

158
00:07:09.439 --> 00:07:11.120
<v Speaker 1>Okay, so walk me through the step by step for

159
00:07:11.160 --> 00:07:11.439
<v Speaker 1>doing this.

160
00:07:11.560 --> 00:07:14.759
<v Speaker 2>Sure, you go to that blank DCA search form before

161
00:07:14.759 --> 00:07:16.920
<v Speaker 2>you touch anything. You go to the view tab in

162
00:07:16.959 --> 00:07:20.120
<v Speaker 2>your browser, down to developer and click on developer tools.

163
00:07:20.639 --> 00:07:22.480
<v Speaker 2>Then you click over to the network tab. It usually

164
00:07:22.480 --> 00:07:24.560
<v Speaker 2>just looks like a blank gray screen at first.

165
00:07:24.560 --> 00:07:25.759
<v Speaker 1>Just waiting for something to happen.

166
00:07:25.839 --> 00:07:28.560
<v Speaker 2>Exactly, leave it open, go back to the web page,

167
00:07:28.959 --> 00:07:31.399
<v Speaker 2>fill out the form just like we did John Kelly Dentist,

168
00:07:31.480 --> 00:07:35.240
<v Speaker 2>and hit submit. The moment you do, that network tab

169
00:07:35.279 --> 00:07:37.120
<v Speaker 2>is going to bustle with activity.

170
00:07:36.879 --> 00:07:39.680
<v Speaker 1>Because it's logging every single piece of data flying back

171
00:07:39.680 --> 00:07:40.120
<v Speaker 1>and forth.

172
00:07:40.279 --> 00:07:43.480
<v Speaker 2>Right, you'll see dozens of items load. You want to

173
00:07:43.800 --> 00:07:46.000
<v Speaker 2>scroll to the very top of that pain. So, like

174
00:07:46.040 --> 00:07:48.600
<v Speaker 2>the main document, request for the URL you're at, and

175
00:07:48.639 --> 00:07:51.040
<v Speaker 2>then look over in the section called headers.

176
00:07:50.879 --> 00:07:51.680
<v Speaker 1>The headers.

177
00:07:51.759 --> 00:07:54.040
<v Speaker 2>Okay, if you scroll down inside the headers, you'll eventually

178
00:07:54.040 --> 00:07:58.519
<v Speaker 2>find a section labeled form data or sometimes payload, and

179
00:07:58.639 --> 00:08:02.199
<v Speaker 2>that's the jack cut is the jackpot. That section literally

180
00:08:02.240 --> 00:08:06.079
<v Speaker 2>reveals the internal blueprint of the sealed envelope. It shows

181
00:08:06.079 --> 00:08:09.240
<v Speaker 2>you the exact variable names the database expects. So instead

182
00:08:09.240 --> 00:08:12.439
<v Speaker 2>of guessing, you see that the server wants like board

183
00:08:12.480 --> 00:08:15.079
<v Speaker 2>code equals twenty and first name equals John.

184
00:08:15.360 --> 00:08:17.720
<v Speaker 1>So now I just take that browsersleuthing and translate it

185
00:08:17.759 --> 00:08:20.120
<v Speaker 1>into Python. Yep, I take all that form data I

186
00:08:20.160 --> 00:08:22.959
<v Speaker 1>just found, and I convert it into a standard Python dictionary.

187
00:08:23.279 --> 00:08:25.879
<v Speaker 1>Just key value pairs wrapped in curly braces.

188
00:08:26.319 --> 00:08:28.959
<v Speaker 2>You've got it. It's just a slight modification to the

189
00:08:28.959 --> 00:08:30.000
<v Speaker 2>script we wrote earlier.

190
00:08:30.160 --> 00:08:32.679
<v Speaker 1>Right, So instead of request dot get, I'm guessing, I

191
00:08:32.799 --> 00:08:35.159
<v Speaker 1>use requests dot post Exactly.

192
00:08:35.320 --> 00:08:38.000
<v Speaker 2>You call request dot post. You pass in the target

193
00:08:38.240 --> 00:08:41.159
<v Speaker 2>url of the form but crucially, you also pass in

194
00:08:41.200 --> 00:08:43.519
<v Speaker 2>your new dictionary using the data parameter.

195
00:08:43.679 --> 00:08:47.919
<v Speaker 1>Oh so I'm literally handing the requests library, the destination address,

196
00:08:47.960 --> 00:08:49.399
<v Speaker 1>and the fully loaded envelope.

197
00:08:49.480 --> 00:08:52.559
<v Speaker 2>Yes, you say data your dictionary, and when you run that,

198
00:08:52.799 --> 00:08:55.240
<v Speaker 2>the server is tricked. It thinks you interacted with the

199
00:08:55.240 --> 00:08:58.600
<v Speaker 2>actual web page. It opens the envelope process of the data,

200
00:08:58.639 --> 00:09:01.399
<v Speaker 2>and your Python scripts zestfully prints out the names of

201
00:09:01.440 --> 00:09:03.039
<v Speaker 2>the eight dentists. Boom.

202
00:09:03.159 --> 00:09:04.799
<v Speaker 1>That is incredibly satisfying.

203
00:09:04.840 --> 00:09:06.960
<v Speaker 2>It's a huge breakthrough when you first get it to work.

204
00:09:07.639 --> 00:09:10.360
<v Speaker 1>So what does this all mean? Like, by mastering these

205
00:09:10.399 --> 00:09:13.519
<v Speaker 1>two techniques extracting query strings from the URLs for debt

206
00:09:13.559 --> 00:09:16.399
<v Speaker 1>t forms and spying on the network tab to send

207
00:09:16.480 --> 00:09:20.559
<v Speaker 1>data dictionaries for post forms, you are essentially empowered to

208
00:09:20.679 --> 00:09:23.240
<v Speaker 1>scrape the vast majority of websites on the Internet.

209
00:09:23.320 --> 00:09:25.840
<v Speaker 2>Absolutely, you're no longer just looking at a static page.

210
00:09:25.840 --> 00:09:28.480
<v Speaker 2>You're actually talking to the database. But you know, we

211
00:09:28.480 --> 00:09:30.720
<v Speaker 2>should probably mention that servers do fight back.

212
00:09:31.000 --> 00:09:33.639
<v Speaker 1>Oh, they definitely do. It's a whole cat and mouse game, right.

213
00:09:33.720 --> 00:09:37.080
<v Speaker 2>Like they use things called CSRF tokens cross site request

214
00:09:37.159 --> 00:09:42.279
<v Speaker 2>forgery tokens, basically a hidden randomized fingerprint the server puts

215
00:09:42.279 --> 00:09:44.399
<v Speaker 2>on the form when a real human loads it.

216
00:09:44.480 --> 00:09:48.000
<v Speaker 1>So if my Python script just blasts the post request

217
00:09:48.080 --> 00:09:49.879
<v Speaker 1>without visiting the page first.

218
00:09:49.679 --> 00:09:53.159
<v Speaker 2>It gets rejected because it's missing the fingerprint. You'd have

219
00:09:53.200 --> 00:09:56.120
<v Speaker 2>to write a script that does a get tea request

220
00:09:56.240 --> 00:09:59.200
<v Speaker 2>first to steal the token, inject it into your dictionary,

221
00:09:59.240 --> 00:10:00.759
<v Speaker 2>and then send the po request.

222
00:10:00.879 --> 00:10:02.320
<v Speaker 1>Wow, so it's a two step dance.

223
00:10:02.480 --> 00:10:04.960
<v Speaker 2>It really is an arms race. Yeah, but that's the

224
00:10:04.960 --> 00:10:08.480
<v Speaker 2>big picture here. Understanding how data moves between a client

225
00:10:08.480 --> 00:10:11.559
<v Speaker 2>and a server, whether it's transparent or hidden. That's the

226
00:10:11.639 --> 00:10:14.840
<v Speaker 2>absolute foundation of both webscraping and cybersecurity.

227
00:10:15.039 --> 00:10:17.799
<v Speaker 1>It totally demystifies the web. You start looking at every

228
00:10:17.840 --> 00:10:20.600
<v Speaker 1>search bar differently, which actually brings me to a quick

229
00:10:20.639 --> 00:10:22.559
<v Speaker 1>review exercise for you as you're listening today.

230
00:10:22.559 --> 00:10:23.519
<v Speaker 2>Oh this is a great idea.

231
00:10:23.720 --> 00:10:27.279
<v Speaker 1>Yeah, next time you are casually browsing a website, maybe

232
00:10:27.360 --> 00:10:29.799
<v Speaker 1>shopping or using a dropdown menu, I want you to

233
00:10:29.840 --> 00:10:32.039
<v Speaker 1>keep an eye on the URL. Does it change to

234
00:10:32.080 --> 00:10:35.279
<v Speaker 1>show your search terms or does it stay exactly the same.

235
00:10:35.600 --> 00:10:37.240
<v Speaker 2>Try to spot if it's a get tea or.

236
00:10:37.240 --> 00:10:40.399
<v Speaker 1>A post exactly, and if it stays the same, try

237
00:10:40.440 --> 00:10:43.519
<v Speaker 1>opening your browsers developer tools head over to the network tab,

238
00:10:43.600 --> 00:10:45.519
<v Speaker 1>or run the search again and see if you can

239
00:10:45.519 --> 00:10:48.600
<v Speaker 1>spot the hidden form data being sent inside the payload.

240
00:10:48.639 --> 00:10:50.799
<v Speaker 1>It's so cool once you see it with your own eyes.

241
00:10:50.919 --> 00:10:53.080
<v Speaker 2>It really builds that critical technical intuition.

242
00:10:53.240 --> 00:10:56.240
<v Speaker 1>It does, but it also leads me with this lingering

243
00:10:56.360 --> 00:11:00.559
<v Speaker 1>question to ponder, oh, what's that Well, if we can

244
00:11:00.679 --> 00:11:04.720
<v Speaker 1>easily write Python scripts that perfectly mimic human behavior like

245
00:11:05.519 --> 00:11:08.679
<v Speaker 1>filling out these hidden povis T forms and tricking servers

246
00:11:08.720 --> 00:11:12.639
<v Speaker 1>into handing over their data, how can website creators truly

247
00:11:12.720 --> 00:11:16.120
<v Speaker 1>tell the difference between a curious learner and an invisible

248
00:11:16.240 --> 00:11:17.600
<v Speaker 1>army of automated bots.

249
00:11:18.000 --> 00:11:21.200
<v Speaker 2>That is the billion dollar question in tech right now, right, and.

250
00:11:21.200 --> 00:11:23.440
<v Speaker 1>What does that mean for the future of navigating the web?

251
00:11:23.960 --> 00:11:26.960
<v Speaker 1>If the line between user and script keeps getting thinner,

252
00:11:27.519 --> 00:11:30.000
<v Speaker 1>we'll think about the invisible logic guarding the gates the

253
00:11:30.039 --> 00:11:31.159
<v Speaker 1>next time you hit search