WEBVTT 1 00:00:00.120 --> 00:00:03.560 Imagine for a second that you are staring at a 2 00:00:03.560 --> 00:00:07.559 web page, okay, and it is just packed with thousands 3 00:00:07.559 --> 00:00:10.519 of data points. Like we are talking, pages upon pages 4 00:00:10.560 --> 00:00:14.960 of you know, randomly generated addresses, phone numbers, product reviews, whatever. 5 00:00:15.080 --> 00:00:16.440 Oh yeah, I know that feeling. 6 00:00:16.239 --> 00:00:18.839 Right, You get that sinking feeling in your stomach, that 7 00:00:19.000 --> 00:00:21.519 creeping dread when you realize you need all of that 8 00:00:21.640 --> 00:00:25.039 data neatly organized in a spreadsheet and the only way 9 00:00:25.079 --> 00:00:27.480 you know how to do it is by uh spending 10 00:00:27.519 --> 00:00:33.759 the next four maybe five hours mindlessly highlighting, copying, and pasting. 11 00:00:33.960 --> 00:00:36.880 Yeah, it's the ultimate digital bottleneck. I mean, your brain 12 00:00:36.960 --> 00:00:40.200 is totally ready to analyze the information and find patterns, 13 00:00:40.399 --> 00:00:44.200 but your hands are just stuck doing this manual, repetitive 14 00:00:44.240 --> 00:00:46.799 labor from like the nineteen nineties exactly. 15 00:00:46.840 --> 00:00:49.119 It completely breaks your momentum. But you know what if 16 00:00:49.119 --> 00:00:50.799 you didn't have to do that, What if you could 17 00:00:50.799 --> 00:00:53.920 write just a few lines of code and instantly download, extract, 18 00:00:53.960 --> 00:00:56.679 and organize every single piece of that data in a 19 00:00:56.719 --> 00:00:59.960 matter of seconds. So today we're taking a deep dive 20 00:01:00.200 --> 00:01:04.359 into the superpower of webscraping and data extraction using Python. 21 00:01:04.480 --> 00:01:05.599 It really is a superpower. 22 00:01:05.640 --> 00:01:08.480 Honestly, it really is, and we're going to map out 23 00:01:08.519 --> 00:01:12.959 a very specific journey for you today. First, we'll figure 24 00:01:12.959 --> 00:01:15.079 out how to get a web page to actually talk 25 00:01:15.159 --> 00:01:19.599 to our Python environment without blocking us. Second, we're going 26 00:01:19.640 --> 00:01:22.760 to explore how to search through all that chaotic raw 27 00:01:22.840 --> 00:01:27.480 code using some incredibly powerful pattern matching tools. And finally, 28 00:01:27.519 --> 00:01:30.680 we will look at how to elegantly carve up the 29 00:01:30.719 --> 00:01:32.840 structural matrix of the web so you can pull out 30 00:01:32.879 --> 00:01:33.840 exactly what you need. 31 00:01:34.200 --> 00:01:36.680 It's a massive shift in how you interact with the Internet. 32 00:01:36.840 --> 00:01:40.599 You basically move from being a passive consumer manually browsing 33 00:01:40.640 --> 00:01:44.799 pages to becoming an active curator who automates the ingestion 34 00:01:44.879 --> 00:01:45.439 of knowledge. 35 00:01:45.480 --> 00:01:47.959 I love that active curator. So let's start at the 36 00:01:48.040 --> 00:01:50.439 very beginning. Before we can parse any data, we actually 37 00:01:50.439 --> 00:01:53.079 have to get the raw web page into our Python environment. 38 00:01:53.319 --> 00:01:56.280 I always picture this as like sending a highly efficient 39 00:01:56.400 --> 00:01:59.239 robot butler to a store to fetch a specific document 40 00:01:59.319 --> 00:02:02.400 for you. Great analogy, thanks, But how do we actually 41 00:02:03.239 --> 00:02:04.159 build that robot. 42 00:02:04.280 --> 00:02:07.000 Well, in the language of the web, setting that robot 43 00:02:07.040 --> 00:02:11.879 butler is called making an HTTP get request. You're pointing 44 00:02:11.879 --> 00:02:15.479 your code at a target URL and asking or getting 45 00:02:15.479 --> 00:02:19.560 the information from that server, right, and within the Python ecosystem, 46 00:02:19.599 --> 00:02:22.759 there are really three primary libraries you'll encounter for this job. 47 00:02:23.280 --> 00:02:25.080 Let's walk through those because I know they each have 48 00:02:25.120 --> 00:02:26.360 a very different philosophy. 49 00:02:26.520 --> 00:02:30.919 They definitely do. So first you have httplib two. This 50 00:02:31.039 --> 00:02:34.719 is actually a really comprehensive, powerful upgrade that was originally 51 00:02:34.759 --> 00:02:36.240 driven by developers at Google. 52 00:02:36.360 --> 00:02:37.759 Oh wow, okay, yeah, and it. 53 00:02:37.719 --> 00:02:41.280 Gives you very fine grained, granular control over your requests. 54 00:02:41.719 --> 00:02:44.319 But you know that power comes with a cost. It 55 00:02:44.360 --> 00:02:47.479 requires you to manually manage a lot of the underlying connection. 56 00:02:47.240 --> 00:02:49.719 Details, which sounds like a headache for a beginner. 57 00:02:49.879 --> 00:02:52.360 It can be. Then you have earlib, which. 58 00:02:52.159 --> 00:02:54.719 I understand is incredibly common just because it comes built 59 00:02:54.719 --> 00:02:57.199 in with standard Python right like, you don't have to 60 00:02:57.240 --> 00:02:59.680 install a third party package to use it exactly. 61 00:02:59.719 --> 00:03:04.840 That is its main appeal. Historically, older versions of Python 62 00:03:04.879 --> 00:03:09.639 actually had two separate, somewhat overlapping libraries, arlib and earlib two. 63 00:03:09.800 --> 00:03:11.039 Oh that sounds confusing. 64 00:03:11.120 --> 00:03:14.039 Its big confusing for developers, so they eventually cleaned house 65 00:03:14.199 --> 00:03:18.240 and merge them into a single unified arlib module. It 66 00:03:18.319 --> 00:03:21.479 sits at a higher level than httplib two, so it 67 00:03:21.520 --> 00:03:24.639 handles some of those low level connection headaches for you, okay, 68 00:03:24.680 --> 00:03:29.080 that's better, But honestly, for most modern developers, the undisputed 69 00:03:29.120 --> 00:03:33.879 favorite is a library simply called Requests just requests. 70 00:03:33.960 --> 00:03:35.879 The name itself kind of tells you it's trying to 71 00:03:35.960 --> 00:03:36.759 be user friendly. 72 00:03:36.919 --> 00:03:41.120 It is beautifully designed. The Request library provides a high level, 73 00:03:41.319 --> 00:03:46.000 incredibly intuitive interface. It abstracts away all the tedious, complex 74 00:03:46.000 --> 00:03:46.840 connection logic. 75 00:03:47.000 --> 00:03:47.319 Nice. 76 00:03:47.439 --> 00:03:50.360 If your primary goal is just to grab data quickly 77 00:03:50.360 --> 00:03:54.080 and cleanly without fighting with connection protocols, Requests is the 78 00:03:54.080 --> 00:03:54.879 tool you reach for. 79 00:03:55.280 --> 00:03:57.919 Okay, So we pick our library, say requests, and we 80 00:03:57.960 --> 00:04:00.759 send our automated requests out to a website. When the 81 00:04:00.800 --> 00:04:03.719 server responds, it doesn't just hand over a clean, readable 82 00:04:03.719 --> 00:04:07.479 text document. The robot comes back with a response object. 83 00:04:07.680 --> 00:04:09.560 What is actually inside that package? 84 00:04:09.680 --> 00:04:12.520 So the response object fundamentally breaks down into two parts, 85 00:04:13.039 --> 00:04:16.920 the headers and the content. The headers are basically the 86 00:04:16.959 --> 00:04:18.959 metadata of the transaction. 87 00:04:18.639 --> 00:04:19.480 Like a shipping label. 88 00:04:19.600 --> 00:04:22.519 Exactly like a shipping label, they tell you the status code, 89 00:04:22.920 --> 00:04:25.399 whether the server actually sound the page or if it crashed, 90 00:04:25.680 --> 00:04:29.199 what kind of server software is running, and various caching rules. 91 00:04:29.360 --> 00:04:32.560 I noticed something interesting in the notes about those headers 92 00:04:32.600 --> 00:04:36.199 regarding security. If you accidentally point your code to an old, 93 00:04:36.480 --> 00:04:41.519 unsecure HTTP address, the server often fires back an HTTP 94 00:04:41.600 --> 00:04:43.279 three oh one redirect status. 95 00:04:43.480 --> 00:04:44.639 Yes, that happens a lot. 96 00:04:44.720 --> 00:04:47.720 It basically intercepts your robot and automatically reroutes it to 97 00:04:47.800 --> 00:04:50.160 the secure HTTPS version of the site. 98 00:04:50.319 --> 00:04:54.439 Modern servers are heavily configured to prioritize encrypted traffic. They'll 99 00:04:54.480 --> 00:04:57.839 seamlessly bounce your automated request to the secure lane, just 100 00:04:57.879 --> 00:04:59.879 like they would for a human typing an address into 101 00:04:59.879 --> 00:05:00.519 a browser. 102 00:05:00.639 --> 00:05:01.240 That makes sense. 103 00:05:01.399 --> 00:05:04.399 Now, underneath those metadata headers is the actual content you 104 00:05:04.439 --> 00:05:06.920 ask for. But as you hinted earlier, it does not 105 00:05:07.079 --> 00:05:11.519 arrive as readable text. It arises raw bytes machine language. 106 00:05:11.079 --> 00:05:13.480 Which looks like absolute garbage to a human eye. If 107 00:05:13.519 --> 00:05:14.279 you try to print it. 108 00:05:14.199 --> 00:05:17.319 Out total garbage. To make it readable, you have to 109 00:05:17.399 --> 00:05:20.639 decode that byte stream. The vast majority of the web 110 00:05:20.680 --> 00:05:23.560 is encoded using a standard called UTF eight. Okay, so 111 00:05:23.639 --> 00:05:26.360 within your Python script, you take those raw bytes, apply 112 00:05:26.439 --> 00:05:30.160 a UTF eight decoding method, and suddenly that machine code 113 00:05:30.199 --> 00:05:34.160 blossoms into the giant, messy HTML string that actually builds 114 00:05:34.199 --> 00:05:34.800 the web page. 115 00:05:34.920 --> 00:05:36.720 I want to pause here for a second, because there 116 00:05:36.800 --> 00:05:41.040 is a glaring real world problem with this robot butler analogy. 117 00:05:41.199 --> 00:05:41.959 Oh what's that? 118 00:05:42.439 --> 00:05:45.800 Well, what if the store doesn't like robots? Like if 119 00:05:45.839 --> 00:05:47.519 I write a script right now and send it to 120 00:05:47.560 --> 00:05:50.000 scrape search results from Google, Google isn't just going to 121 00:05:50.079 --> 00:05:51.680 hand over it to data with a smile. 122 00:05:51.480 --> 00:05:54.040 Right, Oh? Absolutely not. If you try that using the 123 00:05:54.079 --> 00:05:57.800 default settings of erlib or requests, you will immediately hit 124 00:05:57.800 --> 00:06:01.040 a brick wall. Specifically, the server will turn an HTTP 125 00:06:01.120 --> 00:06:02.560 air four h three forbidden. 126 00:06:02.839 --> 00:06:03.800 Yikes, forbidden. 127 00:06:03.879 --> 00:06:07.680 Yeah. Major tech companies actively monitor incoming traffic. When they 128 00:06:07.680 --> 00:06:10.360 see a request, they check the headers, and by default, 129 00:06:10.560 --> 00:06:13.079 your Python library includes a header called the user agent 130 00:06:13.319 --> 00:06:16.240 and essentially wears a giant name pag that says hello, 131 00:06:16.560 --> 00:06:18.160 I'm an automated Python script. 132 00:06:18.279 --> 00:06:21.160 And companies block that because they don't want armies of 133 00:06:21.240 --> 00:06:25.439 bots chewing up their server bandwidth or mass downloading their 134 00:06:25.439 --> 00:06:26.720 proprietary data. 135 00:06:26.759 --> 00:06:27.120 Exactly. 136 00:06:27.279 --> 00:06:28.839 So, how do we get past the bouncer? 137 00:06:29.360 --> 00:06:31.600 You have to change your outfit. Yeah, you modify the 138 00:06:31.680 --> 00:06:34.399 user agent header in your code before you send the request. 139 00:06:34.839 --> 00:06:38.800 Sneaky very. Instead of broadcasting that you're a Python script, 140 00:06:39.079 --> 00:06:42.079 you spoof the identity of a standard web browser. You 141 00:06:42.120 --> 00:06:44.600 pass a string that tells the server, Hey, I'm not 142 00:06:44.680 --> 00:06:47.199 a bot. I'm just a normal user running the Safari 143 00:06:47.240 --> 00:06:48.319 browser on a MacBook. 144 00:06:48.439 --> 00:06:49.560 And that actually works. 145 00:06:49.759 --> 00:06:53.160 It does. Once the server reads that fabricated header, it 146 00:06:53.319 --> 00:06:55.519 drops the velvet rope and lets you in. Wow. 147 00:06:55.639 --> 00:06:59.160 Okay, so we spoof our identity, the bouncer lets us through, 148 00:06:59.399 --> 00:07:02.480 and our script downloads the page. Now we're staring at 149 00:07:02.519 --> 00:07:06.959 a massive, thousands of lines long wall of raw HTML text. 150 00:07:07.399 --> 00:07:10.800 Standard searching methods like hitting ctrlf on your keyboard to 151 00:07:10.800 --> 00:07:13.000 find a word are completely useless here. 152 00:07:13.079 --> 00:07:13.800 Yeah, completely. 153 00:07:13.920 --> 00:07:16.720 If we want to extract thousands of dynamic data points, 154 00:07:16.720 --> 00:07:18.399 we need something much more robust. 155 00:07:18.519 --> 00:07:23.399 We need regular expressions, or as it's often abbreviated, redjax. 156 00:07:23.800 --> 00:07:26.079 You access this in Python through the built in a 157 00:07:26.120 --> 00:07:30.040 EARS module. Rojex isn't just a search tool. It is 158 00:07:30.079 --> 00:07:35.199 an entire micro language dedicated to advanced pattern matching within 159 00:07:35.279 --> 00:07:36.040 strings of text. 160 00:07:36.439 --> 00:07:38.560 I want to break down how we actually deploy this, 161 00:07:38.600 --> 00:07:41.240 because there are several different functions inside that module depending 162 00:07:41.240 --> 00:07:44.000 on your strategy. Sure, let's look at the match function first. 163 00:07:44.560 --> 00:07:49.079 I picture match as like walking into a train station 164 00:07:49.120 --> 00:07:51.399 to see if a specific train is sitting exactly the 165 00:07:51.560 --> 00:07:53.199 very first platform the moment you. 166 00:07:53.279 --> 00:07:57.199 Arrive that analogy holds up perfectly. The match function is 167 00:07:57.240 --> 00:08:00.839 incredibly strict. It only checks for your pattern at the 168 00:08:00.959 --> 00:08:05.160 absolute literal beginning of a string position zero. If the 169 00:08:05.279 --> 00:08:08.120 data you want is buried even one space deep into 170 00:08:08.160 --> 00:08:10.720 the text, match will return absolutely nothing. 171 00:08:10.759 --> 00:08:11.480 Wow. Okay. 172 00:08:11.600 --> 00:08:14.439 There's also full match, which goes a step further. It 173 00:08:14.480 --> 00:08:17.399 demands that the entire string, from the very first character 174 00:08:17.600 --> 00:08:20.439 to the absolute last matches your pattern perfectly. 175 00:08:20.800 --> 00:08:23.480 But in the context of web scraping, I rarely care 176 00:08:23.480 --> 00:08:25.360 if the data is sitting perfectly at the start of 177 00:08:25.399 --> 00:08:27.360 the document. I just want to find it wherever it's 178 00:08:27.480 --> 00:08:30.199 hiding in the chaos. You know. So, if match is 179 00:08:30.319 --> 00:08:34.240 checking the starting platform, the search function is like flying 180 00:08:34.240 --> 00:08:36.960 a helicopter down the entire length of the train tracks. 181 00:08:37.200 --> 00:08:41.159 Yes, the search function scans through the entire string and 182 00:08:41.320 --> 00:08:44.679 stops the exact moment it finds the very first instance 183 00:08:44.720 --> 00:08:48.080 of your pattern. It finds that first train reports its 184 00:08:48.120 --> 00:08:50.240 location and stops looking. 185 00:08:50.440 --> 00:08:52.480 Stops looking entirely entirely. 186 00:08:52.799 --> 00:08:55.720 But in data extraction, stopping at the first hit kind 187 00:08:55.720 --> 00:08:59.279 of defeats the purpose. If you are scraping a directory 188 00:08:59.279 --> 00:09:01.919 for phone numbers, you want every single number on. 189 00:09:01.879 --> 00:09:03.759 The page, right, not just the first one. 190 00:09:03.799 --> 00:09:04.960 For that, you use findal. 191 00:09:05.039 --> 00:09:07.519 Okay, find ol does exactly what it says on the ten. 192 00:09:08.039 --> 00:09:10.840 It sweeps the entire document and hands you back a clean, 193 00:09:11.120 --> 00:09:13.600 comprehensive list of every single matchup. 194 00:09:13.600 --> 00:09:15.840 Fat It's the most common one you'll use, for sure. 195 00:09:16.159 --> 00:09:18.159 I also saw a function called find it or, which 196 00:09:18.159 --> 00:09:20.720 sounds like it's doing something similar but with more complexity. 197 00:09:21.000 --> 00:09:22.919 So find it or is the tool you use when 198 00:09:22.960 --> 00:09:26.360 you need spatial awareness. Findal just gives you the extracted words, 199 00:09:26.639 --> 00:09:29.000 but find it or returns an iterator that provides the 200 00:09:29.080 --> 00:09:32.720 exact character index ranges for every match. Oh interesting, Yeah, 201 00:09:32.759 --> 00:09:35.519 it tells you I found your target data and it 202 00:09:35.600 --> 00:09:39.240 exists precisely from character position one fifty to one sixty two. 203 00:09:39.799 --> 00:09:42.879 It acts like a GPS coordinate system for your data. 204 00:09:42.960 --> 00:09:46.080 That is super useful, and beyond just locating data, you 205 00:09:46.120 --> 00:09:49.039 can manipulate it too. You can use the split function 206 00:09:49.399 --> 00:09:53.399 to chop a massive paragraph into individual tokens based on spaces, 207 00:09:53.799 --> 00:09:57.600 or the subfunction to rapidly substitute or replace specific patterns 208 00:09:57.639 --> 00:09:59.000 across the entire document. 209 00:09:59.159 --> 00:10:01.600 Absolutely, so, here's the real hurdle. 210 00:10:01.639 --> 00:10:04.919 To truly utilize regular expressions. We have to stop searching 211 00:10:04.960 --> 00:10:08.080 for literal words like apple or banana. We need to 212 00:10:08.120 --> 00:10:11.600 search for abstract concepts. We need to find any decimal 213 00:10:11.679 --> 00:10:13.679 number or any capitalized word. 214 00:10:13.799 --> 00:10:16.159 And this is where we introduce special sequences. But a 215 00:10:16.240 --> 00:10:19.480 quick best practice note first, before you write these sequences, 216 00:10:19.639 --> 00:10:22.480 you should prefix your pattern string with a lowercase R. 217 00:10:22.960 --> 00:10:24.679 This creates what's called a raw string. 218 00:10:24.840 --> 00:10:26.240 Wait, why is that necessary? 219 00:10:26.399 --> 00:10:31.120 Well, by default, Python uses backslashes as escape characters for 220 00:10:31.200 --> 00:10:35.240 things like creating new lines. Because Rejects relies heavily on 221 00:10:35.320 --> 00:10:38.919 backslashes for its own syntax, Using a raw string tells 222 00:10:38.960 --> 00:10:42.399 Python to ignore its default rules and treat every backslash 223 00:10:42.440 --> 00:10:45.279 as a literal character for the Rejects engine to interpret. 224 00:10:45.399 --> 00:10:47.840 Oh I see, yeah, it saves you from writing incredibly 225 00:10:47.919 --> 00:10:52.759 messy tangled code. Additionally, you will often wrap your raw 226 00:10:52.799 --> 00:10:56.720 string in the redot compile method. This translates your pattern 227 00:10:56.759 --> 00:11:00.679 into a highly efficient, reusable blueprint, which fads things up 228 00:11:00.759 --> 00:11:02.799 dramatically if you're applying it thousands of times. 229 00:11:02.879 --> 00:11:05.639 Okay, let's talk about these special sequences because they are 230 00:11:05.679 --> 00:11:07.919 the building blocks of that blueprint. But I'll be honest. 231 00:11:07.919 --> 00:11:09.279 When you look at them on paper, they look like 232 00:11:09.320 --> 00:11:10.120 a secret code. 233 00:11:10.679 --> 00:11:13.000 They do require a slight mental chef. Let's look at 234 00:11:13.039 --> 00:11:15.879 those common ones. A backslash followed by a lowercase D. 235 00:11:16.200 --> 00:11:20.279 So backslash D represents any single digit zero through nine. 236 00:11:20.360 --> 00:11:22.919 Got it, And if you capitalize the letter, it flips 237 00:11:22.919 --> 00:11:25.879 the meaning. Right. Yeah, So a backslash and an uppercase 238 00:11:25.960 --> 00:11:29.120 D matches any character that is not a digit exactly. 239 00:11:29.639 --> 00:11:33.960 That capitalization rule applies across the board. A backslash lowercase 240 00:11:34.159 --> 00:11:38.320 S hunts for white space, spaces, tabs, line breaks, A 241 00:11:38.360 --> 00:11:41.879 backslash uppercase S finds anything that is into space. 242 00:11:42.240 --> 00:11:43.399 Okay, tracking so far. 243 00:11:43.679 --> 00:11:47.279 Then there's backslash lowercase W, which is your workhourse for 244 00:11:47.360 --> 00:11:52.120 alphanumeric characters. It matches letters, numbers, and underscores. Capitalize it 245 00:11:52.120 --> 00:11:55.960 to backslash W, and it matches non alphanumeric characters like 246 00:11:56.000 --> 00:11:56.919 punctuation marks. 247 00:11:57.159 --> 00:11:59.960 You can also define your own custom ranges using square back. 248 00:12:00.080 --> 00:12:01.879 That's right. Like if I only want to find numbers 249 00:12:01.919 --> 00:12:04.799 between five and nine, I just write bracket five dash nine, 250 00:12:04.840 --> 00:12:07.759 pracket yeap. It tells the engine only match characters that 251 00:12:07.799 --> 00:12:09.759 fall inside this specific parameter. 252 00:12:09.840 --> 00:12:12.200 You've got it. You also need a way to tell 253 00:12:12.200 --> 00:12:15.039 the engine how many times a character should occur. We 254 00:12:15.159 --> 00:12:19.000 use multipliers for this. An asterisk means the preceding character 255 00:12:19.080 --> 00:12:22.440 can appear zero or more times. A plus sign means 256 00:12:22.440 --> 00:12:24.240 it must appear one or more times. 257 00:12:24.360 --> 00:12:26.799 Okay, I understand the individual puzzle pieces, but when you 258 00:12:26.799 --> 00:12:30.120 string them together, the syntax becomes incredibly intimidating. 259 00:12:30.279 --> 00:12:32.240 It can look a bit overwhelming at first glance. 260 00:12:32.519 --> 00:12:34.639 Let's look at a real world example from a script. 261 00:12:35.000 --> 00:12:38.360 I see a pattern that reads backslash D curly brace 262 00:12:38.440 --> 00:12:41.960 three dash, backslash d curly brace three dash backslash d 263 00:12:42.080 --> 00:12:45.320 curly brace four. That looks like someone dropped a coffee 264 00:12:45.399 --> 00:12:47.679 mug on their keyboard. How do you read that fluently? 265 00:12:48.080 --> 00:12:51.200 It looks chaotic until you apply the rules we just discussed. 266 00:12:51.720 --> 00:12:55.399 Let's translate it piece by piece. The backslash d means 267 00:12:55.480 --> 00:12:58.600 we're looking for a digit. The curly braces the three 268 00:12:58.639 --> 00:13:02.799 inside specify an exact multiplier. We don't want zero or more, 269 00:13:02.879 --> 00:13:06.159 we want exactly three. So backslash d curly brace three 270 00:13:06.320 --> 00:13:08.799 translates to find exactly three numbers in a row. 271 00:13:08.919 --> 00:13:11.240 Okay, Then there is a literal hyphen. 272 00:13:11.039 --> 00:13:13.960 Then another backslash curly brace three, so three more numbers, 273 00:13:13.960 --> 00:13:17.759 another hyphen and finally backslash curly brase four meaning exactly four. 274 00:13:17.679 --> 00:13:21.879 Numbers wait three digits, dash three digits, dash four digits. 275 00:13:21.960 --> 00:13:24.080 When you look at it that way, the gibberish becomes 276 00:13:24.120 --> 00:13:28.279 a surgically precise blueprint for a standard US phone number exactly. 277 00:13:28.440 --> 00:13:31.480 You have just created a custom extraction tool. You can 278 00:13:31.519 --> 00:13:34.759 pass that exact tiny string of characters into a final function, 279 00:13:35.399 --> 00:13:39.440 unleash it on a ten thousand line HTML document, and 280 00:13:39.519 --> 00:13:42.440 it will effortlessly sift through the chaos and hand you 281 00:13:42.480 --> 00:13:45.279 a perfect list of every single phone number on the page. 282 00:13:45.519 --> 00:13:48.799 That is wild. Okay, let's take this new found power 283 00:13:48.799 --> 00:13:51.559 and apply it to the HTML we fetched earlier. Web 284 00:13:51.600 --> 00:13:54.639 pages are built using tags, so the title of a 285 00:13:54.679 --> 00:13:56.480 page is wrapped in title tags. 286 00:13:56.559 --> 00:13:59.639 Yes, and if you want to extract that title using rejects, 287 00:14:00.039 --> 00:14:02.480 your first instinct might be to write a pattern like 288 00:14:02.519 --> 00:14:06.399 title tag, dot asterisk closing title tag. The dot represents 289 00:14:06.440 --> 00:14:10.279 any character, and the asterisk means repeated zero or more times. Right, 290 00:14:10.480 --> 00:14:12.799 it will grab the opening tag, everything in the middle, 291 00:14:13.039 --> 00:14:13.879 and the closing tag. 292 00:14:13.919 --> 00:14:15.600 But the problem with that is you don't actually want 293 00:14:15.600 --> 00:14:18.799 the literal HTML tags in your final spreadsheet, you only 294 00:14:18.879 --> 00:14:22.480 want the human readable text sandwiched between them spot on. 295 00:14:23.080 --> 00:14:26.559 To isolate that text, you introduce capturing groups, which are 296 00:14:26.559 --> 00:14:29.759 created using parentheses. You update your pattern to put parentheses 297 00:14:29.759 --> 00:14:32.840 around the dot asterisk A. The rejex engine sees those 298 00:14:32.840 --> 00:14:36.600 parentheses and understands the instruction. I need you to locate 299 00:14:36.639 --> 00:14:39.600 this entire structure to ensure we are in the right place, 300 00:14:39.639 --> 00:14:42.600 but I only want you to extract and return the 301 00:14:42.639 --> 00:14:45.440 specific data trapped inside the parentheses. 302 00:14:45.679 --> 00:14:49.080 That's brilliant. Let's scale that up. Imagine we are scraping 303 00:14:49.159 --> 00:14:52.440 a directory of employees. The web page lists them using 304 00:14:52.600 --> 00:14:56.039 HTML bullet points, which are designated by lie tags. We 305 00:14:56.080 --> 00:14:58.240 want to extract the names of everyone whose first name 306 00:14:58.279 --> 00:14:59.120 starts with the letter C. 307 00:15:00.120 --> 00:15:03.440 Construct a pattern that looks for the lie tag followed 308 00:15:03.440 --> 00:15:06.720 by a capital C followed by a backslash W plus 309 00:15:06.799 --> 00:15:09.320 to capture the rest of the alphanumeric characters in the name. 310 00:15:10.039 --> 00:15:12.759 You wrap that entire name segment in parentheses, and the 311 00:15:12.799 --> 00:15:15.000 engine will spit out a clean list of first names. 312 00:15:15.240 --> 00:15:17.559 And you can even set up multiple capturing groups in 313 00:15:17.600 --> 00:15:21.159 a single pattern to extract complex data sets. Right like, 314 00:15:21.200 --> 00:15:23.399 you could create one group for a city, another group 315 00:15:23.399 --> 00:15:25.840 for the two letter state code, and a final group 316 00:15:25.840 --> 00:15:27.039 for the five digit zip code. 317 00:15:27.120 --> 00:15:30.440 Exactly. The rejects engine will return those matches as neatly 318 00:15:30.519 --> 00:15:34.559 organized data tipples. You are pulling highly structured, relational data 319 00:15:34.600 --> 00:15:35.879 straight out of raw code. 320 00:15:36.120 --> 00:15:39.279 But as we walk through these HTML examples, a massive 321 00:15:39.360 --> 00:15:44.080 vulnerability becomes obvious to me. Look that HTML is notoriously messy. 322 00:15:44.600 --> 00:15:48.519 Web developers are human. They make mistakes, they update site layouts, 323 00:15:48.519 --> 00:15:52.000 they do weird things. What if a developer decides a 324 00:15:52.039 --> 00:15:54.919 specific employee's name needs to stand out, so they inject 325 00:15:54.960 --> 00:15:57.320 an HTML bold tag right in the middle of the 326 00:15:57.320 --> 00:15:59.919 first name. Oh yeah, Or what if they accidentally leave 327 00:15:59.919 --> 00:16:03.279 a line break where there shouldn't be one. Rajex doesn't 328 00:16:03.320 --> 00:16:07.720 actually understand HTML. It just blindly reads raw characters from 329 00:16:07.759 --> 00:16:08.399 left to right. 330 00:16:08.559 --> 00:16:12.799 That is the fundamental limitation of using regular expressions as 331 00:16:12.840 --> 00:16:16.840 your primary webscribing tool. It completely lacks structural awareness. 332 00:16:16.960 --> 00:16:20.240 It feels like trying to navigate a sprawling, multi level 333 00:16:20.279 --> 00:16:24.759 city while totally blindfolded, relying exclusively on a compass like 334 00:16:24.759 --> 00:16:27.000 the compass might tell you that you are heading directly 335 00:16:27.080 --> 00:16:29.840 north toward your target data, but it has no idea 336 00:16:29.840 --> 00:16:31.399 that there is a brick wall in front of you, 337 00:16:31.639 --> 00:16:33.480 or that the data is actually on the third floor 338 00:16:33.519 --> 00:16:34.039 of a building. 339 00:16:34.159 --> 00:16:37.879 The compass analogy is perfect. Rejects is rigid and literal. 340 00:16:38.840 --> 00:16:41.559 If the underlying structure of the HTML shifts even slightly, 341 00:16:41.879 --> 00:16:46.080 your carefully crafted rejects pattern instantly shatters. Right. If rejects 342 00:16:46.159 --> 00:16:48.960 is a blind compass, what we actually need is a detailed, 343 00:16:48.960 --> 00:16:51.440 three dimensional map of the city. We need a tool 344 00:16:51.480 --> 00:16:54.480 that understands the architecture, and in Python that map is 345 00:16:54.480 --> 00:16:55.879 a library called Beautiful Soup. 346 00:16:55.960 --> 00:16:58.960 Beautiful Soup or BS four for short, HUG. I want 347 00:16:58.960 --> 00:17:01.879 to understand how it actually creates this map, because the 348 00:17:02.039 --> 00:17:05.160 HTML we feed it is still just a chaotic string 349 00:17:05.200 --> 00:17:05.720 of text. 350 00:17:05.960 --> 00:17:08.480 Well, beautiful Soup does not work alone. It relies on 351 00:17:08.559 --> 00:17:12.480 underlying parsing engines, most commonly LexML or HTML five lib. 352 00:17:12.720 --> 00:17:15.559 Think of these parsers as the heavy machinery. Okay, they 353 00:17:15.759 --> 00:17:18.960 ingest the messy, broken HTML string, figure out how all 354 00:17:19.000 --> 00:17:21.160 the open and closed tags relate to each other, and 355 00:17:21.200 --> 00:17:24.359 transform it into a highly structured part tree that Python 356 00:17:24.400 --> 00:17:27.440 can interact with. Beautiful Soup is just the elegant interface 357 00:17:27.440 --> 00:17:29.240 that lets you navigate that tree, and. 358 00:17:29.200 --> 00:17:31.160 One of the first commands you run when you load 359 00:17:31.200 --> 00:17:35.440 that tree is soup dot pritiphy. It takes that unreadable 360 00:17:35.480 --> 00:17:39.599 block of code and magically applies proper indentation. It visually 361 00:17:39.599 --> 00:17:42.920 structures the HTML so a human can instantly see which 362 00:17:42.960 --> 00:17:46.720 tags are nested inside of other tags. It restores your sanity, 363 00:17:46.960 --> 00:17:47.559 it really does. 364 00:17:48.079 --> 00:17:50.720 But the true magic is in the navigation using what 365 00:17:50.839 --> 00:17:52.000 is known as dot notation. 366 00:17:52.440 --> 00:17:55.480 Right with rejex, if we wanted the page title, we 367 00:17:55.559 --> 00:17:58.039 had to build wild cards and capturing groups to sift 368 00:17:58.039 --> 00:18:01.759 through characters. With beautiful soup, you don't use wild cards. 369 00:18:01.799 --> 00:18:03.440 You just type soup dot title. 370 00:18:03.240 --> 00:18:06.359 Dot string exactly. You literally ask the soup object for 371 00:18:06.440 --> 00:18:09.039 the title element and then ask for the string inside it, 372 00:18:09.160 --> 00:18:10.920 and it just hands it to you. If you want 373 00:18:10.920 --> 00:18:13.279 the very first paragraph inside the body of the web page, 374 00:18:13.319 --> 00:18:15.319 you don't write a pattern. You just type soup dot 375 00:18:15.359 --> 00:18:18.240 body dot p It elegantly walks down the branches of 376 00:18:18.240 --> 00:18:18.799 the parch. 377 00:18:18.599 --> 00:18:21.880 Tree that is so clean. It understands the concept of 378 00:18:21.880 --> 00:18:25.160 a hierarchical family tree. It knows what a parent element is, 379 00:18:25.200 --> 00:18:27.599 what a child element is, and what a sibling element is. 380 00:18:28.119 --> 00:18:31.559 Every single navigable piece of data in that tree has 381 00:18:31.599 --> 00:18:35.559 a parent property. So if your code locates a specific 382 00:18:35.640 --> 00:18:37.839 hyperlink in a tag, but you need to know what 383 00:18:37.880 --> 00:18:41.440 container is actually holding that link, you simply call soup 384 00:18:41.880 --> 00:18:46.240 dot a dot parent. The library instantly traverses up the 385 00:18:46.279 --> 00:18:49.759 tree and hands you the surrounding tag, whether it's a paragraph, 386 00:18:49.920 --> 00:18:51.240 a menu list, or a sidebar. 387 00:18:51.480 --> 00:18:53.759 So we have the map. Does that mean we throw 388 00:18:53.759 --> 00:18:55.200 our rejecs compass in the trash? 389 00:18:55.319 --> 00:18:58.119 Not at all. The two tools are meant to be combined. 390 00:18:58.880 --> 00:19:01.759 Beautiful Soup is vastly superior for the heavy lifting of 391 00:19:01.759 --> 00:19:05.519 structural navigation because it understands the architecture of the web page, 392 00:19:05.920 --> 00:19:09.960 but ridjex remains the undisputed king of highly specific string extraction. 393 00:19:10.079 --> 00:19:12.960 I see think about your city analogy. You use Beautiful 394 00:19:12.960 --> 00:19:15.480 Soup's map to confidently navigate down the street, enter the 395 00:19:15.480 --> 00:19:18.079 correct house, walk down the hallway, and step into the 396 00:19:18.119 --> 00:19:21.000 exact right room. Okay, And then once you are safely 397 00:19:21.039 --> 00:19:23.519 in that room, you pull out your regular expressions compass 398 00:19:23.559 --> 00:19:26.000 to search the drawers for the specific item you need. 399 00:19:26.200 --> 00:19:28.599 That makes total sense. You use the tools for what 400 00:19:28.599 --> 00:19:31.119 they're best at. Let's look at the journey we've mapped 401 00:19:31.160 --> 00:19:34.279