Speaker 1: I want you to picture something for a second. You're
Speaker 1: sitting at your computer and you're doing something you do
Speaker 1: probably a dozen times a day. You type a search
Speaker 1: term into a massive e commerce site like Amazon or Walmart,
Speaker 1: right right, yeah, and you hit enter and then boom,
Speaker 1: magic happens. A customized list of products just pops onto
Speaker 1: your screen. But what is actually happening behind the scenes
Speaker 1: when we submit that data to a server, Like, how
Speaker 1: does a process our intent?
Speaker 2: Well, it's funny you call it magic, because you know,
Speaker 2: when you're first learning about programming or data science, it
Speaker 2: really does feel that way totally. But it's not magic.
Speaker 2: It's essentially a digital contract. Basic static websites, they don't
Speaker 2: really require you to know much. Pulling text off a
Speaker 2: flat article is one thing, But interacting with search bars
Speaker 2: and massive databases that is essential for any aspiring tech.
Speaker 1: Enthusiast exactly, which is the whole mission of our Deep
Speaker 1: Die today. We are going to learn how to master
Speaker 1: web scraping by understanding how to programmatically handle these web
Speaker 1: form submissions.
Speaker 2: Yeah, because to get that dynamic data, you can't just
Speaker 2: passively read a page. You have to actively converse with
Speaker 2: the server. And to do that you need to know
Speaker 2: the map, the map. I like that, right, So we're
Speaker 2: going to break down the two dominant types of web
Speaker 2: forms you'll encounter out in the wild. That's get forms
Speaker 2: and post forms, and you know how to actually write
Speaker 2: Python code to handle them.
Speaker 1: Okay, let's unpack this. We'll start with the first type,
Speaker 1: which I think you said is the most common, right,
Speaker 1: the get form.
Speaker 2: Yeah, exactly. A geet form is basically how the vast
Speaker 2: majority of simple web pages work, and it's defining characteristic
Speaker 2: is transparency.
Speaker 1: Transparency. What do you mean by that?
Speaker 2: Well, the parameters of what you're searching for are placed
Speaker 2: directly out in the open, right in the URL. So
Speaker 2: if you go to a huge database like the Internet
Speaker 2: movie database IMDb, and you search for an exact name
Speaker 2: match let's say, an actor name mark.
Speaker 1: Right, I just type mark into the search bar.
Speaker 2: You do, and when you hit enter, you arrive at
Speaker 2: a new page. But if you look up at your
Speaker 2: browser's address bar, the URL has completely changed.
Speaker 1: Right, It's not just MDB dot dot com anymore, exactly.
Speaker 2: It's a long string of text and right at the
Speaker 2: end you'll see a question mark followed by something like
Speaker 2: qequals mark.
Speaker 1: Oh, Wow. I see that all the time and I
Speaker 1: just sort of ignore it. But it's like a postcard.
Speaker 2: A postcard, Yeah, think about it.
Speaker 1: A get form is literally like a postcard. Anyone handling
Speaker 1: it can read the message right on the back. The
Speaker 1: que in that URL literally denotes the query, the term
Speaker 1: you searched for. It's just sitting there in plaintext.
Speaker 2: That's a perfect analogy. Actually, the question mark is the
Speaker 2: dividing line between the server address and your specific instructions.
Speaker 2: And because it's a postcard, anyone, your local router, the
Speaker 2: internet service provider can read it without opening an envelope.
Speaker 1: So if I want to scrape this data using Python,
Speaker 1: I basically just have to forge that postcard. Right.
Speaker 2: You hit the nail on the head. You don't have
Speaker 2: to trick a browser into clicking a button. You just
Speaker 2: generate the right URL.
Speaker 1: Walk me through the actual code for that, like step
Speaker 1: by step.
Speaker 2: Okay, so first you import your necessary packages for this.
Speaker 2: You really just need two foundational Python libraries, Requests and
Speaker 2: beautiful Soup.
Speaker 1: Okay, requests and beautiful Soup.
Speaker 2: Got it. So you use the get mechod from the
Speaker 2: request library. You literally type request stut get and you
Speaker 2: feed it that full URL string with your q equals
Speaker 2: mark parameter.
Speaker 1: And Python basically acts as my browser firing off that
Speaker 1: postcard exactly.
Speaker 2: It retrieves the raw HTML data from the URL. But
Speaker 2: you know, eahml is just a massive wall of text
Speaker 2: and angled brackets. It's really messy, right.
Speaker 1: So how do I pull just the actors names.
Speaker 2: Out of that mess That's where you pass the content
Speaker 2: through the built in parser in beautiful Soup. You don't
Speaker 2: just search the text. You go to the actual web
Speaker 2: page in your regular browser first write click and hit
Speaker 2: inspect ah.
Speaker 1: The inspect element tool. I love that thing.
Speaker 2: It's the best. You use it to find these specific
Speaker 2: HTML tags or CSS classes that hold the actors' names.
Speaker 2: Then you tell beautiful Soup to find those exact elements
Speaker 2: in the raw data it downloaded, and it just.
Speaker 1: Plucks them right out. That is incredibly powerful, And because
Speaker 1: all the data is right there in the URL, gettt
Speaker 1: forms are incredibly easy to share too, Like I can
Speaker 1: copy and paste that link to a friend and they'll
Speaker 1: see the exact same list of actors.
Speaker 2: Yes, because the URL contains the entire state of the search.
Speaker 2: It's brilliant.
Speaker 1: But wait a minute, if geet forms are so easy
Speaker 1: and so shareable, why do we need anything else? Like
Speaker 1: what happens when a website doesn't want to put all
Speaker 1: its cards on the table in the URL?
Speaker 2: Well, think about your postcard analogy. What if you're sending
Speaker 2: your bank account number or a password?
Speaker 1: Oh right, I definitely don't want that written on the
Speaker 1: back of a postcard for my WiFi router to log.
Speaker 2: Exactly when you're dealing with sensitive data or when a
Speaker 2: request actually changes something on the server, like updating a profile,
Speaker 2: you need a sealed envelope. And that introduces us to
Speaker 2: the hidden reality of PST.
Speaker 1: Forms the post form okay.
Speaker 2: And I have to warn you these are fundamentally not
Speaker 2: scrape friendly. Unlike geet forms, PSTR forms do not change
Speaker 2: the URL to reflect the search parameters, so the data
Speaker 2: is just hidden completely. Let me give you a real
Speaker 2: world example. There's this database run by the California Department
Speaker 2: of Consumer Affairs the DCA. They license practitioners across like
Speaker 2: two hundred and fifty five different professions.
Speaker 1: Okay, So let's say I want to find a specific
Speaker 1: dentist in California.
Speaker 2: Right, you go to the site. It's like the Dental
Speaker 2: Board of California. You're from a dropdown. You choose dentist license,
Speaker 2: and let's say you leave the license number blank, but
Speaker 2: you type in the first name John and the last.
Speaker 1: Name Kelly standard search form staff standard stuff.
Speaker 2: You hit search, the database churns, and it spits back
Speaker 2: a result showing say, eight people holding that specific license.
Speaker 1: Here's where it gets really interesting, though. If you check
Speaker 1: the URL on this results page, it gives absolutely no indication.
Speaker 2: Of your search none at all.
Speaker 1: It's just a generic web address. There's no query string,
Speaker 1: no question mark, no John Kelly anywhere. The parameters are
Speaker 1: entirely hidden.
Speaker 2: Right. Building on what we said earlier, a POSTD form
Speaker 2: is like a sealed envelope. You securely hand the data
Speaker 2: to the server, but it's completely hidden from the outside observer.
Speaker 1: And it's hidden from a simple URL scrape too.
Speaker 2: Exactly. If you open Python and try that request dot
Speaker 2: get trick on the generic URL of the results page,
Speaker 2: you will.
Speaker 1: Get nothing back, really just an error.
Speaker 2: Basically, you can even check the response text in Python
Speaker 2: and it'll confirm it's empty or just a blank form
Speaker 2: because the server is looking at your request saying where's
Speaker 2: the envelope?
Speaker 1: That makes total sense. So to succeed, my Python code
Speaker 1: essentially has to pretend to be a human being physically
Speaker 1: filling out the drop downs and clicking the button exactly.
Speaker 2: But how does Python know what to put in the
Speaker 2: envelope if we can't see the URL changing?
Speaker 1: Yeah, that's the million dollar question. How do we crack
Speaker 1: the post form code?
Speaker 2: What's fascinating here is we actually have to spy on ourselves.
Speaker 1: Spy on ourselves, I love it. Tell me more.
Speaker 2: We use a clever workaround leveraging Chrome's developer tools, specifically
Speaker 2: the network tab. Who basically set up a wire tab
Speaker 2: on our own browser to see the hidden request right
Speaker 2: as it's being said.
Speaker 1: Okay, so walk me through the step by step for
Speaker 1: doing this.
Speaker 2: Sure, you go to that blank DCA search form before
Speaker 2: you touch anything. You go to the view tab in
Speaker 2: your browser, down to developer and click on developer tools.
Speaker 2: Then you click over to the network tab. It usually
Speaker 2: just looks like a blank gray screen at first.
Speaker 1: Just waiting for something to happen.
Speaker 2: Exactly, leave it open, go back to the web page,
Speaker 2: fill out the form just like we did John Kelly Dentist,
Speaker 2: and hit submit. The moment you do, that network tab
Speaker 2: is going to bustle with activity.
Speaker 1: Because it's logging every single piece of data flying back
Speaker 1: and forth.
Speaker 2: Right, you'll see dozens of items load. You want to
Speaker 2: scroll to the very top of that pain. So, like
Speaker 2: the main document, request for the URL you're at, and
Speaker 2: then look over in the section called headers.
Speaker 1: The headers.
Speaker 2: Okay, if you scroll down inside the headers, you'll eventually
Speaker 2: find a section labeled form data or sometimes payload, and
Speaker 2: that's the jack cut is the jackpot. That section literally
Speaker 2: reveals the internal blueprint of the sealed envelope. It shows
Speaker 2: you the exact variable names the database expects. So instead
Speaker 2: of guessing, you see that the server wants like board
Speaker 2: code equals twenty and first name equals John.
Speaker 1: So now I just take that browsersleuthing and translate it
Speaker 1: into Python. Yep, I take all that form data I
Speaker 1: just found, and I convert it into a standard Python dictionary.
Speaker 1: Just key value pairs wrapped in curly braces.
Speaker 2: You've got it. It's just a slight modification to the
Speaker 2: script we wrote earlier.
Speaker 1: Right, So instead of request dot get, I'm guessing, I
Speaker 1: use requests dot post Exactly.
Speaker 2: You call request dot post. You pass in the target
Speaker 2: url of the form but crucially, you also pass in
Speaker 2: your new dictionary using the data parameter.
Speaker 1: Oh so I'm literally handing the requests library, the destination address,
Speaker 1: and the fully loaded envelope.
Speaker 2: Yes, you say data your dictionary, and when you run that,
Speaker 2: the server is tricked. It thinks you interacted with the
Speaker 2: actual web page. It opens the envelope process of the data,
Speaker 2: and your Python scripts zestfully prints out the names of
Speaker 2: the eight dentists. Boom.
Speaker 1: That is incredibly satisfying.
Speaker 2: It's a huge breakthrough when you first get it to work.
Speaker 1: So what does this all mean? Like, by mastering these
Speaker 1: two techniques extracting query strings from the URLs for debt
Speaker 1: t forms and spying on the network tab to send
Speaker 1: data dictionaries for post forms, you are essentially empowered to
Speaker 1: scrape the vast majority of websites on the Internet.
Speaker 2: Absolutely, you're no longer just looking at a static page.
Speaker 2: You're actually talking to the database. But you know, we
Speaker 2: should probably mention that servers do fight back.
Speaker 1: Oh, they definitely do. It's a whole cat and mouse game, right.
Speaker 2: Like they use things called CSRF tokens cross site request
Speaker 2: forgery tokens, basically a hidden randomized fingerprint the server puts
Speaker 2: on the form when a real human loads it.
Speaker 1: So if my Python script just blasts the post request
Speaker 1: without visiting the page first.
Speaker 2: It gets rejected because it's missing the fingerprint. You'd have
Speaker 2: to write a script that does a get tea request
Speaker 2: first to steal the token, inject it into your dictionary,
Speaker 2: and then send the po request.
Speaker 1: Wow, so it's a two step dance.
Speaker 2: It really is an arms race. Yeah, but that's the
Speaker 2: big picture here. Understanding how data moves between a client
Speaker 2: and a server, whether it's transparent or hidden. That's the
Speaker 2: absolute foundation of both webscraping and cybersecurity.
Speaker 1: It totally demystifies the web. You start looking at every
Speaker 1: search bar differently, which actually brings me to a quick
Speaker 1: review exercise for you as you're listening today.
Speaker 2: Oh this is a great idea.
Speaker 1: Yeah, next time you are casually browsing a website, maybe
Speaker 1: shopping or using a dropdown menu, I want you to
Speaker 1: keep an eye on the URL. Does it change to
Speaker 1: show your search terms or does it stay exactly the same.
Speaker 2: Try to spot if it's a get tea or.
Speaker 1: A post exactly, and if it stays the same, try
Speaker 1: opening your browsers developer tools head over to the network tab,
Speaker 1: or run the search again and see if you can
Speaker 1: spot the hidden form data being sent inside the payload.
Speaker 1: It's so cool once you see it with your own eyes.
Speaker 2: It really builds that critical technical intuition.
Speaker 1: It does, but it also leads me with this lingering
Speaker 1: question to ponder, oh, what's that Well, if we can
Speaker 1: easily write Python scripts that perfectly mimic human behavior like
Speaker 1: filling out these hidden povis T forms and tricking servers
Speaker 1: into handing over their data, how can website creators truly
Speaker 1: tell the difference between a curious learner and an invisible
Speaker 1: army of automated bots.
Speaker 2: That is the billion dollar question in tech right now, right, and.
Speaker 1: What does that mean for the future of navigating the web?
Speaker 1: If the line between user and script keeps getting thinner,
Speaker 1: we'll think about the invisible logic guarding the gates the
Speaker 1: next time you hit search