Speaker 1: You know, we walk around all day with this massive,
Speaker 1: invisible engine running our modern lives, and most of the
Speaker 1: time we just don't even think about it.
Speaker 2: Oh, for sure, we're completely oblivious to it.
Speaker 1: Right, I'm talking about data. Just think about it for
Speaker 1: a second. For you listening, your smart watch tracks your
Speaker 1: heart rate right, and your sleep cycles.
Speaker 2: And it triggers all those personalized.
Speaker 1: Health alerts exactly. And your browsing habits are being quietly
Speaker 1: collected to you know, serve up those hyper targeted ads
Speaker 1: that sometimes feel it still a little too.
Speaker 2: Accurate, like they're reading your mind.
Speaker 1: Yeah, even your Friday night on the couch is being analyzed,
Speaker 1: like you're viewing habits magically predicting the exact next TV
Speaker 1: show you want to binge.
Speaker 2: Well, the modern digital experience is entirely fueled by data collection.
Speaker 2: I mean, every click, every pause, every service, Yeah, it's
Speaker 2: a data point. It's being processed and utilized to build
Speaker 2: a customized reality just for you.
Speaker 1: But here's the thing that always gets me. If you're
Speaker 1: a data analyst or a data scientist, or just someone
Speaker 1: who likes building cool tech projects, that data isn't just
Speaker 1: magically handed to you on a silver platter.
Speaker 2: No, definitely not.
Speaker 1: You don't just ask the Internet nicely and get a
Speaker 1: perfectly formatted spreadsheet in return. You have to go get
Speaker 1: it right. And one of the biggest, most powerful methods
Speaker 1: forgetting that data is webscraping.
Speaker 2: It is an absolutely foundational skill. I mean, if the
Speaker 2: data is out there on a web page, webscraping is
Speaker 2: how you programmatically extract it, structure it, and you know,
Speaker 2: actually make it useful for your own analysis.
Speaker 1: But the Internet has changed, hasn't it. Like the easy
Speaker 1: days are kind of.
Speaker 2: Over, oh completely, The landscape is entirely different now.
Speaker 1: So the mission of today's deep dive is exactly that
Speaker 1: we are leveling up. We're taking your Python skills from
Speaker 1: basic data extraction and stepping into the gritty, real world
Speaker 1: tactics of advanced webscraping.
Speaker 2: This can be fun.
Speaker 1: We're going to map out exactly how to bypass the
Speaker 1: common roadblocks that modern websites throw at automated scripts.
Speaker 2: And just to set the stage for you listening right now,
Speaker 2: we are operating on the assumption that you already have
Speaker 2: a basic to intermediate grasp of Python programming.
Speaker 1: Right. We're not doing a beginner tutorial here exactly.
Speaker 2: We are going to be covering basic syntax or how
Speaker 2: to install libraries. Today's goal is to understand the architecture
Speaker 2: of resistance on the modern web and how you can
Speaker 2: script your way through it using advanced architectural concepts.
Speaker 1: Okay, let's unpack this because before we start writing scripts,
Speaker 1: we need to define the battlefield.
Speaker 2: Good call you hear.
Speaker 1: The term advanced web scraping a lot, But like, why
Speaker 1: isn't the basic stuff enough anymore? Where exactly is the
Speaker 1: line between a basic scrape and an advanced scrape?
Speaker 2: Well, in our context today, we define the difference by
Speaker 2: looking at the level of resistance. Okay, let's look at
Speaker 2: basic webscraping. In a basic scenario, you target a static
Speaker 2: web page, you send a request, you apply some simple
Speaker 2: scraping process to parse the HTML, and you get your
Speaker 2: required data.
Speaker 1: The server just hands you an HTML document and you
Speaker 1: read it.
Speaker 2: Right exactly minimal resistance.
Speaker 1: It's like reading a public billboard from the highway. Anyone
Speaker 1: driving by can see it, take a picture of it,
Speaker 1: and write down what it says.
Speaker 2: I like that. Yeah, that's basic scraping. But advanced scraping, however,
Speaker 2: is entirely defined by resistance. The roadblocks, right, It's what
Speaker 2: happens when a web page adds layers of difficulty between
Speaker 2: you and the data. Because much of the modern web
Speaker 2: is built on single page applicationss the spas, they use
Speaker 2: frameworks like reactor view, And when you send a basic
Speaker 2: request to one of these sites, the server doesn't hand
Speaker 2: you a nicely formatted HTMIL document with all the data.
Speaker 1: So what does it hand you.
Speaker 2: It hands you a virtually anti skeleton page and a
Speaker 2: massive bundle of JavaScript.
Speaker 1: Oh wow. So if you just run a standard scrape
Speaker 1: on that, you get a blank page.
Speaker 2: Back pretty much.
Speaker 1: You pull down the billboard and it's just a blank canvas.
Speaker 2: You're hitting the nail on the head. The data only
Speaker 2: populates after a browser executes that JavaScript. Ah, and that
Speaker 2: fundamentally breaks basic scraping tools. So advanced scraping is like
Speaker 2: trying to get information from inside an exclusive club, not
Speaker 2: a billboard exactly. Not a billboard. You can't just walk
Speaker 2: in and read the walls. You have to talk to
Speaker 2: a bouncer, show an ID, prove you've been there before,
Speaker 2: and actively participate in the mechanics of the building just
Speaker 2: to find the room you're looking for.
Speaker 1: That makes so much sense. So to get past this
Speaker 1: bouncer and into the club, we need the right tools
Speaker 1: on our tool belts we do. We need an environment
Speaker 1: that lets us test our strategy without starting from scratch
Speaker 1: every time we hit a wall. And for this, Jupiter
Speaker 1: Notebook is still the absolute gold standard for the scraping workflow.
Speaker 2: Oh, without a doubt, Jupiter's fantastic for this because statefulness
Speaker 2: is everything in advanced scraping.
Speaker 1: What do you mean my statefulness?
Speaker 2: Well, when you're testing how a website responds to your
Speaker 2: log in attempts, you don't want to have to run
Speaker 2: a massive script from the very beginning, right like triggering
Speaker 2: a fresh log in navigating three pages deep.
Speaker 1: Oh yeah, every single time you tweak one line of code,
Speaker 1: that would be bru It's.
Speaker 2: A massive waste of time. With Jupiter, you can test
Speaker 2: your connection in one cell, establish your session, and then
Speaker 2: parse the data in another cell while keeping your active
Speaker 2: connection totally intact.
Speaker 1: That's super efficient. And alongside Jupiter you have your foundational tools.
Speaker 1: The Requests library is your scout going out to fetch the.
Speaker 2: Raw data yep, your HTTP scout.
Speaker 1: And Beautiful Soup is your translator turning that raw HTML
Speaker 1: string into a navigable tree structure exactly. But those are
Speaker 1: the tools for the billboard. If we want to get
Speaker 1: into the club, and if we want to deal with
Speaker 1: those react and view websites that require JavaScript to run,
Speaker 1: we need the heavy lifter.
Speaker 2: We a Selenium.
Speaker 1: Selenium which lets Python actually automate a real web browser.
Speaker 2: Which is a complete paradigm shift. You know, you are
Speaker 2: moving from simply downloading a file to puppeteering a complex
Speaker 2: piece of software.
Speaker 1: But wait, I have to push back on this for
Speaker 1: a second, because this always trips people up.
Speaker 2: Go for it.
Speaker 1: If I have Python and I import the Selenium library,
Speaker 1: why do I also need to go out to the
Speaker 1: inn and download a separate, standalone executable called a web
Speaker 1: driver for my specific browser.
Speaker 2: It's a common question like if I use.
Speaker 1: Chrome, I need Chrome driver. Why can't Python just talk
Speaker 1: to my Chrome browser directly.
Speaker 2: What's fascinating here is the architecture of how these programs
Speaker 2: actually communicate. You have to remember Google, Chrome, Firefox, Safari.
Speaker 2: They weren't built with a native backdoor for Python scripts
Speaker 2: to just waltz in and take over their rendering engines. Right.
Speaker 1: They're built for humans exactly.
Speaker 2: They're built to listen to the operating system for human
Speaker 2: inputs like physical mouse clicks and keyboard strokes.
Speaker 1: So Python is basically speaking a completely different language than
Speaker 1: the browser's core engine.
Speaker 2: It is Selenium speaks a generalized automation language. Your specific
Speaker 2: version of Google Chrome speaks its own, highly specific, constantly
Speaker 2: updating language based on its underlying Chromium engine.
Speaker 1: Ah okay.
Speaker 2: So the webdriver acts as a dedicated translator between your
Speaker 2: Python code and the actual bread howser's debugging port.
Speaker 1: So it sits in the middle.
Speaker 2: Right. It takes a Python command like click this button
Speaker 2: translates it into the specific def duels protocol command that
Speaker 2: Chrome version one fourteen understands and executes it.
Speaker 1: Which explains why matching the versions is so critical. Like
Speaker 1: if your Chrome browser updates overnight to version one fifteen,
Speaker 1: but your web driver is still speaking the language of
Speaker 1: version one fourteen.
Speaker 2: The translation fails and your script crashes.
Speaker 1: Wow okay, So setting this up requires you to check
Speaker 1: your exact browser build version, download the matching web driver executable,
Speaker 1: and put it in your system's path variable.
Speaker 2: Yep, And once Selenium knows where that translator lives, it
Speaker 2: can initialize a real visible browser window that starts navigating
Speaker 2: the web on its own.
Speaker 1: It's a little spooky the first time you see it happen,
Speaker 1: just a ghost driving or browser.
Speaker 2: It really is.
Speaker 1: Okay, so our environment is set up. We have Jupiter
Speaker 1: for iterative testing, our scout requests, our translator beautiful Soup,
Speaker 1: and our robotic arm Selenium. Successfully connected to the browser
Speaker 1: via the web driver.
Speaker 2: The squad is ready, the.
Speaker 1: Bouncer is looking at us, and from what I understand,
Speaker 1: the first two major hurdles we have to clear are
Speaker 1: forms and cookies.
Speaker 2: That's right, and a web form is essentially a gateway
Speaker 2: that allows a user to send data back to the
Speaker 2: serfer for processing. In the context of scraping, the most
Speaker 2: common form we care about is the search bar. You
Speaker 2: want to retrieve specific results from a database, Okay. To
Speaker 2: do this, your script needs to know the architectural difference
Speaker 2: between a get to request and a post request.
Speaker 1: Well, I can usually spot a giddy request just by
Speaker 1: looking at my browser right. It basically staples your search
Speaker 1: query right onto the end of the URL.
Speaker 2: Exactly.
Speaker 1: If I search Wikipedia for Python, I can see my
Speaker 1: search terms right there in the address bar at the top,
Speaker 1: appended after a question.
Speaker 2: Mark that visibility makes GIDDY requests incredibly straightforward to scrape.
Speaker 2: You don't even really need to simulate filling out the form.
Speaker 2: You can often just programmatically construct the final URL string
Speaker 2: and Python and send a request directly to it.
Speaker 1: Oh that's easy, But a PS two request doesn't do that.
Speaker 2: No, it doesn't.
Speaker 1: When I log into my bank account, my password doesn't
Speaker 1: suddenly appear in the URL bar for the whole world
Speaker 1: to see. A POSD request packages that data up securely
Speaker 1: behind the scenes.
Speaker 2: Right. The payload of a p post request is embedded
Speaker 2: in the body of the HTTP request itself, alongside specific
Speaker 2: headers that tell the server what kind of data is incoming.
Speaker 1: So how do we script that?
Speaker 2: Well, if you want to scrape data hidden behind a
Speaker 2: PST form, your script has to be tailored to construct
Speaker 2: that exact payload dictionary. Okay, you're mapping the specific input
Speaker 2: field names to your desired values, and then you submit
Speaker 2: it programmatically.
Speaker 1: And if you mess it up, if you format.
Speaker 2: The payload wrong or miss a required hidden field, the
Speaker 2: server will just reject it or return a generic air page.
Speaker 1: Brutal. But here's where it gets really interesting. Though. Submitting
Speaker 1: a form is one thing you get past the bouncer.
Speaker 1: But what if you need to browse five different pages
Speaker 1: behind that.
Speaker 2: Login screen is where a lot of people get stuck.
Speaker 1: My instinct tells me I could just use a standard
Speaker 1: requests dot get command for the next page, But I
Speaker 1: know that fails.
Speaker 2: Why it fails because of a fundamental flaw on the
Speaker 2: HTTP protocol for our purposes, it is inherently stateless, meaning
Speaker 2: it is zero memory.
Speaker 1: Wait, so every time I click a link, the server
Speaker 1: basically treats me like a brand new visitor who just
Speaker 1: walked at the front.
Speaker 2: Door without a mechanism to maintain state. Yes, if you
Speaker 2: log in to a site via post request and then
Speaker 2: immediately send a new request dot get command to access
Speaker 2: your profile page, the server has already forgotten your successful login.
Speaker 2: Oh wow, it will just bounce you back to the
Speaker 2: login screen. And this is why cookies were invented. Cookies
Speaker 2: are just tiny text files the server slips into your browser.
Speaker 2: They contain a unique session identifier.
Speaker 1: Okay, I have an analogy for this. It's like the
Speaker 1: bouncer checking your ID, letting you into the club, and
Speaker 1: stamping your hand.
Speaker 2: I love that.
Speaker 1: Yes, that stamp is the cookie. When you walk from
Speaker 1: the bar, to the VIP lounge. The security guard inside
Speaker 1: doesn't need to see your ID again, he just checks
Speaker 1: for the snamp.
Speaker 2: That is the perfect analogy, and this dictates how we
Speaker 2: write our Python scripts. If we just use the standard
Speaker 2: request dot get method, we are walking into the club,
Speaker 2: refusing the handstamp and wondering why we keep getting thrown out?
Speaker 1: Right, So how do we take the stamp to solve this?
Speaker 2: Instead of standard requests, we initialize a request's session object.
Speaker 1: Ah. So the session object acts as.
Speaker 2: Our hand exactly. It automatically catches the cookies the server sends,
Speaker 2: stores them, and meticulously attaches them to every subsequent request,
Speaker 2: so the server knows it's still us.
Speaker 1: That is so clever.
Speaker 2: Managing that session state programmatically is what elevates a script
Speaker 2: from a basic pull to an advanced operation. It allows
Speaker 2: you to maintain the illusion of a continuous, authenticated user journey.
Speaker 1: So cookies solve our amnesia problem behind the scenes. We
Speaker 1: can log in and we can remember who.
Speaker 2: We are yep.
Speaker 1: But what happens when the data is literally locked behind
Speaker 1: a visual pop up on the screen that requires a
Speaker 1: physical mouse click.
Speaker 2: Ah. The interactive elements.
Speaker 1: Yeah, I'm thinking about those terms and conditions, check boxes
Speaker 1: or radio buttons where you have to select a gender
Speaker 1: or an industry before the next button becomes clickable.
Speaker 2: This where we fully transition into using Selenium to mimic
Speaker 2: human interaction. From a human perspective, a checkbox is just
Speaker 2: a minor annoyance. You see the box, you move your mouse,
Speaker 2: you click. But a standard Python script doesn't have eyes.
Speaker 1: So if I'm blocked by a checkbox, can I just
Speaker 1: tell Python to search the misible text of the page
Speaker 1: for the word agree and click on it.
Speaker 2: You'd think so, But the text you see rendered on
Speaker 2: the screen and the structural code the website reads are
Speaker 2: two different things. Often the text I agree is just
Speaker 2: a visual label floating next to the actual functional element.
Speaker 1: So where is the functional element?
Speaker 2: We have to dive into the underlying HTML document object model,
Speaker 2: the DOM.
Speaker 1: Okay, so imagine the DOM like a massive upside down tree,
Speaker 1: right with the main HTML tag at the route, branching
Speaker 1: out into body tags, div containers, and eventually the tiny
Speaker 1: little leaf that represents our checkbox.
Speaker 2: That's a great visual and to click that leaf we
Speaker 2: need its exact address in the tree. We use a
Speaker 2: locator strategy, most commonly an XPath an XPath, Yeah, an
Speaker 2: XPath is a syntax for navigating through elements and attributes
Speaker 2: in an XML or HTML document.
Speaker 1: But earlier you mentioned React and View making things difficult
Speaker 1: by dynamically generating the page. Don't those frameworks also randomize
Speaker 1: the class names of those elements every time the page loads.
Speaker 2: They absolutely do. It's a nightmare for basic scrapers.
Speaker 1: Right, because if the class name of the checkbox changes
Speaker 1: from box one twenty three to box ninety eighty seven
Speaker 1: on my next visit, a basic XPath looking for box
Speaker 1: one twenty three is going to break immediately.
Speaker 2: Which is why advance scraping requires writing robust relative.
Speaker 1: Xpaths relative xpaths. Okay, how did those work?
Speaker 2: Instead of looking for a hard coded randomized class name,
Speaker 2: you program your XPath to look for structural relationships, like
Speaker 2: what you might tell Selenium. Find the text terms and conditions,
Speaker 2: then find the div container immediately preceding it, and click
Speaker 2: the input tag inside that container.
Speaker 1: Oh, I see, you're triangulating it exactly.
Speaker 2: Once you establish that robust locator, you issue the dot
Speaker 2: click command and Selenium uses the webdriver to simulate a
Speaker 2: fiscal OS level mouse click on those precise coordinates in
Speaker 2: the browser.
Speaker 1: It's like giving our script a tiny robotic finger that
Speaker 1: knows exactly how to navigate a shifting maze. Well said,
Speaker 1: but clicking a box is just one action. There's a
Speaker 1: final boss to this whole operation that requires passing actual
Speaker 1: files through the browser. Uploads.
Speaker 2: Oh, file uploads. They are a significant hurdle, but mastering
Speaker 2: them unlocks a massive amount of potential.
Speaker 1: Give me an example.
Speaker 2: Think about trying to scrape data from a professional networking site.
Speaker 2: You often can't even see other user's data until you
Speaker 2: fully filled out your own profile, which means you have
Speaker 2: to programmatically upload a profile picture.
Speaker 1: Oh raw.
Speaker 2: Maybe you need to submit a PDF resume to a
Speaker 2: job board to access premium employer data.
Speaker 1: Now my brain immediately goes to how I do this?
Speaker 1: As a human? I click the upload button on the
Speaker 1: web page. Then a smaller window pops up on my
Speaker 1: screen like my Macfinder or Windows file explorer, and I
Speaker 1: click through my folders to find the PDF.
Speaker 2: That's the human way.
Speaker 1: But Selenium is controlling the web browser. It doesn't have
Speaker 1: access to my max operating system Windows. So when that
Speaker 1: file explorer pops up, isn't Selenium just totally blind?
Speaker 2: To it. It is entirely blind to it. That operating
Speaker 2: system dialogue box is a walled garden that browser automation
Speaker 2: tools cannot touch. Wait, really, really, if your script clicks
Speaker 2: the upload button and triggers the pop up, your automation
Speaker 2: effectively freezes.
Speaker 1: So how on earth do we get the file from
Speaker 1: our hard drive to the server if we can't use
Speaker 1: the upload window.
Speaker 2: We bypass the visual interface entirely and exploit the underlying
Speaker 2: HTML mechanics.
Speaker 1: Okay, I'm listening.
Speaker 2: When a developer builds an upload feature, they use a
Speaker 2: specific HTML tag. It's an input tag with the attribute
Speaker 2: type file.
Speaker 1: That's the actual engine of the upload, hiding behind the
Speaker 1: pretty styled click here to upload button exactly.
Speaker 2: The visual button is just a trigger. The hidden input
Speaker 2: type filetag is what actually handles the file transfer.
Speaker 1: So we don't click the button at all.
Speaker 2: Nope. Instead of using Selenium to click the visual button
Speaker 2: and trigger the OS window, we use our robust XPath
Speaker 2: to locate that hidden input.
Speaker 1: Tag in the dom, and once we find it.
Speaker 2: We use the dot send keys yeah command in Selenium.
Speaker 1: Wait, usually you use send keys to type text into
Speaker 1: a search bar.
Speaker 2: Right, Yes, normally, but when you use it on an
Speaker 2: input type filetag, and you pass it the absolute filepath
Speaker 2: of your PDF on your hard drive. The browser's engine
Speaker 2: interprets that as a direct file selection.
Speaker 1: Oh, that is brilliant. You just inject the filepath straight
Speaker 1: into the DOM and the browser attaches the file exactly
Speaker 1: as if you would click through the OS window.
Speaker 2: You bypass the pop up window entirely. And the beauty
Speaker 2: of this mechanism is its universality. What do you mean.
Speaker 2: Once you understand the DOM interaction for pushing a jpeg
Speaker 2: to a web server, you understand the process for pushing
Speaker 2: any file type, a PDF, a CSV, a text document.
Speaker 1: It's all the same.
Speaker 2: Yeah, the underlying mechanics of the HTML input tag remain identical.
Speaker 1: So what does this all mean When you pull back
Speaker 1: and look at everything we've unpacked today, it's a lot.
Speaker 1: It is managing post payloads, utilizing session objects to retain
Speaker 1: our cookies so we don't look like an amnesiac, writing
Speaker 1: relative X paths to navigate randomized down trees, and injecting
Speaker 1: filepaths to bypass OS level upload windows.
Speaker 2: It's a full toolkit.
Speaker 1: Advanced webscraping is really just the art of perfectly reverse
Speaker 1: engineering human browser interaction. You are teaching a Python script
Speaker 1: to navigate the complex architecture of the modern web at
Speaker 1: superhuman speeds.
Speaker 2: You are building a digital proxy of yourself, capable of
Speaker 2: handling the same stateful, complex interactions that you do every day.
Speaker 1: To really lock this in, let's do a quick mental
Speaker 1: exercise for you listening. Imagine a real world scenario. Let's
Speaker 1: say you want to build a tool that scrapes a
Speaker 1: major job board to analyze salary trends. Okay, To do this,
Speaker 1: your script needs to execute a sequence. First, it needs
Speaker 1: to log in to your account. Then it needs to
Speaker 1: click a checkbox that says I am seeking remote work right. Next,
Speaker 1: it not's to search the term Python developer, and finally
Speaker 1: it needs to upload your resume to access the premium
Speaker 1: employer's salary data. Mentally, map out which tools you would
Speaker 1: use for each step of that process. Where does a
Speaker 1: tool like beautiful Soup end and where does Selenium begin.
Speaker 2: It's a brilliant exercise in script architecture because the site
Speaker 2: likely relies heavily on JavaScript. To render those premium data
Speaker 2: tabled Beautiful Soup and request the loan cut it, you'd
Speaker 2: initialize your Selenium webdriver right off the bat. You'd use
Speaker 2: it to navigate to the log in page, construct the
Speaker 2: post payload for your credentials, and establish your session cookies.
Speaker 1: Then you'd use your relative x paths to locate that
Speaker 1: remote work checkbox and the hidden file input tag for
Speaker 1: your resume, using sen keys to bypass the RS window.
Speaker 2: Exactly only after all those human like interactions are finished
Speaker 2: and the final page of rich rendered salary data finally
Speaker 2: populates the dom, do you transition.
Speaker 1: To beautiful soup.
Speaker 2: Yep, you can have Selenium extract the fully rendered page
Speaker 2: source and hand that massive htmail string over to beautiful
Speaker 2: soup to quickly and efficiently parse out the numbers the.
Speaker 1: Right tool for the right job. You use the robotic
Speaker 1: arm of Selenium to navigate the maze and open the vault,
Speaker 1: and you use the parser of beautiful Soup to read
Speaker 1: the gold inside.
Speaker 2: It's a beautiful system. But if we connect this to
Speaker 2: the bigger picture, though, it raises a really profound question
Speaker 2: about the future of the Internet. Oh, like, what as
Speaker 2: we get better at this, you know, as our Python
Speaker 2: scripts become more sophisticated at perfectly mimicking human behavior effortlessly
Speaker 2: managing suchion state and seamlessly uploading files. The architecture of
Speaker 2: resistance is going to evolve.
Speaker 1: H The bouncers always get stricter.
Speaker 2: They do. We are already seeing the dawn of security
Speaker 2: systems that don't just look at what you click, but how.
Speaker 1: You click, how you click.
Speaker 2: Yeah, they analyze the micro rhythms of human behavior. They
Speaker 2: track the slight hesitation before a user clicks a SOMEB button.
Speaker 2: They map the subtle nonlinear curve of a human moving
Speaker 2: a physical mouse across a mousepad, comparing it to the perfect,
Speaker 2: mathematically straight line of a Selenium click.
Speaker 1: It's wild.
Speaker 2: They even measure the inconsistent speed of human typing against
Speaker 2: the instantaneous injection of text.
Speaker 1: Oh wow, So if your script locates a robust XPath
Speaker 1: and clicks a checkbox and exactly zero point zero zero
Speaker 1: one milliseconds every single time the page loads the AI,
Speaker 1: a bouncer analyzing the dom interactions knows instantly that you're
Speaker 1: a robot, no matter how good your cookies are.
Speaker 2: Exactly. And when that level of behavioral analysis becomes the
Speaker 2: standard across the web, the question becomes, how will our
Speaker 2: automation scripts have to adapt?
Speaker 1: We can't just be perfect anymore.
Speaker 2: No, it won't be enough to just technically execute the
Speaker 2: actions of a human. Our code will have to learn
Speaker 2: how to feel human. We'll have to program hesitation, mathematical error,
Speaker 2: and Gaussian randomness into our algorithms just to read a
Speaker 2: web page without triggering the alarms.
Speaker 1: That is a wild thought to leave off on. We
Speaker 1: started by talking about getting past a bouncer into an
Speaker 1: exclusive club. It sounds like in the future it won't
Speaker 1: be enough to just have the right ID and know
Speaker 1: the layout of the building. Yep, You'll have to know
Speaker 1: exactly how to stumble through the door to prove you
Speaker 1: actually belong there.
Speaker 2: You'll have to fake being human.
Speaker 1: Keep that in mind next time you architect a script
Speaker 1: to scrape some complex data. Are you being just a
Speaker 1: little too perfect?
Speaker 2: Something to ponder as you refine your code.
Speaker 1: Absolutely well, go build something cool and we'll catch you
Speaker 1: on the next deep dive.