Speaker 1: Welcome to the deep Dive. We're the show that helps
Speaker 1: you cut through the noise, taking stacks of sources and
Speaker 1: finding those key insights so you can get genuinely well
Speaker 1: informed fast. Today we're diving into something that feels well,
Speaker 1: part magic, part engineering, maybe even a little bit detective work.
Speaker 2: It's web scraping, right, the ability to basically write a
Speaker 2: little program that goes out onto the Internet and gathers
Speaker 2: data all by itself.
Speaker 1: Yeah, and seeing a work the first time, there's this
Speaker 1: real rush, like you've unlocked some secret level of the
Speaker 1: web or something.
Speaker 2: Definitely.
Speaker 1: Our main guide today is the book Practical Web Scraping
Speaker 1: for Data Science by Seppie Vanden Brook and Bart Basin's
Speaker 1: really comprehensive.
Speaker 2: Stuff it is. It covers a lot of ground.
Speaker 1: So our mission today to really get you up to
Speaker 1: speed on what webs are graping is why it's so
Speaker 1: important for data science and crucially the things you absolutely
Speaker 1: need to think about technically and maybe even more importantly ethically.
Speaker 2: Yeah, the how, but also the should you.
Speaker 1: Exactly get ready for some aha moments because we're going
Speaker 1: to unpack how you can get a solid handle on
Speaker 1: this pretty powerful skill. Okay, so let's start at the beginning.
Speaker 1: What actually happens under the hood when you just type
Speaker 1: say www dot Google dot com into your browser. Most
Speaker 1: it's just hit enter right right.
Speaker 2: And we take it for granted. But there's this incredible
Speaker 2: coordination happening invisibly, like you said, under the hood before
Speaker 2: you even see anything. All these protocols are firing off.
Speaker 2: DNS is translating that name into an IP address.
Speaker 1: The computer's actual address exactly.
Speaker 2: Then TCP make sure the data gets there reliably. But
Speaker 2: the layer we really care about for scraping the actual
Speaker 2: sort of language the web speaks.
Speaker 1: That's HTTP, Hypertext Transfer Protocol.
Speaker 2: That's the one. It's basically a plaintext conversation, the browser
Speaker 2: sense of request, the server sense of response, with the
Speaker 2: web page content. Understanding that back and forth is fundamental.
Speaker 1: Okay, so HTTP is the conversation. How do we get
Speaker 1: our program to join that conversation. How do we make
Speaker 1: those requests?
Speaker 2: Well, that's where Python's requests library is just fantastic. You
Speaker 2: can use Python's built d stuff.
Speaker 1: Of erlib, but requests is nicer, oh.
Speaker 2: Much nicer, way more user friendly. Think of it like
Speaker 2: a really efficient messenger. You just tell it, hey, go
Speaker 2: get this page using requests dot get or send this
Speaker 2: data with request dot post. It handles a lot of
Speaker 2: the fiddley bits for you automatically, like setting standard headers
Speaker 2: like user agent, which tells the server what kind of
Speaker 2: browser you are or pretending to.
Speaker 1: Be ah, so you can look like a normal browser
Speaker 1: pretty much.
Speaker 2: And crucially, it also lets you change those headers if
Speaker 2: you need to. Sometimes servers are a bit picky about
Speaker 2: who they talk to, so that flexibility is key.
Speaker 1: Right, that makes sense. So requests fetches the page content.
Speaker 1: But then you've got this big blob of well usually HTML, right,
Speaker 1: and looking at raw HTML it can be pretty intimidating
Speaker 1: all those angle brackets.
Speaker 2: Oh yeah, it looks like tag soups sometimes just the jumble.
Speaker 1: So how do we find the actual data we want
Speaker 1: inside that jumble?
Speaker 2: That's the next piece of the puzzle. HTML hypertext markup language.
Speaker 2: It looks messy, but it actually has structure. It uses
Speaker 2: tag it's like A for a link or DIV for
Speaker 2: a section. Then CSS styles it to navigate that structure
Speaker 2: and pull out specific bits. We use another great library,
Speaker 2: beautiful soup. Okay, it takes that messy HTML string and
Speaker 2: turns it into this navigable Python object like a tree
Speaker 2: structure you can walk through, ah.
Speaker 1: Like a family tree for the web page elements.
Speaker 2: Kind of Yeah, and then you can easily say, find
Speaker 2: me all the A tags or find the div with
Speaker 2: this specific ID, or even use these really powerful CSS
Speaker 2: selectors to pinpoint exactly the element you need based on
Speaker 2: its styling or position, and just.
Speaker 1: Building on that for you listening, your browser's developer tools
Speaker 1: are like your secret weapon here. Seriously, invaluable, absolutely cannot
Speaker 1: stress that enough. You hit F twelve usually, and the
Speaker 1: elements tab shows you that nice structured tree view of
Speaker 1: the HTML that beautiful soup will see. You can hover
Speaker 1: over stuff on the page, see the code light up.
Speaker 2: Yeah, it's brilliant for figuring out what tags or what
Speaker 2: CSS selectors you need to target. You can often just
Speaker 2: write click an element and copy it select directly.
Speaker 1: Just one quick tip though, remember view source shows the
Speaker 1: raw HTML the server scent. The elements tab shows what
Speaker 1: the browser has processed, which might include changes made by
Speaker 1: JavaScript after the page loaded.
Speaker 2: That's a really key distinction. Yeah, what you see in
Speaker 2: elements is often closer to what you need if the
Speaker 2: page is dynamic.
Speaker 1: Okay, perfect segway that covers static pages really well. But
Speaker 1: what about those more complex sites, the ones that are
Speaker 1: heavy on JavaScript where content loads dynamically as you scroll,
Speaker 1: or maybe they set cookies using JavaScript. Our requests and
Speaker 1: beautiful soup approach might just stop working there because they
Speaker 1: aren't actually running a browser. They're just fetching the initial
Speaker 1: HTML source.
Speaker 2: You hit the nail on the head. That's a huge
Speaker 2: challenge with modern web development. So many sites are JavaScript heavy.
Speaker 2: The initial HTML might be almost empty just to shell.
Speaker 2: The actual content gets fetched and rendered by JavaScript running
Speaker 2: in your browser, and sometimes that JavaScript is deliberately obfuscated,
Speaker 2: made hard to read to make reverse engineering it almost impossible.
Speaker 1: So you can't easily figure out where it's getting the
Speaker 1: data from exactly.
Speaker 2: Or maybe it sets a special cookie like a not
Speaker 2: a security token using JavaScript, and without that cookie you
Speaker 2: can't make further requests. So if requests can't run JavaScript,
Speaker 2: what do you do?
Speaker 1: And that's where I guess Selenium comes into the picture.
Speaker 1: It's more than just a scraper, isn't it. It's about
Speaker 1: browser automation precisely.
Speaker 2: Selenium's original purpose was actually for automated testing of websites. Yeah,
Speaker 2: making sure, buttons, work forms, submit, et cetera. But that
Speaker 2: makes it incredibly powerful for scraping because it literally drives
Speaker 2: the real web browser, Chrome, Firefox, whatever you can figure.
Speaker 1: So it can run the JavaScript.
Speaker 2: Yes, it loads the page, waits for things to appear,
Speaker 2: clicks buttons, fills in forms, scrolls down the page. Anything
Speaker 2: a human user can do, Selenium can automate.
Speaker 1: Ah. Okay, So for those sites where content loads as
Speaker 1: you scroll, like maybe infinite scrolling on social media or news.
Speaker 2: Sites, perfect example requests would only get the first patch,
Speaker 2: selem can actually perform the scroll action, wait for the
Speaker 2: new content to load because the JavaScript runs, and then
Speaker 2: grab it.
Speaker 1: That's clever. What about waiting Pages don't always load instantly, right?
Speaker 2: Selenium has tools for that too. You can use weights
Speaker 2: telling your script, hey, wait until this specific button is clickable,
Speaker 2: or wait until this piece of text appears before you
Speaker 2: try to interact with it. It makes your scraper much
Speaker 2: more robust against slow loading pages or dynamic elements.
Speaker 1: That sounds incredibly capable, but I imagine driving a full
Speaker 1: browser isn't as lightweight as just making a simple HTTP request.
Speaker 1: Is there a downside?
Speaker 2: Absolutely? There's a trade off. Selenium is significantly slower and
Speaker 2: uses way more memory and CPU resources than requests and
Speaker 2: beautiful soup.
Speaker 1: Because it's literally running Chrome in the.
Speaker 2: Background or something exactly. You're paying for that full browser emulation.
Speaker 2: So it's powerful, essential for those tricky dynamic sites. But
Speaker 2: you always want to check first, can I get this
Speaker 2: data with the simpler, faster request approach Uselen when you have.
Speaker 1: To, okay, makes sense, choose the right tool for the job.
Speaker 1: So let's say we've figured out how to scrape one page,
Speaker 1: maybe even a dynamic one with Selenium. How do we
Speaker 1: scale that up? How do we go from scraping a
Speaker 1: page to well, crawling hundreds or thousands across a whole website.
Speaker 1: That feels like a different beast.
Speaker 2: It is, And that distinction between scraping grabbing data from
Speaker 2: a specific page, and crawling, navigating link by link to
Speaker 2: discover and scrape many pages is really.
Speaker 1: Important, like what search engines do, but on a smaller scale.
Speaker 2: Maybe exactly they crawl the web constantly. For data science,
Speaker 2: if you need to crawl a site, you need a
Speaker 2: more structured approach. Best practices become vital. You'll almost certainly
Speaker 2: want a database, something simple like squilight is often fine.
Speaker 2: Maybe using a helper library like records to keep track
Speaker 2: of everything. What kind of thing, Well, you need a
Speaker 2: list of URLs you plan to visit the crawl frontier.
Speaker 2: You need a list of URLs you've already visited so
Speaker 2: you don't get stuck in loops or scrape the same
Speaker 2: page multiple times. And of course you need to store
Speaker 2: the data you extract. It's also really good practice to
Speaker 2: separate the logic. Have one part of your code responsible
Speaker 2: for finding new links, the crawler, and another part responsible
Speaker 2: for extracting data from a page the scraper makes it
Speaker 2: easier to manage, and.
Speaker 1: You have to be careful not to hammer the website
Speaker 1: absolutely critical.
Speaker 2: You need to build in delays or cool down periods
Speaker 2: between your requests, don't just fire them off as fast
Speaker 2: as possible. You also need air handling. What if a
Speaker 2: page is temporarily down, You need logic to retry later
Speaker 2: and thinking about doing things in peril can speed it up,
Speaker 2: but you have to be even more careful not to
Speaker 2: overload the server. Then it's a balancing act.
Speaker 1: And you mentioned some specific tools for handling URLs.
Speaker 2: Yeah, little things become important when crawling, like earlib dot
Speaker 2: parse dot earl join. Websites often use relative links like
Speaker 2: about us. Your crawler needs to correctly combine that with
Speaker 2: the base you RL get the full address, earl Join
Speaker 2: handles that reliably, and Earl's frag helps remove those fragment
Speaker 2: identifiers the bit after the hashtags. You don't accidentally crawl
Speaker 2: HTML church section one and PA html tag section two
Speaker 2: as if they were different pages.
Speaker 1: So why is this scaling up, this crawling capability so
Speaker 1: important for you our listeners doing data science? What doors
Speaker 1: does it open?
Speaker 2: It opens the door to data sets that just don't
Speaker 2: exist anywhere else or aren't available in a neat packaged format.
Speaker 2: The web is this enormous, constantly updated, incredibly rich source
Speaker 2: of well mostly unstructured.
Speaker 1: Data a real treasure trove if you can access it exactly?
Speaker 2: Imagine you want to build a sentiment analysis model for
Speaker 2: product reviews. You I need thousands, tens of thousands reviews.
Speaker 2: Where do you get them? You call e commerce sites?
Speaker 1: Well? Maybe tracking housing prices perfect.
Speaker 2: Example, collect real estate listings across a whole city or
Speaker 2: region for analysis or visualization. We've seen amazing projects born
Speaker 2: from this. Google Translate got massively better by using scrape
Speaker 2: texts from across the web. There was the Billion Prices
Speaker 2: project at MIT, which scraped online retailers daily to create
Speaker 2: near real time inflation tragging way faster than official government stats.
Speaker 1: Wow.
Speaker 2: Yeah. Or think about monitoring social media for mentions of
Speaker 2: bitcoin to gauge public sentiment, or analyzing job postings to
Speaker 2: see which data science skills are currently in demand. All
Speaker 2: rely on robust crawling. It's about turning the messy, sprawling
Speaker 2: web into structured, valuable information for your data science pipeline.
Speaker 1: Okay, so let's pull back a bit thinking about that
Speaker 1: data science pipeline, maybe using a framework like CRISPADM. Where
Speaker 1: does webscraping fit into the bigger picture?
Speaker 2: Good question. It primarily slots into the early phases data
Speaker 2: understanding and data.
Speaker 1: Preparation, finding and getting the data right.
Speaker 2: Specifically, it's often part of identified data sources, realizing the
Speaker 2: web is a potential source, and then select the data
Speaker 2: and actually collecting it. It's usually about enriching data sets
Speaker 2: you already have, or maybe creating a totally new data
Speaker 2: set from scratch using web data.
Speaker 1: But it's not just a technical task. Is that you
Speaker 1: mentioned managerial concerns?
Speaker 2: Yes, and this is often underestimated. There's this crucial gap
Speaker 2: between building a model using scrape data the model train
Speaker 2: phase and actually deploying that model where it needs ongoing
Speaker 2: scrape data to work the model run phase.
Speaker 1: Ah, because the website might change exactly.
Speaker 2: Websites change all the time, layouts change, HTML structure changes,
Speaker 2: login processes change. A scraper that works perfectly today might
Speaker 2: break tomorrow.
Speaker 1: That warning, so your production model suddenly stops working because
Speaker 1: it's data.
Speaker 2: Feed broke precisely, which means web scrapers require ongoing maintenance.
Speaker 2: Someone has to monitor them, fix them when they break.
Speaker 2: That's real cost, and that's why the golden rule. The
Speaker 2: first piece of advice is always look for an official API.
Speaker 1: First application programming interface, a structured way for programs to
Speaker 1: get data.
Speaker 2: Right, If the website offers an API and it provides
Speaker 2: the data you need and the terms are acceptable, maybe
Speaker 2: it's free or reasonably priced, use the API. It's almost
Speaker 2: always going to be more stable, more reliable, and less
Speaker 2: likely to break than a custom scraper you build yourself.
Speaker 1: Really solid advice. But what if there isn't an API,
Speaker 1: or maybe the EPI exists but it's I don't know,
Speaker 1: super limited in how many requests you can make, or
Speaker 1: it just doesn't have that one specific piece of data
Speaker 1: you absolutely need. When does building the scraper become worth
Speaker 1: the hassle?
Speaker 2: That's the judgment call, isn't it. It's a trade off.
Speaker 2: If the API doesn't cut it for whatever reason, cost
Speaker 2: rate limits, missing data fields, then yeah, building and maintaining
Speaker 2: a scraper might be your only option.
Speaker 1: So you weigh the development and maintenance effort against the
Speaker 1: value of the data exactly.
Speaker 2: But you go into it with your eyes open knowing
Speaker 2: it's likely going to require ongoing work. Is that cat
Speaker 2: and mouse game people talk about. Websites might actively try
Speaker 2: to block scraper, so you might need to adapt your
Speaker 2: techniques constantly.
Speaker 1: And speaking of blocking and well, potential conflicts. The legal
Speaker 1: side of this you mentioned it's complex. It sounds like
Speaker 1: it's not just a technical decision, but a legal and
Speaker 1: ethical one too.
Speaker 2: Absolutely, it's murky waters. Legally speaking, there isn't one single
Speaker 2: law that says webs scraping is legal or web scraping
Speaker 2: is illegal. It depends. Several legal arguments tend to pop
Speaker 2: up in court cases, at least in the US, like what, well,
Speaker 2: there's breach of terms and conditions. If a website's terms
Speaker 2: of service explicitly forbids scraping and you clicked I accept somewhere,
Speaker 2: they might have a case. We saw that with Ryanair
Speaker 2: winning against a flight data scraper.
Speaker 1: Okay, so read the terms definitely.
Speaker 2: Then there's copyright infringement. Is the data itself copyrighted? Usually
Speaker 2: facts aren't, but the presentation might be. The fair use
Speaker 2: doctrine often gets debated here. Think about Google book scanning
Speaker 2: millions of books, lots of legal wrangling there. There's also
Speaker 2: the CFAA, the Computer Fraud and Abuse Act. It's meant
Speaker 2: to target hacking unauthorized access. Sometimes companies try to argue
Speaker 2: that scraping constitutes unauthorized access, especially if you bypass technical.
Speaker 1: Barriers HM that seems like a stretch for public data.
Speaker 2: Courts has struggled with it. There's also older concepts like
Speaker 2: trespass to chattels, basically arguing your scraper is interfering with
Speaker 2: their server resources, especially if you overload it. And then
Speaker 2: there's the robots dot txt file.
Speaker 1: Right, the file that tells bots where they shouldn't go.
Speaker 2: Yeah, it's not strictly legally binding in most cases, but
Speaker 2: ignoring it is definitely not playing nice and signals you're
Speaker 2: disregarding the site owner's wishes. It could be used as
Speaker 2: evidence against you.
Speaker 1: You mentioned a specific case earlier, hi q Labs versus LinkedIn,
Speaker 1: that seemed pretty important for this whole public data question.
Speaker 2: It was, yeah, a really significant case high q Labs
Speaker 2: was scraping data from public LinkedIn profiles. LinkedIn tried to
Speaker 2: stop them technologically and legally, invoking the CFAA.
Speaker 1: And what did the courts say?
Speaker 2: The courts, particularly the Ninth Circuit, basically ruled that scraping
Speaker 2: publicly accessible data, even if the site tries to block
Speaker 2: you with technical measures or says not to in its terms,
Speaker 2: doesn't necessarily violate the cfaas without authorization clause. The key
Speaker 2: was that the data was already opened.
Speaker 1: To the public, So if it's public, maybe it's fair game.
Speaker 2: It leans that way, but it's not a blanket permission slip.
Speaker 2: It highlighted just how blurry the lines are around public
Speaker 2: information and unauthorized access when it comes to the web.
Speaker 2: The legal landscape is definitely still evolving.
Speaker 1: Okay, So with all that complexity technical, ethical, legal, what's
Speaker 1: the takeaway for you, our listener? What are your core
Speaker 1: responsibilities when you decide to scrape data. What's the baseline
Speaker 1: for being a good digital citizen.
Speaker 2: The absolute number one rule is play nice, be respectful,
Speaker 2: don't bombard a website with requests. Think about the impact
Speaker 2: you're having.
Speaker 1: Don't be the reason their site goes down exactly.
Speaker 2: That can cause real financial damage, and that's when legal
Speaker 2: action becomes much more likely. We saw qvc SEW a
Speaker 2: company called resultantly claiming excessive scraping caused outages costing millions.
Speaker 2: You don't want to be that person, so throttle your requests,
Speaker 2: put delays in, Identify yourself with a proper user agent
Speaker 2: header if you can, maybe even include contact info trans
Speaker 2: if appropriate. Yes, always always check the robots dot com
Speaker 2: txt file and the terms of service first, see what
Speaker 2: the site owner explicitly asks for or forbids, and if
Speaker 2: you can, the absolute safest route is to get written
Speaker 2: permission from the website owner.
Speaker 1: That might not always be practical, but it's the ideal.
Speaker 2: It's the gold standard. Yeah, and just pause and think
Speaker 2: is this data truly intended to be public and consumed
Speaker 2: in this way? Or am I accessing something private or
Speaker 2: trying to circumvent a system using common sense and acting
Speaker 2: ethically is just as important as writing clever code.
Speaker 1: Wow, Okay, that was definitely a deep dive. We've gone
Speaker 1: from the basics of HTTP.
Speaker 2: To parsing htmail with beautiful soup.
Speaker 1: Tackling tricky JavaScript with Selenia.
Speaker 2: You're scaling up with crawling techniques.
Speaker 1: I'm wrestling with those really crucial management and legal questions.
Speaker 1: I think you listening should now have a much clearer
Speaker 1: picture not just of the how of webscraping, but the
Speaker 1: really important why and when and when not to.
Speaker 2: Yeah, it's about using this powerful tool effectively but also responsibly.
Speaker 1: Absolutely, So here's a final thought to leave you with
Speaker 1: something to chew on. We've talked about this cat and
Speaker 1: mouse game between scrapers and websites and the shifting legal sands.
Speaker 1: Considering how fast things like AI are evolving and maybe
Speaker 1: new techniques for hiding or protecting data online, how might
Speaker 1: our very definition of publicly available data change in the
Speaker 1: next few years.
Speaker 2: That's a big question, right.
Speaker 1: And what could that changing definition mean for how we
Speaker 1: gather data, how we do analysis, and maybe even the
Speaker 1: kinds of innovation that are possible across well pretty much
Speaker 1: every field.
Speaker 2: Yeah, what does public mean when data is generated by
Speaker 2: AI or locked behind complex interactions, Lots to think about.
Speaker 1: There definitely something to ponder. We'll leave it there for
Speaker 1: this deep dive