Speaker 1: Okay, welcome to the deep dive. Today. We're jumping into webscraping.
Speaker 2: That's right, specifically using PhD based on some key ideas
Speaker 2: from roller source Instant PHPE Webscraping.
Speaker 1: Yeah, it's from twenty thirteen by Jacob Board. So things
Speaker 1: have moved on obviously, but the core ideas they're often
Speaker 1: still relevant, aren't.
Speaker 2: They They really are. The book was aimed at beginners,
Speaker 2: you know, showing how to programmatically crawl websites, download content,
Speaker 2: and basically turn unstructured web stuff into structured data using
Speaker 2: using PHP.
Speaker 1: So our mission here is to pull out those fundamental
Speaker 1: techniques from these excerpts give you a solid grounding in
Speaker 1: how PHPU webscraping works at its core.
Speaker 2: Even if you're maybe adapting these ideas for more modern
Speaker 2: sites later on, the basics often carry.
Speaker 1: Through exactly the source assumes. Maybe not a ton of
Speaker 1: programming experience, though knowing some PHP and HTML definitely.
Speaker 2: Helps, sure, But the focus is really on the scraping
Speaker 2: concepts themselves.
Speaker 1: All right, let's kick things off. Before you scrape, you
Speaker 1: need your tools. What's the basic toolkit according to these sources?
Speaker 2: Okay, so first, obviously you need PHP itself that's the language, right,
Speaker 2: then a good place to write your code an ID
Speaker 2: integrated development environment. The source mentions Eclipse PDT.
Speaker 1: PDT being the PHP development tools for Eclipse, so a
Speaker 1: specialized code editor.
Speaker 2: Yeah. Basically makes coding easier, keeps things organized. And then
Speaker 2: you need a way to run the PHP and.
Speaker 1: Probably a database too, like a local server setup exactly.
Speaker 2: The source recommends xm yp It bundles a patche which
Speaker 2: is the web server, PHP and myseqel the database all
Speaker 2: in one package.
Speaker 1: Ah. Convenient avoids installing everything separately.
Speaker 2: Yeah, and it even includes php I admin oh.
Speaker 1: Right for managing the myseql database visually useful later definitely Okay,
Speaker 1: So you install XAMPP maybe Eclipse any specific setup tweaks needed.
Speaker 2: A couple of key things. The source points out. Setting
Speaker 2: your PHP path variable is good practice. Lets you run
Speaker 2: PHP scripts easily from the command line for testing and stuff, right,
Speaker 2: But the really critical one for scraping is enabling the CURL.
Speaker 1: Extension curl Okay, what is that? Exactly?
Speaker 2: It's a PHP library. You need it enabled in your
Speaker 2: main PHP can fig file the PHP dot i ne
Speaker 2: Without it, your PHP script can't really make web requests easily.
Speaker 1: Ah, So it's essential for fetching pages programmatically.
Speaker 2: Absolutely, And then you know, just test this setup, make
Speaker 2: sure a patche runs. Maybe run a simple finfo script
Speaker 2: to see if curl is listed as enabled.
Speaker 1: Got it, so tool get ready curl enabled. Now the
Speaker 1: first actual step in scraping getting the web page.
Speaker 2: Fetching the content. Yeah, this is where CRL comes into
Speaker 2: play directly.
Speaker 1: Because it handles HTTP requests.
Speaker 2: Exactly unless your script act like a browser, essentially sending
Speaker 2: a request to a URL and getting back the HTML
Speaker 2: source code.
Speaker 1: And the source provides a function example curl get. What's
Speaker 1: the basic flow there, it's pretty logical.
Speaker 2: You initialize a CRL session, think of it as opening
Speaker 2: a connection channel. Okay, Then you set options for that session,
Speaker 2: tell it what you want to.
Speaker 1: Do, like the URL you want to fetch.
Speaker 2: Curl opterill is the main one, and critically kurl opter
Speaker 2: own transfer. You usually want that set to true. Why
Speaker 2: is that so? That's url returns the page content as
Speaker 2: a string variable in your PHP script instead of just
Speaker 2: like printing it straight to the screen. You need it
Speaker 2: as a variable to work with.
Speaker 1: It, right, makes sense? Any other key options?
Speaker 2: Oh yeah, curl opc fallow location is super.
Speaker 1: Useful for redirects like three ozho One's exactly.
Speaker 2: Websites often redirect you. This option tells the URL to
Speaker 2: automatically follow those redirects to the final page. Saves you
Speaker 2: a lot to hassle. Nice.
Speaker 1: What about curl op twuser agent. The source gives them
Speaker 1: an example string ah.
Speaker 2: The user agent. It's basically a string that identifies your
Speaker 2: client your script to the web server one well, partly politeness,
Speaker 2: partly necessity. Some servers block requests that don't have a
Speaker 2: user agent string that looks like it's from a normal
Speaker 2: web browser, so sending one makes your script look less
Speaker 2: like a basic.
Speaker 1: Bot okay, helps avoid immediate blocks potentially.
Speaker 2: Yeah. Then there's curl optt header if you need to
Speaker 2: send custom headers sometimes needed for specific sites and curl
Speaker 2: up tfell on error.
Speaker 1: What does that do?
Speaker 2: It tells curl to treat HDP air codes like four
Speaker 2: O four not found or five hundred server error as
Speaker 2: well actual.
Speaker 1: Script errors instead of just returning an empty page or
Speaker 1: an air page.
Speaker 2: Right, it can be a simple way to detect if
Speaker 2: the request failed badly okay.
Speaker 1: And the source mentions checking the HTTP response code itself.
Speaker 1: Using curl jet info.
Speaker 2: Why bother because knowing the code tells you exactly what happened.
Speaker 2: Two hundred oka means success, you have the page.
Speaker 1: Four oh four means it doesn't exist, right crucial info
Speaker 1: A four h three forbidden means you don't have permission.
Speaker 2: Maybe you need to log in or your IP is blocked.
Speaker 1: Three oh one moved permanently, which follow location handles, but
Speaker 1: good to know.
Speaker 2: Yeah, checking the status code is fundamental for robust error handling.
Speaker 2: You know why something might have failed.
Speaker 1: Okay, so CRL gets you the raw HTML, maybe a
Speaker 1: massive string of code. Now the real challenge finding the
Speaker 1: specific bit of data you want inside all that.
Speaker 2: Extraction time and the main tool the source introduces here
Speaker 2: is XPath XPath.
Speaker 1: I've heard of it with XML. How does it apply
Speaker 1: to HTML?
Speaker 2: Well, HTML isn't always perfect XML, but it's structured right
Speaker 2: with tags and attributes. You can parts that downloaded HTML
Speaker 2: string into something called a DOM a document object model.
Speaker 1: A tree structure of the page.
Speaker 2: Precisely, an XPath is a language specifically for navigating that
Speaker 2: tree and selecting nodes elements attributes text based on their
Speaker 2: path or characteristics.
Speaker 1: So it's more structured than just like searching for keywords
Speaker 1: in the string much more.
Speaker 2: The source shows a function return XPath object. It basically
Speaker 2: takes the HTML strength.
Speaker 1: So when you got from crl right, it.
Speaker 2: Uses PHPs built in don document class to load that HTML,
Speaker 2: even if it's a bit messy.
Speaker 1: I see the source us as an AT symbol before
Speaker 1: load HTML. Is that related to MESSYHTML?
Speaker 2: It is? Real world HTML often have minor errors. The
Speaker 2: AT symbol in PHP suppresses warnings that load HTML might
Speaker 2: generate because of that imperfect markup. It stops your script
Speaker 2: potentially halting on minor issues.
Speaker 1: Ah a practical trick for scraping. Okay, so don document
Speaker 1: lugs the hhamel than what.
Speaker 2: Then you create a dom XPath object from that DOM document,
Speaker 2: and that XPath object is what you use to run
Speaker 2: your queries.
Speaker 1: Okay queries. The source has examples like h one or
Speaker 1: span at class some class exactly.
Speaker 2: Those are XPath expressions. H one means find any H
Speaker 2: one element anywhere in the document.
Speaker 1: And the span it class.
Speaker 2: That's more specific, find any span element that has an
Speaker 2: attribute named class with the exact value some class.
Speaker 1: What about at href at.
Speaker 2: The end of one example that's selecting an attribute, so
Speaker 2: maybe it found a specific link attag and added to
Speaker 2: rev says get the value of its h ref attribute
Speaker 2: the url itself.
Speaker 1: So you run these queries against the XPath object and it.
Speaker 2: Gives you back a list of matching nodes, elements.
Speaker 1: Or attributes, and then the source shows item zero node
Speaker 1: value to get the actual text.
Speaker 2: Right, the query might find multiple matches, so item zero
Speaker 2: usually gets the first one in the list. Then node
Speaker 2: value extracts the text content from inside that element.
Speaker 1: Okay, so XPath is powerful for navigating that structure.
Speaker 2: Very The source has a table with common expressions eight
Speaker 2: headed taro apro using brackets for conditions. That's your vocabulary
Speaker 2: for building these queries.
Speaker 1: What if the data isn't neatly inside a tag or
Speaker 1: the structure's just chaotic, XPath might not work then.
Speaker 2: Exactly, Sometimes XPath is overkill or just plain impossible. That's
Speaker 2: where as the source shows, you might need more direct
Speaker 2: approach custom functions.
Speaker 1: Like the screen between function mentioned.
Speaker 2: Perfect example, it's much simpler conceptually. Its whole job is
Speaker 2: to find a chunk of text that sits between two
Speaker 2: other known unique strings, so you.
Speaker 1: Don't care about HTML tags, just find the text after
Speaker 1: start marker and before end marker.
Speaker 2: Precisely, you give it the whole chunk of text, like
Speaker 2: the page source, the starting string and the ending string.
Speaker 1: How does it work.
Speaker 2: It uses basic PHP string functions stripos to find the
Speaker 2: position of the start and end markers, then subscripted to
Speaker 2: cut out the piece of the string between those positions.
Speaker 1: Simple but effective if the markers are reliable. The source
Speaker 1: uses scraping a Google Analytics ID as an example.
Speaker 2: Yeah, that ID is often embedded in JavaScript between specific
Speaker 2: quote marks or function calls. XPath wouldn't easily grab that,
Speaker 2: but scrape between works perfectly.
Speaker 1: Okay, so we have text extraction covered with XPath and
Speaker 1: custom functions. What about non text content images?
Speaker 2: Good question. You often need to grab images too. The
Speaker 2: process combines things we've discussed how So, First you usually
Speaker 2: find the images url using XPath. You look for an
Speaker 2: mg tag and grab its src attribute.
Speaker 1: So like mng at src exactly.
Speaker 2: That gives you the URL of the image file.
Speaker 1: Then you use currl again.
Speaker 2: Yep, you use your curl git function or similar to
Speaker 2: download the content at that image url.
Speaker 1: But this time you're expecting image data, not HDML.
Speaker 2: Right, binary data and the source suggests a good practice.
Speaker 2: Verify it actually is an image before saving it. How
Speaker 2: PHP has a function get image. You can pass it
Speaker 2: to the downloaded data or the filepath. If you save
Speaker 2: it temporarily, it'll return image dimensions if it's valid or
Speaker 2: false if it's not a recognized image type.
Speaker 1: Smart So you verify it and then then you.
Speaker 2: Just use standard PHP file functions. So open to open
Speaker 2: a local file for writing, right to write the image
Speaker 2: data you got from CURL into it, and F close
Speaker 2: to close.
Speaker 1: The file, and you've saved the image locally.
Speaker 2: You have and that basic method find url, download with CRL,
Speaker 2: save with file functions works for other file types too,
Speaker 2: like pds or whatever.
Speaker 1: Okay, fetching static stuff in images is one thing, but
Speaker 1: lots of data is behind logins or search forms. Yeah,
Speaker 1: how do you interact with sites like that?
Speaker 2: Yeah? This requires simulating form submissions. Forms often use the
Speaker 2: HTTP post method to send data.
Speaker 1: So you need to make post requests with curl exactly.
Speaker 2: The source shows a CURL post function example for this.
Speaker 1: What do you need to know to make that POC
Speaker 1: request work?
Speaker 2: You have to inspect the HTML form on the actual
Speaker 2: web page. First, look for the the form tag itself.
Speaker 2: You need its action attribute. That's the URL you send
Speaker 2: the post request to.
Speaker 1: Okay, the destination you arel yep.
Speaker 2: Then you need to find all the input elements inside
Speaker 2: that form and select or text area too potentially what
Speaker 2: about them? You need their name attributes. Those names become
Speaker 2: the keys in the data you send, and you need
Speaker 2: the value you want to send for each name.
Speaker 1: So if there's an input name username, you send your username.
Speaker 2: Right And crucially don't forget hidden input fields. They often
Speaker 2: contain important stuff like session tokens or form IDs that
Speaker 2: the server expects back. The source login example mentions needing
Speaker 2: email password, but also destination and format which might be
Speaker 2: hidden fields.
Speaker 1: Ah, I got to check the source carefully. What about
Speaker 1: login specifically? Don't they involve cookies?
Speaker 2: Absolutely vital When you log in successfully, the server usually
Speaker 2: sends back cookies to track your session for subsequent request
Speaker 2: or restricted pages. You need to send those cookies back.
Speaker 1: How does Kira r L handle that?
Speaker 2: It has options for it? Curl up pokie jar tells
Speaker 2: Curl to save cookies receives into a specified file. Cookie
Speaker 2: file tells Curl to read cookies from a file and
Speaker 2: send them with the request.
Speaker 1: So you log in, save the cookies, and then use
Speaker 1: those cookies for future requests to stay logged in.
Speaker 2: That's the basic idea. It maintains your session state.
Speaker 1: The source also mentions posting files like simulating and upload.
Speaker 2: Yeah. If a form has an input type file, you
Speaker 2: can simulate uploading a file using curl ob post fields.
Speaker 2: You set the value for that field name to the
Speaker 2: path of your local file, but you prefix the path
Speaker 2: with an AT symbol.
Speaker 1: Crrol understands the AT means upload this file correct.
Speaker 2: It handles reading the file content and sending it appropriately.
Speaker 1: Okay, so you've send the PST request, maybe logged in.
Speaker 1: How do you know if it actually worked.
Speaker 2: The simplest check shown in the source is just to
Speaker 2: look for a specific piece of text in the HTML
Speaker 2: response that you know only appears after a successful submission.
Speaker 1: Like log in, successful or welcome back user exactly.
Speaker 2: You get the response page source from curl and search
Speaker 2: that string for your success message. If it's there, it probably.
Speaker 1: Worked, right. Okay, single pages forms, but the real power
Speaker 1: comes from scraping lots of pages, like product listings or
Speaker 1: search results that span multiple pages. How do you handle
Speaker 1: that pagination?
Speaker 2: Yeah, traversing multiple pages. You start by scraping the first
Speaker 2: page as.
Speaker 1: Usual, get the data, find the image, whatever, right, But.
Speaker 2: While you're doing that, you also use XPath to look
Speaker 2: for the link to the next page.
Speaker 1: Like in a next button or page number link exactly.
Speaker 2: Commonplaces are l elements with class pagination or pager. You'd
Speaker 2: write an XPath query to find a tag inside that,
Speaker 2: maybe specifically the one with text next, and grab its
Speaker 2: href attribute.
Speaker 1: So you get the URL for page two yep.
Speaker 2: Then you scrape page two, and on page two you
Speaker 2: look for the link to page three, and so on.
Speaker 2: You basically collect a list of all the page URLs
Speaker 2: you need to visit.
Speaker 1: Make sure they're full URLs, not relative ones.
Speaker 2: Good point. If they're relative links like page two, you
Speaker 2: need to prepend the base website url to make them
Speaker 2: absolute before fetching with curl.
Speaker 1: Then you just loop through your list of URLs, scraping
Speaker 1: each one.
Speaker 2: Pretty much. I'll fetch page, extract data, fetch next page,
Speaker 2: extract data, repeat.
Speaker 1: Now this sounds like it could hit the server pretty
Speaker 1: fast if you have hundreds of pages.
Speaker 2: It absolutely can, and that brings up a really critical point.
Speaker 2: The source emphasizes politeness.
Speaker 1: Right, don't be a nuisance or worse.
Speaker 2: Get yourself blocked. Hammering a server with rapid fire requests
Speaker 2: is bad form and often triggers automated defenses.
Speaker 1: So how do you be polite?
Speaker 2: The simplest, most common way shown is to just pause
Speaker 2: between requests. Use PHP's sleep function. The sort suggests sleep
Speaker 2: rand one three.
Speaker 1: Way, a random one to three seconds between fetching each page.
Speaker 2: Yeah, it slows your script down, mimics human browsing speed
Speaker 2: a bit more, and drastically reduces the load on their server.
Speaker 2: It's essential for any non trivial scraping.
Speaker 1: Okay, vital tip. So you've scraped politely across many pages,
Speaker 1: extracted tons of data. Where does it all go? Printing
Speaker 1: to screen is useless?
Speaker 2: Now right, you need persistent storage. The obvious choice demonstrated
Speaker 2: is a database. Since XMPP includes my sequel, that's the
Speaker 2: example used.
Speaker 1: First up is setting up the database table.
Speaker 2: Yep, you need to design your table structure. Now define
Speaker 2: columns that match the data points are scraping, like book title, author,
Speaker 2: release date is SBN, etc.
Speaker 1: And you can use food pi admin for that.
Speaker 2: It's a handi graphical tool for creating the database and tables,
Speaker 2: setting data types, all that stuff.
Speaker 1: Okay, tables ready. How does the PHP script connect and
Speaker 1: insert the scraped data?
Speaker 2: The source uses PDOPHP data objects. It's a standard flexible
Speaker 2: way in PHP to talk to databases, including my school.
Speaker 2: You establish a connection using your database name.
Speaker 1: Username, password, then insert the data.
Speaker 2: For inserting lots of items, the best practice shown is
Speaker 2: using prepared statements.
Speaker 1: Why is that better than just building insert strings?
Speaker 2: Two main reasons. Security, It prevents SEQL injection vulnerabilities, and
Speaker 2: often better performance when you're inserting many rows with the
Speaker 2: same structure.
Speaker 1: How do they work.
Speaker 2: You write the insert query once, but use placeholders like
Speaker 2: question marks are named parameters for the actual values. You
Speaker 2: prepare this query structure with the database okay, Then you
Speaker 2: loop through your array of scraped data items like all
Speaker 2: the books you found inside the loop. For each book,
Speaker 2: you bind its specific title, author, etc. To the placeholders
Speaker 2: in the prepared statement, and then you execute it.
Speaker 1: So the queer structure is set once and then just
Speaker 1: the data changes for each execution exactly.
Speaker 2: It's cleaner and safer. One row gets inserted into your
Speaker 2: database table for each item in your scrape data.
Speaker 1: Array, and the source also shows getting data back out.
Speaker 2: Yeah, completes the picture using a select query, again often
Speaker 2: via pdo, to fetch the data you saved, perhaps looping
Speaker 2: through the results to display them in an HTML table
Speaker 2: on a web page.
Speaker 1: Okay, this is getting quite sophisticated fetching, parsing, interacting, storing.
Speaker 1: As scripts get bigger, the code could get messy, right,
Speaker 1: repeating the same CURL setup or XPath.
Speaker 2: Creation definitely can. That's where the source introduces object oriented
Speaker 2: programming or OOP principles as a way to organize things better.
Speaker 1: Making the code reusable and tidier.
Speaker 2: That's the goal. The core idea is creating a class,
Speaker 2: which is like a blueprint for an object. It bundles
Speaker 2: together related data properties and functions that operate on that
Speaker 2: data methods.
Speaker 1: The book uses a human class analogy.
Speaker 2: Yeah, like a human blueprint might define properties like name
Speaker 2: and age, and methods like speak or walk. An object
Speaker 2: is a specific instance created from that blueprint, like Bobby
Speaker 2: eesyl new human, where Bob has his own specific name
Speaker 2: in age.
Speaker 1: So how does the example scrape class in the source
Speaker 1: apply this?
Speaker 2: It takes the common scraping tasks fetching a URL, creating
Speaker 2: the XPath object and put some inside the class definition
Speaker 2: as methods exactly, and it often uses a special method
Speaker 2: called construct What's that do? The constructor runs automatically whenever
Speaker 2: you create a new object from the class. So in
Speaker 2: the scrape class, example, when you write page craper new
Speaker 2: scrape http dot example dot com, the struct method immediately
Speaker 2: takes that.
Speaker 1: URL, the one you just passed in, calls.
Speaker 2: An internal method maybe curl get to fetch the source
Speaker 2: code for that specific URL, and calls another internal method
Speaker 2: like return XPath object to create the XPath object for
Speaker 2: that source. It does the initial setup.
Speaker 1: Work ah, so the object is immediately ready with the
Speaker 1: source and the XPath tool for its specific URL precisely.
Speaker 2: The paid scraper object now holds its own source code
Speaker 2: property and six path object property ready for you to use.
Speaker 2: You'd access them like pagecap er, XPath object query.
Speaker 1: And you could add more methods to the scrape class
Speaker 1: like save image or submit form.
Speaker 2: Absolutely you build up a reusable toolkit within the class.
Speaker 2: Create a scrape object for any URL, and you have
Speaker 2: all your scraping tools ready to work on that specific
Speaker 2: pages content. Makes the main part of your script much cleaner, very.
Speaker 1: Neat okay, final piece automation. You've built this great script,
Speaker 1: maybe using a class it saves to a database. How
Speaker 1: do you make it run automatically? Say every night?
Speaker 2: Yeah, you don't want to manually run it every time.
Speaker 2: This is where scheduling comes in. The source gives an
Speaker 2: example using Windows task scheduler.
Speaker 1: So it's not code within the PHP script itself.
Speaker 2: No, Usually you leverage the operating system scheduling tools Task
Speaker 2: scheduler on Windows, or chron jobs on Linux or Maco.
Speaker 2: They are built for exactly this purpose.
Speaker 1: How does it work?
Speaker 2: Basically, you can figure a task in the scheduler. You
Speaker 2: tell it the scheduled daily at three am for instance. Okay,
Speaker 2: you tell the action to perform. The action is essentially
Speaker 2: run the PHP program and tell it to execute your
Speaker 2: specific scraping script file.
Speaker 1: So you point it to php dot ex and then
Speaker 1: your my scraper not PHP file exactly.
Speaker 2: You provide the full paths. When the schedule time arrives,
Speaker 2: the OS runs php Php execute your script and hopefully
Speaker 2: your database gets updated with fresh data and it just.
Speaker 1: Runs in the background without you doing anything.
Speaker 2: That's the beauty of it. True automation. Wow.
Speaker 1: Okay, so we've really gone from set up getting PHP EXMPP,
Speaker 1: enabling CURL all the way through fetching pages, dealing with readerrects,
Speaker 1: user agents.
Speaker 2: Extracting data with XPath for structure or custom functions like
Speaker 2: scrap between for trickier bits, handling.
Speaker 1: Images, simulating form posts, managing cookies for logins, traversing pagization
Speaker 1: politely with delays.
Speaker 2: Saving the results into a my school database using PDO
Speaker 2: and prepared statements, organizing the code using op with classes.
Speaker 1: And finally automating the whole thing with task scheduler or coronate.
Speaker 1: That's quite the journey.
Speaker 2: It really covers the fundamental life cycle of a web
Speaker 2: scripting PASK based on these source excerpts. Even though the
Speaker 2: web is way more complex now, especially with JavaScript rendering content.
Speaker 1: Right, that's a whole other challenge it is, but these.
Speaker 2: Core concepts fetching HTTP content, parsing structured or semi structured data,
Speaker 2: handling sessions, storing results, they still form the foundation. You
Speaker 2: might add tools like headless browsers to handle JavaScript on top,
Speaker 2: but you still need to understand these basics.
Speaker 1: That's a great takeaway. The principles are enduring even if
Speaker 1: the tools evolve, so thinking about these building blocks how
Speaker 1: might you combine them for something more complex, or maybe
Speaker 1: how does that politeness principle the sleep delay become even
Speaker 1: more important, maybe even in ethical consideration when you think
Speaker 1: about scraping at a really large scale, what's the most
Speaker 1: interesting challenge you could tackle just starting with these foundational
Speaker 1: ideas