Speaker 1: Imagine it is, I don't know, like four point five
Speaker 1: to five pm on a Friday.
Speaker 2: The worst possible time, right.
Speaker 1: At the absolute worst, and your boss drops by your
Speaker 1: desk and hands you what sounds like an incredibly simple assignment.
Speaker 1: They're like, hey, I need you to go to the
Speaker 1: Yahoo Finance technology page and just pull the data for
Speaker 1: every single tech stock listed in their main table.
Speaker 3: It is a quick favor before the weekend, exactly.
Speaker 1: And for every single company you need to grab four
Speaker 1: specific things. You need the company name, the intra day
Speaker 1: stock price, the stock symbol, and then the actual URL
Speaker 1: that links to the company's full profile.
Speaker 2: I mean, on the surface, that sounds like a five
Speaker 2: minute task. You know, you open a spreadsheet, you open
Speaker 2: your web browser, and you just get to.
Speaker 1: Work, right, But then reality actually sets in because you're
Speaker 1: highlighting the first company's name, you're hitting copy, tabbing over
Speaker 1: to your spreadsheet, hitting paste. Yeah, then you're tabbing back
Speaker 1: highlighting the price, copy tab paste, And you know, doing
Speaker 1: that for one company is completely fine, but doing it
Speaker 1: for a company, well, it's brutal. You are making hundreds
Speaker 1: of highly repetitive, like precision clicks, Your risk starts aching,
Speaker 1: your eyes are completely glazing over, and inevitably you start
Speaker 1: making mistakes. You paste a price into the symbol column
Speaker 1: by accident.
Speaker 2: Which throws off the whole sheet exactly.
Speaker 1: It is the kind of mind numbing, soul crushing manual
Speaker 1: data entry that really makes you question your career choices.
Speaker 2: It really does. But you know, humans are just fundamentally
Speaker 2: unsuited for repetitive, high precision tasks over long durations. We
Speaker 2: just lose focus. Computers however, I mean they thrive on
Speaker 2: that exact type of work.
Speaker 1: You don't get bored, No, not at all.
Speaker 2: And this is the precise friction point where programming becomes
Speaker 2: frankly practically.
Speaker 1: Magic, which perfectly brings us to our mission for this
Speaker 1: deep dive today. We're going to build a mental blueprint
Speaker 1: of an automated data extraction pipeline, and we're looking really
Speaker 1: closely at a Python framework called screepy.
Speaker 3: Oh yeah, Scrapy is amazing for this, right.
Speaker 1: It basically serves as the ultimate engine for this kind
Speaker 1: of web scraping.
Speaker 3: It really is a powerhouse.
Speaker 2: And you know, our goal here for everyone listening isn't
Speaker 2: just to talk about writing a quick duty script that
Speaker 2: haphazardly yanks text off a page and calls.
Speaker 3: It a day.
Speaker 1: Right, we want to go deeper.
Speaker 3: Exactly, we are.
Speaker 2: Going to architect a robust, professional grade data pipeline. So
Speaker 2: we will start by examining how to test raw code
Speaker 2: in an isolated sandbox, figure out how to scale that
Speaker 2: up to crawl entire pages, and eventually build a system
Speaker 2: that automatically cleans and formats the data before saving it
Speaker 2: into a pristine spreadsheet.
Speaker 1: Okay, let's unpack this. Because whether you are a developer
Speaker 1: prepping for a major data project, or you know, maybe
Speaker 1: a self taught code or trying to automate your own life,
Speaker 1: or just someone curious about how data actually moves around
Speaker 1: the Internet, understanding this architecture gives you a whole new superpower.
Speaker 1: So where do we even begin? Because if I'm staring
Speaker 1: at that Yahoo Finance page, my first instinct is to just,
Speaker 1: I don't know, write a man subscript and hope it
Speaker 1: grabs the right number.
Speaker 2: Yeah, and plunging straight into writing a massive script is
Speaker 2: the fastest way to get completely.
Speaker 1: Overwhelmed, I can imagine.
Speaker 2: Because before building a factory, you need to test if
Speaker 2: the raw materials are even usable. You need a testing
Speaker 2: ground to see if the data you want is accessible
Speaker 2: in the way you think it is, and in the
Speaker 2: scrape ecosystem, this isolated environment is called the scrape shell.
Speaker 1: The screepy shell. Okay, so I'm picturing like a digital sandbox.
Speaker 3: That's a great way to look at it.
Speaker 1: So you boot up this shell in your computer's terminal,
Speaker 1: and you feed it the specific web address, like our
Speaker 1: Yaku finance page, and it basically goes out, grabs the
Speaker 1: underlying HTML of that page and downloads it right into
Speaker 1: your sandbox exactly.
Speaker 3: It pulls the raw code.
Speaker 1: So now you can experiment with extracting the data without
Speaker 1: the pressure of running a full automated program that might
Speaker 1: break at any second.
Speaker 2: Right, the sandbox isolates the problem.
Speaker 3: You can just tinker.
Speaker 2: But once you have the page data in there, you
Speaker 2: need a designated place to actually put the specific fragments
Speaker 2: of information you plan to extract. Okay, So the source
Speaker 2: material introduces a really foundational concept here called a.
Speaker 1: Scrapy item, Okay, an item. The way the documentation describes
Speaker 1: this item, it actually reminds me of a bento box.
Speaker 3: Oh I like that.
Speaker 1: Yeah, because you have this container, but you don't just
Speaker 1: dump all your food into one big, messy pile. You
Speaker 1: have very specific pre labeled compartments. A compartment for the rice,
Speaker 1: a compartment for the fish.
Speaker 3: Right, it's structured.
Speaker 1: Exactly in Python terms. It's like a highly structured dictionary.
Speaker 1: We define our compartments or feels basically up front. So
Speaker 1: we create a label for company name, one for inter
Speaker 1: day price, one for symbol, and one for link.
Speaker 2: The Bento box analogy perfectly captures the strict structural nature
Speaker 2: of the item based class. When you first instantiate this
Speaker 2: item in scrapy, it exists in memory as a completely
Speaker 2: empty shell. The compartments are there, but they contain absolutely nothing.
Speaker 1: You're just waiting for data, right.
Speaker 2: Filling those compartments requires it to look at the underlined
Speaker 2: structure of the website itself and extract the actual data.
Speaker 1: Okay, and this is where it gets a little abstract,
Speaker 1: I think, but it really is the core mechanism of webscreeping.
Speaker 1: How do we actually tell our computer, hey, go look
Speaker 1: at this massive wall of code and grab the words
Speaker 1: Microsoft Corporation.
Speaker 2: Yeah, it's not like you can just point at the screen.
Speaker 2: So we use a mechanism called an XPath selector.
Speaker 3: An XPath selector right.
Speaker 2: To understand why XPath is necessary, we have to look
Speaker 2: at how websites are actually built. HTML is not just
Speaker 2: a flat text document. It is a complex, nested hierarchy
Speaker 2: like a tree exactly.
Speaker 3: It's a tree of elements.
Speaker 2: You have this massive HTML trunk which branches off into
Speaker 2: body tags, which branch off into dip containers, down into tables, rows,
Speaker 2: and finally individual data cells. So to a computer, a
Speaker 2: webpage looks like a massive set of Russian mesting dolls.
Speaker 1: Okay, so an XPath is essentially like a set of
Speaker 1: navigation directions through that specific family tree. Yes, you are
Speaker 1: telling the scraper, okay, go into the table container of
Speaker 1: the rows and look inside the cell at index two, right, and.
Speaker 2: You've hit on a really vital detail there with index two.
Speaker 2: Oh really, yeah, because in programming we generally start counting from.
Speaker 1: Zero ah right, zero based indexing exactly.
Speaker 2: So if the company name is in the third column
Speaker 2: of the visual tail you see on the screen, it's
Speaker 2: index in the actual code is two.
Speaker 1: Wow.
Speaker 3: Okay, yeah, if.
Speaker 2: You tell your scraper to look at index three, you
Speaker 2: are going to pull the entirely wrong column of data,
Speaker 2: which is a very common trap when people first start
Speaker 2: testing in the scrapey shell.
Speaker 1: That would be so frustrating. Okay, so I write my
Speaker 1: expath directions, I target index two. Does the shell just
Speaker 1: like hand me the text Microsoft Corporation right then and there?
Speaker 3: Uh No, it does not.
Speaker 1: Oh it does. No.
Speaker 2: The XPath selector merely returns a match. It basically hands
Speaker 2: you a selector object, essentially a pointer that says, hey,
Speaker 2: I found the digital.
Speaker 1: Nesting doll you asked for, but it hasn't opened it yet.
Speaker 2: Exactly the text you actually want is still sealed inside
Speaker 2: that doll. So to reach in and extra the human
Speaker 2: readable data, you have to execute a specific function on
Speaker 2: that select or object.
Speaker 1: Okay, looking at the material, it seems we use functions
Speaker 1: like get for this.
Speaker 3: Yes, get is the one.
Speaker 1: So the workflow is basically, use x path to point
Speaker 1: at the cell, then used get to physically pull the
Speaker 1: string of text out, and once we have that text,
Speaker 1: we place it neatly into the company name compartment of
Speaker 1: our Bento box.
Speaker 3: You got it.
Speaker 2: That completes the entire cycle for a single data point.
Speaker 2: We've successfully navigated the HTML tree, extracted the raw text,
Speaker 2: and structured it. But we did this inside an isolated
Speaker 2: sandbox for one single company.
Speaker 1: Right, And here's the fatal flaw with staying in the sandbox.
Speaker 1: We are still manually pointing it at one company at
Speaker 1: a time. If we stop here, we haven't actually automated anything,
Speaker 1: have we. We have just changed the interface from copy
Speaker 1: pasting in a web browser to typing manual commands in
Speaker 1: a terminal.
Speaker 3: Exactly. It's the same amount of work.
Speaker 1: So to build a factory floor that can do this
Speaker 1: one hundred times a second, we need a way to
Speaker 1: scale the logic.
Speaker 2: Yeah, moving from testing to actual automation requires building a spider.
Speaker 1: A spider right.
Speaker 2: When you are ready to scale, you create a full
Speaker 2: scrapy project. Running a simple setup command generates an entire
Speaker 2: architecture of folders and files. It essentially lays the concrete
Speaker 2: for your factory floor.
Speaker 1: Okay, so inside that new folder structure, we have a
Speaker 1: file called items dot pie. I assume this is where
Speaker 1: we officially encode the blueprint for our Bento box. Right,
Speaker 1: listing out those four specific fields we want to fill.
Speaker 2: Yes, you define your item there, but the actual end
Speaker 2: of the thing doing the crawling lives in a separate file,
Speaker 2: maybe something we call a companydetailseem dot.
Speaker 3: Pie got it.
Speaker 1: And that is the spider file exactly.
Speaker 2: And the logic you encode inside the spider is incredibly
Speaker 2: elegant because it relies on loops ah loops.
Speaker 3: Yeah.
Speaker 2: Instead of writing separate XPath directions for every single row
Speaker 2: on the Yahoo finance page, you write a generalized set
Speaker 2: of instructions. You instruct the spider to target the entire
Speaker 2: table first, then you tell it to iterate. So row
Speaker 2: by row right, you say, look at row one, perform
Speaker 2: these actions, then move to row two and repeat.
Speaker 1: Okay, so for row one, it instantiates a brand new
Speaker 1: empty Bento box. It runs the x path directions to
Speaker 1: grab the name, price, symbol, and link. It packs them
Speaker 1: all into the box. But then it uses a command
Speaker 1: called yield. Yes, yield, And here is where I need
Speaker 1: to clarify something, because yield sounds a bit mysterious. If
Speaker 1: I tell the program to yield my carefully packed box,
Speaker 1: am I just throwing it out the window into the void?
Speaker 1: Like how are we actually capturing this data.
Speaker 2: It's a very common point of confusion, actually, because in
Speaker 2: standard Python functions, people are really used to the return command. Right.
Speaker 1: Return just gives you the answer and stops.
Speaker 2: Exactly, it hands back a value and immediately kills the function.
Speaker 2: But yield is fundamentally different. It creates what's called a generator.
Speaker 1: A generator.
Speaker 2: Okay, so when the spider yields the item, it hands
Speaker 2: the Bento box up to the Scrapy engine, basically pauses
Speaker 2: its current state, and then immediately turns back to the
Speaker 2: loop to process road too.
Speaker 1: Oh I see, so it operates almost like a worker
Speaker 1: on an assembly line.
Speaker 3: Yeah, perfect analogy.
Speaker 1: They finish packing a box, they hand it to the
Speaker 1: supervisor standing directly behind them, and then instantly turn back
Speaker 1: to the conveyor belt to pack the next one. They
Speaker 1: don't clock out and go home just because one single
Speaker 1: box is done.
Speaker 3: Exactly.
Speaker 2: The supervisor in your analogy is Scrapy itself. If you
Speaker 2: go to your terminal and simply tope screaby crawl company
Speaker 2: details item, Scrapey spins up the spider, The spider starts
Speaker 2: iterating yielding those boxes, and Scrapey just prints the resulting
Speaker 2: Jason objects right there on your screen, cascading down the terminal.
Speaker 1: I mean, seeing the matrix code fall down the screen
Speaker 1: is really fun. But data printed in a terminal basically
Speaker 1: disappears the second you close the window. We need a
Speaker 1: physical spadsheet we can actually send to our boss.
Speaker 2: Oh absolutely, And scrapey makes saving the data almost shockingly simple.
Speaker 2: Really yeah, you do not need to write complex file
Speaker 2: handling code inside your spider at all. You simply modify
Speaker 2: the terminal command you just ran. Oh nice, By adding
Speaker 2: an out put flag, specifically appending augiocompanydetails dot csv to
Speaker 2: the end of your command, scrape basically catches every single
Speaker 2: item the spider yields and dynamically appends into a neatly
Speaker 2: formatted spreadsheet file.
Speaker 1: Wait, really, so one simple terminal flag replaces dozens of
Speaker 1: lines of file saving logic just one flag.
Speaker 3: That's it.
Speaker 1: That is amazing. We run the command, the spider crawls
Speaker 1: the page, and boom, we have a CSV file sitting
Speaker 1: on our hard drive. But you know, if we open
Speaker 1: that spreadsheet right now, I have a feeling it is
Speaker 1: going to be a disaster. The web is inherently messy,
Speaker 1: right and raw scraped data is rarely usable right out
Speaker 1: of the box.
Speaker 2: Oh, it's almost always unusable. Raw data is rough. The
Speaker 2: source material points out two glaring formatting issues. You will
Speaker 2: immediately encounter when scraping that Yahoo Finance page.
Speaker 1: Okay, what's the first one.
Speaker 2: The first problem has to do with the URLs we
Speaker 2: are extracting for the company profiles. They are what developers
Speaker 2: call relative.
Speaker 1: Links, a relative links, meaning they don't start with HTP
Speaker 1: or wa ww. The HTML might just say something like
Speaker 1: quote MSFT exactly because websites do this to save space, right, yeah,
Speaker 1: they assume the browser already knows you are sitting on
Speaker 1: the Yahoo Finance domain. But if you drop that relative
Speaker 1: link into an offline Excel file and click it, your
Speaker 1: computer has literally no idea where to go. It's a
Speaker 1: completely broken link.
Speaker 2: Right, The spreadsheet has completely lost the context of the website.
Speaker 2: So that is the first major hurdle, appending the base
Speaker 2: domain to every single link on.
Speaker 1: The second issue.
Speaker 2: The second issue is a weird quirk of how Scrape's
Speaker 2: extraction functions work. By default, when you tell scrape to
Speaker 2: extract text, it doesn't just hand you a clean string.
Speaker 2: It returns a Python list object.
Speaker 1: Oh I see. It assumes there might be a multiple
Speaker 1: matches for your expath, so it wraps everything in a
Speaker 1: list just to be.
Speaker 3: Safe, exactly, just in case.
Speaker 1: Which means our spreadsheet doesn't say Microsoft Corporation. It says
Speaker 1: bracket quote Microsoft Corporation quote bracket like it's literally trapped
Speaker 1: inside Microsoft Corporation.
Speaker 2: Yeah, and if you try to hand a spreadsheet full
Speaker 2: of bracketed list objects to a data analyst, they will
Speaker 2: send it right.
Speaker 3: Back to you.
Speaker 1: I would too.
Speaker 2: You cannot perform basic database functions on text that's trapped
Speaker 2: in a list format. We need a way to refine
Speaker 2: this data systematically, and this introduces a vital scrapy component
Speaker 2: called the item loader.
Speaker 1: The item loader. Okay, so instead of the spider just
Speaker 1: stuffing raw, dirty data directly into our bento box fields,
Speaker 1: an item loader acts as an intelligent populator. Yes, we
Speaker 1: feed the raw XPath data into the loader and it processes, it,
Speaker 1: cleans it, and formats it before permanently sealing the box.
Speaker 1: And based on the documentation, it achieves this using two
Speaker 1: distinct mechanisms, input processors and output processor.
Speaker 3: You've got it.
Speaker 2: Let's examine the input processor first. Okay, This mechanism intercepts
Speaker 2: the data the exact millisecond it is scraped off the
Speaker 2: page before it even touches the item. Think about our
Speaker 2: prokin relative URL.
Speaker 1: Problem right, the quote MSFT problem.
Speaker 2: Right In our blueprint file items dot piy, we can
Speaker 2: attach an input process a specifically to the link field.
Speaker 2: And that documentation highlights a really powerful object for this
Speaker 2: called map compose map composed.
Speaker 1: I mean it sounds a bit complicated, but the way
Speaker 1: I understand it, it is essentially a midair interceptor.
Speaker 3: I like that.
Speaker 1: Yes, So you write a tiny helper function let's just
Speaker 1: name it pullings ah. And the only job of pullinks
Speaker 1: is to take a broken piece of text like quote
Speaker 1: MSFT and just gluefinance dot yahoo dot com to the
Speaker 1: front of it exactly.
Speaker 2: And map compose allows you to seamlessly map that custom
Speaker 2: pullings function to your incoming data stream. So the raw
Speaker 2: relative link comes off the web page, map composed catches
Speaker 2: it in midair, feeds it through your helper function, and
Speaker 2: the fully repaired absolute URL is what actually gets placed
Speaker 2: into the Bento box.
Speaker 1: That is so clean, So that handles the data on
Speaker 1: the way in. But what about the bracket problem that
Speaker 1: messy list format the Microsoft Corporation issue?
Speaker 2: Right?
Speaker 1: I assume that is where output processors step in. If
Speaker 1: the input processor is washing the ingredients before they go
Speaker 1: into the box. The output processor operates at the very
Speaker 1: end of the line, right before the spider yields the
Speaker 1: final product.
Speaker 2: Precisely, output processors handle the final data transformation, and Scrapie
Speaker 2: actually provides a built in output processor explicitly designed to
Speaker 2: solve this exact list problem.
Speaker 3: Oh really, yeah, it'll call take first.
Speaker 1: Take first. I was actually trying to think of an
Speaker 1: analogy for this earlier. It's almost like shucking an oyster.
Speaker 2: Shucking an oyster. Okay, how so, Well.
Speaker 1: The data you want is the pearl, but it's trapped
Speaker 1: inside this totally useless hardshell the list brackets. The take
Speaker 1: first processor prize open, the list reaches in, takes the
Speaker 1: very first item it finds, and just hands you the clean,
Speaker 1: unencumbered pearl of text.
Speaker 3: That is a perfect analogy. Yes.
Speaker 2: By setting take first as the default output processor for
Speaker 2: your item loader, every single field, the company, name, the price,
Speaker 2: the symbol is automatically shucked from those annoying brackets. Nice.
Speaker 2: They all become clean, plain strengths. So now when you
Speaker 2: save to your CSV, the spreadsheet looks completely professional.
Speaker 1: Okay, I am really loving this architecture. Item loaders give
Speaker 1: us structural cleaning on the fly. But let's push this
Speaker 1: even further.
Speaker 3: Let's do it.
Speaker 1: What if we want to apply actual business logic, Like,
Speaker 1: what if we aren't just formatting text, but we want
Speaker 1: to make active decisions based on the data itself after
Speaker 1: the whole extraction process is complete.
Speaker 2: Well, structural cleaning is one thing, but applying business logic
Speaker 2: requires the final and perhaps most powerful piece of this
Speaker 2: scrapy architecture. Data pipelines.
Speaker 1: Data pipelines. I visualize a pipeline as a quality assurance
Speaker 1: conveyor belt on a factory floor.
Speaker 3: Oh absolutely so.
Speaker 1: Our item loaders have packed these perfectly formatted bento boxes.
Speaker 1: Now those completed boxes are rolling down a long conveyor belt,
Speaker 1: and positioned along this belt are different QA stations. As
Speaker 1: a box rolls up to a station, an inspector opens it,
Speaker 1: looks at the data inside based on very strict mathematical rules,
Speaker 1: and decides what to do. They can modify the data,
Speaker 1: they can drop the box entirely into the trash, or
Speaker 1: they can simply wave it through to the next station.
Speaker 2: The conveyor belt analogy is incredibly accurate in your scraping project.
Speaker 2: These pipelines live in their own dedicated file simply named
Speaker 2: pipelines dot pi makes sense, and inside this file you
Speaker 2: can construct as many of these QA stations as you need.
Speaker 2: Each station is written as a standard Python class, and
Speaker 2: each class must contain a specific mandatory function called process item.
Speaker 1: Because its entire existence is dedicated to processing the item currently.
Speaker 3: Rolling down the belt exactly.
Speaker 1: The source material actually provides a highly practical example of
Speaker 1: this business logic involving the stock prices. Let's walk through
Speaker 1: how we build that.
Speaker 3: Sure.
Speaker 2: So imagine you are building a tool for a financial
Speaker 2: analyst who only cares about extreme stock price movements. They
Speaker 2: want to flag any stock over one hundred dollars or
Speaker 2: under fifty dollars. Okay, we can build two separate stations
Speaker 2: on our conveyor belt. We define our first class and
Speaker 2: name it above one hundred price check.
Speaker 1: Station one is operational the item rolls up. What actually
Speaker 1: happens inside that process item function.
Speaker 2: First, it looks at the inter day price field. Now,
Speaker 2: remember the price is currently stored as a string of text.
Speaker 2: You cannot perform mathematical operations on text.
Speaker 1: Right, Python would just throw an error exactly.
Speaker 2: So the first thing the function does is parse that
Speaker 2: text and convert it into a numerical float. Once it
Speaker 2: is a real number, it as a simple logical question,
Speaker 2: is this value greater than one hundred?
Speaker 1: Okay, let's say the extracted price is a one oh
Speaker 1: five point fifty. The answer is yes. What does the
Speaker 1: station actually do?
Speaker 2: It actively overwrites the scrape data. It modifies the item,
Speaker 2: changing the price field from one oh five point fifty
Speaker 2: to a hard coded string that simply reads one hundred.
Speaker 1: Oh wow, so it replaces the number entirely.
Speaker 3: Yeah.
Speaker 2: It categorizes the data, and once the modification is done,
Speaker 2: it passes the item down the belt.
Speaker 1: Okay, So the item continues rolling and arrives at our
Speaker 1: second class below fifty price check station two. I assume
Speaker 1: the logic mirror is the first. It converts the price
Speaker 1: to a number and checks if it is under fifty dollars. So,
Speaker 1: if the stock is trading at say forty dollars, it
Speaker 1: overwrites the field to read fifty precisely.
Speaker 2: But the really fascinating part of this pipeline is what
Speaker 2: happens to the data that followed perfectly in the middle.
Speaker 2: Consider a stock trading at exactly seventy five dollars.
Speaker 1: Okay, So the seventy five dollars box rolls up to
Speaker 1: station one. The above one hundred check runs seventy five
Speaker 1: is not greater than one hundred, so the inspector does nothing.
Speaker 1: The box passes through completely untouched. It reaches station two.
Speaker 1: The below fifty check runs seventy five is not less
Speaker 1: than fifty. Again, the inspector does nothing, and.
Speaker 2: So the seventy five dollars item arrives at the very
Speaker 2: end of the conveyor belt completely untouched, retaining its original
Speaker 2: precise numerical value.
Speaker 1: That is so cool.
Speaker 2: You see, you are not just blindly scraping data anymore.
Speaker 2: You are dynamically categorizing and filtering information in real time.
Speaker 1: And if you are listening to this and wondering why
Speaker 1: go through all this trouble, why not just dump the
Speaker 1: raw numbers into Excel and run a simple filter formula later.
Speaker 3: Right? People always ask.
Speaker 1: That it's because of automation. By building this logic directly
Speaker 1: into the pipeline, this script can run unattended on a
Speaker 1: server every single day for a year. You wake up
Speaker 1: and your perfectly categorized, cleaned spreadsheet is just sitting there
Speaker 1: waiting for you, no manual cleanup required.
Speaker 2: Ever, it's huge it's just your time from cleaning data
Speaker 2: to actually analyzing it exactly. But building a pipeline with
Speaker 2: multiple stations introduces a really critical logistical problem.
Speaker 1: Oh what's that?
Speaker 2: Well, if you have a massive pipeline with twenty different
Speaker 2: stations doing complex transformations, the order in which those stations.
Speaker 3: Operate is paramount.
Speaker 1: Oh I see.
Speaker 3: Yeah.
Speaker 2: If station one deletes a piece of data entirely, station
Speaker 2: two cannot check it, it's gone.
Speaker 1: Okay, So how does scrapee manage the traffic? Like, how
Speaker 1: does it know that the above one hundred check should
Speaker 1: run before the blow fifty check?
Speaker 2: Scrapeye requires you to explicitly declare the order in a
Speaker 2: configuration file called settings dot pie. Now, by default, Scrapeye
Speaker 2: is a very lightweight crawler, so pipelines are turned off
Speaker 2: to save memory.
Speaker 1: Oh I didn't realize that.
Speaker 2: Yeah, So to activate your conveyor belt, you must open
Speaker 2: settings dot pie and actually uncomment a specific dictionary named itempipelines.
Speaker 1: Ah. So settings dot pie is essentially the control room
Speaker 1: of the factory. You uncomment the code to power up
Speaker 1: the conveyor.
Speaker 2: Belt exactly, and inside that dictionary you list have the
Speaker 2: names of your classes, your QA stations, and next to
Speaker 2: each class name, you assigned an integer a priority number. Okay,
Speaker 2: and Scrape operates on a very strict rule. Here, a
Speaker 2: lower number dictates a higher priority.
Speaker 1: So it's exactly like a race. First place gets the
Speaker 1: number one, so they go first.
Speaker 3: Exactly.
Speaker 1: So if I assign above one hundred price check a
Speaker 1: priority number of one hundred and below fifty price check
Speaker 1: a priority number of two hundred, Scrape basically guarantees that
Speaker 1: the one hundred check executes first every single time.
Speaker 2: Yeap, you hold absolute control over the sequence of transformations.
Speaker 2: You can inject new stations, you can reorder them or
Speaker 2: turn them off completely, all from the central settings dot
Speaker 2: PI file without ever having to touch your underlying spider code.
Speaker 1: Man, let's zoom out and look at what we have
Speaker 1: constructed today. We didn't just write a script. We architected
Speaker 1: a complete resilient system.
Speaker 3: We really did.
Speaker 1: We started in the scriptshall testing the HTML dawntree to
Speaker 1: figure out our XPath navigation. We built a structural blueprint
Speaker 1: for our data using the item bento box. We deployed
Speaker 1: a spider to loop through rows and continually yield items.
Speaker 1: We utilized item loaders acting as mid air interceptors and
Speaker 1: oyster shuckers to clean up broken URLs and messy brackets,
Speaker 1: and finally we routed everything through a QA conveyor belt
Speaker 1: in the pipeline to apply custom business logic.
Speaker 2: Moving from a manual task to a fully automated self
Speaker 2: cleaning data pipeline completely alters how you viewed data on
Speaker 2: the Internet.
Speaker 1: It really does. So to make sure the core mechanisms
Speaker 1: of this architecture really stick with you, let's see if
Speaker 1: we actually internalize the details. A quick review question for
Speaker 1: everyone listening. What's your if I run my spider and
Speaker 1: I open my CSV file and I find that my
Speaker 1: spreadsheet is full of data trapped inside those annoying list brackets,
Speaker 1: what is the one word fix I forgot to implement
Speaker 1: in my item loader?
Speaker 2: You forgot your oyster shuck the take first output processor
Speaker 2: take first.
Speaker 1: It is such a tiny detail in the grand architecture,
Speaker 1: but you know, knowing the default tools available in the
Speaker 1: framework is exactly what separates a fragile, messy script from
Speaker 1: a professional, production ready pipeline.
Speaker 2: Absolutely, a professional pipeline is designed to be resilient, but
Speaker 2: you know there is an external factor that even the
Speaker 2: best pipeline cannot inherently control.
Speaker 1: The web itself.
Speaker 2: Exactly the web itself. Our entire beautifully calibrated system relies
Speaker 2: on XPath selectors. We gave our spider rigid directions based
Speaker 2: on the current lay out of the website. We told
Speaker 2: it navigate into the table, drop down into the rows,
Speaker 2: find the solid index too, right.
Speaker 1: So what happens when Yahoo Finance decides to just refresh
Speaker 1: their brand overnight, like they roll out massive UI redesign.
Speaker 1: They ditch HTML tables entirely and move to a modern
Speaker 1: grid layout.
Speaker 2: The moment they push that update, the structure of the
Speaker 2: HTML tree fundamentally changes. Your XPath directions become completely useless.
Speaker 2: The spider navigates to where index two used to be
Speaker 2: and finds nothing or worse. Or worse, it finds entirely
Speaker 2: different data, like grabbing the market caap instead of the
Speaker 2: company name, and blindly feeds that bad data straight through
Speaker 2: your beautiful pipeline into your final spreadsheet.
Speaker 1: The pipeline executed perfectly, but the map it used was
Speaker 1: completely obsolete. It highlights a massive challenge in this field,
Speaker 1: because now that you understand how to build a scripey
Speaker 1: pipeline that collects cleans and transforms data. The next logical
Speaker 1: frontier is resiliency. How do you design systems that don't
Speaker 1: just extract data based on today's layout, but can intelligently
Speaker 1: hunt for that data tomorrow, adapting seamlessly when the digital
Speaker 1: landscape shifts completely underneath them. Something to think about.