Speaker 1: Imagine a world running on data that looks well, kind
Speaker 1: of like a spreadsheet. We're talking banks, insurance, retail, government.
Speaker 1: This isn't the really flashy stuff like AI making pictures
Speaker 1: or music. This is the absolute bedrock of our digital economy.
Speaker 1: So today we're doing a deep dive into this fascinating
Speaker 1: and honestly often overlooked world of machine learning for tabular data.
Speaker 1: Our goal here is to cut through some of the noise,
Speaker 1: give you a shortcut to understanding what tabular data actually is,
Speaker 1: why it matters so much, and how we apply the
Speaker 1: well the most powerful mL techniques to it. We'll touch
Speaker 1: on everything from cleaning up messy data, pitting classic mL
Speaker 1: against deep learning, and even deploying these models out into
Speaker 1: the real world. You might find some surprising facts, maybe
Speaker 1: a few aha moments. Okay, let's get into it. So
Speaker 1: let's start right at the beginning. What exactly is tabular data?
Speaker 1: I think of just a simple table, maybe one, listing
Speaker 1: currencies for different countries. Each row that's an observation, like
Speaker 1: all the details for Australia's currency, and then each column
Speaker 1: is of feature things like currency name or units per u.
Speaker 2: Pretty straightforward, right, Yeah, exactly it's a format everyone gets,
Speaker 2: which is why it's so fundamental. And while it might
Speaker 2: only be say, ten percent or so of all the
Speaker 2: digital data out there, by some estimates that ten percent
Speaker 2: is absolutely critical. It's structure that row and columns set
Speaker 2: up makes it super easy to input, retrieve, manage, analyze.
Speaker 2: It really is the lifeblood for countless businesses, spreadsheets, huge databases,
Speaker 2: you name it.
Speaker 1: Okay, So here's something that's always kind of puzzled me.
Speaker 1: If it's so fundamental, why haven't deep learning models completely
Speaker 1: taken over this space, I mean, the way they have
Speaker 1: for images or audio or text. What's the key difference there?
Speaker 2: That is a really great question. The main thing is
Speaker 2: that tabular data has that typical matrix shape rows, columns
Speaker 2: pretty distinct. It's not like unstructured data audio, waves, pixels, texts,
Speaker 2: which is much more well unordered and varied. And because
Speaker 2: of that unique structure, tabula data comes with its own
Speaker 2: set of let's call them pathologies, common problems you absolutely
Speaker 2: have to fix before you can do any serious analysis pathologies.
Speaker 1: Okay, what kind of problems are we talking about?
Speaker 2: Well, first off, you often find constant or quasi constant
Speaker 2: columns features that just don't change much or at all.
Speaker 2: They offer almost no information to a model. Then there
Speaker 2: are duplicated and highly call near features, so information that's
Speaker 2: either just copied or it's so similar it's basically saying
Speaker 2: the same thing twice. With linear models especially, this can
Speaker 2: cause real conceptual misunderstandings. Makes it hard to figure out
Speaker 2: what's actually driving a prediction.
Speaker 1: Right, like having two columns that both measure basically the
Speaker 1: same temperature scale.
Speaker 2: Yeah, redundant exactly. Then you have irrelevant features, stuff that
Speaker 2: just doesn't help predict what you want to predict, and
Speaker 2: the big one missing data. This is crucial because some
Speaker 2: mL algorithms just flat out won't run if there are gaps,
Speaker 2: and these gaps aren't always random. Sometimes data is missing
Speaker 2: completely at random, sometimes just at random, or sometimes it's
Speaker 2: missing not at random. Think about it. A missing review
Speaker 2: score might actually mean there are no reviews, which is
Speaker 2: itself information.
Speaker 1: All right, That's a subtle but important distinction.
Speaker 2: Definitely. We also deal with rare categories, features with tons
Speaker 2: of unique values, or values that show up super infrequently,
Speaker 2: hard for models to learn from those, and my personal favorite,
Speaker 2: just plain errors in the data. You know, Misspelling's like
Speaker 2: Toyota instead of Toyota. Hmm, this isn't just a typo Cosmetically,
Speaker 2: it splits what should be one category into multiple noisy ones.
Speaker 2: Really confuses the model.
Speaker 1: It sounds like a minefield.
Speaker 2: It can be. The key insight here is like a
Speaker 2: slightly blurry pixel in an image might just make it
Speaker 2: less clear, but a single misspelled category or an unhandled
Speaker 2: missing value in a table that can fundamentally mislead your
Speaker 2: model force it to make decisions based on completely wrong information.
Speaker 2: It's like having a great map, but with a few
Speaker 2: key cities just randomly renamed, you can't navigate.
Speaker 1: Huh, the bane of every data scientist's existence. It's like
Speaker 1: trying to find Waldo, but half the time he skilled
Speaker 1: Waldo completely messing up your search algorithm. So it really
Speaker 1: sounds like forget the fancy algorithms for a second, real
Speaker 1: hard work. Maybe the biggest challenge is just understanding and
Speaker 1: prepping your data. Is that where something like exploratory data
Speaker 1: analysis EDA comes in. Is it indispensable?
Speaker 2: Absolutely one percent. Getting reliable insights starts with good EDA.
Speaker 2: And it's not just about making pretty charts though that helps.
Speaker 2: It's really about systematically spotting and fixing these pathologies before
Speaker 2: they wreck your model downstream. We use tools like histograms,
Speaker 2: box plots, things like that to actually see how the
Speaker 2: data is distributed. You know, spot things like heavy tails,
Speaker 2: extreme values and prices maybe which can seriously skew your results.
Speaker 1: Okay, when you find those extremes, like maybe a house
Speaker 1: listed for ten billion dollars by mistake, you know, just
Speaker 1: deleted to you. You mentioned windsorizing. How does that work
Speaker 1: and why is it often better than just tossing the
Speaker 1: data point or letting it mess everything up?
Speaker 2: Right? Good question. Windsorizing basically means if a value is
Speaker 2: way out there, maybe be on the top one percent
Speaker 2: or bottom one percent of your data range, you just
Speaker 2: capped it, place it with the value at that one
Speaker 2: percent or ninety nine percent mark. So you keep the
Speaker 2: data point, but you prevent that extreme outlier, maybe a
Speaker 2: data entry error, from having this huge disproportionate influence on
Speaker 2: your model. Similarly, for those categorical features with tons of
Speaker 2: unique labels we call high cardinality features, we can aggregate
Speaker 2: the really rare categories, group all those one off values
Speaker 2: into a single other category. Do this simplifies things for the
Speaker 2: model and Honestly, the unsung hero that makes a lot
Speaker 2: of this data wrangling possible is the panda's data frame
Speaker 2: in Python. It's just incredibly flexible and efficient for managing
Speaker 2: and manipulating tabular data.
Speaker 1: Okay, Pandas got it, So let's shift. Here's a bit.
Speaker 1: There's this ongoing debate in the data science world, right
Speaker 1: when you're tackling these tabular data problems, what's better classical
Speaker 1: machine learning techniques or deep learning. Maybe we can unpack
Speaker 1: this using that airbb example you mentioned, predicting listing prices
Speaker 1: in New York City.
Speaker 2: Yeah, that's a great way to look at it. We
Speaker 2: can compare these two approaches. Let's say classical mL represented
Speaker 2: by xg boost a popular choice, and deep learning may
Speaker 2: be using keras across a few key things. First, simplicity
Speaker 2: in that Airbnb case study, using xg boost often meant
Speaker 2: much simpler code to define and train the model, sometimes
Speaker 2: literally just one line following the standard psychic learned pattern.
Speaker 2: Lots of people know. Kearras for deep learning usually needed
Speaker 2: quite a few more lines, especially for defining all the
Speaker 2: network layers and setting up things like efficient training callbacks.
Speaker 1: Right, that definitely matters for day to day work. But
Speaker 1: simplicity aside, what about understanding why the model makes a prediction?
Speaker 1: You know, transparency and explainability. How do they stack up there?
Speaker 2: That's a huge point. Classical models like decision trees, which
Speaker 2: are kind of the building blocks for xg boost, can
Speaker 2: often be visualized or explained. You could, for example, show
Speaker 2: a non specialist how a simple decision tree predicts how
Speaker 2: long a property might stay on the market step by step.
Speaker 2: Deep neural networks, well, they often rely on these analogies
Speaker 2: to biological neurons, which were frankly a bit controversial and
Speaker 2: don't really clarify how the model arrives at its decision. Internally,
Speaker 2: it's more of a.
Speaker 1: Black box the black box problem.
Speaker 2: Exactly, and related to that is feature importance, what actually
Speaker 2: drove the prediction. Xg boost has built in methods that
Speaker 2: easily tell you which features had the biggest impact, like
Speaker 2: for the Airbnb prices, room type might pop up as
Speaker 2: the most important factor. Deep learning frameworks like Keras they
Speaker 2: don't usually have that built right in. You need external tools,
Speaker 2: often more complex ones, to try and get that same
Speaker 2: kind of insight.
Speaker 1: So it sounds like for understanding and explaining classical methods
Speaker 1: often have an edge.
Speaker 2: Often yes, And if we look at the bigger picture,
Speaker 2: like research trends, the amount of research specifically on deep
Speaker 2: learning for tabular data is actually just a tiny fraction
Speaker 2: of all deep learning research being published. There's just no
Speaker 2: unambiguous winner yet in terms of raw predictive power on
Speaker 2: tables for tabular data, the jury is definitely still out.
Speaker 1: Okay, that's really interesting. Why do you think that is?
Speaker 1: Why hasn't deep learning just dominated here like it has elsewhere?
Speaker 2: Well? One theory is that tabular nata often already presents
Speaker 2: features in a highly structured, kind of interpretable way, you know, price, location,
Speaker 2: number of bedrooms. Deep learning's real superpower is often extracting
Speaker 2: hierarchical features from raw unstructured stuff, like finding edges and
Speaker 2: shapes than objects in an image or grammar patterns in text.
Speaker 2: But with tabular data, that powerful automatic feature extraction might
Speaker 2: not be the huge advantage it is elsewhere. The features
Speaker 2: are often pretty meaningful already. In fact, sometimes deep learning
Speaker 2: might even pick up on spurious correlations and tables, essentially
Speaker 2: finding patterns and noise because it's so powerful at pattern finding.
Speaker 1: Gotcha, So it might be too powerful in a way
Speaker 1: for this kind of data sometimes. So if deep learning
Speaker 1: isn't the clear raining champ here, what is generally considered,
Speaker 1: you know, state of the art for most tabular data
Speaker 1: problems right now.
Speaker 2: Right now, that title really belongs to gradient boosting decision
Speaker 2: trees or gbdts. These models have really become the workhourses
Speaker 2: for tabular data tasks gbdts.
Speaker 1: Okay, how do they actually work? How do they get
Speaker 1: such good predictions?
Speaker 2: They're a really cool example of what's called an ensemble method,
Speaker 2: basically getting multiple models to work together. But unlike some
Speaker 2: other ensemble methods like random forests, where models are built independently,
Speaker 2: gbdts build models sequentially. Think of it like building the
Speaker 2: prediction piece by piece, almost like a chain. Each new
Speaker 2: tree model tries to correct the errors made by the
Speaker 2: previous ones. So it's this iterative process of improvement. It
Speaker 2: learns from the mistakes of the models that came before
Speaker 2: it in the sequence.
Speaker 1: Ah. Okay, So it's like a team where each member
Speaker 1: learns from the last one's.
Speaker 2: Attempt precisely, not just averaging independent guesses, but actively refining
Speaker 2: the prediction.
Speaker 1: And you mentioned two big names leading the GBDT charge
Speaker 1: x you boost and light GBM. They got famous through competitions.
Speaker 2: Right, that's right. They really gained prominence by winning or
Speaker 2: performing incredibly well in data science competitions like Caggles Higgs
Speaker 2: Boson machine Learning Challenge years ago that really put them.
Speaker 1: On the map. So what makes them so good? Is
Speaker 1: it just the boosting idea or is it more to it?
Speaker 2: There's definitely more to it. They achieve their speed and
Speaker 2: accuracy through some really clever technical innovations. Xg boost, for instance,
Speaker 2: uses smart ways to find the best splits in the
Speaker 2: data very quickly, like histogram splitting and a unique weighted
Speaker 2: quantile sketch. Light GBM uses techniques like leafwise tree growth.
Speaker 2: Instead of building the tree level by level symmetrically, it
Speaker 2: focuses its effort on the nodes the leaves where it
Speaker 2: can reduce the air the most. This can lead to
Speaker 2: faster training and smaller trees. Light GBM also uses smart
Speaker 2: sampling like gradient based one side sampling or GOSS to
Speaker 2: focus on the data points that are harder to predict,
Speaker 2: and exclusive feature bundling EFB to kind of group sparse
Speaker 2: features together efficiently.
Speaker 1: Wow, Okay, that sounds pretty sophisticated Under the hood, it is.
Speaker 2: Think of light GBM like a really efficient data assistant.
Speaker 2: It knows exactly where the most important information is likely
Speaker 2: to be and how to summarize things without losing crucial details.
Speaker 2: That makes it fast.
Speaker 1: And you mentioned something earlier that really caught my attention.
Speaker 1: They handle missing data automatically. That sounds almost too good
Speaker 1: to be true. How does that work? Does it mean
Speaker 1: we could just be a bit lazier with cleaning our
Speaker 1: data if we use.
Speaker 2: These Huh, well, it is a huge advantage. It's not
Speaker 2: really about being lazy though. Both xg boosts and light
Speaker 2: GBM have this built in capability where at each split
Speaker 2: point in a tree, they learn which direction left or
Speaker 2: right branch missing values should go to minimize the overall
Speaker 2: error the loss function, so the model itself learns the
Speaker 2: best way to handle those gaps based on the data patterns.
Speaker 2: It's quite robust and another key practical thing they both
Speaker 2: do is early stopping. They watch performance on a separate
Speaker 2: validation data set during training, and if the performance stops
Speaker 2: improving for a certain number of rounds, they just stop training.
Speaker 2: This is crucial to prevent overfitting, making sure the model
Speaker 2: works well on new data, not just the data it
Speaker 2: was trained on.
Speaker 1: Okay, that makes sense, prevents it from just memorizing the
Speaker 1: training set. So we have these two powerhouses, xg boost
Speaker 1: and light GBM. How do you actually choose between them
Speaker 1: for a specific priser.
Speaker 2: Yeah, that's a common question. Based on the sources we
Speaker 2: looked at and general experience in the field, there are
Speaker 2: some general guidelines. Light GBM often tends to perform better
Speaker 2: or at least train faster, when you have really large
Speaker 2: amounts of data. Its leafwise growth is very efficient then,
Speaker 2: but that same leafwise growth can sometimes cause it to
Speaker 2: overfit a bit more easily on smaller data sets. Xg boost,
Speaker 2: on the other hand, is often considered slightly more robust
Speaker 2: maybe builds more stable models, especially on smaller BEATA samples.
Speaker 2: Speed wise, light GBM is typically faster on CPUs, but
Speaker 2: xg boost is often seen as more scalable for distributed
Speaker 2: computing and has had perhaps slightly more mature GPU support historically,
Speaker 2: though light GBM is catching up fast there too.
Speaker 1: Okay, So it depends on the scale of your data,
Speaker 1: maybe your hardware. Interesting trade offs. So it seems like
Speaker 1: gradient boosting is incredibly powerful for tables. But deep learning
Speaker 1: isn't completely out of the picture, right and stepping back,
Speaker 1: getting any model boosting or deep learning actually working in
Speaker 1: the real world that involves a lot more than just
Speaker 1: hitting train, doesn't it?
Speaker 2: Oh, absolutely far more, And yes, deep learning still has
Speaker 2: a role. While classical mL specially gbtt's often performs very
Speaker 2: competitively or even better on many tabular tasks, frameworks like
Speaker 2: Keras built on TensorFlow, and fasti, which is built on PyTorch,
Speaker 2: are definitely making inroads. They often incorporate sophisticated preprocessing layers
Speaker 2: right into the deep learning model itself, handling data transformations
Speaker 2: efficiently within the network architecture.
Speaker 1: Right. And once you've picked your approach, say XG boost
Speaker 1: or a Keris model, you need to tune it right
Speaker 1: make it perform its best. That's where hyper parameter optimization
Speaker 1: comes in, finding those perfect setting exactly.
Speaker 2: You need to find the ideal settings the hyper parameters
Speaker 2: for your specific model and data, and there are several
Speaker 2: ways to do that. There's the classic grid search, which
Speaker 2: is exhaustive. It literally tries every single combination of parameter
Speaker 2: values you give it, like trying every key on a
Speaker 2: giant keychain. Then there's random search. You just randomly sample combinations. Surprisingly,
Speaker 2: this often works just as well or even better than
Speaker 2: grid search, especially if only a few hyper parameters really matter.
Speaker 2: It's often much more efficient.
Speaker 1: Randomly trying things works better. That seems counterintuitive it.
Speaker 2: Does, but imagine you have ten settings, but only two
Speaker 2: really impact performance. Grid search spends most of its time
Speaker 2: trying useless combinations of the other eight. Random search has
Speaker 2: a better chance of hitting good values for the important
Speaker 2: too much faster. Then you have smarter methods. Success of
Speaker 2: having is like running a tournament. You start many models
Speaker 2: with few resources, quickly discard the bad ones and give
Speaker 2: more resources to the promising candidates. And then there's Beaesian
Speaker 2: optimization using tools like optuna. This is really clever. It
Speaker 2: builds a statistical model of how the hyper parameters seem
Speaker 2: to affect performance, and uses that model to intelligently decide
Speaker 2: which combinations to try next. It's an informed search, much
Speaker 2: more efficient than just randomly guessing or trying everything.
Speaker 1: Okay, beaesian optimization sounds powerful. So you've trained your model,
Speaker 1: you've tuned it. Now the really hard part getting it
Speaker 1: out of lab and actually used. This is where mL
Speaker 1: ops machine learning operations becomes essential.
Speaker 2: Right, absolutely critical. MLOPS is huge and why is it
Speaker 2: so crucial? Well, first, just running your train model on
Speaker 2: some new unseen data points before deploying is vital. This
Speaker 2: helps detect things like data leakage. That's when somehow information
Speaker 2: from the future or even from the target variable you're
Speaker 2: trying to predict, accidentally sneaks into your training data. This
Speaker 2: makes your model look amazing during development, but then it
Speaker 2: completely fails in the real world because that leaked information
Speaker 2: isn't available. Then MLUPS practices help catch this ah.
Speaker 1: The dreaded data leakage, like predicting stock prices using tomorrow's
Speaker 1: closing price somehow precisely.
Speaker 2: MLOPS also helps you validate the model's actual performance in
Speaker 2: a scenario that mimix production. We saw examples of maybe
Speaker 2: doing a basic web deployment with something simple like flask,
Speaker 2: which is great for demos, but for real world, reliable
Speaker 2: applications you almost always need the robustness and scalability of
Speaker 2: public cloud platforms like Google Cloud awsure. These clouds offer
Speaker 2: comprehensive MLOPS environments. They handle things like model monitoring, tracking
Speaker 2: accuracy over time to see if it degrades. They ensure
Speaker 2: resiliency and uptime so your service stays available, and they
Speaker 2: support sophisticated mL pipelines.
Speaker 1: Okay, tell me more about the mL pipeline. That sounds
Speaker 1: like the real engine behind m elopes. What does it
Speaker 1: actually automate?
Speaker 2: It really is a game changer. An mL pipeline is
Speaker 2: essentially a coded, automated workflow. It takes you all the
Speaker 2: way from the raw input data right through to a deployed,
Speaker 2: monitored model. It automates the data cleanup, the feature engineering,
Speaker 2: the model training, the evaluation, the tuning, the deployment, the
Speaker 2: whole nine yards. This ensures consistency. Every time you run
Speaker 2: the pipeline, you get the same steps applied in the
Speaker 2: same way. It ensures repeatability and this is absolutely essential
Speaker 2: in dynamic environments like real estate pricing, where the market
Speaker 2: data changes constantly and you need to retrain and update
Speaker 2: your models frequently and reliably.
Speaker 1: That makes total sense. Automation and consistency are key for
Speaker 1: anything real world. So, thinking about everything we've discussed, if
Speaker 1: classical mL like GBDTS is strong and deep learning has
Speaker 1: its place, it makes you wonder can you actually combine
Speaker 1: them get the best of both worlds.
Speaker 2: That's exactly what some of the most interesting recent work explores,
Speaker 2: and the answer seems to be a definite yes. Going
Speaker 2: back to that Tokyo Airbnb pricing problem mentioned in the
Speaker 2: source material, they actually tried blending the predictions. They took
Speaker 2: an optimized XG boost model and a fine tuned deep
Speaker 2: learning model using FASTAI, and they found the best results.
Speaker 2: The lowest prediction error came from a fifty to fifty
Speaker 2: ensemble just averaging the predictions of the two models.
Speaker 1: How a fifty to fifty split was optimal, not leaning
Speaker 1: more heavily on one or the other in.
Speaker 2: That specific case. Yes, it really challenges that narrative. You
Speaker 2: sometimes hear that deep learning is all you need for
Speaker 2: tabular data. It seems that's often not true. Combining the
Speaker 2: strengths of gbdt's maybe their robustness with structured features and explainability,
Speaker 2: with the potential pattern finding power of deep learning, that
Speaker 2: collaborative approach which yielded the best results. It really reinforces
Speaker 2: that core idea, doesn't it That knowledge is most valuable
Speaker 2: when you understand it and can apply it creatively, and
Speaker 2: that considering multiple perspectives multiple approaches usually leads to a richer,
Speaker 2: better outcome.
Speaker 1: Absolutely, a blend often works best. So what a journey
Speaker 1: You've just taken a deep dive with us into this well,
Speaker 1: surprisingly complex, but absolutely vital world machine learning for tabular data.
Speaker 1: We've gone from dealing with messy spreadsheets and weird data
Speaker 1: quirks to pitting these powerful algorithms like XG boost and
Speaker 1: deep learning against each other and seeing how we actually
Speaker 1: bring them to life with mlops. It really makes you think, though,
Speaker 1: if even these incredibly sophisticated machine learning models can get
Speaker 1: tripped up by something as simple as a misspelling like toyota,
Speaker 1: or by subtle dependencies between rows or that missing value
Speaker 1: that actually means something, what does that really tell us
Speaker 1: about the fundamental importance of truly understanding your data, getting
Speaker 1: your hands dirty with it, exploring it, cleaning it before
Speaker 1: you even think about pressing trains. Maybe a thought worth
Speaker 1: mulling over. We really hope this deep dive helps you
Speaker 1: be even more informed and maybe more curious about the
Speaker 1: data that powers so much of our world.