Speaker 1: Welcome to the deep dive. You know, we hear so
Speaker 1: much about the theory of data science, all the algorithms
Speaker 1: in math. But today we're going to try and pull
Speaker 1: back the curtain a bit look at the real world.
Speaker 1: And our guide for this is The Practitioner's Guide to
Speaker 1: Data Science by Huielin and Mingly. It feels less like
Speaker 1: a textbook honestly, and more like an insider's view of
Speaker 1: what it really takes to do data science day to day.
Speaker 2: That's exactly right. What really jumped out at me was
Speaker 2: how practical it is, you know, how grounded it is.
Speaker 2: The authors they don't just give you the what. They
Speaker 2: really dig into the how, things like the soft skills
Speaker 2: which often get missed, right, and the whole context of
Speaker 2: the big data cloud environment. Yeah, that's huge. It's really
Speaker 2: about well navigating the messiness of actual data projects. Okay.
Speaker 1: Yeah, And it seems like they really push for hands
Speaker 1: on learning. I just appreciate that they've got these R
Speaker 1: and Python code notebooks all ready to go. You can
Speaker 1: grab them on GitHub the links http LA three three
Speaker 1: seven CD four's and they basically say, hey, get your
Speaker 1: hands dirty, take this code, use your own data to
Speaker 1: try it on problems.
Speaker 2: Yeah, make it tangible, and that focus on reproducibility using
Speaker 2: things like Google co Lab. Yeah, that's so important. Now
Speaker 2: it's not just about following steps, it's about giving you
Speaker 2: the power to take these techniques and actually build something,
Speaker 2: apply them to whatever challenges you're facing. It makes data
Speaker 2: science feel like a real tool you can use, not
Speaker 2: just concepts.
Speaker 1: The book kicks off with a bit of history too,
Speaker 1: which I've found pretty useful just for context. It traces
Speaker 1: things from the early days like least squares, linear discriminate analysis,
Speaker 1: the real foundations, all the way up to how cloud
Speaker 1: computing just completely changed the game for data engineering and management.
Speaker 1: It really shows how far we've come and fast.
Speaker 2: Oh definitely thinking about that evolution the cloud, it's just
Speaker 2: been massive, a total game changer. Suddenly you have access
Speaker 2: to all this computing power, storage. It kind of democratized
Speaker 2: working with huge data sets, you know, and that shifted
Speaker 2: data engineering. It's less about physical boxes now and more
Speaker 2: about orchestrating data pipelines up in the cloud. It's fundamental shift.
Speaker 1: Okay, So the authors then break down data science roles.
Speaker 1: They talk about three main skill tracks engineering, analysis and modeling, inference,
Speaker 1: So for engineering, it's about building the infrastructure right, the
Speaker 1: data pipelines, automated collection, managing the data itself, the plumbing
Speaker 1: basically right.
Speaker 2: And it's so critical you need that solid engineering foundation.
Speaker 2: Everything else is built on top of it. If your
Speaker 2: data infrastructure isn't reliable, well, the analysts and modelers, they
Speaker 2: just can't do their jobs properly. It's often the unseen work,
Speaker 2: but it's absolutely essential for getting good outcomes.
Speaker 1: Then there's the analysis track. This sounds like it's really
Speaker 1: about understanding the business side. What's the question, what's the
Speaker 1: data telling us, and then translating that business need into
Speaker 1: a data problem you can actually solve. The book really
Speaker 1: hits hard on domain knowledge and communication skills here.
Speaker 2: Exactly asking the right questions. Understanding the business context is crucial.
Speaker 2: The analyst is like a translator, you know, bridging the
Speaker 2: gap between the business folks who have the problems and
Speaker 2: the data scientists who might have solutions. It's definitely not
Speaker 2: just about crunching numbers. It's about insights that lead to
Speaker 2: actual decisions.
Speaker 1: Okay, and finally modeling inference. This is where we get
Speaker 1: into applying all the different learning methods. Supervised learning like
Speaker 1: regression and classification for predictions, but also unsupervised learning for
Speaker 1: finding patterns, and even causal inference trying to figure out
Speaker 1: cause and effect.
Speaker 2: Yeah, and the range of tools here is fascinating. You
Speaker 2: can forecast trends, categorize things, try to understand why something
Speaker 2: is happening. Each technique gives you a different lens on
Speaker 2: the data. A good practitioner knows which tool to pull
Speaker 2: out for which.
Speaker 1: Job, which brings us to the kinds of questions data
Speaker 1: science can actually answer prediction, classification, optimization like forecasting, sales spotting, fraud,
Speaker 1: finding efficient routes. But and I think this is really important.
Speaker 1: The book also points out the limitations. It's not magic, right.
Speaker 2: Being honest about that builds trust, manages xs. Sometimes the
Speaker 2: data just isn't there, or maybe the problem isn't really
Speaker 2: a data problem at all. Knowing what data science can't
Speaker 2: do is just as important as knowing what it can.
Speaker 1: They also talk a bit about team structure, like should
Speaker 1: you build your own team or outsource, and they stress
Speaker 1: how vital collaboration is across different departments. It seems like
Speaker 1: a data scientist working alone probably won't get very far.
Speaker 2: Oh absolutely not. Data science is inherently collaborative. It has
Speaker 2: to be. You need domain experts to frame the problem right,
Speaker 2: You need engineering support, you need buy in from leaders
Speaker 2: so the insights actually get used. It doesn't matter if
Speaker 2: the team is internal or external. Those connections are fundamental.
Speaker 1: Now. The book introduces this idea. I found really neat
Speaker 1: that the three pillars of knowledge for a data scientist. First,
Speaker 1: the core analytics stuff, stats, machine learning techniques, the tools. Second,
Speaker 1: domain knowledge plus collaboration, communication, leadership skills, and the third
Speaker 1: pillar that's big data management and the IT skills for
Speaker 1: the modern cloud world.
Speaker 2: Those three pillars really capture how multi fascinating data science
Speaker 2: is now. You can't just be good at one or two.
Speaker 2: You really need a solid base in all three to
Speaker 2: handle complex real world projects. Think about say, predicting customer churn.
Speaker 2: You need the analytics chops for the model, but you
Speaker 2: also need the domain knowledge to know which customer behaviors matter,
Speaker 2: and you need the IT skills to actually get and
Speaker 2: process all that data from the cloud.
Speaker 1: Okay, then the book gets into the actual project cycle.
Speaker 1: It breaks projects down by type like offline training, offline application,
Speaker 1: offline training, online application, online training, online application. It's interesting
Speaker 1: how the tech needs and the business value change depending
Speaker 1: on whether it's real time or batch.
Speaker 2: Yeah, that's a really practical way to categorize them. Knowing
Speaker 2: upfront if you need a weekly report versus say a
Speaker 2: real time recommendation engine on a website, well that changes everything.
Speaker 2: How you get data, how you prefit, model, test, deploy.
Speaker 2: It all follows from that.
Speaker 1: And the book really hammers home the importance of those
Speaker 1: early stages problem formulation and project planning. They stress using
Speaker 1: data in the planning, really understanding the business value and
Speaker 1: why data scientists have to be involved early. It helps
Speaker 1: avoid solving completely the wrong problem or setting totally unrealistic timelines.
Speaker 2: That's exactly where projects can go off the rails right
Speaker 2: at the start, spending that time up front to clearly
Speaker 2: define the business problem, figure out the desired outcome, make
Speaker 2: a realistic plan. It's foundational. Data scientists bring that unique perspective.
Speaker 2: They understand the business and what's actually possible with the data.
Speaker 1: And when they talk about project modeling, it's described as
Speaker 1: very iterative, not just picking a model. It involves all
Speaker 1: that hard work, data cleaning, wrangling, exploratory analysis to really
Speaker 1: get the data. Then translating the business problem into stats
Speaker 1: or machine learning terms. It's rarely finding the perfect model
Speaker 1: first try.
Speaker 2: That iterative nature is totally key. It's just how it works.
Speaker 2: You break down big problems into smaller analytical questions, apply
Speaker 2: different methods. You need feedback, loops, communication, You got to
Speaker 2: be willing to learn and adjust as you go.
Speaker 1: Finally, in the intersection, they flag a couple of super
Speaker 1: common mistakes. We mentioned solving the wrong problem, but the
Speaker 1: second one is underestimating timelines. They say that data exploration
Speaker 1: and prep, the unglamorous stuff, can eat up like sixty
Speaker 1: to eighty percent of the total project time.
Speaker 2: WHOA, Yeah, that number really hits home, doesn't it. It
Speaker 2: highlights all that hidden effort needed just to get raw,
Speaker 2: messy data ready for modeling. If you don't budget time
Speaker 2: for that wrangling and exploring properly, your project's almost certainly
Speaker 2: going to hit delays or worse, you build on shaky data.
Speaker 1: Okay, let's shift gears a bit and dig into some
Speaker 1: of the more technical details. Starting with data preprocessing. The
Speaker 1: book spends a good amount of time here, and well,
Speaker 1: like we just said, raw data is rarely model ready.
Speaker 1: Data cleaning is usually step one right, finding and dealing
Speaker 1: with weird stuff negative age percentages over one hundred. The
Speaker 1: book talks about different strategies like just deleting those rows
Speaker 1: or maybe treating them as missing values and imputing them later.
Speaker 2: And that's a strategic choice. When when do you delete
Speaker 2: versus impute? The book suggests if your data set's big
Speaker 2: enough and the bad data seems random, maybe deletion is okay,
Speaker 2: but imputation lets you keep more data. They cover simple
Speaker 2: methods mean median mode and more complex ones like Kenearest neighbors.
Speaker 2: The even point to the impute function in ours impute
Speaker 2: Missings package for.
Speaker 1: That right, and that leads straight into missing values generally,
Speaker 1: which are just everywhere in real data. Again, imputation is key.
Speaker 1: They detail the basic methods. kNN even mentioned maybe using
Speaker 1: bagging trees for imputation sometimes.
Speaker 2: Yeah, And the imputation method you choose, it can actually
Speaker 2: affect your model's performance down the line. Like the book says,
Speaker 2: simple mean imputation ignores relationships between variables and can kind
Speaker 2: of distort things, especially if lots of data is missing.
Speaker 2: More advanced methods try to use those relationships to make
Speaker 2: better guesses.
Speaker 1: Centering and scaling also get covered, basically getting all your
Speaker 1: variables onto a similar scale. The book mentions pre process
Speaker 1: in r's carrot package using center, and this is super
Speaker 1: important for lots of algorithms that are sensitive to how
Speaker 1: big the numbers are. Like imagine comparing height in centimeters
Speaker 1: and income in thousands of dollars, totally different scales. Right,
Speaker 1: some algorithms might just focus on the income because the
Speaker 1: numbers are.
Speaker 2: Bigger exactly, algorithms like gradient descent, which trains so many models,
Speaker 2: they just work better, converge faster when features are on
Speaker 2: a similar scale. It stops variables with big ranges from
Speaker 2: just dominating the learning process unfairly.
Speaker 1: Okay, Next up skewness and outliers. The book talks about
Speaker 1: using visualizations, box plots, histograms to spot these, and also
Speaker 1: statistical methods like Z scores or the modified Z score
Speaker 1: using the mad function in R. Finding these is important
Speaker 1: because they can really mess up certain models, like one
Speaker 1: huge income could totally skew the average and mislead a
Speaker 1: linear model.
Speaker 2: Yeah, and knowing how different models react to outliers is key.
Speaker 2: Linear regression logistic regression pretty sensitive based models usually more
Speaker 2: robust and the book rightly says outliers aren't always errors.
Speaker 2: They could be real just unusual, so deciding what to
Speaker 2: do remove transform leave alone needs sought. Maybe domain knowledge.
Speaker 2: They mentioned transformations like spatial sign in R that can
Speaker 2: kind of dampen the influence of outliers without removing them.
Speaker 1: Colinearity is another big one when your predictor variables are
Speaker 1: highly correlated with each other. The book points to find
Speaker 1: correlation in Carrot for finding these. If predictors are too correlated,
Speaker 1: it makes model coefficients unstable and hard to interpret, like
Speaker 1: trying to separate the effect of Facebook AdSpend from Instagram
Speaker 1: AdSpend if they always move together precisely.
Speaker 2: High multi collinearity inflates the variance of coefficient estimates in
Speaker 2: linear models makes it hard to see the independent effect
Speaker 2: of each variable, so you might remove one variable, combine them,
Speaker 2: or use dimensionality reduction techniques to handle it.
Speaker 1: They also cover sparse variables predictors that barely change across
Speaker 1: the data set, very low variance, near zero var in
Speaker 1: Carrot helps find these based on unique values and frequency ratios. Basically,
Speaker 1: if a variable is almost constant, it's not really helping
Speaker 1: your model tell things apart.
Speaker 2: Yeah, they're just not adding much information. Removing them can
Speaker 2: simplify the model, make it more stable, maybe train faster
Speaker 2: without really hurting performance.
Speaker 1: And the last preprocessing step mentioned is re encoding dummy variables.
Speaker 1: That's just converting categorical things like colors, product types into
Speaker 1: numbers usually binaries, zeros and one so algorithms can understand them.
Speaker 2: Fundamental step for categorical data creates those binary dummy variables
Speaker 2: so the model can tree each category is its own
Speaker 2: feature and learn its relationship to the outcome.
Speaker 1: Okay, shifting now to data wrangling, The book really highlights
Speaker 1: ours deeplayer package for manipulating data. They go through functions
Speaker 1: like select for picking columns, filter for rows, arrange for sorting,
Speaker 1: dot mutate for making new variables, and summarize with groupie
Speaker 1: for calculating stats across groups. They even give a customer
Speaker 1: segmentation example showing how you'd use these sociatyrize metrics for
Speaker 1: different customer types like average age, spending transaction counts for
Speaker 1: say conspicuous versus price conscious customers.
Speaker 2: Oh yeah, deeplayer really changed the game for data manipulation
Speaker 2: and r The syntax just so intuitive makes common tasks
Speaker 2: much clearer and more efficient. That customer segmentation example is
Speaker 2: great shows exactly how you use these tools to pull
Speaker 2: out meaningful insights about different groups in your data.
Speaker 1: They do you give a quick nod to base our
Speaker 1: functions too, like apply lapply supply, acknowledging that while deeplayer
Speaker 1: is great, sometimes you need the flexibility of the base
Speaker 1: functions for trickier stuff.
Speaker 2: Right. Deep player streamlines a lot, but basar gives you
Speaker 2: that fine grain control for maybe more complex or custom operations.
Speaker 2: It's good to know both, really, all.
Speaker 1: Right, let's talk model tuning. The book starts with the
Speaker 1: classic variance bias tradeoff, the idea that a really complex
Speaker 1: model might fit your training data perfectly low bias, but
Speaker 1: then it fails on new data because it learned the
Speaker 1: noise high variance overfitting, while a too simple model won't
Speaker 1: even capture the basic patterns high bias underfit. Tuning is
Speaker 1: finding that balance for good generalization.
Speaker 2: Yeah, that's like machine learning one oh one, isn't it?
Speaker 2: But absolutely crucial. You want the model to learn the
Speaker 2: real signal, not the random noise. Overfitting is like memorizing
Speaker 2: test answers. Great for that test useless. Otherwise, underfitting is
Speaker 2: like not studying at all, and.
Speaker 1: Data splitting and resampling are the main tools for managing
Speaker 1: this trade off. The book talks about the basic train
Speaker 1: to split, build on training data, evaluate on unseen test data.
Speaker 1: It also mentions a fancier technique, maximum dissimilarity sampling, using
Speaker 1: MAXDESEM and carrot. The goal there is to make the
Speaker 1: test set really diverse, covering more possibilities. Simple random splitting
Speaker 1: can sometimes give you unrepresentative train or test sets just
Speaker 1: by luck, which biases your performance estimate. Maximum dissimilarity sampling
Speaker 1: tries to build a test set that really spans the
Speaker 1: range of your data, giving a more robust evaluation of
Speaker 1: how the model might do in the wild. Then you
Speaker 1: have resampling methods for getting more stable performance estimates, especially
Speaker 1: with limited data. The book covers cross validation like kfold
Speaker 1: and bootstrapping. With kfold, you split data into k parts,
Speaker 1: train on K one, test on the last one, repeat
Speaker 1: four times and average the results. Gives a more reliable picture.
Speaker 2: Yeah, resampling is invaluable for confidence in your performance metrics.
Speaker 2: Cross validation avoids the risk of getting a misleading score
Speaker 2: just from one lucky or unlucky train to split. Bootstrapping
Speaker 2: involves resampling with replacement to create lots of simulated data
Speaker 2: sets than training and testing on those. It gives you
Speaker 2: a sense of the stability and uncertainty around your metrics.
Speaker 1: So how do we actually measure performance? The book says
Speaker 1: it depends if it's regression or classification. For regression predicting numbers,
Speaker 1: common metrics are URMC, ROOTMANE squared error tells you the
Speaker 1: average error size, and ARE squared the proportion of variants explained.
Speaker 1: Though they caution that high R squared isn't everything, and
Speaker 1: they mention adjusted R squared, which penalizes extra unhelpful predictors.
Speaker 2: Right armac is nice because it's in the same units
Speaker 2: as your target variable easy to grasp. Our square tells
Speaker 2: you how much better your model is than just guessing
Speaker 2: the average, but yeah, doesn't guarantee it's a good model
Speaker 2: or will generalize. Adjusted R squared pushes towards simpler models,
Speaker 2: which is often.
Speaker 1: Good for classification predicting categories. The book gets into the
Speaker 1: confusion matrix true positives, false positives, etc. N metrics like
Speaker 1: accuracy specificity, finding the true negatives and the Kappa statistic
Speaker 1: Kappa using Kappa dot test in rs FMSD package measures
Speaker 1: agreement beyond what you'd expect by chance.
Speaker 2: Useful, Yet the confusion matrix breaks it all down, not
Speaker 2: just if the model was right, but how it was wrong.
Speaker 2: Simple accuracy can be really misleading with unbalanced classes. If
Speaker 2: ninety nine percent or negative, a dumb model predicting negative
Speaker 2: all the time gets ninety nine percent accuracy. Specificity Kappa
Speaker 2: they give a much better picture, especially Kappa accounting for
Speaker 2: chance agreement.
Speaker 1: They also cover ROC curves and AUC area under the
Speaker 1: curve using proc dot rock in R that helps evaluate
Speaker 1: classifiers across different thresholds, and gain and lift charts, which
Speaker 1: are more business focused. They show how much better your
Speaker 1: model is at finding positive cases compared to just random selection.
Speaker 2: For marketing campaigns, RC curves visualize that trade off between
Speaker 2: finding true positives and avoiding false positives. As you change
Speaker 2: the decision threshold, higher AEC is generally better. GAT and
Speaker 2: lift charts translate that into business terms. How much more
Speaker 2: efficiently can you reach your target audience using the model?
Speaker 2: Very practical? Okay.
Speaker 1: Finally, the book walks through a bunch of different regression models.
Speaker 1: Start with the basics ordinary lease squares OLS, linear regression
Speaker 1: covers its assumptions, linearity, independence, constant error variants, normal residuals,
Speaker 1: and diagnostic plots to check them. Then moves to things
Speaker 1: like principal component regression PCR and partial lease squares PLS
Speaker 1: for handling many possibly correlated predictors.
Speaker 2: Understanding those OLS assumptions is so important for trusting the results.
Speaker 2: If they're violated, your coefficients and predictions might be off.
Speaker 2: Diagnostics help check that PCR and PLS are eight dimensionality
Speaker 2: reduction tools, especially when multiicolinearity makes standard linear aggression unstable.
Speaker 1: It also covers regularization methods Ridge, LASSO and elastic net.
Speaker 1: These shrink coefficients to prevent overfitting handle collinearity, and LASSO
Speaker 1: can even do feature selection by zeroing out some coefficients
Speaker 1: mentions in neet angle net.
Speaker 2: From there, yeah, regularization is super powerful for building more
Speaker 2: robust models, especially with lots of features. Ridge shrinks everything
Speaker 2: towards zero, last looking for some coefficients to zero doing
Speaker 2: automatic feature selection. Elastic net is a mix of both.
Speaker 1: Then tree based methods get introduced. Decision trees plus ensembles
Speaker 1: like bagging tree, bagging carrot, random forests are FING CARROT
Speaker 1: and gradient boosted machines GBM and CARROT. These are good
Speaker 1: for nonlinear patterns and handling different data types. Touches on
Speaker 1: splitting criteria to many information gain and pruning to avoid overfitting.
Speaker 2: Trees are incredibly versatile. Oftent top performers great at finding
Speaker 2: complex patterns without needing tons of feature engineering. Ensembles like
Speaker 2: random forests and gradium boosting combine many trees to get
Speaker 2: even better more stable predictions. Understanding splitting and pruning is
Speaker 2: key to making them work well.
Speaker 1: And lastly, a quick intro to deep learning. Feed Forward
Speaker 1: neural networks FFNNs, convolutional neural networks CNNs for images, Recurrent
Speaker 1: neural networks RNNs for sequences like text briefly covers applications,
Speaker 1: components like neurons, activation functions, sigmoid or LU layers, optimization,
Speaker 1: gradient descent ADAM, regularization dropout points to the CARAS package.
Speaker 1: In art.
Speaker 2: Deep learning has had amazing success, especially with images, language, speech.
Speaker 2: The book just gives a taste, but hits the core concepts,
Speaker 2: the building blocks, how they learn how to control them.
Speaker 2: It's a huge field, but that's a good starting point.
Speaker 1: So wrapping up this deep dive on the Practitioner's Guide
Speaker 1: to Data Science, it really feels like a valuable bridge. Yeah, definitely.
Speaker 1: And as you, our listener, think about all this, consider
Speaker 1: how these ideas might apply to what you're working on
Speaker 1: or learning about. Maybe you're prepping for a meeting, trying
Speaker 1: to understand a new area, or just curious that ability
Speaker 1: to work effectively with data it's just becoming so critical everywhere.
Speaker 1: Perhaps you're looking at customer behavior or analyzing trends for research.
Speaker 1: The kinds of practical steps and thinking outlined in this
Speaker 1: book offer a really solid way to approach those kinds
Speaker 1: of challenges. Something to think about.