Speaker 1: Okay, So have you ever felt like you're just drowning
Speaker 1: in information, you know, for a project, or maybe getting
Speaker 1: ready for a meeting, or even just trying to learn
Speaker 1: something new, and you just wish someone could kind of
Speaker 1: boil it all down.
Speaker 2: Yeah, just give you the essentials, right, what really.
Speaker 1: Matters exactly, and maybe, you know, throw in a few
Speaker 1: surprising bits to keep it interesting. Well, if that sounds
Speaker 1: like you, you are definitely in the right place, because
Speaker 1: that's what we do here on the Deep Dive. We're
Speaker 1: sort of your shortcut to getting properly informed, and today
Speaker 1: we're taking a deep dive into your source material. These
Speaker 1: are excerpts from Easily Practical Machine Learning Algorithms with Python
Speaker 1: by doctor Darren Thomas.
Speaker 2: Yeah, and this isn't like your standard dance textbook. The author,
Speaker 2: doctor Thomas, he's got a PhD, loads the teaching experience.
Speaker 1: And get this a background and saxophone performance, right, but.
Speaker 2: His passion for machine learning led him to use these
Speaker 2: algorithms in education. He's a lecturer now at Asia Pacific
Speaker 2: International University and.
Speaker 1: The book's aim, which is really really key for us today,
Speaker 1: is to be simple, easy to follow, a kind of
Speaker 1: condensed guide for actually using these algorithms with Python exactly.
Speaker 2: He even says his goal was always to show what
Speaker 2: to do, rather than talk a lot about how to
Speaker 2: do it. So less heavy theory, more hands on application.
Speaker 1: Okay, so important. Note, then, the book and this deep
Speaker 1: dive too, sort of assumes you've already got some background
Speaker 1: in Python. Maybe data science stats.
Speaker 2: Yeah, it's more for folks looking to build on existing skills,
Speaker 2: maybe not for absolute beginners to data science itself.
Speaker 1: Right, So our mission today we want to unpack some
Speaker 1: of the most common machine learning algorithms. We'll look at
Speaker 1: classification that's predicting.
Speaker 2: Categories like spam or not spam.
Speaker 1: Exactly, and numeric prediction predicting continuous values like say, house prices.
Speaker 1: Will get into how they're used, their surprising upsides, their challenges,
Speaker 1: and crucially, how you actually figure out if the models
Speaker 1: any good and how to make it better.
Speaker 2: Yeah, judging and improving them key.
Speaker 1: Totally, So ready to jump in. Let's unpack this.
Speaker 2: Let's do it, Okay.
Speaker 1: First up, decision trees. You can think of these as
Speaker 1: like the foundation for a lot of predictive stuff.
Speaker 2: Right At its heart, a decision tree is just a
Speaker 2: way to classify things or predict numbers by splitting your
Speaker 2: data up. Splitting it how It basically keeps dividing the
Speaker 2: sample into smaller and smaller groups, trying to make each
Speaker 2: little group as similar as possible inside.
Speaker 1: So it looks like a tree visually like a flow.
Speaker 2: Shart exactly like a float chart. You start at the
Speaker 2: top the root node. That's your first big decision point.
Speaker 2: Then you follow branches down through more decision nodes, making more.
Speaker 1: Splits until you hit the end the leaf nodes.
Speaker 2: Yep, the leaf nodes. That's where you get your final prediction.
Speaker 1: And how does it decide where to split?
Speaker 2: Well? For classification, it often uses something called entropy. High
Speaker 2: entropy means things are really mixed up. The tree tries
Speaker 2: to make splits that reduce that entropy, creating purer groups.
Speaker 1: Lower entropy, more homogeneous, got it. And for predicting numbers.
Speaker 2: Prediction that uses metrics like means squared error msee or
Speaker 2: maybe R square to guide the splits.
Speaker 1: Okay, so what's great about them? Why start here?
Speaker 2: Well? A big plus is flexibility. They handle missing data
Speaker 2: pretty well. They don't really care if your data isn't,
Speaker 2: you know, perfectly normal. A nice bell curve right, and
Speaker 2: you don't even have to use all your variables. Plus,
Speaker 2: and this is a big one, they're relatively easy to interpret.
Speaker 1: Ah, so you don't need a math PhD to figure
Speaker 1: out what it's doing.
Speaker 2: Pretty much. You can literally trace a path down the
Speaker 2: tree and see the reasoning that.
Speaker 1: Transparency sounds really useful, especially if you need to explain
Speaker 1: why a prediction was made.
Speaker 2: Absolutely imagine telling someone why their loan was denied. Showing
Speaker 2: them a simple tree is way easier than explaining you know,
Speaker 2: complex equations from some other models. Builds trust.
Speaker 1: Okay, but there's always a catch, right, what's the downside?
Speaker 2: The main one is that they can get really complex
Speaker 2: if you let them grow too much, really deep trees,
Speaker 2: and that leads to that often leads to overfitting.
Speaker 1: Overfitting like it learns the training data too well.
Speaker 2: Exactly, it fits the specific sample data perfectly, maybe even
Speaker 2: the noise. But then it can generalize well to new
Speaker 2: data it hasn't seen before.
Speaker 1: Like memorizing answers instead of understanding the concept perfect analogy.
Speaker 2: And of course, a super complex tree, even though it's visual,
Speaker 2: can still be hard to explain easily.
Speaker 1: Okay, let's make it concrete. The source had an example, right,
Speaker 1: a cancer data.
Speaker 2: Set, Yeah, predicting health status alive or dead. The model
Speaker 2: mainly used variables like time and age for its splits.
Speaker 1: Then how did it perform?
Speaker 2: It got seventy eight percent accuracy on the data it
Speaker 2: trained on, okay, but then on the unseen test data
Speaker 2: it drops slightly to seventy three percent.
Speaker 1: That drop five percent is that bad or expected?
Speaker 2: That's actually pretty common and often expected. It shows it's generalizing. Okay,
Speaker 2: still performing decently on new stuff. It learned patterns and
Speaker 2: applied them.
Speaker 1: Right now, What about using it for numeric prediction, same
Speaker 1: data set, but predicting aid yep.
Speaker 2: So here, instead of looking at purity like with dagony,
Speaker 2: the tree uses mse means squared error in the nodes,
Speaker 2: and the leaves predict the average age for that group.
Speaker 1: And which variables were important there.
Speaker 2: The source mentioned pH dot Carno Andmeal dot.
Speaker 1: Cow and the results.
Speaker 2: Well, the correlation between the actual age and predicted age
Speaker 2: was okay on the training data about point five to
Speaker 2: four moderate moderate yeah, But on the test data it
Speaker 2: dropped way down to point one eight.
Speaker 1: Ouch, big drop? What about the error the.
Speaker 2: MSc MC was sixty one point eight on the training set,
Speaker 2: but jumped up to eighty five point twenty four on
Speaker 2: the test set.
Speaker 1: So again, that drop tells us.
Speaker 2: It tells us while it learned something, it really struggled
Speaker 2: to generalize the age prediction to new people. Highlights that
Speaker 2: overfitting risk with single trees, which.
Speaker 1: Leads us perfectly into the next one. Random forest. This
Speaker 1: sounds cool.
Speaker 2: Yeah, this is where it gets really interesting. Random Forest
Speaker 2: tackles that overfitting problem head on. How instead of building
Speaker 2: just one decision tree, it builds hundreds, maybe even thousands
Speaker 2: of them.
Speaker 1: Wow? Okay, where does the random part come?
Speaker 2: In? Two places? First, each tree is built using a
Speaker 2: different random sample of your data drawn with replacement called
Speaker 2: boots trapping. Second, at each split point in a tree,
Speaker 2: it only considers a random subset of your available features.
Speaker 1: So not every tree sees all the data, and not
Speaker 1: every split considers all the factors exactly.
Speaker 2: And the idea is you get lots of slightly different trees,
Speaker 2: none of them perfect, but hopefully their errors are kind
Speaker 2: of random and cancel each other out.
Speaker 1: So how does it make a final prediction with all
Speaker 1: those trees?
Speaker 2: It's pretty democratic. Actually, for classification, it's just a majority
Speaker 2: vote whichever prediction most trees make.
Speaker 1: Wins simple enough, and for predicting numbers.
Speaker 2: It just averages the predictions from all the individual trees.
Speaker 1: And this whole wisdom of the crowd thing really helps
Speaker 1: with overfitting massively.
Speaker 2: That aggregation step makes the model much more robust and
Speaker 2: way less prone to overfitting compared to a single complex
Speaker 2: decision tree, so.
Speaker 1: The benefits seem clear less. Overfitting works well even if
Speaker 1: you don't have tons of data handles missing values.
Speaker 2: Sounds great, It often is. It's a very popular, reliable
Speaker 2: algorithm for those reasons.
Speaker 1: Okay, but the drawback you mentioned transparency with decision trees,
Speaker 1: what about here?
Speaker 2: Oh yeah, that's the main trade off. With potentially thousands
Speaker 2: of trees. You can't just visualize it like a single
Speaker 2: flow chart.
Speaker 1: Anymore, so it becomes a black box pretty much.
Speaker 2: You know what goes in, you know what comes out,
Speaker 2: but explaining exactly how it arrived at that specific prediction
Speaker 2: is really hard.
Speaker 1: That's a problem when.
Speaker 2: When you need to explain the why, like we said,
Speaker 2: loan applications, medical diagnoses, if you can't explain the reasoning,
Speaker 2: it can cause issues with trust or even regulations that
Speaker 2: demand transparency.
Speaker 1: Right, so that lack of interpretability is a real consideration.
Speaker 1: You might get great predictions but lose the explanation.
Speaker 2: It's a definite trade off you have to weigh.
Speaker 1: Let's look at the example the doctor Aus data set
Speaker 1: predicting gender.
Speaker 2: Right, the source noted income and age came out as
Speaker 2: strong predictors, pointing out the known differences in salaries and
Speaker 2: life expectancy. And the performance impressive on the training data
Speaker 2: ninety three percent accuracy. Wow, but then quite a big
Speaker 2: drop on the test data, down to sixty six percent.
Speaker 1: Oo. That's a nearly thirty percent draw. What does that
Speaker 1: tell us?
Speaker 2: It tells us that while the model learned the training
Speaker 2: data extremely well, almost perfectly, it really struggled to generalize
Speaker 2: that learning.
Speaker 1: So even random forest isn't immune to some overfitting. Or
Speaker 1: maybe the training data just wasn't fully representative.
Speaker 2: Could be either or both. It's a stark reminder that
Speaker 2: high training accuracy is nice, but test accuracy is what
Speaker 2: really counts for real world use.
Speaker 1: Okay, and the numeric prediction example predicting income from the
Speaker 1: same doctor Aus data.
Speaker 2: Set here age was the most important variable by far
Speaker 2: makes intuitive sense, right. Yeah, people often earn more as
Speaker 2: they get older.
Speaker 1: Sure, and the numbers.
Speaker 2: Strong correlation on the training data point eight three, but
Speaker 2: again a drop on the test data down to point
Speaker 2: four to eight.
Speaker 1: Still a decent drop, and the error ms.
Speaker 2: MSc was low on the training set point zero four
Speaker 2: and higher on the test set point one to one.
Speaker 1: So similar story. Good at learning the training data, but
Speaker 1: only moderately good at generalizing the income prediction.
Speaker 2: Yeah, pretty much. It captures the relationship it sees, but
Speaker 2: applying it to new unseen individuals is where the real
Speaker 2: test lies.
Speaker 1: Okay, moving on K nearest neighbor or k NN, this
Speaker 1: one sounds neighborly, huh.
Speaker 2: Yeah, it's actually quite intuitive. The core idea is predicting
Speaker 2: by proximity, like.
Speaker 1: Your example of walking into a classroom of twelve year olds.
Speaker 1: If another kid walks in, you guess they're also twelve.
Speaker 2: Exactly like that, Yeah, kNN looks at an unknown data
Speaker 2: point and finds the k known data points that are
Speaker 2: nearest to it in the feature space.
Speaker 1: And nearest is usually measured by.
Speaker 2: Typically Euclidian distance, just the straight line distance between points
Speaker 2: in that multi dimensional space.
Speaker 1: So K is just how many neighbors you look at
Speaker 1: like the three nearest or five nearest.
Speaker 2: Yep, K is the number of you consider.
Speaker 1: And how do those neighbors make the prediction for classification?
Speaker 2: They vote. That's why K is usually an odd number
Speaker 2: to avoid ties. For numeric prediction, it's just the average
Speaker 2: of the neighbor's values.
Speaker 1: What's it good for.
Speaker 2: It's pretty good with non linear data where the boundary
Speaker 2: is in a straight line. And it's non parametric, meaning
Speaker 2: it doesn't make strong assumptions about how your data is distributed.
Speaker 2: Makes it flexible.
Speaker 1: Sounds simple enough. Any hidden traps.
Speaker 2: Well, it's sometimes called a lazy learning algorithm. Lazy because
Speaker 2: it doesn't really build a model during training. It basically
Speaker 2: just stores all the training data. The real work happens
Speaker 2: only when you ask for a prediction.
Speaker 1: Ah. So it doesn't give you much insight into why
Speaker 1: variables are important.
Speaker 2: Not really no abstraction. And because it stores everything, it
Speaker 2: can struggle with really large data sets. Needs a lot
Speaker 2: of memory.
Speaker 1: And there was another crucial point something about scale.
Speaker 2: Yes, critically important for K and N it is scale sensitive.
Speaker 1: Okay, break that down. What does scale sensitive mean? Practically?
Speaker 2: Imagine you have age maybe zero to one hundred, and
Speaker 2: another variable like own scar, which is just zero or one.
Speaker 2: When Cainean calculates distance, the age difference will totally swamp
Speaker 2: the own scar difference, just because the numbers are so
Speaker 2: much bigger.
Speaker 1: So age will have way more influence on who's considered nearest.
Speaker 2: Exactly, even if owning a car is actually super important
Speaker 2: for the prediction. Yeah, so you have to scale your
Speaker 2: data first.
Speaker 1: How do you scale it?
Speaker 2: Common ways are a minmax scaling where you squish everything
Speaker 2: into a zero to one range, or standardization where you
Speaker 2: give variables a mean of zero and standard deviation of one.
Speaker 2: It puts everything on a level playing.
Speaker 1: Field, right, So age isn't shouting louder than the other variables.
Speaker 1: Makes sense. The example use the turnout data set to
Speaker 1: predict if someone voted yeah.
Speaker 2: And they specifically mentioned scaling the data first. The model
Speaker 2: got almost eighty percent accuracy on training, and importantly, it
Speaker 2: held up really well on the test data too.
Speaker 1: That consistency is good, right, suggests it's generalizing.
Speaker 2: Exactly what you want to see, not just memorizing.
Speaker 1: And for predicting income from that same turnout.
Speaker 2: The correlation was zer point six y three on training
Speaker 2: dropped a bit too. Point four to eight on test,
Speaker 2: but the MSc values were very close point zero two
Speaker 2: to one for training, point zero two nine for testing.
Speaker 1: So again, even if the correlation isn't super strong, the
Speaker 1: similar error rates suggest it's performing consistently on new data.
Speaker 2: YEP indicates stable performance, which is often more important than
Speaker 2: hitting the absolute highest correlation number.
Speaker 1: Okay, Next algorithm, Support vector machines SVM. This sounds a
Speaker 1: bit more complex.
Speaker 2: It combines ideas from K and N and linear models,
Speaker 2: but with a really clever twist for classification, which is
Speaker 2: its main goal is to find the best dividing line
Speaker 2: or plane or hyperplane in higher dimensions.
Speaker 1: Hyperplane fancy word for boundary.
Speaker 2: Right. It wants the boundary that creates the biggest possible
Speaker 2: gap or margin between the different classes.
Speaker 1: A bigger buffer zone exactly.
Speaker 2: And the data points that sit right on the edge
Speaker 2: of that margin. Those are the support vectors. They're the
Speaker 2: critical ones that actually define the boundary.
Speaker 1: Interesting, does it always have to be a straight line?
Speaker 1: What if the data is all mixed up?
Speaker 2: Good question. It prefers straight lines, but it has tricks.
Speaker 2: It can allow some misclassifications using a slack variable, a
Speaker 2: bit of wiggle room, or for really messy nonlinear data.
Speaker 2: It uses the kernel trick.
Speaker 1: The kernel trick sounds like magic, It kind of is.
Speaker 2: It projects the data into much higher dimensional space where
Speaker 2: hopefully a simple linear boundary can separate the classes.
Speaker 1: Like unfolding a crumpled paper to separate dots. I like
Speaker 1: that analogy. So sbms are flexible, can handle messy data.
Speaker 2: Very flexible, Yeah, I think can be incredibly accurate even
Speaker 2: on complex problems.
Speaker 1: Okay, sounds powerful. Downside, is it another black box?
Speaker 2: Often? Yes, especially with those kernel tricks and higher dimensions.
Speaker 2: Explaining why SBM made a particular decision gets very abstract, very.
Speaker 1: Quickly right, hard to explain to the boss.
Speaker 2: Can be Also, choosing the right kernel isn't always obvious,
Speaker 2: And like K and N, it's scales sensitive. You need
Speaker 2: to rescale your data.
Speaker 1: Got it scaling again. The example was predicting mortgage status.
Speaker 1: Yes know from the working hours data.
Speaker 2: Set correct and data prep was key. They combined some
Speaker 2: child related variables and rescaled everything. Then they compared two
Speaker 2: common kernels, linear and RBF radio basis function.
Speaker 1: Oh do they do?
Speaker 2: The linear kernel hit eighty seven percent accuracy on training
Speaker 2: and impressively held that exact same accuracy on the test data.
Speaker 1: Wow, perfect generalization in that case.
Speaker 2: Fantastic result. The RBF kernel was just slightly lower and.
Speaker 1: For SVM regression predicting education level from the same data set.
Speaker 2: Yeah. Again. Comparing linear and RBF kernels, the linear one
Speaker 2: showed a pretty weak correlation, only point three to eight
Speaker 2: on training and point four zero on tests. That doesn't
Speaker 2: sound great, but the MS values were low and very
Speaker 2: stable point zero one five eight nine on training, point
Speaker 2: zero one eight three to two on testing.
Speaker 1: So weak relationship overall, but the model makes consistent predictions exactly.
Speaker 2: Suggests it's generalizing well, even if it's not explaining huge
Speaker 2: amount of the variants. The source also mentioned outliers might
Speaker 2: be affecting the correlation metric more here.
Speaker 1: Interesting. So metrics can sometimes tell slightly different stories.
Speaker 2: Definitely, you need to look at them together, all right.
Speaker 1: Artificial neural networks ann's the brain inspired ones kind of.
Speaker 2: They were initially inspired by biological neurons. Yeah. You have inputs,
Speaker 2: some processing happens, and you get outputs.
Speaker 1: And deep learning is just when you have lots of
Speaker 1: layers of these neurons.
Speaker 2: Multiple hidden layers. Yes, that's the essence of deep learning.
Speaker 1: So how do they actually work? Simply put, think.
Speaker 2: Of inputs like signals arriving. Each input gets a weight
Speaker 2: how important it is. They get summed up and then
Speaker 2: hit an activation function like.
Speaker 1: The neuron deciding whether to fire.
Speaker 2: Sort of yeah, that function decides if the signal is
Speaker 2: strong enough to pass on to the next layer. Usually
Speaker 2: the information flows one way feed forward. It's a whole
Speaker 2: cascade of these simple weighted sums and activations.
Speaker 1: And the big advantage why all the hype, They.
Speaker 2: Really shine with massive amounts of data. Given enough data,
Speaker 2: they can learn incredibly complex suttle patterns that other algorithms
Speaker 2: might miss entirely.
Speaker 1: So flexibility is huge. Image recognition, self driving cars.
Speaker 2: Exactly, they tower a lot of cutting edge AI tasks.
Speaker 1: But the catch they need tons of data.
Speaker 2: Typically, yes, massive data sets for optimal performance, and training
Speaker 2: them can take a lot of computing power and time.
Speaker 2: Plus sometimes simpler networks can struggle to converge, meaning they
Speaker 2: don't actually learn effectively.
Speaker 1: Okay, example time predicting union membership from the wages data set.
Speaker 2: Right, And a key step here was turning categorical variables
Speaker 2: like occupation into dummy.
Speaker 1: Variables numerical flags basically.
Speaker 2: Yep, zeros and ones. The network can understand. The model
Speaker 2: achieved a solid seventy percent accuracy, and importantly, it was
Speaker 2: consistent between the training and test data.
Speaker 1: Good stability again and the regression example predicting wages.
Speaker 2: Also from the wages data set, it showed a moderate
Speaker 2: correlation point five to six on training, very close point
Speaker 2: five to four odd tests, and the MSS values were
Speaker 2: also very so between the two.
Speaker 1: Sets, So reasonably good generalization there too. Consistent if not
Speaker 1: spectacular prediction.
Speaker 2: Looks like it reliable performance on new data.
Speaker 1: Now for something completely different, k means you said this
Speaker 1: one's unsupervised learning.
Speaker 2: That's right. This is fascinating because, unlike everything else we've discussed,
Speaker 2: there's no right answer or target variable we're trying.
Speaker 1: To predict, no gender, no income, no voted label exactly.
Speaker 2: KMES isn't trying to predict anything specific. It's just trying
Speaker 2: to find natural groupings or clusters within the data itself,
Speaker 2: based on similarity.
Speaker 1: How does it find these groups?
Speaker 2: It starts by randomly guessing the locations of K cluster
Speaker 2: centers or centroids, K.
Speaker 1: Being the number of clusters you think are in the data.
Speaker 2: Precisely. Then it assigns each data point to its nearest centroid.
Speaker 2: After that, it recalculates the position of each centroid to
Speaker 2: be the actual center of all the points assigned to it.
Speaker 1: And it repeats that. Assigned points move centers yep.
Speaker 2: It iterates back and forth, assigned points update centroids until
Speaker 2: the centroids start moving much until things stabilize, and.
Speaker 1: The researcher has to decide on K the number of clusters.
Speaker 1: That sounds tricky, it can be.
Speaker 2: It's a key challenge.
Speaker 1: So what's the big benefit of doing this.
Speaker 2: It's fantastic for exploring your data, for segmentation, finding hidden
Speaker 2: patterns you didn't even know existed, understanding what makes different
Speaker 2: subgroups within your data.
Speaker 1: Distinct, discovering natural segments exactly.
Speaker 2: But the drawbacks stem from that unsupervised nature. Since there's
Speaker 2: no right answer, evaluating how good the clusters are is
Speaker 2: more subjective, relying on the researcher's interpretation.
Speaker 1: Hell me guess scale sensitive.
Speaker 2: You got it requires data normalization or scaling, just like
Speaker 2: kNN and SVM because it relies on distance calculations, and yeah,
Speaker 2: choosing the right K is tough.
Speaker 1: Are there ways to help choose K.
Speaker 2: There are methods, yeah, like the elbow method, where you
Speaker 2: plot a measure of cluster cohesion against different values of
Speaker 2: K and look for an elbow point where adding more
Speaker 2: clusters doesn't improve things much.
Speaker 1: Okay, the example used the act SAT scores age education.
Speaker 2: Yes, and they stressed normalizing the data first, because scores,
Speaker 2: age and education level are all on different scales.
Speaker 1: Did the elbow method work?
Speaker 2: It suggested K two two clusters, but the author actually
Speaker 2: decided to go with K three, looking for maybe a
Speaker 2: bit more detail in the groupings.
Speaker 1: And what did those three clusters reveal.
Speaker 2: Pluster zero had the oldest students, most education, highest test
Speaker 2: scores generally, Cluster one was kind of medium across the board.
Speaker 2: It's second oldest, second highest education, and Cluster two was
Speaker 2: much younger with weaker test performance.
Speaker 1: Seems logical age and achievement grouping together.
Speaker 2: Yeah, But here's the kicker, the really surprising part. When
Speaker 2: they looked closer, what did they find? A really clear
Speaker 2: separation by gender emerged Cluster one all males, Cluster zero
Speaker 2: and two all females.
Speaker 1: WHOA, The algorithm wasn't told about gender, right, It just
Speaker 1: found that pattern exactly.
Speaker 2: It revealed this underlying structure, and it helped explain things
Speaker 2: like why cluster one one all males had particularly high
Speaker 2: quantitative set scores tying into statistical trends.
Speaker 1: That's amazing. So kymines didn't just group by scores. It
Speaker 1: uncovered this fundamental demographic split. What does that tell us? Generally?
Speaker 2: It shows how these unsupervised methods can reveal really deep insights,
Speaker 2: sometimes biases or strong correlations we weren't even looking for.
Speaker 2: It forces you to ask why the data is structured
Speaker 2: that way.
Speaker 1: So the final interpretation was cluster.
Speaker 2: Zero older educated women, cluster one young males, Cluster two
Speaker 2: very young women, a much richer picture than just high medium,
Speaker 2: low scores.
Speaker 1: Incredible. Okay, so we've seen all these algorithms, but just
Speaker 1: getting an answer isn't enough. We need to know if
Speaker 1: the answer is good.
Speaker 2: Assessing models absolutely critical, otherwise you're just generating numbers without
Speaker 2: knowing if they're meaningful.
Speaker 1: And for classification, the go to tool is the confusion matrix.
Speaker 2: Definitely a fundamental starting point. It just lays out your
Speaker 2: predictions against the actual truth in a simple table.
Speaker 1: Remind us of the four boxes you've.
Speaker 2: Got true negatives TN correctly saying some thing isn't there.
Speaker 2: False negatives FN missing something that is there. Big problem
Speaker 2: in medical tests.
Speaker 1: For instance, right false positives.
Speaker 2: FP seeing something is there when it isn't, like a
Speaker 2: spam filter blocking a real email.
Speaker 1: And true positives TP.
Speaker 2: Correctly identifying something that is there the spam filter catching
Speaker 2: actual spam.
Speaker 1: So the cancer decision tree example had four ten zero FN,
Speaker 1: twenty eight FP, and eighty four TP. Lots of numbers.
Speaker 1: If I'm building, say a system to detect faulty products
Speaker 1: on an assembly line, which metrics should I care most about?
Speaker 2: Ooh, good question. For faulty products, you probably care a
Speaker 2: lot about. Recall also called sensitivity. That's true positives divided
Speaker 2: by all the actual positives TP plus FN. You want
Speaker 2: to catch as many faulty products as possible, even if
Speaker 2: you accidentally flag a few good ones. Missing a faulty
Speaker 2: one could be costly or dangerous.
Speaker 1: So recall is about minimizing misses. For the cancer example,
Speaker 1: recall was point seventy six yes.
Speaker 2: And precision, which is true positives divided by all the
Speaker 2: ones the model predicted is positive TP plus FP was
Speaker 2: also point seven to six. Precision is about how trustworthy
Speaker 2: A positive prediction is when the model says it's cancer.
Speaker 2: How often is it right? And the F measure that
Speaker 2: just combines precision and recall into one score also point
Speaker 2: seventy six in this case because there were zero false negatives,
Speaker 2: which is unusual.
Speaker 1: What about the negative specificity?
Speaker 2: Specificity is true negatives divided by all the actual negatives
Speaker 2: tmplus FP. It measures how well the model identifies the
Speaker 2: true negatives. For the cancer example, it was only point
Speaker 2: one three, quite poor, meaning it wasn't good at correctly
Speaker 2: identifying healthy people as healthy.
Speaker 1: Okay, so different metrics matter depending on the goal. What
Speaker 1: about plain old accuracy?
Speaker 2: Accuracy is just tmplus TP divided by the total all
Speaker 2: the correct predictions. It was seventy six percent here. But
Speaker 2: accuracy can be misleading, especially if one class is way
Speaker 2: more common.
Speaker 1: Than the other, like if ninety nine percent of emails
Speaker 1: aren't spam. A model predicting not spam all the time
Speaker 1: gets ninety nine percent accuracy, but is useless Exactly.
Speaker 2: That's why you need these other metrics. An error is
Speaker 2: just one minus accuracy, so twenty four percent here.
Speaker 1: And finally, kappa.
Speaker 2: Kappa measures accuracy but accounts for how much agreement you'd
Speaker 2: expect just by chance, closer to one is better. Here
Speaker 2: was point one seven to one, which is pretty low.
Speaker 2: Suggests the model's performance wasn't much better than random guessing
Speaker 2: once you factor chance in. Wow.
Speaker 1: Okay, so looking at all of them gives a much
Speaker 1: fuller picture.
Speaker 2: Definitely.
Speaker 1: Now the ROC curve this is a cool backstory. Wwii.
Speaker 2: Yeah. Radar engineers used it to figure out if a
Speaker 2: blip was an enemy plane a true positive or just
Speaker 2: noise a false positive. High stake stuff.
Speaker 1: Now we use it for models.
Speaker 2: Yep. It plots the true positive rate against the false
Speaker 2: positive rate at various thresholds. Ideally, you want the curve
Speaker 2: to shoot up towards the top left corner high true positives,
Speaker 2: low false positives.
Speaker 1: A diagonal line is bad like random guessing.
Speaker 2: Exactly the area under this curve AUC gives a single number,
Speaker 2: zero to one. The cancer model had an AEC of
Speaker 2: zero point seven nine, which is generally considered acceptable. Maybe
Speaker 2: fairy good okay.
Speaker 1: Another key technique cross validation. This is for checking generalizability.
Speaker 2: Right. Instead of just one train test split, you divide
Speaker 2: your training data into say five or ten folds like slices. Yeah.
Speaker 2: Then you train the model five times. Each time, you
Speaker 2: train on four folds and test on the one fold
Speaker 2: left out.
Speaker 1: Using a different slice for testing each time exactly.
Speaker 2: Then you average the results from those five tests. It
Speaker 2: gives you a much better idea of how stable the
Speaker 2: performance is and how well it's likely to perform on
Speaker 2: genuinely new data.
Speaker 1: And the real test set, the one you held back
Speaker 1: at the start.
Speaker 2: Crucially, you only touch that once, right at the very end,
Speaker 2: after you've done all your model selection and tuning, using
Speaker 2: cross validation on the training data, don't peak.
Speaker 1: Got it? For the cancer decision tree, cross validation showed
Speaker 1: about seventy three percent accuracy, but with a standard deviation
Speaker 1: of eleven percent.
Speaker 2: Yeah, that standard deviation tells you there was quite a
Speaker 2: bit of variability and performance across the different folds. Maybe
Speaker 2: do the small sample size making the full quite different
Speaker 2: from each other.
Speaker 1: Okay, shifting to assessing regression models predicting numbers. No confusion
Speaker 1: matrix here, right.
Speaker 2: Right, we use different metrics. You'd start by comparing basic dats,
Speaker 2: mean standard deviation, core tiles of the actual values versus
Speaker 2: your predicted values. Ideally they should look pretty similar.
Speaker 1: Like for the age prediction the means were close, but
Speaker 1: standard deviations.
Speaker 2: Differed a bit. Hey exactly. Then you look at the
Speaker 2: correlation between actual and predicted for age that was point
Speaker 2: five to four on the training set.
Speaker 1: We also saw means squared error MS.
Speaker 2: Lower is better, lower is better, Yes, for age, it
Speaker 2: was sixty one point eight on train eighty five point
Speaker 2: twenty four on test. The different shows that drop in
Speaker 2: performance on unseen data and are squared. Our squared tells
Speaker 2: you how much of the variation in the outcome variable
Speaker 2: your model explains. Ranges from zero to one. For age prediction,
Speaker 2: that was point twenty nine, which the source called not exciting.
Speaker 2: It means the model only explained about twenty nine percent
Speaker 2: of the variation in age.
Speaker 1: And cross validation applies here too.
Speaker 2: Absolutely, you'd cross validate metrics like MSE and R two
Speaker 2: for that SVM regression predicting education. The cross validated MS
Speaker 2: was similar to the original test MAC, which is good,
Speaker 2: But the R two, the cross validated R two had
Speaker 2: a high standard deviation point one nine. So I guess
Speaker 2: the model's ability to explain the variance wasn't very stable
Speaker 2: across different subsets of the data. Again, maybe sample size
Speaker 2: issues or just a complex relationship.
Speaker 1: Okay, we've built models, we've assessed them now making them better.
Speaker 1: We know we can get more data, change variables, switch algorithms,
Speaker 1: But what about tuning.
Speaker 2: Ah hyper parameter tuning. This is really interesting. It's about
Speaker 2: tweaking the settings of the algorithm itself before it starts learning.
Speaker 1: So not changing the data, but changing how the algorithm
Speaker 1: learns from the data exactly.
Speaker 2: Things like the K and kN N, or the C
Speaker 2: penalty termin SVM, or how deep you let a decision
Speaker 2: tree grow. These are hyper.
Speaker 1: Parameters, and how do you tune them?
Speaker 2: Guessing more systematic than that, Usually you define a grid
Speaker 2: of possible values for the hyper parameters you want to tune,
Speaker 2: like Trika values from one to twenty, try different distance.
Speaker 1: Metrics, and the computer tries all the combinations.
Speaker 2: YEP, often using cross validation within the grid search. It
Speaker 2: runs models for all combinations and tells you which set
Speaker 2: of hyper parameters gave the best average performance on the
Speaker 2: cross validation folds. It's a bit a trial and error
Speaker 2: guided by data.
Speaker 1: Let's look at the kNN tuning example. Original model predicting
Speaker 1: voted was around seventy nine percent accurate.
Speaker 2: Right, the tuned K one thirteen the weights uniform versus
Speaker 2: distance based and the distance metric Manhattan versus Kowski clidion.
Speaker 2: That created ninety six different kNN models to test.
Speaker 1: Wow, and the best one.
Speaker 2: The best combination found through cross validation, achieved seventy four
Speaker 2: percent accuracy.
Speaker 1: That's lower than the original seventy nine percent.
Speaker 2: On cross validation. Yes, but here's the crucial part. When
Speaker 2: they tested that tuned model back on its own training data,
Speaker 2: its accuracy jumped to ninety nine point seven percent.
Speaker 1: WHOA nearly perfect. But that sounds suspicious.
Speaker 2: I really suspicion. That huge gap between the near perfect
Speaker 2: training accuracy and the seventy four percent cross validated accuracy
Speaker 2: screams overfitting. The tuning process found settings that basically memorize
Speaker 2: the training data.
Speaker 1: So this is a massive warning sign about just looking
Speaker 1: at training accuracy, especially after tuning.
Speaker 2: Absolutely reinforces why cross validation and that final untouched test
Speaker 2: set are non negotiable.
Speaker 1: Okay, and the SVM regression tuning predicting education original mESC
Speaker 1: was about point zero one five eight nine.
Speaker 2: They tune the c's cost parameter, the kernel type linear
Speaker 2: versus RBF and degree for polynomial kernels and the result
Speaker 2: the best combo nudged the MSE down slightly to point
Speaker 2: zero one, five, three, eight.
Speaker 1: A tiny improvement worth it.
Speaker 2: Depends entirely on the context. Sometimes shaving even a tiny
Speaker 2: bit off the air can be hugely valuable. It shows
Speaker 2: the potential, even if it's not always dramatic.
Speaker 1: Finally, taking it one step further, combining algorithms the ensemble
Speaker 1: approach or stacking.
Speaker 2: Yes, stacking is a pretty sophisticated way to combine models.
Speaker 2: The basic idea is you train several different models, like
Speaker 2: a random forest, a SVM, a kNN okay. Then instead
Speaker 2: of just averaging their outputs, you use their predictions as
Speaker 2: inputs for a final metal model.
Speaker 1: So a model that learns from the predictions of other.
Speaker 2: Models exactly, it learns how to best combine the strengths
Speaker 2: and potentially correct the weaknesses of the base models to
Speaker 2: make a final, hopefully better prediction.
Speaker 1: Trying to create a supermodel.
Speaker 2: That's the goal maximize strengths, minimize weaknesses.
Speaker 1: The example used the doctor Aus data set again predicting gender,
Speaker 1: combining random forest, SVM, and kNN YEP individually.
Speaker 2: Cross validation showed random forest was the best performer on
Speaker 2: its own well, SVM struggled a.
Speaker 1: Bit, So did stacking them beat the random forest?
Speaker 2: Interestingly? No, No, the initial stacked model actually performed worse
Speaker 2: than just using the random forest by itself.
Speaker 1: Huh So more complex isn't always better?
Speaker 2: Definitely not automatically. Just throwing models together can sometimes add
Speaker 2: noise or compound errors.
Speaker 1: Did they try tuning the models within the stack?
Speaker 2: Did they tuned hyper parameters for the random forest, the SVM,
Speaker 2: and the kN N within the ensemble.
Speaker 1: Structure and the final result after all that work, The.
Speaker 2: Final tuned ensemble model landed at about seventy percent accuracy
Speaker 2: on both training and test data.
Speaker 1: So after all that stacking and tuning, it didn't really
Speaker 1: improve much, if at all, over the simpler single random
Speaker 1: forest model.
Speaker 2: Pretty much in this specific case, the added complexity didn't
Speaker 2: yield significantly better performance.
Speaker 1: That's a really important takeaway, A huge one. Wow. Okay,
Speaker 1: we've covered a lot of ground today, from decision trees
Speaker 1: kind of the building block.
Speaker 2: To random forests using the wisdom of the.
Speaker 1: Crowd, kNN using proximity, SVM finding that optimal boundary an
Speaker 1: n's sort of mimicking the brain, then.
Speaker 2: K meanes finding hidden groups without even being told what
Speaker 2: to look.
Speaker 1: For and crucially, how to actually measure if these models
Speaker 1: are any good using things like the confusion matrix, RC curves,
Speaker 1: cross validation.
Speaker 2: And how to potentially improve them through careful hyper parameter
Speaker 2: tuning and even stacking them together.
Speaker 1: And remember, listener, these aren't just you know, abstract ideas.
Speaker 1: This deep dive should give you a real foundation for
Speaker 1: understanding how these tools work, yeah.
Speaker 2: Their quirks, their strengths, their weaknesses, and importantly, how to
Speaker 2: think critically about their results and choose the right approach
Speaker 2: for your specific problem.
Speaker 1: And that final point from the ensemble example really sticks
Speaker 1: with me. As the source material concludes, and this is powerful.
Speaker 1: Complexity is not a cure all for better performance, and
Speaker 1: simple is almost always better.
Speaker 2: It's a fantastic lesson, it really is.
Speaker 1: And it leaves us with a question for you to
Speaker 1: maybe mull over in whatever you're working on, whether it's
Speaker 1: data or just a life, where might you be over
Speaker 1: complicating things? Where could a simpler approach actually lead to clearer,
Speaker 1: more powerful insights.
Speaker 2: Something to think about.