Speaker 1: Okay, let's unpack this today. We're embarking on a deep
Speaker 1: dive into the fascinating, sometimes maybe often hyped, but fundamentally
Speaker 1: important world of data science.
Speaker 2: Definitely hyped at times, right, But.
Speaker 1: Our mission here is really to demystify what data science
Speaker 1: truly is. We want to explore its core processes, the
Speaker 1: essential tools, and also confront some of the well the
Speaker 1: critical real world challenges that come with working with data.
Speaker 2: And the ethical ones too. They're huge.
Speaker 1: Absolutely. We'll be drawing our insights primarily from Rachel Schutt's
Speaker 1: pioneering book Doing Data Science, Straight Talk from the front Line,
Speaker 1: which came out of her course at Columbia.
Speaker 2: University, a really groundbreaking course back then.
Speaker 1: Exactly. So by the end of this deep dive you
Speaker 1: should have a much clearer understanding of this field, hopefully
Speaker 1: equipped with the knowledge to you know, cut through the
Speaker 1: noise and see why it's irrelevant. So let's dive straight
Speaker 1: into the heart of it. What is data science? It's
Speaker 1: a question that even the pioneers of the field really
Speaker 1: grappled with.
Speaker 2: They really did until.
Speaker 1: Shuts introduction to data science course at Columbia I think
Speaker 1: it started fall twenty twelve that really acted as an
Speaker 1: incubator for this whole idea.
Speaker 2: Yeah, it was a starting point, and Kathy O'Neill form
Speaker 2: the mathbave dot org blog. She was instrumental in bringing
Speaker 2: these ideas out, specifically pushing back against all the marketing hype.
Speaker 1: There was a lot of hype back then.
Speaker 2: Oh yeah. A crucial point here is that initial sort
Speaker 2: of bewilderment around it. The term was just vague. People
Speaker 2: were throwing around phrases like masters of the universe for
Speaker 2: data scientists.
Speaker 1: Right, which must have annoyed some statisticians.
Speaker 2: You can imagine they felt like, hey, that's our feel.
Speaker 2: The science of data identity theft almost. But the core
Speaker 2: argument in the book, and I think it holds up,
Speaker 2: is that data science isn't just rebranding.
Speaker 1: Not just a new buzzword exactly.
Speaker 2: It's genuinely a new idea, maybe still a bit fragile
Speaker 2: or evolving, but it uniquely combines foundations from statistics and
Speaker 2: computer science. Plus it has this distinct process tied to it.
Speaker 1: And part of that newness I think came from this
Speaker 1: idea of datification. Kenneth Kookier and Victor Merhr Schoenberger talked
Speaker 1: about this in foreign affairs maybe mid two.
Speaker 2: Thirteeneen Riise Big Data. Yeah.
Speaker 1: They defined datification as basically taking all aspects of life
Speaker 1: and turning them into data.
Speaker 2: Which sounds huge.
Speaker 1: It is. Think about it, Google glassdentifying your gaze, Twitter,
Speaker 1: turning stray thoughts into data points LinkedIn mapping out your
Speaker 1: professional life. Everything becomes potentially quantifiable.
Speaker 2: Which immediately makes you ask, Okay, who is this we
Speaker 2: doing the datafying and what kind of value are they
Speaker 2: actually creating? Often it's well modelers, entrepreneurs.
Speaker 1: Looking for efficiency, automation pretty much.
Speaker 2: Yeah, And what's really striking is that so much of
Speaker 2: this wasn't bubbling up in academia initially. It was happening
Speaker 2: in industry, in tech companies. That's quite different from how
Speaker 2: statistics traditionally develop.
Speaker 1: So if it's this broad new thing happening in industry,
Speaker 1: what does a data scientist actually look like? Should have
Speaker 1: this interesting exercise for.
Speaker 2: Her students, the self profiling, Yeah.
Speaker 1: Right, rate yourself on computer science, math, stats, machine learning,
Speaker 1: domain expertise, communication VIZ data visualization, and.
Speaker 2: The results were all over the PLAYFFERNT thing. It showed
Speaker 2: pretty clearly that you know, no single person is going
Speaker 2: to be brilliant at all of those things.
Speaker 1: Yeah, unicorns are rare exactly, which led to this idea.
Speaker 2: Maybe it's more useful to define a data science team
Speaker 2: than one perfect data scientist.
Speaker 1: Makes sense Like that Josh Will's quote, Oh.
Speaker 2: Yeah, the classic person who is better at statistics than
Speaker 2: any software engineer, and better at software engineering than any statistician.
Speaker 1: That captures it pretty well.
Speaker 2: It does. Fundamentally, a data scientist extracts, meaning interprets data.
Speaker 2: They need tools from stats and machine learning, sure, but
Speaker 2: also crucially human intuition, and let's be honest, a huge
Speaker 2: part of the job is just collecting, cleaning, and munging data. Yeah,
Speaker 2: wrestling with it because real world data is just in
Speaker 2: apparently messy always Okay.
Speaker 1: So moving from the who to the how? How does
Speaker 1: this actually get done? We hear big data all the time,
Speaker 1: but it's kind of a.
Speaker 2: Vague term it really is.
Speaker 1: Yeah.
Speaker 2: The book breaks it down nicely though, three parts. One,
Speaker 2: it's a set of technologies. Two, it's potentially a revolution
Speaker 2: in how we measure things, and a point of view,
Speaker 2: really a philosophy about how decisions are going to be
Speaker 2: made in the future based on data.
Speaker 1: Right and connecting big data back to basic stats like
Speaker 1: populations and samples.
Speaker 2: That seems important, Oh, absolutely critical. There's this dangerous assumption
Speaker 2: sometimes with big data that nal You know, you have
Speaker 2: all the.
Speaker 1: Data, but you never really do.
Speaker 2: You pretty much never do. There's always something missing, some
Speaker 2: context you don't have. Kate Crawford's talk on the Hurricane
Speaker 2: Sandy tweets is such a powerful example of this.
Speaker 1: Well was it gist of that?
Speaker 2: Well, looking at the tweets, you might think New Yorkers
Speaker 2: were just casually shopping before the storm and partying after.
Speaker 2: But that's because they were the ones tweeting heavily.
Speaker 1: Ah, so it missed the people really.
Speaker 2: Affected exactly, coastal New Jerseyans whose homes were being destroyed.
Speaker 2: They weren't tweeting about their grocery runs. It just shows
Speaker 2: how subjective the whole process is. You, the data scientist,
Speaker 2: are turning the world into data. It's not objective.
Speaker 1: Data doesn't just speak for itself.
Speaker 2: Never be very skeptical if someone claims it does.
Speaker 1: Okay, so data is subjective. We need context. Then we
Speaker 1: get to modeling. This sounds like where the magic.
Speaker 2: Happens or the hard work maybe both, and when we
Speaker 2: say model here, we don't mean like a database scheme.
Speaker 2: We mean a statistical model.
Speaker 1: Like a mathematical function.
Speaker 2: Yeah, one that tries to capture the uncertainty, the randomness,
Speaker 2: and how the data was generated and building these it's
Speaker 2: definitely part art, part science. Textbooks don't really give you
Speaker 2: a step by step guide. You have to make assumptions,
Speaker 2: a lot of assumptions about reality. But yeah, we'll get
Speaker 2: into how that works.
Speaker 1: And you mentioned a big pitfall here overfitting.
Speaker 2: Yes, get ready to hear about fitting a lot, possibly
Speaker 2: until you have nightmares.
Speaker 1: About Okay, okay, So what is it?
Speaker 2: It's when your model gets too good at explaining the
Speaker 2: specific data you train it on, including all the random
Speaker 2: noise and quirks in that sample.
Speaker 1: So it learns the noise, not the signal.
Speaker 2: Precisely, and then it fails, often badly, when you try
Speaker 2: to use it on new unseen data. It hasn't learned
Speaker 2: the general pattern, just the specifics of the test it's
Speaker 2: studied for, so to speak.
Speaker 1: Right, it can't generalize. So before we even get to
Speaker 1: complex models, what's the first step.
Speaker 2: Exploratory data analysis? Eighty A. Yeah, it's absolutely fundamental.
Speaker 1: And that's more than just plotting things, oh much more.
Speaker 2: It's a mindset. It's about getting intuition, understanding the shape
Speaker 2: of your data, feeling how it connects back to the
Speaker 2: real world process that created it.
Speaker 1: So what does it help you do practically well?
Speaker 2: Gain intuition? Obviously, make comparisons, do basic sanity checks. Is
Speaker 2: the data on the right scales, it the right format,
Speaker 2: Spot missing values or crazy outliers, summarize things, even debug
Speaker 2: how the data was logged in the first place.
Speaker 1: Okay, like the example with the New York Times ad
Speaker 1: data NYT one dot csv through NYT three one dot csv.
Speaker 2: Exactly, the students had to plot distributions of ad impressions
Speaker 2: and click through rates the CTR for different age groups,
Speaker 2: and segment users by whether they clicked or not using
Speaker 2: r In that case, it forces you to really look
Speaker 2: at the data first.
Speaker 1: And this whole process it kind of mirrors the scientific method,
Speaker 1: doesn't it.
Speaker 2: It really does. You ask a question, you research, explore
Speaker 2: the data, you form a hypothesis, you test it, build
Speaker 2: a model, analyze the results, communicate them.
Speaker 1: But with a twist.
Speaker 2: Yeah, The big difference is the feedback loop. When you
Speaker 2: build a data product like a stam filter or a
Speaker 2: recommendation engine. It goes out into the world, people use it,
Speaker 2: Their interactions generate more data.
Speaker 1: Which feeds back into the system.
Speaker 2: Right, it's a dynamic cycle. It's not like predicting the weather,
Speaker 2: where your forecast doesn't actually change tomorrow's weather. Here the
Speaker 2: model influences the world, which generates new data for them, and.
Speaker 1: The data scientist is involved all the.
Speaker 2: Way through, absolutely, from deciding what data to even collect,
Speaker 2: to asking the first questions, planning the attack, and yeah,
Speaker 2: writing the code.
Speaker 1: The Real Direct case study sounds like a good example
Speaker 1: of this using data in real estate.
Speaker 2: Yeah, Doug Pearlson's company. Yeah, the traditional real estate broker
Speaker 2: system was well broken in terms of data. Brokers guarded
Speaker 2: their info fiercely. Public data was months out of date.
Speaker 1: So what did Real Direct do?
Speaker 2: They heided agents who pooled their knowledge, use data driven tips,
Speaker 2: built real time recommendations, tried to get live feeds on searches, offers,
Speaker 2: closing times.
Speaker 1: All that stuff, and the business model reflected that efficiency.
Speaker 2: Right, a subscription model plus lower commission because the data
Speaker 2: supposedly made things more efficient. The exercise for the students
Speaker 2: was literally, okay, you're advising the CEO, define a data strategy.
Speaker 2: What data do you need, where do you get it?
Speaker 2: How do you clean it, explore it, summarize it, puts
Speaker 2: it all together.
Speaker 1: Okay, let's shift gears to the algorithms the engines driving this.
Speaker 1: Machine learning versus statistical modeling always confusing.
Speaker 2: It is confusing because there's so much overlap. mL algorithms,
Speaker 2: mostly from computer science, do prediction classification clustering. Statistical modeling
Speaker 2: from SaaS environments does well prediction classification clustering.
Speaker 1: So what's the real difference? Then?
Speaker 2: Often it's about the goal and the origin. Many mL algorithms,
Speaker 2: especially the ones driving AI image recognition, speech recommenders, they
Speaker 2: weren't typically part of a core stats curriculum, and crucially,
Speaker 2: they're often not designed to help you infer the underlying why.
Speaker 1: They just want the best prediction exactly.
Speaker 2: Maximum accuracy is usually the goal, whereas statistical modeling often
Speaker 2: puts more emphasis on understanding the relationships the uncertainty. But honestly,
Speaker 2: good data scientists use both. They know when each approach
Speaker 2: is more valuable.
Speaker 1: Right, and the warning you mentioned don't be a hammer
Speaker 1: looking for a nail.
Speaker 2: Precisely, don't just grab the algorithm you know best and
Speaker 2: force it onto the problem. First, understand the problem text,
Speaker 2: figure out its mathematical structure, then see which algorithms fit
Speaker 2: makes sense.
Speaker 1: Let's start with a classic linear regression.
Speaker 2: Ah, yes, your bread and butter. For predicting a continuous
Speaker 2: outcome like price or temperature, using one or more predictors.
Speaker 1: We usually start thinking about simple lines like why will
Speaker 1: twenty five x deterministic?
Speaker 2: Right? But the key mental shift is moving to stochastic functions,
Speaker 2: acknowledging that there's randomness uncertainty. The line represents the average trend,
Speaker 2: but the points will scatter around it, and.
Speaker 1: How do you find the best line?
Speaker 2: You minimize the distance between the points and the line,
Speaker 2: specifically the sum of the squared vertical distances. That's the
Speaker 2: mean squared.
Speaker 1: Error, and you evaluate it with things like P values.
Speaker 2: Yeah, P values help you test if your predictors actually
Speaker 2: have a statistically significant effect. Are their coefficients likely different
Speaker 2: from zero? You can add more predictors. That's multiple linear regression,
Speaker 2: which then raises the question of feature selection, which predictors
Speaker 2: matter most?
Speaker 1: And simulating data can help understand.
Speaker 2: This oh hugely useful, especially in learning. You create fake
Speaker 2: data where you know the true relationship, then you see
Speaker 2: if your model can recover it. How sample size effects things?
Speaker 2: What happens if you add irrelevant variables? It builds intuition?
Speaker 1: Okay, what about classifying things? Finding similar items?
Speaker 2: That sounds like CA nearest neighbors or kNN.
Speaker 1: Right knnn ad. How does that work?
Speaker 2: The idea is simple. To classify a new unlabeled item,
Speaker 2: you look at its K closest neighbors data set where
Speaker 2: you do have labels. Then you assign the classes most
Speaker 2: common among those neighbors. Examples could be anything classifying emails
Speaker 2: as spam NOTT spam based on similar emails, assessing credit
Speaker 2: risk based on similar applicants, recommending restaurants based on what
Speaker 2: similar users like find The neighbors and the key choices
Speaker 2: are two main things. First, how do you define closest?
Speaker 2: You need a distance metric Euclidian is common for points,
Speaker 2: Cosign for text, Hamming for strings, Manhattan for grid like paths.
Speaker 2: Depends on the data. Second, choosing K how many neighbors
Speaker 2: do you consult? One, five, twenty. That's a tuning parameter.
Speaker 1: And this is where it gets interesting. The curse of dimensionality.
Speaker 2: I asked. The curse kNN works great in low dimensions
Speaker 2: like recognizing handwritten digits, where pixels in a two hundred
Speaker 2: and fifty six dimension space have a natural closeness. But
Speaker 2: imagine text data with thousands of dimensions.
Speaker 1: Words things get spread out exactly.
Speaker 2: In high dimensions, everything is kind of far away from
Speaker 2: everything else. Your nearest neighbors might not be very similar
Speaker 2: at all in a meaningful sense. K and N breaks down.
Speaker 2: That's why it's usually bad for spam filtering.
Speaker 1: Good point other K and N pitfalls.
Speaker 2: Definitely need to scale your variables. If income is in dollars,
Speaker 2: in ages in years, income will dominate the distance calculation
Speaker 2: unless you scale them, and overfitting is a risk, especially
Speaker 2: of K one. Then you're just copying the label of
Speaker 2: the single closest point, which might be noise. Correlated features
Speaker 2: can also distort distances.
Speaker 1: Okay, so kNN needs labels. What if you don't have labels,
Speaker 1: but you suspect there are groups in your data.
Speaker 2: Then you're talking about unsupervised learning, and K means clustering
Speaker 2: is a common technique there.
Speaker 1: Unsupervised So the algorithm finds the groups itself precisely.
Speaker 2: You tell it how many clusters K you think exist,
Speaker 2: and the algorithm iteratively assigns points to the nearest cluster
Speaker 2: center centroid, and then recapculates the centroids until things stabilize.
Speaker 1: Why would you do that?
Speaker 2: Lots of reasons. Maybe you want to segment users for
Speaker 2: different marketing or product experiences, or build separate predictive models
Speaker 2: for distinct customer groups. K means helps you discover those
Speaker 2: groups automatically instead of you trying to define them with
Speaker 2: arbitrary rules or thresholds.
Speaker 1: So it automates finding clusters in like many dimensions.
Speaker 2: Yeah, that's the power, but it has its quirks. Choosing
Speaker 2: the right K is often more art than science, and
Speaker 2: sometimes the algorithm can get stuck in a suboptimal solution
Speaker 2: depending on where it starts.
Speaker 1: Is it an old algorithm?
Speaker 2: The basic idea goes back to the fifties Steinhaus and
Speaker 2: Lloyd m the term K means in sixty seven. There
Speaker 2: are newer versions like K means plus plus from two
Speaker 2: thousand and seven that try to start the algorithm off better.
Speaker 1: Okay, so we said canon isn't great for spam filtering
Speaker 1: because of hi dimensions. What does work well that.
Speaker 2: Brings us to naive base? A surprisingly effective probabilistic.
Speaker 1: Approach based on Bayes Law.
Speaker 2: Exactly remember baes Law from stats PA B p G.
Speaker 1: Vaguely like the disease testing example, probability you're sick given
Speaker 1: a positive test.
Speaker 2: That's the one we applied the same logic to spam.
Speaker 2: What's the probability in email is spam? Given that contains
Speaker 2: the word viagra, p spam, word peace, damp, word.
Speaker 1: Needs sense what's the naive part?
Speaker 2: The naive assumption is that the words in the email
Speaker 2: appear independently of each.
Speaker 1: Other, which isn't true. Right, free and viagra probably appeared
Speaker 1: together more often than by chance in spam.
Speaker 2: Totally untrue. But the simplification makes the math tractable, and
Speaker 2: surprisingly it often works really well in practice, especially for text.
Speaker 1: Any pitfall just counting words.
Speaker 2: Oh yeah, if a word like viagra only appeared in
Speaker 2: spam in your training data, the model might assign a
Speaker 2: one hundred percent probability of spam if it sees that
Speaker 2: word again. It's overfitting. Also, what if you see a
Speaker 2: word you've never seen before? The probability would be zero,
Speaker 2: which messes up the calculation.
Speaker 1: So how do you fix that?
Speaker 2: With laplace smoothing sometimes called additive smoothing, you basically add
Speaker 2: a small pseudo count to every word count, pretending you've
Speaker 2: seen each word at least once or a fraction of
Speaker 2: a time. It prevents zero probabilities and generally makes the
Speaker 2: estimates more robust.
Speaker 1: And this was used in that NYT article classification exercise.
Speaker 2: Yes, exactly Jake's exercise. Download two thousand articles from different
Speaker 2: sections arts, business, sports, et cetera. Using the API. Train
Speaker 2: a naive base model specifically Bernoulli ni bays here to
Speaker 2: classify them, tune the smoothing parameters, evaluate with a confusion matrix.
Speaker 2: See which words were most indicative of each section. Great
Speaker 2: hands on example.
Speaker 1: Cool again, Another big one. Logistic regression. How's that different
Speaker 1: from linear regression?
Speaker 2: Linear regression predicts a continuous value, right like a house price.
Speaker 2: Logistic regression predicts the probability of a binary outcome, something
Speaker 2: that's either yes or no zero.
Speaker 1: One like will a user click and add? Is this
Speaker 1: email spam while this customer churn?
Speaker 2: Exactly those kinds of things binary outcomes?
Speaker 1: And how does it predict a probability? Doesn't a linear
Speaker 1: model output any number.
Speaker 2: It starts with a linear combination of features, just like
Speaker 2: linear regression. Else plus matrox. But then it feeds that
Speaker 2: result through a special function called the logistic function or sigmoid.
Speaker 1: Function the S shaped curve that's the one.
Speaker 2: Pt one plus et. This function squishes any input value
Speaker 2: into an output between zero and one, perfect for representing
Speaker 2: a probability.
Speaker 1: So alpha and beta still means something yep.
Speaker 2: Alpha spain is related to the baseline probability. The overall
Speaker 2: odds and the betas are the weights for each feature,
Speaker 2: telling you how much each feature changes the law odds
Speaker 2: of the outcome.
Speaker 1: How do you find the best alpha and betas?
Speaker 2: Usually with maximum likelihood estimation, you find the parameters that
Speaker 2: make the observed data most probable. This often involves optimization
Speaker 2: algorithms like Newton's method, or, especially for huge data sets,
Speaker 2: stochastic gradient descent SGD.
Speaker 1: SGD sounds familiar.
Speaker 2: Very common in large scale machine learning. It updates the
Speaker 2: parameters using just one data point or a small batche
Speaker 2: at a time, making it efficient for massive data sets,
Speaker 2: especially sparse ones. Tools like mahood or valpol wabbit use
Speaker 2: it heavily.
Speaker 1: Now, evaluating these models, you said, accuracy isn't always great.
Speaker 2: Right, especially with imbalanced classes. If only one percent of
Speaker 2: emails are spam. A model predicting not spam one hundred
Speaker 2: percent of the time is ninety nine percent accurate, but useless.
Speaker 1: So what should we use instead?
Speaker 2: Look at precision of the times you predicted spam, how
Speaker 2: often were you right? And recall of all the actual
Speaker 2: spam how much did you catch often? There's a trade off.
Speaker 1: And F score AUC.
Speaker 2: F score tries to combine precision and recall into one number.
Speaker 2: AUC area under the ROC curve is really good because
Speaker 2: it measures performance across all possible thresholds and isn't thrown
Speaker 2: off by imbalanced classes. It's base rate invariant.
Speaker 1: But even these metrics might not capture the real goal exactly.
Speaker 2: Your model might have great AUC, but does it actually
Speaker 2: increase revenue or user engagement. That's where AB testing comes in,
Speaker 2: the gold standard for real world impact.
Speaker 1: YE, run a controlled experiment, show the old system to
Speaker 1: group A, the new model to group B, and measure
Speaker 1: the actual business outcome you care about. Google's paper on
Speaker 1: experimentation really drives this.
Speaker 2: Home and Media six degrees M six D use logistic
Speaker 2: regression for predicting AD clicks.
Speaker 1: Yeah, a classic application user level conversion prediction, highly scalable,
Speaker 1: and effective for binary outcomes.
Speaker 2: So okay, let's get into some real world messiness. Time
Speaker 2: stamps seems simple, but you said they're tricky. Oh they are.
Speaker 2: You get tons of time stamped event data, user clicks,
Speaker 2: check ins, sensor readings. That's big data right there. But
Speaker 2: they introduce subtle problems like what the biggest is causality.
Speaker 2: You cannot use information from the future to predict the
Speaker 2: present or past. Sounds obvious, but it's easy to accidentally
Speaker 2: leak future information into your training data if you're not
Speaker 2: careful with time stamps.
Speaker 1: Ah the time travel problem exactly.
Speaker 2: You also have to be super careful distinguishing in sample
Speaker 2: training data from out of sample testing data based on time,
Speaker 2: and often you need running estimates, like a running average,
Speaker 2: not a single average calculated over all past data, because
Speaker 2: the world changes like in finance. Finance is a great example,
Speaker 2: they often use log returns instead of simple percentage returns
Speaker 2: because log returns handle compounding better and are more symmetric.
Speaker 2: Volatility is key, and techniques like exponential downweighting let you
Speaker 2: calculate a volatility estimate that gives more weight to recent data,
Speaker 2: efficiently updating with just the latest info.
Speaker 1: And financial markets have weak signals.
Speaker 2: Extremely weak these days, so many algorithms are looking for
Speaker 2: the same patterns that the signals get arbitraged away quickly.
Speaker 2: You might aim for just a tiny positive correlation like
Speaker 2: three percent over a day. Yet linear regression is still
Speaker 2: used because it's robust even with all that noise.
Speaker 1: And they use things like priors and regularization yes, to.
Speaker 2: Keep the model stable. Priors incorporate existing beliefs, and regularization
Speaker 2: adds penalties to stop coefficients from getting too large or
Speaker 2: varying too wildly. It helps prevent overfitting in noisy high
Speaker 2: dimensional data. Mathematically, it often simplifies to adding a term
Speaker 2: to the covariance matrix.
Speaker 1: Which brings us to figuring out what data to even
Speaker 1: put into the model. Feature engineering the art of data science.
Speaker 2: Some say it's about deciding which variables features to use
Speaker 2: and how to transform them to be most effective for
Speaker 2: your model. Should's four quadrants idea highlights thinking about relevance, usefulness,
Speaker 2: whether something's even logged. Your creativity is a limit.
Speaker 1: How do you select the best features?
Speaker 2: Several ways filters rank features individually by predictive power like correlation,
Speaker 2: but they miss interactions. Step rise regression ads or removes
Speaker 2: features one by one from a model watching metas like
Speaker 2: R squared or P values, but careful, it can easily overfit.
Speaker 1: What about embedded methods.
Speaker 2: These methods build feature selection right into the model. Training
Speaker 2: decision trees are a prime example.
Speaker 1: How do they work?
Speaker 2: They recursively split the data based on the feature that
Speaker 2: provides the most information game at each step, essentially creating
Speaker 2: a flow chart to classify things. They're very interpretable. You
Speaker 2: can see the rules.
Speaker 1: Like the Titanic survival example.
Speaker 2: Classic decision trees can figure out rules like if female survive,
Speaker 2: if male and child survive. They handle continuous variables by
Speaker 2: finding optimal split points, but you often need to prune
Speaker 2: the tree, cutting back branches to stop it from overfitting
Speaker 2: the training data. In random Force, they take decision trees
Speaker 2: to the next level. They build many decision trees on
Speaker 2: different random subsets of the data and features bagging, and
Speaker 2: then average their predictions. Much more accurate and robust than
Speaker 2: a single tree, But you lose that easy interpretability.
Speaker 1: Now when selecting features that causation versus correlation issue seems
Speaker 1: critical huge.
Speaker 2: Is user played ten times a feature describing the user's behavior,
Speaker 2: or is you show ten ads a future describing your action.
Speaker 2: If you want actionable insights to change outcomes, you need
Speaker 2: to focus on features you can actually control.
Speaker 1: David Huffacker at Google talked about mixing message.
Speaker 2: Yeah, they're a hybrid approach, combining qualitative insights from small
Speaker 2: user interviews with quantitative analysis of large scale log data.
Speaker 2: The Google Plus circles feature apparently came from that kind
Speaker 2: of mix interviews, sparking ideas tested with big data moving
Speaker 2: from description to prediction, which.
Speaker 1: Also raises privacy concerns.
Speaker 2: Absolutely, anytime you're dealing with human data, what are people
Speaker 2: worried about? Identity, theft, financial loss, creepy ads. The book
Speaker 2: mentions ideas like clearer data flow diagrams, privacy controls, sensible defaults,
Speaker 2: big thorny issues.
Speaker 1: Let's talk visualization again. Mark Hanson framed it as more
Speaker 1: than just nice plots right.
Speaker 2: As an art and information discipline that actually changes how
Speaker 2: we see things. Change the instruments, and you change the theory.
Speaker 2: His examples Million Dollar Blocks, Project Cascade, the NYT Lobby
Speaker 2: display showed data as a powerful communication and exploration tool, sometimes.
Speaker 1: Even as art and Ian Wong at Square connected this
Speaker 1: directly to fraud.
Speaker 2: Detection exactly. Square deals with massive potential fraud. The used
Speaker 2: machine learning heavily, but Wong stressed the importance of visualization alongside.
Speaker 2: It helps him understand why the model flags something, especially
Speaker 2: with high class imbalance.
Speaker 1: Where fraud is rare so accuracy is useless.
Speaker 2: Decisely, they focus on precision and recall. Wong's tips were
Speaker 2: great too. Models aren't black boxes. Iterate quickly like experiments.
Speaker 2: Keep your code clean and reusable.
Speaker 1: And productionizing the models.
Speaker 2: Making them work in real time, minimizing the gap between
Speaker 2: offline tests and online reality. Visualization isn't just for building
Speaker 2: the model. It helps the human operations team review transactions efficiently.
Speaker 2: It augments their intelligence like an exoskeleton.
Speaker 1: Okay, this is a really important distinction. Prediction versus causality.
Speaker 2: Yes, this is where things get philosophically deep. Maybe recommending
Speaker 2: a book because you read a similar one that's prediction.
Speaker 2: Understanding what causes someone to buy a book, or get
Speaker 2: sick or click an add that's causality.
Speaker 1: The methods might look similar, sometimes they might.
Speaker 2: Use similar stats, but the intent is different, and that
Speaker 2: changes everything about how you design your analysis and interpret
Speaker 2: the results.
Speaker 1: And observational data is tricky for causality.
Speaker 2: Very ory. Steelman mentioned that Okay, Cupid, example, may be
Speaker 2: beautiful in emails correlated with responses, but did it cause
Speaker 2: them or were people writing beautiful also doing other things
Speaker 2: differently confounders everywhere.
Speaker 1: So the gold standard is.
Speaker 2: Randomize controlled trials. RCTs randomly assign people to treatment or control.
Speaker 2: If the group starts statistically identical thanks to randomization, any
Speaker 2: difference in outcome can be attributed to the treatment. That's
Speaker 2: causal Inference.
Speaker 1: And ab tests are like RCTs for tech.
Speaker 2: Pretty much much easier logistically than clinical trials, usually less
Speaker 2: it's stay more control if you have the infrastructure randomly
Speaker 2: show version A or version B of a web page,
Speaker 2: measure the difference in clicks or conversions.
Speaker 1: But watching out for things like Simpson's paradox, Oh.
Speaker 2: Yeah, a huge pitfall and observational data, a trend appears
Speaker 2: in the overall data, but reverses when you break it
Speaker 2: down to the subgroups, so as you can't just trust
Speaker 2: aggregated numbers.
Speaker 1: David Madigan's work sounds intense. Using the Reuben causal model.
Speaker 2: Right a formal framework to think about what you can
Speaker 2: and can't know from observational studies. His almop project analyzed
Speaker 2: huge medical databases.
Speaker 1: Two hundred million people.
Speaker 2: Yeah, and the shocking finding was that for many medical
Speaker 2: questions using the same database but different standard analysis methods,
Speaker 2: you could get opposite conclusions, like whether a drug caused
Speaker 2: cancer or not.
Speaker 1: Wow. That really underlines the challenge of inferring cause from
Speaker 1: observed data.
Speaker 2: It really does choices in analysis matter profoundly.
Speaker 1: Another scary problem data leakage. Claudia or Perlic called it
Speaker 1: a huge problem.
Speaker 2: It is both in petitions and the real world. It's
Speaker 2: when your model accidentally learns from information it wouldn't have
Speaker 2: in a real prediction scenario. It learns the noise for signal.
Speaker 1: How does that happen?
Speaker 2: Could be implicit future information, like using diagnosis codes assigned
Speaker 2: after a prediction should have been made, or non random sampling.
Speaker 2: Perlik gave examples like patient IDs accidentally encoding clinic location,
Speaker 2: making cancer prediction trivial, or pneumonia models exploiting diagnosis codes.
Speaker 1: So how do you avoid it?
Speaker 2: Specific advice, Yes, very specific. Strict temporal cutoff. Remove everything
Speaker 2: learned even microseconds before the event you're predicting time, stamp
Speaker 2: everything based on when it was known, not just when
Speaker 2: it happened. Start clean with raw data, and most importantly,
Speaker 2: understand how the data was generated.
Speaker 1: Also important is model calibration. Are the probabilities right crucial?
Speaker 2: A model might be good at ranking risks, but are
Speaker 2: its predicted probabilities accurate? Like when it says seventy percent chance,
Speaker 2: does it happen seventy percent of the time? Unprune decision
Speaker 2: trees she showed often give probabilities that are too extreme,
Speaker 2: while logistic progression tends to be better calibrated.
Speaker 1: Okay, Last, big area data engineering, scale and complexity, map
Speaker 1: reduce and hodup do data scientists need to know?
Speaker 2: This? Often not directly coding map produced jobs anymore with
Speaker 2: newer tools abstracting it, but understanding the concepts is still
Speaker 2: really valuable.
Speaker 1: Why was map reduce developed? What problem does it solve?
Speaker 2: Think about counting word frequencies in terabytes of text. You
Speaker 2: can't load it all into memory on one machine, and
Speaker 2: even if you could split it, combining the accounts back together,
Speaker 2: it creates a bottleneck the fan end problem. Map reduce
Speaker 2: provides a framework for distributing the work across many machines,
Speaker 2: handling failures automatically and managing the.
Speaker 1: Complexity, but it can be unintuitive.
Speaker 2: Converting some algorithms into map processed chunks and reduce aggurate
Speaker 2: results steps isn't always obvious, and how evenly the data
Speaker 2: is spread across the machines is critical for performance.
Speaker 1: And Joshuall's big data economic law love that one.
Speaker 2: No single record is that valuable, but every record, the
Speaker 2: whole collection is incredibly valuable. The aggregate tells the story.
Speaker 1: And tools like Prigle and mahoot build on this.
Speaker 2: Yeah, prigles for large scale graph processing, mahoot for machine
Speaker 2: learning algorithms implemented on top of doop map produce handling
Speaker 2: the scale.
Speaker 1: So wrapping up this deep dive, we've covered a lot.
Speaker 1: The goal was really to show what it's like to
Speaker 1: be a data scientist and also how some of this
Speaker 1: work gets done.
Speaker 2: From defining the field to specific algorithms and those tough
Speaker 2: real world.
Speaker 1: Challenges right and revisiting that core question what is data science?
Speaker 1: The book suggests it's maybe a set of best practices
Speaker 1: used in tech companies tackling problems with data, sometimes scientifically, but.
Speaker 2: Also always be wary of the hype. It's not magic
Speaker 2: it's work.
Speaker 1: It's a process absolutely, and thinking about the future the
Speaker 1: hope for next gen data scientists. It's not just about
Speaker 1: technical skill or salary.
Speaker 2: No, it's about being good problem solvers, asking the right questions,
Speaker 2: thinking deeply about design process ethics.
Speaker 1: Like Jeff Hammerbacker's famous quote about the best minds working
Speaker 1: on clicking ads that sucks. The aspiration is to use
Speaker 1: these powerful tools responsibly to make things better, not worse.
Speaker 2: Which comes back to critical thinking always remembering data does
Speaker 2: not speak for itself. It needs interpretation, criticism, evaluation. It
Speaker 2: requires dealing with messy, incomplete, often inconclusive data. It's a
Speaker 2: human process full of judgment calls.
Speaker 1: So for you, our listener, as you maybe explore data
Speaker 1: science yourself, what does all this mean. It's complex, it's evolving,
Speaker 1: it has huge potential and pitfalls. You've heard the practicalities,
Speaker 1: the ethical tightropes, the human effort involved.
Speaker 2: It's not just about the algorithms.
Speaker 1: Definitely not. So here's something to think about as you
Speaker 1: continue your journey. What ethical questions will you inevitably face
Speaker 1: when you start applying these powerful tools. How will you
Speaker 1: handle that tension between what data can show you and
Speaker 1: what should be done with that knowledge, especially when different
Speaker 1: people have different interests. Ultimately, how will you try to
Speaker 1: ensure that your work, your insights are used to make
Speaker 1: the world genuinely better and not just, you know, to
Speaker 1: make people click