Speaker 1: Imagine, right, imagine a master chef, like a world class chef,
Speaker 1: starring in this massive primetime thirty minute cooking show.
Speaker 2: Okay, I'm picturing it right.
Speaker 1: So the camera rolls, the steak is perfectly seared, the
Speaker 1: sauce is just flawless, and the audience goes completely wild.
Speaker 1: But what you don't see, though, what's hidden off camera,
Speaker 1: is that this chef spent the previous eight hours standing
Speaker 1: in a back alley literally crying while peeling potatoes and
Speaker 1: shopping like ten thousand onions.
Speaker 2: Yeah, that sounds about right for the industry.
Speaker 1: It's crazy. And in the technology world, that chef is
Speaker 1: a data scientist. So today we are looking at how
Speaker 1: enterprise organizations are finally you know, firing the potato peelers.
Speaker 2: It is a critical shift. Honestly, we are looking at
Speaker 2: a fundamental rewrite of the architecture, Like the actual plumbing
Speaker 2: of predictive analytics is being completely re routed at the
Speaker 2: enterprise level just to eliminate these massive systemic inefficiencies.
Speaker 1: Exactly. So, if you want a shortcut to understanding how
Speaker 1: big businesses are actually transfer forming their standard everyday databases
Speaker 1: into these automated predictive engines. Well, this is it. Today's
Speaker 1: deep dive is based on excerpts from the book Data
Speaker 1: Science using Oracle Data Minor and Oracle r Enterprise.
Speaker 2: Which is a fantastic resource by the.
Speaker 1: Way, Oh totally. We're exploring how bringing the math directly
Speaker 1: to the data solves honestly one of the single biggest
Speaker 1: bottlenecks in modern tech. So okay, let's unpack this because
Speaker 1: the way a lot of organizations currently execute data science
Speaker 1: is just fundamentally broken, it really is.
Speaker 2: What's fascinating here is that the real secret to effective
Speaker 2: data science at scale isn't necessarily about inventing a more
Speaker 2: complex neural network or a bitter algorithm. It's really about
Speaker 2: where those algorithms are physically executed, right.
Speaker 1: The location matters exactly.
Speaker 2: Moving massive amounts of data across networks just to analyze
Speaker 2: it is a critical, costly mistake, and the infrastructure we
Speaker 2: were exploring today was built specifically to stop that movement entirely.
Speaker 1: So to understand the fix, I feel like we first
Speaker 1: have to understand why data scientists are stuck chopping those
Speaker 1: onions in the first place. Because if you look at
Speaker 1: standard industry frameworks like the crispym methodology. You'd probably assume
Speaker 1: the actual modeling, like the glamorous machine learning part, is
Speaker 1: where all the time goes.
Speaker 2: Yeah, that is the assumption, but the reality is heavily,
Speaker 2: heavily skewed. In almost any enterprise deployment, data preparation takes
Speaker 2: up a staggering sixty to eighty percent of the total
Speaker 2: project effort.
Speaker 1: Sixty to eighty percent. That's insane.
Speaker 2: It is because real world data is inherently dirty, it's skewed,
Speaker 2: it's real with missing values. It's just a mess. Right.
Speaker 1: So if the vast majority of the job is just
Speaker 1: cleaning up missing variables and formatting timestamps, why is the
Speaker 1: titled data scientist and not data janitor.
Speaker 2: Well, because that janitoring actually dictates the success or failure
Speaker 2: of the entire model. In predictive analytics, there is basically
Speaker 2: an ironcloud rule. A simple regression model built on perfectly
Speaker 2: clean data will consistently outperform a highly sophisticated deep learning
Speaker 2: model that has been fed dirty data.
Speaker 1: Wow. Really, so the clean data beats the complex math every.
Speaker 2: Single time, because if you don't handle the anomaies, your
Speaker 2: model simply learns the noise. It just memorizes the mistakes.
Speaker 1: But spending all that time cleaning. I mean, that's the
Speaker 1: antithesis of business agility, right Like, if a telecom company
Speaker 1: wants to predict customer churn this month, they can't afford
Speaker 1: to spend three weeks manually cleaning billing data first.
Speaker 2: Exactly, And that naturally leads us to the data science
Speaker 2: automation pyramid. From the source material. To move fast, you
Speaker 2: have to automate the data pipeline.
Speaker 1: Okay, So walk us through this pyramid. What's at the bottom.
Speaker 2: At the base, you have problem specific automation. This is
Speaker 2: automating a single, rigid workflow, like maybe a monthly sales
Speaker 2: forecast that runs exactly the same way every time.
Speaker 1: Got it, Just basic scripting right.
Speaker 2: Then above that you have repetitive task automation, which is
Speaker 2: where you build generalize scripts to automatically handle missing values
Speaker 2: or transform columns across various different data sets.
Speaker 1: So that's taking away a lot of the manual janitor
Speaker 1: work exactly.
Speaker 2: It frees up the human to do actual science.
Speaker 1: Okay, So what's at the very top of the pyramid.
Speaker 2: Then the automated statistician Ooh, that sounds intense. It is
Speaker 2: This is an environment where the system evaluates the underlying
Speaker 2: data structures, learns the patterns, and automatically selects the most
Speaker 2: optimal algorithm without requiring a human to manually tune the
Speaker 2: hyper parameters.
Speaker 1: Wait, getting to that top tier sounds incredible, but I
Speaker 1: mean the glaring issue here is the friction of traditional architecture.
Speaker 2: Right.
Speaker 1: Historically, when a data scientist wanted to run those repetitive
Speaker 1: data cleaning scripts, they were using client side tools like
Speaker 1: Python or open source R or SaaS on their laptops.
Speaker 2: Yeah, which means they had to extract the data. Traditional
Speaker 2: analytical environments basically sit on a separate application server or
Speaker 2: on the data scientist's local machine. So to run your model,
Speaker 2: you have to query your central enterprise database, extract gigabytes
Speaker 2: or sometimes terabytes of data, push it over the network,
Speaker 2: it into the memory of your analytical tool, process it,
Speaker 2: and then attempt to write the results back.
Speaker 1: It sounds exhausting just describing it. So, bringing back to
Speaker 1: our chef analogy, traditional data science is like storing all
Speaker 1: your raw ingredients in this massive warehouse all the way
Speaker 1: across town. Yes, exactly, and every single time you want
Speaker 1: to test a new recipe, You literally have to drive
Speaker 1: a semi truck across the city, load up the ingredients,
Speaker 1: drive back to your kitchen, cook the meal, and then
Speaker 1: drive the leftovers back to the warehouse.
Speaker 2: It's catastrophic for efficiency. You hit network io bottlenecks immediately,
Speaker 2: you hit integration failures, and most importantly, client based tools
Speaker 2: simply choke because they require the data set to be
Speaker 2: loaded into active RAM.
Speaker 1: Right. You can't just cram everything in there.
Speaker 2: No, you absolutely cannot load a two terabyte customer table
Speaker 2: into the RAM of a standard application server. It will crash.
Speaker 1: Here's where it gets really interesting, though, because the solution
Speaker 1: presented in this architecture, specifically Oracle Advanced Analytics, basically just
Speaker 1: builds the kitchen and inside the warehouse precisely.
Speaker 2: Oracle Advanced Analytics or OAA operates directly on top of
Speaker 2: the Oracle database kernel. The data never actually moves.
Speaker 1: So you're just cooking where the food is exactly.
Speaker 2: By eliminating data extraction, you effectively achieve zero latency in
Speaker 2: your data pipeline.
Speaker 1: And I imagine you bypass the memory limits of a
Speaker 1: local machine entirely right, because the database is already optimized
Speaker 1: to query and process data directly from storage.
Speaker 2: Oh absolutely, Plus you don't have to worry about security
Speaker 2: protocols breaking down over some open network connection, right.
Speaker 1: Because once the data leaves the database, you've kind of
Speaker 1: lost control over who sees it.
Speaker 2: Security is a massive factor here. The data remains governed
Speaker 2: by the strict native security policies of the database itself.
Speaker 2: But from a purely performance standpoint, executing inside the kernel
Speaker 2: allows the algorithms to leverage oracles parallel processing capabilities.
Speaker 1: Meaning instead of one computer churning through the data row
Speaker 1: by row by row, the database can split the job
Speaker 1: across dozens of internal processors simultaneously.
Speaker 2: Exactly, and when predictive models run directly inside the kernel,
Speaker 2: the whole business posture shifts. You aren't extracting data to
Speaker 2: run some post mortem analysis of what happened last quarter.
Speaker 1: Yeah, it's not looking backward anymore.
Speaker 2: Right, You are queering the database in real time to ask,
Speaker 2: what is the probability this specific transaction happening right now
Speaker 2: is fraudulent?
Speaker 1: Okay, So the kitchen is inside the warehouse, which is great,
Speaker 1: but we still have to do the sixty to eighty
Speaker 1: percent of the workload that involves data preparation, right, I mean,
Speaker 1: bringing the math to the data doesn't magically clean it.
Speaker 1: We still need the tools to handle anomalies.
Speaker 2: Yes we do, and that is handled through an in
Speaker 2: database plseql package called DBMS data meaning transformation.
Speaker 1: Okay, quite a mouseful, it is, yeah.
Speaker 2: But basically this is the toolkit for managing that massive
Speaker 2: data preparation phase without ever leaving the database.
Speaker 1: So let's talk about how this toolkit actually works specifically withouts,
Speaker 1: Because if you have an e commerce platform in your analyze,
Speaker 1: and say average order value one user buying a fifty
Speaker 1: thousand dollars watch is going to completely skew your standard distribution.
Speaker 2: Oh absolutely, So.
Speaker 1: The toolkit handles this using what they call winsorizing or trimming.
Speaker 2: Right, unhandled extreme values will drag the mean of your
Speaker 2: data so far from the median that any distance based
Speaker 2: algorithm you use will generate completely erroneous clusters. It'll just
Speaker 2: ruin the model.
Speaker 1: So if you're using this plcql package, how do these
Speaker 1: two methods winsorizing and trimming mechanically solve that watch problem.
Speaker 2: Well, trimming is the brute force approach. It literally clips
Speaker 2: the extreme tail ends of your distribution, say the top
Speaker 2: one percent of values and just sets them to NUL.
Speaker 1: Just delete them basically effectively.
Speaker 2: Yes. Windsorizing, on the other hand, is much more elegant.
Speaker 2: Instead of removing the data point entirely, it cacks it.
Speaker 2: It replaces those extreme tail values with a specified maximum parameter,
Speaker 2: pulling the outlier back into the acceptable edge of the distribution.
Speaker 1: Oh I see, So windsorizing is like taking a person
Speaker 1: who's screaming through a megaphone in a crowded room and
Speaker 1: forcing them to just whisper. While trimming is you're just
Speaker 1: throwing them out of the building entirely so they don't
Speaker 1: ruin the party.
Speaker 2: That's a great way to visualize it. Yes, but dealing
Speaker 2: without liars is just the first step. You also have
Speaker 2: to normalize the data before you apply the algorithm.
Speaker 1: Right normalization, which is bringing variables to a uniform scale
Speaker 1: using min max or z score calculations. Because and correct
Speaker 1: me if I'm wrong. If you feed an algorithm a
Speaker 1: customer's age, which is a two digit number, alongside their
Speaker 1: annual income, which is a six digit number, the geometry
Speaker 1: of the algorithm will mathematically assume the income is exponentially
Speaker 1: more important, just simply because the integer is larger.
Speaker 2: You nailed it. The algorithm operates on mathematical distance. If
Speaker 2: you don't scale the inputs to a uniform magnitude, your
Speaker 2: model is practically useless.
Speaker 1: It just gets confused by the big numbers exactly.
Speaker 2: But the toolkit goes beyond scaling. It also performs complex binning,
Speaker 2: which is transforming continuous data into discrete categories.
Speaker 1: Yeah. The supervised binning feature is what really caught my
Speaker 1: eye in the source text because instead of a human
Speaker 1: arbitrarily deciding that high income starts at exactly one hundred
Speaker 1: thousand dollars, supervised binning automates the logic it does.
Speaker 2: It uses a decision tree algorithm under the hood, so
Speaker 2: the system analyzes the data's relationship to your target outcome,
Speaker 2: like whether a customer churned or not. If the decision
Speaker 2: tree determines that a massive spike inchurn happens specifically when
Speaker 2: income drops below let's say sixty four three hundred dollars,
Speaker 2: it sets the bin boundary exactly there.
Speaker 1: Oh wow.
Speaker 2: Yeah, it lets the predictive power of the data dictate
Speaker 2: the categorization completely, removing human bias.
Speaker 1: Well, wait, if I'm just a standard database administrator or
Speaker 1: a business analyst running this, How do I know if
Speaker 1: the algorithm I want to use requires minmac scaling or
Speaker 1: a Z score or supervised binning like I wouldn't.
Speaker 2: Know that, and you frequently don't need to know. Oracle
Speaker 2: utilizes a feature called Automatic data Preparation or ADP nice.
Speaker 2: When enabled, ADP intercepts your request evaluates the specific algorithm
Speaker 2: you've chosen, say a support vector machine, which strictly requires
Speaker 2: normalized inputs, and it automatically executes the correct mathematical transformations
Speaker 2: inside the kernel before running the model.
Speaker 1: That is so cool. It handles the prerequisites dynamically, so
Speaker 1: the data is prepped, the environment is secure, and we
Speaker 1: are finally ready for the top tier of that pyramid.
Speaker 1: We talked about the automated statistician.
Speaker 2: Yes, this is where we look at the DBM's predictive
Speaker 2: analytics package. It contained three highly automated APIs, Predict, Explain
Speaker 2: and Profile.
Speaker 1: Right Predict automatically generates an outcome variable. Explain ranks the
Speaker 1: importance of the independent variables, and Profile extracts the core
Speaker 1: business rules the model found. You literally just pass it
Speaker 1: the table name and the target column and it does
Speaker 1: the rest.
Speaker 2: It really is that straightforward.
Speaker 1: So what does this all mean? It sounds like we
Speaker 1: are completely democratizing machine learning, allowing average SQL users to
Speaker 1: perform data science without knowing the math.
Speaker 2: It absolutely democratizes access, But in this raises an important question.
Speaker 2: Is it safe to lower the barrier to entry that far?
Speaker 1: That's a very fair point.
Speaker 2: When an analyst just presses a predict button, they are
Speaker 2: essentially trusting a black box. The system is making vast
Speaker 2: mathematical assumptions on their behalf.
Speaker 1: I agree, and frankly, I'm a bit skeptical. If you
Speaker 1: let an average user bypass the math and just blindly
Speaker 1: apply predictive models to their company's revenue data, aren't we
Speaker 1: just accelerating how fast they can make a catastrophic business decision.
Speaker 2: That is the inherent risk of democratization, without a doubt,
Speaker 2: If the user doesn't understand the underlying assumptions of the models,
Speaker 2: the results can be dangerous. Most powerful parametric machine learning
Speaker 2: models assume that your underlying data follows a normal Bell
Speaker 2: curve distribution. Right If you feed them highly skewed, non
Speaker 2: normal data, the predictions will be mathematically invalid, period.
Speaker 1: Which is why having statistical tests built directly into the
Speaker 1: databas is so critical, I guess you don't have to
Speaker 1: blindly trust the black box. You can use native sequel
Speaker 1: functions like the Shapiro Wilks test to evaluate the normality
Speaker 1: of your distribution right there in the query exactly.
Speaker 2: Shapiro Wokes evaluates the null hypothesis that your sample came
Speaker 2: from a normally distributed population.
Speaker 1: Okay, so if I run that SQL query and it
Speaker 1: returns a P value.
Speaker 2: Of zero, you instantly know your data is non normal.
Speaker 2: You can test your assumptions without having to extract the
Speaker 2: data to a specialized statistical software package. It's all right there.
Speaker 1: And the analytical capabilities of modern sequel don't stop there.
Speaker 1: The source material dives into functions like lag lead and
Speaker 1: these really complex windowing functions. And these aren't just convenient syntax,
Speaker 1: they are massive performance life savers.
Speaker 2: They really are lag and let allow you to access
Speaker 2: data from previous or subsequent rows in the exact same
Speaker 2: result set without having to use a clunky self joint.
Speaker 1: So like if a retailer is calculating year over year's
Speaker 1: sales growth across ten thousand stores, they don't have to
Speaker 1: pull millions of rows into a Python data frame just
Speaker 1: to calculate a rolling average. They can use a SQL
Speaker 1: windowing function to calculate that moving average directly on the
Speaker 1: storage disc and.
Speaker 2: By processing that rolling average inside the database. Using SQL,
Speaker 2: you are leveraging the internal optimizer. It completes the calculation
Speaker 2: in a fraction of the time and only returns the
Speaker 2: final aggregated insight to the application layer.
Speaker 1: Okay, so SQL is incredibly powerful. But let's play Devil's
Speaker 1: advocate for a second.
Speaker 2: Let's do it.
Speaker 1: What if your company's lead data scientist is like a
Speaker 1: hardcore statistical researcher, you know the type. They spend eight
Speaker 1: years getting a PhD. They live and breathe the open
Speaker 1: source R programming language, and they rely on these massive,
Speaker 1: crowdsourced libraries of cutting edge algorithms that standard seql just
Speaker 1: doesn't natively support.
Speaker 2: Are they just forced to abandon R and write Oracle
Speaker 2: plseql not at all.
Speaker 1: That exact friction is what Oracle Are Enterprise or ORE
Speaker 1: was engineered to eliminate.
Speaker 2: Because open source R has the same architectural flaw we
Speaker 2: talked about earlier. Right, it's entirely client based. Has to
Speaker 2: load everything into the local laptops around exactly.
Speaker 1: Open source R is brilliant for innovation, but it is
Speaker 1: fundamentally incapable of handling true enterprise big data. If you
Speaker 1: try to run an advanced clustering algorithm on a billion
Speaker 1: rows of transaction data using open source R, the memory
Speaker 1: limit will immediately crash the session.
Speaker 2: Just a blue screen of death.
Speaker 1: Basically pretty much.
Speaker 2: So how does already solve this without forcing that PHGD
Speaker 2: data scientist to learn a whole new language. The source
Speaker 2: outlines a three layer architecture to make our database compatible. Right,
Speaker 2: so layer one is simply the client R engine. The
Speaker 2: data scientist sits at their laptop and write standard R
Speaker 2: code in their normal ide. They don't change their workflow
Speaker 2: at all.
Speaker 1: Okay, and layer two is where the magic happens.
Speaker 2: Right.
Speaker 1: The database has this transparency layer.
Speaker 2: Yes, this is the crucial translation mechanism. When the data
Speaker 2: scientist writes an OUR command to filter a data set
Speaker 2: or apply a transformation, the transparency layer intersects that command. Oh, okay,
Speaker 2: does not pull the data to the laptop. Instead, it
Speaker 2: dynamically translates the R syntax into a highly optimized SEQL query.
Speaker 2: It maps the R data frames directly to Oracle tables
Speaker 2: or views.
Speaker 1: That is wild. So the data scientist thinks they are
Speaker 1: manipulating a local R data frame, but behind the scenes,
Speaker 1: Oracle is essentially spoofing the environment and executing a native
Speaker 1: SQL query on the server exactly.
Speaker 2: It's totally seamless. And then layer three consists of spawned
Speaker 2: R engines running directly on the database server itself. If
Speaker 2: the data scientist uses an ORE package like ort M,
Speaker 2: which maps directly to Oracle data mining algorithms, the execution
Speaker 2: happens entirely inside the kernel using parallel processing.
Speaker 1: But wait, what if they are using a custom third
Speaker 1: party R package that Oracle doesn't natively map to. How
Speaker 1: do you keep the memory from crashing.
Speaker 2: Then that's where functions like or dot row apply come in.
Speaker 2: It allows the database server to partition the massive data
Speaker 2: set into manageable chunks, spawn multiple R engines directly on
Speaker 2: the server in parallel, feed the data chunks to those engines,
Speaker 2: and then reassemble the results at the end.
Speaker 1: Oh that's incredibly smart.
Speaker 2: Yeah, you get the full analytical power of custom our
Speaker 2: packages without ever moving the data across the network or
Speaker 2: overwhelming a single machines RAM.
Speaker 1: If we connect this to the bigger picture. This integration
Speaker 1: is the ultimate bridge. You are taking the rapid innovation
Speaker 1: and the massive crowd source brilliance of the open source
Speaker 1: our community, and you're seamlessly plugging it directly into the
Speaker 1: heavy duty, industrial scale processing power of an enterprise database.
Speaker 2: You get the best of both worlds. The agility of
Speaker 2: open source statistical libraries combined with the scalability, parallel execution,
Speaker 2: and strict security of a Tier one database. You just
Speaker 2: don't have to compromise anymore.
Speaker 1: This has been a really fascinating exploration. We've completely broken
Speaker 1: down why the traditional paradigm of data science, you know,
Speaker 1: extracting the data from the source and moving it to
Speaker 1: the math, is just a fragile, bottlenecked system. And we've
Speaker 1: seen how inverting that paradigm, bringing them directly to the
Speaker 1: data through Oracle Data Minor advanced SQL analytics and that
Speaker 1: awesome transparency layer of Oracle our Enterprise allows organizations to
Speaker 1: execute real time, highly scalable predictions without moving a single
Speaker 1: byte of data.
Speaker 2: It permanently alters the velocity in which an enterprise can
Speaker 2: generate actionable foresight.
Speaker 1: It changes everything, and for you listening understanding this architectural
Speaker 1: shift puts you at a massive advantage, whether you are
Speaker 1: architecting a back end, leading a business unit, or just
Speaker 1: tracking the evolution of AI. Knowing that data movement and
Speaker 1: data preparation are the true hidden constraints of machine learning
Speaker 1: really changes how you should evaluate every new tech solution
Speaker 1: on the market.
Speaker 2: It absolutely should dictate your strategy and want to leave
Speaker 2: you with a final thought. Tom All over, we spend
Speaker 2: a lot of time discussing the top tier of that
Speaker 2: automation pyramid, the automated statistician with tools actively translating code,
Speaker 2: automatically normalizing variables, and letting decision trees handle data prep.
Speaker 2: The mechanical friction of data science is vanishing.
Speaker 1: Yeah, it's getting so automated.
Speaker 2: So if this trend accelerates over the next decade, what
Speaker 2: happens to the human data scientist? Will the prestigious role
Speaker 2: of data scientists eventually pivot away from writing code entirely,
Speaker 2: transforming them into business strategists who simply understand how to
Speaker 2: ask a database the right strategic question.
Speaker 1: It's an incredible thought. From spending eight hours peeling potatoes
Speaker 1: to finally just sitting at the chef's table and designing
Speaker 1: the menu thanks for taking the deep dive with us.