Speaker 1: All right, so we've got a ton of information here
Speaker 1: about how to actually set up an analytics system.
Speaker 2: Yeah, it seems like you're really looking to get into
Speaker 2: the modern best practices for this.
Speaker 1: Absolutely, and I think this is going to be really
Speaker 1: fascinating because we've got some excerpts from the Analytics set
Speaker 1: Up Guidebook by Holistics.
Speaker 2: Yeah, and Holistics they've got years of experience helping companies
Speaker 2: build their business intelligence capabilities exactly, so they definitely know
Speaker 2: what they're talking about.
Speaker 1: Yeah, they know their stuff. So what we're going to
Speaker 1: do is give you a really clear roadmap to building
Speaker 1: a modern and scalable analytics system and.
Speaker 2: Hopefully cut through a lot of the jargon that's out there. Yeah, sure,
Speaker 2: and really highlight some of the most fascinating bits from
Speaker 2: this guidebook.
Speaker 1: Now, they start out by emphasizing that any analytics system,
Speaker 1: no matter how complex, it ultimately boils down to three
Speaker 1: key actions.
Speaker 2: Yeah, what they call the core trio, so loading the data,
Speaker 2: transforming the data, and then actually using the data.
Speaker 1: Okay, I like that the core trio.
Speaker 2: Yeah, it's a simple framework, but it's really powerful just
Speaker 2: because it helps you kind of break down, you know,
Speaker 2: this whole complex system into like manageable steps.
Speaker 1: Yeah, for sure. So step one then is loading data.
Speaker 1: Seems pretty straightforward, right, just get all your data into
Speaker 1: one central location.
Speaker 2: Yeah in principle, yeah, but in reality it can quickly
Speaker 2: become a logistical nightmare. I can imagine, you know, think
Speaker 2: about all the different sources you might have, Like you know,
Speaker 2: you've got data in your app, you've got your CRM,
Speaker 2: all your marketing platforms, and then you know, maybe even
Speaker 2: those dreaded spreadsheets lurking in the shadows.
Speaker 1: Yeah.
Speaker 2: So consolidating all of that into a central repository it's
Speaker 2: no small feet, No.
Speaker 1: It's not. This makes me think of like herding cats.
Speaker 2: You know what I mean, it really is, Yeah, except
Speaker 2: the cats are data points from a dozen different systems.
Speaker 1: Yeah, exactly.
Speaker 2: So this is where this concept of a data warehouse
Speaker 2: comes in. It's basically, you know, it's a massive centralized
Speaker 2: storage system that's really specifically designed for housing and managing
Speaker 2: all of your organization's data.
Speaker 1: Okay, so that's our central hub, the data warehouse.
Speaker 2: Yeah.
Speaker 1: But you know, the guidebook mentions that some early stage
Speaker 1: companies might try to skip this step. Is that ever
Speaker 1: a good idea?
Speaker 2: It's a delicate balance. You know, if you're truly early stage,
Speaker 2: maybe you just have one data source not a lot
Speaker 2: of traffic, you might be able to get away with
Speaker 2: querying directly from your production database for a short period
Speaker 2: of time.
Speaker 1: I mean, but that just seems like that would put
Speaker 1: such a strain on your system, potentially even affecting your
Speaker 1: users exactly.
Speaker 2: And that's one of the big risks, right It's like
Speaker 2: it's like trying to run a marathon while carrying a
Speaker 2: heavy backpack. Okay, you know, you might make it to
Speaker 2: the finish line, but it's going to be slow and painful, ye,
Speaker 2: and you run the risk of tripping and injuring yourself,
Speaker 2: which in this analogy, would be you know, equivalent to
Speaker 2: causing downtime for your application.
Speaker 1: Right right now, that makes perfect sense. And what about
Speaker 1: companies that are using you know, no secal databases like
Speaker 1: Mango dB. Are those suitable for analytics.
Speaker 2: Generally no no SQL databases. They're great for handling specific
Speaker 2: types of data and workloads, but they're not really optimized
Speaker 2: for the complex queries that are essential for business intelligence. Okay,
Speaker 2: so trying to perform in depth analytics on a no
Speaker 2: SQL database, it's it's kind of like trying to you know,
Speaker 2: trying to write a novel on a typewriter that's designed
Speaker 2: for short memos. Right.
Speaker 1: Okay, so even for startups that are just starting out,
Speaker 1: you're advocating for them to move towards you know, this
Speaker 1: dedicated analytics setup as soon as possible.
Speaker 2: Yeah. Absolutely. I mean there is this you know, dump
Speaker 2: and load method which is basically just exporting data to
Speaker 2: like local files, you know, to analyze it. And that
Speaker 2: might work in the very very early stages, but it
Speaker 2: quickly becomes unsustainable, Okay. It just it lacks the structure
Speaker 2: and the scalability and automation that you get with a
Speaker 2: proper analytics system.
Speaker 1: Okay, so we've established that this data warehouse is essential,
Speaker 1: but how do we actually get all of our data
Speaker 1: into it? That seems like a really big task.
Speaker 2: Yeah, for sure. And that's where data consolidation comes in.
Speaker 2: And this is you know, a critical part of building
Speaker 2: a modern analytics stack. And the guidebook talks about a
Speaker 2: key shift in thinking here, moving from this traditional ETL
Speaker 2: approach to a more modern ELT approach.
Speaker 1: Okay, so ETL versus ELT. What's the difference and why
Speaker 1: is this shift so important? Right?
Speaker 2: So, ETL stands for extract, Transform load, and that's the
Speaker 2: traditional method where you extract a data from all your
Speaker 2: various sources, you transform it into a usable format outside
Speaker 2: of the data warehouse, and then finally you load it
Speaker 2: into the warehouse.
Speaker 1: Okay, so it's like before you put it into storage,
Speaker 1: you're meticulously cleaning and organizing everything.
Speaker 2: Yeah, exactly. But this approach, it has some significant drawbacks.
Speaker 2: You know, as the volume and variety of data increases,
Speaker 2: transforming everything before loading it into the warehouse can really
Speaker 2: create a massive bottleneck.
Speaker 1: So it's like having a single narrow doorway into our
Speaker 1: data warehouse.
Speaker 2: Great way to put it, Yeah, and it really slows
Speaker 2: everything down. So ELT, which stands for extract load transform,
Speaker 2: basically flips the script to extract the data, you load
Speaker 2: it into the data warehouse in its raw, unprocessed format,
Speaker 2: and then you transform it within the warehouse itself.
Speaker 1: So we're dumping everything into the warehouse first and then
Speaker 1: organizing later. It just seems so counterintuitive.
Speaker 2: Yeah, I know it seems that way, right.
Speaker 1: But what are the benefits of this approach.
Speaker 2: Well, there are several first eliminates that bottleneck that we
Speaker 2: talked about. That's caused by transforming everything outside the warehouse. Okay,
Speaker 2: and cloud data warehouses are incredibly powerful these days, you know,
Speaker 2: so you can actually leverage that processing power for the transformation.
Speaker 1: It's like using one of those, you know, industrial sized
Speaker 1: vacuum cleaners versus like a tiny handheld vacuum cleaner.
Speaker 2: Exactly. Yeah, it's a much more efficient use of resources.
Speaker 1: Okay.
Speaker 2: Secondly, ELT allows for a more agile, dumb first, transform
Speaker 2: later approach, so you can load all your raw data
Speaker 2: into the warehouse without having to like meticulously define every
Speaker 2: transformation upfront.
Speaker 1: Okay, so you're saying that's particularly valuable if your company's
Speaker 1: data needs are still evolving.
Speaker 2: Exactly. Yeah, if you're not entirely sure what insights you're
Speaker 2: looking for yet, you can just get all the data
Speaker 2: in there and then worry about the transformation later.
Speaker 1: Okay, that makes sense.
Speaker 2: And then here's another key advantage. ELT empowers data analysts
Speaker 2: to actually take ownership of the transformation process.
Speaker 1: Wait, so the analysts are actually writing the transformations themselves.
Speaker 1: I always thought that was like a data engineer's job.
Speaker 2: Yeah, traditionally it has been, right, but with ELT, analysts
Speaker 2: can actually use SQL okay, which is the language they're
Speaker 2: already familiar with to define and manage those transformations within
Speaker 2: the data warehouse.
Speaker 1: So it freeze them up. They don't have to rely
Speaker 1: on the data engineers as much.
Speaker 2: Exactly. It reduces their dependency on the engineers, and then
Speaker 2: they can be much more agile and responsive to you know,
Speaker 2: the changing business needs.
Speaker 1: Yeah, I mean that must be incredibly empowering for them.
Speaker 2: Yeah.
Speaker 1: Now, before we move on from this section, I know
Speaker 1: data laks are a part of this modern data ecosystem,
Speaker 1: but I'm not exactly sure how they fit into this
Speaker 1: this whole ELT picture.
Speaker 2: Right. So, a data lake is basically a vast, unstructured
Speaker 2: repository for all of your raw data, regardless of its format.
Speaker 2: Think of it as a massive data holding tank. So
Speaker 2: you can dump everything into the data lake, and then
Speaker 2: you selectively pull out the specific data that you need
Speaker 2: for analysis using ELT.
Speaker 1: Okay, so we can dump everything into the data lake
Speaker 1: and then pull out what we need when we're ready
Speaker 1: to actually analyze it.
Speaker 2: Exactly. Yeah, It's kind of like having a giant pantry
Speaker 2: where you store all of your ingredients and then you
Speaker 2: pull out the specific ones you need when you're ready
Speaker 2: to cook a particular dish.
Speaker 1: Okay, all right, I like that analogy. So now that
Speaker 1: we've got you know, our data into the warehouse, what
Speaker 1: happens next.
Speaker 2: Well, now it's time for step two of the core trio,
Speaker 2: which is transforming that raw data into something we can
Speaker 2: actually use to gain insights.
Speaker 1: So we're molding it to fit our needs.
Speaker 2: Yeah, exactly, And this is where we start shaping and
Speaker 2: molding the data to fit our specific business needs. And
Speaker 2: you know, there are a lot of potential benefits to
Speaker 2: doing good data transformations. It's not just about making the
Speaker 2: data look pretty. It's about ensuring consistency, making it easier
Speaker 2: to analyze, and even potentially saving money.
Speaker 1: Right, Okay, so let's dive into those benefits a little
Speaker 1: bit more. What are some of the key advantages of
Speaker 1: really well designed data transformations.
Speaker 2: Well, one of the key benefits is that it helps
Speaker 2: ensure consistency across the entire organization. Okay, so by creating
Speaker 2: standardized definitions and calculations, you can avoid this dreaded thing
Speaker 2: called metric drift.
Speaker 1: Well, metric drift, yeah, I've heard of that.
Speaker 2: Which is basically where different departments end up calculating the
Speaker 2: same thing in different ways. Leading to confusion and inaccurate reporting.
Speaker 1: Right, So we want to make sure that like everyone
Speaker 1: speaking the same language exactly.
Speaker 2: Yeah, like everyone in the organization is using the same dictionary. Okay,
Speaker 2: so when we talk about key metrics, we all know
Speaker 2: what we're talking about, right, We're all on the same page, exactly.
Speaker 2: Another advantage is reusability, okay. So by defining transformations as
Speaker 2: like modular components, you can easily reuse them across different
Speaker 2: analyzes and reports, okay, which saves you a lot of
Speaker 2: time and effort. But it also ensures that all your
Speaker 2: insights are based on the same underlying logic.
Speaker 1: So we're not reinventing the wheel every time we need
Speaker 1: to pull a new report or something. Yeah, okay.
Speaker 2: And that brings us to the third major benefit, which
Speaker 2: is performance and cost effectiveness.
Speaker 1: Okay.
Speaker 2: You know, well designed transformations can optimize your queries, make
Speaker 2: them run faster and more efficiently, which can translate into
Speaker 2: significant cost savings, especially when you're dealing with large volumes
Speaker 2: of data in a cloud data.
Speaker 1: Warehouse, right where you're paying for compute time okay exactly.
Speaker 2: So yeah, good data transformations. They're like building a solid
Speaker 2: foundation for a house. You know, it might not be
Speaker 2: the most glamorous part, but it's essential for stability, efficiency,
Speaker 2: and long term success.
Speaker 1: Okay, I like that. So it's like the foundation. So
Speaker 1: what are some what does a data transformation look like
Speaker 1: in practice?
Speaker 2: Yeah, So to give you a practical sense of what
Speaker 2: this looks like, the guidebook actually walks us through a
Speaker 2: really basic example of using SQL to transform raw booking
Speaker 2: data into a daily summary. And it's really quite elegant
Speaker 2: in its simplicity.
Speaker 1: Oka.
Speaker 2: It really demystifies that process of data transformation and shows
Speaker 2: you how with a little bit of SQL knowledge, analysts
Speaker 2: can really take ownership of this crucial step.
Speaker 1: Yeah, it's really accessible. Yeah. So okay, so we've loaded
Speaker 1: the data, we've transformed it. What is the final step
Speaker 1: in this core trio?
Speaker 2: Right? So the final step, of course, is actually using
Speaker 2: that data. Right.
Speaker 1: It's all about using it getting value out of it exactly.
Speaker 2: And the guidebook refers to this as data servicing.
Speaker 1: Data servicing, okay, And they.
Speaker 2: Talk about how the role of data analysts has evolved
Speaker 2: significantly in recent years.
Speaker 1: You know, I was just thinking about that the other day, like,
Speaker 1: how how much that has changed?
Speaker 2: Yeah, it's really fascinating. How this role has evolved, and
Speaker 2: they actually frame it as a tale of three jobs.
Speaker 1: A tale of three jobs.
Speaker 2: Okay, highlighting you know, the journey of data analysts from
Speaker 2: like report monkeys to self service enablers.
Speaker 1: Okay, I definitely want to hear more about this.
Speaker 2: Yeah, it's a really interesting story. So in the early
Speaker 2: days of BI, you know, data analysts were often relegated
Speaker 2: to the role of report monkeys.
Speaker 1: Okay.
Speaker 2: They basically spent most of their time manually generating reports
Speaker 2: for decision makers.
Speaker 1: Okay.
Speaker 2: It was really tedious and reactive, and they often felt
Speaker 2: like they were just churning out spreadsheets okay, and not
Speaker 2: really having any opportunity to actually analyze the data.
Speaker 1: Right, They're just you know, putting together spreadsheets.
Speaker 2: Exactly, And it was a recipe for burnout.
Speaker 1: Oh absolute, I can imagine.
Speaker 2: But then came the advent of self service BI tools
Speaker 2: like Tableau, oh, Tableau, which promised to liberate analysts from
Speaker 2: you know, the shackles of manual reporting and empower business
Speaker 2: users to actually explore the data themselves.
Speaker 1: Okay, so they're empowered. That sounds like a really good thing.
Speaker 2: Freedom It certainly seemed that way initially, but as the
Speaker 2: guidebook points out this shift to self service BI, while
Speaker 2: empowering in many ways, also introduced some new challenges.
Speaker 1: Okay, so there's a butt coming.
Speaker 2: Yeah.
Speaker 1: What were some of the downsides of this self service revolution?
Speaker 2: Well, one of the biggest issues was the rise of
Speaker 2: metric drift.
Speaker 1: Okay.
Speaker 2: With no centralized definition of governance around key metrics, you know,
Speaker 2: different departments started calculating things in slightly different ways, leading
Speaker 2: to you know, inconsistencies and making it difficult to really
Speaker 2: get a clear picture of what was actually happening.
Speaker 1: Right, So again, like everyone speaking a different language exactly.
Speaker 2: Yeah, it's like everyone having their own version of the truth, right,
Speaker 2: which doesn't sound like a recipe for good decision making.
Speaker 1: Oh no, it doesn't. So is there something else where
Speaker 1: do we go from there?
Speaker 2: Right? So that's where the third stage in the evolution
Speaker 2: of the data analyst comes in. And this is where
Speaker 2: you know, tools like Looker and Holistics enter the scene
Speaker 2: with their emphasis on data modeling layers and a more
Speaker 2: balanced approach to self service.
Speaker 1: Okay, so data modeling layers, I think you've mentioned this before.
Speaker 1: Can you just explain what those are and why they're
Speaker 1: so important?
Speaker 2: Right? A data Modeling Layer is basically an abstraction layer
Speaker 2: that sits on top of your raw data, and it
Speaker 2: allows you to define business logic, calculations, relationships, and standardized
Speaker 2: definitions for all your key metrics in a centralized location.
Speaker 1: So rather than just having like the business logic baked
Speaker 1: into every individual report or dashboard exactly.
Speaker 2: Yeah, you're basically creating a central repository, okay, of all
Speaker 2: those definitions and calculations that can be reused to cross
Speaker 2: the organization.
Speaker 1: It's like a blueprint for your data.
Speaker 2: Exactly, So everyone is working from the same set of plans.
Speaker 2: And this not only helps to prevent metric drift, but
Speaker 2: it also makes it much easier for business users to
Speaker 2: actually explore the data independently without you know, getting lost
Speaker 2: in the technical complexities.
Speaker 1: Yeah, you know. And there's there's this great anecdote in
Speaker 1: the book about a CEO who needed specific data and
Speaker 1: how data modeling layer empowered them to be able to
Speaker 1: get that data themselves without having to go through the
Speaker 1: data team.
Speaker 2: Yeah, that's a great anecdote. It illustrates how this can
Speaker 2: really bridge the gap, you know, yeah, between technical and
Speaker 2: non technical users and make data accessible to everyone. You know,
Speaker 2: so instead of having to rely on the data team
Speaker 2: for every single request, business users can actually explore the
Speaker 2: data themselves, okay, armed with the confidence that they're working
Speaker 2: with accurate and consistent definitions.
Speaker 1: Yeah, it seems like a win win. The data analysts
Speaker 1: don't have to just focus on those mundane reporting tasks,
Speaker 1: and business users are able to kind of answer their
Speaker 1: own questions exactly.
Speaker 2: And you know, the guidebook highlights the key benefits of
Speaker 2: this approach. It's increased self service analytics, okay, more efficient
Speaker 2: use of your data team's resources, and a well documented
Speaker 2: and consistent layer of data knowledge that everyone can access.
Speaker 1: Okay, So data modeling layers very important piece of the
Speaker 1: puzzle for sure. So let's talk about data modeling itself
Speaker 1: a little bit more. You sent me some really interesting
Speaker 1: stuff on Kimball's dimensional data modeling, with I think is
Speaker 1: a very important framework in this field. Yeah.
Speaker 2: Absolutely, Kimball's Dimensional data Modeling. It's it's a classic for
Speaker 2: a reason. You know, even in today's cloud first world,
Speaker 2: the core concepts are incredibly relevant.
Speaker 1: Okay.
Speaker 2: It's it's like learning the fundamentals of music theory. You know.
Speaker 2: You can always add your own flare later on, yeah,
Speaker 2: but those basics they're essential, right, So what are some.
Speaker 1: Of those core concepts those essentials.
Speaker 2: Well, one of the core concepts is this idea of
Speaker 2: a data model itself. So a data model is essentially
Speaker 2: an abstract representation of your business data.
Speaker 1: Okay.
Speaker 2: It defines the entities, the attributes, the relationships that are
Speaker 2: important for your analysis. It's kind of like creating a map,
Speaker 2: you know.
Speaker 1: Okay, So it's like our map of our data exactly. Okay,
Speaker 1: So what else do we need to navigate this terrain?
Speaker 2: Well? Another key concept is what's called relationship mapping.
Speaker 1: Okay.
Speaker 2: So, just like in a relational database, you need to
Speaker 2: define how different data models relate to each other. So,
Speaker 2: for example, a customer model might have a relationship with
Speaker 2: an orders model okay, right, which allows you to analyze
Speaker 2: customer purchasing behavior.
Speaker 1: So it's like connecting the docks between different data.
Speaker 2: Points exactly, Yeah, to reveal that bigger picture.
Speaker 1: Okay.
Speaker 2: Then there's what's called custom field logic, which allows you
Speaker 2: to create calculated fields within your modelsky. For example, you
Speaker 2: might define total revs new as the sum of all
Speaker 2: sales transactions.
Speaker 1: Right, So we're going beyond the raw data we're adding calculations.
Speaker 2: To it exactly. Yeah, you're adding layers of meaning and
Speaker 2: calculations to make it more relevant to your specific business needs. Right.
Speaker 2: And then finally, there's this idea of models built on
Speaker 2: top of other models, okay. And this allows you to
Speaker 2: create increasingly sophisticated and nuanced representations of your data.
Speaker 1: Okay.
Speaker 2: So, for example, you could combine a customer model with
Speaker 2: a product model okay, and create what's called a customer
Speaker 2: segmentation model okay, right, which allows you to group customers
Speaker 2: based on their purchasing patterns.
Speaker 1: So we're building like a hierarchy of models.
Speaker 2: Exactly, Yeah, each one adding more you know, granularity and insight.
Speaker 1: Okay. So, and this brings us to the star of
Speaker 1: the show, which I think you mentioned earlier, the Star schema. Yes,
Speaker 1: the Star schema, So tell me about that.
Speaker 2: It's a classic for a reason, okay. And it's it's
Speaker 2: basically a specific way of organizing your data for analysis
Speaker 2: that makes it incredibly efficient for querying and reporting. It's
Speaker 2: called a star schema because visually it kind of resembles
Speaker 2: a star. You have a central fact table and that's
Speaker 2: surrounded by multiple dimension tables.
Speaker 1: Okay. So the fact table is the heart of the
Speaker 1: star exactly, and then the dimensions are like the points
Speaker 1: radiating outward, like what goes into each of those?
Speaker 2: Right? So the fact table contains the core metrics you
Speaker 2: want to analyze, okay. Often these are numerical values like
Speaker 2: sales figures or website visits, you know, customer interactions. It's
Speaker 2: like the what of your analysis okay, the what right,
Speaker 2: and the dimension tables provide the context okay, and descriptive
Speaker 2: information about those facts okay, you know, like the who, what, where,
Speaker 2: when and why of your data.
Speaker 1: Okay. So, for example, if we're looking at you know,
Speaker 1: sales data, our fact table might contain like date, product ID, quantity,
Speaker 1: souled price exactly. Yeah. And then our dimension tables would
Speaker 1: be things like the customer, the product, you know, the
Speaker 1: location and things like that precisely. Yeah. Yeah.
Speaker 2: And by linking these tables together, you can start to
Speaker 2: answer those, you know, complex questions about your data, like
Speaker 2: which products are selling best in which regions? Or what
Speaker 2: are the demographics of our highest spending customers.
Speaker 1: So we're adding layers of details exactly.
Speaker 2: Yeah. And the guidebook does a great job of, you know,
Speaker 2: walking us through a practical example of designing a staff
Speaker 2: schema using the case of modeling data from a point
Speaker 2: of sale system, and they emphasize this importance of choosing
Speaker 2: the right grain, you know, which refers to the level
Speaker 2: of detail that's captured in that fact table. Okay, and
Speaker 2: getting this right it's really crucial for efficient and accurate analysis.
Speaker 1: Yeah, because if you get it wrong, it can lead
Speaker 1: to all kinds of problems, right, like performance issues, data redundancy,
Speaker 1: and even inaccurate results exactly.
Speaker 2: Yeah. It's it's like, you know, trying to build a
Speaker 2: house with bricks that are too big or too small.
Speaker 2: It's just not going to work well.
Speaker 1: Okay. So, like they use the analogy of tracking website traffic, right,
Speaker 1: and if your grain is too coarse, you might only
Speaker 1: be tracking visits at you know, the website level, but
Speaker 1: you're missing details about like the page views right exactly.
Speaker 1: But if it's too fine, you're tracking every single click
Speaker 1: and mouse movement.
Speaker 2: Yeah, and you end up with this massive, unwieldy data set.
Speaker 1: Okay, So it's really important to find that that sweet
Speaker 1: spot exactly, so you know, you get the information you
Speaker 1: need without overwhelming yourself with data you don't need, right, now,
Speaker 1: you know, I know the world of data warehousing has
Speaker 1: changed a lot since Kimball first introduced this. Oh yeah,
Speaker 1: I mean, especially with cloud data warehouses and their power
Speaker 1: and affordability.
Speaker 2: Absolutely, and the guidebook emphasizes that, you know, while Kimball's
Speaker 2: principles are still really relevant, some of those specific techniques
Speaker 2: that he talked about can be adapt or even bypassed
Speaker 2: in the context of a modern cloud data warehouse.
Speaker 1: Okay, so there's some things you can skip, yeah, exactly.
Speaker 1: You know, like they gave the example of inventory management,
Speaker 1: and traditionally, you know, if you wanted to track inventory
Speaker 1: levels over time, you'd need these complex snapshot fact tables.
Speaker 1: But with a modern data warehouse, a lot of the
Speaker 1: time you can just work with the raw data exactly.
Speaker 2: Yeah, and that's because of the sheer processing power and
Speaker 2: storage capacity of these modern cloud data warehouses. Yeah, it
Speaker 2: allows us to be much more agile and iterative in
Speaker 2: our in our approach to data modeling. You know, we
Speaker 2: can load the raw data first and then experiment with
Speaker 2: different modeling techniques within the warehouse itself.
Speaker 1: Okay, so you don't have to like figure everything out ufront,
Speaker 1: you can.
Speaker 2: Just experiment exactly, yeah, it's much more flexible, I.
Speaker 1: See, okay, And they also talked about how features like
Speaker 1: table partitioning can simplify how you handle slowly changing dimensions
Speaker 1: or seds.
Speaker 2: Yeah for sure, which used.
Speaker 1: To require you know, these plax modeling.
Speaker 2: Techniques, right, right, So table partitioning is basically a really
Speaker 2: powerful technique. It allows you to divide, you know, a
Speaker 2: large table into smaller, more manageable chunks, okay, based on
Speaker 2: criteria like date ranges. And this can really improve query
Speaker 2: performance and make it a lot easier to manage that
Speaker 2: historical data.
Speaker 1: So instead of having like this one massive table for
Speaker 1: all of our customer data, we might partition it by
Speaker 1: year exactly. Yeah, so if we need data from a
Speaker 1: specific year, it's much faster to get to.
Speaker 2: Yeah, precisely. And then when it comes to ScDs, you know,
Speaker 2: which track changes to dimension attributes over time, table partitioning
Speaker 2: can actually be used to store different versions of a
Speaker 2: dimension in separate partitions, okay, so you can easily track
Speaker 2: those historical changes without having to resort to those complex
Speaker 2: modeling techniques.
Speaker 1: So I mean, with these cloud data warehouses, it seems
Speaker 1: like there's just so much more flexibility.
Speaker 2: Yeah, absolutely, we're no longer limited by those constraints of
Speaker 2: the old traditional on premise data warehouses. So yeah, we
Speaker 2: can experiment more freely, iterate more quickly, and adapt our
Speaker 2: models as our business needs evolve.
Speaker 1: Okay, And you know, they have a really great case
Speaker 1: study that they talk about about their own company, Holistics,
Speaker 1: and they talk about how they implemented a snowcloud analytics system,
Speaker 1: right yeah, and how they took this very iterative approach
Speaker 1: to data modeling.
Speaker 2: It's a great example of putting this putting theory into practice.
Speaker 2: You know, they didn't try to model everything up front.
Speaker 2: They just started with a basic model and then iteratively
Speaker 2: refined it based on how people were actually using the data.
Speaker 1: So the key takeaway is, let usage guide your modeling
Speaker 1: decisions exactly, you know, don't try to over engineer things
Speaker 1: from the start. Start simple and see how people are
Speaker 1: interacting with the data, and then adjust accordingly.
Speaker 2: And they highlight some key principles that guided their process,
Speaker 2: like embedding business logic in the data models sometulf okay,
Speaker 2: rather than individual queries and aiming for you know, self
Speaker 2: serve and analytics by really empowering those business users to
Speaker 2: explore the data independently.
Speaker 1: So we're creating a system that's both you know, robust
Speaker 1: and flexible and sure it's consistency, but also allows for exploration.
Speaker 2: Yeah, finding that balance.
Speaker 1: Okay, And this brings us to this this fascinating concept
Speaker 1: of the arc of adoption, right, Yeah, which describes the
Speaker 1: typical stages of data usage evolution within an organization.
Speaker 2: Yeah.
Speaker 1: It's a really great framework for understanding like how data
Speaker 1: driven thinking takes root and.
Speaker 2: Matures exactly, and understanding where your organization sits on this
Speaker 2: arc can help you make much more informed decisions.
Speaker 1: Okay, So tell me about this arc of adoption. I mean,
Speaker 1: you know, we've already talked about the fact that, like
Speaker 1: in those early stages, there's this heavy reliance on you know,
Speaker 1: spreadsheets and ad hoc analysis, right, So, what are some
Speaker 1: of the telltale signs that a company's in this phase
Speaker 1: and what are some of the challenges they might face.
Speaker 2: Well, Well, in this initial phase, the data is often
Speaker 2: scattered across various sources, and there's this heavy reliance on
Speaker 2: manual processes to extract you know, basic insights. Think about
Speaker 2: you know those late nights you spend cobbling together reports
Speaker 2: and excel. Oh, Yeah, I've been there trying to make
Speaker 2: sense of data from different departments and systems, and you know,
Speaker 2: it's functional, but it's so time consuming and prone to errors,
Speaker 2: and it just doesn't scale well as your organization grows,
Speaker 2: Like trying.
Speaker 1: To build a skyscraper with hand tools exactly.
Speaker 2: Yeah, you might be able to lay a few bricks,
Speaker 2: but you're not going to.
Speaker 1: Get very far right right.
Speaker 2: And the guidebook it highlights some of the key challenges
Speaker 2: of this stage, like data inconsistency, lack of standardization. You know,
Speaker 2: it's really difficult to collaborate, and there's a very limited
Speaker 2: ability to answer those more complex business questions.
Speaker 1: It's a lot of frustration from both you know, the
Speaker 1: people putting together the reports and then the decision makers
Speaker 1: who are trying to use that information.
Speaker 2: For sure, and this is often the point where companies ree, okay,
Speaker 2: we need a more robust and scalable solution, and they
Speaker 2: begin to invest in bi tools, which kind of moves
Speaker 2: them to that second stage of the arc of adoption, okay,
Speaker 2: which is charpitorized by those you know, static reports and dashboards.
Speaker 2: So this is where we start to see you know,
Speaker 2: those colorful charts and graphs that executives love, right right, Yeah,
Speaker 2: and dashboards they provide this uh, you know, more centralized
Speaker 2: and visual way to track those key metrics, so it
Speaker 2: makes it easier to monitor progress, identify trends, and bi
Speaker 2: tools play a much more prominent role at this stage,
Speaker 2: host speaker.
Speaker 1: Right, they start automating those reporting processes exactly.
Speaker 2: Yeah, and they make data more accessible to a wider audience.
Speaker 1: Okay, so it's like, you know, we've gone from that
Speaker 1: hand drawn map to the GPS.
Speaker 2: That's a great way to put it.
Speaker 1: Yeah, we get a better picture of you know, where
Speaker 1: we are and where we're going. Yeah, but we're still
Speaker 1: kind of on a predetermined.
Speaker 2: Route, right And dash towards you know, they provide valuable insights,
Speaker 2: but they don't necessarily empower users to you know, explore
Speaker 2: the data freely or ask those what if questions that
Speaker 2: often lead to those really groundbreaking discoveries.
Speaker 1: And that brings us to like, you know, the the
Speaker 1: holy grail of data maturity, right, self service analytics exactly
Speaker 1: the third stage of this arc of adoption, And this
Speaker 1: is where our users can you know, they have the
Speaker 1: ability to access, analyze, and visualize data independently without having
Speaker 1: to go to the data team for every single request exactly.
Speaker 2: Self service analytics really represents this shift from a centralized,
Speaker 2: it driven model to a much more decentralized user driven approach.
Speaker 2: It's like, you know, having a fully interactive map, you
Speaker 2: can zoom in and out, explore different routes, discover like
Speaker 2: those hidden gems, you know, and even create your own
Speaker 2: personalized maps.
Speaker 1: So incredibly empowering for those users. Yeah, but I know
Speaker 1: that the guidebook cautions that just giving people BI tools
Speaker 1: isn't enough, no at all, to really you know, experience
Speaker 1: self service analytics.
Speaker 2: Yeah, simply giving everyone access to data without the right foundation,
Speaker 2: it's it's a recipe for disaster. Okay, you know, it's
Speaker 2: it's like giving someone like a really powerful sports car, okay,
Speaker 2: without any driving lessons or you know, knowledge of traffic laws.
Speaker 1: Right, you know, they might be able to get a
Speaker 1: moving but it's probably going to end in a crash.
Speaker 2: Yeah, that's a that's a scary thought. So what are
Speaker 2: those essential ingredients?
Speaker 1: Yeah, so the guidebook they highlight three key pillars data literacy,
Speaker 1: data governance, and a robust data modeling layer.
Speaker 2: Okay, let's break those down a little bit.
Speaker 1: Yeah, so data literacy basically means that users need to
Speaker 1: understand how to interpret and work with data, how to
Speaker 1: ask the right questions okay, and then how to draw
Speaker 1: you know, meaningful conclusions from the insights they discovered. Okay.
Speaker 2: So it's not just about having access to thet it's
Speaker 2: knowing what to do with it.
Speaker 1: Yeah, exactly. It's about being able to actually speak the
Speaker 1: language of data. Data governance ensures that the data is accurate, consistent,
Speaker 1: and trustworthy.
Speaker 2: Okay.
Speaker 1: It involves establishing processes for data quality management, defining those
Speaker 1: clear roles and responsibilities for data stewership, and then implementing
Speaker 1: those security measures to protect sensitive information. So we're making
Speaker 1: sure everyone's using the data responsibly exactly.
Speaker 2: Yeah, it's about creating that framework that ensures everyone is
Speaker 2: using the data responsibly and ethically.
Speaker 1: Okay. And then finally, the data modeling layer, which I
Speaker 1: think we've talked about a lot already, but that's the
Speaker 1: foundation really that makes this all possible.
Speaker 2: Yeah. Absolutely. And by defining those business logics and those calculations,
Speaker 2: you know, those relationships in that centralized location, we're creating
Speaker 2: this single source of truth that everyone in the organization
Speaker 2: can actually trust.
Speaker 1: So the modeling layer is our guide exactly.
Speaker 2: Yeah. It ensures that everyone's you know, speaking the same
Speaker 2: language and interpreting the data consistently.
Speaker 1: Okay, So without that things could go very wrong.
Speaker 2: Oh yeah, absolutely. Without a robust data modeling layer, you know,
Speaker 2: self service analytics can really quickly descend into chaos.
Speaker 1: Okay.
Speaker 2: You know you have users potentially creating their own definitions,
Speaker 2: their own calculations, okay, which can lead to all sorts
Speaker 2: of inconsistencies and inaccurate reporting.
Speaker 1: Okay. So you know, they make it very clear that
Speaker 1: that self service analytics is a journey. It's not like
Speaker 1: you just arrived there.
Speaker 2: No, absolutely, yeah, it's it's an ongoing process of learning, adapting,
Speaker 2: and refining. And as organizations become more data driven, the
Speaker 2: demands on that data team they also increase.
Speaker 1: Yeah we talked about that earlier.
Speaker 2: Right, and the guidebook delves into the challenges of scaling
Speaker 2: your BI tools and processes to match the growing data
Speaker 2: needs of the organization.
Speaker 1: Yeah, you know how that team can be overwhelmed too
Speaker 1: many requests for reports and dashboards at hoc analyses.
Speaker 2: Right, Yeah, it's like that small kitchen staff trying to
Speaker 2: cater a banquet for you know, hundreds of guests. It's
Speaker 2: just not sustainable in the long run.
Speaker 1: Right, it's not. So how do we prevent that? What
Speaker 1: are some of those strategies?
Speaker 2: So scalability becomes really really crucial here, and the guidebook
Speaker 2: highlights several strategies for scaling your BI infrastructure, things like
Speaker 2: choosing the right BI tools that can handle those increasing
Speaker 2: data volumes and that user concurrency, you know, implementing a
Speaker 2: really robust data governance process okay to streamline those workflows,
Speaker 2: and then fostering that culture of data literacy throughout the
Speaker 2: entire organization.
Speaker 1: Okay, so working smarter, not harder, exactly.
Speaker 2: Yeah, you know, investing in those right tools, processes, and
Speaker 2: skills really helps those data teams keep pace with those
Speaker 2: growing demands and ensure that the organization can continue to
Speaker 2: make those really data driven decisions as it scales.
Speaker 1: Okay, So technology is obviously, you know, a very important
Speaker 1: part of this, but it's really the people that are
Speaker 1: driving that success.
Speaker 2: Yeah, absolutely, you know, technology is an essential enabler, but
Speaker 2: it's it's ultimately the people who drive that data driven success. So,
Speaker 2: you know, fostering that culture of data literacy, empowering those
Speaker 2: business users and then investing in the development of your
Speaker 2: data professionals. Those are all crucial for building a truly
Speaker 2: data driven organization.
Speaker 1: Okay. So the guidebook concludes by emphasizing that this world
Speaker 1: of business intelligence, I mean it's constantly changing.
Speaker 2: Oh yeah, absolutely, always evolving.
Speaker 1: New tools, new technologies, new approaches. I mean, what's what
Speaker 1: works today might not work tomorrow. So you know we
Speaker 1: got to stay curious, experiment, yeah, definitely, and continue to
Speaker 1: learn and adapt.
Speaker 2: You know, it's like surfing. Now you need to be
Speaker 2: aware of those changing tides, adjust your balance, and you know,
Speaker 2: stay ahead of the curve to really avoid wiping out.
Speaker 1: Okay, I like that. So what does this mean for
Speaker 1: our listeners, Well, you know.
Speaker 2: Building a modern analytics system, it's not just about choosing
Speaker 2: those right tools, it's about really understanding those underlying principles,
Speaker 2: you know, adapting them to your organization's specific needs and growth, Okay,
Speaker 2: and then embracing that culture of continuous learning and improvement.
Speaker 2: So this, this deep diet has really equipped you with
Speaker 2: the knowledge to navigate this really exciting and rapidly evolving field.
Speaker 2: But the journey doesn't end.
Speaker 1: Here, no, it doesn't.
Speaker 2: So keep exploring, keep experimenting and keep pushing the boundaries
Speaker 2: of what's possible with data.
Speaker 1: It really is, and it can help you make those
Speaker 1: decisions about your bi strategy. So let's go back to
Speaker 1: these stages and unpack them a little bit more. You know,
Speaker 1: we know that in the early stages, companies rely heavily
Speaker 1: on spreadsheets and ad hoc analysis. What are some of
Speaker 1: the telltale signs that a company's in this phase and
Speaker 1: what are some of the challenges they might face.
Speaker 2: Yeah, so in this initial phase, data is often, like
Speaker 2: you said, scattered across different sources, and there's a really
Speaker 2: heavy reliance on those manual processes to extract, you know,
Speaker 2: just those basic insights. Think of those late nights, like
Speaker 2: you said, spent cobbling together reports and Excel trying to
Speaker 2: make sense of data from different departments or systems, and
Speaker 2: you know, it's a functional approach, but it's incredibly time consuming,
Speaker 2: prone to errors, and it just doesn't scale well as
Speaker 2: the organization grows.
Speaker 1: It's like trying to build a skyscraper.
Speaker 2: With handles exactly. Yeah, you might be able to lay
Speaker 2: a few bricks, but you're not going to get very far.
Speaker 2: And the guidebook highlights some of those key challenges of
Speaker 2: this stage, you know, data inconsistency, lack of standardization, difficulty
Speaker 2: in collaborating, and a very limited ability to answer those
Speaker 2: those those complex business questions.
Speaker 1: Frustration from both you know, the people who are putting
Speaker 1: together the reports and the decision makers who are trying
Speaker 1: to use it exactly.
Speaker 2: And this is often the point where companies realize, okay,
Speaker 2: we need a more bust and scalable solution, so they
Speaker 2: start to invest in BI tools, which kind of moves
Speaker 2: them to that second stage of the arc of adoption, okay,
Speaker 2: which is characterized by those you know, static reports and dashboards.
Speaker 2: So this is where we start to see those you know,
Speaker 2: colorful charts and graphs that executives love.
Speaker 1: Right and dashboards. They provide a more centralized and visual
Speaker 1: way to track those key metrics, making it easier to
Speaker 1: monitor progress, identify trends, and BI tools play a much
Speaker 1: more prominent role at this stage, automating those reporting processes
Speaker 1: and making data more accessible to a wider audience. So
Speaker 1: we've gone from that hand drawn map to a GPS system.
Speaker 1: We've got a clearer view of where we are and
Speaker 1: where we're going, but we're still following that pre determined route.
Speaker 2: That's a great way to put it. Yeah, dashboards, you know,
Speaker 2: they provide valuable insights, but they don't necessarily empower users
Speaker 2: to explore the data freely or ask those what if
Speaker 2: questions that often lead to those you know, groundbreaking discoveries.
Speaker 1: Right, and that brings us to the holy grail, self
Speaker 1: service analytics exactly. Yeah, the third stage of this arc
Speaker 1: of adoption. Yeah, and this is where our users have
Speaker 1: the ability to access, analyze, and visualize data independently without
Speaker 1: having to go to the data team for every request.
Speaker 2: Exactly. Self service analytics represents a shift from that centralized,
Speaker 2: it driven model to a much more decentralized user driven approach.
Speaker 2: It's like having a fully interactive map. You can zoom
Speaker 2: in and out, explore different routes, discover those hidden gems,
Speaker 2: you know, and even create your own personalized maps.
Speaker 1: Okay, so very empowering for the user. But I know
Speaker 1: the guidebook cautions that just giving people BI tools isn't
Speaker 1: enough to actually be able to do this.
Speaker 2: No, not at all. Simply giving everyone access to data
Speaker 2: without the right foundation is a recipe for disaster. It's
Speaker 2: like giving someone a really powerful supports car you know,
Speaker 2: without any driving lessons or knowledge of traffic laws, they
Speaker 2: might be able to get it moving, but it's probably
Speaker 2: going to end in a crash.
Speaker 1: That's a pretty scary thought. So what are those essential
Speaker 1: ingredients for self service analytics?
Speaker 2: Yes, so the guidebook highlights three key pillars data literacy,
Speaker 2: data governance, and a robust data modeling layer. So data
Speaker 2: literacy basically means that users need to understand how to
Speaker 2: interpret and work with data, how to ask the right questions,
Speaker 2: and then how to draw you know, meaningful conclusions from
Speaker 2: the insights they discover.
Speaker 1: So it's not just about having access to the data,
Speaker 1: it's knowing what to do with it.
Speaker 2: Yeah, exactly, it's about being able to speak the language
Speaker 2: of data data governance that ensures that the data is accurate, consistent,
Speaker 2: and trustworthy. It involves establishing those processes for data quality management,
Speaker 2: defining clearer roles and responsibilities for data stewardship, and then
Speaker 2: implementing those security measures to protect sensitive information, but.
Speaker 1: Making sure everyone's using the data responsibly and.
Speaker 2: Exactly, yeah, it's about creating a framework to make sure
Speaker 2: that happens. And then finally, you know, we have that
Speaker 2: data modeling layer and This is that foundation that makes
Speaker 2: self service analytics possible. By defining those business logics and calculations,
Speaker 2: you know, those relationships in a centralized location, we're creating
Speaker 2: that single source of truth that everyone in the organization
Speaker 2: can trust.
Speaker 1: So that modeling layer is our guide. It ensures that
Speaker 1: everyone speaking the same language interpreting the data consistently exactly.
Speaker 2: And without that robust data modeling layer, you know, self
Speaker 2: service analytics can really descend into chaos where you have
Speaker 2: users potentially creating their own definitions, their own calculations, which
Speaker 2: leads to inconsistencies and inaccurate reporting.
Speaker 1: So it sounds like this is really a journey. It's
Speaker 1: not a.
Speaker 2: Destination, absolutely, yeah, it's it's an ongoing process of learning, adapting,
Speaker 2: and refining. And as organizations become more data driven, the
Speaker 2: demands on the data team also increase, and the guidebook
Speaker 2: really delves into the challenges of scaling your bi tools
Speaker 2: and processes to match those you know, growing data needs
Speaker 2: of the organization.
Speaker 1: Right, so, the data team can easily become overwhelmed too
Speaker 1: many requests for reports, dashboards, ad hoc analyses. How do
Speaker 1: we prevent that?
Speaker 2: What are some strategy so scalability becomes really crucial here,
Speaker 2: and the guidebook highlights several strategies for scaling your BI infrastructure.
Speaker 2: You know, things like choosing the right BI tools that
Speaker 2: can handle those increasing data volumes and that user concurrency,
Speaker 2: implementing a really robust data governance process to streamline workflows,
Speaker 2: and then fostering a culture of data literacy throughout the.
Speaker 1: Organization, so working smarter, not harder, investing the right tools, processes,
Speaker 1: skills to help that data team keep pace with those
Speaker 1: growing demands exactly.
Speaker 2: Yeah, you know, investing in those right tools, those processes,
Speaker 2: and those skills really helps those data teams keep pace
Speaker 2: with those growing demands and ensure that the organization can
Speaker 2: continue to make data driven decisions as it scales.
Speaker 1: Okay, so technology is a really important enabler, but at
Speaker 1: the end of the day, it's really the people that
Speaker 1: are going to be driving data driven.
Speaker 2: Success, right, Yeah, absolutely, technology is an essential enabler, but
Speaker 2: it's ultimately the people who drive that data driven success.
Speaker 2: So fostering that culture of data literacy and power uring
Speaker 2: those business users, and investing in that development of data professionals,
Speaker 2: those are all crucial for building a truly data driven organization.
Speaker 1: And the guide, but concludes by saying that this world
Speaker 1: of business intelligence is constantly evolving with new tools and technologies.
Speaker 1: What works today might be outdated tomorrow, so it's really
Speaker 1: essential to stay curious, experiment, and continue to learn and adapt.
Speaker 2: You know, It's like surfing. You need to be aware
Speaker 2: of those changing tides, adjust your balance, and stay ahead
Speaker 2: of the curve to avoid wiping out.
Speaker 1: Yeah, I like that analogy. So what does all this
Speaker 1: mean for listeners?
Speaker 2: Well, building a modern analytics system, it's not just about
Speaker 2: choosing the right tools, it's about really understanding those underlying principles,
Speaker 2: adapting them to your organization's needs and growth, and embracing
Speaker 2: that culture of continuous learning and improvement.
Speaker 1: This deep dive has really given you the knowledge to
Speaker 1: navigate this exciting and rapidly evolving field. But remember it
Speaker 1: doesn't end here. Keep exploring, keep experimenting, and keep pushing
Speaker 1: the boundaries of what's possible with data.