Speaker 1: Okay, so you're trying to get a real grip on
Speaker 1: something complex, right, and you want it fast, but without
Speaker 1: getting totally buried in jargon and detail.
Speaker 2: Yeah, information overload is real.
Speaker 1: Exactly, So think of this as your shortcut. We're diving
Speaker 1: deep into real time analytics today, trying to give you
Speaker 1: that core understanding, you know, without all the noise.
Speaker 2: And we're basing this on the book Building real Time
Speaker 2: Analytics Systems. It just came out September twenty twenty three, first.
Speaker 1: Edition, right, and the book's goal seems pretty practical, just
Speaker 1: helping you, the listener, get your job done if you're
Speaker 1: working in this space.
Speaker 2: Pretty much, it cuts through the theory to the how
Speaker 2: to now.
Speaker 1: Think back maybe early two thousands data analytics. It's often
Speaker 1: felt like something you did after everything else happened, you know,
Speaker 1: batch process.
Speaker 2: Oh definitely, reports would tell you what happened yesterday or
Speaker 2: last week hindsight basically.
Speaker 1: But things have really shifted, haven't they. There's this like
Speaker 1: massive appetite now for knowing things the moment they happen
Speaker 1: real time.
Speaker 2: Absolutely. The book uses fraud detection as a great example. Yeah,
Speaker 2: finding out about fraud hours later?
Speaker 1: Uh uh, too late? Right, the money's probably done exactly.
Speaker 2: The real wind is spotting it now, flagging it, maybe
Speaker 2: blocking it instantly. That immediacy is key. It's not just
Speaker 2: nice to have anymore often, it's well essential.
Speaker 1: And that brings us to this idea of streaming. It's
Speaker 1: not about waiting for a whole file to finish downloading
Speaker 1: or collecting.
Speaker 2: No, not at all. Think of it more like a
Speaker 2: continuous flow, a river of data that just keeps coming.
Speaker 2: It never really ends.
Speaker 1: And the crucial part is you can dip into that
Speaker 1: river and act on what you see right then and there.
Speaker 2: Precisely. A data stream fundamentally is just a series of
Speaker 2: data points ordered by time. Each one represents some kind
Speaker 2: of event or a change.
Speaker 1: Like what give us an example, well, like.
Speaker 2: Every single purchase on an e commerce site, or every
Speaker 2: reading from an IoT sensor, maybe temperature of pressure. It's
Speaker 2: like a constant pulsive information.
Speaker 1: Okay, okay, And here's a point the book really stresses,
Speaker 1: which I found fascinating. Events have a shelf.
Speaker 2: Life, a very short one.
Speaker 1: Sometimes their value can just like plummet super fast. Think
Speaker 1: about an online shopping cart someone just abandon right if.
Speaker 2: You can ping them with an SMS or an email.
Speaker 2: Maybe with a little discount voucher, Like.
Speaker 1: Immediately you might get that sale back.
Speaker 2: You've got a decent shot. Yeah, but wait, even just
Speaker 2: a couple of hours, they've moved.
Speaker 1: On, bought somewhere else, or just changed their mind exactly.
Speaker 2: The timing that immediate reaction makes all the difference, and
Speaker 2: that is the heart of real time analytics or RTA.
Speaker 2: It's all about squeezing value from those events basically as
Speaker 2: soon as they happen.
Speaker 1: The book mentions soft real time. What's that about. Does
Speaker 1: that mean it's not quite real time?
Speaker 2: Well, yeah, kind of. It just acknowledges that, you know,
Speaker 2: perfection is hard. There might be tiny delays milliseconds maybe
Speaker 2: seconds because of network latency or system hiccups. It's not instantaneous,
Speaker 2: but it's very very close.
Speaker 1: Okay, so practical real time. The big difference is compared
Speaker 1: to batch processing right batches.
Speaker 2: Where you collect data over time maybe an hour, maybe
Speaker 2: a day, put it in a big chunk, and then
Speaker 2: analyze it.
Speaker 1: We used to set up these artificial deadlines. Didn't we
Speaker 1: run the report at midnight for yesterday's data.
Speaker 2: Yeah, those time boundaries. The problem is your analysis is
Speaker 2: always looking backwards, you're getting insights about what was happening.
Speaker 1: Which might be stale news by the time you get
Speaker 1: it totally.
Speaker 2: RTA aims to give you a view of the present,
Speaker 2: so your decisions are actually relevant to now.
Speaker 1: So our mission for this deep dive drawing from the
Speaker 1: book is to really get into those core concepts and
Speaker 1: importantly the benefits. What do you actually gain from this?
Speaker 2: Let's talk benefits. Then. Speed is obviously a big one.
Speaker 2: The book argues it's often a decisive factor. Market leaders
Speaker 2: tend to be faster.
Speaker 1: Faster at understanding, faster at reacting.
Speaker 2: Exactly, and RTA helps achieve that. For one thing, it
Speaker 2: can actually open up totally new revenue streams now. So well,
Speaker 2: think about turning your real time data itself into a
Speaker 2: product offering your end users. Maybe customers the ability to
Speaker 2: query data with analytical capabilities almost live. They'd likely pay
Speaker 2: for that kind of access.
Speaker 1: Ah, interesting, So the insight itself becomes a premium service.
Speaker 1: That makes sense. It's not just about making more money, though,
Speaker 1: is it. The book talks infrastructure costs too.
Speaker 2: Yes, that's a really important one. Traditional BATGE systems often
Speaker 2: tie storage and compute together very tightly.
Speaker 1: Meaning if your data grows.
Speaker 2: Your costs for both storage and the processing power needed
Speaker 2: can just explode, often exponentially ouch. But with RTA, you're
Speaker 2: processing data more incrementally as it arrives. It sort of
Speaker 2: breaks that tight coupling. You don't necessarily need to store
Speaker 2: everything forever just to process it later in huge.
Speaker 1: Batches, so you avoid building those massive, expensive legacy systems
Speaker 1: just for batch jobs.
Speaker 2: Potentially, yes, significant cost savings are possible there. You're handling
Speaker 2: smaller streams continuously.
Speaker 1: Like managing a steady creek instead of building dams for
Speaker 1: unpredictable floods. And what about us, the customers? How does
Speaker 1: this improve customer experience?
Speaker 2: Well, think about customer support. Traditionally it's reactive, Right, you
Speaker 2: have a problem, you call their email, They investigate.
Speaker 1: And maybe fix it eventually.
Speaker 2: Maybe. With RTA, companies can constantly monitor streams of data
Speaker 2: usage patterns. ERA logus sensor data looking for anomalies or
Speaker 2: signs of trouble.
Speaker 1: Ah, so they can spot problems before I even notice them.
Speaker 2: That's the goal. They can potentially identify and even resolve
Speaker 2: issues proactively automatically, maybe reroute traffic, restart a service, or
Speaker 2: even reach out to you before it becomes a major headache.
Speaker 1: That sounds much better, moving from reactive firefighting to proactive.
Speaker 2: Care exactly, it leads to much higher customer satisfaction. It
Speaker 2: feels like the company is actually looking out for you.
Speaker 1: Okay, So RTA sounds powerful but also complex. The book
Speaker 1: introduces this term the real time analytics ecosystem or stack.
Speaker 1: What is that? In simple terms?
Speaker 2: Yeah, you'll hear ecosystem, stack, streaming stack. The basic mean
Speaker 2: the same thing. It's the whole collection of tools, technologies,
Speaker 2: and the processes you use to get from those raw,
Speaker 2: unending streams.
Speaker 1: Of data to actual insights you can use.
Speaker 2: Precisely, it's the entire pipeline, all the components working together.
Speaker 1: And why is understanding that whole picture important?
Speaker 2: Well, if you're an architect designing these systems, or developer
Speaker 2: building the apps, or even an operator keeping it all running,
Speaker 2: you need to understand how the pieces fit together.
Speaker 1: To make the right choices about tools and how they connect.
Speaker 2: Absolutely, it helps you build systems that are robust, scalable,
Speaker 2: and actually deliver those real time insights effectively.
Speaker 1: Okay. Now, before diving into the modern stack, the book
Speaker 1: briefly mentions something called the Lambda architecture. Sounds a bit
Speaker 1: I don't know dated.
Speaker 2: It is a bit older. Yeah.
Speaker 1: Yeah.
Speaker 2: It was kind of an early attempt to deal with
Speaker 2: having both real time needs and needing accurate historical analysis
Speaker 2: on huge data.
Speaker 1: Sets, trying to do both at once sort of.
Speaker 2: It had three layers, a big, slow batch layer for
Speaker 2: processing all the historical data accurately, a fast speed layer
Speaker 2: for handling the incoming real time streams providing quick, maybe
Speaker 2: slightly less perfect answers. And the third layer a serving
Speaker 2: layer that would try to merge the results from both
Speaker 2: the batch and speed layers when you actually queried the system.
Speaker 1: Okay, so it tried to give you fast answers and
Speaker 1: eventually correct complete answers. What was the upside?
Speaker 2: The main benefit was that your original raw data was
Speaker 2: kept safe and sound in the batch layer, so if
Speaker 2: you messed up your processing logic or wanted to try
Speaker 2: a new analysis.
Speaker 1: You could always go back and rerun it on the
Speaker 1: original data exactly. Data I mutability was a plus, But
Speaker 1: I sense a butt coming. The book implies it wasn't
Speaker 1: the perfect solution. What were the drawbacks?
Speaker 2: There were quite a few. Actually, First, it was complex.
Speaker 2: You essentially had to build and maintain two separate data pipelines,
Speaker 2: Batch and speed. That's a lot of engineering.
Speaker 1: Effort, double the work, potentially double the problem pretty much.
Speaker 2: Also, many early stream processors relied heavily on the JVM,
Speaker 2: the Java Virtual Mass, which was fine if you were
Speaker 2: a Java shop, but maybe less ideal otherwise.
Speaker 1: Fender lock in or skill set mismatch.
Speaker 2: Yeah, and maybe the biggest headache was often having to
Speaker 2: write and maintain the same or very similar processing logic
Speaker 2: in both the batch and the speed layers.
Speaker 1: Duplication. That sounds like a nightmare for consistency and updates,
Speaker 1: it really was.
Speaker 2: Keeping them perfectly in sync was hard, leading to potential
Speaker 2: inconsistencies in the final results. So yeah, lots of overhead
Speaker 2: and complexity.
Speaker 1: Okay, so LAMB deserved a purpose, But we've move on.
Speaker 1: What does a more modern real time analytics stack look like?
Speaker 1: What are the essential pieces?
Speaker 2: Right? The contemporary approach is generally more streamlined. It typically
Speaker 2: starts with event producers.
Speaker 1: The things generating the data in the first place.
Speaker 2: Exactly, systems that detect something happen to state change and
Speaker 2: fire off an event. Like an order management system sees
Speaker 2: a new order and generates an order received event, and that.
Speaker 1: Event contains the details like order ID, customer info items.
Speaker 2: All the relevant data and A key thing here mentioned
Speaker 2: in the book is you really need to benchmark your
Speaker 2: producers make sure they can actually handle the volume and
Speaker 2: speed of events you expect without becoming a bottleneck. Scalability
Speaker 2: and latency are critical right from the start.
Speaker 1: Okay, makes sense, The source needs to keep up. Where
Speaker 1: do those events go next?
Speaker 2: They flow into the event streaming platform. This is like
Speaker 2: the central highway or message bus for all your events.
Speaker 2: The backbone, yeah, exactly. Its job is to ingest potentially
Speaker 2: huge volumes of events, store them reliably, usually for some
Speaker 2: configurable period, and deliver them to whatever needs to consume them.
Speaker 2: A patch Kafka is probably the most well known example here.
Speaker 1: Right, Kofka comes up a lot. What makes a good
Speaker 1: streaming platform?
Speaker 2: Key things are scalability, Can it handle growth? Fault tolerance?
Speaker 2: Does it lose data if a server fails? High throughput?
Speaker 2: Can it handle a massive continuous flow and low latency?
Speaker 2: How quickly does data get through? Got it?
Speaker 1: So? Data producers feed events onto this Sofka like highway?
Speaker 2: Then what then? You typically have a stream processing platform
Speaker 2: This is where the real time analysis starts happening.
Speaker 1: This is where the magic happens, well.
Speaker 2: Some of it. This is where you take those raw
Speaker 2: event streams and transform them, maybe enrich them by joining
Speaker 2: them with other data streams or static data filter them,
Speaker 2: aggregate them, run calculations. Basically turn raw data into intermediate insights.
Speaker 1: Can you give examples of tools here?
Speaker 2: Sure. Popular ones include a Patche flink, which is a
Speaker 2: powerful stream processing framework. There's also a Patche Spark streaming,
Speaker 2: which extends the Spark batch engine for streaming, and Kaffka streams,
Speaker 2: which is a library that lets you build stream processing
Speaker 2: apps directly on top of Kafka.
Speaker 1: What are the important features for these stream processors?
Speaker 2: You need things like good state management because your analysis
Speaker 2: often depends on past events when doing capabilities for doing
Speaker 2: calculations over specific time periods like the last five minutes.
Speaker 2: Fault tolerance obviously so processing doesn't stop if something breaks,
Speaker 2: and support for different data formats okay.
Speaker 1: Using happens insights are generated, how do we actually use
Speaker 1: them or see them?
Speaker 2: That's the final piece, usually the serving layer. This is
Speaker 2: the system that stores the results of your real time
Speaker 2: processing and makes them available for querying fast.
Speaker 1: This is what applications or dashboards actually talk to exactly.
Speaker 2: It's the primary access point. Now, this serving layer could
Speaker 2: be a few different types of systems like what It
Speaker 2: could be a fast key value store think Mango dB,
Speaker 2: maybe elastic search or rettis. These are great if you
Speaker 2: primarily need to look up results based on a specific key,
Speaker 2: like getting the current status for a particular user.
Speaker 1: ID quick lookups. What's the alternative?
Speaker 2: The alternative, especially for more complex analytics, is a real
Speaker 2: time ol APP database. Ol app stands for online analytical processing.
Speaker 1: Ah okay designed for analysis right.
Speaker 2: Tools like a Pacupine, Apache, Druid, rock Set, or ClickHouse
Speaker 2: fall into this category. They are built for slicing and
Speaker 2: dicing data, running aggregations, filtering across lots of dimensions, much
Speaker 2: more complex queries than just a key lookup.
Speaker 1: So if I want to see, say, sales trends by
Speaker 1: region and product category for the last hour, I'd want
Speaker 1: an ol APP database.
Speaker 2: Generally, Yes, that's where they shine. The crucial thing for
Speaker 2: any serving layer in this context is speed. You need
Speaker 2: really fast data ingestion. The results from the stream processor
Speaker 2: need to show up almost instantly, and query latency needs
Speaker 2: to be low often in the.
Speaker 1: Millisecond well, well seconds again.
Speaker 2: Wow yeah, and it also needs to handle high concurrency,
Speaker 2: potentially thousands or even hundreds of thousands of queries per
Speaker 2: second depending on the application.
Speaker 1: That's incredible scale. How do you choose between key value
Speaker 1: and real time ol app beyond just the query type?
Speaker 2: Well, query type is the main driver, but you also
Speaker 2: look at how data gets in. Does it support direct
Speaker 2: streaming ingestion from COFKA or flink or do you need
Speaker 2: an extra step? How fast is that ingestion? Really? Can
Speaker 2: it handle your expected data volume and rate? Does it
Speaker 2: need complex indexing or pre aggregation to meet your query
Speaker 2: speed goals? Lots to consider, definitely, and the book makes
Speaker 2: a very sensible point. Don't just trust the marketing hype.
Speaker 2: Do your own benchmarking with your own data and query patterns.
Speaker 2: See what actually works best for your specific needs.
Speaker 1: Test it yourself. Always good advice. Okay, so we have producers,
Speaker 1: the streaming platform, the process, or the serving layer. How
Speaker 1: do people like actual users see this stuff?
Speaker 2: Ah, the front end? Good point. If your users are
Speaker 2: internal like data analysts or engineers, maybe they querry the
Speaker 2: serving layer directly using SQL or an.
Speaker 1: API okay, But for less technical users or external customer.
Speaker 2: Then you'll likely need a user interface a front end,
Speaker 2: and you've got a few options here.
Speaker 1: What are they?
Speaker 2: You could go fully custom build your own web application
Speaker 2: using standard tools like react as Angular viewjas gives you
Speaker 2: total control over the look, feel, and functionality.
Speaker 1: The Highffert high control option. What else?
Speaker 2: Then there are low code frameworks, things like Streamlet or plotly,
Speaker 2: dash or popular especially in the Python world. They let
Speaker 2: you build interactive dashboards and web apps with much less
Speaker 2: front end coding effort.
Speaker 1: Faster development, maybe, less customization generally yes.
Speaker 2: And the third category is data visualization tools they ca Apache, Superset, redash, Grfauna.
Speaker 2: These often provide drag and drop interfaces to build dashboards
Speaker 2: directly on top of your data sources, often with no
Speaker 2: coding required at all.
Speaker 1: The quickest way to get a dashboard up.
Speaker 2: Often yes, So how you choose depends on a few things.
Speaker 2: What's the front end coding skill level of your team,
Speaker 2: how much time do you realistically have, and who are
Speaker 2: the user's internal experts or external customers needing a polished experience.
Speaker 1: A spectrum of choices matching needs and resources makes sense
Speaker 1: now the book also notes that sometimes the lines between
Speaker 1: these components get fuzzy.
Speaker 2: Yeah, technology evolves and tools sometimes wear multiple hats. Apache
Speaker 2: Pulsar is a good examples, mainly an event streaming platform
Speaker 2: like Kafka, but it also has built in capabilities called
Speaker 2: Pulsar functions that let you do some lightweight stream processing
Speaker 2: directly within Pulsar itself.
Speaker 1: Ah, so the streaming platform is doing some processing tasks exactly.
Speaker 2: It blurs the line a bit between the streaming platform
Speaker 2: and the stream processing platform. It just shows that these
Speaker 2: categories aren't always rigid silos. The landscape is pretty dynamic.
Speaker 1: Okay, that's a great tour through the stack. So wrapping
Speaker 1: up this main section, what's the big takeaway from the
Speaker 1: book about building these systems?
Speaker 2: I think the fundamental message is that embracing real time
Speaker 2: analytics isn't just a technical upgrade. It's a strategic move
Speaker 2: that can give you a serious competitive.
Speaker 1: Edge by making you faster, more informed.
Speaker 2: And ultimately making more accurate, relevant decisions because you're acting
Speaker 2: on what's happening now, not what happened yesterday.
Speaker 1: Fantastic, and this deep dive has been really an introduction.
Speaker 1: We've touched on the core ideas, the benefits those key
Speaker 1: building blocks of the RTA stack all pulled from the
Speaker 1: insights in building real time analytics systems.
Speaker 2: Absolutely, it just scratches the surface, but hopefully gives you
Speaker 2: a solid foundation.
Speaker 1: So a final thought for you, the listener to chew on,
Speaker 1: think about your own work, your own organization. Where could
Speaker 1: real time analytics unlock something new, maybe a new data
Speaker 1: product or a way to significantly improve a process you
Speaker 1: already have.
Speaker 2: Yeah, ask yourself, what's the current shelf life of your data?
Speaker 2: Is its value decaying rapidly? What can you gain by
Speaker 2: acting on it immediately? And maybe think about that stack.
Speaker 2: We discussed producers, streaming, processing, serving front end. Which piece
Speaker 2: might offer the biggest immediate win for your specific situation?
Speaker 1: Something to definitely consider. Where could that immediate incite make
Speaker 1: the biggest difference