Speaker 1: Welcome to the deep dive. Today. We're diving into well,
Speaker 1: a really fundamental challenge in computing these days, handling that
Speaker 1: absolute flood of data, you know, the high speed stuff.
Speaker 2: Yeah, it's relentless. I think social media, IoT sensors, everywhere,
Speaker 2: web clicks, it just keeps coming.
Speaker 1: And it's not just the sheer volume, right, it's how
Speaker 1: bursty it is. You get these sudden peaks exactly.
Speaker 2: That borestiness is the real killer. Our sources. They show
Speaker 2: that building systems that don't just fall over when you
Speaker 2: get a sudden rush, like say everyone trying to check
Speaker 2: out a bike in a smart city right at five pm,
Speaker 2: that needs special thinking. Traditional approaches just won't.
Speaker 1: Cut it right. So our mission today is basically to
Speaker 1: give you a solid shortcut to understanding these kinds of architectures.
Speaker 1: Will use the Amazon Kinesis family, you know, data streams,
Speaker 1: fire hose, data analytics, video streams, kind of as our
Speaker 1: case study.
Speaker 2: Yeah. The goal is by the end you'll really get
Speaker 2: the difference between these tools, and more importantly, the core
Speaker 2: idea is behind making these systems scalable and crucially fast.
Speaker 1: Okay, let's kick off with the big why why is
Speaker 1: speed so important in data analysis. Now why the big rush?
Speaker 2: Well, it's become a strategic must have. Really, we've moved
Speaker 2: way beyond just looking backwards. Yeah, like those old batch
Speaker 2: jobs running overnight giving you a report.
Speaker 1: On yesterday, right, a snapshot of the past exactly.
Speaker 2: Now, it's all about immediate insights, the freshest possible data.
Speaker 2: Let's you make the best decisions right now, perishable insights.
Speaker 1: Basically, I've heard about this thing, the O day loop observe, orient, decide, act.
Speaker 1: It comes from military strategy. I think, how does real
Speaker 1: time data fit in there?
Speaker 2: It fits perfectly. Real time analytics fundamentally strengths that observe window.
Speaker 2: Something happens, you know about.
Speaker 1: It instantly, which means you can orient, decide, and act
Speaker 1: much faster.
Speaker 2: Precisely gives you a huge edge. Think about frog detection
Speaker 2: or optimizing ad placements on the fly, or in that
Speaker 2: smart city example, rerouting maintenance crews instantly based on real
Speaker 2: time bike usage patterns.
Speaker 1: Okay, so that's the why. Now the how? A single
Speaker 1: server obviously can't cope, so we use distributed systems. Lots
Speaker 1: of servers network together. But does not just explode the
Speaker 1: complexity failure modes everywhere.
Speaker 2: Oh absolutely, It gets complex fast, but engineers figured it
Speaker 2: out ways to manage it. Two main patterns really help
Speaker 2: tame that initial complexity.
Speaker 1: Beast Okay, what were they.
Speaker 2: First standardized interfaces? Things like APIs This really paved the
Speaker 2: way for micro service architectures.
Speaker 1: Ah, so breaking the big system down into smaller independent
Speaker 1: services exactly.
Speaker 2: It lets different teams own their piece and you know,
Speaker 2: iterate faster without stepping on each other's toes.
Speaker 1: It sounds a bit like Conway's law in action, the
Speaker 1: system design mirroring the ORG structure.
Speaker 2: That's a great way to put it. Yeah, small independent
Speaker 2: teams tend to build small independent services. Those standardized interfaces
Speaker 2: are what make it work smoothly.
Speaker 1: Okay, interfaces help services talk. But what about failures? If
Speaker 1: one micro service crashes while it's processing data, how do
Speaker 1: you stop it from dragging down everything else that depends
Speaker 1: on it?
Speaker 2: Right, that's the castgating failure problem. And that's where the
Speaker 2: second pattern comes in. Decoupling. You use an asynchronous message broker.
Speaker 1: A message broker like a middleman for data sort of.
Speaker 2: Yeah. It acts as the stable buffer, an invariant in
Speaker 2: the system. If a downstream service, a consumer fails, the
Speaker 2: broker just holds onto the incoming messages. It builds up
Speaker 2: a backlog.
Speaker 1: So the producer can keep sending data unaware of the
Speaker 1: downstream problem.
Speaker 2: Exactly, the failure is contained. Once the consumer service comes
Speaker 2: back online, it just starts processing the backlog from the broker.
Speaker 2: No data lost, no castkating crash.
Speaker 1: That makes sense. The broker isolates the fault. But if
Speaker 1: that backlog keeps growing, that sounds like trouble brewing. What
Speaker 1: metrics do we watch?
Speaker 2: The key capacity metric is transactions per second tps. That's
Speaker 2: usually limited by either the number of records per second
Speaker 2: or the total data size like megabytes per second.
Speaker 1: Okay, tps, but what's the danger signal the real data.
Speaker 2: Your signal is back pressure. That's when your producers are
Speaker 2: sending data faster than your consumers can process it. Input
Speaker 2: tps is consistently higher than output tps.
Speaker 1: So back to the bike sharing rush hour hits every
Speaker 1: bike starts pinging its location like crazy. How do you
Speaker 1: handle that surge, that back pressure without the system just drowning.
Speaker 2: You've got a few strategies ranging from let's say gentle
Speaker 2: to pretty aggressive. Okay, you can throttle the producers tell
Speaker 2: the bikes, hey, maybe only report your location every minute
Speaker 2: instead of every second during peak times.
Speaker 1: Reduce the input rate makes.
Speaker 2: Sense, Or you can scale the consumers. If you're in
Speaker 2: the cloud, you might automatically spin up more instances of
Speaker 2: your processing application to handle the load.
Speaker 1: Add more workers, right.
Speaker 2: You can also use bigger buffers in the message broker
Speaker 2: itself to just absorb temporary spikes. But the most drastic
Speaker 2: option is to just drop messages drop data.
Speaker 1: That sounds risky.
Speaker 2: It is. You'd only ever do that for non critical data.
Speaker 2: Like maybe dropping some routine sensor readings is okay if
Speaker 2: the systems overloaded, but you'd never drop something like a
Speaker 2: customer order completion message. Never.
Speaker 1: Okay, got it? Critical versus non critical. Let's dig into
Speaker 1: the stream itself. What are the absolute basic parts of
Speaker 1: any streaming system for main components?
Speaker 2: Yeah, you can break it down pretty simply. First you
Speaker 2: have the producers. These are the apps sending the data,
Speaker 2: our bike sensors, the user's mobile app checking out a bike,
Speaker 2: that kind of thing.
Speaker 1: Okay, producers send data.
Speaker 2: Then you have the messages or records. That's the actual
Speaker 2: data payload, usually small, maybe I a few kill bytes
Speaker 2: often capped around a megabyte. Each record has the data
Speaker 2: itself and a header, usually with a unique ID the
Speaker 2: broker assigns.
Speaker 1: Got it, producers messages.
Speaker 2: Third is the stream or the broker. That's the component
Speaker 2: doing the buffering, like canisis itself.
Speaker 3: Or Kafka the fault isolator we talked about, right, And
Speaker 3: finally you have the consumers, the applications that pull data
Speaker 3: from the stream and actually do something with it, analysis, storage, triggering, alerts.
Speaker 1: Whatever, producers, messages, stream consumers. Okay, now latency you mentioned
Speaker 1: speed is critical the source of stress. We need to
Speaker 1: be really precise here. It's not just latency. They're two
Speaker 1: specific measures.
Speaker 2: Yes, absolutely critical distinction. There's propagation delay that's simply the
Speaker 2: time it takes from the moment of producer writes a
Speaker 2: message to the moment a consumer reads it raw transmission speed.
Speaker 1: Basically okay, right to read time. What's the other one?
Speaker 2: The other and often more telling metric is the age
Speaker 2: of the message. This measures how long a message has
Speaker 2: actually been sitting in the stream before a consumer picked
Speaker 2: it up.
Speaker 1: Ah, So propagation delay could be low, but the message
Speaker 1: might still be old. If the consumers are lagging exactly.
Speaker 2: If that average message age starts creeping up, that's your
Speaker 2: big warning sign. It means your consumers can't keep up.
Speaker 2: The backlog is growing, and performance problems are right around
Speaker 2: the corner, even if the network itself is fast.
Speaker 1: That's a really important distinction. Now, failure modes distributed systems
Speaker 1: retries network glitches. It means data streams all and have
Speaker 1: this at least once delivery guarantee right, a message might
Speaker 1: show up more than once.
Speaker 2: That's the standard reality. Yes, because ensuring exactly once delivery
Speaker 2: across a distributed system is incredibly hard. Most brokers guarantee
Speaker 2: at least once. The producer might retry sending if it
Speaker 2: doesn't get confirmation the consumer by process and in crash
Speaker 2: before confirming. Duplicates happen, and that.
Speaker 1: Sounds potentially disastrous. If you're, say, processing payments or updating
Speaker 1: inventory accounts, double counting.
Speaker 2: It would be disastrous, which is why the responsibility for
Speaker 2: handling duplicates shifts downstream to the consumer or the final
Speaker 2: destination system. Those systems must be designed to be idempatant idempatent,
Speaker 2: meaning meaning processing the same message multiple times has the
Speaker 2: exact same effect as processing it just once.
Speaker 1: How do you achieve that two main ways.
Speaker 2: Either the operation itself is naturally idempatant like setting a
Speaker 2: value as adempatent adding to a count, or is not,
Speaker 2: or you use that unique ideed. It's typically in the
Speaker 2: message payload to explicitly check if you've already processed this
Speaker 2: specific message before de duplication.
Speaker 1: Basically okay, So the consuming application needs to be smart
Speaker 1: about duplicates. What about errors within a message? If a
Speaker 1: consumer gets a record it just can't process.
Speaker 2: That leads to a particularly nasty problem called the poison pill.
Speaker 1: Poisman pill sounds bad, It is.
Speaker 2: Because message streams usually guarantee the order of messages, at
Speaker 2: least within a certain context. Like all messages for a
Speaker 2: specific bike, you need to process the unlock event before
Speaker 2: the return event takes sense.
Speaker 1: Order matters.
Speaker 2: But if one message in that sequence causes an error
Speaker 2: in the consumer, maybe it's malformed bad data, and the
Speaker 2: consumer keeps retrying and failing on that specific message, it
Speaker 2: gets stuck. It gets completely stuck. That single poison pill
Speaker 2: record blocks the processing of all the perfectly valid records
Speaker 2: sitting behind it in the stream for that same bike
Speaker 2: or whatever the ordering context is. The whole sequence grinds
Speaker 2: to a halt because of one bad message.
Speaker 1: Wow, so a single bite of bad data could effectively
Speaker 1: block a whole partition of the stream, causing a latency
Speaker 1: for everything behind it to skyrocket exactly.
Speaker 2: It really highlights why robust error handling and potentially dead
Speaker 2: letter cues for problematic messages are so crucial in consumer design.
Speaker 1: Okay, that sets the stage really well. Let's move on
Speaker 1: to the specific tools Amazon offers with Kinesis to handle
Speaker 1: all this at scale. Starting with the foundation canesis data
Speaker 1: streams or KDS. When do you reach for this?
Speaker 2: KTS is your go to when you need maximum control,
Speaker 2: high durability, and really low latency. We're talking subsecond processing
Speaker 2: for your own custom applications that read directly from the stream.
Speaker 1: And the core unit of KDS is the shard, right,
Speaker 1: what does that actually do?
Speaker 2: The shard is the fundamental unit of capacity and parallelism
Speaker 2: in KDS. Each chard has fixed limits on how much
Speaker 2: data it can ingest typically one megabyte per second or
Speaker 2: one thousand records per second, whichever comes first, and also
Speaker 2: how much data can be read out.
Speaker 1: If I need more capacity, I just add more shards
Speaker 1: pretty much.
Speaker 2: Yeah, you scale the stream by scaling the number of shards.
Speaker 1: Okay, and if we're tracking millions of bike journeys, how
Speaker 1: does KDS decide which shard a specific bike's data goes
Speaker 1: to and keep it in order.
Speaker 2: That's determined by the partition key. When your producer application
Speaker 2: sends a record to KDS, it includes a partition key.
Speaker 2: This could be the bike ID for instance.
Speaker 1: And KTS uses that key to route.
Speaker 2: The data exactly. KTS hashes the partition key and uses
Speaker 2: the result to assign the record to a specific shard.
Speaker 2: All records with the same partition key always go to
Speaker 2: the same.
Speaker 1: Chard ah, and that's how it guarantees order. Within that key,
Speaker 1: all data for bike one twenty three lands on, say,
Speaker 1: shard five, in the correct sequence precisely.
Speaker 2: But this also highlights a potential pitfall, the hot partition
Speaker 2: key or hot shard. If one partition ke, like a
Speaker 2: super popular bike or a single central sensor, sends way
Speaker 2: more data than others, its shark can get overwhelmed while
Speaker 2: others are idle.
Speaker 1: So choosing a good part two key that distributes data
Speaker 1: evenly is critical for performance, maybe not just the bike idea.
Speaker 1: If usage is very uneven.
Speaker 2: Right, sometimes using a more random key or combining fields
Speaker 2: is necessary to spread the load effectively.
Speaker 1: Now reading the data, Let's say we have multiple apps
Speaker 1: wanting that bike data, one for maintenance alarts, one for
Speaker 1: route analysis. How do they read efficiently? You mentioned Enhanced
Speaker 1: fan out EFO.
Speaker 2: Yes, so, standard KDS consumers work on a pull model.
Speaker 2: They pull the shard for data, but each chard has
Speaker 2: a total read capacity limit about two megabytes per second
Speaker 2: that all standard consumers share.
Speaker 1: So if you have lots of consumers pulling the same shard,
Speaker 1: they start slowing each other down exactly.
Speaker 2: They compete for that shared bandwidth. Enhanced fan out EFO
Speaker 2: solves this with EFO. Each registered consumer gets its own
Speaker 2: dedicated throughput limit per shard, delivered via push model from Kinesis.
Speaker 1: So EFO consumers don't interfere with each other.
Speaker 2: Correct. It allows multiple real time applications to consume from
Speaker 2: the same stream at high speed independently. If you need
Speaker 2: scale on the consumer side, EFO is often essential.
Speaker 1: Okay, KTS for flexible, low latency custom apps. What about
Speaker 1: Kinesis Data fire Hose KDF. People often call it the
Speaker 1: easy loader.
Speaker 2: Why because it is KDF is fully managed, totally serverless
Speaker 2: and Its whole purpose is to make it super simple
Speaker 2: to capture streaming data and load it into specific destinations.
Speaker 2: Think S three data laks, redshift data warehouses, elastic search
Speaker 2: for search, that kind of thing.
Speaker 1: How easy is easy?
Speaker 2: Often zero code is needed for the delivery part. You
Speaker 2: can figure fire Hose in the console, point it at
Speaker 2: your stream source which could even be KDS, tell it
Speaker 2: where to send the data, and it handles the scaling, buffering, delivery,
Speaker 2: and retries automatically.
Speaker 1: Does it do anything else like transform the data?
Speaker 2: Yes, you can. It has built in capabilities for data
Speaker 2: format conversion like Jayson to park, and it can also
Speaker 2: invoke an AWS lambda function to perform custom inline transformations
Speaker 2: basic epl before the data lands and the destination.
Speaker 1: Okay, KTS gives us speed and flexibility. KDF gives us
Speaker 1: ease of use for loading data, sinks. What's the catch
Speaker 1: with KDF? What's the tradeoff?
Speaker 2: The main trade off is latency because fire Hose buffers
Speaker 2: data internally, often for several seconds or even minutes to
Speaker 2: batch it efficiently for delivery to the destination.
Speaker 1: Ah, It's not truly real time like KDS.
Speaker 2: Can be exactly. It optimizes for delivery throughput and cost
Speaker 2: effectiveness by batching, not for subsecond end to end latency,
Speaker 2: So if you need that immediate subsecond processing, KDS is
Speaker 2: the choice. If your goal is reliable, easy loading into
Speaker 2: F three or redshift and near real time is good enough,
Speaker 2: KDF is often way simpler.
Speaker 1: Makes sense speed versus simplicity. Now, analysis kinesis data analytics kDa.
Speaker 1: How does this fit in analyzing the stream as it flows?
Speaker 2: Precisely? kDa is also serverless, and it lets you run
Speaker 2: continuous queries or applications against your streaming data, either from
Speaker 2: KDS or KDF. It offers two different.
Speaker 1: Engines for this, Okay, what are the engine choices?
Speaker 2: First is a sqel engine. If you're familiar with standard SQL,
Speaker 2: you can use ans icql to query the stream. kDa
Speaker 2: presents the stream data within windows, like tumbling windows of the
Speaker 2: last five minutes or sliding windows, so.
Speaker 1: You could write a query like select count from bike
Speaker 1: stream where status in US, group by location district over
Speaker 1: a rolling time.
Speaker 2: Window exactly that kind of thing. It treats the incoming
Speaker 2: stream like a continuously updating table. You can query. Very
Speaker 2: powerful for many common analytics tasks and accessible if you
Speaker 2: know SQL.
Speaker 1: And the second engine you mentioned too.
Speaker 2: The second is based on a patche flink. This gives
Speaker 2: you much more power and flexibility. You typically write your
Speaker 2: applications in Javar Scala.
Speaker 1: What can flink do that the SQL engine can't.
Speaker 2: Well, flink apps can handle much more complex logic, maintain
Speaker 2: sophisticated state across events, and importantly can connect the data
Speaker 2: sources and sinks outside of just kinesis and aws like
Speaker 2: maybe an on premises KOFKA cluster or a custom database.
Speaker 1: And I remember reading flink is key for achieving exactly
Speaker 1: once processing semantics. How does it manage that?
Speaker 2: Yes, that's a major advantage. Flank achieves exactly once by
Speaker 2: carefully managing the application's internal state and periodically saving checkpoints
Speaker 2: of that state to durable storage typically S three or
Speaker 2: something like rock sysdb. If there's a failure, it can
Speaker 2: restore from the last successful checkpoint and resume processing without
Speaker 2: missing data or processing duplicates.
Speaker 1: So SQL for simpler windowed queries, think for complex stateful
Speaker 1: potentially exactly one's processing and connecting anywhere.
Speaker 2: That's a good summary.
Speaker 1: Yeah, okay, one more service kinesis video streams KBS. This
Speaker 1: seems more specialized. Video and audio.
Speaker 2: Yeah, KBS is specifically designed for handling time encoded data streams,
Speaker 2: primarily video and audio, but potentially other time series data too,
Speaker 2: like radar or liar feeds.
Speaker 1: How does play out in our smart city bike scenario?
Speaker 2: KBS actually addresses two pretty distinct use cases. There first
Speaker 2: is real time low latency interaction. Imagine a user wanting
Speaker 2: to see a live video feed of a bike stand
Speaker 2: on their phone to check if bikes.
Speaker 1: Are actually available okay, live streaming.
Speaker 2: KVS provides capabilities, often using Web RTC standards for that
Speaker 2: kind of low latency peer to peer or small group
Speaker 2: video streaming. It manages the signaling channels and things like
Speaker 2: stunt and turn servers needed to establish the connections. So
Speaker 2: KVS WebRTC.
Speaker 1: For live use got it? And the second use case.
Speaker 2: The second is more about storage, playback and analysis. This
Speaker 2: is KVS storage. You stream video from say security cameras
Speaker 2: that the bike stands into KVS for durable storage.
Speaker 1: And then what just store it?
Speaker 2: You can store it for compliance or later review, but
Speaker 2: the real power comes from integrating it with AI and
Speaker 2: mL services. For instance, you could feed that stored video
Speaker 2: stream into Amazon Recognition.
Speaker 1: AH for analysis.
Speaker 2: What like running facial recognition to detect known vandals near
Speaker 2: the bike stands in real time, or maybe object detection
Speaker 2: to count bikes automatically or spot obstructions. Recognition analyzes the
Speaker 2: frames ingested via KVS and can trigger alerts or other actions.
Speaker 1: So KVS handles both the live interaction piece and the
Speaker 1: jest for later analysis piece for video.
Speaker 2: Data correct two sides of the video coin.
Speaker 1: That's a really helpful tour of the whole Kinesis family.
Speaker 1: Let's try to quickly recap the core job of each
Speaker 1: one for the listener.
Speaker 2: Okay, KTS, that's your foundational low latency, high throughput stream
Speaker 2: for custom apps needing maximum flexibility, thinks subsecond speed right
Speaker 2: KTF KDF the easy serverlust loader, great for getting streaming
Speaker 2: data into S three redshift elastic search without much code,
Speaker 2: but sacrifices that subsecond latency for batching efficiency. kDa the
Speaker 2: real time Analytics engine, use it SQL interface for simpler
Speaker 2: windowed queries, or the Apache frink engine for complex stateful
Speaker 2: processing exactly one semantics and broader connectivity.
Speaker 1: And finally, KVS KDS.
Speaker 2: The specialist for video and time encoded data, handles both
Speaker 2: low latency web RTC streaming for live interaction and gerbil
Speaker 2: ingestion for storage playback, and AML analysis like recognition.
Speaker 1: Perfect summary. Now, before we wrap up, there's a crucial
Speaker 1: security point that underpins all of this distributed complexity. How
Speaker 1: to secure all these producers and consumers talking to kinesis, Yeah, this.
Speaker 2: Is super important. The absolute standard best practice is use
Speaker 2: IMA roles. Do not embed long term access keys or
Speaker 2: secrets directly into your producer or consumer applications. Why roles specifically,
Speaker 2: because IM roles provide temporary, short lived security credentials that
Speaker 2: are automatically generated and rotated by AWS. Your application running
Speaker 2: on EC two or Lambda or wherever assumes a role
Speaker 2: gets temporary permissions, does its work, and those credentials expire.
Speaker 1: So even if an application instance gets compromised somehow, the
Speaker 1: attacker doesn't get permanent keys.
Speaker 2: Exactly, the potential blast radius is dramatically reduced compared to static,
Speaker 2: long lived keys being compromised. It's all about implementing least
Speaker 2: privileged access using temporary role based credentials. It's fundamental to
Speaker 2: secure cloud architecture.
Speaker 1: That's a critical takeaway. Okay, let's finish with that provocative
Speaker 1: thought we talked about at least once delivery and the
Speaker 1: need for dempotency. How should listeners think about that?
Speaker 2: Right, we established that GETT duplicates is just a fact
Speaker 2: of life in most high performance, resilient streaming systems like
Speaker 2: those built on Kinesis data streams. You have to design
Speaker 2: for it.
Speaker 1: So the provocative thought.
Speaker 2: Is think about how profoundly that changes how you have
Speaker 2: to design your applications. Yeah, you can never assume an
Speaker 2: input an event, A message will arrive only once, might
Speaker 2: arrive twice or even more time. So that's rare. So
Speaker 2: how does that constraint The absolute certainty that you might
Speaker 2: get duplicates ripple through your entire design? How does it
Speaker 2: affect your database schemas, your transaction logic, your state management.
Speaker 2: Realizing that you must build identity consumers isn't just a detail,
Speaker 2: it's a fundamental shift in mindset needed to truly master
Speaker 2: streaming architectures.
Speaker 1: A great point to ponder designing for repeats, not just requests.
Speaker 1: Thanks for breaking all that down.
Speaker 2: My pleasure. It's complex, but hopefully a bit clearer now.