Speaker 1: Welcome to the deep dive, your shortcut to truly understanding
Speaker 1: complex topics. Today, we're plunging into Apache Koffka. It's really
Speaker 1: a foundational technology underpinning so much of the modern data world.
Speaker 2: Absolutely, it's everywhere, even if you don't see it directly.
Speaker 1: We've pulled together a stack of detailed sources for you,
Speaker 1: particularly some really great insights from a patchi Kofka in action,
Speaker 1: and our mission really is to unpack its full potential.
Speaker 2: Yeah, get beyond just the buzzwords exactly.
Speaker 1: We'll explore everything from its basic building blocks to how
Speaker 1: it ensures rock solid reliability, which is critical goose performance
Speaker 1: to incredible levels, and fits into the most advanced enterprise systems.
Speaker 1: Think of it this way. If you've ever wondered how
Speaker 1: a massive online retailer processes millions of real time orders,
Speaker 1: updates inventory instantly, or personalizes your shopping experience on the fly, Kuffa.
Speaker 2: Is probably in the mix somewhere. It's off in that
Speaker 2: silent powerhouse.
Speaker 1: Get ready for some serious aha moments, because this deep
Speaker 1: dive is your essential guide. We want you to not
Speaker 1: just know what Kolica is, but deeply understand how it
Speaker 1: works and why it's so critical for today's real time
Speaker 1: data needs.
Speaker 2: And hopefully, without feeling overwhelmed by the jargon.
Speaker 1: Let's untack this.
Speaker 2: It's truly a system that transforms how organizations handle data.
Speaker 2: It enables that shift, you know, from waiting.
Speaker 3: For daily reports still a batch world.
Speaker 2: Right to getting instant insights and acting on them immediately.
Speaker 1: That real time capability is absolutely where the magic happens. Okay,
Speaker 1: So for anyone looking to understand Kafka, where do we
Speaker 1: even begin? What are its absolute foundational elements?
Speaker 2: Okay? So at its core, cof Go works with messages
Speaker 2: sometimes called records. Okay, These are essentially just by rays.
Speaker 2: Think of them like small data envelopes, and for efficiency,
Speaker 2: they're often grouped into batches before being sent saves overhead.
Speaker 1: Got it? Batches of messages, and these messages are organized
Speaker 1: into topics like categories exactly.
Speaker 2: Think of a topic as a dedicated channel or category
Speaker 2: for bundling messages of a specific business type, much like
Speaker 2: tables in a database maybe, but really designed for a
Speaker 2: continuous stream of events. So, for that online retailer we mentioned,
Speaker 2: you might have a customer orders topic or maybe a
Speaker 2: product inventory updates topic.
Speaker 1: Right, So if I place an order that becomes a
Speaker 1: message in the customer orders topic. Simple enough, But how
Speaker 1: does KOFKA handle the sheer volume millions, maybe billions of
Speaker 1: messages and ensure it can scale.
Speaker 2: Ah? That's where partitions come in, and they are truly
Speaker 2: like the backbone of kofka's performance and scalability.
Speaker 1: Partitions Okay, hash.
Speaker 2: Topic is divided into one or more partitions. This division
Speaker 2: is what enables parallel processing lots of things happening at once.
Speaker 1: Makes sense, divide and conquer and to.
Speaker 2: Ensure high availability and fault tolerance, which is crucial. These
Speaker 2: partitions are replicated across different COFCA servers. The servers which
Speaker 2: you call we call them brokers. So if one broker
Speaker 2: goes down to debta, is safe and accessible in another one?
Speaker 2: No panic?
Speaker 1: Got it? Mess topics partitions on brokers. Okay, So you
Speaker 1: have producers sending messages and consumers receiving them. How did
Speaker 1: they interact with these partitions and brokers?
Speaker 2: Good question. Producers are the application sending the messages your
Speaker 2: order service. Maybe they send them to the designated leader
Speaker 2: of a partition.
Speaker 1: Leader one broker is in charge for that partition, right, and.
Speaker 2: The producer selects that partition using something called a partitioner.
Speaker 2: Often it's based on a message key, which we'll get to.
Speaker 1: Okay.
Speaker 2: On the other side, consumers receive and process messages. They're
Speaker 2: quite flexible actually, they can read from multiple partitions, even
Speaker 2: multiple topics at once.
Speaker 1: And the brokers themselves, what's their main job.
Speaker 2: The brokers are the Kafka servers. They manage the storage, distribution,
Speaker 2: retrieval of messages, all that, and they share replicas and
Speaker 2: processing tasks pretty evenly among themselves. It's a distributed system.
Speaker 1: This sounds like a lot of moving parts all needing
Speaker 1: to coordinate. Who is the leader? Is this broker alive?
Speaker 1: How does Kafka manage that internal choreography?
Speaker 2: Right? That's the role coordination pluster. Historically this was a
Speaker 2: page zookeeper, a whole separate system you had to manage.
Speaker 1: I remember a zoo keeper could be complex.
Speaker 2: It could. But now Koka is increasingly moving to craft
Speaker 2: kr aft ok and this isn't just a name change.
Speaker 2: It's a pretty significant evolution. Craft simplifies the entire Kofka
Speaker 2: architecture because it removes that external dependency on zookeeper, so
Speaker 2: Kofka manages itself more exactly, it becomes self managing for
Speaker 2: these critical coordination tasks. Overseeing partition assignments, handling leader elections,
Speaker 2: continuously monitoring broker health means fewer moving parts for you
Speaker 2: to manage, which is a huge operational win, especially for
Speaker 2: large dynamic clusters.
Speaker 1: Okay, that gives us the basic anatomy messages and topics
Speaker 1: split into partitions managed by brokers, with producers and consumers
Speaker 1: all coordinated by craft or zookeeper. But here's where it
Speaker 1: gets really interesting and a bit well mind bending for me. Initially,
Speaker 1: the sources describe Kofka's core nature as a distributed log.
Speaker 2: Yes, this is fundamental.
Speaker 1: Can you elaborate on that? Why is thinking of it
Speaker 1: as a log so important?
Speaker 2: Absolutely? What's truly fascinating here is that Kafka is fundamentally
Speaker 2: a distributed log. You need to sort of forget about
Speaker 2: it being just a message queue for a second. Okay,
Speaker 2: think of it more like an immutable personal diary, or
Speaker 2: maybe better the commit log of a database. It answers
Speaker 2: the question what happened? It focuses on the history of
Speaker 2: events rather than just what is which is the current state?
Speaker 1: Right history versus snapshot?
Speaker 2: Precisely So for our online retailer, it's not just current
Speaker 2: inventory is fifty shirts. It's more like a shirt was
Speaker 2: sold at ten point zero one am than another at
Speaker 2: ten point zero two am. Then we receive stock at
Speaker 2: ten point zero five am, the whole sequence.
Speaker 1: So it's about the sequence of actions, the journey, not
Speaker 1: just the final destination. What are the key properties of
Speaker 1: such a log that make it so powerful?
Speaker 2: Logs have distinct crucial properties. First order and sorting. Messages
Speaker 2: are always sorted time within a partition, oldest entry.
Speaker 1: The beginning partition, got it.
Speaker 2: Second, writing and reading direction. You always append new entries
Speaker 2: to the end of the log, like adding to a diary,
Speaker 2: and you typically read from old to new using what
Speaker 2: Kafka calls offsets to track your position.
Speaker 1: Offsets like bookmarks kinda yeah.
Speaker 2: And crucially, immutability. Once an entry is written, you can't
Speaker 2: easily change or remove it. It's like writing in permanent ink.
Speaker 1: That immutability has profound implications for data integrity. I imagine,
Speaker 1: and I've heard this concept of time travel mentioned with logs.
Speaker 1: How does that actually work and why is it such
Speaker 1: a game changer?
Speaker 2: That's right, time travel because the log is immutable and ordered.
Speaker 2: You can literally reconstruct the state of the world at
Speaker 2: any point in time by simply replaying the entries from
Speaker 2: the beginning of the log or from a specific offset.
Speaker 2: For our online retailer, this means you could replay all
Speaker 2: order placed events from last year to reconstruct exactly how
Speaker 2: many items were sold during a specific promotion wow, or
Speaker 2: even rebuild an entire system state if a database with
Speaker 2: some lost just from the Kofka log. This capability is
Speaker 2: really how Kafka helps businesses transition from traditional batch oriented
Speaker 2: processing you know, waiting for those overnight reports.
Speaker 1: Right at the end of day summary is to real.
Speaker 2: Time data handling, getting instant, up to the minute insights.
Speaker 1: That's incredibly powerful replaying history. But if a log is
Speaker 1: conceptually simple, why does it need to be distributed? Why
Speaker 1: not just one gigantic, super fast log on one machine. Yeah,
Speaker 1: good question, And this feels like where Kofka truly transforms
Speaker 1: from a concept into an industrial strength powerhouse. Must have
Speaker 1: big implications for like data resilience exactly.
Speaker 2: The challenge with the single log is clear speed, scalability
Speaker 2: and resilience a single system, a single server. It's often
Speaker 2: just not reliable enough hardware fails networks glitch, it's common.
Speaker 2: Kafka addresses this through horizontal scaling. Me Instead of buying bigger,
Speaker 2: more powerful servers, vertical scaling you use more servers. When
Speaker 2: you're existing brokers are getting busy, you just add another
Speaker 2: one to the cluster.
Speaker 3: Scale out, not up precisely.
Speaker 2: This is cheaper, much more flexible, and fits the reality
Speaker 2: that individual machines aren't perfectly reliable.
Speaker 1: So that's a fundamental architectural philosophy behind COFKA. Expect failures,
Speaker 1: build around them.
Speaker 2: It absolutely is. This approach is crucial because individual IT
Speaker 2: systems are seen as inherently unreliable. Horizontal scaling also enables parallelization.
Speaker 1: More work done at the same time.
Speaker 2: Right, allowing Kofka to process far more messages per unit
Speaker 2: of time than a single server ever could. And this
Speaker 2: works sufficiently because in KOFKA, data for one logical entities,
Speaker 2: say all events related to a specific product or maybe
Speaker 2: a single customer's order history can be kept in its
Speaker 2: own log partition.
Speaker 1: Using that message key you mentioned earlier.
Speaker 2: Exactly using the key, this ensures correct ordering for that
Speaker 2: specific entity, even if the overall order across all products
Speaker 2: isn't strictly sequential globally.
Speaker 1: Okay, that makes sense order within a context.
Speaker 2: So the implication for you, as someone listening and maybe
Speaker 2: trying to build robust systems, is that Kofka is designed
Speaker 2: from the ground up to be highly available and resilient
Speaker 2: even if parts of it fail. It distributes data and
Speaker 2: work across many machines. It's built for reliability in an
Speaker 2: unreliable world.
Speaker 1: That's a fantastic overview of the core architecture, very clear.
Speaker 1: Let's maybe talk about the messages themselves. Now. The source
Speaker 1: mentions Kofka's data agnosticism. What does that actually mean for
Speaker 1: what you can send through it?
Speaker 2: It means Kafka doesn't really care about the content of
Speaker 2: your messages. It treats all messages as raw BTE arrays.
Speaker 1: Just sequences of bytes yep.
Speaker 2: This flexibility is a key design choice. It allows Kafka
Speaker 2: to handle any kind of data, whether it's JSON, AVRO
Speaker 2: proto buff, plaintext, whatever, regardless of its format or structure.
Speaker 2: It doesn't try to interpret the data, which is actually
Speaker 2: a big part of its high performance.
Speaker 1: Like a postal service.
Speaker 2: Exactly like a postal service that delivers any package, big
Speaker 2: or small without needing to know what's inside.
Speaker 1: That sounds incredibly flexible, But doesn't that mean it's complete
Speaker 1: agnostic to the meaning or structure of the data. What
Speaker 1: are the implications of that for say, data governance or
Speaker 1: ensuring consistency down the line.
Speaker 2: Ah, you've hit on a crucial point there. While Kofka
Speaker 2: itself is agnostic, in practice, it is optimized for many
Speaker 2: small structured messages.
Speaker 1: Small unstructure.
Speaker 2: Okay, the default maximum message size is only one megabyte.
Speaker 1: Oh, that's smaller than I might have thought.
Speaker 2: It is, and why you can technically adjust this. It's
Speaker 2: generally advised against larger messages can severely impact performance disk space.
Speaker 2: It's just not what it's designed for. Kofka is built
Speaker 2: for high throughput of many small events, not for transferring
Speaker 2: large files like PDFs or big video files. I mean,
Speaker 2: look at LinkedIn. They famously use Kafka to process something
Speaker 2: like seven trillion messages a day across roughly one hundred
Speaker 2: clusters back in twenty nineteen.
Speaker 1: Seven trillion a day.
Speaker 2: That's an astonishing number of small messages. Really drives home
Speaker 2: the point definitely.
Speaker 1: So if most messages are small, what are the common
Speaker 1: types of messages you typically see in a real world
Speaker 1: KOFCA system, what patterns emerge in practice?
Speaker 2: Most systems use a mix of message types. You often
Speaker 2: see states, states, yeah, messages that describe the complete current
Speaker 2: state of an object, like all the details for a product,
Speaker 2: its current price, stock level, description, everything, and if you
Speaker 2: only care about the latest state. KOFKA has a feature
Speaker 2: called log compaction, which uses message keys to save space
Speaker 2: by keeping only the most recent version of a particular record.
Speaker 1: Okay, so state is the full picture right now? What else?
Speaker 2: Then? There are deltas. These contain only the changes in state,
Speaker 2: like just a stock quantity adjustment of negative five because
Speaker 2: an item sold or plus ten because stock arrived.
Speaker 1: Ah, just the change. That sounds way more efficient for
Speaker 1: data volume.
Speaker 2: It is much smaller messages, but they're less useful on
Speaker 2: their own.
Speaker 1: How so, what are the challenges if you only have
Speaker 1: deltas and need to know, say, the product's total stock
Speaker 1: right now?
Speaker 2: That's a great question. If you only have deltas, you'd
Speaker 2: have to process all previous deltas for that product just
Speaker 2: to reconstruct the current state. That could be computationally intensive
Speaker 2: for the consumer.
Speaker 1: Right you have to sum them all up exactly.
Speaker 2: Which is why events are often preferred. They describe what happened,
Speaker 2: but add context. Like instead of just MIGHTO five stock,
Speaker 2: the event might be order fulfilled event, which contains the
Speaker 2: stock change but also the order ID, customer ID timestamp
Speaker 2: more more meaning VAT, adjustment or promotion started, or other examples. Logs,
Speaker 2: in fact, are really just a special kind of event stream.
Speaker 1: Okay, events give context and you mentioned one more type.
Speaker 2: Yes, commands. These are used to instruct other systems to
Speaker 2: perform actions like ship this order command or process payment command.
Speaker 2: Unlike events, where the cener often doesn't care who listens,
Speaker 2: commands usually require a response or a specific action from
Speaker 2: the recipient system.
Speaker 1: That distinction between events and commands feels important. Commands expect
Speaker 1: a reaction. Now you mentioned, messages aren't just a singular
Speaker 1: blob of data. They have structure. Break that down for
Speaker 1: us again. What are the parts of a Kafka message?
Speaker 2: Yes, a Kafka message or record technically is composed of
Speaker 2: a few key elements. First, the value that's the primary payload,
Speaker 2: the actual information you want to convey, like the details
Speaker 2: of a customer order.
Speaker 3: Usually the biggest part the core data.
Speaker 2: And there's an optional key which is incredibly important even
Speaker 2: though it's optional, so important it's used to categorize messages,
Speaker 2: and critically, messages with the same key are guaranteed by
Speaker 2: Kafka to go to the same partition.
Speaker 1: Ah. So that's how you ensure order for a specific entity,
Speaker 1: like all updates for product one two over.
Speaker 2: Exactly, if you send all updates for product one twenty
Speaker 2: three with a key one twenty three, they land in
Speaker 2: the same partition in order, it guarantees their order relative
Speaker 2: to each other, at least from a single producer. The
Speaker 2: key is also essential for that log compaction feature we
Speaker 2: mentioned where Kafka retains only the latest message for a
Speaker 2: given key, very useful for topics representing current state.
Speaker 1: So the key is crucial for ordering and compaction, not
Speaker 1: just as a ID. What else is in a message?
Speaker 2: There are also optional custom headers. These are meant for
Speaker 2: technical metadata, things like tracing IDs for distributed systems, maybe
Speaker 2: security token stuff like that, not really for business data.
Speaker 1: Keep business data in the value generally yes.
Speaker 2: And finally, there's a timestamp. This records the time the
Speaker 2: message was created by the producer or potentially when it
Speaker 2: was appended to the broker log, depending on configuration. This
Speaker 2: timestamp is vital for many real time analytics scenarios, especially
Speaker 2: when you start dealing with time windows in stream processing.
Speaker 1: Fascinating how much detail goes into what seems like a
Speaker 1: simple message. Loads of potential there. Now, let's pivot to
Speaker 1: something absolutely critical for any data system, reliability. How does
Speaker 1: kofka build trust with your data? How does it ensure
Speaker 1: nothing gets lost or hopelessly out of order?
Speaker 2: Right? Reliability in Kofka is built on a few core pillars. First,
Speaker 2: replication and leaders followers.
Speaker 1: We touched on this leaders and followers for partitions exactly.
Speaker 2: For each partition, one broker acts as the leader. It
Speaker 2: handles all the incoming produce requests and outgoing consumer requests
Speaker 2: for that partition. The other brokers holding replicas for that
Speaker 2: partition are called followers, and they just continuously replicate or
Speaker 2: copy new messages from that leader. This creates redundant copies
Speaker 2: of your data across different machines.
Speaker 1: That sounds incredibly robust, multiple copies, But what actually happens
Speaker 1: behind the scenes when a leader fails let's say the
Speaker 1: machine crashes. Is the switch to a follower instantaneous? Are
Speaker 1: there any potential downsides or edge cases? A listener should be.
Speaker 2: Aware of good question. When a leader fails, Kafka automatically
Speaker 2: detects this and elects a new leader from its set
Speaker 2: of in sync replicas or ISRs. These are followers that
Speaker 2: are caught up with a leader's.
Speaker 1: Log ISRs, in sync replicas.
Speaker 2: YEAH, or sometimes eligible leader replicas elrs, depending on the setup.
Speaker 2: The goal is to ensure the topic remains accessible. Producers
Speaker 2: and consumers are designed to automatically detect this change and
Speaker 2: switch to the new leader, usually with minimal interruption. We're
Speaker 2: talking milliseconds typically.
Speaker 1: Okay, so it's fast. Any downsides?
Speaker 2: The main downside is that during that brief election period,
Speaker 2: that specific partition might be temporarily unavailable for writing. Reading
Speaker 2: might still be possible from followers depending on config, but
Speaker 2: writs need the leader. Also, once the original preferred leader
Speaker 2: comes back online and catches up, KAKA often aims to
Speaker 2: reinstate it as leader. This helps rebalance the leadership load
Speaker 2: across the cluster over time.
Speaker 1: That's excellent to know. Automatic failover is key. So how
Speaker 1: do acknowledgements or ACKs play into this? How do producers
Speaker 1: know their messages are safely persisted across these replicas before
Speaker 1: they move on?
Speaker 2: ACKs? Are precisely how producers control the durability guarantee and
Speaker 2: ensure messages are safely persisted. There are three main strategies
Speaker 2: controlled by the act's producer. Can fig with x zero
Speaker 2: it's basically fire and forget send in hope pretty much.
Speaker 2: It gives the highest performance because the producer doesn't wait
Speaker 2: for any confirmation at all, but it offers the lowest
Speaker 2: reliability comforable. Maybe to UDP networking. You could lose messages
Speaker 2: if the broker fails immediately.
Speaker 1: When would you ever use that?
Speaker 2: It's acceptable if some data loss is tolerable, maybe high
Speaker 2: volume sensor data where only the latest reading matters and
Speaker 2: losing an occasional reading isn't catastrophic, like a temperature sensor
Speaker 2: in a non critical system.
Speaker 1: Okay, what about AX one. That sounds like a middle ground.
Speaker 2: It is AX one means the producer gets a response
Speaker 2: and acknowledgment as soon as the leader broker successfully receives
Speaker 2: and writes the message to its local log. This offers
Speaker 2: much better latency than waiting for all replicas, but data
Speaker 2: loss is still possible if the leader receives the message
Speaker 2: sends the akaz act back to the producer but then
Speaker 2: crashes before that message gets replicated to its followers, that
Speaker 2: message is lost.
Speaker 1: Ah okay, so it's confirmed by the leader, but not
Speaker 1: guaranteed replicated.
Speaker 2: Yet exactly, which brings us to AXOL or you can
Speaker 2: write as a medico one. This has actually been the
Speaker 2: default setting since Kafka three.
Speaker 1: Point zero, the safest option.
Speaker 2: Yes, AXOL offers the highest reliability with this setting. The
Speaker 2: leader waits until all of the current in sync replica's
Speaker 2: ISRs have successfully persisted the data to their logs before
Speaker 2: sending that final ACK back to the producer.
Speaker 1: So you know it's on multiple machines, right.
Speaker 2: This is what you definitely want for critical data like
Speaker 2: those customer orders, financial transactions, anything you absolutely cannot lose.
Speaker 1: So for guaranteed delivery, AXOL is the gold standard. Is
Speaker 1: there a way to fine tune exactly how many InSync
Speaker 1: replicas need to acknowledge before the leader confirms? Maybe you
Speaker 1: don't need all of them, just a majority.
Speaker 2: Yes. Absolutely. That's where men dot nsync dot replicas comes in.
Speaker 2: It's a topic level configuration setting that works hand in
Speaker 2: hand with AXOL.
Speaker 1: How does it work?
Speaker 2: It specifies the minimum number of ISRs, including the leader
Speaker 2: itself that must acknowledge the right before the leader confirms
Speaker 2: receipt back to the producer. So if you have a
Speaker 2: replication factor of three and you set men dot nsync
Speaker 2: dot replicas two, then the right succeeds as long as
Speaker 2: the leader and at least one follower confirm it. If
Speaker 2: only the leader is available, well, the producer will get
Speaker 2: an error and can retry, preventing potential data loss if
Speaker 2: too many replicas are temporarily down or slow.
Speaker 1: That gives you really fine grain control over the durability
Speaker 1: versus availability trade off. Very useful, but what about guaranteeing
Speaker 1: messages are written exactly once and in the correct order,
Speaker 1: especially if a producer has to retry sending due to
Speaker 1: a temporary network issue or something that sounds like a
Speaker 1: classic distributed systems headache.
Speaker 2: It is a tough problem, but Kaofka has solutions for that.
Speaker 2: We turn to idempatance and transactions.
Speaker 1: Idempatance, meaning doing something multiple times, has the same effect
Speaker 1: as doing.
Speaker 2: It once, precisely by setting enabled dot idempatance true on
Speaker 2: the producer, which is actually the default now too. Alongside
Speaker 2: acts all, Kafka ensures that messages are written in the
Speaker 2: correct order. Within a partition and are present exactly once,
Speaker 2: even if the producer retries sending.
Speaker 1: How does it do that without much overhead?
Speaker 2: It uses sequence numbers assigned by the producer and tracked
Speaker 2: by the broker. The performance loss is negligible, maybe one
Speaker 2: percent or less, but the gain in data integrity is huge.
Speaker 2: Imagine if an order place message got duplicated because of
Speaker 2: a retry, I defidence prevents that.
Speaker 1: Okay, so I dumpetance handles duplicates from producer retries. What
Speaker 1: about transactions? When do they come in?
Speaker 2: Transactions are for achieving exactly once semantics eos When you're
Speaker 2: doing more complex things, especially involving multiple partitions or transferring
Speaker 2: data atomically between COFKA and other external systems like databases
Speaker 2: or other COFKA topics.
Speaker 1: Like a multi step process that needs to succeed or
Speaker 1: fail entirely exactly.
Speaker 2: A producer can begin a transaction, send messages to multiple partitions,
Speaker 2: and then either commit the transaction, making all messages visible
Speaker 2: to consumers, or abort it, discarding them all. It's atomic.
Speaker 1: How do transactions affect consumers? Do they need to do
Speaker 1: anything special?
Speaker 2: Yes? Crucially, consumers that need transactional guarantees must set their
Speaker 2: isolation dot level configuration to read.
Speaker 3: Committed, read committed Okay.
Speaker 2: This ensures they only read messages that are part of
Speaker 2: successfully committed transactions, filtering out any messages from aborted transactions
Speaker 2: or ongoing ones. It's actually good practice to set this
Speaker 2: even if you don't use transactions initially, just to be safe.
Speaker 1: And how does Kafka manage this atomicity across potentially multiple
Speaker 1: brokers and partitions.
Speaker 2: It uses a variation of the classic two phase commit protocol. Internally,
Speaker 2: it involves transaction coordinators on the brokers and special control messages.
Speaker 2: It's complex under the hood, but it provides that strong
Speaker 2: guarantee of atonic rights across multiple partitions, ensuring data consistency
Speaker 2: throughout your entire data flow.
Speaker 1: That's a truly comprehensive approach to reliability from replication and
Speaker 1: ACKs right through to idempaitence and transactions. Very impressive. Now,
Speaker 1: let's talk about speed. Kafka is famous for its performance.
Speaker 1: It's throughput. How does it achieve such high speeds and
Speaker 1: what are the key configurations that truly make it fly?
Speaker 2: Yeah, performance is definitely one of Kafka's hallmarks. It's inherently
Speaker 2: tuned for performance right out of the box. It's design
Speaker 2: basically assumes that hard disks are relatively cheap these days
Speaker 2: and memory is quite abundant, okay, and it heavily prioritizes
Speaker 2: horizontal scaling as we discussed. But to truly make it fly,
Speaker 2: optimization through careful configuration settings across all the components producers,
Speaker 2: brokers and consumers is vital. It's not just one magic switch.
Speaker 1: And you mentioned partitions are key to performance earlier? Can
Speaker 1: you elaborate on how they directly contribute to speed and throughput?
Speaker 2: Absolutely, partitions are key because they enable massive parallel processing
Speaker 2: and load balancing. Remember, topics are divided into partitions and
Speaker 2: these partitions are then distributed across the different brokeram machines.
Speaker 2: Producers determine which partition to send messages to, usually based
Speaker 2: on the key and on the consumer side, we use consumer.
Speaker 1: Groups consumer groups. What are those exactly?
Speaker 2: A consumer group is just a set of consumer instances
Speaker 2: that cooperate to consume from a topic. Kafka automatically assigns
Speaker 2: the partitions of a topic across the available consumers in
Speaker 2: a group. So if a topic has ten partitions and
Speaker 2: you have five consumers in a group. Each consumer will
Speaker 2: handle two partitions in parallel.
Speaker 1: AH, so the group processes the topic together in parallel
Speaker 1: across partitions exactly.
Speaker 2: This allows you to scale out your consumption by simply
Speaker 2: adding more consumer instances to the group up to the
Speaker 2: number of partitions, it drastically increases the overall message processing throughput.
Speaker 1: I understand that more partitions can mean more potential parallelism,
Speaker 1: more throughput, but it feels like there could be a
Speaker 1: point of diminishing returns or even negative consequences. What are
Speaker 1: the implications of having too many partitions? Is their downside?
Speaker 2: You're absolutely right to be cautious. There definitely is a downside.
Speaker 2: While increasing partitions can boost throughput up to a point,
Speaker 2: it introduces significant complexity and overhead if you go too far.
Speaker 2: Like what well, each partition demand client resources, memory on
Speaker 2: producers and consumers on the brokers, Each partition is a
Speaker 2: log file on disc requiring file handles, memory for indexing,
Speaker 2: and CPU for replication. Thousands or tens of thousands of
Speaker 2: partitions can really strain broker resources. It can also lead
Speaker 2: to prolonged unavailability during certain failure scenarios like leader elections,
Speaker 2: especially with older zookeeper managed clusters, where a zookeeper itself
Speaker 2: could become.
Speaker 1: A bottleneck, So finding the right number is important crucial.
Speaker 2: Imagine our online retailer suddenly deciding to have a million
Speaker 2: tiny partitions, one for every single product desk cue. While
Speaker 2: it might sound organized for ordering, the overhead of managing
Speaker 2: all those partitions, leader, elections, replication traffic, client connections would
Speaker 2: likely overwhelm the system and actually reduce overall performance instability.
Speaker 1: And can you easily change the number later?
Speaker 2: That's another catch. Reducing the number of partitions for a
Speaker 2: topic isn't really possible without potentially losing data or complex
Speaker 2: manual steps. Increasing partitions is easier. You could do that online,
Speaker 2: but increasing partitions disrupts message ordering guarantees for existing keys.
Speaker 2: Why because the partitioner usually calculates the target partition using
Speaker 2: something like hash key percent number of partitions. If you
Speaker 2: chang ange the number of partitions, the result changes and
Speaker 2: messages with the same key start going to different partitions
Speaker 2: than before, breaking strict ordering for that key until all
Speaker 2: the old data expires.
Speaker 1: Wow. Okay, so choosing the initial partition count and planning
Speaker 1: for future growth is really important. Get it wrong, It's
Speaker 1: hard to fix. Easily.
Speaker 2: Exactly optimal balance is key usually found through careful testing, monitoring,
Speaker 2: and understanding your data access patterns. Don't just pick a
Speaker 2: huge number upfront.
Speaker 1: That's a very clear warning. So partitions are critical for scaling.
Speaker 1: How does the producer specifically contribute to this incredible performance?
Speaker 1: Beyond just sending messages?
Speaker 2: Producer performance relies heavily on batching. This is super.
Speaker 1: Important batching grouping messages.
Speaker 2: Yes. Instead of sending each message individually over the network
Speaker 2: as soon as it's ready, producers collect messages destined for
Speaker 2: the same partition on the same broker and group them
Speaker 2: into larger batches than they send the entire batch in
Speaker 2: one go.
Speaker 1: That must save a lot of network round trips.
Speaker 2: Drastically, It significantly enhances performance and reduce uses network load
Speaker 2: by sending fewer, larger chunks of data rather than many
Speaker 2: small ones.
Speaker 1: How do you control that batching?
Speaker 2: You can figure it mainly with two settings. Yeah, batch
Speaker 2: dot size which sets the maximum batch size and bytes,
Speaker 2: and linger dot ms, which is the maximum time in milliseconds.
Speaker 2: The producer will wait to try and fill up a
Speaker 2: batch before sending it, even if it's not full yet.
Speaker 1: So a trade off between latency and throughput exactly.
Speaker 2: Larger linger dot m's values like five meters, ten meters
Speaker 2: or even more increase latency slightly because messages wait longer,
Speaker 2: but they also increase the chance of bigger batches, leading
Speaker 2: to much better throughput and efficiency. Finding the sweet spot
Speaker 2: depends on your application's latency requirements.
Speaker 1: And what about compression? Does that happen at the producer?
Speaker 2: Yes, and it's another big performance booster. The producer can
Speaker 2: compress the entire batch of messages just once before sending.
Speaker 1: It the whole batch, not message by message, whole batch.
Speaker 2: This is much more efficient than compressing individual messages. Common
Speaker 2: compression types like snappy, gzip, LZ for zstd are supported.
Speaker 2: This significantly reduces the amount of data sent over the
Speaker 2: network and also saves hard disk space on the brokers
Speaker 2: as they store and transmit the compressed batches unchanged.
Speaker 1: Clever batching and compression working together. And earlier you mentioned
Speaker 1: zero copy transfer as a neat trick for brokers to
Speaker 1: achieve high performance. How does that actually make the brokers faster?
Speaker 1: What are they doing or rather not doing right?
Speaker 2: Broker performance is maximized largely by keeping them as simple
Speaker 2: and efficient as possible. Their primary job is in complex computation.
Speaker 2: It's really about efficiently pushing bytes from network sockets to
Speaker 2: disc when receiving from producers, reducing and pushing bytes from
Speaker 2: disc back to network sockets when sending to consumers.
Speaker 1: Consuming just moving data pretty much.
Speaker 2: And this is where zero copy transfer, a feature available
Speaker 2: in Linux and other Unix like operating systems, comes into play.
Speaker 2: It's a fundamental reason Kofka can achieve such incredible speeds
Speaker 2: on commodity hardware.
Speaker 1: So what does zero copy actually avoid?
Speaker 2: What cop Imagine the traditional way data moves from the
Speaker 2: disc into the operating system kernel's memory page cash, then
Speaker 2: copied into the applications memory, the coca broker process, then
Speaker 2: copied back into the kernel socket buffer memory, and finally
Speaker 2: copied out to the network card. That's multiple copies and memory.
Speaker 1: Sounds inefficient, it is.
Speaker 2: Zero copy allows the kernel to directly transfer data from
Speaker 2: the disc cache page cash straight to the network socket
Speaker 2: buffer without needing that intermediate copy into the applications Kafka's
Speaker 2: memory space.
Speaker 1: It cuts out the middleman the application buffer exactly.
Speaker 2: It avoids unnecessary data copies and CPU context switches between
Speaker 2: kernel mode and user mode. This makes the broker astonishingly efficient,
Speaker 2: acting more like a super fast data pipeline or router
Speaker 2: than a heavy processing engine. And because Coffka's message format
Speaker 2: on disc is the same as it's over the wire format,
Speaker 2: this works beautifully.
Speaker 1: That's a really key optimization. And you also mentioned Kofka
Speaker 1: often relies on the OS for flushing data to disc
Speaker 1: rather than doing it manually after every rite.
Speaker 4: Yes.
Speaker 2: Generally Kafka avoids forcing manual sink operations after every message
Speaker 2: right for performance reasons, and FOLCNC forces the OS to
Speaker 2: physically write data from its cashes to the disc hardware immediately,
Speaker 2: which it can be slow.
Speaker 1: So it risks losing data if the OS crashes before flushing.
Speaker 2: In theory, yes, for data that's only in the OS cash,
Speaker 2: but Kofka relies on its replication mechanism for durability. By
Speaker 2: the time a producer gets an ax AL confirmation, the
Speaker 2: data is safely replicated to multiple brokers, OS caches and
Speaker 2: likely heading to disk soon via background OS processes. Relying
Speaker 2: on replication plus the OS's background flushing provides high throughput
Speaker 2: and strong durability guarantees in practice.
Speaker 1: Okay, that makes sense. Reliability through replications speed through avoiding
Speaker 1: forced sinks, so producers batch and compress brokers use zero copy.
Speaker 1: What about the consumer side? Is there a bottleneck there
Speaker 1: or is COFKA just pushing data as fast as the
Speaker 1: network allows?
Speaker 2: Consumer performances is definitely configurable. You have settings like fetch
Speaker 2: dot min dot bytes, which tells the broker the minimum
Speaker 2: amount of data to send back in one go.
Speaker 1: So the consumer isn't getting piny responses all the time.
Speaker 2: Right, and fetch dot max dot wheat dot ms the
Speaker 2: maximum time the broker will wait for that minimum amount
Speaker 2: of data to accumulate before sending back whatever it has.
Speaker 2: These help tune the balance between latency and throughput on
Speaker 2: the consumer side, similar to the producer's lingered MS.
Speaker 1: Okay, but where's the real limit?
Speaker 2: Usually here's a crucial insight and something many people overlook
Speaker 2: when troubleshooting performance. Consumer performance is typically limited by how
Speaker 2: fast the consumer application processes the data, not by COFA itself.
Speaker 1: So it's my code that's slow, not COFKA.
Speaker 2: Often, yes, COFA brokers are incredibly efficient at serving data.
Speaker 2: Our own performance tests and many others often reveal that
Speaker 2: consumers can easily pull data much faster than typical producers
Speaker 2: can even send it. The bottleneck is frequently your own
Speaker 2: application logic. How quickly can your online retailer's inventory serve,
Speaker 2: look up product details, updated's database and acknowledge the item
Speaker 2: sold event it just received from kofca. Or sometimes it's
Speaker 2: simply the network bandwidth available to the consumer, but rarely
Speaker 2: is it Kafka's ability to deliver the bytes.
Speaker 1: That's a really key distinction to keep in mind for
Speaker 1: optimization and troubleshooting. Focus on the consumer application logic first. Okay,
Speaker 1: we've covered the core mechanics, reliability, performance, fantastic foundation. Now
Speaker 1: let's talk about how coofka integrates with the wider world
Speaker 1: of systems and enables that exciting realm of real time
Speaker 1: data analysis. How does coofca connect fit into this picture?
Speaker 1: What is it?
Speaker 2: Coppa connect is a really powerful framework and tool that's
Speaker 2: part of the Apache Kofka project. Its purpose is to
Speaker 2: make it easy to integrate COFCA with external systems.
Speaker 1: External systems like what think.
Speaker 2: Databases, key value stores, search indexes, file systems, cloud storage
Speaker 2: like S three, messaging queues like JMS, pretty much anything
Speaker 2: you'd want to get data at kofka from or get
Speaker 2: data out of Kofka into Okay.
Speaker 1: So it's like a universe data bridge builder for COFKA.
Speaker 2: That's a great way to put it. And crucially, it
Speaker 2: aims to do this without you having to write custom
Speaker 2: integration code for every single system you use pre built
Speaker 2: or community connectors.
Speaker 1: So why should you, the listener care about COOFCA connect?
Speaker 1: What's the big benefit?
Speaker 2: The big benefit is that it helps automate and standardize
Speaker 2: data flow to and from Kofka. It massively simplifies building
Speaker 2: and managing these data pipelines. For our online retailer example,
Speaker 2: it means effortlessly getting customer profile updates from a CRM
Speaker 2: system into a COSTA topic or pushing processed order data
Speaker 2: from Kofka into a downstream data warehouse or fulfillment system,
Speaker 2: all using configuration rather than complex custom code.
Speaker 1: Less code, more configuration sounds good. How does it work? Architecturally?
Speaker 2: A Kafka connect deployment runs as a cluster of workers.
Speaker 2: These are just JVM processes that execute the integration tasks.
Speaker 2: They handle skivelling offsets, configuration, and distributing.
Speaker 1: The actual work workers running the tasks and the tasks themselves.
Speaker 2: The actual integration logic is encapsulated in connectors. These are
Speaker 2: plugins that you deploy to your connect pluster. There are
Speaker 2: two main types, source connectors and sync connectors.
Speaker 1: Source and sync easy enough.
Speaker 2: Source connectors import data from an external system into Kofka topics.
Speaker 2: For example, a GDBC source connector can pull a database
Speaker 2: table for new rows or more powerfully, connectors like Debisium
Speaker 2: perform change data capture CDC by reading database transaction logs
Speaker 2: and sending every single row level change, insert, update, delete
Speaker 2: to Kafka in real time.
Speaker 1: CDC is huge for real time data warehousing and replication.
Speaker 2: Absolutely and then sync connectors do the opposite. They export
Speaker 2: data from Coffka topics to external systems, like writing records
Speaker 2: from a Cofka topic into lastic search for searching, or
Speaker 2: hdfs for archiving, or maybe calling a rest api.
Speaker 1: Very powerful. Can you do any like light transformations on
Speaker 1: the messages as they pass through connect maybe clean things
Speaker 1: up a bit before they hit kafka or before they
Speaker 1: go to the sync system.
Speaker 2: Yes, you can. Coffka connects support something called Sync Message
Speaker 2: transformations or smts smmt's. These allow you to perform simple,
Speaker 2: stateless record level transformations on messages within the connect pipeline.
Speaker 2: They could be applied before a message is written to
Speaker 2: Kafka by a source connector, or before a message is
Speaker 2: written to an external system by a sync.
Speaker 1: Connector what kind of transformations.
Speaker 2: Common examples include things like renaming fields, replaced field, dropping fields,
Speaker 2: replace field again or drop, extracting a field from the
Speaker 2: message value to become the message key value, toke, pulling
Speaker 2: out a specific field, extract field, or even masking sensitive
Speaker 2: fields by setting them to null or a fixed value
Speaker 2: mask field for privacy or compliance reasons.
Speaker 1: That sounds really useful for basic cleanup or shaping.
Speaker 2: It is for light work, but there's an important warning here.
Speaker 2: Smts are not a fully fledged ETL extract transform load tool.
Speaker 2: They are designed for simple, stateless transformations on individual messages.
Speaker 2: If you need complex transformations, stateful operations, joins between different
Speaker 2: data sources, or heavy computation, smts are not the right tool.
Speaker 1: So for the heavy lifting.
Speaker 2: For that you typically turn to a dedicated stream processing framework.
Speaker 1: AH stream processing. Perfect segue. Welcome to the world of
Speaker 1: stream processing. What exactly is that and why is it
Speaker 1: such a natural fit such a big deal When talking
Speaker 1: about Kofka.
Speaker 2: Stream processing is essentially about processing data continuously as it arrives,
Speaker 2: typically in real time or near real time, instead of
Speaker 2: collecting data into batches and processing it hours later the
Speaker 2: old way. Right, you process potentially unbounded streams of data
Speaker 2: events as they flow through the system, like data flowing
Speaker 2: through Coofka topics and the benefit This enables instant analysis,
Speaker 2: immediate reactions to business changes, and the creation of applications
Speaker 2: that are always up to date. Think back to our
Speaker 2: online retailer. Instead of knowing total sales only at the
Speaker 2: end of the day from a Bash report, stream processing
Speaker 2: allows them to see real time sales trends as they happen.
Speaker 2: They can detect potentially fraudulent transactions within seconds, or trigger
Speaker 2: personalized offers based on a customer's immediate browsing behavior on
Speaker 2: their website. It unlocks truly real time capabilities.
Speaker 1: That makes sense moving from batch latency to real time responsiveness.
Speaker 1: What are some common frameworks people use for this with Kofka.
Speaker 2: There are several powerful frame works out there. Kafka streams
Speaker 2: is a very popular choice because it's actually a Java
Speaker 2: library that's part of the Apache Kofka project itself. It
Speaker 2: makes it easy to build stream processing applications that read
Speaker 2: from and write to.
Speaker 1: Kaofka tightly integrated.
Speaker 2: Very then you have other major open source players like
Speaker 2: Apache Flank, which is known for its sophisticated state management
Speaker 2: and event time processing capabilities. There's also Apache spark streaming,
Speaker 2: though it's more microbatch oriented. Historically, and in other ecosystems,
Speaker 2: you might see things like Scala's AKA streams or Python's
Speaker 2: faust library.
Speaker 1: Lots of choices, Let's maybe focus on Kofka streams. Since
Speaker 1: it's part of kofka. How do you actually process these
Speaker 1: streams using it? What are the building blocks.
Speaker 2: And Kafka streams. If you define your processing logic as
Speaker 2: a topology of processors, kind of like chaining together operations,
Speaker 2: you typically start with a source processor which reads data
Speaker 2: from one or more Cofka topics into a stream called
Speaker 2: a k stream.
Speaker 1: Okay, get the data in then what Then.
Speaker 2: You apply various transformation or processing steps. You can filter
Speaker 2: messages based on some condition, maybe only keep orders with
Speaker 2: the value over one hundred dollars. You can use map
Speaker 2: or map values to transform the messages. Map values just
Speaker 2: changes the message value. Well, map can change both the
Speaker 2: key and the value, or even the type of.
Speaker 1: The message useful for reshaping data definitely.
Speaker 4: You can merge two different data streams together into one,
Speaker 4: or you can split a single stream into multiple downstream
Speaker 4: topics based on different conditions, like routing high value orders
Speaker 4: to one topic and standard orders.
Speaker 1: To another, branching the flow exactly.
Speaker 2: And A very common and powerful operation is aggregation. Like count,
Speaker 2: This coount's occurrence is per key. For example, continuously counting
Speaker 2: how many times each product was viewed or.
Speaker 1: Added to a cart. That sounds like it needs to
Speaker 1: remember things over time.
Speaker 2: It does. Operations like count. Some reduce are stateful. They
Speaker 2: need to maintain and update some internal state based on
Speaker 2: the incoming messages. Kofka Streams manages this state reliably interesting.
Speaker 1: I've also heard of streaming squel being used with Kafka.
Speaker 1: Is that like running familiar SQL queries but on live,
Speaker 1: constantly changing data streams instead of static tables.
Speaker 2: Precisely, streaming SQL offers a higher level declarative way to
Speaker 2: define stream processing logic using SQL like syntax. Frameworks like
Speaker 2: KSQLDB built on Coofka Streams or flink sql allow you
Speaker 2: to write queries like selection productive count from clicks group
Speaker 2: by producted directly on data streams.
Speaker 1: But what does that query return? A stream doesn't have
Speaker 1: an end good point.
Speaker 2: Unlike a traditional database query that runs once and returns
Speaker 2: a single final result set, a streaming SEQL query typically
Speaker 2: runs continuously and produces a new data stream of changes.
Speaker 1: A stream of changes.
Speaker 2: Yeah, so for that at count query, the output stream
Speaker 2: would contain messages indicating the updated count for a productive
Speaker 2: Every time the count changes due to a new click
Speaker 2: event arriving, it continuously refines the result.
Speaker 1: So the result itself is a stream. That's a different
Speaker 1: way of thinking it is.
Speaker 2: And maybe frameworks like flank sql also support a headless
Speaker 2: mode where you can deploy pre defined SQL queries that
Speaker 2: just run continuously in the background, perhaps writing their continuously
Speaker 2: updated results back to another Kafka topic or an external database.
Speaker 1: You mentioned state full operations like counting and how Kaffa
Speaker 1: streams manage a state. You also hear about stream states
Speaker 1: and tables. How do those fit in? Especially with aggregations.
Speaker 2: Right, stream processing frameworks need a way to reliably store
Speaker 2: the state required for operations like aggregations, counts, sums, averages,
Speaker 2: or for joins between streams in Kaffa streams. This state
Speaker 2: is typically stored in local state stores on the machine
Speaker 2: running the application instance. These are often backed by embedded
Speaker 2: databases like rockstd for performance local storage.
Speaker 1: What happens if the the application instance crashes is the state.
Speaker 2: Loss ah good question. That's where Kafka's own reliability comes in.
Speaker 2: These local state stores are backed by internal change log
Speaker 2: topics in Kafka.
Speaker 1: Itself changelog topics.
Speaker 2: Yes, every update made to the local state store is
Speaker 2: also written as a message to a compacted Kofka topic.
Speaker 2: If your application instance crashes and restarts, possibly on a
Speaker 2: different machine, Kafka streams can automatically restore its local state
Speaker 2: by replaying the messages from that changelog topic. It makes
Speaker 2: the state fault tolerant.
Speaker 1: Clever using Kafka to back up the state of the
Speaker 1: stream processor exactly so.
Speaker 2: Those aggregations like some or Average, use these state stores
Speaker 2: to keep track of the running calculation. The result of
Speaker 2: an aggregation in Kafka streams is often represented as a
Speaker 2: K table.
Speaker 1: A K table. What's that compared to a K stream.
Speaker 2: Think of a K stream as representing the raw sequence
Speaker 2: of events, the history what happened. A K table, on
Speaker 2: the other hand, represents the current state derived from that stream,
Speaker 2: like an up to date view or materialize few What
Speaker 2: is the current value? So the output of our account
Speaker 2: aggregation would be a K table where the key is
Speaker 2: the producted and the value is its latest.
Speaker 1: Count stream effects table of current state. Got it? And
Speaker 1: what about combining data from different streams or enriching a
Speaker 1: stream with data from a table. How do streaming joins work?
Speaker 1: In this world?
Speaker 2: Joins are essential for enriching data. Stream processing frameworks support
Speaker 2: various types of joins. You can do stream table joins.
Speaker 2: This is common for enrichment. Imagine you have a K
Speaker 2: stream of order events and a K table containing customer
Speaker 2: profile information keep by customer ID. You can join the
Speaker 2: order stream with a customer table to add the customer's
Speaker 2: name and address to each order event as it flows through.
Speaker 2: The join is typically triggered when a new event arrives
Speaker 2: on the stream and it looks up the corresponding key.
Speaker 4: In the table.
Speaker 1: Okay, enriching events with static ish data. Right.
Speaker 2: You can also do table table joints, where changes in
Speaker 2: either table can trigger updates to the joined result. This
Speaker 2: is useful for combining two evolving data sets. And then
Speaker 2: you have stream stream joins joining two potentially infinite streams
Speaker 2: of events. This usually requires defining a time window, because
Speaker 2: you need to specify how long the system should wait
Speaker 2: for a matching event to arrive on the other stream
Speaker 2: before giving up. For example, joining AD impressions with AD
Speaker 2: clicks based on a user ID within say a five
Speaker 2: minute window.
Speaker 1: Windows become crucial for stream stream joins. And you mentioned
Speaker 1: earlier that for joins to work efficiently, data often needs
Speaker 1: to be copartitioned. Can you remind us what that means? Sure?
Speaker 2: Copartitioning is a prerequisite for efficient joins and some aggregations
Speaker 2: in many distributed stream processing systems, including cough To streams.
Speaker 2: It means that records from the different topics being joined,
Speaker 2: which share the same joint key, must reside in partitions
Speaker 2: with the same ID number across.
Speaker 1: Those topics, same key, same partition number, even if in
Speaker 1: different topics exactly.
Speaker 2: Think of it like this. If you have customer orders
Speaker 2: in one Kafka topic orders and customer addresses in another topic,
Speaker 2: dresses both potentially partitioned across multiple brokers. For Kafa streams
Speaker 2: to efficiently join an order with its corresponding address based
Speaker 2: on customer it needs to ensure that the order for
Speaker 2: a customer one twenty three Q one twenty three and
Speaker 2: the address for customer one twenty three Q one twenty
Speaker 2: three both land in say, partition five of their respective topics.
Speaker 2: Why is that necessary Because then the Kaffka stream's task
Speaker 2: responsible for processing partition five will have local access to
Speaker 2: both the order and the address data for customer one
Speaker 2: twenty three. It doesn't need to make slow network calls
Speaker 2: to fetch data from other partitions or other instances. It
Speaker 2: keeps the joint operation efficient and scalable.
Speaker 1: It's like ensuring related files are in the same filing
Speaker 1: cabinet drawer, even across different cabinets, so you only have
Speaker 1: to look in one place.
Speaker 2: That's a perfect analogy. If data isn't naturally co partitioned
Speaker 2: by key when it arrives, kopfcas streams often needs to
Speaker 2: perform an internal repartitioning step, which involves writing the data
Speaker 2: to an intermediate correctly partitioned topic before the join can happen.
Speaker 2: This adds some overhead, but ensures correctness.
Speaker 1: That co partitioning requirement makes perfect sense for distributed joints.
Speaker 1: Now time itself seems like a really crucial and potentially
Speaker 1: tricky concept in stream processing. You mentioned event time earlier.
Speaker 1: What are the different time concepts we need to be
Speaker 1: aware of and why does it matter which one you use?
Speaker 2: Yes, Understanding time is absolutely vital and choosing the wrong
Speaker 2: time semantic can lead to inaccurate results. There are generally
Speaker 2: four key time concepts people talk about. First, event time.
Speaker 2: This is when the event actually occurred in the real world,
Speaker 2: like the timestamp generated by the sensor when it took
Speaker 2: a reading, or the exact moment a customer clicked a
Speaker 2: button on the website.
Speaker 1: The time it happened.
Speaker 2: Second, create time. This is the time when the producer
Speaker 2: application created the COFCA message. This is often close to
Speaker 2: event time, but could be later if there's a delay
Speaker 2: in the producing system. This is actually the default time
Speaker 2: stamp stored in COFCA messages if the producer doesn't explicitly
Speaker 2: sell one. Third, log a pen time this is the
Speaker 2: time when the coffer broker received the message and appended
Speaker 2: it to the partition's log. This timestamp is assigned by
Speaker 2: the broker itself. Messages within a partition are strictly ordered
Speaker 2: by log.
Speaker 1: A pen time Okay, broker received time.
Speaker 2: And finally, stream time or sometimes called processing time. This
Speaker 2: is the time when the message is actually processed by
Speaker 2: the stream processing application.
Speaker 1: Instance.
Speaker 2: This is usually the latest of all the time stamps
Speaker 2: and can be affected by processing delays, network latency, etc.
Speaker 1: So event time create time log a pen time, stream
Speaker 1: processing time. Why does the choice matter so much?
Speaker 2: It matters because if you want accurate results that reflect
Speaker 2: the real world sequence of events, especially when dealing with
Speaker 2: data that might arrive late or out of order, which
Speaker 2: is common in distributed systems, you generally want to use
Speaker 2: event time.
Speaker 1: Even if messages arrive out of sequence.
Speaker 2: Yes, processing based on event time allows the system to
Speaker 2: correctly handle out of order data and produce results consistent
Speaker 2: with when things actually happened. Example, if you're calculating hourly
Speaker 2: sales totals, using event time ensures a sale that occurred
Speaker 2: at ten five h zero five am but arrive late
Speaker 2: at eleven five zero five am still gets counted in
Speaker 2: the ten point zero zero eleven point zero am window.
Speaker 2: Using processing time would put it in the wrong hour.
Speaker 1: Ah. That makes a huge difference for accuracy. But processing
Speaker 1: based on event time sounds more complex.
Speaker 2: It is more complex. The stream processor needs mechanisms to
Speaker 2: handle potentially late arriving data, often using concepts like water
Speaker 2: marks to track the progress of event time and decide
Speaker 2: when it's safe to finalize calculations for a given time.
Speaker 2: Window processing time is simpler, but less accurate for many
Speaker 2: use cases.
Speaker 1: Okay, that clarifies the time concepts and building on that
Speaker 1: you mentioned, time windows are often used with stream processing,
Speaker 1: especially for aggregations or joints. What are some common types
Speaker 1: of windows and one would you use them? Come?
Speaker 2: Windows are fundamental for defining boundaries for calculations on unbanded streams.
Speaker 2: Common types include tumbling windows. These are fixed size, non
Speaker 2: overlapping windows. Think of them like slicing time into consecutive chunks,
Speaker 2: for example, calculating total sales for each distinct hour ten
Speaker 2: point zero zero, eleven point zero zero, eleven point zero zero, zero, twelve,
Speaker 2: de verits, et cetera. Great for curiotic.
Speaker 1: Reports fixed separate blocks. Got it?
Speaker 2: Then you have sliding windows. These also have a fixed length,
Speaker 2: but they slide forward continuously by a specified slide interval,
Speaker 2: meaning the windows overlap. For example, calculating the average website
Speaker 2: response time over the last five minutes updated every one minute.
Speaker 2: Useful for monitoring moving averages or recent trends.
Speaker 1: Overlapping continuously updated.
Speaker 2: Few right, they're also hopping windows. These are similar to
Speaker 2: sliding windows fixed length overlapping, but defined by both the
Speaker 2: window size and a fixed advancement interval the hop For example,
Speaker 2: calculating a daily report covering the last seven days where
Speaker 2: the window is seven days long and it hops forward
Speaker 2: by one day each day.
Speaker 1: Okay, like a sliding window, but maybe advancing less frequently
Speaker 1: kind of.
Speaker 2: And finally, session windows these are quite different. Their boundaries
Speaker 2: aren't based on fixed time intervals, but on peer of
Speaker 2: inactivity in the data stream.
Speaker 1: Grouped by key inactivity. How does that work?
Speaker 2: You define a session gap duration for a given key
Speaker 2: like a user ID. All events arriving within that gap
Speaker 2: duration of each other are grouped into the same session window.
Speaker 2: If no event arrives for that key for longer than
Speaker 2: the gap, the session is considered closed and the next
Speaker 2: event starts a new session. Perfect for tracking user sessions
Speaker 2: on a website where session ends after say thirty minutes
Speaker 2: of no clicks from that user.
Speaker 1: That's really clever for activity based analysis. Lots of windowing options,
Speaker 1: so we have all these powerful stream processing capabilities with
Speaker 1: Kafka streams transformations, state management, joins Windows. How does Kafka
Speaker 1: streams actually achieve parallelization for all this work? If I
Speaker 1: run multiple instances of my streaming application, how do they coordinate?
Speaker 2: Kaffka Streams leverages Costca's own partitioning model for parallelization, which
Speaker 2: is really elegant. It effectively splits the processing topology your
Speaker 2: chain of K streams and K tables into independent units
Speaker 2: called tasks. Each task is responsible for processing data from
Speaker 2: one or more specific partitions of the input Cofka topics.
Speaker 1: So one task per input partition roughly generally.
Speaker 2: Yes, although a task might process partitions from multiple topics
Speaker 2: if they're part of a join or merge. These tasks
Speaker 2: are the smallest unit of parallelization. Kaffka Streams then automatically
Speaker 2: distributes these tasks as evenly as possible across all the
Speaker 2: running instances of your application that share the same application
Speaker 2: dot eight ah.
Speaker 1: The application dot A links the instances together.
Speaker 2: Correct It acts like the consumer group i D we
Speaker 2: discussed earlier. If you have say, ten partitions in your
Speaker 2: input topic, and you run five instances of your Coffa
Speaker 2: streams application with the same id Kaffa streams will assign
Speaker 2: two tasks and thus two partitions to each instance to
Speaker 2: process in parallel.
Speaker 1: And what happens if one of those application instances fails Coffa.
Speaker 2: Streams handles that automatically too, using Kafka's underlying consumer group
Speaker 2: rebalancing protocol. If an instance fails or leaves the group,
Speaker 2: its task are automatically redistributed among the remaining healthy instances. Similarly,
Speaker 2: if you add a new instance, tasks will be migrated
Speaker 2: to it to rebalance the load. This provides elasticity and
Speaker 2: fault tolerance for your stream processing very resilient.
Speaker 1: You mentioned repartitioning earlier for joins, Does that involve these
Speaker 1: tasks too?
Speaker 2: Yes. For operations like key based joins or aggregations group
Speaker 2: by key count, et cetera that require data to be
Speaker 2: grouped by key, Kofka streams might need to perform that
Speaker 2: repartitioning step we talked about. This is done internally by
Speaker 2: writing the relevant data stream to a special intermediate KOFCA
Speaker 2: topic often called a repartition topic, which is correctly partitioned
Speaker 2: by the required key. Then downstream tasks read from this
Speaker 2: repartition topic. This effectively shuffles the data across the tasks
Speaker 2: based on the key, ensuring all messages for a specific
Speaker 2: key are processed by the same task, even if they
Speaker 2: originally came from different input partitions. This might split your
Speaker 2: overall processing logic into what COFFCA Streams calls sub topologies
Speaker 2: connected by these interns ernal repartition topics, optimizing dataflow and correctness.
Speaker 1: That's incredibly powerful and quite sophisticated under the hood, allowing
Speaker 1: complex stateful processing to scale out and be fault tolerant.
Speaker 1: But with all this power comes responsibility. Right, we've built
Speaker 1: this amazing real time data nervous system. Let's talk about
Speaker 1: management and security. How do you keep KOFKA healthy, compliant,
Speaker 1: and secure, especially in a large production environment with many
Speaker 1: teams using it.
Speaker 2: You're absolutely right. Once KOFKA becomes central to your data architecture,
Speaker 2: effective governance becomes crucial. Without it, things can quickly descend
Speaker 2: into chaos.
Speaker 1: Chaos how well.
Speaker 2: Imagine different teams producing data to the same topic but
Speaker 2: using slightly different formats or field names. Without agreement. Downstream
Speaker 2: consumer's break data becomes inconsistent, trust or roads. You risk
Speaker 2: that garbage in garbage out scenario.
Speaker 1: We mentioned so defining clear rules is step one.
Speaker 2: Yes, even if you don't use a formal tool initially,
Speaker 2: data always has a schema, implicit or explicit. Documenting and
Speaker 2: agreeing on schemas between producers and consumers is paramount For
Speaker 2: our online retailer. The team producing order events needs to
Speaker 2: agree with the teams consuming them, fulfillment analytics, etc. On
Speaker 2: exactly what fields are present, their types, and whether they
Speaker 2: are required or optional.
Speaker 1: Okay, agree on the schema, But schemas evolve. What about
Speaker 1: handling changes? How do you ensure a change made by
Speaker 1: one team doesn't break everyone else.
Speaker 2: That's where schema compatibility levels come into play. These are
Speaker 2: rules that define how schemas are allowed to evolve over time.
Speaker 2: They cover in changes like adding or deleting fields or
Speaker 2: changing types.
Speaker 1: What are the common levels?
Speaker 2: You typically have no compatibility or no nell anything goes.
Speaker 2: Any change is allowed, even breaking ones like renaming a
Speaker 2: required field very risky. Backward compatibility new schemas must be
Speaker 2: readable by applications using older schemas. This usually means you
Speaker 2: can add optional fields or delete existing fields, but not
Speaker 2: add required fields or rename existing ones. Consumers run older
Speaker 2: code won't break when encountering new data.
Speaker 1: Okay, new consumers can read old data. Wait, no other
Speaker 1: way around. Old consumers can read new data. Let me rephrase.
Speaker 1: Backward means consumers using the new schema can still process
Speaker 1: data written with the old schema. You can add optional
Speaker 1: fields or delete fields. Ah.
Speaker 2: Yes, let me clarify that. Backward compatibility means consumers using
Speaker 2: the new schema can process data produced with the old schema.
Speaker 2: This usually allows deleting fields or making required fields optional.
Speaker 1: Got it, new code reads old data.
Speaker 2: Then forward compatibility, consumers using an old schema can process
Speaker 2: data produced with the new schema. This usually allows adding
Speaker 2: new fields typically optional, or making optional fields required. Old
Speaker 2: code won't break when seeing new data.
Speaker 1: Formats old code reads new data.
Speaker 2: And finally, full compatibility. This combines both backward and forward
Speaker 2: Both new and old consumers can read both new and
Speaker 2: old data. This is the safest but often the most restrictive,
Speaker 2: typically only allowing adding or moving optional fields.
Speaker 1: Choosing the right compatibility level seems critical for managing change smoothly.
Speaker 2: Absolutely, it prevents unexpected breakages as your data landscape evolves.
Speaker 1: And that leads us naturally to schema registries. I assume
Speaker 1: are these the tools that enforce these compatibility rules exactly.
Speaker 2: Schema registries like the popular Confluence Schema Registry or alternatives
Speaker 2: like care Paths or EPICUREA registry act as a central
Speaker 2: authority or single source of truth for all schemas used
Speaker 2: within your Kafka ecosystem.
Speaker 1: What do they do?
Speaker 2: They store schemas like Avro, Poto, Boof or Jason schema definitions,
Speaker 2: manage different versions of those schemas, and critically, they enforce
Speaker 2: the compatibility rules you've defined for each topic or subject.
Speaker 2: In registry terms, when a producer tries to send data,
Speaker 2: it often first checks with the registry if the schema
Speaker 2: it's using is compatible with the registered versions for that
Speaker 2: topic according to the configured compatibility level. If not, the
Speaker 2: registry can reject the attempt, preventing incompatible data from ever
Speaker 2: entering cope. Consumers also use a registry to fetch the
Speaker 2: correct schema to de serialize incoming data.
Speaker 1: So the registry acts like a gatekeeper for schema quality
Speaker 1: and evolution.
Speaker 2: Precisely, it's a cornerstone of good Kafka governance.
Speaker 1: Okay governance handles the data structure. Moving to security, what
Speaker 1: are the core concerns for protecting your Kafka cluster and
Speaker 1: the data flowing through it from unauthorized access or potential breaches.
Speaker 1: This must be top of mind for anyone running Cofka
Speaker 1: with sensitive data.
Speaker 2: Security is absolutely paramount, and it involves several layers. First,
Speaker 2: you need authentication. This verifies who a client, producer, consumer, broker,
Speaker 2: dual claims to be. Common mechanisms include using mutual TLS MTLs,
Speaker 2: where both the client and server present certificates to verify
Speaker 2: each other's identity.
Speaker 1: Encrypted and authenticated connection.
Speaker 2: Right, or using SSL simple authentication and security layer which
Speaker 2: supports various pluggable methods like Curbaro's common and traditional enterprises
Speaker 2: plane user name password used with TLS scram, more secure challenges,
Speaker 2: sponsor passwords or even o opatonide connect via SaaS loft
Speaker 2: bearer for integration with modern identity providers.
Speaker 1: So confirm identity first.
Speaker 2: Then what Once a client is authenticated? You need authorization.
Speaker 2: This defines what an authenticated client is actually allowed to do.
Speaker 2: Can it read from topic A? Can it write to
Speaker 2: topic B? Can it create new topics?
Speaker 1: Controlling permissions exactly?
Speaker 2: Kafka typically handles this via Access control Lists ACLS. You
Speaker 2: define rules specifying which user principle has which permission read, write, create, describe, etc.
Speaker 2: On which resource, topic, group cluster. The best practice here
Speaker 2: is always the least privileged principle. Grant only the permissions
Speaker 2: absolutely necessary for a client to perform its function, nothing more.
Speaker 1: Don't give the marketing analytics consumer rate access to the
Speaker 1: payment processing topic.
Speaker 2: Definitely not. ACLS prevent that.
Speaker 1: Okay, Authentication and authorization control access. What about protecting the
Speaker 1: data itself, both as it moves across the network and
Speaker 1: when it's sitting on the broker's discs.
Speaker 2: Good point. That involves encryption for data in transit between
Speaker 2: clients and brokers or between brokers themselves. You use transport encryption,
Speaker 2: typically TLS Transport Layer Security, the successor to SSL. Enabling.
Speaker 2: TLS encrypts all the COOFKA traffic, preventing eavesdropping on the network.
Speaker 1: Is there a performance hit?
Speaker 2: Yes, TLS does introduce some CPU overhead for the encryption
Speaker 2: decryption process, so there's a performance cost, but it's often
Speaker 2: a necessary cost for security. KOFKA cleverly supports configuring multiple
Speaker 2: listeners per broker, For example, one plaintext listener on port
Speaker 2: nine zero nine two and one TLS listener on part
Speaker 2: nine zero nine three. This allows for gradual migration of
Speaker 2: clients to TLS without downtime.
Speaker 1: Okay TLS for data in motion. What about encryption at
Speaker 1: rest when data is stored on the broker discs.
Speaker 2: Kofka itself doesn't provide built in features for encrypting data
Speaker 2: stored within its log files on disc. For encryption at rest,
Speaker 2: you typically rely on capabilities of the underlying operating system
Speaker 2: or storage system, for instance, using filesystem level encryption like
Speaker 2: Linux's dmcrypt LUKS or features provided by cloud storage volumes
Speaker 2: like ebs encryption on AWS.
Speaker 1: So handle it at the infrastructure.
Speaker 2: Layer generally yes. Alternatively, a pattern sometimes used as employing
Speaker 2: a secure KOFCA proxy that encrypts message values before they
Speaker 2: are even produced to Kafka and decrypts them after consumption.
Speaker 2: This has complexity, but ensures the data on the broker
Speaker 2: disc is encrypted at the application level.
Speaker 1: And what about true end to end encryption where only
Speaker 1: the original producer and final consumer can decrypt the message
Speaker 1: and even the brokers can't see the plaintext.
Speaker 2: That offers the highest level of confidentiality for critical data. However,
Speaker 2: it must be implemented entirely by the client applications themselves.
Speaker 2: The producer incrudes the message value before sending, and the
Speaker 2: consumer decrypts it after receiving. KOFKA just transports the encrypted
Speaker 2: byt RAY. There's no widely adopted standard library or built
Speaker 2: in KOFKA feature for this, so it requires careful implementation
Speaker 2: and key management on the client side.
Speaker 1: Okay, so multiple layers often off c TLS encryption and
Speaker 1: transit infrastructure encryption at rest, an optional client side end
Speaker 1: to end encryption seems comprehensive. What if you have an
Speaker 1: existing cluster that was set up without security enabled. Is
Speaker 1: it a massive disruptive project to secure it later without
Speaker 1: causing significant downtime?
Speaker 2: Fortunately, no, it's usually manageable. You can secure an unsecured
Speaker 2: cluster gradually without affecting availabilities significantly. The key is using
Speaker 2: those multiple listeners. You can add new secure listeners example
Speaker 2: saslpls alongside the existing plaintex listeners on all brokers, usually
Speaker 2: requiring rolling restarts of the brokers one by one. Then
Speaker 2: you update the inner broker communication protocol to use the
Speaker 2: secure listener another rolling restart. Finally, you migrate your client
Speaker 2: applications incrementally to connect to the secure listeners and configure
Speaker 2: their authentication credentials. Once all clients are migrated, you can
Speaker 2: optionally disable the old plaintex listeners, a well defined process
Speaker 2: designed to minimize disruption, so.
Speaker 1: It's not an all or nothing, big bang cutover. That's
Speaker 1: very reassuring for organizations looking to improve their security posture. Now,
Speaker 1: what about managing resource usage? We talked about performance, but
Speaker 1: how do you prevent a single misbehaving or poorly coded
Speaker 1: application from consuming excessive resources and potentially impacting the entire
Speaker 1: cluster and all other users.
Speaker 2: That's where resource allocations, specifically quotas come into play.
Speaker 1: Quotas limits.
Speaker 2: Yes, Kafka allows administrators to set quotas on resource consumption
Speaker 2: for clients. Their primary purpose is really to protect the
Speaker 2: cluster from excessive load caused by misconfigured, buggy, or even
Speaker 2: malicious clients. It's generally not intended as a mechanism to
Speaker 2: artificially limit well behaved clients or enforced strict service tiers,
Speaker 2: though some use it that way.
Speaker 1: What kind of resources can you limit?
Speaker 2: The main quota is control produce and consume throughput rates
Speaker 2: measured invites per second. You can set a producer bitter
Speaker 2: quota and a consumer bitter quota. There's also a request
Speaker 2: percentage quota that limits the percentage of CPU time a
Speaker 2: client's request can utilize on the broker's network and IO threads,
Speaker 2: preventing CPU starvation.
Speaker 1: Bandwidth and CPU usage limits. Can you set these limits
Speaker 1: broadly or target specific users or applications.
Speaker 2: You can define quotas at different levels. You can set
Speaker 2: them based on the client dot ID property configured in
Speaker 2: the producer or consumer application. However, client dot ID can
Speaker 2: sometimes be easily spoofed or shared across instances. A more
Speaker 2: robust approach is to set quotas at the authenticated user level,
Speaker 2: based on the principle derived from SASEL or TLS. Authentication.
Speaker 2: User level quotas are generally considered more reliable for enforcement.
Speaker 2: You can also set default quotas per user or per
Speaker 2: client ID.
Speaker 1: What's the best practice for setting these quota values? Should
Speaker 1: you be really strict.
Speaker 2: The general best practice is actually to set quotas quite generously,
Speaker 2: well above the normal expected peak usage for a client,
Speaker 2: monitor the actual usage closely. The idea is to use
Speaker 2: quotas as a safety net or a safeguard to catch
Speaker 2: runaway clients or denial of service attempts, rather than using
Speaker 2: them as a strict bottleneck that clients regularly bump up
Speaker 2: against during normal operation. They're like circuit breakers, not fine grain.
Speaker 1: Traffic shapers use them as guardrails, not speed limits for
Speaker 1: everyday traffic. Makes sense. We've talked a lot about Kofka's internals,
Speaker 1: capabilities management, What about actually running it? What are the
Speaker 1: common deployment models? How do people typically deploy and operate
Speaker 1: KOFCA clusters in the real world.
Speaker 2: You have several options, each with its own set of
Speaker 2: trade offs regarding control, cost and operational effort. The traditional
Speaker 2: approach is running Kafka on your own hardware in your
Speaker 2: own data centers. This gives you maximum control over the
Speaker 2: environment hardware selection and configuration, but it also requires the
Speaker 2: most significant operational expertise for planning, provisioning, automation, monitoring, patching, upgrades.
Speaker 1: Everything, high control, high responsibility exactly.
Speaker 2: A very common variation is running Kafka in virtualized environments
Speaker 2: like VMware OpenStack on top of your own hardware or
Speaker 2: private cloud infrastructure. This adds a layer of virtualization management,
Speaker 2: but follows similar principles. Key considerations here are distributing brokers,
Speaker 2: evenly across physical VM hosts to avoid single points of failure,
Speaker 2: and carefully considering storage performance, preferring local SSDs over shared
Speaker 2: network attached storage SANDS and NASS if latency and throughput
Speaker 2: are critical, as network storage can sometimes introduce unpredictable performance.
Speaker 1: Okay, self managed on prem or private cloud? What about
Speaker 1: public cloud? Kumernetes seems popular these days?
Speaker 4: Right?
Speaker 2: Running Kafka on Kubernetes has become increasingly popular, especially for
Speaker 2: organizations that already have strong operational teams comfortable with managing
Speaker 2: stateful workloads on k ads.
Speaker 1: Isn't running something stateful like Cofka on Kubernetes tricky?
Speaker 2: It can be. Managing storage, networking and upgrades requires care. However,
Speaker 2: the ecosystem has matured significantly. The community standard for this
Speaker 2: is generally considered to be Strimsy Strimsey dot io. It's
Speaker 2: a Kubernetes operator specifically designed for deploying and managing Kofka
Speaker 2: clusters along with components like Coffca, Connect, mirror, Maaker, etc.
Speaker 2: On Kubernetes automate. It's many complex operational tasks like provisioning,
Speaker 2: configuration management, rolling upgrade certificate management, making it much more manageable,
Speaker 2: but it still requires a solid understanding of both Cofka
Speaker 2: and Kubernetes.
Speaker 1: So Strimsey helps bridge the gap for Kubernetes users. What
Speaker 1: about just using a fully managed service.
Speaker 2: That's the other major direction, using public cloud managed services
Speaker 2: platforms like conflent cloud from the original creators of Kafka,
Speaker 2: Amazon MSK Managed Streaming for Kafka, Avan for Apache Kofka
Speaker 2: as your HD on sidez Kafka, Google Cloud pub sub.
Speaker 2: Though different, sometimes used as alternative, they have stracked away most,
Speaker 2: if not all, of the underlying infrastructure.
Speaker 1: Management sounds appealing. What's the catch?
Speaker 2: The appeal is obvious, faster deployment, reduced operational burden, built
Speaker 2: in scalability and reliability features. The catch, or rather the
Speaker 2: trade offs, are typically reduced control over the fine grain configuration, potential,
Speaker 2: vendor lock in, and cost which can sometimes be higher
Speaker 2: than self managing a very large scale. Even with a
Speaker 2: managed service, you still need significant in house Kofka expertise.
Speaker 2: You need to understand Kofka concepts to design your applications correctly,
Speaker 2: choose the right service tier, understand the service's limitations like
Speaker 2: maximum retention periods, throughput cabs, partition limits, troubleshoot application level issues,
Speaker 2: and manage costs effectively. It's not no OPS, it's different OPS.
Speaker 1: Managed service simplifies infrastructure, but not application design or KOFCA knowledge.
Speaker 1: Good point. Finally, regardless of how you deploy it, keeping
Speaker 1: an eye on everything once it's running seems essential. Monitoring
Speaker 1: and alerting must be crucial for a distributed system like
Speaker 1: Kafka that's likely underpinning critical applications absolutely crucial.
Speaker 2: While Kofka is designed to be robust and fault tolerant,
Speaker 2: it's not invulnerable. Things can still go wrong. Brokers can
Speaker 2: run out of disk space, networks can become saturated, consumers
Speaker 2: can fall behind, partitions can become under replicated.
Speaker 1: Why is monitoring so important? What are you trying to achieve?
Speaker 2: The primary goals of monitoring are to detect problems quickly,
Speaker 2: ideally before they cause a major outage or data loss,
Speaker 2: and provide insights needed for troubleshooting and capacity planning. You
Speaker 2: want to prevent a small issue on one broker from
Speaker 2: cascading into a complete cluster failure.
Speaker 1: What are some of the absolute key metrics you should
Speaker 1: be watching the critical vital signs?
Speaker 2: There are many metrics available via JMX, but some key
Speaker 2: ones include Under replicated partitions, this metric, exposed by the
Speaker 2: controller broker, counts the number of partitions that currently don't
Speaker 2: have their full set of InSync replicas. This value should
Speaker 2: ideally always be zero. If it's greater than zero for
Speaker 2: a sustained period, it indicates a problem with replication and
Speaker 2: reduced fault tolerance. That's a top priority alert.
Speaker 1: Underreplicated equals bad got it.
Speaker 2: Active controller count should be exactly one across the entire cluster.
Speaker 2: If it's zero or one, the clusters in a bad state,
Speaker 2: offline petitions count, similar to underreplicated discounts. Partitions whose leader
Speaker 2: is offline should also be zero. Broker level metrics like
Speaker 2: leader count and partition count, These should be relatively balanced
Speaker 2: across all brokers in the cluster. If one broker has
Speaker 2: vastly more leaders or partitions than others, it indicates a
Speaker 2: load imbalance. Replication lag metrics like max lag under Kafka
Speaker 2: dot server replica fectro managers show the maximum lag between
Speaker 2: a leader and its followers. High lag means followers are
Speaker 2: falling behind, increasing risks during failover.
Speaker 1: Lag is important for consumers too write definitely.
Speaker 2: Consumer lag, often calculated externally by monitoring tools by comparing
Speaker 2: the consumer group's committed offset with the latest offset in
Speaker 2: the partition, is critical. It shows how far behind a
Speaker 2: consumer group is in processing messages. High or constantly increasing
Speaker 2: lag indicates the consumers can't keep up. Also, watch basic
Speaker 2: throughput bites in PERSEC bites OP per SC messages in
Speaker 2: per sec at the broker and topic level to understand
Speaker 2: load and detect anomalies. Network and request metrics request Q
Speaker 2: size should be near zero. Request latent zemes P ninety
Speaker 2: nine or P nine ninety nine are useful to spot
Speaker 2: processing bottlenecks and for consumer groups COFFA, dot com consumer
Speaker 2: dot consumer coordinator. Metrics related to rebalances, repellance, latency, max, rebalanced, total,
Speaker 2: frequent or long rebalances can indicate instability in the consumer group, example,
Speaker 2: consumers crashing repeatedly.
Speaker 1: That's a good list of critical signals. Yeah, so, how
Speaker 1: do you approach alerting based on these metrics without getting
Speaker 1: drowned in constant notifications, especially in dynamic environments.
Speaker 2: That's the art of good alerting. First, not every metric
Speaker 2: needs an alert. Focus on metrics that indicate a real
Speaker 2: actionable problem impacting service health or data integrity, like under
Speaker 2: replicated partitions. High consumer lag beyond a threshold brokers being down. Second,
Speaker 2: set meaningful thresholds based on your baseline performance and SLOs.
Speaker 2: Don't alert if consumer lag spikes for one minute during
Speaker 2: a brief load burst if it recovers quickly, alert if
Speaker 2: it stays high for ten minutes. Third, allows systems time
Speaker 2: for self healing where appropriate. If you're running on Kubernetes,
Speaker 2: maybe don't page the on call engineer immediately. If for
Speaker 2: broker pod crashes, Kubernetes might restart it successfully within a minute.
Speaker 2: Alert only if the problem persists, be on the expected
Speaker 2: auto recovery time.
Speaker 1: Build in some tolerance for self heal right.
Speaker 2: And finally, always enrich your alerts with context. An alert
Speaker 2: message should clearly state what is wrong, where, which cluster,
Speaker 2: which topic, group, the severity, and ideally provide links to
Speaker 2: relevant dashboards for investigation or even point to specific troubleshooting
Speaker 2: steps or alert playbooks. An alert for our online retailer
Speaker 2: that just says metric excess high is useless. An alert
Speaker 2: saying critical order processing consumer group lag one million messages
Speaker 2: for fifteen men's on broad COSTCA cluster dashboard, dot link playbook.
Speaker 2: Dot link is much more effective.
Speaker 1: Context makes alerts actionable, not just noise. Excellent advice. We've
Speaker 1: covered a huge amount of ground how Kofka works, how
Speaker 1: it performs, how it integrates, how it's managed and secured.
Speaker 1: Truly a deep dive. But as with any complex distributed system,
Speaker 1: things can still go wrong on a larger scale. Let's
Speaker 1: talk disaster management. How do you prepare for the really
Speaker 1: bad scenarios like an entire data center going offline unexpectedly.
Speaker 2: Disaster management is absolutely critical for business continuity, especially when
Speaker 2: Kafka is underpinning core operations. Kaffa's own architecture provides some
Speaker 2: inherent resilience here. Its asyncreitous nature, for example, helps mitigate
Speaker 2: temporary network partitions or brief compute failures. If a producer
Speaker 2: can't reach the broker, it can often buffer messages and
Speaker 2: retry later. If a consumer gets disconnected, it can usually
Speaker 2: reconnect and resume from where it left off using its
Speaker 2: committed offsets. Data isn't typically lost just because of transient
Speaker 2: connectivity issues.
Speaker 1: Okay, it handles temporary glitches, Well, what about more permanent
Speaker 1: compute failures like a single broker machine dying completely within
Speaker 1: a cluster.
Speaker 2: As we discuss with reliability, Kafka is explicitly designed to
Speaker 2: handle individual compute failures gracefully. Assuming you've configured adequate replication
Speaker 2: with a replication factor of say three and min dot
Speaker 2: insnc dot replica set to two, the cluster can tolerate
Speaker 2: the loss of one broker per partition without any loss
Speaker 2: of data availability for rights all of yours or reads.
Speaker 1: Automatic leader fail over kicks in exactly.
Speaker 2: The cluster heals itself by electing new leaders from the
Speaker 2: remaining in sync replicas. This is a core part of
Speaker 2: Kofka's high availability promise within a single cluster deployment.
Speaker 1: That's good for failures within a deployment, but what about
Speaker 1: the really big one, a full data center failure. If
Speaker 1: your entire Kofka cluster is in one DC and that
Speaker 1: DC goes dark power, loss, network outage, natural disaster, you're offline,
Speaker 1: right you are?
Speaker 2: If the entire cluster lives in that single DC, that's
Speaker 2: a major risk for critical applications. So for true disaster
Speaker 2: recovery across data centers, you need more sophisticated strategies which
Speaker 2: significantly increase complexity and cost.
Speaker 1: What are the main approaches.
Speaker 2: One approach is to build a stretched cluster.
Speaker 1: Stretched.
Speaker 2: Yes, you deploy single logical Kofka cluster, but the brokers
Speaker 2: are physically distributed across multiple data centers or availability zones ASS.
Speaker 2: For example, you might place brokers in three different ass
Speaker 2: within the same geographic region.
Speaker 1: So it operates as one cluster, just geographically spread out.
Speaker 2: Correct. The coordination mechanism craft or zookeeper also needs to
Speaker 2: be stretched across these locations. If one entire data center
Speaker 2: AZ fails, the Kafka cluster can potentially remain operational as
Speaker 2: long as a majority of coordinators remain online and partitions
Speaker 2: still have a live leader in one of the surviving
Speaker 2: locations and enough ISRs meet the men dot InSync dot
Speaker 2: replicas requirement.
Speaker 1: What are the downsides of stretching?
Speaker 2: The main downside is latency. Network latency between data centers
Speaker 2: is typically much higher than within a single DC. This
Speaker 2: increased latency impacts produced request times, especially for AXOL replication
Speaker 2: lag and failover times. It generally requires dcs that are
Speaker 2: relatively close geographically with low latency, high bandwidth links between them,
Speaker 2: and you typically need at least three locations dcs or
Speaker 2: azs to reliably maintain a quorum for the coordination cluster
Speaker 2: if one location fails.
Speaker 1: So stretched clusters offer high availability, become with latency trade offs,
Speaker 1: and require careful network planning. What's the alternative if stretching
Speaker 1: isn't feasible?
Speaker 2: The other common approach is using multiple independent Kofka clusters
Speaker 2: and replicating data between them using tools like Kofka's own
Speaker 2: mirror Maker, specifically mirror Maker two, which is built on
Speaker 2: the Connect framework.
Speaker 1: Mirroring copying data between clusters exactly.
Speaker 2: You continuously copy messages from topics in one cluster to
Speaker 2: topics in another cluster, usually located in a different data
Speaker 2: center or region. You can set various topologies with mirroring,
Speaker 2: like what a common one is? Active passive. You have
Speaker 2: your primary active cluster in one DC where producers send
Speaker 2: data and consumers normally read from mirror Maker then copies
Speaker 2: this data to a passive cluster in a separate dr
Speaker 2: disaster recovery data center.
Speaker 1: What's the passive cluster used for?
Speaker 2: It serves as a hot standby. If the active cluster fails,
Speaker 2: you would need to manually or via automation fail over
Speaker 2: your producers and consumers to start using the passive cluster instead.
Speaker 2: This setup is also often used for data migration between
Speaker 2: clusters or for centralizing data for analysis. Each regional clusters
Speaker 2: mirroring data to a main headquarters.
Speaker 1: Cluster, so failover is typically a manual step in active passive.
Speaker 1: What about active active does that avoid manual failover?
Speaker 2: An active active setup involves having two or more independent clusters,
Speaker 2: both actively serving producer and consumer traffic and configured to
Speaker 2: mirror data to each other. For example, users in Europe
Speaker 2: connect to the EU cluster, uses in the US connect
Speaker 2: to the US cluster, and mirror Maker copies data in
Speaker 2: both directions between them.
Speaker 1: That sounds complex to manage, especially avoiding infinite loops of
Speaker 1: mirroring the same message back and forth.
Speaker 2: It is complex. Mirror Maker two handles the looping issue
Speaker 2: by automatically prefixing mirrored topics with the name of the
Speaker 2: source cluster. For example, data from topic A in the
Speaker 2: USE cluster arrives in the UTH cluster as USE cluster
Speaker 2: dot Topic A. This prevents it from being mirrored back again.
Speaker 2: The bigger challenge with active active is often on the
Speaker 2: application side. You need to ensure your consumer services can
Speaker 2: handle potentially processing the same logical event twice, once from
Speaker 2: the local cluster, once mirrored from the remote cluster, or
Speaker 2: implement logic to de duplicate or process only based on origin.
Speaker 2: Our online retailer would need very careful application designed to avoid, say,
Speaker 2: double charging a customer shipping an order twice. If using
Speaker 2: active active, it provides higher availability and potentially lower latency
Speaker 2: for users connecting to their local cluster, but demands more
Speaker 2: sophisticated application logic.
Speaker 1: Active Active sounds powerful, but requires careful thought about idempotency
Speaker 1: or deduplication and consumers any other mirroring topologies.
Speaker 2: Another useful one is hub and spoke. This is great
Speaker 2: for distributed organizations. Imagine a central hub Kafka cluster at
Speaker 2: headquarters and multiple smaller spoke clusters at regional offices, factories,
Speaker 2: or even retail stores like.
Speaker 1: Our supermarket chain example.
Speaker 2: Exactly, espoke cluster might handle local operations and then mirror
Speaker 2: relevant data like sales summaries up to the central hub
Speaker 2: for company wide aggregation and analysis. The hub might also
Speaker 2: mirror down commands or configuration updates to the spokes. This
Speaker 2: allows spokes to operate somewhat independently. Even if connectivity to
Speaker 2: the hub is intermittent, and then sync up later. Think
Speaker 2: cruise ships needing to operate autonomously at sea and then
Speaker 2: sink data when they reach port.
Speaker 1: Hub and spoke seems well suited for those kinds of
Speaker 1: distributed environments. So stretched clusters or mirroring with various patterns
Speaker 1: offer ways to handle large scale disasters. Kafka is clearly
Speaker 1: incredibly versatile, powerful and resilient when used correctly. But just
Speaker 1: as important as knowing when and how to use it
Speaker 1: is knowing when not to use Kaofka. It's not a
Speaker 1: silver bullet for every data problem, is it. What are
Speaker 1: some crucial limitations or scenarios where kofka might be the
Speaker 1: wrong tool for the job.
Speaker 2: That's a vital point to cover. Kaffa is amazing, but
Speaker 2: applying it inappropriately leads to frustration and poor results. There
Speaker 2: are several key anti patterns. First, and perhaps most importantly,
Speaker 2: Kaffa is not a relational database.
Speaker 1: We touched on this with the log versus state idea.
Speaker 2: Right Coffee excels at storing the history of events what happened,
Speaker 2: It's generally poor at representing and querying the current state
Speaker 2: of complex entities, especially if that requires complex joints across
Speaker 2: multiple normalized tables or point lookups based on arbitrary criteria
Speaker 2: like finding a user by email address when the topic
Speaker 2: is key by user ID. Don't try to replace your
Speaker 2: operational databases postgress, MySQL, et cetera with Kofka for storing
Speaker 2: the canonical queriable state of your application entities. Use databases
Speaker 2: for that. Messages flowing through Kafka should ideally be denormalized,
Speaker 2: meaning a single message contains all the relevant information needed
Speaker 2: for that event, potentially duplicating data found elsewhere, rather than
Speaker 2: requiring complex lookups for joins later, use relational databases for
Speaker 2: your service's internal data persistence and complex querying needs.
Speaker 1: So you wouldn't use kofcas, CASEQL or streams API to
Speaker 1: directly serve a request like find customer excess current shipping address.
Speaker 1: If that address might have changed many times, You'd query
Speaker 1: a proper database that stores the current address exactly.
Speaker 2: You might build that database view using data stream from Kofka.
Speaker 2: But Kafka itself isn't the primary query engine for current
Speaker 2: state lookups in that way.
Speaker 1: Okay, not a database replacement. What else is it not?
Speaker 2: Second Kafka is not a synchronous communication interface or a
Speaker 2: request response system, meaning avoid using KOFKA for interactions where
Speaker 2: a client sends a request and needs an immediate blocking
Speaker 2: response back on the same connection. Think of a web
Speaker 2: browser calling a back end API to fetch user profile
Speaker 2: data that needs a quick HTTP response. Kofka is designed
Speaker 2: for asynchronous data exchange. Producers send messages and typically don't
Speaker 2: wait for or even know about the consumers. Consumers process
Speaker 2: messages at their own pace. Trying to force synchronous request
Speaker 2: response patterns over Kafka e g. Producer sends topic A,
Speaker 2: waits for a consumerative process, and send a apply to
Speaker 2: topic B. Producer reads from topic B is usually complex, brittle,
Speaker 2: and inefficient compared to using standard RPC, rest APIs or
Speaker 2: gRPC for synchronous needs.
Speaker 1: So if your online stores checkout page needs that instant
Speaker 1: payment accepted or payment declined feedback to show the user,
Speaker 1: you'd use a direct API call to the payment service,
Speaker 1: not send a command via Kafka and hope for a
Speaker 1: quick reply message.
Speaker 2: Precisely use the right tool for the job. KOFKA for
Speaker 2: asynchronous event streams APIs for synchronous interactions.
Speaker 1: Makes sense and what about sending large files? You mentioned
Speaker 1: the one mebe default limit earlier right.
Speaker 2: Third, KOFKA is not a file exchange platform. It is
Speaker 2: highly optimized for processing a large volume of relatively small messages,
Speaker 2: typically under that ONEVAB default and often much smaller kilobytes.
Speaker 2: These messages are usually structured machine readable data like JSON
Speaker 2: or AVRO. Trying to send large binary files like multi
Speaker 2: megabyte PDFs, high resolution images, video files directly inside KOFCA
Speaker 2: message values is generally a very bad idea. Why specifically
Speaker 2: it performs poorly It consumes excessive broker disk space, puts
Speaker 2: heavy load on network bandwidth, increases memory pressure on clients
Speaker 2: and brokers, and can lead to processing timeouts and instability.
Speaker 2: Kofka's internal mechanisms aren't designed for efficiently handling huge blobs.
Speaker 1: Like that, So what's the alternative? If you need to
Speaker 1: signal that a large file is ready for processing, The.
Speaker 2: Standard pattern is to store the large file in a
Speaker 2: dedicated object storage system like AWSS three, Google Cloud Storage,
Speaker 2: or an internal file server, and then send a small
Speaker 2: COFFCA message containing metadata about the file, including a reference
Speaker 2: or pointer like the S three URL to its location.
Speaker 2: The consumer reads the small notification message from COFCA and
Speaker 2: then uses the reference to fetch the large file directly
Speaker 2: from the storage system for processing. Keep large payloads out
Speaker 2: of cofka itself.
Speaker 1: Use COFCA for the notification, not the file transfer. Got it.
Speaker 1: Does it always make sense to implement coofka even for
Speaker 1: smaller projects or simpler data needs. Is there a complexity threshold?
Speaker 2: That's a good question. Fourth coofka is used for small
Speaker 2: applications can sometimes be questionable. While kafka brings immense power
Speaker 2: and scalability, it also introduces operational complexity. Setting up, managing, monitoring,
Speaker 2: securing and upgrading a COFFCA cluster, even a small one,
Speaker 2: or even using managed service, which still requires configuration and understanding,
Speaker 2: involves overhead. For very small scale applications or simple point
Speaker 2: to point integration needs, the complexity of introducing kofka might
Speaker 2: outweigh its benefits. Sometimes a simpler solution, like using a
Speaker 2: message queue built into your application framework, relying on database triggers,
Speaker 2: or polling using a lightweight cloud queue service, or even
Speaker 2: just direct API calls might be perfectly sufficient and much
Speaker 2: easier to manage. Don't introduce kafka just because it's popular,
Speaker 2: make sure the problem warrants its capabilities. Don't bring a
Speaker 2: bazooka to a knife fight, so to speak.
Speaker 1: Evaluate the trade offs, don't over engineer makes sense. And finally,
Speaker 1: the source material has this great warning which you alluded
Speaker 1: to earlier. If garbage is produced into Kofka, garbage will
Speaker 1: also come out at the consumer side. What does that
Speaker 1: truly mean in practice and why is it such an
Speaker 1: important closing thought?
Speaker 2: It means that ultimately Kafka is not a substitute for
Speaker 2: good architecture and good data practices upstream. Kaffa itself provides
Speaker 2: incredible freedom and flex dixability and how you produce and
Speaker 2: consume data. It faithfully and reliably transports the bite arrays
Speaker 2: you give it, but it doesn't inherently understand or validate
Speaker 2: the semantic meaning or business correctness of that data beyond
Speaker 2: basic scheme of validation. If you use a schema registry, it.
Speaker 3: Just moves the bytes exactly.
Speaker 2: So, if the data quality being produced into Kafka is poor,
Speaker 2: if producers send malform messages, inconsistent values, incorrect calculations, or
Speaker 2: just plain wrong information, Kafka will diligently deliver that poor
Speaker 2: data to all interested consumers. The consumers will then either
Speaker 2: crash trying to process the garbage, make bad decisions based
Speaker 2: on the garbage, or have to implement complex, defensive logic
Speaker 2: to try and clean up the garbage. This highlights the
Speaker 2: Kafka's power, lized not just in the technology itself, but
Speaker 2: crucially in the thoughtful architecture, the data modeling, the schema governance,
Speaker 2: the validation logic and producers, and the overall data quality
Speaker 2: strategy you build around it. Coffa is a powerful tool,
Speaker 2: but its effectiveness depends entirely on how well you design
Speaker 2: and manage the entire gata ecosystem it enables. You won't
Speaker 2: magically fix up stream data problems.
Speaker 1: Garbage in, garbage out, faithfully delivered at scale. A very
Speaker 1: sobering and important reminder while you have just completed a
Speaker 1: very deep dive into apatche Kofka. We went from its
Speaker 1: fundamental components, messages, topics, partitions, brokers, and its unique nature
Speaker 1: as a.
Speaker 2: Distributed log all the way through to replication acknowledgments, idempotence
Speaker 2: for reliability, and.
Speaker 1: How it achieves incredible performance through batching, compressions, zero copy
Speaker 1: and parallel processing. With partitions.
Speaker 2: We explored its power for integrating systems using KOFCA connect
Speaker 2: and SMTS, and dove into the world of real time
Speaker 2: stream processing with Kofka streams, covering state tables, joins, and
Speaker 2: time concepts.
Speaker 1: And we didn't forget management and security, governance schema registries, authentication, authorization,
Speaker 1: encryption quotas, deployment models, and crucial monitoring and alerting.
Speaker 2: Plus how to handle disasters using stretched clusters or mirroring
Speaker 2: patterns like active passes in Hubbin.
Speaker 1: Spoke and critically, we also covered the scenarios where Kaffka
Speaker 1: might not be the best fit, ensuring you have that
Speaker 1: balanced and practical understanding.
Speaker 2: It's been quite a journey through the Kafka ecosystem.
Speaker 1: Indeed, the key takeaway for you listening is hopefully clear.
Speaker 1: Kofka isn't just another messaging system off the shelf. It's
Speaker 1: a truly foundational technology that, when you strategically and thoughtfully,
Speaker 1: can genuinely transform how your organization handles data, enabling that
Speaker 1: shift towards real time operations and insights.
Speaker 2: That transformation potential is real.
Speaker 1: But as we've seen, Kafka empowers you with incredible flexibility,
Speaker 1: and this brings us back to that final crucial point.
Speaker 1: Remember this, Kaffka doesn't care about the messages it transports.
Speaker 1: It's agnostic. If garbage is produced into Kafka, garbage will
Speaker 1: also come out of the consumer side.
Speaker 2: It faithfully delivers what it's given exactly.
Speaker 1: It's true power lies not just in the impressive technology itself,
Speaker 1: but in the thoughtful architecture, the rigorous data quality practices,
Speaker 1: and the careful you bring to it.
Speaker 2: It really underscores the importance of that end to end
Speaker 2: thinking about your data pipelines.
Speaker 1: So as you reflect on your own data challenges and opportunities,
Speaker 1: what stands out to you from today's deep dive. What
Speaker 1: aspects of Kofka's architecture, its capabilities, or even its limitations
Speaker 1: might you explore further for your own needs or projects.
Speaker 1: Keep asking those questions, keep digging deeper, because in the
Speaker 1: world of data, the learning never truly stops