Kafka Consumer Rebalancing with KIP-848

Adding another Apache Kafka^® consumer should help an application keep up with its incoming records but the group also has to agree which partitions each instance will read. That reassignment can interrupt processing during deployments and failures even when the brokers have plenty of capacity.

David Jacot, Guozhang Wang and Jason Gustafson proposed KIP-848 to replace the group-wide coordination behind those rebalances with a protocol managed by the broker. Its server-side consumer assignment became generally available in Kafka 4.0 after several years of implementation and testing. We’re excited to try this with larger consumer groups and grateful to everyone who has worked on its implementation and testing.

I think KIP-848 is worth adopting for new consumer applications once the brokers and client library support it. A partition handoff no longer needs every member of the group to finish the same coordination round so a slow consumer whose partitions are unchanged need not delay that move. I would prioritise it for existing applications where deployments or changes in consumer numbers regularly cause rebalance delays.

For a small group that already rebalances without trouble I would plan the migration alongside a normal upgrade. Our local test below produced similar recovery times for cooperative rebalancing and KIP-848 so it would be wrong to promise faster processing just from changing the protocol. The consumer giving up a partition still has to release it and a crashed process still has to be detected.

Let’s follow a partition through a normal handoff and a consumer failure before looking at the settings needed to use this protocol.

What a Consumer Rebalance Changes

We’ll use a payments topic with six partitions and an application called payment-workers. Consumers A, B and C each read two partitions as members of the same consumer group.

Adding Consumer D gives Kafka another place to assign work. In the example below it moves partition P1 from A to D while B and C keep their existing partitions. The exact assignment depends on the assignor and group membership so this is one possible allocation rather than a promise that Kafka will always move P1.

No log segments move between brokers during this operation because consumer rebalancing changes who reads the data. Our Kafka cluster balancing article covers the separate process of moving partition replicas between brokers.

KIP-848 also keeps the consumer-group rule that each partition is assigned to one member at a time. A seventh consumer cannot take half of P1’s records while another member owns it. KIP-932 share groups provide a different model for independent jobs that can be processed out of order.

Eager, Cooperative and KIP-848

With eager rebalancing all consumers release their partitions when the group rebalances so all six partitions would be revoked in our example. Cooperative rebalancing already avoids revoking every partition by releasing only the ones that need a new owner.

Both cooperative and KIP-848 release only P1 here but they organise that handoff differently. Let’s follow classic cooperative rebalancing first with a separate picture for each step. A to D are consumer applications and the small P0 to P5 boxes show the partitions they own.

1. All Consumers Join the First Round

When D joins the group all four consumers take part in a JoinGroup and SyncGroup round through the broker. One consumer acts as the group leader and calculates the assignment. The result asks A to release P1 while D waits for that release.

2. Consumer A Releases P1

A completes its revocation callback and stops using P1 while keeping P0. B and C retain their existing partitions but D cannot start reading P1 until another coordination round gives it the assignment.

3. All Consumers Join the Second Round

All four consumers take part in another JoinGroup and SyncGroup round so the leader can assign the released P1 to D. B and C have to participate even though neither has a partition moving to another consumer.

A consumer that is slow to rejoin can delay this process even when its own assignment is unchanged. Keeping a partition assigned therefore does not remove the need to coordinate with the rest of the group.

Our consumer-group rebalancing guide covers the protocol history and assignment strategies. The coordinator here is a Kafka broker responsible for the group rather than the KRaft controller.

The Same Partition Move with KIP-848

Start again with A owning P0 and P1 before D joins. This time the broker coordinator calculates the new assignment and exchanges updates through ConsumerGroupHeartbeat requests and responses. The grey dashed arrows show continuing heartbeat exchanges while the coloured arrow highlights the message for that step.

1. Consumer D Joins the Group

D contacts the coordinator with its first ConsumerGroupHeartbeat request to join the group. This triggers the broker to calculate an assignment that includes D. In our example P1 is selected to move from A to D but D cannot read it while A still owns it.

2. The Broker Asks A to Release P1

The coordinator’s response to A’s heartbeat asks it to keep P0 and release P1. D still has to wait because A has not yet finished the revocation.

3. A Confirms It Has Released P1

A completes its revocation callback and reports that it now owns only P0 in its next heartbeat. An application using manual commits should commit the next offset after the work it has completed before releasing P1.

4. D Receives P1

Once the coordinator has recorded A’s release it can return P1 in D’s heartbeat response. D can then resume reading from the group’s committed offset without waiting for B and C to join a shared round.

The partition move is the same in both protocols. KIP-848 removes the shared coordination rounds so B and C do not hold up a handoff they are not involved in. D still depends on A releasing P1 and a slow revocation callback on A can delay that release.

If A has successfully processed through offset 1042 and committed 1043 then D resumes at 1043. If A processed some records without committing their offsets then D can receive those records again. Changing the rebalance protocol does not make an external database write or payment request exactly once.

The coordinator tracks versions of group and member state through epochs so it can distinguish an old assignment from the current one. Its changes are recorded in __consumer_offsets rather than existing only in the leader consumer’s memory. The released coordinator implementation handles those state changes and the Java consumer membership implementation runs the assignment callbacks before acknowledging the reconciled assignment.

Those callbacks still run on the application’s consumer thread. A slow callback can delay that consumer’s processing even though the other group members no longer have to wait at a group-wide barrier. Keeping work bounded inside callbacks is therefore just as important after migration.

When a Consumer Fails

For the failure case we’ll start again with A, B and C so A still owns P0 and P1. What happens if A crashes without releasing either partition? The coordinator cannot assume that a missing heartbeat means the process has stopped because a network interruption can make a live consumer temporarily unreachable.

It waits for the applicable timeout before removing that member and making its partitions available to the survivors. B and C can keep their own assignments while the group waits for A’s failure to be detected. A graceful shutdown can hand work back sooner because the consumer can leave the group rather than disappearing without notice.

After a network interruption an old process might still be executing an external request even after Kafka has removed it from the group. Kafka’s membership checks cannot cancel that request so the application still needs idempotency or fencing where duplicate side effects would be harmful.

Shortening the session timeout reduces failure-detection delay but also makes pauses and brief connection problems more likely to remove healthy members. That trade-off needs to be tested with the application’s actual processing and network behaviour.

Local Rebalance Tests

We compared the three approaches with Kafka 4.1.2 and the matching Java clients in an isolated local container. A producer continuously wrote to six partitions while three consumer processes read them. We then started a fourth process and later killed Consumer A without a graceful shutdown.

The classic runs used StickyAssignor and CooperativeStickyAssignor respectively while KIP-848 used the broker’s uniform assignor. All runs used six-second session timeouts and one-second heartbeats to keep the failure check short. These are lab settings rather than production recommendations.

The results below describe callback revocations and application reads rather than a production throughput benchmark. The broker and clients shared one machine and the runs used different assignor implementations.

Both cooperative rebalancing and KIP-848 avoided revoking the survivors’ existing partitions during this failure check. Neither bypassed the timeout needed to detect A’s disappearance. The read-gap chart measures the longest interval between logged application reads on either of A’s partitions across the failure and recovery.

On this small group the eager handoff was also quick despite revoking all six partitions. These results don’t establish a universal latency improvement from KIP-848 because the application had almost no work to finish and no slow member holding up the classic rebalance. A useful production comparison needs the real callbacks and processing delays as well as a representative group size.

Enabling the Consumer Protocol

For a compatible Java consumer the application continues using KafkaConsumer and subscribe() with the same processing loop. These are the protocol-specific settings used in our test alongside the application’s existing connection and deserializer configuration.

group.id=payment-workers
group.protocol=consumer
group.remote.assignor=uniform

group.remote.assignor selects an assignor offered by the broker. Omitting it uses the coordinator’s configured default rather than the old client-side partition.assignment.strategy.

Before changing an existing application check where each of its settings now belongs.

The Kafka 4.1.2 Java configuration checks reject explicitly supplied classic assignor and heartbeat/session settings with group.protocol=consumer. Leaving them in a shared properties file can therefore prevent the consumer from starting rather than simply leaving those values unused.

The broker configuration supplies the consumer protocol’s heartbeat and session defaults. Kafka 4.1.2 also supports group-level overrides named consumer.heartbeat.interval.ms and consumer.session.timeout.ms so check the effective group configuration before tuning the broker defaults. Those overrides belong to the group rather than the application’s properties file. Our consumer configuration guide covers the surrounding application settings.

Migrating an Existing Group

Keep the same group.id when the intention is to preserve the group’s committed offsets. A new group ID creates independent progress tracking and can replay retained data or start at the end depending on its reset policy.

1. Check broker and client support. The protocol is generally available from Kafka 4.0 but each client library must implement it. Upgrading brokers alone does not switch a classic consumer application to the new protocol.
2. Review the assignor and subscriptions. Check custom assignment logic and regex subscriptions against the chosen client release instead of assuming every classic configuration transfers unchanged.
3. Choose an online or stopped-group migration. Rolling migration requires compatible classic members and an upgrade-enabled group.consumer.migration.policy. The conversion implementation also rejects custom assignor data it cannot preserve. A stopped-group migration makes the group empty before restarting it with the new protocol and deliberately includes a processing interruption.
4. Test the change with representative processing. Include consumer startup and shutdown as well as a killed process. Verify commits and duplicate handling around the moved partitions.
5. Roll out with a rollback plan. Monitor assignment changes and lag while members are replaced. Returning a non-empty group to classic mode also depends on the migration policy and compatible clients.

Kafka Streams has its own work under KIP-1071 and this is not a replacement for Kafka Connect’s worker-group protocol. Restrict this migration procedure to the ordinary consumer applications it is intended for.

Monitoring the Change with AxonOps

For a rollout I’d compare the same consumer group’s lag before and during the deployment in AxonOps and check whether the affected partitions resume making progress after each instance is replaced. Broker load and errors help establish whether the application is waiting for reassignment or struggling to fetch and process its records.

Pair those observations with the consumer application’s assignment callbacks and processing timings. A broker-side lag chart cannot establish how long an external payment service took to complete a request or prove that the request succeeded exactly once. Our consumer lag article explains the differences between fetched records and committed progress.

KIP-848 is useful when group coordination is contributing to the interruption. If one partition receives more work than its consumer can process then changing the protocol will not divide that partition between several consumers or make the handler faster. Testing the handoff alongside the application’s processing gives us a way to establish which problem we’re actually trying to solve.