Apache Kafka® 4.4.0 was released on 22 October with changes that go beyond the brokers. Share groups can send rejected records to a dead-letter queue and Kafka Streams can retain committed local state after errors that previously required a rebuild. Kafka Connect also gets a way to stop workers from quietly creating missing internal topics.
I’m particularly interested in the recovery changes because restarting an application and getting it processing again can be very different things when there’s a large state store to restore. There is enough to explain here without squeezing every feature into one article so I’ve split the release into four parts.
| Article | Coverage |
|---|---|
| Part 1 | Broker metrics and storage changes with upgrade checks |
| Part 2 | Share-group dead-letter queues and what happens to failed records |
| Part 3 | Transactional Streams state stores and recovery |
| Part 4 | Explicit creation of Kafka Connect’s internal topics |
We’re excited to try these improvements at AxonOps and grateful to everyone who has put time into developing and testing them.
Broker Monitoring
Kafka already exposed the bytes read from and written to storage on Linux. KIP-1291 by Sahil Devgon adds the remaining fields from /proc/self/io so you can also see the I/O requested by the Kafka process and the number of system calls it makes.
| Metric | Unit | What it counts |
|---|---|---|
linux-disk-read-bytes | Bytes | Reads attributed to storage I/O |
linux-disk-write-bytes | Bytes | Writes attributed to storage I/O |
linux-disk-rchar | Bytes | Data returned by read-related calls including reads served without storage I/O |
linux-disk-wchar | Bytes | Data passed through write-related calls |
linux-disk-syscr | Calls | Read-related system calls |
linux-disk-syscw | Calls | Write-related system calls |
linux-disk-cancelled-write-bytes | Bytes | Writes cancelled before reaching storage |
The first two already existed and the other five are new. These are cumulative process-level measurements rather than per-topic or per-disk measurements so use their change over time when comparing them with request latency and replication traffic.
For example you might see read calls increasing while storage reads stay low because much of the requested data is already cached. I wouldn’t turn that comparison into a precise Kafka page-cache hit percentage though. Linux’s process I/O accounting includes activity beyond reads from Kafka log files and the counters don’t all measure the same operations.
Our Kafka monitoring metrics guide explains the request and replication measurements to compare alongside them. Verify the new names in your collection configuration before depending on them in an alert.
Group Coordinator Metrics
If a dashboard suddenly shows no groups after a future upgrade you don’t want to discover that it was still reading a deprecated metric. Nick Guo’s KIP-1301 completes more of the move from Yammer metrics to Kafka Metrics while leaving the old measurements available during the deprecation period.
| Existing measurement | Kafka Metrics replacement |
|---|---|
NumGroups | group-count with protocol=classic |
NumOffsets | offset-count |
NumGroupsPreparingRebalance | classic-group-count with state=PreparingRebalance |
NumGroupsCompletingRebalance | classic-group-count with state=CompletingRebalance |
NumGroupsStable | classic-group-count with state=Stable |
NumGroupsDead | classic-group-count with state=Dead |
NumGroupsEmpty | classic-group-count with state=Empty |
The replacements belong to the kafka.server group-coordinator metrics rather than kafka.coordinator.group. The 4.4.0 implementation marks the old metrics for removal in Kafka 5.0. Compare old and new values while both are present and check the exported labels as well as the metric names.
Tiered Storage Uploads
With tiered storage a closed log segment can exist both on a broker’s disk and in remote storage during the local retention period. KIP-1241 adds a way to postpone that remote copy when you don’t need it immediately.
| Topic setting | Default | Purpose |
|---|---|---|
remote.copy.lag.ms | 0 | Time-based upload eligibility |
remote.copy.lag.bytes | -1 | Size-based eligibility measured using newer local log data |
The default time setting keeps immediate eligibility. A non-active segment becomes eligible when either the applicable time check or the size check is satisfied. Setting either to 0 allows immediate upload. A value of -1 derives that check from the corresponding local retention setting and an unlimited local retention value disables that check.
Delaying uploads can reduce the time you pay to keep the same data remotely while it remains local. It also means remote readers wait longer for those segments and an upload outage can leave more data needing space locally. Review both retention limits before changing this because a shorter remote-storage bill doesn’t remove the need for local disk headroom. The option changes upload timing rather than Kafka’s replication factor.
Storage Formatting
KIP-1170 by José Armando García Sancio brings bootstrap metadata into the KRaft checkpoint mechanism. One practical consequence is easy to miss when a deployment system formats disks with one container image and starts Kafka with another.
A Kafka node must be at least as new as the storage tool used to format its directory. Choosing an older --release-version with the 4.4 tool doesn’t make that directory suitable for starting an older Kafka executable.
Keep the formatting tool and the Kafka installation together in your provisioning process. This concerns formatting new storage and does not mean existing production directories should be reformatted during an upgrade.
Controller Registrations
A retired controller can leave a registration in cluster metadata that still advertises its supported features. That stale registration can then block a feature upgrade even though the machine is no longer participating.
Kevin Wu’s KIP-1312 adds kafka-cluster.sh unregister-controller and a --unregister option to kafka-metadata-quorum.sh remove-controller. The operation requires metadata.version=4.4-IV2.
Removing a controller’s registration and removing a voting member from the quorum are different operations. Check the current voter set and the intended replacement before using either command so cleanup of an old registration doesn’t become an attempt to remove a controller that is still needed.
Client and Command Changes
There are a few smaller changes worth checking in application and administration scripts.
| Change | What to check |
|---|---|
broker.id is deprecated | Use node.id in broker configuration |
kafka-cluster.sh api-versions replaces the deprecated kafka-broker-api-versions.sh | Update scripts and choose --bootstrap-server or --bootstrap-controller for the target |
bootstrap.resolve.timeout.ms adds asynchronous bootstrap DNS resolution | Default 0 retains synchronous behaviour. A positive value changes when failures reach the application. Streams and Connect do not yet support this mode |
| Producer performance tests gain key-distribution controls | Use --record-key-range and --key-distribution when testing keyed traffic rather than assuming an unkeyed test represents the workload |
Before Upgrading
I’d start by recording the current workload and recovery behaviour so there’s something useful to compare after the upgrade. The Kafka monitoring guide covers the broker and application checks.
- Record produce and fetch latency alongside consumer lag and broker resource use under comparable traffic.
- Check the supported upgrade path from your current release and test it with your client and connector versions.
- Check provisioning scripts for mismatched storage-tool versions and dashboards for deprecated coordinator metrics.
- Complete the rolling upgrade and verify the cluster before enabling new feature levels. Treat changes to
metadata.versionandshare.versionas separate decisions with their own rollback constraints. - Test the failure scenarios that affect your applications rather than judging the upgrade only by steady-state throughput.
Kafka 4.4.0 doesn’t establish a general throughput or latency improvement for every workload. The more interesting comparison for a stateful application may be how long it takes to resume processing after failure. Part 3 explains that change while Part 2 follows a failed share-group record through the new dead-letter queue support.