Cassandra

Apache Cassandra 6.0 Part 3: Performance Optimisations

Hayato Shimizu · June 16, 2026 · Updated September 7, 2026 · 14 min read

Performance Optimisations

I started using Cassandra in 2008, when it was widely regarded as one of the strongest choices for sustained write throughput and horizontal scale. Its architecture was a good fit for workloads that needed to keep accepting writes while nodes were added or lost, and Cassandra gained a strong performance reputation in that area.

Over the following years, other databases raised the performance bar with different storage-engine, scheduling, and implementation choices. ScyllaDB has been one of the most prominent competitors in that conversation. Benchmark results need a representative workload because hardware, schema design, read/write mix, consistency level, data shape, and operational limits all affect a result.

Cassandra 5.0 and the work coming in the 6.0 line narrow some of the gaps that developed over that period. The 6.0 items are individual engineering changes across allocation, hot paths, compaction behaviour, flushing, and reads. They have a direct effect on long-running clusters where GC activity, compaction pressure, and tail latency are part of normal operations.

Cassandra 6.0 is pre-release software. This post reflects 6.0-alpha3 and must be checked again against the release notes and upgrade documentation for the exact version being deployed.

Cassandra 5.0 and 6.0

Cassandra 5.0 was already a substantial storage-engine release. It added direct I/O support for the commit log through Java native APIs, a trie-based memtable implementation, the trie-indexed bti SSTable format, and Unified Compaction Strategy. Those changes affected durability I/O, in-memory data structures, point lookups, and compaction scheduling. Cassandra 6.0 builds on that base by removing work from paths that remain busy after a memtable is frozen, an SSTable is read during compaction, or a coordinator serializes a read response.

Area	Cassandra 5.0 baseline	Cassandra 6.0 work
Durable write I/O	Direct I/O became available for commit log files.	Direct I/O reaches compaction reads for compressed SSTables, where large sequential scans can otherwise displace query data from the page cache.
Memtables and flushing	TrieMemtable improved memory use, GC efficiency, and lookup performance while data is in memory.	The flush path is specialized to remove allocations, repeated checks, mapping work, and megamorphic calls as the frozen memtable becomes an SSTable.
SSTable lookups and compaction	The `bti` format and Unified Compaction Strategy gave 5.0 new storage and compaction options.	Read, protocol, metadata, and response-serialization paths receive allocation and CPU reductions around those storage operations.
JVM operational noise	GC work and its logging remain visible under sustained allocation.	Async GC logging is enabled on JDKs that support it so log-file I/O is less likely to stall application threads.

This is not a claim that a Cassandra 6.0 cluster will match every ScyllaDB result. It is a set of changes that addresses specific sources of CPU, allocation, and page-cache pressure which are visible in Cassandra clusters over time.

Important examples from the 6.0 branch include the following.

direct I/O support for compaction reads;
memtable flush logic optimization;
reduced allocations in the read path, including read-path allocation reductions;
avoiding ByteBuffer wrapping in selector input rows through CASSANDRA-21362;
cheaper calculation of BTreeRow.minDeletionTime through CASSANDRA-21414;
protocol and metadata allocation improvements such as CASSANDRA-21199;
metric optimizations using thread-local counters and timers;
async GC logging where supported to reduce hiccups from GC log I/O.

Direct I/O for compaction reads

The important comparison with 5.0 is that direct I/O was already available for the commit log, which is the durability path for incoming mutations. Compaction reads were different. Cassandra read the SSTable components needed for a compaction through the operating system page cache. That is a reasonable default for ordinary reads, where a recently read SSTable page may be useful again. It is a poor fit for a large sequential compaction scan over data that is about to be merged and replaced.

In the buffered 5.0 path, the kernel admits pages read by the compaction into the page cache. If the input SSTables are larger than available cache, those sequential reads can evict pages that would otherwise serve live queries. Cassandra then has to read the live data again, while the kernel is also reclaiming pages and writing dirty compaction output. The effect is often most obvious when a large STCS compaction overlaps a read workload with a hot set that would normally fit in memory, although every compaction strategy can create the same pressure.

CASSANDRA-19987 adds a direct path for compaction reads from compressed SSTables. Direct I/O bypasses the page cache for the selected I/O, so the compaction input no longer competes for cache residency with query data. Uncompressed SSTables continue to use buffered reads. Cassandra still performs the merge, decompression, reconciliation, and output work; the change is specifically about how the input SSTable bytes reach the process. Direct I/O also has alignment and buffering requirements, which is why this work followed Cassandra’s internal buffering support rather than being a single flag added to the old reader.

The 6.0 setting is deliberately narrow.

# cassandra.yaml
# auto inherits from disk_access_mode
# direct bypasses the OS page cache for compaction reads
compaction_read_disk_access_mode: auto

Setting compaction_read_disk_access_mode to direct does not turn every client read into direct I/O and it does not remove normal disk contention. The storage device still has to serve compaction and query requests. What changes is that compaction stops filling the page cache with a sequential working set that is unlikely to be reused.

I am grateful to the people doing this work in the open. CASSANDRA-19987 was reported by Jon Haddad and assigned to Sam Lightfoot. The Apache Cassandra PR describes the patch as by Sam Lightfoot, reviewed by Ariel Weisberg and Maxwell Guo. Sam also published a detailed write-up, Direct I/O for Cassandra Compaction: Cutting p99 Read Latency by 5x, showing the page-cache problem and benchmark results from his test setup.

I would still benchmark this on the storage and workload I actually run. The relevant comparison is buffered compaction reads versus direct compaction reads with the same schema, SSTable shape, compaction concurrency, device, and application traffic. Capture p95 and p99 query latency, cache hit and major-fault behaviour, device queue depth, disk latency, compaction throughput, and the time taken for the cache to recover after compaction. The direct path can protect query latency while reducing the throughput available to the compaction scan on some devices, so the useful result is the cluster-level trade-off rather than one isolated I/O number.

CASSANDRA-21134 extends the same direct-I/O work to compressed SSTable writes from background operations such as compaction, streaming, cleanup, repair, and upgrades. It does not apply to memtable flushes, which remain buffered because recently flushed data can benefit from the page cache. Part 7 covers the configuration and storage test plan for that write path.

Memtable flush optimization

The Cassandra 5.0 TrieMemtable work improved the way Cassandra holds data before a flush, including memory use, garbage-collection efficiency, and lookups. That does not make the transition from a frozen memtable to an SSTable free. A flush walks partitions, rows, and cells, serializes them, updates SSTable metadata, builds indexes, and writes the result to disk. On a write-heavy node, that work runs continuously enough to determine whether memtable memory is released before the next round of writes needs it.

Flush speed therefore bounds sustainable write throughput in a practical way. If flushing cannot release memtable memory at the rate the application fills it, Cassandra applies backpressure or moves towards memory pressure. Flush threads also compete with mutation processing threads for CPU, which means a hot flush path can reduce write throughput before disk bandwidth is fully used.

CASSANDRA-21083, reported and implemented by Dmitry Konstantinov, works through that path at a much lower level than a new flush scheduler or a new compaction strategy. The changes include the following.

Flush-path work	Why it changes CPU or allocation cost
Update `MetadataCollector` clustering values only for the first and last clustering key in a partition.	The SSTable metadata needs partition bounds, not the same update for every row written in the partition.
Split `Cell.Serializer` and `MetadataCollector.update(Cell)` call sites.	A call site that sees many different concrete cell types becomes megamorphic, which prevents the JIT from making the inlining decisions available to a monomorphic or lower-polymorphism call site.
Precalculate counter-column information and move guardrail checks outside the per-row loop.	The common flush path avoids repeated type checks, guardrail lookups, and hidden boxing for logging parameters.
Return early from row and deletion checks when no complex deletion or tombstone work is present.	A normal live row does not pay for iteration or metadata checks required only by less common deletion cases.
Reduce `NativeClustering` serialization allocation and avoid remapping columns when the mapping has not changed.	Serialization performs less object creation and avoids rebuilding data that is already in the expected form.
Use a flush iterator without column filtering.	A flush writes the full frozen memtable, so it does not need the general filtering machinery used by other readers.

The emphasis on megamorphic calls is worth spelling out. Cell operations can potentially run millions of times per second during a busy flush. When a JVM call site sees several unrelated concrete implementations, it cannot reliably inline one fast path. Splitting the call site gives the JIT a more stable type profile. The individual saving may be small, although it repeats for every cell written by every flush thread.

Dmitry’s post on the work credits Branimir Lambov for review. The ticket, profiles, benchmarks, code review, and regression testing are the work that turns a collection of micro-optimizations into a change that can be used by operators. That effort is as important as the code that appears in the release.

I would expect this to show up under sustained write load rather than in a short benchmark with clean caches and no pressure. Compare Cassandra 5.0 and 6.0 with the same memtable implementation, flush writer count, compression configuration, data model, and device. Measure flush duration and bytes per second, write throughput, write latency, allocator stalls, pending flushes and compactions, heap allocation rate, CPU time in flush threads, and GC activity. A higher peak write result is not enough if it is followed by compaction backlog or an unstable latency profile.

Read-path allocation and CPU work

The same 6.0 changelog contains a group of smaller changes in the read and response paths. Cassandra 5.0 has the trie memtable and bti SSTable options, but a read still crosses coordinator planning, replica reads, row merging, selection, result serialization, protocol handling, and native transport. Allocations in any of those stages become allocation rate in the JVM, and allocation rate becomes young-generation collection work even when retained heap is stable.

The 6.0 changes target several specific pieces of that path.

Ticket	Code-path change	Effect to test
CASSANDRA-21199	Allocation improvements in `ProtocolVersion`, `StorageProxy`, and `MerkleTree`.	Lower per-request and metadata housekeeping allocation under coordinator and repair-related activity.
CASSANDRA-21360	Removes allocations from miscellaneous read-path locations.	Lower allocation rate under the query patterns that exercise those paths.
CASSANDRA-21362	Avoids wrapping `ByteBuffer` values in `cql3.selection.Selector.InputRow`.	Fewer short-lived wrapper objects while processing selected values.
CASSANDRA-21414	Reduces the cost of calculating `BTreeRow.minDeletionTime`.	Less CPU spent on row metadata that is examined repeatedly by read and storage paths.
CASSANDRA-21285	Uses a lightweight moving average to size `LocalDataResponse` output buffers before row serialization.	Fewer buffer resizes, allocations, and array copies for responses that exceed the old initial buffer size.

The response-buffer change is a useful example of why allocation work should not be dismissed as cosmetic. Starting an output buffer with a small fixed size means a larger response repeatedly allocates a new array, copies the previous content, and doubles again until it fits. Cassandra 6.0 keeps a lightweight estimate of recent response sizes so the common response can start closer to its final size. This does not change the bytes sent to the client or the CQL result; it removes the intermediate buffers created while producing it.

Compare 5.0 and 6.0 with result sizes that resemble the application, including paging, wide rows, selected collections, and the consistency levels used in production. Java Flight Recorder allocation profiles are useful here, especially for Native-Transport-Requests and coordinator activity. Pair allocation rate with p95 and p99 latency, CPU, request throughput, and GC pause or concurrent-GC CPU time. A reduction in allocated bytes is only operationally useful when it reduces the work that competes with the request path.

GC logging

CASSANDRA-21372 enables asynchronous GC logging on JDK versions that support it. This does not make the garbage collector itself faster and it does not reduce the allocation rate. It removes one narrower source of disturbance when a GC event waits on file I/O while writing its log. That distinction is important when comparing versions. Treat it as a way to reduce logging-related stalls, then verify the actual GC behaviour separately through allocation rate, pause time, and concurrent collector CPU.

Contributors

The direct I/O work credits Jon Haddad for the issue, Sam Lightfoot for the implementation, and Ariel Weisberg and Maxwell Guo for review. The flush work was reported and implemented by Dmitry Konstantinov, with Branimir Lambov credited for review. C. Scott Andreas authored the LocalDataResponse allocation work, Caleb Rackliffe is also credited as a co-author, and Caleb Rackliffe and Dmitry Konstantinov reviewed it.

Those names cover only the tickets where the attribution is explicit. The 6.0 performance work also depends on people who review patches, write benchmarks, run CI, maintain test infrastructure, resolve regressions, prepare releases, document behaviour, and support the users who find problems in real clusters. I am grateful for that wider contribution to Cassandra.

Cassandra 5.0 and 6.0 test plan

Use the same hardware, JDK, heap, schema, replication settings, compaction strategy, data volume, and client workload for both versions. A version comparison with changed SSTable formats, a different compaction configuration, or a warm cache on one run and a cold cache on another does not isolate the code changes described above.

Test	Cassandra 5.0 baseline	Cassandra 6.0 comparison	Measurements
Compaction with a live hot read set	Buffered compaction reads, using the normal page cache path.	Repeat with `compaction_read_disk_access_mode: direct`.	p95/p99 read latency, major faults, page-cache activity, device queue depth, read/write latency, compaction throughput, and time to recover the hot set.
Sustained write and flush	Same memtable implementation and `memtable_flush_writers` setting.	Repeat after 6.0 flush-path changes.	Write throughput and latency, allocator stalls, flush duration, flush CPU time, allocation rate, GC activity, pending flushes, and compaction backlog.
Read response serialization	A result size and paging profile that represents the application.	Repeat with 6.0 response-buffer and selection allocation changes.	Native transport allocation rate, buffer-copy allocation, coordinator CPU, p95/p99 latency, response size, and throughput.
Mixed workload	Reads, writes, repairs, and compaction at production-like concurrency.	Repeat with the same traffic while enabling only the 6.0 setting under evaluation.	Coordinator and replica latency separately, read amplification, SSTables per read, tombstones scanned, disk saturation, GC pauses, and concurrent-GC CPU.

An ordinary JVM dashboard can show a change in heap or pause time, but it cannot establish whether table-level read amplification, tombstone pressure, an overloaded compaction strategy, or a page-cache eviction event is causing the latency that users see. Capture Cassandra metrics, operating-system I/O data, JFR or another allocation profile, logs, and the exact configuration with every run.

Series

This post is part of the Cassandra 6.0 series.

Demo Sandbox

Thank You from AxonOps: Current London 2026 Recap