Cassandra

Apache Cassandra 6.0 Part 7: Cursor Compaction and SSTable Writes

Hayato Shimizu · July 31, 2026 · Updated September 7, 2026 · 16 min read

Cursor Compaction and SSTable Writes

Part 3 covered direct I/O reads during compaction and the flush-path work in Cassandra 6.0. This post focuses on the compaction implementation itself. It reads several SSTables, reconciles their partitions and cells, purges data when it is safe to do so, and writes replacement SSTables.

Cassandra 6.0 is pre-release software. This post reflects 6.0-alpha3 and must be checked again against the release notes and upgrade documentation for the exact version being deployed.

CASSANDRA-20918 adds a cursor-based compaction implementation that avoids creating a normal Java object for every partition, row, clustering value, tombstone, and cell encountered during a merge. The same development line includes direct I/O support for compressed background SSTable writes through CASSANDRA-21134, cursor support for the compaction read disk-access mode in CASSANDRA-21147, and more useful compaction history in CASSANDRA-20081.

The potential benefit is significant for a node that is continually compacting, although it is not an instruction to enable every new option immediately. Cursor compaction remains experimental. The standard cassandra.yaml leaves it disabled, while cassandra_latest.yaml enables it. The direct-I/O write mode changes how the kernel handles background writes, while trickle fsync changes when buffered data is forced out. Each deserves a test against the actual table shapes, devices, and service-level objectives of the cluster.

Change	Potential benefit	Condition to verify before using it
Cursor compaction	Much lower compaction allocation rate and less CPU spent constructing short-lived row and cell objects.	The table, source SSTable format, partitioner, schema, and compaction operation are supported by the cursor path.
Direct compaction reads	Large sequential reads do not displace hot query pages from the operating-system page cache.	Query latency, device queueing, and compaction throughput improve together on the target storage.
Direct background writes	Compressed SSTable output from maintenance work avoids filling the page cache.	The lower cache pressure is worth the direct-I/O buffering and device behaviour on that workload.
Trickle fsync	Buffered output is synchronized in bounded intervals instead of accumulating into a larger writeback burst.	The chosen interval protects read latency without reducing maintenance throughput too far.

Cassandra 5.0 and 6.0

Cassandra 5.0 was already a significant storage-engine release. TrieMemtable changed how in-memory data is represented, the trie-indexed bti SSTable format provided a new on-disk option, and Unified Compaction Strategy gave operators more control over the balance between write amplification, read amplification, and space usage. Those features affect which SSTables are produced and when they are selected for maintenance.

The ordinary compaction implementation still had to turn the contents of the selected SSTables into Cassandra’s normal in-memory representations as it merged them. For a large compaction this means repeated allocation of objects representing partition keys, clustering values, liveness information, deletion times, rows, and cells. Most of those objects are only useful for the short period between decoding an input entry and serializing its replacement. Their allocation adds CPU work and increases young-generation collection activity even when the heap does not retain much more data.

Cassandra 6.0 addresses a different layer of that same operation. Cursor compaction reads the SSTable byte stream through reusable descriptors, merges those descriptors, and writes the result without first building the usual per-entry object graph. It is not a different compaction strategy and it does not change reconciliation rules, tombstone safety, or the selected set of SSTables. It changes the execution path used after a strategy has selected the work.

Area	Cassandra 5.0 baseline	Cassandra 6.0 work
In-memory data before flush	TrieMemtable reduces memory and garbage-collection costs while mutations are resident.	The flush path receives specialized serialization and metadata work, covered in Part 3.
Compaction selection	Unified Compaction Strategy and the established strategies decide which SSTables need maintenance.	The same strategy can use a cursor merge path when the table and operation meet its eligibility checks.
Merge execution	The iterator path materializes normal Cassandra row and cell structures while it reconciles input SSTables.	Reusable cursor descriptors parse and merge the SSTable stream with far less short-lived allocation.
Compaction I/O	Buffered reads and writes use the page cache unless another disk mode is selected.	Direct read mode is available to cursor compaction; compressed background writes can also use direct I/O.

The cursor work should not be read as a general statement that one version will outperform another for every workload, or that an enabled path always runs for every table. It is a carefully bounded optimization for a particularly expensive part of storage maintenance.

Cursor compaction mechanics

The normal compaction job has not changed in principle. Cassandra takes overlapping SSTables that are already sorted by partition key and clustering order, produces the latest live version of each logical item, retains tombstones that cannot yet be removed safely, then writes new SSTables in sorted order. The difficult part is doing that at scale without allowing the implementation overhead to become a sizeable part of the job.

The cursor path has three main components in the Cassandra source. They are SSTableCursorReader, CursorCompactor, and SSTableCursorWriter. The reader moves through an SSTable with a small state machine. It recognizes partition starts, static rows, rows, cell headers, cell values, range tombstone markers, partition ends, and end of input. Reusable PartitionDescriptor, UnfilteredDescriptor, liveness, deletion-time, and cell structures hold the current decoded values rather than allocating a new object graph each time the reader advances.

The compactor keeps one cursor for each input SSTable. It orders the cursors by their current partition and clustering position, identifies the input records that represent the same logical value, applies Cassandra’s timestamp and deletion reconciliation rules, and asks the writer to emit only the surviving data. The writer serializes the output SSTable components directly, updates index and metadata information, and retains only bounded scratch buffers where the format requires data such as row headers or complex-column markers to be written after a row is known to be complete.

Step	Conventional iterator path	Cursor path
Read a partition	Decode the partition into standard objects used by the iterator pipeline.	Advance an `SSTableCursorReader` and load the current partition descriptor into reusable storage.
Read rows and cells	Construct row, cell, liveness, and deletion representations as the stream is traversed.	Advance through reader states such as row start, cell header, and cell value while reusing descriptors.
Reconcile input SSTables	Merge iterator entries and apply deletion and liveness rules.	Sort the active cursors, merge matching descriptors, and apply the same rules without retaining a per-entry object graph.
Write output	Serialize merged objects into a new SSTable.	`SSTableCursorWriter` writes the surviving descriptors and updates the BIG-format index and metadata.
Move to the next partition	Discard temporary objects and repeat.	Reuse the same descriptor and scratch objects for the next partition.

This is why the allocation change can be so large. The CASSANDRA-20918 ticket describes the early benchmark suite as holding heap allocation around 20 MB for cursor compaction, where the regular implementation exceeded 5 GB for the tested cases. The same ticket reports three to five times faster compaction in several test mixes. Those are development benchmarks, not a performance promise for every cluster. Compaction work that no longer creates a large volume of temporary objects leaves more CPU and GC capacity for the service.

The source also makes clear that this is more than a read-side shortcut. A cursor writer has to preserve SSTable ordering, row indexes, partition metadata, checksums, and the serialized structure of simple and complex columns. Replacing mature iterator code with a low-allocation path requires a substantial body of test coverage because an optimization in this area must never change which value wins or which tombstone remains.

Eligibility and fallback

Cursor compaction is deliberately guarded. cursor_compaction_enabled remains false in cassandra.yaml, while cassandra_latest.yaml sets it to true. The configuration calls it an experimental garbage-free compaction path, so the enabled setting still needs production-like testing.

# cassandra.yaml
cursor_compaction_enabled: true

When enabled, Cassandra still checks each compaction. It uses the conventional iterator path whenever the cursor implementation does not support the table or the requested operation. This fallback is important because an unsupported condition costs the opportunity for the optimization, not the correctness of the compaction.

In alpha3, CASSANDRA-21463 adds non-frozen collection and UDT support to the cursor path. Counter tables still use the iterator path, as does an SSTable header that contains a dropped multi-cell column. The cursor path remains experimental because alpha releases found real output divergences, including index-offset overflow, same-timestamp tie breaks, dropped-column filtering, and materialized-view row resurrection. Those fixes are covered by CASSANDRA-21462, CASSANDRA-21255, and CASSANDRA-21152.

Situation	Cursor-compaction behaviour	Operational implication
Plain supported table and current BIG SSTables	Cursor compaction can run when enabled.	Establish the allocation, CPU, compaction-throughput, and query-latency baseline before enabling it broadly.
Table with counters	Cassandra uses the iterator path.	A cluster can contain both paths. Do not assume the setting changes every table’s maintenance profile.
Non-frozen collections and UDTs	Cursor compaction can run when the other eligibility checks pass.	Alpha3 removed the earlier collection fallback, but this still needs schema-specific tests.
Dropped multi-cell column retained in an SSTable header	Cassandra uses the iterator path.	Test tables with the schema history and SSTable generations that exist in production, not only a newly created table.

The practical result is that this feature should be evaluated table by table. An environment that looks simple from its current CQL schema can have older SSTables and schema history that alter eligibility. A test that forces one compacted table into a clean state cannot show how the cluster behaves while it has a mixture of active tables, old files, normal background compaction, repair-related maintenance, and client traffic.

Direct I/O for SSTable reads and writes

Cursor compaction participates in the compaction read access mode introduced with the direct-I/O read work. CASSANDRA-21147 connects the cursor reader to the same configurable SSTable disk-access mode used by the iterator path. This lets a supported cursor compaction set compaction_read_disk_access_mode to direct, which bypasses the operating-system page cache for the selected compaction input reads.

That setting is useful when compaction scans a large input set which is unlikely to be read again, while live queries depend on a hot working set in the page cache. It does not make client reads direct I/O, and it does not remove device contention between a compaction and the query workload. Its purpose is to prevent the compaction input from evicting useful query pages simply because it is large and sequential.

The write side has a separate scope. Cassandra 6.0 provides background_write_disk_access_mode for compressed SSTable output produced by background operations. These include compaction, streaming, cleanup, repair, and upgrade SSTables.

# cassandra.yaml
# Applies only to compressed SSTable writes from background operations.
background_write_disk_access_mode: direct

# Per concurrent background writer. This uses off-heap memory.
direct_write_buffer_size: 1MiB

CASSANDRA-21134 is explicit about the boundaries. Uncompressed tables continue to use buffered writes. Memtable flushes also remain buffered because recently flushed data often benefits from being available in the page cache for subsequent reads. Direct background writes need a staging buffer that is aligned to filesystem requirements; the configured buffer is allocated per concurrent background writer, so its off-heap footprint increases with compaction and streaming concurrency.

The direct-I/O options are therefore not a blanket storage-performance mode. They make one trade-off. Background SSTable maintenance can preserve page-cache residency for query data, while the device has to perform the writes without the kernel buffering that normally absorbs and schedules them. The appropriate test has concurrent queries, writes, compaction, and enough data to exceed memory. An idle compaction benchmark cannot show whether the setting improves the user-facing cluster.

Trickle fsync

Trickle fsync affects the buffered path rather than bypassing it. Cassandra’s 6.0 configuration enables trickle_fsync and sets trickle_fsync_interval to 10240KiB by default. During sequential writes, Cassandra calls fsync() at intervals instead of allowing a much larger amount of dirty data to accumulate and be flushed in one burst.

The interval is counted in uncompressed bytes. For a compressed SSTable, this can mean that the fsync occurs after fewer physical bytes have reached the device than the configured interval suggests. A smaller interval limits the size of each forced writeback event and can protect read tail latency, while increasing the rate of fsync calls and reducing write throughput on some devices. A larger interval reduces that overhead until it allows the kernel’s normal dirty-page writeback behaviour to dominate again.

Storage condition	What to compare	Signals that decide the setting
SSD or NVMe with concurrent reads	Default interval against the intended change under the same compaction load.	p95/p99 read latency, write latency, device queue depth, fsync duration, compaction progress, and host CPU.
Network-attached or cloud block storage	Buffered and direct background writes, with the same compaction concurrency.	Tail latency during storage stalls, IOPS and throughput limits, write throttling, and compaction completion time.
Spinning disks	Default trickle fsync against a carefully larger interval.	Read latency variability, seek pressure, throughput, and whether maintenance falls behind.
Write-heavy tables that are rarely read after flush	Buffered flush behaviour should remain the reference case.	Page-cache usefulness after flush, read latency when a hot set exists, and disk behaviour during flush plus compaction.

Compaction evidence

Compaction is not isolated from the rest of Cassandra. Faster work can lower the time SSTables overlap, which may reduce read amplification, while a higher compaction rate can also change when new work is selected. The CASSANDRA-20918 review includes benchmark and profiling work from Dmitry Konstantinov that showed a substantial reduction in compaction-thread allocation and a roughly two-times throughput improvement for his VInt-heavy test case. It also notes that faster compaction can leave less time for SSTables to accumulate into one batch during an intensive write workload. That is a useful reminder to examine queueing and SSTable shape, not only one throughput number.

Cassandra 6.0 also improves the evidence available to the operator. CASSANDRA-20081 extends nodetool compactionhistory with compaction type, strategy name, and level. This makes it easier to establish whether a result came from STCS, LCS, TWCS, or UCS and to preserve the execution context with the event.

Unified Compaction Strategy also gains parallel output-shard compaction. The parallelize_output_shards option splits one UCS compaction into separate output-shard tasks, which can reduce its duration. Major compactions can use the same parallelism through nodetool compact --jobs, with Cassandra limiting the default to half of the available compaction threads so a major compaction does not starve background work.

Test	Baseline	Cassandra 6.0 comparison	Measurements
Cursor implementation	`cursor_compaction_enabled: false` with the production table definition and SSTable generations.	Enable it only for tables that are eligible, then confirm which path actually runs from logs and profiling.	Compaction throughput, compaction-thread allocation rate, CPU, GC activity, pending compactions, SSTables per read, and read p95/p99.
Compaction reads	Buffered read access for compaction.	`compaction_read_disk_access_mode: direct`, tested with the cursor path where eligible.	Page-cache behaviour, major faults, device queue depth, compaction rate, and read latency during and after compaction.
Compressed background writes	`background_write_disk_access_mode: standard`.	Direct background writes with a controlled `direct_write_buffer_size`.	Off-heap memory per active writer, device latency, write throughput, compaction completion time, and client read latency.
Trickle fsync	The supplied interval with the same storage and write concurrency.	One controlled interval change at a time.	Fsync latency, tail read and write latency, dirty writeback behaviour, device utilization, and maintenance backlog.
Mixed production traffic	Reads, writes, repair-related maintenance, compaction, and normal operational concurrency.	Repeat after each independent setting change.	Coordinator and replica latency, table-level SSTables per read, tombstones scanned, heap and off-heap allocation, logs, and full configuration.

Capture the selected compaction strategy, its options, SSTable format, compression configuration, storage compatibility mode, device model, filesystem, kernel, JDK, concurrent compactor count, table schema, data volume, and data age distribution with every run. Without that information a good result is difficult to reproduce and a bad result is difficult to explain.

AxonOps is useful here because this evaluation needs several kinds of evidence at once. That includes Cassandra compaction and table metrics, host I/O, client-facing latency, JVM allocation and GC activity, logs, events, and the configuration that selected the path. Looking at one graph after the fact rarely identifies whether an observed latency change came from page-cache eviction, storage queueing, a shift in SSTable count, or a compaction backlog.

Contributors

Nitsan Wakart authored the cursor-compaction implementation. The merged Apache Cassandra pull request credits Branimir Lambov and Dmitry Konstantinov for review. The ticket records Benedict Elliott Smith’s guidance, David Capwell’s benchmark work, and Josh McKenzie’s help with the open-source contribution process and merge. It also records the test, profiling, and QA work that was needed to establish the behaviour beyond a microbenchmark.

Sam Lightfoot reported and implemented the direct background-write and cursor direct-read follow-up work. Brad Schoening raised the request for more informative compaction history; Arvind Kandpal implemented it, with Maxwell Guo and Jyothsna Konisa credited for review.

Those names are the explicit credits attached to these tickets and commits. The feature also depends on the people who maintain the compaction tests, run CI, investigate failures, review follow-up changes, prepare releases, write documentation, and support the users who exercise Cassandra under real operational pressure. I am grateful for the care that goes into all of that work.

Demo Sandbox

Thank You from AxonOps: Current London 2026 Recap