Cassandra

Apache Cassandra 6.0 Part 4: Repair, Guardrails, and Observability

Hayato Shimizu · June 25, 2026 · Updated September 7, 2026 · 17 min read

Repair, Guardrails, and Observability

Repair is one of the operations that separates a healthy Cassandra deployment from one that has simply been left running. Replicas can diverge because a node was unavailable, hints expired, a write was missed, or data was changed while a replica was out of service. Deletes make the timing more important because tombstones only protect against resurrection while they remain available on every relevant replica.

For much of Cassandra’s history, teams have used scripts or external services to decide what to repair, when to run it, how much parallelism to allow, and how to record proof that it completed. Those systems can work well, although every deployment has had to make its own choices around repair history, scheduling, retries, disk headroom, alerting, topology changes, and failures that occur halfway through a run.

The Cassandra 6.0 line brings more of that operational control into the database. It also adds guardrails and diagnostic surfaces that allow a cluster to reject or warn about known risks before they become less predictable production incidents. The important question is not whether a new setting exists. It is whether the team can show what it does under normal load, failure conditions, and a recovery workflow.

Cassandra 6.0 is pre-release software. This post reflects 6.0-alpha3 and must be checked again against the release notes and upgrade documentation for the exact version being deployed.

Area	Cassandra 5.0 baseline	Cassandra 6.0 work to validate
Repair control	Operators commonly coordinate repair with external schedules and retain their own repair history.	CEP-37 introduces a scheduler and replicated repair-history state inside Cassandra for full, incremental, and preview-repaired categories.
Disk protection	A disk guardrail can protect a replica or token range as capacity becomes unsafe.	A keyspace-wide option can stop writes across all replicas of a keyspace when one participating node crosses the failure threshold.
Client compatibility	Driver inventory and enforcement are usually external release-process work.	A server-side driver-version guardrail can warn or reject a declared driver type and version below a configured minimum.
Diagnosis	Logs, JMX metrics, tracing, and external profilers are the usual evidence sources.	Slow-query records, integrated async profiling, administrative history, and richer table metrics give operators more Cassandra-native evidence.

Automated repair architecture

CASSANDRA-19918 is the Apache Cassandra Unified Repair Solution, also known as CEP-37. Its purpose is not to change anti-entropy itself. Cassandra still compares replica data and streams differences when required. The change is that Cassandra can own the scheduling and history of repair instead of requiring every user to build a separate controller for the entire ring.

The scheduler keeps replicated repair history in the system_distributed keyspace. That history records enough state for nodes to determine what has been repaired and to assign follow-up work across the ring. A scheduler thread pool coordinates repair work, while assignments can be split into smaller token ranges so an individual repair does not become an unbounded task. The shipped repair types are full, incremental, and preview-repaired. Paxos repair is proposed separately and is not a scheduler type in 6.0-alpha3.

The core flow is straightforward, but its operational implications need attention.

Step	Scheduler activity	Why the detail is operationally important
1. Read repair history	Nodes use replicated state to establish which table and token ranges need attention.	Repair selection has an auditable history rather than existing only in a cron job or an external service database.
2. Create bounded assignments	The scheduler divides the work by token range, data size, or partition count according to its configuration.	Assignment size controls the burst of disk, CPU, network, and compaction work introduced by repair.
3. Select eligible work	It considers repair type, node and replica constraints, schedules, retry state, major-version safety, and configured data-center scope.	The system needs to avoid turning an already overloaded node or an incompatible rolling upgrade into more repair work.
4. Run and record repair	Cassandra carries out the selected repair and updates replicated history as it progresses.	Completion must be distinguishable from a job that was started, retried, interrupted, or failed after streaming began.
5. Revisit the ring	The scheduler returns to ranges and tables according to its interval and repair policy.	The useful measure is repair freshness across the full data set, not an isolated successful session.

The supplied Cassandra 6.0 configuration contains an auto_repair section with global settings and per-repair-type overrides. It allows separate handling for full, incremental, and preview-repaired work. Relevant controls include bytes_per_assignment, partitions_per_assignment, a maximum number of tables per assignment, a minimum interval before the same node is repaired again, retry limits, and a backoff period. Operators can choose whether to repair only primary token ranges, exclude data centers, group tables by keyspace, and control the number or percentage of nodes repairing in parallel.

The auto_repair section ships commented out, so the scheduler is disabled until an operator configures it. The supplied example enables only full repair. Treat that as a starting point for testing rather than a default production policy.

This level of configuration is necessary because repair cannot be reduced to a single daily schedule. A cluster with high write throughput, large tables, multi-region replication, or limited disk headroom must decide how quickly it needs to revisit data and what impact it can accept while doing so. A cluster that cannot finish its repair cycle before its safety window expires has a capacity or operational-design problem even if every individual repair command returns successfully. Adaptive Regulation of Cassandra Repair explains the feedback model AxonOps uses to adapt repair velocity and parallelism to live cluster conditions while completing repair within the required window.

Repair state and failure handling

Repair state must be treated as shared coordination data. A scheduler that allows one node to remove a history entry while another updates it can create state that no longer reliably describes the work. CASSANDRA-20996 proposes lightweight transactions for all auto-repair history mutations. It remains open in 6.0-alpha3, and the current implementation uses lightweight transactions for only a subset of those operations.

That choice has cost, because LWT is coordination work, but the history is not a high-volume application table. It is the record used to ensure that multiple schedulers do not make conflicting decisions about the same repair state. The relevant test is not only that a scheduled repair runs; it is that node restarts, a retry, a history cleanup, and an overlapping scheduler decision leave the history internally consistent.

Disk headroom is the second important failure boundary. Repair can require streaming and anti-compaction work, which can temporarily increase disk use. Cassandra 6.0 includes a repair disk-headroom rejection threshold and a configurable compaction threshold that can prevent repair work from being started when pending compaction is already too high. Those controls should be set from measured free space and the largest repair assignment that the cluster will permit, not from the point at which a disk is nearly full.

Failure or pressure condition	Required evidence	Expected operating response
A node restarts while repair history is updated	History record state before and after the restart, scheduler logs, and repaired-range freshness.	Confirm the scheduler does not duplicate or lose the assignment and that its retry state is explicit.
Repair is delayed by compaction pressure	Pending compactions, repair-rejection event, table write rate, and disk queueing.	Reduce repair parallelism or assignment size, address the backlog, then resume with a documented schedule.
Disk headroom falls below the repair threshold	Free space by data directory, repair activity, compaction output, and retention trend.	Stop adding repair pressure, recover capacity, and establish why the headroom model was wrong.
Rolling major-version upgrade is in progress	Node version inventory, repair configuration, and scheduler events.	Follow the mixed-version guardrail rather than allowing automated work to mask an upgrade issue.
Network or replica failure interrupts a session	Session state, retry count, streamed bytes, timeout and error logs, and post-retry repair history.	Verify the retry is bounded and the range remains visible until it has actually completed.

The scheduler reduces the amount of custom orchestration required. It does not remove the need to understand the repair lifecycle, measure its impact, retain evidence, or make a decision about what happens when the cluster cannot keep up.

Repair control under live load

CEP-37 is a welcome step for Cassandra because it gives operators a built-in scheduler, replicated repair history, token-range assignments, retries, protective thresholds, and a clearer repair surface inside the database. That will be valuable for teams that need a sound native repair baseline without deploying and maintaining another scheduler.

AxonOps Adaptive Repair is designed to keep repair on track while the cluster is changing underneath it. It uses high-resolution Cassandra and Linux telemetry to regulate repair velocity and parallelism against the time remaining before each table’s gc_grace_seconds deadline. It does not assume that a repair plan created at the start of a run remains appropriate after client traffic, compaction pressure, replica latency, or host I/O wait changes.

Control question	CEP-37 scheduler	AxonOps Adaptive Repair
How is work organised?	Cassandra maintains repair history and schedules bounded assignments with repair-type, concurrency, retry, and policy controls.	AxonOps plans token-range segments against the table volume and repair deadline, then continuously regulates their launch rate.
What is the operating input?	Configured assignment, interval, concurrency, safety, and retry settings within Cassandra’s scheduler.	Current repair progress plus 5-second Cassandra and Linux telemetry, including pending `ReadStage` and `MutationStage` work, coordinator and replica latency drift, and I/O wait.
What happens as the cluster becomes busy?	The configured scheduler and protective thresholds govern whether work remains eligible.	The feedback controller can reduce or increase repair velocity and parallelism as the live performance state changes, while preserving the completion target.
What is the target?	Reliable automatic repair orchestration within Cassandra.	Complete repair inside the required safety window without treating a live cluster as if it were idle.

This is not a reason to dismiss CEP-37. It establishes useful capabilities in Cassandra and moves repair in the right direction. Teams that need repair to remain responsive to live application load, uneven table sizes, changing compaction pressure, and host-level I/O conditions should evaluate Adaptive Regulation of Cassandra Repair. That article explains the feedback model and why fine-grained telemetry is central to this approach.

Disk and driver guardrails

Guardrails convert conditions that are often discovered late into explicit warnings or failures. They do not replace capacity planning, schema review, client-release management, or early alerting. Their role is to protect the database and make a dangerous condition impossible to ignore.

CASSANDRA-21024 adds a keyspace-wide disk-usage option. Earlier disk protection could reject writes for token ranges whose replicas included a node past the configured disk failure threshold. That protects the node, but it can leave an application with partial write availability that differs by token range. The new option allows Cassandra to stop writes for the whole keyspace across all nodes that replicate it when any participating node exceeds the threshold.

This is a deliberate availability trade-off. The token-level policy can preserve writes for unaffected replica sets. The keyspace-wide policy makes the failure mode simpler and more predictable for the application, while sacrificing writes that may otherwise have succeeded. Neither setting fixes a disk-capacity problem. The choice should be based on the application’s ability to handle partial keyspace availability and the operational preference for a clear, consistent failure boundary.

CASSANDRA-21146 provides a driver-version guardrail. Cassandra can evaluate the driver type and version reported during connection startup against configured warning or failure minima. The guardrail is disabled until configured, and it is particularly useful in development or pre-production environments where teams need a clear signal before an outdated client reaches a database upgrade.

The configuration separates warn and reject maps. An operator can declare a minimum version per known driver type, then decide whether versions below the threshold should generate a warning or fail the connection. The maps can also contain unknown and unset entries for clients that do not report an identity. This still needs a real client inventory because a nonstandard driver or an old service can otherwise create a rollout surprise.

CASSANDRA-17258 adds client warnings when a write targets a partition already above a tracked size or tombstone threshold. The guardrail is disabled by default through write_thresholds_enabled: false, with thresholds unset. It builds on node-local top-partition information. This is useful because an oversized partition usually becomes expensive long before it becomes an obvious outage. It can increase read work, compaction cost, repair time, streaming load, and failure recovery time.

Guardrail	What Cassandra can do	What it does not do
Disk usage	Warn or reject according to configured thresholds; optionally fail keyspace writes across participating replicas.	Add disk, choose retention, rebalance data, or recover space already consumed by an unsafe table.
Driver version	Warn or reject connections that report a configured driver type below the minimum version.	Test an application migration or prove that every service reports its version accurately.
Large-partition write	Warn a client when it adds data to a partition already tracked as large or tombstone-heavy.	Redesign the partition key, reduce cardinality, or remove existing oversized partitions.
Repair disk protection	Decline repair preparation or streaming under unsafe free-space or compaction conditions.	Create the capacity required to catch up on the repair backlog.

The right operating model has earlier warning thresholds in monitoring than the database’s reject thresholds. A disk guardrail firing should be the final safety control, not the first notification that a keyspace is growing too quickly.

Slow queries, profiling, and metrics

Cassandra 6.0 improves several sources of diagnostic evidence. CASSANDRA-13001 makes slow-query logging available through a virtual-table appender instead of only a debug log destination. That gives an operator a Cassandra-native queryable record of slow work, while a monitoring system can still collect and retain the event with the surrounding metrics and logs.

CASSANDRA-20854 integrates the async-profiler library through JMX and nodetool profile. The point is to make a low-overhead profile available through the normal operational interface when CPU or allocation analysis is needed. It does not mean profiling is free, and it must be governed carefully on a busy production node, but it removes the need to improvise an attach workflow during an incident.

The 6.0 line also adds or improves table and virtual-table metrics, including total rows read and mutated, rows-mutated-per-write histograms, prepared-statement cache information, hint metrics, timer percentile information, uncaught-exception visibility, and administrative history. These signals are most useful when they are interpreted together.

Symptom	Cassandra evidence to correlate	Questions to answer
Read latency increases	Slow-query records, rows read, SSTables per read, tombstones scanned, read timeouts, device latency, and compaction activity.	Is the request reading more data, touching more SSTables, waiting on disk, or competing with maintenance work?
Write latency increases	Rows mutated per write, commitlog and memtable metrics, flush and compaction state, disk queueing, and client errors.	Did mutation size change, did flushing fall behind, or is the storage path saturated?
Node CPU is high	Async profile, executor activity, allocation rate, GC information, query and repair rate, and recent configuration events.	Which code path is consuming the time and is it driven by application traffic, repair, compaction, or a regression?
A production change precedes drift	`nodetool history`, configuration history, schema events, topology events, and the before-and-after workload profile.	What changed, when did it propagate, and does the timing align with the observed behaviour?

I do not want a Cassandra incident to be diagnosed from a vague heap graph or a single node-level CPU number. A latency alert should lead to the affected table, request pattern, storage state, and recent operational change. Cassandra’s additional signals make that path more direct, while AxonOps retains the metrics, logs, events, configurations, topology context, and repair-control decisions together over time. That history is what allows Adaptive Repair to respond to the cluster that is actually running, rather than a static view of the one that existed when the schedule was created.

Cassandra 5.0 and 6.0 validation

The useful comparison is not whether a feature can be enabled in an isolated test cluster. It is whether the 6.0 operating model produces better evidence and safer behaviour under the work that the production cluster already does.

Test	Cassandra 5.0 baseline	Cassandra 6.0 comparison	Measurements
Repair scheduling	Existing external schedule and its repair history.	Configure the scheduler with conservative assignment, concurrency, and retry settings.	Repair freshness by range and table, bytes repaired, failed and retried sessions, compaction backlog, disk headroom, read/write latency, and history consistency.
Disk-pressure response	Existing alerts and token-level write rejection behaviour.	Test configured warning and failure thresholds, including the keyspace-wide option, in a controlled environment.	Per-node free space, rejected writes by token and keyspace, application error handling, replica availability, and recovery after capacity is restored.
Driver compatibility	Inventory drivers outside the database.	Warn in pre-production first, then test failure handling with an obsolete declared driver version.	Connection warnings or failures, service rollout readiness, reporting accuracy, and no unplanned application outage.
Slow-query analysis	Logs, tracing, JMX, and external profiling workflow.	Add virtual-table slow-query records and a controlled async profile to the incident path.	Query identifiers, table and coordinator context, profile overhead, CPU and allocation evidence, and retained incident history.
Large-partition protection	Existing top-partition metrics and manual review.	Test client warnings while a known large partition receives writes.	Warning delivery, partition size and tombstone evolution, read cost, compaction impact, and follow-up schema remediation.

Contributors

Jaydeepkumar Chovatia proposed and led the Unified Repair Solution work. Kristijonas Zalys proposed the still-open work on concurrent auto-repair history mutations, while Isaac Reath developed the keyspace-wide disk-usage guardrail. Brad Schoening raised the driver-version guardrail request, with Stefan Miklosovic assigned to the work. David Capwell raised the large-partition client-warning work, which Minal Kyada implemented. Jon Haddad proposed the slow-query virtual-table appender, and Yaman Ziadeh with Bernardo Botella Corbi brought async profiling into Cassandra’s JMX and nodetool surface.

Those ticket credits are only the visible part of the effort. Repair and observability features need design review, distributed-system testing, regression investigation, CI capacity, documentation, packaging, release management, and the reports from operators who encounter difficult cluster states. I am grateful to everyone who contributes that work to Cassandra.

Demo Sandbox

Thank You from AxonOps: Current London 2026 Recap