Cassandra

Apache Cassandra 6.0 Part 10: Upgrade and Production Validation

Hayato Shimizu · August 28, 2026 · Updated September 7, 2026 · 14 min read

Upgrade and Production Validation

An Apache Cassandra upgrade is a distributed-system change, not a sequence of package installations. A node joins a ring with a particular protocol implementation, storage format, schema state, JVM, configuration, driver population, monitoring stack, and history of SSTables. The upgrade is complete only when all of those elements are understood well enough to operate the cluster under normal traffic and a failure condition.

This series describes work on the Cassandra 6.0 line. Cassandra 6.0 is pre-release software, and this post reflects 6.0-alpha3. The first rule of a production plan is to use the release documentation, release notes, and compatibility guidance for the exact alpha, beta, release candidate, or GA version being deployed. A branch changelog is useful for understanding the direction and components of the work, but it is not a substitute for the final upgrade procedure. Do not infer a release guarantee or a production sequence from a feature branch alone.

The second rule is to keep the binary upgrade separate from optional feature adoption. Accord, Zstd dictionary compression, cursor compaction, direct I/O modes, automated repair, SAI changes, JDK 21, Generational ZGC, and new guardrails each have different operational effects. Enabling them together makes a regression difficult to attribute and makes a rollback decision less clear.

Change category	Include in the binary upgrade	Validate as a separate controlled change
Cassandra binary and supported configuration	Yes, after compatibility and rolling-upgrade tests.	N/A
Drivers and application behaviour	Inventory and test before any production node is upgraded.	Upgrade individual drivers and services through their own release process.
Storage format and compatibility mode	Follow the exact release documentation and do not advance the format early.	Advance only after every node and operational tool has been validated.
Cursor compaction and direct I/O	Leave at the known baseline unless the feature test already supports a change.	Test table eligibility, disk path, page-cache impact, and rollback separately.
Accord, compression dictionaries, SAI, constraints	Do not make application semantic changes during the database rollout.	Test each feature with a specific workload and application migration.
Automated repair, guardrails, JDK 21	Retain the known operating model through the binary rollout.	Introduce scheduler policy, reject thresholds, or JVM choice after baseline stability is established.

Build the upgrade inventory

An upgrade plan should begin with an inventory that is sufficiently detailed to explain a later regression. The critical information is not limited to Cassandra version numbers. It includes the table definitions and storage characteristics that decide whether a node behaves differently after it restarts.

Inventory area	Record before testing	Why it changes the upgrade outcome
Cassandra and Java	Node version, exact package or image, JDK vendor and patch level, JVM options, heap, direct-memory configuration, and process limits.	A JVM or package change can look like a Cassandra regression when the binary is not the source of the difference.
Topology and networking	Data centers, racks, token allocation, replication, seed and discovery configuration, TLS, internode ports, and node replacement procedure.	Bootstrap, streaming, repair, and metadata changes need the actual topology rather than an idealized ring diagram.
Schema and storage	Keyspaces, table schemas, counters, collections, indexes, compaction strategies, compression, TTLs, table size, SSTable format, disk layout, and largest partitions.	Storage compatibility, cursor eligibility, index cost, repair volume, and compaction behaviour depend on the real tables.
Application clients	Driver type and version, authentication, consistency levels, retry policies, prepared-statement use, request sizes, paging, timeouts, and service ownership.	Client compatibility and retry behaviour can turn a short node restart into an application incident.
Operations tooling	Metrics, logs, tracing, JMX, agents, backup, restore, repair controller, alerting, deployment automation, incident runbooks, and access controls.	An upgrade is not safe if operators lose the ability to diagnose or recover the cluster.
Capacity baseline	Disk free space by data directory, compaction backlog, repair freshness, read/write latency, timeout rate, CPU, memory, network, and workload seasonality.	A node under existing pressure is a poor canary and can conceal the reason an upgrade test fails.

The inventory should be an artifact stored with the test results, not a collection of assumptions spread across tickets and chat messages. It becomes the reference for determining whether the canary node, the first rack, and the fully upgraded ring remain comparable to the original cluster.

Before the rolling upgrade

Confirm the supported source version, the exact Cassandra 6.0 release, rolling order, and release-specific instructions.
Test a cluster with representative data, old SSTables, tombstones, indexes, repair history, compaction pressure, backups, streaming, and client traffic.
Check every application and administrative driver, including infrequent jobs. The 6.0 driver-version guardrail can help find declared gaps before production.
Restore a representative backup with the normal tooling.
Test bootstrap, replacement, decommission, restart, streaming, repair, compaction, schema changes, and a node failure.
Confirm metrics, logs, JMX, profiling, backups, alerts, access control, and incident tooling work with Cassandra 6.0.
Capture the pre-upgrade ring, schema, configuration, disk space, repair age, compaction state, table-level reads and writes, SSTables per read, tombstones, latency, timeouts, CPU, heap, direct memory, and network use.
Identify and disable automation that can change schema, bootstrap, decommission, move, replace, or assassinate nodes during the CMS transition.
Choose the CMS members and failure-domain placement before starting. The post-initialization CMS must be expanded from one member to at least three.

Rolling upgrade procedure

Follow the exact release procedure and supported rolling order for the version being installed.
Upgrade the Cassandra binary only where the source runtime is already a supported JDK. Cassandra 6.0 supports JDK 11, 17, and 21, so a 4.0 or 4.1 cluster on JDK 8 needs a planned JDK move before or as part of its supported upgrade sequence. Keep drivers, storage compatibility settings, repair policy, and optional 6.0 features unchanged otherwise.
Upgrade one representative node. Do not use a node already under disk, compaction, repair, or incident pressure.
Return the node to service and check topology, client traffic, logs, schema and metadata state, streaming, compaction, repair, monitoring, backup, and host resources.
Leave the node under normal traffic and maintenance work. Compare it with nodes that have not changed.
Upgrade the next small group only when the previous group is healthy. Pause the rollout when latency, timeouts, pending work, disk use, CPU, memory, or telemetry changes unexpectedly.
After every node is on Cassandra 6.0, run nodetool cms initialize on one node. Until that step succeeds, schema changes, bootstrap, decommission, move, replacement, and assassinate operations are prohibited. If initialization reports an upgrade mismatch, follow the version-specific NEWS.txt recovery procedure, including its documented stop-and-retry sequence and ignore option.
Run nodetool cms reconfigure to expand the initial one-member CMS to at least three members, distributed across the intended failure domains. Treat storage compatibility changes and optional 6.0 features as separate later changes.

Optional features

Do not turn on every new capability as soon as the last node has restarted. Accord changes application behaviour. Compression, cursor compaction, direct I/O, repair policy, SAI, guardrails, and the JVM each change a different part of the system. Give every change its own test and rollout so that a regression has an obvious starting point.

Change	First production use	Check
Accord transactions	Test one application workflow with defined transaction boundaries, contention, timeout, retry, and recovery behaviour.	Correct committed results, latency under contention, failure recovery, driver support, and operational diagnostics.
Zstd dictionary compression	Test one representative table and dictionary lifecycle.	Compression ratio, CPU, flush and compaction rate, read latency, memory use, dictionary distribution, and rollback behaviour.
Cursor compaction	Enable only for eligible tables after source SSTable, index, schema, and partitioner checks.	Cursor-path selection, allocation rate, compaction throughput, GC, query latency, and fallback behaviour.
Direct I/O modes	Change reads or compressed background writes one setting at a time.	Page-cache impact, device queueing, direct-write buffer use, compaction rate, and read/write tail latency.
Automated repair	Start with conservative assignment and concurrency policy.	Repair freshness, history consistency, retries, streamed bytes, compaction pressure, disk headroom, and application impact.
SAI and constraints	Test an explicit query or data-validation requirement.	Index build state, selectivity, write cost, query latency, rejected-mutation handling, migration compatibility, and a safe rollback.
JDK 21 and Generational ZGC	Compare the existing JVM and collector under identical workload and host conditions.	End-to-end latency, throughput, allocation, pauses, concurrent GC CPU, process RSS, direct memory, agents, logs, and restart behaviour.

When something changes, this gives the team a starting point. It is much easier to investigate a regression after one controlled change than after a rollout that altered the binary, storage settings, drivers, repair policy, and JVM together.

When to pause the rollout

Agree in advance what will make you stop. “The cluster looks unhealthy” is not useful when people are under pressure and half the ring has already changed. Use thresholds that the team understands, name the people who can pause the rollout, and collect the state of the cluster before changing anything else.

If this happens	Check first	Then
Sustained p99 read or write latency regression beyond the agreed SLO	Coordinator and replica latency separately, timeouts, retries, request shape, compaction, disk queueing, GC, and recent deployment event.	Pause the rollout and compare the affected node or rack to the preflight baseline before changing more variables.
Timeout or unavailable rate rises	Consistency level, replica availability, driver retries, node health, internode errors, and topology state.	Stop expanding the change, protect application traffic, and follow the documented recovery or rollback path.
Repair or compaction backlog grows unexpectedly	Pending work, table write rate, disk free space, device latency, repair assignment state, and host CPU.	Reduce operational pressure, stop the rollout, and determine whether the binary, configuration, or existing capacity caused the backlog.
Schema or metadata disagreement	Schema/version state, CMS or topology evidence, node logs, recent DDL, and mixed-version inventory.	Halt schema and topology changes, preserve evidence, and use the documented recovery procedure for the Cassandra 6.0 release.
Monitoring or backup coverage is lost	Missing metrics/logs, failed agent or JMX connections, backup status, restore evidence, and alert state.	Do not continue with an unsupported operating model; restore diagnostic and recovery coverage first.
Disk headroom becomes unsafe	Free space by directory, streaming, compaction output, repair work, retention, and write rate.	Halt work that expands disk use, recover capacity, and re-evaluate the rollout schedule.

Rollback is not always as simple as reinstalling the previous package. A storage-format change, compatibility-mode advance, schema change, or application feature can remove part of the path back. Work out the rollback boundary before the upgrade, keep optional features off until the binary change has settled, and test rollback or replacement in the representative environment. Include drivers, monitoring, backups, repair, and topology in that test.

Production acceptance

The production rollout is ready to close only after the upgraded cluster has operated through enough normal work to show that the new baseline is stable. That includes a full repair and compaction cycle appropriate to the data set, ordinary backup and restore checks, expected workload peaks, node restart or replacement evidence, and a review of the observability surfaces used during the rollout.

Acceptance area	Minimum proof
Availability	The application meets its agreed availability and latency objectives across all data centers and node groups.
Data protection	Repair freshness is within the required window, backup succeeds, and a representative restore has been verified.
Storage health	Disk headroom, compaction, SSTable counts, tombstones, and page-cache or device behaviour are stable for the workload.
Topology and schema	Nodes agree on required metadata, CMS initialization has completed, the CMS has at least three appropriately placed members, topology operations are understood, and application schema changes are controlled.
Client compatibility	Every production service, job, and administrative tool is operating through a supported and observed driver path.
Operations	Metrics, logs, events, configuration history, alerts, JMX, profiling, backup, and incident runbooks work with Cassandra 6.0.
Recovery	The team has exercised and documented the rollback, replacement, and failure response boundaries that remain available.

AxonOps provides the historical context needed during this process. It includes Cassandra metrics, host metrics, logs, events, configuration history, topology, repair state, and workload changes. The key is not to collect everything without structure. It is to retain the before-and-after evidence needed to answer what changed, when it changed, and which part of the system moved with it.

Contributors

The Cassandra 6.0 work described across this series comes from people implementing consensus and metadata changes, repair scheduling, guardrails, compaction, compression, indexing, CQL validation, JVM compatibility, test infrastructure, release engineering, documentation, package maintenance, security work, review, and support for users who report production issues. The individual posts credit the people named on the relevant tickets and pull requests.

I am grateful to all of them. A release that operators can deploy with confidence depends just as much on compatibility testing, CI, release preparation, documentation, and difficult bug investigation as it does on the headline code changes.

Demo Sandbox

Thank You from AxonOps: Current London 2026 Recap