Cassandra

Transactional Cluster Metadata (TCM) and Cluster Metadata Service (CMS) in Cassandra 6.0

Hayato Shimizu · · · 15 min read

Transactional Cluster Metadata (TCM) and Cluster Metadata Service (CMS) in Cassandra 6.0

The Cassandra 6.0 line includes important recovery, inspection, and topology-operation work around TCM, short for Transactional Cluster Metadata. TCM is the architecture introduced by CEP-21. The Cluster Metadata Service, or CMS, is the small quorum-backed service within that architecture which accepts, orders, and publishes cluster metadata changes.

TCM is not a new name for gossip. Cassandra still uses gossip for liveness and transient information, but TCM moves correctness-critical metadata onto a linearized event log. That metadata includes topology, token ownership, data placements, schema, and the information required to decide which replicas are valid for a request.

Bootstrap, replacement, decommission, token movement, and schema changes are the operations where an eventually propagated view of the ring is hardest to reason about. TCM gives those changes an ordered history, an epoch, precomputed placements, and rules that prevent the next step from starting before the affected replica groups have observed the preceding one.

In Cassandra 6.0, the work around TCM covers CMS rediscovery when addresses change, metadata-log inspection, offline recovery tooling, peer-table repair, and topology-safe data-centre and rack changes for live nodes.

Cassandra 6.0 is pre-release software. This post reflects 6.0-alpha3 and must be checked again against the release notes and upgrade documentation for the exact version being deployed.

Why Cassandra moved metadata out of gossip

Having operated Cassandra for almost twenty years, schema disagreement has been one of the most painful issues to investigate and recover from because a node can miss a schema change while unavailable, then return with a different view of a table, column, or type. Before TCM, several parts of cluster metadata were propagated through eventually consistent mechanisms. That approach is resilient when nodes are joining and leaving, although it does not provide one authoritative order for cluster-wide changes.

Cassandra deliberately has no central master to impose that order. Its masterless design has been valuable in difficult environments, although the trade-off is that membership, ownership, and schema changes need careful coordination across independent nodes. TCM is the architecture I have been looking forward to using because it brings a durable, ordered history to those changes while keeping Cassandra’s normal data path distributed.

With gossip propagation of topology changes, coordinators can learn a new placement at different times. During range movement, one coordinator can build a read plan from the old placement while another builds a write plan from the new placement, leaving the two plans without a guaranteed intersection. CEP-21 gives an example where one coordinator reads from {A,B} while another writes to {C,X} after a range movement. Both requests can appear successful even though the write is not visible to the earlier read quorum. In a worse timing window, a write can be left only on replicas that are about to stop owning the range.

TCM addresses this by recording topology transitions as ordered metadata events and by tracking separate read placements and write placements while data is moving. The temporary extra write replicas are deliberate. They protect writes while the cluster moves from one safe replica group to the next.

Schema was the other major reason for CEP-21. When gossip-based schema propagation is delayed, unavailable nodes can retain a different schema view from the rest of the cluster. Cassandra 4.x schema changes used the regular timestamped, last-write-wins path, so separately valid operations could race and nodes would independently pull or push schema to catch up. Those exchanges can create migration storms and do not give the cluster a single order in which to validate dependent changes.

The result is more serious than a temporary schema-version mismatch in nodetool describecluster. A coordinator and a replica can disagree about a column or table definition while serving a request, and concurrent DDL can leave a schema which requires manual intervention. CEP-21 gives an example of table creation that depends on a user-defined type at the same time as a DROP TYPE. Each coordinator can validate against an earlier view, then leave a table which cannot load after restart because its type has been removed. TCM serializes metadata changes against the latest committed state and publishes the resulting log entry in order instead of resolving the conflict after the fact.

I am grateful to the Cassandra committers and the wider contributor community for work like this. The implementation is visible, but the testing, review, compatibility work, release preparation, documentation, and communication that make a change of this scope usable are just as important. Cassandra has benefited from that sustained effort for many years, and TCM is a substantial example of it.

TCM architecture

CMS members are a subset of the Cassandra nodes. A node proposing a metadata change sends an event to a CMS member. The service validates the event against the latest metadata, rejects changes that conflict with an in-flight operation or would violate replication invariants, then commits an immutable log entry through a CMS quorum. The commit receives a monotonically increasing epoch.

The CMS then disseminates the committed entry to the cluster. Every node applies the ordered events to build its local immutable ClusterMetadata view. A metadata event is not required to synchronously reach every node before it can be committed. The operation rules instead control when the next topology step may proceed, ensuring that the affected replica groups have acknowledged the prior epoch.

CEP-21 deliberately keeps this control plane inside Cassandra. The CMS is not ZooKeeper, etcd, or a new external dependency; its members are Cassandra nodes which have taken on a metadata-consensus role. Normal data requests retain Cassandra’s coordinator and replica path, using the local placement snapshot rather than routing through the CMS.

TCM architecture diagram showing a metadata event entering a CMS quorum, becoming an epoch in the metadata log, disseminating to Cassandra nodes, and epochs exchanged during normal reads and writes.

The split between the CMS and the data plane is intentional. A client read or write does not need every node to participate in metadata consensus. It uses the local placement snapshot, then includes the relevant epoch in internode messages. A replica that has fallen behind can catch up to the required epoch before it processes the request. A replica that is ahead returns its epoch, allowing the coordinator to catch up and validate its replica plan again.

This is where the architecture changes the safety of a topology operation. Metadata propagation can still be asynchronous, but an older coordinator cannot quietly complete a quorum request against a placement that has been superseded without learning that its view is stale.

Epochs, metadata log entries, and placement snapshots

An epoch is the position of a committed event in the metadata history. It is not a wall-clock timestamp and it does not describe the age of a node. It identifies a particular immutable cluster metadata state.

For a topology operation, Cassandra can precompute the read and write placements associated with an epoch. This moves placement calculation away from the hot request path and gives the coordinator a stable replica plan for the metadata it knows. When a request encounters a newer epoch, Cassandra can retrieve the missing log entries, update the local snapshot, and check whether the original plan still satisfies the requested consistency level.

The metadata event itself is subject to CMS quorum consensus. The rollout of that event to every Cassandra node is asynchronous. Those are two different parts of the design.

StageWhat is ordered or checkedWhy it exists
Event submissionThe CMS validates schema, topology, ownership, and in-flight operations against the current metadata.A decommission that breaks a required replication factor, or a conflicting range movement, can be rejected before it changes placement.
CMS commitA quorum of CMS members appends the event and assigns the next epoch.There is one durable order for metadata changes.
Metadata publicationNodes receive the new log entry and derive an immutable ClusterMetadata snapshot.Nodes converge on the same directory, schema, and placements without relying on a race between gossip messages.
Topology-step gateA majority of the relevant pre-change and post-change replica group acknowledges the previous epoch.Read and write quorums retain an overlap while ownership is changing.
Request-time epoch exchangeCoordinators and replicas compare the epoch carried with a mutation or read request.A stale replica plan can be caught up, rejected, or retried rather than being accepted silently.

How TCM bootstraps a node without breaking quorum intersection

CEP-21 describes a four-step bootstrap. Consider a keyspace with RF=2 and existing nodes A, B, and C. A new node X joins with token 150, splitting the range between tokens 100 and 200.

The initial state contains two ranges. (0,100] has replicas {A,B}, while (100,200] has replicas {B,C}. The first event splits the latter range at 150, without changing replica ownership. The next events change write placement before read placement, then retire the superseded write replicas only after the affected nodes have observed the preceding state.

TCM bootstrap diagram showing the four CEP-21 steps. Cassandra splits ranges, adds the joining node to write placements, streams and switches read placements, then retires old write replicas.

The order is the important detail.

  1. Cassandra splits ranges to represent the new token, leaving existing read and write placements unchanged.
  2. It adds X to the write placements for the ranges it will own. Reads still use the old replica groups, while writes are sent to the old replicas and X.
  3. X bootstraps by streaming the required data. Only after that succeeds can Cassandra add X to the read placement and remove the outgoing replica from the read placement.
  4. Cassandra removes the outgoing replica from the write placement after the required acknowledgements show that no coordinator can still read using the older placement without learning about the newer epoch.

Writing to the outgoing replica after it has been removed from read placement can look counterintuitive. It closes the gap between a coordinator that still has an old view and a coordinator that has already learned the new view. Without it, a read quorum based on an older placement and a write quorum based on a newer placement could fail to intersect.

The same pattern applies in reverse during decommission. Cassandra first adds the recipient nodes to write placements, streams from the leaving node, switches read placements, then removes the leaving node from writes and merges equivalent adjacent ranges. Replacement, removal, token movement, and other ownership operations are decomposed into equivalent metadata transitions.

Failure scenarios and recovery behaviour

TCM does not remove the need for careful operations. It changes the evidence available to the operator and gives the system a durable way to resume or compensate for a partially completed change.

ScenarioWhat TCM records or detectsOperational consequence
A coordinator has an old epochA replica response carries a newer epoch than the coordinator used for its plan.The coordinator catches up from the metadata log and checks the plan again. If the plan is no longer sufficient for the requested consistency level, the request fails rather than claiming success against an obsolete placement.
A bootstrap stops after X becomes a write replicaThe metadata log retains the pending bootstrap state and its epoch.Cassandra does not let each node independently erase the pending state based only on local liveness. X must catch up and complete streaming before read placement can change; an unrecoverable operation needs the version-specific recovery procedure.
Two topology changes overlapThe CMS evaluates an event against in-flight range movements.Disjoint operations may proceed concurrently. Operations that would interfere can be rejected until the first movement completes.
A node is down while metadata advancesThe node’s persisted metadata is behind the log tail.On restart it discovers the CMS or a peer, replays the missing events, rebuilds its local metadata snapshot, and then continues from the current state.
A node has a stale schema viewThe coordinator or replica identifies a newer epoch during internode communication and can replay the ordered metadata entries.Cassandra does not rely on independently timed schema pulls and pushes to decide the DDL order. A request that cannot satisfy its consistency requirements with the current metadata fails rather than reporting a success against a conflicting view.
CMS member addresses change while a node is downA persisted address list can be stale even when CMS membership is still valid.CASSANDRA-20476 adds a discovery protocol that builds a temporary address lookup, contacts the current CMS, and allows the agreed address change to be committed and disseminated.
A CMS quorum is unavailableThe CMS cannot commit a new metadata event.Existing stable data traffic has its normal Cassandra behaviour, but new schema or topology changes that need CMS consensus must wait for CMS recovery. Treat this as a cluster-control-plane incident with a version-specific recovery runbook.

TCM request-path diagram showing a coordinator with an old epoch, a replica returning a newer epoch, metadata catch-up, and replica-plan validation before the request can succeed.

The distinction between a failed operation and a failed CMS is useful in practice. A bootstrap can be paused with a durable pending state while the CMS remains healthy. Losing CMS quorum is different because the cluster lacks the quorum required to order the next metadata event. Operators should not use generic resets or ad hoc edits in either case. The correct recovery process depends on the Cassandra version, the state of the metadata log, and the latest published operational procedure.

What Cassandra 6.0 adds around TCM operations

TCM first ships in the Cassandra 6.0 line. It was developed on trunk while that line was numbered 5.1, then became part of the unreleased 6.0 branch. The changes described here cover the cases that become visible once the system is operated through restarts, address changes, recovery work, and real topology maintenance.

  • CASSANDRA-20476 adds CMS member rediscovery for nodes that restart after CMS addresses have changed. It works with node-ID based CMS membership so a node can find the current endpoint mapping before it has caught up.
  • CASSANDRA-20528 adds topology-safe data-centre and rack changes for live nodes, bringing location changes under the same metadata-driven safety model.
  • CASSANDRA-19151 adds an offline cluster metadata tool for inspection and recovery work when a running node is not the right place to diagnose state.
  • CASSANDRA-20525 adds a nodetool command for inspecting the cluster metadata log and directory virtual tables.
  • CASSANDRA-21187 adds repair tooling for inconsistent system.peers and system.peers_v2 information.

These are not commands to run casually. They give an operator more direct visibility into the metadata state that previously had to be inferred from logs, gossip output, peer tables, and the observed symptoms of a failed topology operation.

Initializing CMS after an upgrade

After every node in a rolling upgrade is on Cassandra 6.0, an operator must run nodetool cms initialize on one node. The TCM upgrade is not complete until that step has finished.

From the first Cassandra 6.0 restart until CMS initialization finishes, do not make schema changes, bootstrap, decommission, move, replace, or assassinate nodes. Disable any automation that can perform those operations for the duration. Cassandra’s upgrade guidance treats this as a prohibited-operations window, not a recommendation to defer unrelated maintenance.

CMS starts with one member after initialization, which is not suitable for a real cluster. Use nodetool cms reconfigure to expand it. Cassandra recommends a minimum of three CMS members and three to seven members per data centre, chosen across failure domains rather than on every node. A node that is behind can catch up from metadata-log entries or a metadata snapshot, which can be inspected with nodetool cms snapshot.

Scaling TCM and the CMS

The CMS does not make the entire Cassandra cluster a consensus group. Its members are a smaller subset of nodes that agree on metadata events. That keeps the consensus cost tied to the control plane rather than placing it on every normal read and write.

The data plane still has work to do during a topology change. New nodes must stream. Existing nodes may receive temporary additional writes. Metadata events must reach the affected replica groups, and the next stage waits for the required acknowledgements. Large clusters therefore benefit from careful operation scheduling rather than treating TCM as permission to run unlimited bootstraps, decommissions, or DC changes in parallel.

CEP-21 allows operations on disjoint replica groups to proceed concurrently when the replication invariants are preserved. That gives Cassandra a path to scale topology work without allowing overlapping movements to compromise placement. The practical limit depends on the ranges involved, available streaming bandwidth, disk headroom, compaction pressure, and the ability of the CMS and affected nodes to publish and acknowledge each epoch.

CMS member placement also needs thought. The members must survive the failures the cluster is designed to tolerate and must retain a quorum during maintenance. This is a control-plane availability decision, separate from ordinary replica placement. Making every Cassandra node a CMS member increases the consensus group without improving a normal data request, while too few or poorly distributed CMS members make metadata changes unavailable during a failure domain event.

Benefits and constraints

BenefitWhat it changesConstraint to keep in view
Ordered metadata historyEvery committed topology or schema event has an epoch and immutable position in the log.Operators need to understand whether a node is behind, whether an event is pending, and whether the CMS still has quorum.
Safer range movementRead and write placements change in staged transitions with majority acknowledgement gates.Bootstrap and decommission still need streaming capacity, disk headroom, and monitoring.
Better request-time safetyEpoch exchange exposes a coordinator or replica using an obsolete placement.A request may fail during a metadata race rather than returning a false success. Clients should retain normal retry discipline.
Resumable operationsPending metadata state is durable rather than being inferred independently from local liveness.A permanently failed operation may require a deliberate cancellation or recovery procedure.
More inspectable control planeMetadata log, directory state, and CMS tooling provide concrete evidence during diagnosis.Tool output needs to be captured with the change record and compared to the intended placement.

TCM operational checks

For a topology change, capture the pre-change metadata epoch, CMS membership, keyspace replication settings, intended placements, and relevant system.peers state. During the operation, track streaming, disk headroom, compaction load, CMS and metadata-log events, and the epoch acknowledgements that permit each transition. Afterwards, retain the final metadata view alongside the topology change record.

AxonOps can bring those signals together with the node configuration, logs, disk activity, repair state, compaction backlog, and latency history. That is useful when the question is not merely whether a node reached UN, but whether the cluster took the expected route through the metadata transition and whether the data movement created a new operational problem.

Series

Sources