Columnar SSTable Experiments in Apache Cassandra (Part 2)

OK, curiosity got the better of me. There are so many combinations of compression settings and row-group sizes to try that I wanted to see how they affect our queries.

In Part 1 of our columnar SSTable experiments the custom format reduced table storage by about two thirds compared with BTI while using Zstd on the encoded columns and again on the serialized SSTable stream. This time I wanted to test removing one of those two Zstd layers and measure the effect on storage and CPU use as well as query throughput and latency. I also tried larger row groups to see whether they helped longer reads at the expense of short queries.

We’ve now moved the work to our Cassandra 6.0 fork and tested six compression combinations against the same 1.21 billion metric points. Cassandra and the load client each have their own server this time so the client no longer limits ingestion before we can see the differences between formats.

For this numerical time-series workload we saw no substantial degradation in query performance with columnar or Parquet alongside 66% less table storage with our selected columnar combination and 57% less with Parquet. Short reads stayed close to BTI and longer range queries were faster in these runs though each combination was tested only once.

The Table and Workload

This experiment uses the same table as Part 1 for time series with one floating-point reading per timestamp. Every storage and performance figure comes from this table with synthetic numerical readings sampled every five seconds so the results do not establish how these formats would behave with text payloads or wider tables.

CREATE TABLE metrics.v2metrics5 (
    orgid text,
    clusterhash bigint,
    metricid bigint,
    time int,
    value float,
    PRIMARY KEY ((orgid, clusterhash, metricid), time)
) WITH CLUSTERING ORDER BY (time DESC);

Each partition holds one metric series identified by (orgid, clusterhash, metricid). Within that partition the time column holds Unix timestamps in seconds and each row has one 32-bit floating-point value with the newest readings first.

Compression and Encoding

For these numerical columns we use established lossless encodings developed for time-series and columnar storage. The custom columnar file format encodes time with delta-of-delta following the timestamp approach described in the Gorilla paper and encodes value with ALP (Adaptive Lossless Floating-Point) by Azim Afroozeh and colleagues.

For the Parquet file format we used its standard byte stream split encoding for the floating-point values with Zstd page compression. The benchmark table definition includes a Cassandra compressor setting but the Parquet writer does not apply that outer compression.

The data-layout diagrams in Part 1 show how readings become consecutive encoded sections in a row group. Both formats still preserve Cassandra’s write timestamps and deletion metadata for normal CQL reads.

Compression Layers

Cassandra’s outer compression operates on fixed-size chunks of the serialized byte stream. Our tests use 16 KiB uncompressed chunks so this does not mean compressing an entire file as a single block. Chunk boundaries do not have to align with columns or row groups.

The custom columnar format can also compress each encoded column chunk inside a row group. It keeps the compressed version only when it is smaller than the encoded input.

The outer layer can compress metadata as well as the encoded column data because it sees the whole serialized stream. The inner layer sees only the individual chunks selected for per-column compression. On reads the outer block must be decompressed before any inner-compressed chunks inside it can be decompressed and their values decoded.

ALP and SIMD

Our ALP encoder uses the JDK 21 Vector API through jdk.incubator.vector to try candidate decimal exponents and check that values round-trip exactly. SIMD instructions let it perform those checks on several values at once.

The server used here has AVX2 with 256-bit vectors. The implementation can use 512-bit vectors where AVX-512 is available but we haven’t measured that hardware in this experiment. ALP decoding is currently scalar so the read results do not include a vectorized decoder.

Compression Combinations

We tested BTI against four columnar compression combinations and Parquet to compare compression layers and row-group sizes. The letters A to F identify these combinations in the results below.

B and C isolate the effect of per-column compression when outer compression is already enabled. Comparing B with D isolates the outer layer at the same row-group size. D and E then compare group sizes with only per-column compression.

The Zstd dictionary section below covers an additional BTI run with dictionary compression.

Still 66% Less Table Storage

Our selected columnar combination C stored the same 1.21 billion metric points in 6.13 GB instead of BTI’s 18.15 GB after compaction. That is about one third of the table storage for this narrow metrics workload.

Combination C uses 128-row groups with outer Zstd compression and no per-column compression. The query results compare this setup with BTI on reads ranging from the latest point to seven days of data.

Query Throughput and Latency

This single-partition query uses ? bind markers for the partition-key values and time bounds supplied by the client.

SELECT time, value FROM metrics.v2metrics5
WHERE orgid = ? AND clusterhash = ? AND metricid = ?
  AND time >= ? AND time < ?;

For the one-hour test the client supplies time bounds 3,600 seconds apart and our dataset has one reading every five seconds so this query returns 720 rows. The five-minute test uses a shorter range while the 24-hour and seven-day tests return 17,280 rows and 120,960 rows from the selected partition.

These are end-to-end CQL measurements with 32 concurrent readers against the same dataset on the same server. Each query type ran for 10 seconds of warm-up followed by 30 seconds of measurement with a randomly selected series for each request. We measured each combination once and did not control the page-cache state.

1-hour range query throughput

Queries per second · higher is better

A · BTI11,661 queries/s

C · Columnar14,209 queries/s

08,00016,000 queries/s

1-hour range query median latency

Milliseconds · lower is better

A · BTI2.44 ms

C · Columnar2.10 ms

01.53 ms

The table below compares BTI with columnar C across all five query ranges. Latency is shown as p50 / p99 in milliseconds with p50 representing the median.

This columnar setup completed about 6% fewer latest-point queries and about 2% fewer five-minute queries per second while the short-read latencies were close to BTI. From an hour onwards it completed 20% to 22% more queries per second with lower median and p99 latency in this run.

Ingestion with this setup was about 6% slower with 7% more CPU time than BTI. The storage saving remains the largest change for this workload and the read results need repeated runs before we can rely on those differences.

Benchmark Setup

Changes Since Part 1

We ported the formats from our 5.0 fork to a build based on the cassandra-6.0-rustyrazorblade branch. Jon Haddad and Patrick McFadin continue to put a great deal of work into Cassandra performance and we’re grateful to be building on their fork. Our additions are separate from that public branch and we’ll publish them once they are ready for review.

When we first repeated the shared-VM setup on 6.0 the client limited most combinations to roughly 975,000 points per second. Giving Cassandra and the client separate servers let us measure ingestion with the Cassandra node’s CPU fully occupied.

The experimental sstable_format_parameters table option now lets us choose the row-group size per table. We used it to compare 128-row and 512-row groups without creating another named format entry in cassandra.yaml. We also increased read concurrency from 16 to 32 because the extra network round trip was limiting the shorter queries.

Hardware and Workload

The two VMs share a network with a measured round-trip time of 0.42 ms and about 7 Gbit/s of bandwidth between them. Cassandra could use all 16 vCPUs without core pinning.

• Data volume was seven days for 10,000 series at one point every five seconds for a total of 1,209,600,000 points.
• Writes used unlogged batches of 12 points within one partition with 256 requests in flight. Each point had its own write timestamp and a TTL of 30 days minus its age.
• Concurrent reads during ingestion consisted of 20 single-series queries per second with each query returning 720 readings over a one-hour range.
• Before the read tests we flushed the memtables and waited for pending compactions to finish.

The node used cassandra_latest.yaml as its base with offheap_objects memtables and periodic commit-log sync. Compaction ran with eight concurrent compactors and a throughput limit of 64 MiB/s. BTI and custom columnar flushes used LZ4 before compaction applied each table’s selected compression. Parquet used its own page compression.

Query Performance

Each query type ran for 10 seconds of warm-up followed by 30 seconds of measurement with a randomly selected series for each request. The range queries use the partition key and time bounds shown above while the latest-point query relies on the descending clustering order to return the newest reading.

SELECT time, value FROM metrics.v2metrics5
WHERE orgid = ? AND clusterhash = ? AND metricid = ?
LIMIT 1;

Query	Points returned per query
Latest point	1
5-minute range	60
1-hour range	720
24-hour range	17,280
7-day range	120,960

At 16 concurrent readers the latest-point and five-minute queries used only three to five of Cassandra’s 16 vCPUs. Doubling concurrency roughly doubled their throughput which is consistent with the client waiting for responses rather than keeping the server busy. Longer queries already used about 12 vCPUs and gained another 19% to 35% at 32 readers.

All reported read results therefore use 32 concurrent readers. They are end-to-end CQL measurements which include the server’s work and delivery of the returned rows over the network.

Table Settings

These options belong to our unpublished implementation. The named formats are configured in cassandra.yaml while BTI remains the default.

storage_compatibility_mode: NONE

sstable:
  selected_format: bti
  configurations:
    columnar:
      format: columnar
    parquet_bss_512:
      format: parquet
      parameters:
        float_encoding: byte_stream_split
        row_group_rows: '512'
        iceberg_mirror: 'true'

All six combinations share the columns and compaction settings below. This shared definition stops at default_time_to_live so that one of the A-F clauses below completes the CREATE TABLE statement for that run.

CREATE TABLE metrics.v2metrics5 (
    orgid text,
    clusterhash bigint,
    metricid bigint,
    time int,
    value float,
    PRIMARY KEY ((orgid, clusterhash, metricid), time)
) WITH CLUSTERING ORDER BY (time DESC)
    AND compaction = {
        'class': 'TimeWindowCompactionStrategy',
        'compaction_window_size': '1',
        'compaction_window_unit': 'DAYS',
        'max_threshold': '32',
        'min_threshold': '4'
    }
    AND default_time_to_live = 2592000

The benchmark also enabled unsafe_aggressive_sstable_expiration. That option skips SSTable overlap safety checks and is omitted from the shared definition above because it is not a general recommendation. These examples show the format variations rather than a complete benchmark reproduction package.

Each clause below belongs to a separate run of the same table. They are alternatives for creating a new table rather than successive changes to a populated table.

sstable_format_parameters overrides the parameters of the selected named format entry. Our columnar format accepts row_group_rows and row_group_size_in_bytes so tables can vary these without another entry in cassandra.yaml. Changes apply to subsequently written SSTables and do not rewrite existing files immediately.

A. BTI

The table uses BTI from selected_format: bti in cassandra.yaml with Zstd applied to 16 KiB chunks of the serialized SSTable data.

    AND compression = {'class': 'ZstdCompressor', 'chunk_length_in_kb': '16'};

B. Columnar with Both Zstd Layers

This keeps the two-layer arrangement from Part 1 with Zstd on the encoded column chunks and on the outer byte stream using the columnar format’s default 128-row groups.

    AND compression = {'class': 'ZstdCompressor', 'chunk_length_in_kb': '16'}
    AND sstable_format = 'columnar'
    AND column_encodings = {'time': 'delta_of_delta', 'value': 'alp'}
    AND column_compression = {'time': 'zstd', 'value': 'zstd'};

C. Columnar with Outer Zstd Only

This new table keeps outer Zstd and 128-row groups while omitting column_compression so the encoded column chunks are not compressed separately.

    AND compression = {'class': 'ZstdCompressor', 'chunk_length_in_kb': '16'}
    AND sstable_format = 'columnar'
    AND column_encodings = {'time': 'delta_of_delta', 'value': 'alp'};

D. Columnar with Per-Column Zstd Only

Setting compression to {'enabled': 'false'} removes the outer layer while retaining per-column Zstd and the default 128-row groups.

    AND compression = {'enabled': 'false'}
    AND sstable_format = 'columnar'
    AND column_encodings = {'time': 'delta_of_delta', 'value': 'alp'}
    AND column_compression = {'time': 'zstd', 'value': 'zstd'};

E. Per-Column Zstd with 512-Row Groups

This uses the same compression settings as D but overrides row_group_rows at table level to test 512-row groups.

    AND compression = {'enabled': 'false'}
    AND sstable_format = 'columnar'
    AND column_encodings = {'time': 'delta_of_delta', 'value': 'alp'}
    AND column_compression = {'time': 'zstd', 'value': 'zstd'}
    AND sstable_format_parameters = {'row_group_rows': '512'};

F. Parquet with Zstd Pages

The parquet_bss_512 entry in cassandra.yaml supplies 512-row groups with byte stream split for floating-point values. The compression clause was present in the benchmark DDL but is ignored by this writer so Parquet uses only Zstd page compression.

    AND compression = {'class': 'ZstdCompressor', 'chunk_length_in_kb': '16'}
    AND sstable_format = 'parquet_bss_512'
    AND column_encodings = {'time': 'delta_of_delta', 'value': 'byte_stream_split'}
    AND column_compression = {'time': 'zstd', 'value': 'zstd'};

The Parquet writer maps the timestamp encoding request to Parquet’s DELTA_BINARY_PACKED encoding as described in Part 1. It does not write our custom delta-of-delta representation into Parquet pages.

Results Across All Six Combinations

Disk Usage

Table sizes were measured after ingestion and the final flush and compactions. Peak size is the largest table size recorded during ingestion.

B and C differ by only 0.01 GB at the reported precision. Delta-of-delta and ALP leave little repetition within an individual encoded column chunk so the optional per-column compressor rarely produces a smaller result.

Keeping outer compression reduced storage by about 13% compared with D. The encoded values are only part of the file because each group also contains row flags and liveness information alongside TTLs and chunk headers. Outer compression can find repetition across these sections that a compressor working on one small column chunk cannot see. We still need to measure how much of the saving belongs to each metadata field.

Increasing the row-group size from 128 to 512 without outer compression made E about 9% smaller than D. A larger group spreads its headers over more rows but also means decoding a larger group when a query needs only a few values.

Read Throughput

All results use 32 concurrent readers and show completed queries per second. A is BTI and F is Parquet while B to E are the four columnar combinations.

The latest-point query was fastest with BTI while D had the highest five-minute throughput. Relative to BTI the shortest-query results were within about 7% in either direction so there isn’t a consistent short-read benefit here.

For queries returning 720 rows over a one-hour range every columnar combination and Parquet completed 11% to 22% more queries per second. The advantage increased to 16% to 25% for 17,280-row reads and 18% to 24% for 120,960-row reads.

BTI reconstructs individual rows and cells while the columnar readers decode groups of values together. The files are also substantially smaller but we did not measure physical device reads separately from cached reads. These results don’t tell us how much of the improvement came from decoding versus I/O.

Points Returned per Second

Multiplying completed queries by the number of returned points gives the following rates in millions of points per second.

The rates for queries returning 17,280 and 120,960 rows are close for columnar and Parquet. With one run per combination we shouldn’t treat differences of a few percent as a reliable ranking.

Read Latency

Each cell below gives p50 / p99 in milliseconds at the same concurrency of 32.

With 32 readers sharing eight physical cores these latencies include contention and are not isolated single-query timings. Our selected columnar combination reduced median latency from 62.8 ms to 51.3 ms for 17,280-row reads and from 430 ms to 358 ms for 120,960-row reads.

Ingestion and CPU

The node’s CPU remained busy throughout ingestion in all six combinations. BTI ingested the most points per second and used the least CPU time for the full load.

CPU-seconds are accumulated process CPU time during ingestion rather than elapsed time or a utilisation percentage. Final compaction is the elapsed time needed to finish pending compactions after ingestion.

The custom columnar combinations used 7% to 17% more ingestion CPU than BTI and loaded 5% to 8% fewer points per second. Parquet needed 37% more CPU and loaded about 20% fewer points per second.

We haven’t profiled the extra CPU cost yet. Both experimental formats encode columns during flush and again during compaction which adds work to those paths. They finished the remaining compactions in less than half BTI’s time but that depends on the files left at the end of ingestion as well as the smaller byte volume.

Write Latency

These latencies are for the client’s batch requests.

Combination C’s median write latency was close to BTI while its p99 rose from 7.05 ms to 8.34 ms. Parquet’s p99 was just over twice BTI’s which would need attention before choosing it for a write-heavy workload.

Zstd Dictionary Compression

Replacing ordinary Zstd on BTI with Cassandra 6.0’s dictionary compressor did not reduce disk usage for this numerical time-series table.

The run included some SSTables without a dictionary but the fully dictionary-compressed daily windows also showed no reduction.

Zstd dictionaries help when the same byte sequences recur across compression chunks. We suspect ordinary Zstd already finds most of the repeated row structure inside each 16 KiB chunk while the full-precision floats and jittered write timestamps leave little extra repetition for a dictionary to exploit. That explanation still needs testing with smaller chunks but these results give us no reason to use dictionary compression for this numerical dataset.

Part 3 tests columnar storage with text and ASCII columns because repeated strings provide a different test for compression than the timestamps and floating-point readings used here.

Conclusions

The summary below compares the storage and performance of BTI with columnar C and Parquet.

Parquet used 57% less storage than BTI but ingestion was about 20% slower and needed 37% more CPU time. Its longer-range read throughput was close to the custom columnar combinations.

For the next round of tests we’d use C with outer Zstd and 128-row groups because it retained the largest storage saving with ingestion and CPU use close to BTI. B’s extra per-column compression didn’t reduce storage meaningfully or give a consistent read improvement so there is little reason to keep that extra layer for this dataset.

Without outer compression the table was about 15% larger with 128-row groups or 4.6% larger with 512-row groups than C. Increasing the group size recovered some space but E’s shortest reads were slower than D’s while its day and week queries were faster. We’d want repeated runs before choosing a row-group size on those read differences.

Parquet’s longer-range reads were close to the custom format but it occupied 26% more storage than C and needed substantially more ingestion CPU. Its compatibility with external readers may justify that cost for some uses. Those readers still need explicit handling of Cassandra’s expiry and deletion rules as discussed in Part 1.

BTI ingested about 6% more points per second than C and had lower p99 write latency while C returned longer ranges faster. For our monitoring workload we’re primarily interested in storing the same history in much less space with relatively little change to ingestion throughput and CPU requirements.

Further Testing

These results give us a better starting combination but several questions still need measurements.

• Repeat the runs to establish variation before relying on small differences between combinations.
• Control page-cache state because the dataset fits in the server’s memory and restarting Cassandra before the reads did not establish cold-cache conditions.
• Test columnar storage with text and ASCII columns in the next blogs alongside wider tables and real metric samples. One float column with synthetic values cannot represent every Cassandra workload.
• Extend the dictionary tests with smaller BTI chunks and different training samples.
• Test different CPU types and core counts because this server has eight physical cores with AVX2. Wider vectors or more cores may change the encoding cost.
• Vectorize ALP decoding and measure whether it improves range reads without increasing the cost of short queries.
• Investigate metadata storage per partition to see whether reducing repeated row-group metadata can recover some of the saving currently provided by outer compression.
• Run multi-node repair and streaming tests before drawing conclusions about cluster-wide behaviour from the single-node results.

For this narrow metrics workload we could store the same 1.21 billion points in roughly one third of the space while keeping ingestion and CPU use close to BTI. Adding a trained dictionary to BTI did not reduce storage in the additional run so the disk saving remains the main reason to keep developing the columnar format. We still need repeated runs with wider tables and production-like data before extending that conclusion to other Cassandra workloads.