Measuring Compression Impact on Storage and Query Latency

The Symptom

The telemetry collection has grown to 68 GB on disk using Snappy compression. Storage costs are increasing linearly with data volume. The team wants to reduce storage without significantly impacting query performance.

The Cause

Snappy prioritizes speed over compression ratio. For the telemetry data (repetitive numeric values, repeated field names, similar document structures), a better algorithm can achieve significantly higher compression ratios because the data has high redundancy within pages.

The Benchmark

Create identical collections with different compression algorithms and measure storage and query performance:

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.MICROSECONDS)
@Warmup(iterations = 2, time = 10)
@Measurement(iterations = 3, time = 15)
@Fork(1)
@State(Scope.Benchmark)
public class CompressionBenchmark {

    private MongoCollection<Document> snappyCollection;
    private MongoCollection<Document> zstdCollection;
    private MongoCollection<Document> noneCollection;

    @Setup
    public void setup() {
        MongoClient client = MongoClients.create("mongodb://localhost:27017");
        var db = client.getDatabase("compression_bench");

        snappyCollection = db.getCollection("readings_snappy");
        zstdCollection = db.getCollection("readings_zstd");
        noneCollection = db.getCollection("readings_none");
    }

    @Benchmark
    public List<Document> readSnappy() {
        return snappyCollection.find(
            Filters.eq("sensorId", "sensor-00042")
        ).sort(Sorts.descending("ts"))
         .limit(100)
         .into(new ArrayList<>());
    }

    @Benchmark
    public List<Document> readZstd() {
        return zstdCollection.find(
            Filters.eq("sensorId", "sensor-00042")
        ).sort(Sorts.descending("ts"))
         .limit(100)
         .into(new ArrayList<>());
    }

    @Benchmark
    public List<Document> readNone() {
        return noneCollection.find(
            Filters.eq("sensorId", "sensor-00042")
        ).sort(Sorts.descending("ts"))
         .limit(100)
         .into(new ArrayList<>());
    }
}

Storage comparison for 10 million identical documents:

Compression	Data size	Index size	Total	Compression ratio
None	3.4 GB	480 MB	3.88 GB	1.0x
Snappy	1.36 GB	480 MB	1.84 GB	2.5x (data only)
Zstd	0.68 GB	480 MB	1.16 GB	5.0x (data only)

Query latency comparison (100 documents, warm cache):

Benchmark                           Mode  Cnt    Score    Error  Units
CompressionBenchmark.readNone      avgt    3  280.000 ± 15.000  us/op
CompressionBenchmark.readSnappy    avgt    3  290.000 ± 18.000  us/op
CompressionBenchmark.readZstd      avgt    3  295.000 ± 16.000  us/op

With a warm cache, compression has near-zero impact on read latency because data is decompressed in cache. The 10-15us difference is within the error margin.

Query latency with cold cache (pages must be read from disk and decompressed):

Compression	p50 (cold)	p95 (cold)	p99 (cold)
None	1.2ms	3.5ms	8.0ms
Snappy	1.1ms	3.2ms	7.5ms
Zstd	1.0ms	2.8ms	6.5ms

Compressed data is actually faster on cold reads because less data is read from disk. Zstd reads 5x less data, and the decompression time (negligible on modern CPUs) is less than the I/O time saved.

The Fix

Switch the telemetry collection to Zstd compression. This requires recreating the collection or using compact with the new settings:

// Option 1: compact with new compression (MongoDB 4.4+)
db.runCommand({
  compact: "readings",
  force: true
  // Note: compact does NOT change compression. Must recreate collection.
})

// Option 2: Create new collection, copy data
db.createCollection("readings_new", {
  storageEngine: {
    wiredTiger: { configString: "block_compressor=zstd" }
  }
})

// Copy data in batches
var batch = [];
db.readings.find().forEach(doc => {
  batch.push(doc);
  if (batch.length >= 10000) {
    db.readings_new.insertMany(batch);
    batch = [];
  }
});
if (batch.length > 0) db.readings_new.insertMany(batch);

// Rename
db.readings.renameCollection("readings_old");
db.readings_new.renameCollection("readings");

For new collections, set Zstd as the default:

// FAST: Create new collections with Zstd compression
database.createCollection("readings",
    new CreateCollectionOptions().storageEngineOptions(
        new Document("wiredTiger",
            new Document("configString", "block_compressor=zstd"))
    ));

The Proof

After migrating the telemetry collection from Snappy to Zstd:

Metric	Snappy	Zstd
Data file size	68 GB	27 GB
Storage reduction	baseline	60%
Read p50 (warm cache)	8ms	8ms
Read p99 (cold cache)	7.5ms	6.5ms
Write p50	3ms	3.5ms
Write p99	15ms	18ms
CPU during checkpoint	12%	18%

The Trade-off

Zstd uses more CPU during writes (checkpoint and eviction) because compression is CPU-intensive. The 6% CPU increase during checkpoints is acceptable on a 16-core server but may be problematic on a 4-core instance. Write p99 increased from 15ms to 18ms due to the compression overhead during page writes.

For write-heavy workloads on CPU-constrained servers, Snappy remains the better choice. For read-heavy or storage-constrained workloads, Zstd provides significant storage savings with minimal read latency impact.

Index compression is separate from data compression. WiredTiger uses prefix compression for indexes by default, which compresses common key prefixes. This is orthogonal to block compression and should always be enabled.