Network Compression and End-to-End Payload Optimization

The Symptom

The telemetry platform’s application servers are in a different availability zone from the MongoDB servers. Network monitoring shows 450 Mbps of traffic between the application and database tiers during peak load. The inter-AZ bandwidth limit is 5 Gbps, but the 450 Mbps contributes $1,800/month in data transfer costs. Additionally, the network latency between AZs is 1.5ms, and larger payloads take proportionally longer to transfer.

The Cause

The MongoDB wire protocol sends BSON documents uncompressed by default. Each query response with 100 documents at 340 bytes each sends 34 KB of uncompressed BSON. At 1,000 queries/sec, that is 34 MB/sec of query responses alone. Add write acknowledgments, cursor batches, and keepalives: 450 Mbps total.

The Benchmark

Test network compression with different compressors:

// Configure driver with network compression
MongoClientSettings snappySettings = MongoClientSettings.builder()
    .applyConnectionString(connString)
    .compressorList(List.of(
        MongoCompressor.createSnappyCompressor()
    ))
    .build();

MongoClientSettings zstdSettings = MongoClientSettings.builder()
    .applyConnectionString(connString)
    .compressorList(List.of(
        MongoCompressor.createZstdCompressor()
    ))
    .build();

MongoClientSettings zlibSettings = MongoClientSettings.builder()
    .applyConnectionString(connString)
    .compressorList(List.of(
        MongoCompressor.createZlibCompressor()
            .withProperty(MongoCompressor.LEVEL, 6)
    ))
    .build();

Results for 100-document query response (34 KB uncompressed):

Compressor	Compressed size	Ratio	Compression time	Decompression time
None	34 KB	1.0x	0	0
Snappy	15.3 KB	2.2x	0.02ms	0.01ms
Zstd	8.9 KB	3.8x	0.08ms	0.02ms
Zlib-6	9.5 KB	3.6x	0.15ms	0.05ms

End-to-end query latency (same AZ, 0.2ms RTT):

Benchmark                           Mode  Cnt    Score    Error  Units
NetworkCompressionBench.noCompression  avgt  5  310.000 ± 12.000  us/op
NetworkCompressionBench.snappy        avgt    5  295.000 ± 10.000  us/op
NetworkCompressionBench.zstd          avgt    5  285.000 ± 11.000  us/op
NetworkCompressionBench.zlib          avgt    5  305.000 ± 14.000  us/op

Same AZ: negligible difference. The payload is small and network transit time is minimal.

End-to-end query latency (cross-AZ, 1.5ms RTT):

Compressor	p50	p95	p99
None	5.2ms	8.5ms	15ms
Snappy	4.5ms	7.2ms	12ms
Zstd	4.0ms	6.5ms	10ms
Zlib	4.2ms	7.0ms	11ms

Cross-AZ: Zstd reduces p99 by 33%. The smaller payload transfers faster across the 1.5ms RTT link.

The Fix

Enable Zstd network compression in the MongoDB driver:

// FAST: Enable Zstd network compression
MongoClientSettings settings = MongoClientSettings.builder()
    .applyConnectionString(connString)
    .compressorList(List.of(
        MongoCompressor.createZstdCompressor(),
        MongoCompressor.createSnappyCompressor()  // Fallback
    ))
    .applyToConnectionPoolSettings(builder -> builder
        .maxSize(200)
        .minSize(20)
    )
    .build();

The driver negotiates compression with the server during the handshake. The first compressor in the list is preferred. If the server does not support Zstd (pre-4.2), it falls back to Snappy.

Enable on the server side:

# mongod.conf
net:
  compression:
    compressors: zstd,snappy

The Proof

After enabling Zstd network compression:

Metric	No compression	Zstd compression
Network bandwidth (peak)	450 Mbps	140 Mbps
Monthly data transfer cost	$1,800	$560
Cross-AZ query p99	15ms	10ms
Driver CPU overhead	baseline	+3%
Server CPU overhead	baseline	+2%

The Trade-off

Network compression adds CPU overhead on both the driver (compression) and the server (decompression for incoming requests, compression for outgoing responses). The 5% total CPU increase is negligible on modern multi-core servers.

For same-datacenter deployments with sub-millisecond RTT, network compression provides minimal latency improvement. The CPU cost may outweigh the benefit. Enable it only if storage bandwidth is constrained or data transfer costs are significant.

For cross-region deployments (10-100ms RTT), network compression provides substantial latency improvement by reducing the bytes on the wire. The compression/decompression time (0.02-0.08ms) is dwarfed by the RTT savings.

The compressor list is negotiated per connection, not per operation. All operations on a connection use the same compressor. You cannot compress specific queries and leave others uncompressed. If compression overhead matters for specific operations, consider using separate MongoClient instances: one with compression for batch operations and one without for latency-sensitive operations.