sq/todos/SQ-025-compression-performance.md

SQ-025: Compression & Performance Tuning

Status: [ ] TODO Blocked by: SQ-024 Priority: Low

Description

Add zstd compression to S3 segment shipping and create a benchmark suite.

Files to Create/Modify

• crates/sq-storage/src/object_store/shipper.rs - add zstd compression
• crates/sq-storage/src/object_store/reader.rs - add zstd decompression
• crates/sq-storage/benches/ - throughput benchmarks

Benchmarks

• Write throughput: messages/sec at various payload sizes
• Read throughput: messages/sec sequential scan
• Compression ratio: raw vs compressed segment size
• S3 round-trip: write, ship, trim, read from S3

Acceptance Criteria

• Compressed segments round-trip correctly (write -> compress -> upload -> download -> decompress -> read)
• Compression ratio metrics are logged
• Benchmarks produce readable output
• No correctness regressions (all existing tests still pass)