29
todos/SQ-025-compression-performance.md
Normal file
29
todos/SQ-025-compression-performance.md
Normal file
@@ -0,0 +1,29 @@
|
||||
# SQ-025: Compression & Performance Tuning
|
||||
|
||||
**Status:** `[ ] TODO`
|
||||
**Blocked by:** SQ-024
|
||||
**Priority:** Low
|
||||
|
||||
## Description
|
||||
|
||||
Add zstd compression to S3 segment shipping and create a benchmark suite.
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
- `crates/sq-storage/src/object_store/shipper.rs` - add zstd compression
|
||||
- `crates/sq-storage/src/object_store/reader.rs` - add zstd decompression
|
||||
- `crates/sq-storage/benches/` - throughput benchmarks
|
||||
|
||||
## Benchmarks
|
||||
|
||||
- Write throughput: messages/sec at various payload sizes
|
||||
- Read throughput: messages/sec sequential scan
|
||||
- Compression ratio: raw vs compressed segment size
|
||||
- S3 round-trip: write, ship, trim, read from S3
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] Compressed segments round-trip correctly (write -> compress -> upload -> download -> decompress -> read)
|
||||
- [ ] Compression ratio metrics are logged
|
||||
- [ ] Benchmarks produce readable output
|
||||
- [ ] No correctness regressions (all existing tests still pass)
|
||||
Reference in New Issue
Block a user