feat: add initial

Signed-off-by: kjuulh <contact@kjuulh.io>
This commit is contained in:
2026-02-26 21:52:50 +01:00
commit 3162971c89
48 changed files with 3041 additions and 0 deletions

View File

@@ -0,0 +1,29 @@
# SQ-018: S3 Read Fallback
**Status:** `[ ] TODO`
**Blocked by:** SQ-017
**Priority:** Medium
## Description
When a consumer requests an offset from a trimmed segment, fetch it from S3 instead.
## Files to Create/Modify
- `crates/sq-storage/src/engine.rs` - update read path with S3 fallback
- `crates/sq-storage/src/object_store/reader.rs` - download + decompress segment from S3
## Behavior
1. Read path checks local WAL first
2. If segment not found locally, check if it's in S3 via the index
3. Download segment from S3, decompress (zstd)
4. Read messages from the downloaded segment
5. Optionally cache downloaded segment locally for subsequent reads
## Acceptance Criteria
- [ ] Write messages, ship to S3, trim locally, read from S3 -> messages correct
- [ ] CRC validation on S3-fetched data passes
- [ ] S3 fetch failure: returns appropriate error
- [ ] Performance: subsequent reads of same trimmed segment use local cache