29
todos/SQ-018-s3-read-fallback.md
Normal file
29
todos/SQ-018-s3-read-fallback.md
Normal file
@@ -0,0 +1,29 @@
|
||||
# SQ-018: S3 Read Fallback
|
||||
|
||||
**Status:** `[ ] TODO`
|
||||
**Blocked by:** SQ-017
|
||||
**Priority:** Medium
|
||||
|
||||
## Description
|
||||
|
||||
When a consumer requests an offset from a trimmed segment, fetch it from S3 instead.
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
- `crates/sq-storage/src/engine.rs` - update read path with S3 fallback
|
||||
- `crates/sq-storage/src/object_store/reader.rs` - download + decompress segment from S3
|
||||
|
||||
## Behavior
|
||||
|
||||
1. Read path checks local WAL first
|
||||
2. If segment not found locally, check if it's in S3 via the index
|
||||
3. Download segment from S3, decompress (zstd)
|
||||
4. Read messages from the downloaded segment
|
||||
5. Optionally cache downloaded segment locally for subsequent reads
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] Write messages, ship to S3, trim locally, read from S3 -> messages correct
|
||||
- [ ] CRC validation on S3-fetched data passes
|
||||
- [ ] S3 fetch failure: returns appropriate error
|
||||
- [ ] Performance: subsequent reads of same trimmed segment use local cache
|
||||
Reference in New Issue
Block a user