30 lines
942 B
Markdown
30 lines
942 B
Markdown
# SQ-018: S3 Read Fallback
|
|
|
|
**Status:** `[x] DONE`
|
|
**Blocked by:** SQ-017
|
|
**Priority:** Medium
|
|
|
|
## Description
|
|
|
|
When a consumer requests an offset from a trimmed segment, fetch it from S3 instead.
|
|
|
|
## Files to Create/Modify
|
|
|
|
- `crates/sq-storage/src/engine.rs` - update read path with S3 fallback
|
|
- `crates/sq-storage/src/object_store/reader.rs` - download + decompress segment from S3
|
|
|
|
## Behavior
|
|
|
|
1. Read path checks local WAL first
|
|
2. If segment not found locally, check if it's in S3 via the index
|
|
3. Download segment from S3, decompress (zstd)
|
|
4. Read messages from the downloaded segment
|
|
5. Optionally cache downloaded segment locally for subsequent reads
|
|
|
|
## Acceptance Criteria
|
|
|
|
- [ ] Write messages, ship to S3, trim locally, read from S3 -> messages correct
|
|
- [ ] CRC validation on S3-fetched data passes
|
|
- [ ] S3 fetch failure: returns appropriate error
|
|
- [ ] Performance: subsequent reads of same trimmed segment use local cache
|