sq/todos/SQ-018-s3-read-fallback.md

SQ-018: S3 Read Fallback

Status: [ ] TODO Blocked by: SQ-017 Priority: Medium

Description

When a consumer requests an offset from a trimmed segment, fetch it from S3 instead.

Files to Create/Modify

• crates/sq-storage/src/engine.rs - update read path with S3 fallback
• crates/sq-storage/src/object_store/reader.rs - download + decompress segment from S3

Behavior

1. Read path checks local WAL first
2. If segment not found locally, check if it's in S3 via the index
3. Download segment from S3, decompress (zstd)
4. Read messages from the downloaded segment
5. Optionally cache downloaded segment locally for subsequent reads

Acceptance Criteria

• Write messages, ship to S3, trim locally, read from S3 -> messages correct
• CRC validation on S3-fetched data passes
• S3 fetch failure: returns appropriate error
• Performance: subsequent reads of same trimmed segment use local cache