sq/todos/SQ-016-object-store-shipping.md

SQ-016: Object Store Shipping

Status: [x] DONE Blocked by: SQ-007 Priority: Medium

Description

Background process that ships closed WAL segments to S3-compatible object storage for long-term durability.

Files to Create/Modify

• crates/sq-storage/src/object_store/mod.rs - ObjectStore trait + S3 impl + Noop impl
• crates/sq-storage/src/object_store/shipper.rs - SegmentShipper as notmad::Component
• crates/sq-storage/src/object_store/layout.rs - S3 key naming convention

ObjectStore Trait

pub trait ObjectStore: Send + Sync {
    async fn put(&self, key: &str, data: Vec<u8>) -> Result<()>;
    async fn get(&self, key: &str) -> Result<Vec<u8>>;
    async fn list(&self, prefix: &str) -> Result<Vec<String>>;
    async fn delete(&self, key: &str) -> Result<()>;
}

S3 Key Layout

{cluster_id}/{topic}/{partition}/{base_offset}-{end_offset}.sqseg

Acceptance Criteria

• Closed segment is detected and uploaded to object store
• S3 key matches expected layout
• Noop object store works for testing (stores in memory)
• Upload failure: segment stays local, retried on next cycle
• Successful upload is recorded (segment marked as "shipped")
• Uses zstd compression before upload

Notes

• Uses object_store crate with AWS S3 features (same as nostore)
• Shipper runs as a notmad::Component in the background
• Poll interval: every 5 seconds check for closed segments