46 lines
1.4 KiB
Markdown
46 lines
1.4 KiB
Markdown
# SQ-016: Object Store Shipping
|
|
|
|
**Status:** `[x] DONE`
|
|
**Blocked by:** SQ-007
|
|
**Priority:** Medium
|
|
|
|
## Description
|
|
|
|
Background process that ships closed WAL segments to S3-compatible object storage for long-term durability.
|
|
|
|
## Files to Create/Modify
|
|
|
|
- `crates/sq-storage/src/object_store/mod.rs` - ObjectStore trait + S3 impl + Noop impl
|
|
- `crates/sq-storage/src/object_store/shipper.rs` - SegmentShipper as notmad::Component
|
|
- `crates/sq-storage/src/object_store/layout.rs` - S3 key naming convention
|
|
|
|
## ObjectStore Trait
|
|
|
|
```rust
|
|
pub trait ObjectStore: Send + Sync {
|
|
async fn put(&self, key: &str, data: Vec<u8>) -> Result<()>;
|
|
async fn get(&self, key: &str) -> Result<Vec<u8>>;
|
|
async fn list(&self, prefix: &str) -> Result<Vec<String>>;
|
|
async fn delete(&self, key: &str) -> Result<()>;
|
|
}
|
|
```
|
|
|
|
## S3 Key Layout
|
|
|
|
`{cluster_id}/{topic}/{partition}/{base_offset}-{end_offset}.sqseg`
|
|
|
|
## Acceptance Criteria
|
|
|
|
- [ ] Closed segment is detected and uploaded to object store
|
|
- [ ] S3 key matches expected layout
|
|
- [ ] Noop object store works for testing (stores in memory)
|
|
- [ ] Upload failure: segment stays local, retried on next cycle
|
|
- [ ] Successful upload is recorded (segment marked as "shipped")
|
|
- [ ] Uses zstd compression before upload
|
|
|
|
## Notes
|
|
|
|
- Uses `object_store` crate with AWS S3 features (same as nostore)
|
|
- Shipper runs as a notmad::Component in the background
|
|
- Poll interval: every 5 seconds check for closed segments
|