45
todos/SQ-016-object-store-shipping.md
Normal file
45
todos/SQ-016-object-store-shipping.md
Normal file
@@ -0,0 +1,45 @@
|
||||
# SQ-016: Object Store Shipping
|
||||
|
||||
**Status:** `[ ] TODO`
|
||||
**Blocked by:** SQ-007
|
||||
**Priority:** Medium
|
||||
|
||||
## Description
|
||||
|
||||
Background process that ships closed WAL segments to S3-compatible object storage for long-term durability.
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
- `crates/sq-storage/src/object_store/mod.rs` - ObjectStore trait + S3 impl + Noop impl
|
||||
- `crates/sq-storage/src/object_store/shipper.rs` - SegmentShipper as notmad::Component
|
||||
- `crates/sq-storage/src/object_store/layout.rs` - S3 key naming convention
|
||||
|
||||
## ObjectStore Trait
|
||||
|
||||
```rust
|
||||
pub trait ObjectStore: Send + Sync {
|
||||
async fn put(&self, key: &str, data: Vec<u8>) -> Result<()>;
|
||||
async fn get(&self, key: &str) -> Result<Vec<u8>>;
|
||||
async fn list(&self, prefix: &str) -> Result<Vec<String>>;
|
||||
async fn delete(&self, key: &str) -> Result<()>;
|
||||
}
|
||||
```
|
||||
|
||||
## S3 Key Layout
|
||||
|
||||
`{cluster_id}/{topic}/{partition}/{base_offset}-{end_offset}.sqseg`
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] Closed segment is detected and uploaded to object store
|
||||
- [ ] S3 key matches expected layout
|
||||
- [ ] Noop object store works for testing (stores in memory)
|
||||
- [ ] Upload failure: segment stays local, retried on next cycle
|
||||
- [ ] Successful upload is recorded (segment marked as "shipped")
|
||||
- [ ] Uses zstd compression before upload
|
||||
|
||||
## Notes
|
||||
|
||||
- Uses `object_store` crate with AWS S3 features (same as nostore)
|
||||
- Shipper runs as a notmad::Component in the background
|
||||
- Poll interval: every 5 seconds check for closed segments
|
||||
Reference in New Issue
Block a user