feat: add initial

Signed-off-by: kjuulh <contact@kjuulh.io>
This commit is contained in:
2026-02-26 21:52:50 +01:00
commit 3162971c89
48 changed files with 3041 additions and 0 deletions

View File

@@ -0,0 +1,45 @@
# SQ-016: Object Store Shipping
**Status:** `[ ] TODO`
**Blocked by:** SQ-007
**Priority:** Medium
## Description
Background process that ships closed WAL segments to S3-compatible object storage for long-term durability.
## Files to Create/Modify
- `crates/sq-storage/src/object_store/mod.rs` - ObjectStore trait + S3 impl + Noop impl
- `crates/sq-storage/src/object_store/shipper.rs` - SegmentShipper as notmad::Component
- `crates/sq-storage/src/object_store/layout.rs` - S3 key naming convention
## ObjectStore Trait
```rust
pub trait ObjectStore: Send + Sync {
async fn put(&self, key: &str, data: Vec<u8>) -> Result<()>;
async fn get(&self, key: &str) -> Result<Vec<u8>>;
async fn list(&self, prefix: &str) -> Result<Vec<String>>;
async fn delete(&self, key: &str) -> Result<()>;
}
```
## S3 Key Layout
`{cluster_id}/{topic}/{partition}/{base_offset}-{end_offset}.sqseg`
## Acceptance Criteria
- [ ] Closed segment is detected and uploaded to object store
- [ ] S3 key matches expected layout
- [ ] Noop object store works for testing (stores in memory)
- [ ] Upload failure: segment stays local, retried on next cycle
- [ ] Successful upload is recorded (segment marked as "shipped")
- [ ] Uses zstd compression before upload
## Notes
- Uses `object_store` crate with AWS S3 features (same as nostore)
- Shipper runs as a notmad::Component in the background
- Poll interval: every 5 seconds check for closed segments