sq/todos/SQ-007-storage-engine-facade.md

SQ-007: Storage Engine Facade

Status: [ ] TODO Blocked by: SQ-006 Priority: High

Description

Unified read/write interface wrapping WAL writer + reader + offset index. This is the single entry point for all storage operations used by the server.

Files to Create/Modify

• crates/sq-storage/src/engine.rs - StorageEngine

StorageEngine API

pub struct StorageEngine<F: FileSystem, C: Clock> {
    fs: Arc<F>,
    clock: Arc<C>,
    config: WalConfig,
    writers: HashMap<(String, u32), WalWriter<F>>,
    index: OffsetIndex,
}

impl<F: FileSystem, C: Clock> StorageEngine<F, C> {
    pub async fn new(fs: Arc<F>, clock: Arc<C>, config: WalConfig) -> Result<Self>;
    pub async fn append(&mut self, topic: &str, partition: u32, key: Option<&[u8]>, value: &[u8], headers: &[Header]) -> Result<u64>;
    pub async fn read(&self, topic: &str, partition: u32, from_offset: u64, limit: usize) -> Result<Vec<Message>>;
    pub async fn recover(&mut self) -> Result<()>;  // Rebuild state from existing WAL files on startup
    pub fn closed_segments(&self) -> Vec<ClosedSegment>;  // For S3 shipper
}

Acceptance Criteria

• Write 1000 messages, read from offset 0, verify all present
• Write, read from offset 500, verify correct slice returned
• Read with limit, verify at most N messages returned
• Write to multiple topics/partitions, verify isolation (no cross-contamination)
• Recovery: write messages, drop engine, create new engine, call recover(), read all messages back
• Segment rotation happens transparently during append