sq/todos/SQ-004-wal-segment-writer.md

SQ-004: WAL Segment Writer

Status: [x] DONE Blocked by: SQ-002, SQ-003 Priority: High

Description

Implement the WAL segment writer that appends records to segment files with fsync for durability. Handles segment rotation when size or time thresholds are exceeded.

Files to Create/Modify

• crates/sq-storage/src/wal/writer.rs - WalWriter with append + fsync + rotation
• crates/sq-storage/src/wal/segment.rs - segment header encoding/decoding

Segment Header Format (32 bytes)

[magic: [u8; 4]]    = b"SQWL"
[version: u16]       = 1
[topic_len: u16]
[topic: [u8; 20]]    (padded/truncated)
[partition: u32]

WalWriter API

pub struct WalWriter<F: FileSystem> {
    fs: Arc<F>,
    config: WalConfig,
    topic: TopicName,
    partition: u32,
    active_segment: Option<Box<dyn FileHandle>>,
    segment_base_offset: u64,
    segment_position: u64,
    next_offset: u64,
    segment_opened_at: Instant,
}

impl<F: FileSystem> WalWriter<F> {
    pub async fn new(fs: Arc<F>, config: WalConfig, topic: TopicName, partition: u32) -> Result<Self>;
    pub async fn append(&mut self, key: Option<&[u8]>, value: &[u8], headers: &[Header], timestamp_ms: u64) -> Result<u64>;  // returns offset
    pub async fn close_active_segment(&mut self) -> Result<Option<ClosedSegment>>;
    pub fn next_offset(&self) -> u64;
}

Acceptance Criteria (using InMemoryFileSystem)

• Write 1 message, verify segment file exists with correct header + record
• Write 100 messages, verify all offsets are monotonically increasing (0, 1, 2, ...)
• Segment rotation: write until size > max_segment_bytes, verify new segment created
• Segment rotation: advance clock past max_segment_age, verify rotation on next write
• fsync failure: set fault on InMemoryFS, verify append() returns error
• fsync failure: offset is NOT advanced (can retry the write)
• Segment directory structure: {data_dir}/{topic}/{partition}/{base_offset}.wal

Notes

• sq-storage depends on sq-sim for the FileSystem trait
• Writer must call fsync after every append (or batch of appends)
• ClosedSegment contains the path and offset range of the completed segment