63
todos/SQ-004-wal-segment-writer.md
Normal file
63
todos/SQ-004-wal-segment-writer.md
Normal file
@@ -0,0 +1,63 @@
|
||||
# SQ-004: WAL Segment Writer
|
||||
|
||||
**Status:** `[ ] TODO`
|
||||
**Blocked by:** SQ-002, SQ-003
|
||||
**Priority:** High
|
||||
|
||||
## Description
|
||||
|
||||
Implement the WAL segment writer that appends records to segment files with fsync for durability. Handles segment rotation when size or time thresholds are exceeded.
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
- `crates/sq-storage/src/wal/writer.rs` - WalWriter with append + fsync + rotation
|
||||
- `crates/sq-storage/src/wal/segment.rs` - segment header encoding/decoding
|
||||
|
||||
## Segment Header Format (32 bytes)
|
||||
|
||||
```
|
||||
[magic: [u8; 4]] = b"SQWL"
|
||||
[version: u16] = 1
|
||||
[topic_len: u16]
|
||||
[topic: [u8; 20]] (padded/truncated)
|
||||
[partition: u32]
|
||||
```
|
||||
|
||||
## WalWriter API
|
||||
|
||||
```rust
|
||||
pub struct WalWriter<F: FileSystem> {
|
||||
fs: Arc<F>,
|
||||
config: WalConfig,
|
||||
topic: TopicName,
|
||||
partition: u32,
|
||||
active_segment: Option<Box<dyn FileHandle>>,
|
||||
segment_base_offset: u64,
|
||||
segment_position: u64,
|
||||
next_offset: u64,
|
||||
segment_opened_at: Instant,
|
||||
}
|
||||
|
||||
impl<F: FileSystem> WalWriter<F> {
|
||||
pub async fn new(fs: Arc<F>, config: WalConfig, topic: TopicName, partition: u32) -> Result<Self>;
|
||||
pub async fn append(&mut self, key: Option<&[u8]>, value: &[u8], headers: &[Header], timestamp_ms: u64) -> Result<u64>; // returns offset
|
||||
pub async fn close_active_segment(&mut self) -> Result<Option<ClosedSegment>>;
|
||||
pub fn next_offset(&self) -> u64;
|
||||
}
|
||||
```
|
||||
|
||||
## Acceptance Criteria (using InMemoryFileSystem)
|
||||
|
||||
- [ ] Write 1 message, verify segment file exists with correct header + record
|
||||
- [ ] Write 100 messages, verify all offsets are monotonically increasing (0, 1, 2, ...)
|
||||
- [ ] Segment rotation: write until size > max_segment_bytes, verify new segment created
|
||||
- [ ] Segment rotation: advance clock past max_segment_age, verify rotation on next write
|
||||
- [ ] fsync failure: set fault on InMemoryFS, verify append() returns error
|
||||
- [ ] fsync failure: offset is NOT advanced (can retry the write)
|
||||
- [ ] Segment directory structure: `{data_dir}/{topic}/{partition}/{base_offset}.wal`
|
||||
|
||||
## Notes
|
||||
|
||||
- sq-storage depends on sq-sim for the FileSystem trait
|
||||
- Writer must call fsync after every append (or batch of appends)
|
||||
- `ClosedSegment` contains the path and offset range of the completed segment
|
||||
Reference in New Issue
Block a user