64 lines
2.2 KiB
Markdown
64 lines
2.2 KiB
Markdown
# SQ-004: WAL Segment Writer
|
|
|
|
**Status:** `[ ] TODO`
|
|
**Blocked by:** SQ-002, SQ-003
|
|
**Priority:** High
|
|
|
|
## Description
|
|
|
|
Implement the WAL segment writer that appends records to segment files with fsync for durability. Handles segment rotation when size or time thresholds are exceeded.
|
|
|
|
## Files to Create/Modify
|
|
|
|
- `crates/sq-storage/src/wal/writer.rs` - WalWriter with append + fsync + rotation
|
|
- `crates/sq-storage/src/wal/segment.rs` - segment header encoding/decoding
|
|
|
|
## Segment Header Format (32 bytes)
|
|
|
|
```
|
|
[magic: [u8; 4]] = b"SQWL"
|
|
[version: u16] = 1
|
|
[topic_len: u16]
|
|
[topic: [u8; 20]] (padded/truncated)
|
|
[partition: u32]
|
|
```
|
|
|
|
## WalWriter API
|
|
|
|
```rust
|
|
pub struct WalWriter<F: FileSystem> {
|
|
fs: Arc<F>,
|
|
config: WalConfig,
|
|
topic: TopicName,
|
|
partition: u32,
|
|
active_segment: Option<Box<dyn FileHandle>>,
|
|
segment_base_offset: u64,
|
|
segment_position: u64,
|
|
next_offset: u64,
|
|
segment_opened_at: Instant,
|
|
}
|
|
|
|
impl<F: FileSystem> WalWriter<F> {
|
|
pub async fn new(fs: Arc<F>, config: WalConfig, topic: TopicName, partition: u32) -> Result<Self>;
|
|
pub async fn append(&mut self, key: Option<&[u8]>, value: &[u8], headers: &[Header], timestamp_ms: u64) -> Result<u64>; // returns offset
|
|
pub async fn close_active_segment(&mut self) -> Result<Option<ClosedSegment>>;
|
|
pub fn next_offset(&self) -> u64;
|
|
}
|
|
```
|
|
|
|
## Acceptance Criteria (using InMemoryFileSystem)
|
|
|
|
- [ ] Write 1 message, verify segment file exists with correct header + record
|
|
- [ ] Write 100 messages, verify all offsets are monotonically increasing (0, 1, 2, ...)
|
|
- [ ] Segment rotation: write until size > max_segment_bytes, verify new segment created
|
|
- [ ] Segment rotation: advance clock past max_segment_age, verify rotation on next write
|
|
- [ ] fsync failure: set fault on InMemoryFS, verify append() returns error
|
|
- [ ] fsync failure: offset is NOT advanced (can retry the write)
|
|
- [ ] Segment directory structure: `{data_dir}/{topic}/{partition}/{base_offset}.wal`
|
|
|
|
## Notes
|
|
|
|
- sq-storage depends on sq-sim for the FileSystem trait
|
|
- Writer must call fsync after every append (or batch of appends)
|
|
- `ClosedSegment` contains the path and offset range of the completed segment
|