feat: add initial

Signed-off-by: kjuulh <contact@kjuulh.io>
This commit is contained in:
2026-02-26 21:52:50 +01:00
commit 3162971c89
48 changed files with 3041 additions and 0 deletions

View File

@@ -0,0 +1,63 @@
# SQ-004: WAL Segment Writer
**Status:** `[ ] TODO`
**Blocked by:** SQ-002, SQ-003
**Priority:** High
## Description
Implement the WAL segment writer that appends records to segment files with fsync for durability. Handles segment rotation when size or time thresholds are exceeded.
## Files to Create/Modify
- `crates/sq-storage/src/wal/writer.rs` - WalWriter with append + fsync + rotation
- `crates/sq-storage/src/wal/segment.rs` - segment header encoding/decoding
## Segment Header Format (32 bytes)
```
[magic: [u8; 4]] = b"SQWL"
[version: u16] = 1
[topic_len: u16]
[topic: [u8; 20]] (padded/truncated)
[partition: u32]
```
## WalWriter API
```rust
pub struct WalWriter<F: FileSystem> {
fs: Arc<F>,
config: WalConfig,
topic: TopicName,
partition: u32,
active_segment: Option<Box<dyn FileHandle>>,
segment_base_offset: u64,
segment_position: u64,
next_offset: u64,
segment_opened_at: Instant,
}
impl<F: FileSystem> WalWriter<F> {
pub async fn new(fs: Arc<F>, config: WalConfig, topic: TopicName, partition: u32) -> Result<Self>;
pub async fn append(&mut self, key: Option<&[u8]>, value: &[u8], headers: &[Header], timestamp_ms: u64) -> Result<u64>; // returns offset
pub async fn close_active_segment(&mut self) -> Result<Option<ClosedSegment>>;
pub fn next_offset(&self) -> u64;
}
```
## Acceptance Criteria (using InMemoryFileSystem)
- [ ] Write 1 message, verify segment file exists with correct header + record
- [ ] Write 100 messages, verify all offsets are monotonically increasing (0, 1, 2, ...)
- [ ] Segment rotation: write until size > max_segment_bytes, verify new segment created
- [ ] Segment rotation: advance clock past max_segment_age, verify rotation on next write
- [ ] fsync failure: set fault on InMemoryFS, verify append() returns error
- [ ] fsync failure: offset is NOT advanced (can retry the write)
- [ ] Segment directory structure: `{data_dir}/{topic}/{partition}/{base_offset}.wal`
## Notes
- sq-storage depends on sq-sim for the FileSystem trait
- Writer must call fsync after every append (or batch of appends)
- `ClosedSegment` contains the path and offset range of the completed segment