sq/todos/SQ-002-wal-record-encoding.md

SQ-002: WAL Record Encoding/Decoding

Status: [ ] TODO Blocked by: SQ-001 Priority: High

Description

Implement binary encoding and decoding of individual WAL records, independent of file I/O. Each record is CRC32-protected for corruption detection.

Files to Create/Modify

• crates/sq-storage/src/wal/mod.rs - module declaration
• crates/sq-storage/src/wal/record.rs - encode_record, decode_record, CRC validation
• crates/sq-storage/src/lib.rs - re-export wal module

Record Binary Format

[crc32: u32]         - CRC32 over everything after this field
[length: u32]        - total byte length of record (excluding crc32 and length)
[offset: u64]        - monotonic offset
[timestamp_ms: u64]  - wall clock millis
[key_len: u32]       - 0 = no key
[key: [u8; key_len]]
[value_len: u32]
[value: [u8; value_len]]
[headers_count: u16]
[for each header:]
  [hdr_key_len: u16]
  [hdr_key: [u8; hdr_key_len]]
  [hdr_val_len: u32]
  [hdr_val: [u8; hdr_val_len]]

Acceptance Criteria

• encode_record(&Message) -> Vec<u8> produces correct binary
• decode_record(&[u8]) -> Result<(Message, usize)> parses correctly (returns bytes consumed)
• Roundtrip: encode then decode, verify equality
• Corruption test: flip a byte, decode returns CRC error
• Edge cases: empty value, empty key, no headers, many headers, max-size value
• Uses crc32fast crate for CRC computation

Notes

• Use little-endian byte order throughout
• The length field allows skipping records without full parsing
• CRC is over the bytes AFTER the CRC field itself