sq/todos/SQ-006-sparse-offset-index.md

SQ-006: Sparse Offset Index

Status: [x] DONE Blocked by: SQ-005 Priority: Medium

Description

In-memory sparse offset index that maps offsets to segment file locations for fast consumer seeks.

Files to Create/Modify

• crates/sq-storage/src/index.rs - OffsetIndex

OffsetIndex API

pub struct OffsetIndex {
    // Per topic-partition: sorted vec of (offset, segment_path, byte_position)
    entries: BTreeMap<(TopicName, u32), Vec<IndexEntry>>,
    sample_interval: u64, // e.g. every 1000th offset
}

pub struct IndexEntry {
    pub offset: u64,
    pub segment_path: PathBuf,
    pub byte_position: u64,
}

impl OffsetIndex {
    pub fn new(sample_interval: u64) -> Self;
    pub fn add_entry(&mut self, topic: &TopicName, partition: u32, entry: IndexEntry);
    pub fn lookup(&self, topic: &TopicName, partition: u32, offset: u64) -> Option<&IndexEntry>;
    pub fn build_from_segments<F: FileSystem>(fs: &F, segments: &[PathBuf]) -> Result<Self>;
}

Acceptance Criteria

• Build index from a set of written segments, look up first offset -> correct segment
• Look up offset in the middle -> returns nearest lower indexed entry
• Look up offset beyond all segments -> returns None
• Multiple topic-partitions are isolated
• Sample interval works: only every Nth offset is indexed