feat: add initial

Signed-off-by: kjuulh <contact@kjuulh.io>
This commit is contained in:
2026-02-26 21:52:50 +01:00
commit 3162971c89
48 changed files with 3041 additions and 0 deletions

View File

@@ -0,0 +1,44 @@
# SQ-006: Sparse Offset Index
**Status:** `[ ] TODO`
**Blocked by:** SQ-005
**Priority:** Medium
## Description
In-memory sparse offset index that maps offsets to segment file locations for fast consumer seeks.
## Files to Create/Modify
- `crates/sq-storage/src/index.rs` - OffsetIndex
## OffsetIndex API
```rust
pub struct OffsetIndex {
// Per topic-partition: sorted vec of (offset, segment_path, byte_position)
entries: BTreeMap<(TopicName, u32), Vec<IndexEntry>>,
sample_interval: u64, // e.g. every 1000th offset
}
pub struct IndexEntry {
pub offset: u64,
pub segment_path: PathBuf,
pub byte_position: u64,
}
impl OffsetIndex {
pub fn new(sample_interval: u64) -> Self;
pub fn add_entry(&mut self, topic: &TopicName, partition: u32, entry: IndexEntry);
pub fn lookup(&self, topic: &TopicName, partition: u32, offset: u64) -> Option<&IndexEntry>;
pub fn build_from_segments<F: FileSystem>(fs: &F, segments: &[PathBuf]) -> Result<Self>;
}
```
## Acceptance Criteria
- [ ] Build index from a set of written segments, look up first offset -> correct segment
- [ ] Look up offset in the middle -> returns nearest lower indexed entry
- [ ] Look up offset beyond all segments -> returns None
- [ ] Multiple topic-partitions are isolated
- [ ] Sample interval works: only every Nth offset is indexed