45 lines
1.3 KiB
Markdown
45 lines
1.3 KiB
Markdown
# SQ-006: Sparse Offset Index
|
|
|
|
**Status:** `[x] DONE`
|
|
**Blocked by:** SQ-005
|
|
**Priority:** Medium
|
|
|
|
## Description
|
|
|
|
In-memory sparse offset index that maps offsets to segment file locations for fast consumer seeks.
|
|
|
|
## Files to Create/Modify
|
|
|
|
- `crates/sq-storage/src/index.rs` - OffsetIndex
|
|
|
|
## OffsetIndex API
|
|
|
|
```rust
|
|
pub struct OffsetIndex {
|
|
// Per topic-partition: sorted vec of (offset, segment_path, byte_position)
|
|
entries: BTreeMap<(TopicName, u32), Vec<IndexEntry>>,
|
|
sample_interval: u64, // e.g. every 1000th offset
|
|
}
|
|
|
|
pub struct IndexEntry {
|
|
pub offset: u64,
|
|
pub segment_path: PathBuf,
|
|
pub byte_position: u64,
|
|
}
|
|
|
|
impl OffsetIndex {
|
|
pub fn new(sample_interval: u64) -> Self;
|
|
pub fn add_entry(&mut self, topic: &TopicName, partition: u32, entry: IndexEntry);
|
|
pub fn lookup(&self, topic: &TopicName, partition: u32, offset: u64) -> Option<&IndexEntry>;
|
|
pub fn build_from_segments<F: FileSystem>(fs: &F, segments: &[PathBuf]) -> Result<Self>;
|
|
}
|
|
```
|
|
|
|
## Acceptance Criteria
|
|
|
|
- [ ] Build index from a set of written segments, look up first offset -> correct segment
|
|
- [ ] Look up offset in the middle -> returns nearest lower indexed entry
|
|
- [ ] Look up offset beyond all segments -> returns None
|
|
- [ ] Multiple topic-partitions are isolated
|
|
- [ ] Sample interval works: only every Nth offset is indexed
|