# SQ (SeedQueue)

A fast, durable message queue. Written in Rust.

- **Custom WAL** with configurable fsync for durability
- **Cap'n Proto data plane** for high-throughput publish/subscribe (~2M msg/s)
- **gRPC control plane** for topic management, cluster ops, and health checks
- **Simple quorum clustering** with gossip-based membership
- **S3 tiered storage** for long-term segment offloading
- **OpenTelemetry** traces and metrics out of the box

## Quickstart

```bash
# Build
cargo build --release -p sq-server

# Run a single node
./target/release/sq-server serve

# Default ports:
#   6064  Cap'n Proto data plane (producer/consumer)
#   6060  gRPC control plane
#   6062  HTTP (health, metrics)
```

Or with [mise](https://mise.jdx.dev):

```bash
mise run develop    # cargo run -p sq-server -- serve
```

## SDK

Add the dependency:

```toml
[dependencies]
sq-sdk = { path = "crates/sq-sdk" }
```

Publish:

```rust
use sq_sdk::{Producer, ProducerConfig, ProducerMessage};

let mut producer = Producer::connect(ProducerConfig::default()).await?;

producer.send("orders", None, b"hello").await?;

// Or batch:
let batch = vec![
    ProducerMessage::new("orders", b"msg-1"),
    ProducerMessage::new("orders", b"msg-2"),
];
producer.send_batch(batch).await?;
```

Subscribe:

```rust
use sq_sdk::{Consumer, ConsumerConfig};

let mut consumer = Consumer::connect(ConsumerConfig {
    topic: "orders".into(),
    consumer_group: "my-group".into(),
    auto_commit: true,
    ..Default::default()
}).await?;

loop {
    let messages = consumer.poll().await?;
    for msg in &messages {
        println!("[{}] offset={}", msg.topic, msg.offset);
    }
}
```

A `BatchProducer` is available for high-throughput fire-and-forget patterns with automatic batching.

The gRPC transport is still accessible via `GrpcProducer`, `GrpcConsumer`, etc.

## Configuration

All flags can also be set via environment variables.

| Flag | Env | Default | Description |
|------|-----|---------|-------------|
| `--grpc-host` | `SQ_GRPC_HOST` | `127.0.0.1:6060` | gRPC listen address |
| `--capnp-host` | `SQ_CAPNP_HOST` | `127.0.0.1:6064` | Cap'n Proto listen address |
| `--http-host` | `SQ_HTTP_HOST` | `127.0.0.1:6062` | HTTP listen address |
| `--node-id` | `SQ_NODE_ID` | `node-1` | Unique node identifier |
| `--data-dir` | `SQ_DATA_DIR` | `./data` | WAL storage directory |
| `--seeds` | `SQ_SEEDS` | | Comma-separated seed node addresses |
| `--cluster-id` | `SQ_CLUSTER_ID` | `default` | Cluster identifier |
| `--sync-policy` | `SQ_SYNC_POLICY` | `every-batch` | `every-batch`, `none`, or interval in ms |
| `--s3-bucket` | `SQ_S3_BUCKET` | | S3 bucket for tiered storage |
| `--s3-endpoint` | `SQ_S3_ENDPOINT` | | S3 endpoint (for MinIO, etc.) |
| `--s3-region` | `SQ_S3_REGION` | | S3 region |

Observability is activated by setting `OTEL_EXPORTER_OTLP_ENDPOINT` (e.g., `http://jaeger:4317`).

## Running a cluster

```bash
# Node 1
sq-server --node-id node-1 --grpc-host 0.0.0.0:6060 --capnp-host 0.0.0.0:6064 serve

# Node 2
sq-server --node-id node-2 --grpc-host 0.0.0.0:6070 --capnp-host 0.0.0.0:6074 \
  --seeds 10.0.0.1:6060 serve

# Node 3
sq-server --node-id node-3 --grpc-host 0.0.0.0:6080 --capnp-host 0.0.0.0:6084 \
  --seeds 10.0.0.1:6060,10.0.0.2:6070 serve
```

## Docker Compose

A full stack is provided in `templates/docker-compose.yaml`:

```bash
docker compose -f templates/docker-compose.yaml up -d
```

This starts a 3-node SQ cluster with Jaeger (tracing), Prometheus (metrics), Grafana (dashboards), and MinIO (S3-compatible object storage).

## Development

Requires Rust 2024 edition (nightly or stable 1.85+).

```bash
mise run check       # cargo check --workspace
mise run test        # cargo nextest run --workspace
mise run clippy      # cargo clippy --workspace
mise run build       # cargo build --workspace
mise run local:up    # start docker-compose services
mise run local:down  # stop docker-compose services
mise run develop     # run server in dev mode
```

Proto codegen (requires [buf](https://buf.build)):

```bash
mise run generate:proto
```

## CI

CI runs via [Dagger](https://dagger.io) (containerized pipelines, no YAML). Requires the Dagger CLI.

```bash
mise run ci:pr       # PR pipeline: check + test + build
mise run ci:main     # Main branch pipeline
```

The pipeline runs `cargo check`, `cargo test`, and builds a release Docker image — all inside containers with dependency caching.

## Project structure

```
crates/
  sq-server          Server binary + library (gRPC, Cap'n Proto, HTTP)
  sq-sdk             Client SDK (Producer, Consumer, BatchProducer)
  sq-storage         WAL, storage engine, S3 object store
  sq-cluster         Membership and replication
  sq-grpc-interface  Protobuf/tonic generated types
  sq-capnp-interface Cap'n Proto schema and codec
  sq-models          Shared domain types
  sq-sim             Deterministic I/O simulation for testing
ci/                  Dagger CI pipeline
examples/
  publish_subscribe  Example publisher and subscriber
templates/
  docker-compose.yaml  Full observability stack
  sq-server.Dockerfile Multi-stage release build
```

## Testing

```bash
# Unit tests (~100 tests, instant)
cargo test --workspace --lib

# Integration tests (cluster, data plane)
cargo test -p sq-server --test cluster_test
cargo test -p sq-server --test data_plane_test

# Stress tests (100K+ messages, benchmarks)
cargo test -p sq-server --test stress_test -- --nocapture
cargo test -p sq-server --test capnp_stress_test -- --nocapture

# Storage benchmarks
cargo bench -p sq-storage
```