# post3 — Pluggable S3-Compatible Storage

## Project Overview

**post3** = **Post**greSQL + S**3**. An S3-compatible storage system with pluggable backends. Objects can be stored in PostgreSQL (split into 1 MiB blocks in `bytea` columns) or on the local filesystem.

## Architecture

- **`crates/post3/`** — Core library crate. Contains the `StorageBackend` trait, `PostgresBackend`, `FilesystemBackend`, repository layer, models, error types, and SQL migrations.
- **`crates/post3-server/`** — Binary + lib crate. S3-compatible HTTP server using axum. Generic over `B: StorageBackend` — works with any backend.
- **`crates/post3-sdk/`** — Client SDK wrapping `aws-sdk-s3` with ergonomic defaults (dummy creds, path-style, us-east-1). Re-exports `aws_sdk_s3` for advanced use.
- **`ci/`** — Custom CI pipeline using `dagger-sdk` directly. Builds, tests, and packages in containers.

## Development Commands (mise)

```sh
mise run up              # Start PostgreSQL (docker compose)
mise run down            # Stop PostgreSQL + remove volumes
mise run check           # cargo check --workspace
mise run dev             # Run the server (localhost:9000)
mise run test            # Run all tests (starts PG first)
mise run test:integration # Run S3 integration tests only
mise run db:shell        # psql into dev database
mise run db:reset        # Wipe and restart PostgreSQL
mise run build           # Release build
mise run ci:pr           # Run CI PR pipeline via Dagger
mise run ci:main         # Run CI main pipeline via Dagger
mise run example:basic   # Run basic SDK example (requires server)
mise run example:metadata # Run metadata example (requires server)
mise run example:aws-sdk # Run raw aws-sdk-s3 example (requires server)
mise run example:cli     # Run AWS CLI example (requires server + aws CLI)
mise run example:curl    # Run curl example (requires server)
mise run example:large   # Run large file stress test (requires server)
```

## Environment

- **DATABASE_URL**: `postgresql://devuser:devpassword@localhost:5435/post3_dev`
- **POST3_HOST**: `127.0.0.1:9000`
- PostgreSQL 18 on port **5435** (avoids conflicts with other projects)

## Key Patterns

- **`StorageBackend` trait** — Pluggable storage via `impl Future<...> + Send` desugared async methods (edition 2024). Server is generic over `B: StorageBackend`.
- **`PostgresBackend`** (alias `Store`) — PostgreSQL backend using sqlx repos + 1 MiB block chunks
- **`FilesystemBackend`** — Local filesystem backend using percent-encoded keys, JSON metadata, atomic writes
- **notmad 0.11** for component lifecycle (native async traits, no async_trait)
- **sqlx** with `PgPool` for database access; migrations at `crates/post3/migrations/`
- **axum 0.8** with `{param}` path syntax and `{*wildcard}` for nested keys
- Trailing slash routes duplicated for AWS SDK compatibility (`/{bucket}` + `/{bucket}/`)
- Body limit set to 5 GiB via `DefaultBodyLimit`
- S3 multipart upload supported: CreateMultipartUpload, UploadPart, CompleteMultipartUpload, AbortMultipartUpload, ListParts, ListMultipartUploads
- Query param dispatch: PUT/GET/DELETE/POST on `/{bucket}/{*key}` dispatch by `?uploads`, `?uploadId`, `?partNumber`
- Handlers use turbofish `::<B>` in router for generic dispatch
- Tests use `aws-sdk-s3` with `force_path_style(true)` and dummy credentials

## Database Schema

7 tables: `buckets`, `objects`, `object_metadata` (KV registry), `blocks` (1 MiB chunks), `multipart_uploads`, `multipart_upload_metadata`, `upload_parts`. All use `ON DELETE CASCADE` for cleanup.

## Testing

- **PostgreSQL integration tests** in `crates/post3-server/tests/s3_integration.rs` — spin up a real server per test on an ephemeral port. Each test gets its own `PgPool` and cleans the database. Tests must run with `--test-threads=1` to avoid DB conflicts.
- **Filesystem integration tests** in `crates/post3-server/tests/fs_integration.rs` — same HTTP-level tests but using `FilesystemBackend` with a temp directory. No PostgreSQL required.
- **Filesystem unit tests** in `crates/post3/src/fs.rs` — direct backend method tests.

## Roadmap (see `todos/`)

- **POST3-008**: Client SDK crate — **Done** (`crates/post3-sdk/`)
- **POST3-009**: CI pipeline — **Done** (custom `ci/` crate using `dagger-sdk` directly)
- **POST3-010**: Production Docker Compose (Dockerfile, health endpoint, compose)
- **POST3-011**: Usage examples — **Done** (Rust examples, AWS CLI, curl, large file stress test)
- **POST3-012**: Authentication (SigV4 verification, API keys table, admin CLI)

## CI Pattern

Custom `ci/` crate using `dagger-sdk` (v0.19) directly — self-contained, no external dagger-components dependency. Subcommands: `pr` (check + test + build + package) and `main` (same, no publish yet). Uses PostgreSQL 18 as a Dagger service container for integration tests. Skeleton source + dependency-only prebuild for cargo layer caching. mold linker for fast linking. Final image: `debian:bookworm-slim`.