feat: add post3 s3 proxy for postgresql

Signed-off-by: kjuulh <contact@kjuulh.io>
This commit is contained in:
2026-02-27 11:37:48 +01:00
commit 21bac4a33f
67 changed files with 14403 additions and 0 deletions

77
CLAUDE.md Normal file
View File

@@ -0,0 +1,77 @@
# post3 — Pluggable S3-Compatible Storage
## Project Overview
**post3** = **Post**greSQL + S**3**. An S3-compatible storage system with pluggable backends. Objects can be stored in PostgreSQL (split into 1 MiB blocks in `bytea` columns) or on the local filesystem.
## Architecture
- **`crates/post3/`** — Core library crate. Contains the `StorageBackend` trait, `PostgresBackend`, `FilesystemBackend`, repository layer, models, error types, and SQL migrations.
- **`crates/post3-server/`** — Binary + lib crate. S3-compatible HTTP server using axum. Generic over `B: StorageBackend` — works with any backend.
- **`crates/post3-sdk/`** — Client SDK wrapping `aws-sdk-s3` with ergonomic defaults (dummy creds, path-style, us-east-1). Re-exports `aws_sdk_s3` for advanced use.
- **`ci/`** — Custom CI pipeline using `dagger-sdk` directly. Builds, tests, and packages in containers.
## Development Commands (mise)
```sh
mise run up # Start PostgreSQL (docker compose)
mise run down # Stop PostgreSQL + remove volumes
mise run check # cargo check --workspace
mise run dev # Run the server (localhost:9000)
mise run test # Run all tests (starts PG first)
mise run test:integration # Run S3 integration tests only
mise run db:shell # psql into dev database
mise run db:reset # Wipe and restart PostgreSQL
mise run build # Release build
mise run ci:pr # Run CI PR pipeline via Dagger
mise run ci:main # Run CI main pipeline via Dagger
mise run example:basic # Run basic SDK example (requires server)
mise run example:metadata # Run metadata example (requires server)
mise run example:aws-sdk # Run raw aws-sdk-s3 example (requires server)
mise run example:cli # Run AWS CLI example (requires server + aws CLI)
mise run example:curl # Run curl example (requires server)
mise run example:large # Run large file stress test (requires server)
```
## Environment
- **DATABASE_URL**: `postgresql://devuser:devpassword@localhost:5435/post3_dev`
- **POST3_HOST**: `127.0.0.1:9000`
- PostgreSQL 18 on port **5435** (avoids conflicts with other projects)
## Key Patterns
- **`StorageBackend` trait** — Pluggable storage via `impl Future<...> + Send` desugared async methods (edition 2024). Server is generic over `B: StorageBackend`.
- **`PostgresBackend`** (alias `Store`) — PostgreSQL backend using sqlx repos + 1 MiB block chunks
- **`FilesystemBackend`** — Local filesystem backend using percent-encoded keys, JSON metadata, atomic writes
- **notmad 0.11** for component lifecycle (native async traits, no async_trait)
- **sqlx** with `PgPool` for database access; migrations at `crates/post3/migrations/`
- **axum 0.8** with `{param}` path syntax and `{*wildcard}` for nested keys
- Trailing slash routes duplicated for AWS SDK compatibility (`/{bucket}` + `/{bucket}/`)
- Body limit set to 5 GiB via `DefaultBodyLimit`
- S3 multipart upload supported: CreateMultipartUpload, UploadPart, CompleteMultipartUpload, AbortMultipartUpload, ListParts, ListMultipartUploads
- Query param dispatch: PUT/GET/DELETE/POST on `/{bucket}/{*key}` dispatch by `?uploads`, `?uploadId`, `?partNumber`
- Handlers use turbofish `::<B>` in router for generic dispatch
- Tests use `aws-sdk-s3` with `force_path_style(true)` and dummy credentials
## Database Schema
7 tables: `buckets`, `objects`, `object_metadata` (KV registry), `blocks` (1 MiB chunks), `multipart_uploads`, `multipart_upload_metadata`, `upload_parts`. All use `ON DELETE CASCADE` for cleanup.
## Testing
- **PostgreSQL integration tests** in `crates/post3-server/tests/s3_integration.rs` — spin up a real server per test on an ephemeral port. Each test gets its own `PgPool` and cleans the database. Tests must run with `--test-threads=1` to avoid DB conflicts.
- **Filesystem integration tests** in `crates/post3-server/tests/fs_integration.rs` — same HTTP-level tests but using `FilesystemBackend` with a temp directory. No PostgreSQL required.
- **Filesystem unit tests** in `crates/post3/src/fs.rs` — direct backend method tests.
## Roadmap (see `todos/`)
- **POST3-008**: Client SDK crate — **Done** (`crates/post3-sdk/`)
- **POST3-009**: CI pipeline — **Done** (custom `ci/` crate using `dagger-sdk` directly)
- **POST3-010**: Production Docker Compose (Dockerfile, health endpoint, compose)
- **POST3-011**: Usage examples — **Done** (Rust examples, AWS CLI, curl, large file stress test)
- **POST3-012**: Authentication (SigV4 verification, API keys table, admin CLI)
## CI Pattern
Custom `ci/` crate using `dagger-sdk` (v0.19) directly — self-contained, no external dagger-components dependency. Subcommands: `pr` (check + test + build + package) and `main` (same, no publish yet). Uses PostgreSQL 18 as a Dagger service container for integration tests. Skeleton source + dependency-only prebuild for cargo layer caching. mold linker for fast linking. Final image: `debian:bookworm-slim`.