feat: add post3 s3 proxy for postgresql
Signed-off-by: kjuulh <contact@kjuulh.io>
This commit is contained in:
77
CLAUDE.md
Normal file
77
CLAUDE.md
Normal file
@@ -0,0 +1,77 @@
|
||||
# post3 — Pluggable S3-Compatible Storage
|
||||
|
||||
## Project Overview
|
||||
|
||||
**post3** = **Post**greSQL + S**3**. An S3-compatible storage system with pluggable backends. Objects can be stored in PostgreSQL (split into 1 MiB blocks in `bytea` columns) or on the local filesystem.
|
||||
|
||||
## Architecture
|
||||
|
||||
- **`crates/post3/`** — Core library crate. Contains the `StorageBackend` trait, `PostgresBackend`, `FilesystemBackend`, repository layer, models, error types, and SQL migrations.
|
||||
- **`crates/post3-server/`** — Binary + lib crate. S3-compatible HTTP server using axum. Generic over `B: StorageBackend` — works with any backend.
|
||||
- **`crates/post3-sdk/`** — Client SDK wrapping `aws-sdk-s3` with ergonomic defaults (dummy creds, path-style, us-east-1). Re-exports `aws_sdk_s3` for advanced use.
|
||||
- **`ci/`** — Custom CI pipeline using `dagger-sdk` directly. Builds, tests, and packages in containers.
|
||||
|
||||
## Development Commands (mise)
|
||||
|
||||
```sh
|
||||
mise run up # Start PostgreSQL (docker compose)
|
||||
mise run down # Stop PostgreSQL + remove volumes
|
||||
mise run check # cargo check --workspace
|
||||
mise run dev # Run the server (localhost:9000)
|
||||
mise run test # Run all tests (starts PG first)
|
||||
mise run test:integration # Run S3 integration tests only
|
||||
mise run db:shell # psql into dev database
|
||||
mise run db:reset # Wipe and restart PostgreSQL
|
||||
mise run build # Release build
|
||||
mise run ci:pr # Run CI PR pipeline via Dagger
|
||||
mise run ci:main # Run CI main pipeline via Dagger
|
||||
mise run example:basic # Run basic SDK example (requires server)
|
||||
mise run example:metadata # Run metadata example (requires server)
|
||||
mise run example:aws-sdk # Run raw aws-sdk-s3 example (requires server)
|
||||
mise run example:cli # Run AWS CLI example (requires server + aws CLI)
|
||||
mise run example:curl # Run curl example (requires server)
|
||||
mise run example:large # Run large file stress test (requires server)
|
||||
```
|
||||
|
||||
## Environment
|
||||
|
||||
- **DATABASE_URL**: `postgresql://devuser:devpassword@localhost:5435/post3_dev`
|
||||
- **POST3_HOST**: `127.0.0.1:9000`
|
||||
- PostgreSQL 18 on port **5435** (avoids conflicts with other projects)
|
||||
|
||||
## Key Patterns
|
||||
|
||||
- **`StorageBackend` trait** — Pluggable storage via `impl Future<...> + Send` desugared async methods (edition 2024). Server is generic over `B: StorageBackend`.
|
||||
- **`PostgresBackend`** (alias `Store`) — PostgreSQL backend using sqlx repos + 1 MiB block chunks
|
||||
- **`FilesystemBackend`** — Local filesystem backend using percent-encoded keys, JSON metadata, atomic writes
|
||||
- **notmad 0.11** for component lifecycle (native async traits, no async_trait)
|
||||
- **sqlx** with `PgPool` for database access; migrations at `crates/post3/migrations/`
|
||||
- **axum 0.8** with `{param}` path syntax and `{*wildcard}` for nested keys
|
||||
- Trailing slash routes duplicated for AWS SDK compatibility (`/{bucket}` + `/{bucket}/`)
|
||||
- Body limit set to 5 GiB via `DefaultBodyLimit`
|
||||
- S3 multipart upload supported: CreateMultipartUpload, UploadPart, CompleteMultipartUpload, AbortMultipartUpload, ListParts, ListMultipartUploads
|
||||
- Query param dispatch: PUT/GET/DELETE/POST on `/{bucket}/{*key}` dispatch by `?uploads`, `?uploadId`, `?partNumber`
|
||||
- Handlers use turbofish `::<B>` in router for generic dispatch
|
||||
- Tests use `aws-sdk-s3` with `force_path_style(true)` and dummy credentials
|
||||
|
||||
## Database Schema
|
||||
|
||||
7 tables: `buckets`, `objects`, `object_metadata` (KV registry), `blocks` (1 MiB chunks), `multipart_uploads`, `multipart_upload_metadata`, `upload_parts`. All use `ON DELETE CASCADE` for cleanup.
|
||||
|
||||
## Testing
|
||||
|
||||
- **PostgreSQL integration tests** in `crates/post3-server/tests/s3_integration.rs` — spin up a real server per test on an ephemeral port. Each test gets its own `PgPool` and cleans the database. Tests must run with `--test-threads=1` to avoid DB conflicts.
|
||||
- **Filesystem integration tests** in `crates/post3-server/tests/fs_integration.rs` — same HTTP-level tests but using `FilesystemBackend` with a temp directory. No PostgreSQL required.
|
||||
- **Filesystem unit tests** in `crates/post3/src/fs.rs` — direct backend method tests.
|
||||
|
||||
## Roadmap (see `todos/`)
|
||||
|
||||
- **POST3-008**: Client SDK crate — **Done** (`crates/post3-sdk/`)
|
||||
- **POST3-009**: CI pipeline — **Done** (custom `ci/` crate using `dagger-sdk` directly)
|
||||
- **POST3-010**: Production Docker Compose (Dockerfile, health endpoint, compose)
|
||||
- **POST3-011**: Usage examples — **Done** (Rust examples, AWS CLI, curl, large file stress test)
|
||||
- **POST3-012**: Authentication (SigV4 verification, API keys table, admin CLI)
|
||||
|
||||
## CI Pattern
|
||||
|
||||
Custom `ci/` crate using `dagger-sdk` (v0.19) directly — self-contained, no external dagger-components dependency. Subcommands: `pr` (check + test + build + package) and `main` (same, no publish yet). Uses PostgreSQL 18 as a Dagger service container for integration tests. Skeleton source + dependency-only prebuild for cargo layer caching. mold linker for fast linking. Final image: `debian:bookworm-slim`.
|
||||
Reference in New Issue
Block a user