post3/CLAUDE.md

post3 — Pluggable S3-Compatible Storage

Project Overview

post3 = PostgreSQL + S3. An S3-compatible storage system with pluggable backends. Objects can be stored in PostgreSQL (split into 1 MiB blocks in bytea columns) or on the local filesystem.

Architecture

• crates/post3/ — Core library crate. Contains the StorageBackend trait, PostgresBackend, FilesystemBackend, repository layer, models, error types, and SQL migrations.
• crates/post3-server/ — Binary + lib crate. S3-compatible HTTP server using axum. Generic over B: StorageBackend — works with any backend.
• crates/post3-sdk/ — Client SDK wrapping aws-sdk-s3 with ergonomic defaults (dummy creds, path-style, us-east-1). Re-exports aws_sdk_s3 for advanced use.
• ci/ — Custom CI pipeline using dagger-sdk directly. Builds, tests, and packages in containers.

Development Commands (mise)

mise run up              # Start PostgreSQL (docker compose)
mise run down            # Stop PostgreSQL + remove volumes
mise run check           # cargo check --workspace
mise run dev             # Run the server (localhost:9000)
mise run test            # Run all tests (starts PG first)
mise run test:integration # Run S3 integration tests only
mise run db:shell        # psql into dev database
mise run db:reset        # Wipe and restart PostgreSQL
mise run build           # Release build
mise run ci:pr           # Run CI PR pipeline via Dagger
mise run ci:main         # Run CI main pipeline via Dagger
mise run example:basic   # Run basic SDK example (requires server)
mise run example:metadata # Run metadata example (requires server)
mise run example:aws-sdk # Run raw aws-sdk-s3 example (requires server)
mise run example:cli     # Run AWS CLI example (requires server + aws CLI)
mise run example:curl    # Run curl example (requires server)
mise run example:large   # Run large file stress test (requires server)

Environment

• DATABASE_URL: postgresql://devuser:devpassword@localhost:5435/post3_dev
• POST3_HOST: 127.0.0.1:9000
• PostgreSQL 18 on port 5435 (avoids conflicts with other projects)

Key Patterns

• StorageBackend trait — Pluggable storage via impl Future<...> + Send desugared async methods (edition 2024). Server is generic over B: StorageBackend.
• PostgresBackend (alias Store) — PostgreSQL backend using sqlx repos + 1 MiB block chunks
• FilesystemBackend — Local filesystem backend using percent-encoded keys, JSON metadata, atomic writes
• notmad 0.11 for component lifecycle (native async traits, no async_trait)
• sqlx with PgPool for database access; migrations at crates/post3/migrations/
• axum 0.8 with {param} path syntax and {*wildcard} for nested keys
• Trailing slash routes duplicated for AWS SDK compatibility (/{bucket} + /{bucket}/)
• Body limit set to 5 GiB via DefaultBodyLimit
• S3 multipart upload supported: CreateMultipartUpload, UploadPart, CompleteMultipartUpload, AbortMultipartUpload, ListParts, ListMultipartUploads
• Query param dispatch: PUT/GET/DELETE/POST on /{bucket}/{*key} dispatch by ?uploads, ?uploadId, ?partNumber
• Handlers use turbofish ::<B> in router for generic dispatch
• Tests use aws-sdk-s3 with force_path_style(true) and dummy credentials

Database Schema

7 tables: buckets, objects, object_metadata (KV registry), blocks (1 MiB chunks), multipart_uploads, multipart_upload_metadata, upload_parts. All use ON DELETE CASCADE for cleanup.

Testing

• PostgreSQL integration tests in crates/post3-server/tests/s3_integration.rs — spin up a real server per test on an ephemeral port. Each test gets its own PgPool and cleans the database. Tests must run with --test-threads=1 to avoid DB conflicts.
• Filesystem integration tests in crates/post3-server/tests/fs_integration.rs — same HTTP-level tests but using FilesystemBackend with a temp directory. No PostgreSQL required.
• Filesystem unit tests in crates/post3/src/fs.rs — direct backend method tests.

Roadmap (see todos/)

• POST3-008: Client SDK crate — Done (crates/post3-sdk/)
• POST3-009: CI pipeline — Done (custom ci/ crate using dagger-sdk directly)
• POST3-010: Production Docker Compose (Dockerfile, health endpoint, compose)
• POST3-011: Usage examples — Done (Rust examples, AWS CLI, curl, large file stress test)
• POST3-012: Authentication (SigV4 verification, API keys table, admin CLI)

CI Pattern

Custom ci/ crate using dagger-sdk (v0.19) directly — self-contained, no external dagger-components dependency. Subcommands: pr (check + test + build + package) and main (same, no publish yet). Uses PostgreSQL 18 as a Dagger service container for integration tests. Skeleton source + dependency-only prebuild for cargo layer caching. mold linker for fast linking. Final image: debian:bookworm-slim.