feat: add post3 s3 proxy for postgresql

Signed-off-by: kjuulh <contact@kjuulh.io>

todos/POST3-001-workspace-skeleton.md

@@ -0,0 +1,21 @@
+# POST3-001: Create workspace skeleton
+
+**Status:** Done
+**Priority:** P0
+**Blocked by:** —
+
+## Description
+
+Set up the Rust workspace with both crates, Docker Compose for PostgreSQL 18, and mise.toml dev tasks.
+
+## Acceptance Criteria
+
+- [ ] `Cargo.toml` workspace root with `crates/*` members
+- [ ] `crates/post3/Cargo.toml` — library crate with sqlx, tokio, bytes, chrono, md-5, hex, thiserror, uuid, tracing, serde
+- [ ] `crates/post3/src/lib.rs` — empty module declarations
+- [ ] `crates/post3-server/Cargo.toml` — binary crate depending on post3, axum, clap, notmad, quick-xml, etc.
+- [ ] `crates/post3-server/src/main.rs` — minimal tokio main
+- [ ] `templates/docker-compose.yaml` — PostgreSQL 18 on port 5435
+- [ ] `mise.toml` — tasks: up, down, dev, test, db:shell, db:migrate
+- [ ] `cargo check --workspace` passes
+- [ ] `mise run up` starts PostgreSQL successfully

todos/POST3-002-schema-models-errors.md

@@ -0,0 +1,24 @@
+# POST3-002: Database schema, models, and error types
+
+**Status:** Done
+**Priority:** P0
+**Blocked by:** POST3-001
+
+## Description
+
+Define the PostgreSQL schema (buckets, objects, object_metadata, blocks), create Rust model types with sqlx::FromRow, and define the Post3Error enum.
+
+## Acceptance Criteria
+
+- [ ] `crates/post3/migrations/20260226000001_initial.sql` with all 4 tables + indexes
+- [ ] `crates/post3/src/models.rs` — BucketRow, ObjectRow, BlockRow, MetadataEntry, ObjectInfo, ListObjectsResult
+- [ ] `crates/post3/src/error.rs` — Post3Error enum (BucketNotFound, BucketAlreadyExists, ObjectNotFound, BucketNotEmpty, Database, Other)
+- [ ] Migration runs successfully against PostgreSQL
+- [ ] `cargo check -p post3` passes
+
+## Schema Details
+
+- `buckets` — id (UUID PK), name (TEXT UNIQUE), created_at
+- `objects` — id (UUID PK), bucket_id (FK CASCADE), key, size, etag, content_type, created_at; unique on (bucket_id, key)
+- `object_metadata` — id (UUID PK), object_id (FK CASCADE), meta_key, meta_value; unique on (object_id, meta_key)
+- `blocks` — id (UUID PK), object_id (FK CASCADE), block_index, data (BYTEA), block_size; unique on (object_id, block_index)

todos/POST3-003-repository-layer.md

@@ -0,0 +1,26 @@
+# POST3-003: Repository layer and Store API
+
+**Status:** Done
+**Priority:** P0
+**Blocked by:** POST3-002
+
+## Description
+
+Implement the repository layer (raw SQL CRUD for each table) and the high-level Store API that orchestrates them with transactions and chunking logic.
+
+## Acceptance Criteria
+
+- [ ] `repositories/buckets.rs` — create, get_by_name, list, delete, is_empty
+- [ ] `repositories/objects.rs` — upsert, get, delete, list (with prefix + pagination)
+- [ ] `repositories/blocks.rs` — insert_block, get_all_blocks (ordered by block_index)
+- [ ] `repositories/metadata.rs` — insert_batch, get_all (for an object_id)
+- [ ] `store.rs` — Store struct with all public methods:
+  - create_bucket, head_bucket, delete_bucket, list_buckets
+  - put_object (chunking + MD5 ETag + metadata, all in a transaction)
+  - get_object (reassemble blocks + fetch metadata)
+  - head_object, delete_object, list_objects_v2
+  - get_object_metadata
+- [ ] put_object correctly splits body into 1 MiB blocks
+- [ ] get_object correctly reassembles blocks in order
+- [ ] Overwriting an object deletes old blocks+metadata via CASCADE
+- [ ] `cargo check -p post3` passes

todos/POST3-004-s3-server-skeleton.md

@@ -0,0 +1,20 @@
+# POST3-004: S3 HTTP server skeleton
+
+**Status:** Done
+**Priority:** P0
+**Blocked by:** POST3-003
+
+## Description
+
+Build the post3-server binary with CLI (clap), state management, notmad component lifecycle, and axum router with all S3 routes wired up.
+
+## Acceptance Criteria
+
+- [ ] `main.rs` — dotenvy + tracing_subscriber + cli::execute()
+- [ ] `cli.rs` — clap App with `serve` subcommand
+- [ ] `cli/serve.rs` — ServeCommand with --host flag, starts notmad::Mad with S3Server
+- [ ] `state.rs` — State struct (PgPool + Store), runs migrations on new()
+- [ ] `s3/mod.rs` — S3Server implementing notmad::Component
+- [ ] `s3/router.rs` — all 9 routes mapped to handler functions
+- [ ] Server starts, binds to port, responds to requests
+- [ ] `cargo check -p post3-server` passes

todos/POST3-005-xml-responses-extractors.md

@@ -0,0 +1,20 @@
+# POST3-005: XML response builders and extractors
+
+**Status:** Done
+**Priority:** P1
+**Blocked by:** POST3-004
+
+## Description
+
+Implement S3-compatible XML response serialization and request query parameter extraction.
+
+## Acceptance Criteria
+
+- [ ] `s3/responses.rs`:
+  - `list_buckets_xml(buckets)` — ListAllMyBucketsResult with Owner
+  - `list_objects_v2_xml(bucket, result, max_keys)` — ListBucketResult with Contents
+  - `error_xml(code, message, resource)` — S3 Error response
+- [ ] `s3/extractors.rs`:
+  - `ListObjectsQuery` — list-type, prefix, max-keys, continuation-token, start-after, delimiter
+- [ ] XML output matches S3 format (xmlns, element names, date format ISO 8601)
+- [ ] All responses include `x-amz-request-id` header (UUID)

todos/POST3-006-s3-handlers.md

@@ -0,0 +1,30 @@
+# POST3-006: S3 bucket and object handlers
+
+**Status:** Done
+**Priority:** P1
+**Blocked by:** POST3-005
+
+## Description
+
+Implement all S3 HTTP request handlers that bridge the S3 REST API to the core Store API.
+
+## Acceptance Criteria
+
+### Bucket handlers (`s3/handlers/buckets.rs`)
+- [ ] CreateBucket — PUT /{bucket} → 200 + Location header
+- [ ] HeadBucket — HEAD /{bucket} → 200 or 404
+- [ ] DeleteBucket — DELETE /{bucket} → 204 (409 if not empty)
+- [ ] ListBuckets — GET / → 200 + XML
+
+### Object handlers (`s3/handlers/objects.rs`)
+- [ ] PutObject — PUT /{bucket}/{*key} → 200 + ETag header; reads x-amz-meta-* from request headers
+- [ ] GetObject — GET /{bucket}/{*key} → 200 + body + ETag + Content-Type + Content-Length + Last-Modified + x-amz-meta-* headers
+- [ ] HeadObject — HEAD /{bucket}/{*key} → 200 + metadata headers (no body)
+- [ ] DeleteObject — DELETE /{bucket}/{*key} → 204
+- [ ] ListObjectsV2 — GET /{bucket}?list-type=2 → 200 + XML
+
+### Error handling
+- [ ] NoSuchBucket → 404 + XML error
+- [ ] NoSuchKey → 404 + XML error
+- [ ] BucketAlreadyOwnedByYou → 409 + XML error
+- [ ] BucketNotEmpty → 409 + XML error

todos/POST3-007-integration-tests.md

@@ -0,0 +1,27 @@
+# POST3-007: Integration tests with aws-sdk-s3
+
+**Status:** Done
+**Priority:** P1
+**Blocked by:** POST3-006
+
+## Description
+
+End-to-end integration tests using the official AWS S3 Rust SDK to validate the full stack.
+
+## Acceptance Criteria
+
+- [ ] `tests/common/mod.rs` — TestServer helper:
+  - Starts server on ephemeral port (port 0)
+  - Configures aws-sdk-s3 with force_path_style, dummy creds, custom endpoint
+  - Cleans database between tests
+- [ ] Test: create + list buckets
+- [ ] Test: head bucket (exists + not exists)
+- [ ] Test: delete bucket
+- [ ] Test: put + get small object (body roundtrip)
+- [ ] Test: put large object (5 MiB, verify chunked storage + reassembly)
+- [ ] Test: head object (size, etag, content-type)
+- [ ] Test: delete object (verify 404 after)
+- [ ] Test: list objects v2 with prefix filter
+- [ ] Test: overwrite object (verify latest version)
+- [ ] Test: user metadata roundtrip (x-amz-meta-* headers)
+- [ ] All tests pass with `cargo nextest run`

todos/POST3-008-client-sdk.md

@@ -0,0 +1,26 @@
+# POST3-008: Client SDK crate
+
+**Status:** Done
+**Priority:** P0
+**Blocked by:** —
+
+## Description
+
+Create a `crates/post3-sdk/` client crate that wraps `aws-sdk-s3` with post3-specific defaults.
+
+## What was built
+
+- [x] `crates/post3-sdk/Cargo.toml` — depends on aws-sdk-s3, aws-credential-types, aws-types, aws-config
+- [x] `Post3Client` struct wrapping `aws_sdk_s3::Client`
+- [x] `Post3Client::new(endpoint_url)` — builds client with force_path_style, dummy creds, us-east-1
+- [x] `Post3Client::builder()` — for advanced config (custom creds, region, etc.)
+- [x] Re-exports: `aws_sdk_s3` and `bytes`
+- [x] Convenience methods:
+  - `create_bucket(name)`, `head_bucket(name)`, `delete_bucket(name)`, `list_buckets()`
+  - `put_object(bucket, key, body: impl AsRef<[u8]>)`
+  - `get_object(bucket, key)` → `Result<Bytes>`
+  - `head_object(bucket, key)` → `Result<Option<ObjectInfo>>`
+  - `delete_object(bucket, key)`
+  - `list_objects(bucket, prefix)` → `Result<Vec<ObjectInfo>>`
+- [x] `inner()` access to `aws_sdk_s3::Client`
+- [x] Unit tests + doc-tests pass

todos/POST3-009-ci-dagger.md

@@ -0,0 +1,30 @@
+# POST3-009: CI pipeline with Dagger Rust SDK
+
+**Status:** Done
+**Priority:** P1
+**Blocked by:** —
+
+## Description
+
+Set up a Dagger-based CI pipeline using a custom self-contained `ci/` crate with `dagger-sdk` directly (not the external `cuddle-ci` / `dagger-rust` components, which are too opinionated for post3's context).
+
+## What was built
+
+- [x] `ci/` added as workspace member
+- [x] `ci/Cargo.toml` with dependencies: dagger-sdk, eyre, tokio, clap
+- [x] `ci/src/main.rs` — custom pipeline with:
+  - `pr` and `main` subcommands (clap CLI)
+  - Source loading with dependency caching (skeleton files pattern from dagger-components)
+  - `rustlang/rust:nightly` base with clang + mold 2.3.3 for fast linking
+  - Dagger cache volumes for target/ and cargo registry
+  - `cargo check --workspace` compilation check
+  - PostgreSQL 18 as Dagger service container for integration tests
+  - `cargo test --workspace -- --test-threads=1` against Dagger PG
+  - Release binary build + packaging into `debian:bookworm-slim`
+  - `post3-server --help` sanity check in final image
+- [x] `mise.toml` tasks: `ci:pr`, `ci:main`
+- [x] No container publish (deferred until registry is decided)
+
+## Reference
+
+Pattern inspired by dagger-components (`/home/kjuulh/git/git.kjuulh.io/kjuulh/dagger-components`) but self-contained — no external git dependencies.

todos/POST3-010-docker-compose-production.md

@@ -0,0 +1,34 @@
+# POST3-010: Production Docker Compose setup
+
+**Status:** Todo
+**Priority:** P1
+**Blocked by:** POST3-009
+
+## Description
+
+Create a production-oriented Docker Compose setup that runs post3-server alongside PostgreSQL, with proper networking, health checks, and configuration.
+
+## Acceptance Criteria
+
+- [ ] `Dockerfile` (multi-stage) for post3-server:
+  - Builder stage: rust image, compile release binary
+  - Runtime stage: debian-slim or alpine, copy binary + migrations
+  - Health check endpoint (add `GET /health` to router)
+  - Non-root user
+- [ ] `templates/docker-compose.production.yaml`:
+  - `postgres` service (PostgreSQL 18, persistent volume, health check)
+  - `post3` service (built image, depends_on postgres healthy, DATABASE_URL from env)
+  - Named volumes for PostgreSQL data
+  - Internal network
+  - Port 9000 exposed for post3
+- [ ] `templates/.env.example` — sample env file for production
+- [ ] `GET /health` endpoint on the server (returns 200 when DB is reachable)
+- [ ] `mise.toml` tasks:
+  - `prod:up` — start production compose
+  - `prod:down` — stop production compose
+  - `prod:build` — build the Docker image
+- [ ] README section on production deployment
+
+## Notes
+
+The CI pipeline (POST3-009) will produce the container image. This ticket handles the compose orchestration for self-hosted deployment.

todos/POST3-011-examples.md

@@ -0,0 +1,29 @@
+# POST3-011: Usage examples
+
+**Status:** Done
+**Priority:** P1
+**Blocked by:** POST3-008
+
+## Description
+
+Create runnable examples demonstrating how to use post3 with both the SDK and shell tools.
+
+## What was built
+
+### Rust examples (`crates/post3-sdk/examples/`)
+- [x] `basic.rs` — create bucket, put/get/delete object, list objects with prefix filter
+- [x] `metadata.rs` — put object with custom metadata (x-amz-meta-*), retrieve via head/get
+- [x] `aws_sdk_direct.rs` — use aws-sdk-s3 directly (without post3-sdk wrapper), shows raw config
+
+### Script examples (`examples/`)
+- [x] `aws-cli.sh` — shell script demonstrating all operations via `aws` CLI
+- [x] `curl.sh` — shell script demonstrating raw HTTP calls with curl
+
+### mise tasks
+- [x] `example:basic` — runs the basic Rust example
+- [x] `example:metadata` — runs the metadata Rust example
+- [x] `example:aws-sdk` — runs the raw aws-sdk-s3 example
+- [x] `example:cli` — runs the AWS CLI example script
+- [x] `example:curl` — runs the curl example script
+
+All examples tested and verified working against live server.

todos/POST3-012-authentication.md

@@ -0,0 +1,70 @@
+# POST3-012: Authentication system
+
+**Status:** Todo
+**Priority:** P1
+**Blocked by:** —
+
+## Description
+
+Add authentication to post3-server. Currently the server accepts any request regardless of credentials. We need to support API key-based authentication that is compatible with the AWS SigV4 signing process (so the official AWS SDKs and CLI work transparently).
+
+## Approach
+
+### Phase 1: API key authentication (simple)
+
+Use a shared secret key pair (access_key_id + secret_access_key) configured via environment variables. The server validates that the `Authorization` header contains a valid AWS SigV4 signature computed with the known secret.
+
+- [ ] Database table `api_keys`:
+  ```sql
+  CREATE TABLE api_keys (
+      id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+      access_key_id   TEXT NOT NULL,
+      secret_key      TEXT NOT NULL,  -- stored hashed or plaintext for SigV4
+      name            TEXT NOT NULL,  -- human-readable label
+      is_active       BOOLEAN NOT NULL DEFAULT true,
+      created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW()
+  );
+  CREATE UNIQUE INDEX idx_api_keys_access_key ON api_keys (access_key_id);
+  ```
+- [ ] SigV4 signature verification middleware (axum layer)
+- [ ] Extract access_key_id from `Authorization` header
+- [ ] Look up secret_key from `api_keys` table
+- [ ] Recompute the SigV4 signature and compare
+- [ ] Return `403 AccessDenied` XML error on mismatch
+- [ ] Environment variable `POST3_AUTH_ENABLED=true|false` to toggle (default: false for backward compat)
+
+### Phase 2: Per-bucket ACLs (future)
+
+- [ ] `bucket_permissions` table linking api_keys to buckets with read/write/admin roles
+- [ ] Enforce permissions in handlers
+- [ ] Admin API for managing keys and permissions
+
+### Phase 3: Admin CLI
+
+- [ ] `post3-server admin create-key --name "my-app"` — generates and prints access_key_id + secret_access_key
+- [ ] `post3-server admin list-keys` — list all API keys
+- [ ] `post3-server admin revoke-key --access-key-id AKIA...` — deactivate a key
+
+## Migration
+
+- [ ] New migration file for `api_keys` table
+- [ ] Existing deployments unaffected (auth disabled by default)
+
+## SDK Integration
+
+- [ ] `Post3Client::builder().credentials(access_key, secret_key)` passes real credentials
+- [ ] When auth is disabled, dummy credentials still work
+
+## Testing
+
+- [ ] Test: request with valid signature succeeds
+- [ ] Test: request with invalid signature returns 403
+- [ ] Test: request with unknown access_key_id returns 403
+- [ ] Test: auth disabled mode accepts any credentials
+- [ ] Test: admin CLI key management commands
+
+## Notes
+
+SigV4 verification requires access to the raw request (method, path, headers, body hash). The `aws-sigv4` crate from the AWS SDK can help with signature computation on the server side. Alternatively, implement the HMAC-SHA256 chain manually — it's well-documented.
+
+The secret_key must be stored in a form that allows recomputing signatures (SigV4 uses the secret directly in HMAC, not a hash of it). This means secret_keys are stored as plaintext or with reversible encryption. This is inherent to SigV4's design.

todos/POST3-013-s3-compliance.md

@@ -0,0 +1,68 @@
+# POST3-013: S3 Compliance Testing with Ceph s3-tests
+
+## Status: Done
+
+## Summary
+
+Integrate Ceph s3-tests (the industry-standard S3 conformance suite) to validate post3's S3 compatibility. Uses the filesystem backend (`--backend fs`) for fast, database-free test runs.
+
+## Results
+
+**124 tests passing, 0 failures, 0 errors** out of 829 total tests (705 deselected for unimplemented features).
+
+## What was done
+
+### Phase 1 — Missing S3 operations (blocking for s3-tests)
+- [x] `ListObjectVersions` stub — `GET /{bucket}?versions` (returns objects as version "null")
+- [x] `DeleteObjects` batch delete — `POST /{bucket}?delete`
+- [x] `ListObjects` v1 — `GET /{bucket}` without `list-type=2`
+- [x] `GetBucketLocation` — `GET /{bucket}?location`
+- [x] `--backend fs/pg` CLI flag + `--data-dir`
+- [x] Bucket naming validation (S3 rules: 3-63 chars, lowercase, no IP format)
+
+### Phase 2 — Delimiter & listing compliance
+- [x] Delimiter + CommonPrefixes in `list_objects_v2` (both backends)
+- [x] V1 and V2 XML responses emit delimiter/common_prefixes
+- [x] MaxKeys limits total objects+common_prefixes combined (sorted interleave)
+- [x] MaxKeys=0 returns empty, non-truncated result
+- [x] StartAfter + ContinuationToken echo in v2 response
+- [x] Owner element in v1 Contents
+- [x] Empty delimiter treated as absent
+
+### Phase 3 — Test infrastructure
+- [x] s3-tests git submodule (pinned at `06e2c57`)
+- [x] `s3-compliance/s3tests.conf.template`
+- [x] `s3-compliance/run-s3-tests.sh`
+- [x] mise tasks: `test:s3-compliance` and `test:s3-compliance:dry`
+
+### Phase 4 — Compliance fixes from test runs
+- [x] ETag quoting normalization in multipart completion (both backends)
+- [x] ListObjectVersions pagination (NextKeyMarker/NextVersionIdMarker when truncated)
+- [x] ListObjectVersions passes key-marker and delimiter from query params
+- [x] EntityTooSmall validation (non-last parts must be >= 5 MB)
+- [x] DeleteObjects 1000 key limit
+- [x] delete_object returns 404 for non-existent bucket
+- [x] Common prefix filtering by continuation token
+
+## Usage
+
+```sh
+mise run test:s3-compliance        # run filtered s3-tests
+mise run test:s3-compliance:dry    # list which tests would run
+```
+
+## Excluded test categories
+
+Features post3 doesn't implement (excluded via markers/keywords):
+ACLs, bucket policy, encryption, CORS, lifecycle, versioning, object lock,
+tagging, S3 Select, S3 website, IAM, STS, SSE, anonymous access, presigned URLs,
+CopyObject, logging, notifications, storage classes, auth signature validation,
+Range header, conditional requests, public access block.
+
+## Future work
+
+- [ ] Add CI step (`ci/src/main.rs`) for automated s3-compliance runs
+- [ ] Gradually reduce exclusion list as more features are implemented
+- [ ] Range header support (would enable ~10 more tests)
+- [ ] CopyObject support (would enable ~20 more tests)
+- [ ] Idempotent CompleteMultipartUpload (Ceph-specific, 2 excluded tests)