feat: add post3 s3 proxy for postgresql
Signed-off-by: kjuulh <contact@kjuulh.io>
This commit is contained in:
68
todos/POST3-013-s3-compliance.md
Normal file
68
todos/POST3-013-s3-compliance.md
Normal file
@@ -0,0 +1,68 @@
|
||||
# POST3-013: S3 Compliance Testing with Ceph s3-tests
|
||||
|
||||
## Status: Done
|
||||
|
||||
## Summary
|
||||
|
||||
Integrate Ceph s3-tests (the industry-standard S3 conformance suite) to validate post3's S3 compatibility. Uses the filesystem backend (`--backend fs`) for fast, database-free test runs.
|
||||
|
||||
## Results
|
||||
|
||||
**124 tests passing, 0 failures, 0 errors** out of 829 total tests (705 deselected for unimplemented features).
|
||||
|
||||
## What was done
|
||||
|
||||
### Phase 1 — Missing S3 operations (blocking for s3-tests)
|
||||
- [x] `ListObjectVersions` stub — `GET /{bucket}?versions` (returns objects as version "null")
|
||||
- [x] `DeleteObjects` batch delete — `POST /{bucket}?delete`
|
||||
- [x] `ListObjects` v1 — `GET /{bucket}` without `list-type=2`
|
||||
- [x] `GetBucketLocation` — `GET /{bucket}?location`
|
||||
- [x] `--backend fs/pg` CLI flag + `--data-dir`
|
||||
- [x] Bucket naming validation (S3 rules: 3-63 chars, lowercase, no IP format)
|
||||
|
||||
### Phase 2 — Delimiter & listing compliance
|
||||
- [x] Delimiter + CommonPrefixes in `list_objects_v2` (both backends)
|
||||
- [x] V1 and V2 XML responses emit delimiter/common_prefixes
|
||||
- [x] MaxKeys limits total objects+common_prefixes combined (sorted interleave)
|
||||
- [x] MaxKeys=0 returns empty, non-truncated result
|
||||
- [x] StartAfter + ContinuationToken echo in v2 response
|
||||
- [x] Owner element in v1 Contents
|
||||
- [x] Empty delimiter treated as absent
|
||||
|
||||
### Phase 3 — Test infrastructure
|
||||
- [x] s3-tests git submodule (pinned at `06e2c57`)
|
||||
- [x] `s3-compliance/s3tests.conf.template`
|
||||
- [x] `s3-compliance/run-s3-tests.sh`
|
||||
- [x] mise tasks: `test:s3-compliance` and `test:s3-compliance:dry`
|
||||
|
||||
### Phase 4 — Compliance fixes from test runs
|
||||
- [x] ETag quoting normalization in multipart completion (both backends)
|
||||
- [x] ListObjectVersions pagination (NextKeyMarker/NextVersionIdMarker when truncated)
|
||||
- [x] ListObjectVersions passes key-marker and delimiter from query params
|
||||
- [x] EntityTooSmall validation (non-last parts must be >= 5 MB)
|
||||
- [x] DeleteObjects 1000 key limit
|
||||
- [x] delete_object returns 404 for non-existent bucket
|
||||
- [x] Common prefix filtering by continuation token
|
||||
|
||||
## Usage
|
||||
|
||||
```sh
|
||||
mise run test:s3-compliance # run filtered s3-tests
|
||||
mise run test:s3-compliance:dry # list which tests would run
|
||||
```
|
||||
|
||||
## Excluded test categories
|
||||
|
||||
Features post3 doesn't implement (excluded via markers/keywords):
|
||||
ACLs, bucket policy, encryption, CORS, lifecycle, versioning, object lock,
|
||||
tagging, S3 Select, S3 website, IAM, STS, SSE, anonymous access, presigned URLs,
|
||||
CopyObject, logging, notifications, storage classes, auth signature validation,
|
||||
Range header, conditional requests, public access block.
|
||||
|
||||
## Future work
|
||||
|
||||
- [ ] Add CI step (`ci/src/main.rs`) for automated s3-compliance runs
|
||||
- [ ] Gradually reduce exclusion list as more features are implemented
|
||||
- [ ] Range header support (would enable ~10 more tests)
|
||||
- [ ] CopyObject support (would enable ~20 more tests)
|
||||
- [ ] Idempotent CompleteMultipartUpload (Ceph-specific, 2 excluded tests)
|
||||
Reference in New Issue
Block a user