69 lines
2.9 KiB
Markdown
69 lines
2.9 KiB
Markdown
# POST3-013: S3 Compliance Testing with Ceph s3-tests
|
|
|
|
## Status: Done
|
|
|
|
## Summary
|
|
|
|
Integrate Ceph s3-tests (the industry-standard S3 conformance suite) to validate post3's S3 compatibility. Uses the filesystem backend (`--backend fs`) for fast, database-free test runs.
|
|
|
|
## Results
|
|
|
|
**124 tests passing, 0 failures, 0 errors** out of 829 total tests (705 deselected for unimplemented features).
|
|
|
|
## What was done
|
|
|
|
### Phase 1 — Missing S3 operations (blocking for s3-tests)
|
|
- [x] `ListObjectVersions` stub — `GET /{bucket}?versions` (returns objects as version "null")
|
|
- [x] `DeleteObjects` batch delete — `POST /{bucket}?delete`
|
|
- [x] `ListObjects` v1 — `GET /{bucket}` without `list-type=2`
|
|
- [x] `GetBucketLocation` — `GET /{bucket}?location`
|
|
- [x] `--backend fs/pg` CLI flag + `--data-dir`
|
|
- [x] Bucket naming validation (S3 rules: 3-63 chars, lowercase, no IP format)
|
|
|
|
### Phase 2 — Delimiter & listing compliance
|
|
- [x] Delimiter + CommonPrefixes in `list_objects_v2` (both backends)
|
|
- [x] V1 and V2 XML responses emit delimiter/common_prefixes
|
|
- [x] MaxKeys limits total objects+common_prefixes combined (sorted interleave)
|
|
- [x] MaxKeys=0 returns empty, non-truncated result
|
|
- [x] StartAfter + ContinuationToken echo in v2 response
|
|
- [x] Owner element in v1 Contents
|
|
- [x] Empty delimiter treated as absent
|
|
|
|
### Phase 3 — Test infrastructure
|
|
- [x] s3-tests git submodule (pinned at `06e2c57`)
|
|
- [x] `s3-compliance/s3tests.conf.template`
|
|
- [x] `s3-compliance/run-s3-tests.sh`
|
|
- [x] mise tasks: `test:s3-compliance` and `test:s3-compliance:dry`
|
|
|
|
### Phase 4 — Compliance fixes from test runs
|
|
- [x] ETag quoting normalization in multipart completion (both backends)
|
|
- [x] ListObjectVersions pagination (NextKeyMarker/NextVersionIdMarker when truncated)
|
|
- [x] ListObjectVersions passes key-marker and delimiter from query params
|
|
- [x] EntityTooSmall validation (non-last parts must be >= 5 MB)
|
|
- [x] DeleteObjects 1000 key limit
|
|
- [x] delete_object returns 404 for non-existent bucket
|
|
- [x] Common prefix filtering by continuation token
|
|
|
|
## Usage
|
|
|
|
```sh
|
|
mise run test:s3-compliance # run filtered s3-tests
|
|
mise run test:s3-compliance:dry # list which tests would run
|
|
```
|
|
|
|
## Excluded test categories
|
|
|
|
Features post3 doesn't implement (excluded via markers/keywords):
|
|
ACLs, bucket policy, encryption, CORS, lifecycle, versioning, object lock,
|
|
tagging, S3 Select, S3 website, IAM, STS, SSE, anonymous access, presigned URLs,
|
|
CopyObject, logging, notifications, storage classes, auth signature validation,
|
|
Range header, conditional requests, public access block.
|
|
|
|
## Future work
|
|
|
|
- [ ] Add CI step (`ci/src/main.rs`) for automated s3-compliance runs
|
|
- [ ] Gradually reduce exclusion list as more features are implemented
|
|
- [ ] Range header support (would enable ~10 more tests)
|
|
- [ ] CopyObject support (would enable ~20 more tests)
|
|
- [ ] Idempotent CompleteMultipartUpload (Ceph-specific, 2 excluded tests)
|