This is the multi-page printable view of this section. .
PGSTY SILO Blog
- 1: Posts
-
2: Release Notes
- 2.1: SILO 20261026: Server, mcli and KES
- 2.2: SILO 20261018 Release Notes
- 2.3: silo-pkg 3.14.3 Released
- 2.4: PGSTY KES 20261018 Released
- 2.5: SILO Console 2.4.2 Released
- 2.6: mcli 20261018 Released
- 2.7: silo-pkg 3.14.2 Released
- 2.8: SILO 20261015: Superseded Draft Record
- 2.9: SILO Console 2.4.1 Released
- 2.10: silo-pkg 3.14.1 Released
- 2.11: SILO 20260916 Release Notes
- 2.12: mcli 20260916 Released
- 2.13: mcli 20260913 Released
- 2.14: silo-pkg 3.14.0 Released
- 2.15: SILO Console 2.4.0 Released
- 2.16: Silo Console 2.2.0 Release Notes
- 2.17: SILO Console 2.5: Interface and Administration Updates
- 2.18: silo-pkg 3.15: Go 1.27.3 and S3 SDK Fixes
- 2.19: Silo Console 2.1.0 Released
- 2.20: Silo Console 2.0.0 Released
- 2.21: silo-pkg 3.13.0 Released
- 2.22: silo-pkg 3.12.0 Released
- 2.23: Silo Pkg 3.11.0 Released
- 2.24: Silo 20260903 Released
- 2.25: Explicit Version Deletes Now Require DeleteObjectVersion
- 2.26: mcli 20260903 Released
- 2.27: mcli 20260806 Released
- 2.28: Silo 20260806 Released
- 2.29: Silo 20260804 Released
- 2.30: mcli 20260804 Released
- 2.31: Silo 20260618 Released
- 2.32: Silo 20260417 Released
- 2.33: Silo 20260325 Released
- 2.34: Silo 20260321 Released
- 2.35: Silo 20260314 Released
- 2.36: Silo 20260214 Released
- 2.37: Silo 20251203 Released
-
3: SILO Security Chronicle
- 3.1: October Security Update: Object Lock, Admin Credentials and JWK Parsing
- 3.2: SN-2026-015 & SN-2026-016: S3 Object Lock Governance Retention Bypass and Version-Addressed Overwrites
- 3.3: September Security Repairs: Payload Integrity, IAM Revocation and Console Sharing
- 3.4: SN-2026-011: Fix and Release Status
- 3.5: CVE-2025-62506: Session-Policy Privilege Escalation
- 3.6: SILO 20260903 Security Notes: SN-2026-006 through 010
- 3.7: CVE-2026-32285: The jsonparser Advisory That Required No Patch
- 3.8: CVE-2026-33322: OIDC JWT Algorithm Confusion
- 3.9: CVE-2026-33419: LDAP STS Enumeration and the Throttling Chain
- 3.10: CVE-2026-34204: Replication Metadata Injection
- 3.11: CVE-2026-39414: Oversized S3 Select Records and a SIMD Bypass
- 3.12: CVE-2026-40344: Snowball Auto-Extract Authentication Bypass
- 3.13: CVE-2026-41145: Unsigned-Trailer Query Authentication Bypass
- 3.14: CVE-2026-42600: ReadMultiple Storage-REST Path Traversal
- 3.15: SN-2026-002: Internode Path Containment Audit — Paying Off What CVE-2026-42600 Left Owing
- 3.16: SN-2026-003: s3:versionid Condition Evaluation and Request Condition Hardening
- 3.17: SN-2026-004: Object-Scoped Policy Grants Reaching Bucket Mutations
- 3.18: SN-2026-018: Client Source Address Trust, Forwarded Headers, and aws:SourceIp
- 3.19: SN-2026-019: Notification Keyspace Registration and Validation Redaction
- 3.20: Duplicate Part Numbers in CompleteMultipartUpload
-
4: Design Records
- 4.1: SILO, KES and OpenBao: Key Ownership and Recovery
- 4.2: Pinned TLS Parameters and Handshake Compatibility
- 4.3: Bucket Configuration Replication: Source Times, Deletions, and Deterministic Convergence
- 4.4: SigV4 Signed-Header Coverage: Rejecting Unsigned x-amz-* Headers
- 4.5: Durable IAM Revocations
- 4.6: Federated CopyObject: Preserve the Destination Contract
- 4.7: Multi-Pool Object Consistency
- 4.8: Object Lock Replication Ordering
- 4.9: SSE-C Replica Integrity
- 4.10: Replication Reliability: Delete Completion, MRF Visibility, and Resync Cancellation
- 4.11: FULL_OBJECT Checksum Replication: Preserve the Type at Upload Initiation
- 4.12: IAM Policy Reloads: Lock Scope, Cancellation, and Time Budgets
- 4.13: LDAP Long DNs: Restoring Login Without Hiding Revocation Errors
- 4.14: Replicated Tag Ordering: Revision Timestamps, Tombstones, and Resurrection
- 4.15: SILO Server 20260903 Pre-release Review
- 4.16: Overwrite Pool Affinity: Why #245 Was Not Adopted
- 4.17: Replica Metadata Normalization: What a Trusted Copy May Not Re-Inject
- 4.18: Request-Header Deadlines: Absolute Header Limits and Rolling Body Idle Timeouts
- 4.19: Conditional DELETE: One Condition, One Logical Object
- 4.20: DSN-Only Database Notifications: A Compatibility Boundary for #53
- 4.21: Preview Text, Never Execute It: SILO Console Text Preview Design
- 4.22: Multi-Pool Deletion: Preserve the Latest Copy on Failure
- 4.23: Go 1.27 TLS Defaults and OIDC Discovery Failure Modes
- 4.24: No I/O Before Auth, No Privilege From Headers
- 4.25: One Endpoint, Two Privileges: Separating User and Group Status
- 4.26: IAM Admin API Authorization Boundaries
- 4.27: Config Environment Files Are Not Shell Scripts
- 4.28: Two SSE-C Keys, One CopyObject Response
- 4.29: ListMultipartUploads Behavior and Design Trade-Offs
- 4.30: Why CompleteMultipartUpload Must Return ChecksumType: Review of PR #57
- 4.31: When the Total Is Unknown: Folder Download Progress
- 4.32: A Listing Must Not Drop a Null Version That Still Has Quorum
- 4.33: A ListObjects Shortcut Must Not Turn a Missing Bucket into an Empty One
- 4.34: Read-Only Checksum Audit and Reliable CLI Output
- 4.35: Optional Checksums, Mandatory Failure: Repairing UploadPart and UploadPartCopy Compatibility
- 4.36: BadDigest, InvalidRequest, and the CompleteMultipartUpload Checksum Contract
- 4.37: Clearing Stale Fields in Legacy Metadata and Scanner Responses
- 4.38: LIST Recovery: Confirm Only Uncertain Names and Bound the Waiting Cost
- 4.39: Object Lock Retention Bypass and Version-Overwrite Repair Design (SN-2026-015 / SN-2026-016)
1 - Posts
Essays and analysis about MinIO, S3-compatible object storage, and the SILO community fork.
1.1 - SILO at One Year: What Actually Changed Since Upstream MinIO?
On October 15, 2025, MinIO shipped its final public open-source release, RELEASE.2025-10-15T17-29-55Z (commit 9e49d5e7a648). Over the next four months, upstream placed the repository into “maintenance mode,” deleted the minio/console repository outright, and in February 2026 archived the main server repository—routing all subsequent CVE patches and bug fixes exclusively to its commercial AIStor product.
On October 26, 2026, SILO tagged RELEASE.2026-10-26T00-00-00Z (commit c3ad836aed0d). Across 376 days, the server repository accumulated 604 commits, evolving alongside its companion repositories silo-console, mcli, and silo-pkg.
A question we hear often from operators evaluating the fork is straightforward: Over the past year, did SILO merely rebrand the binary and apply surface-level CVE patches, or did it fix substantive storage and protocol bugs? Compared with the final upstream MinIO release, what actually changed?
Here is the short answer: SILO preserved MinIO’s erasure-coding storage layout (xl.meta v1.3), HighwayHash bitrot protection, and fundamental read/write quorum rules, while focusing a full year of engineering on security boundaries, S3 protocol and checksum correctness, replication and lifecycle convergence, failed-write rollback and multi-pool safety, and a restored administration console and CLI.
| Area | Key Substantive Changes (2025-10-15 → 2026-10-26) |
|---|---|
| Security & Authorization | Patched 7 post-fork CVEs (OIDC algorithm confusion, LDAP STS enumeration and brute-force, replication SSE header injection, S3 Select unbounded record OOM, unsigned-trailer signature bypass, Snowball extraction authorization bypass, and /rmpl path traversal) plus 18 SN-2026-* security boundary hardenings (enforced SigV4 SignedHeaders for X-Amz-* operational headers, policy condition key isolation, bucket-level write action boundaries, service-account delegation ceilings, Object Lock bypass validation, and durable IAM revocations against stale-event resurrection) |
| S3 Protocol & Checksums | Replaced upstream’s NotImplemented stubs with full per-bucket CORS persistence and OPTIONS preflight; implemented conditional DELETE (If-Match) and multi-pool conditional PUT / CompleteMultipartUpload; fixed plaintext checksums and logical part sizes across multipart uploads, CopyObject, SSE-C, and server-side compression; changed tampered-encryption errors from HTTP 206 to 500 |
| Replication & Lifecycle | Preserved FULL_OBJECT checksum types during multipart replication initiation (#255); preserved raw SSE-C ciphertext and seals across trusted replication peers; added modification timestamps to object tags and Object Lock retention for out-of-order convergence; prevented versioned lifecycle expiration from deleting remote tier data when only adding a delete marker (#253) |
| Storage & Multi-Pool Safety | Two-stage multi-pool bulk deletion (#257), migration version checks, LIST recovery under drive disagreement (#239), and storage RPC delete options for overwrite rollback; existing timeout/cancellation semantics remain, including the possibility of late commits, delayed rollback and stale reads |
| Console & mcli | Restored and rebuilt the full administration UI in SILO Console v2.5.1 (<3 MB zero-telemetry bundle, EN/ZH toggle, cursor pagination, Tasks center, Metrics V3, high-latency WebSocket diagnostic repairs); shipped mcli 20261026 with read-only checksum verify, nonzero exit codes on partial or permission failures, and safe mirror delete propagation |
| Operations & Delivery | Wired --read-header-timeout / MINIO_READ_HEADER_TIMEOUT (30s absolute HTTP/1 header deadline); added trusted-proxy settings (MINIO_API_TRUSTED_PROXIES) and native silo healthcheck; permanently removed self-update and SUBNET call-home; built a verified multi-arch supply chain on Go 1.27.3, UBI 9.8 / distroless images, RPM/DEB/APK packages, Windows ZIP archives, GPG signatures, SPDX SBOMs, and Sigstore attestations |
Equally important are the boundaries that have not changed:
- No distributed fencing, and no claim that every consistency problem is solved. Late commits after lease expiration or network partitions remain possible; strict generation read refusal, automatic cross-pool read recovery, public persistent
ListMultipartUploadsscanning (#79), access-frequency pool tiering, and multi-pool overwrite affinity (#245) were tested and withdrawn or deferred when their capacity or availability costs proved unacceptable. - No head-to-head full-cluster throughput benchmark between the two tags. We record targeted microbenchmarks and amortized overhead checks, not synthetic cluster-wide throughput claims.
- On-disk compatibility does not mean arbitrary mixed-version operation or rolling downgrade. Durable IAM revocations (
siloParentRevocation) and replication metadata tombstones require coordinated upgrades across participating nodes.
The sections below walk through each area in detail, with exact commit references, design records, and operational boundaries.
1. Baseline and Product Stack: What Changed and What Stayed the Same
Our comparison uses two fixed points:
- Upstream baseline:
minio/minioRELEASE.2025-10-15T17-29-55Z(commit9e49d5e7a648f00e26f2246f4dc28e6b07f8c84a, October 15, 2025). Note: between that final release and archiving the repo in February 2026, upstream merged 9 unreleased commits (such as POST policy tag constraints infa18589d1cand conditional PUT read-quorum adjustments in18f97e70b1), which never shipped in an upstream release and were addressed independently in SILO. - SILO target:
pgsty/siloRELEASE.2026-10-26T00-00-00Z(commitc3ad836aed0d8a7d23c2e44a5eaa3009d08525e3), released alongside mcli / KES 20261026, Console v2.5.1, and silo-pkg v3.15.2. See the release notes and component matrix.
Across 9e49d5e7a648..c3ad836aed0d, pgsty/silo contains 604 reachable commits spanning feature and security repairs, regression suites, packaging, and documentation. The sections below organize those changes by delivered runtime behavior.
The Four-Component Coordinated Stack
Because the server embeds the Console frontend and imports mc administration packages alongside shared policy and encryption libraries, maintaining the server alone is not enough. SILO maintains a coordinated four-component product graph (plus optional external KES):
| Component | Upstream 2025-10-15 State |
SILO 20261026 State |
Key Evolution |
|---|---|---|---|
Server (pgsty/silo) |
Go 1.24.0, minio binary |
Go 1.27.3, silo binary (c3ad836aed0d) |
Retains github.com/minio/minio module path; delivers 7 CVE fixes, 18 security boundary hardenings, and storage/replication repairs |
Console (pgsty/silo-console) |
Stripped-down v1.7.7 (object browser only) |
v2.5.1 (fff1618d9abb) |
Upstream deleted minio/console; SILO restored the full admin console with a <3 MB zero-telemetry bilingual frontend |
Client (pgsty/mc) |
mc cf909e1063a9 |
mcli 20261026 (0ceb298cdcd0) |
Shipped as mcli (avoiding collision with Midnight Commander mc), adding checksum verify and honest error exit codes |
Shared Pkg (pgsty/silo-pkg) |
minio/pkg/v3 v3.1.3 |
v3.15.2 (0416eb1f4505) |
Maintained directly as github.com/pgsty/silo-pkg/v3 for IAM policy evaluation, condition key isolation, TLS trust, and KMS |
S3 SDK (minio/minio-go/v7) |
v7.0.91 |
v7.4.2-0.20261025165926-320af050c445 |
Uses the verified upstream module commit (including short-stream multipart, negative-offset ReadAt, and batch-delete fixes) |
KMS (pgsty/kes) |
External optional service | KES 20261026 (e9b4f1acebc1) |
Go 1.27.3; OpenBao KV v2 and local Moto Secrets Manager compatibility checks; updated AWS and GCP clients |
Foundations Intentionally Preserved
So existing MinIO deployments can switch to pgsty/silo in place without data migration or client changes, SILO keeps the core storage and wire contracts unchanged:
- Erasure coding and bitrot protection: Reed-Solomon erasure coding and per-block HighwayHash256 verification remain unchanged.
- Read and write quorum rules: Read quorum remains $K$ (data shards); write quorum remains $K$ when data and parity shard counts differ, and $K+1$ when data shards equal parity shards (for example, 16-drive 12+4 uses R12/W12, while 8+8 uses R8/W9).
- On-disk format and system namespace:
xl.metastays at version1.3and.minio.sysretains its layout, allowing existing drives to mount directly. - Wire and configuration compatibility:
MINIO_*,MC_*, andCONSOLE_*environment variables,/minio/health/*probes,/minio/v2/metrics/*and/minio/metrics/v3scrape endpoints, andX-Minio-*headers remain supported (see the compatibility overview and migration guide).
2. Security, Authentication, and Authorization: 7 CVEs and 18 Security Boundary Hardenings
Because SILO retains the github.com/minio/minio Go module path while upstream advisories publish open-ended affected version ranges, automated scanners frequently flag SILO for every historical MinIO CVE.
To separate fact from scanner noise: vulnerabilities such as CVE-2025-62506 (service-account privilege escalation), CVE-2025-31489, and CVE-2025-27414 were already fixed in upstream’s 2025-10-15 baseline. By contrast, the 7 CVEs below were disclosed after 2025-10-15, left unpatched in upstream open-source MinIO, and fixed and verified in SILO:
| CVE / Advisory | Severity | Root Cause and Impact | SILO Fix and Code Evidence |
|---|---|---|---|
| CVE-2026-33322 | Critical (9.8) | OIDC JWT algorithm confusion: when a token header specified HMAC (HS256), upstream verified it using the OIDC ClientSecret as a symmetric key, letting anyone with ClientSecret forge arbitrary identities including consoleAdmin |
Enforce asymmetric JWKS public-key verification only and reject HMAC-signed OIDC tokens (cmd/jwt.go, internal/config/identity/openid/) |
| CVE-2026-33419 | High | LDAP STS endpoint leaked whether usernames existed and lacked rate limiting, enabling username enumeration and password brute-forcing for STS credentials | Unify bind-failure authentication errors and enforce sliding-window rate limiting keyed by trusted client IP + normalized username without penalizing successful logins (cmd/sts-handlers.go) |
| CVE-2026-34204 | High | Unauthenticated X-Minio-Replication-* headers on ordinary PUT/COPY requests could inject broken encryption metadata and render objects permanently unreadable |
Strip unauthorized replication headers and SSE replication metadata at request ingress (cmd/auth-handler.go, cmd/object-handlers-common.go) |
| CVE-2026-39414 | High | S3 Select accumulated newline-free CSV or JSON Lines streams without bound, allowing a single request to trigger an out-of-memory (OOM) crash | Enforce strict per-record byte limits in the CSV and JSON Lines parsers (internal/s3select/) |
| CVE-2026-41145 | High | STREAMING-UNSIGNED-PAYLOAD-TRAILER write paths bypassed signature verification on specific routes, allowing anonymous or forged-signature object writes |
Enforce full authentication and signature validation across all aws-chunked and unsigned-trailer ingress paths (cmd/auth-handler.go) |
| CVE-2026-40344 | High | Snowball archive batch-extraction upload path lacked entry-point authentication and per-entry authorization checks | Require caller authentication at ingress and evaluate bucket/prefix policy authorization for every extracted archive entry (cmd/bucket-handlers.go, cmd/object-handlers.go) |
| CVE-2026-42600 | High | Internode storage endpoint /rmpl (ReadMultiple) lacked drive-root path containment, allowing path traversal outside the drive root |
Remove the unused ReadMultiple endpoint from the internode storage REST protocol and enforce strict drive-root path containment across all storage RPC handlers (cmd/storage-rest-server.go) |
18 SN-2026-* Security Boundary Hardenings
Beyond public CVEs, adversarial code review and multi-node testing uncovered 18 additional security boundary defects tracked in our Security Advisories ledger:
- SigV4 SignedHeaders Integrity and Policy Condition Isolation (SN-2026-011, SN-2026-003):
- Upstream allowed callers to attach unsigned
X-Amz-*operational headers (such as governance bypass, tagging, or checksum headers) to an otherwise valid SigV4 request. SILO requires everyX-Amz-*operational header present on the request to be covered bySignedHeaders. - Upstream populated IAM policy condition maps directly from client headers and query parameters, allowing callers to spoof server-authoritative condition keys such as
s3:prefix,s3:versionid,s3:ExistingObjectTag/*, oraws:PrincipalType. SILO isolates server-authoritative condition keys insilo-pkgand bindss3:VersionIdchecks to the actual target version inDeleteObjectsandGetObject.
- Upstream allowed callers to attach unsigned
- Bucket-Level Write Boundaries, Version-Deletion Authorization, and Admin Action Splits (SN-2026-004, SN-2026-005, SN-2026-009 and 010, SN-2026-017):
- Tightened resource ARN matching for 12 bucket-level write actions (
s3:CreateBucket,s3:DeleteBucket,s3:PutBucketPolicy,s3:PutLifecycleConfiguration, etc.) so object wildcard grants (arn:aws:s3:::bucket/*) or barearn:aws:s3:::ARNs cannot mutate bucket configurations. - Require
s3:DeleteObjectVersionfor permanent deletion of an explicit version ID, includingnull; an ordinary delete that creates a delete marker still usess3:DeleteObject. Batch deletion checks each entry independently. - Split user status toggling out of
admin:CreateUserintoadmin:EnableUser/admin:DisableUser; separated self-service password changes intoadmin:ChangeMyPassword(see password-permission migration); and enforced parent-account ownership checks across IAM admin access-key creation, inspection, and listing endpoints.
- Tightened resource ARN matching for 12 bucket-level write actions (
- Service-Account Delegation Ceilings and Object Lock Governance Bypass (SN-2026-015 and 016):
- Fixed inline session-policy evaluation so parent
Denyrules are never dropped when the inline policy omits a matchingAllow, and re-verified live parent privileges on temporary STS and service-account sessions. - Required
x-amz-bypass-governance-retention: truewhenPutObjectoverwrites aGOVERNANCE-locked object, and validated retention XML payloads before evaluating Object Lock governance bypass inPutObjectRetention.
- Fixed inline session-policy evaluation so parent
- Durable IAM Revocations, Policy Reload Lock Isolation, and LDAP Long DNs (SN-2026-013):
- Previously, in multi-node and site-replicated clusters, delayed site-replication events or missed peer notifications could resurrect deleted users, policies, or service accounts. SILO persists deletion tombstones, revision numbers, and parent revocation cutoffs (
siloParentRevocation) across both object-backed IAM and STS enumeration, and reloads committed state on peer delete notifications. - Moved slow IAM policy loads and KMS decryption outside the hot authentication-cache lock while propagating KMS request cancellation (design record); fixed LDAP
StartTLSprecedence and hex-escaped long DN path-component limits (design record).
- Previously, in multi-node and site-replicated clusters, delayed site-replication events or missed peer notifications could resurrect deleted users, policies, or service accounts. SILO persists deletion tombstones, revision numbers, and parent revocation cutoffs (
3. S3 Protocol Completeness and End-to-End Checksums
On the S3 protocol surface, SILO focused on three goals: implementing stubbed-out bucket APIs, aligning multi-pool conditional write semantics, and ensuring end-to-end plaintext checksum and size accuracy across multipart, encrypted, and compressed objects.
| Feature / Scenario | Upstream MinIO (2025-10-15) |
SILO (20261026) |
Design Record |
|---|---|---|---|
| Per-bucket CORS | GetBucketCors/PutBucketCors/DeleteBucketCors returned NotImplemented stubs |
Full per-bucket CORS XML persistence (up to 100 rules, 64 KiB), dynamic OPTIONS preflight matching, site replication, and bucket metadata export/import |
Per-Bucket CORS |
Conditional DELETE |
No If-Match support on DELETE |
Single-object DELETE supports If-Match: "<etag>" and If-Match: * under the shared object lock; batch DeleteObjects does not support per-entry ETag conditions |
Conditional DELETE |
Multi-pool conditional PUT & multipart completion |
Checked only the target write pool, allowing stale pool copies to accept outdated ETags or reject current ones | Resolves the logical current object across all storage pools (including draining pools) before evaluating If-Match / If-None-Match |
Multi-Pool Object Consistency |
| Server-computed part checksums | A checksum-enabled multipart upload rejected parts without explicit checksum headers, including copied parts | Computes plaintext part checksums using the algorithm declared at upload initialization, before compression or encryption, and persists them in xl.meta; uploads without a declared algorithm do not acquire a default CRC32C checksum |
Part Checksums |
CompleteMultipartUpload verification |
Permissive on duplicate part numbers, missing checksums, or FULL_OBJECT type mismatches |
Enforces strictly ascending part numbers (InvalidPartOrder on duplicates) and validates part checksums and FULL_OBJECT / COMPOSITE rollups |
Checksum Integrity Repairs |
CopyObject / UploadPartCopy checksums |
Server-side copies of encrypted or compressed objects could copy ciphertext bitrot hashes and omit ChecksumType in responses |
Computes or verifies checksums against plaintext logical bytes and returns accurate ChecksumType in CopyObjectResult / CopyPartResult |
Checksum Integrity Repairs |
| SSE-C same-object key rotation & zero-byte objects | In-place CopyObject with a new SSE-C key updated metadata without re-encrypting payload bytes; zero-byte SSE-C objects and part attributes skipped unsealing |
Rewrites ciphertext on same-object SSE-C key rotation and validates customer keys on zero-byte SSE-C reads and GetObjectAttributes part listings |
SSE-C Key Rotation, Zero-Byte Validation |
| Logical part sizes for encrypted/compressed objects | GetObjectAttributes and ?partNumber=N could report physical encrypted/compressed shard sizes |
Returns exact plaintext logical part sizes across GetObjectAttributes and part-number GET/HEAD requests |
Multipart Part Logical Size |
ObjectTampered HTTP status |
Returned HTTP 206 Partial Content when stored encryption metadata was tampered with |
Returns HTTP 500 (XMinioObjectTampered) so clients and caches never treat tampered objects as valid partial responses |
ObjectTampered 500 Status |
DNS federation CopyObject |
Cross-cluster federated copies could stream raw .minio.sys ciphertext, leaving the destination object undecryptable |
Streams decrypted plaintext via standard S3 client reads and re-seals at the destination | Federated CopyObject |
4. Replication and Lifecycle Correctness
In clusters running active-active replication or remote tiering, the most damaging bugs are silent divergence between sites or premature deletion of tiered data. Over the past year, SILO fixed six replication and lifecycle failure modes:
| Area | Upstream Behavior and Risk | SILO Behavior |
|---|---|---|
Multipart FULL_OBJECT checksum replication (#255) |
When replicating a multipart object with a CRC32/CRC32C FULL_OBJECT checksum, the source omitted x-amz-checksum-type: FULL_OBJECT on NewMultipartUpload, causing the target to validate as COMPOSITE and fail CompleteMultipartUpload with BadDigest |
Propagates ChecksumType in putReplicationOpts so multipart replication and CopyObject share the full checksum header contract (design record) |
| SSE-C replica integrity | Replicating SSE-C objects regenerated KMS/DARE seals or corrupted the customer-key seal on the target replica | Preserves raw SSE-C ciphertext, random IVs, and cryptographic seals across trusted replication peers (design record) |
| Timestamped tag deletion & out-of-order convergence | Tag deletion sent an empty string without advancing a revision timestamp, letting delayed replication events resurrect deleted tags; SSE-KMS copies also dropped tag revision timestamps | Adds replication-tag-modtime timestamps and deletion tombstones, preserves timestamps across SSE-KMS copies, and rejects stale out-of-order tag events (design record) |
| Object Lock retention ordering | Concurrent cross-site Object Lock retention updates lacked deterministic ordering, allowing stale events to overwrite newer retention dates | Adds replication-retention-modtime and converges retention state under persistence locks (design record) |
| Delete-marker replication & version purge | Exact-version purges could create delete markers on drives missing the version, and MRF retry queues could conflate marker creation with permanent version purge | Separates delete-marker creation from permanent version purge, re-checks source state under the replication lock before sending queued marker creations, and requires write-quorum majority confirmation for missing-version purges (design record) |
Tiered versioned lifecycle expiration (#253) |
When a lifecycle Expiration rule on a versioned bucket (or versioning-suspended bucket with non-null versions) only wrote a delete marker, upstream could still delete the remote tier payload backing historical versions |
Distinguishes delete-marker insertion from permanent version expiration, keeping remote tier payloads intact whenever expiration only adds a delete marker |
SILO also isolated per-job and per-target cancellation contexts and fixed ReplicatedCount / FailedCount accounting in replication resync (Replication Resync Correctness), and added per-bucket locks plus optional field-level tombstones (MINIO_SITE_REPLICATION_METADATA_TOMBSTONES) for bucket configuration convergence (Bucket Metadata Convergence).
5. Storage Mutation, Rollback, and Multi-Pool Safety
In an erasure-coded storage engine, the happy path is straightforward; edge-case bugs hide at the intersection of slow drives, partial-write rollbacks, concurrent overwrites, and multi-pool scaling. The 20261026 release ships three groups of storage-layer safeguards:
1. Commit Error Classification and Overwrite Rollback RPC Forwarding
- Missing staging
xl.metaclassified as503 SlowDownWrite: When a slow drive or concurrent cleanup causes a stagingxl.metafile in.minio.sys/tmpto go missing during commit, SILO classifies the condition aserrFileNotFoundand returns HTTP503 SlowDownWriteso clients can retry safely instead of receiving a false404 ObjectNotFound(error classification design). - Overwrite rollback option forwarding and timeout boundaries: SILO forwards storage RPC deletion options (including the previous data directory needed for overwrite rollback) while preserving existing drive/RPC deadlines, request cancellation, object locking, read quorum, and healing rules. Note that a client-visible timeout does not mean underlying disk or RPC I/O has stopped or that the write failed to commit—late commits or delayed
UndoWritecalls can still occur after partial failures, lease loss, or network partitions (the admitted-I/O draining waits and per-drive mutation guards explored in the October maintenance draft were withdrawn indcc2f20ba; see the final maintenance boundaries).
2. Multi-Pool Locking, Two-Stage Bulk Deletion, and Migration Safety
For multi-pool deployments:
- Shared cross-pool namespace locks: Fixed pool-local lock isolation where concurrent writes, deletes, and metadata updates on different pools failed to serialize against one another.
- Two-stage multi-pool deletion (
#257): Previously, multi-poolDeleteObjectsissued parallel deletes across all pools; if the pool holding the newest version succeeded while a pool holding an older copy failed due to a drive error, the stale copy resurfaced as the current object. SILO now deletes in two stages—cleaning non-authoritative duplicate pools first and removing the authoritative copy only after stale copies are gone, while protecting referenced remote tier objects on partial failure (Multi-Pool Delete Safety). - Rebalance and decommission safeguards: Preserved object tags and revision timestamps across pool migration and prevented decommission from overwriting newer versions or dropping delete markers on the destination pool (Multi-Pool Object Consistency).
3. LIST Recovery Under Drive Disagreement (#239)
During rolling restarts with concurrent overwrites, minority drive disagreements or unresolved entries could cause ListObjects to omit readable keys. While keeping the lock-free fast path intact, SILO now confirms eligible unresolved latest-object entries on demand, distinguishes real deletions from drive read errors, and propagates terminal errors through pagination (LIST Recovery Design Record).
6. Restoring the Full Control Plane: SILO Console v2.5.1 and mcli 20261026
For day-to-day operators, the most visible difference between SILO and upstream MinIO is the administration control plane.
SILO Console v2.5.1: From a Stripped Browser Back to a Full Admin Console
In mid-2025, upstream stripped the embedded community Console down to v1.7.7—removing cluster administration, IAM, policies, lifecycle, replication, diagnostics, and monitoring panels to leave only a bare object browser—and subsequently deleted github.com/minio/console.
Building on the community restoration initiated by Georg Mangold (@gmangold), SILO shipped six major Console releases (v2.0.0 through v2.5.1), restoring the full administration suite while making the frontend smaller, faster, and telemetry-free:
| Dimension | Upstream 2025-10-15 Embedded Console (v1.7.7) |
SILO Console v2.5.1 |
|---|---|---|
| Administrative Scope | Basic bucket and object browser only; all admin panels removed | Full administration console: Overview dashboard, Buckets, Object Browser, Lifecycle, Replication, Tiering, Notifications, IAM (Users/Groups/Service Accounts/Policies with built-in policy Reset), IDP (OIDC/LDAP), KMS, Diagnostics (Health/Trace/Logs/Watch/Speedtest/Profile), and background Tasks |
| Bundle Size & Telemetry | ~10 MB bundle with external font/script dependencies and call-home hooks | < 3 MB byte-for-byte reproducible bundle, zero telemetry, zero external network dependencies, responsive light/dark themes |
| Localization | English only | Native English and Simplified Chinese toggle across UI, error messages, and documentation links |
| Metrics Integration | Legacy endpoints or removed | Native Metrics V3 integration that distinguishes real zeros from missing node data |
| Large-Bucket Browsing & Previews | Unbounded directory fetches prone to browser freezes | Cursor pagination (default 100, up to 1,000 per page), bounded 1 MiB text/Markdown/CSV/code preview, and native streaming ZIP multi-object downloads |
High-Latency Diagnostics (#63) |
Health-report downloads over high-latency links failed with WebSocket 1006 / 1011 truncation |
Fixed WebSocket close handshakes across all six diagnostic streams and repaired the madmin v3 stream decoder pre-read bug, verified under 250 ms each-way injected latency |
Console 2.5.1 also fixes group membership during status changes, OIDC group permissions, TLS-proxy session cookies, credential exports and keyboard focus. See the Console 2.5 series notes.
mcli 20261026: Honest Exit Codes and Read-Only Checksum Verification
On the client side, pgsty/mc ships as mcli (while preserving MC_* environment variables and ~/.mc / ~/.silo configuration compatibility) and fixes long-standing CLI reliability gaps:
- Read-only
mcli checksum verify: Verifies end-to-endFULL_OBJECTchecksums across buckets, prefixes, or object versions directly against the server without mutating objects or writing local files, supporting--versions,--Algo, and--json. - Nonzero exit codes on partial or permission failures:
mcli admin info --jsonnow exits nonzero when access is denied or a node reportsstatus: error(upstream exited0);mcli mirror,mcli mv,mcli retention,mcli legalhold, andmcli sqlall return nonzero exit codes on partial object failures, source-deletion errors, or permission denials;- Downloads interrupted mid-stream fail immediately rather than reporting a truncated file as complete.
- Safe
mirrordelete propagation and--dry-runrestarts: Preventedmirror --removefrom deleting destination objects when source listing fails; added--dry-runtomcli admin service restart; and honored optional persistedregionsettings in alias configurations.
The 20261026 client also isolates cached identities, drains resync JSON through terminal states, fixes rate overflow, returns nonzero for partially failed start/cancel operations, and adds batch pending output with bounded fallback when live metrics are missing. See the mcli changes and script migration notes.
7. Operations, Configuration, and Supply-Chain Delivery
Across server operations and artifact delivery, SILO made four practical changes:
- Enforced request-header timeouts and trusted-proxy boundaries:
- Upstream defined
--read-header-timeout/MINIO_READ_HEADER_TIMEOUTwithout wiring it intohttp.Server, leaving HTTP/1 connections vulnerable to slowloris-style trickle headers. SILO wired the setting into the HTTP server with a 30-second absolute HTTP/1 header deadline (design record). - Added
MINIO_API_TRUSTED_PROXIESandMINIO_IDENTITY_LDAP_STS_TRUSTED_PROXIESsoX-Forwarded-Foris never trusted blindly by default; addedCONSOLE_WS_MAX_MESSAGE_BYTESandCONSOLE_WS_MAX_DECOMPRESSED_BYTESto bound embedded Console WebSocket frames. - Added a native
silo healthchecksubcommand for shell-free distroless containers and tightenedMINIO_CONFIG_ENV_FILEparsing to reject malformed lines.
- Upstream defined
- Hardened event notification targets:
- Fixed NATS reconnect handling and JetStream async reply subjects, AMQP confirm-mode lock contention, PostgreSQL/MySQL TLS configuration loading, and preserved Kafka’s default protocol version
2.1.
- Fixed NATS reconnect handling and JetStream async reply subjects, AMQP confirm-mode lock contention, PostgreSQL/MySQL TLS configuration loading, and preserved Kafka’s default protocol version
- Permanently removed self-update and call-home:
- Removed in-place binary self-update (
MINIO_UPDATE) and allSUBNETregistration, telemetry, license checks, and phone-home endpoints. Upgrades are managed cleanly through OS packages (RPM/DEB/APK) and container images.
- Removed in-place binary self-update (
- Rebuilt a verifiable multi-architecture supply chain:
- Built all components with Go 1.27.3 (including built-in
crypto/x509distrusted-intermediate checks), UBI 9.8 and distroless container variants, staticcurl 8.23.0, and Red Hat’sRHSB-2026-012root CA blocklist. - Published Linux (
amd64/arm64/ppc64le/s390x), macOS (amd64/arm64), and Windows (amd64/arm64ZIP) archives, RPM/DEB/APK packages, and multi-arch Docker Hub images (pgsty/silo,pgsty/silo-console,pgsty/mc,pgsty/kes) backed by GPG signatures, SPDX SBOMs, and GitHub Sigstore build attestations (see Downloads).
- Built all components with Go 1.27.3 (including built-in
8. Honest Boundaries: Withdrawn Experiments and Open Limits
Maintaining storage software in production requires knowing what to roll back when an idea does not survive real capacity or failure testing. Over the past year, we implemented, benchmarked, and deliberately withdrew or closed four designs:
- Withdrew public persistent
ListMultipartUploadsdisk scanning (#79):- Goal: Upstream’s
ListMultipartUploadsreads only a node-local in-memory cache, losing visibility of incomplete uploads across restarts. In September we implemented a persistent scan across storage pools and erasure sets. - Why we rolled it back: Capacity testing showed scan latency grew linearly with prefix depth and abandoned upload count, timing out at 20,000 incomplete uploads and amplifying drive I/O. In Server 20261018 we withdrew public persistent scanning and retired the optional
strictmode, returning to the node-local cache plus exact-key lookup (see ListMultipartUploads Design and Capacity Evidence).
- Goal: Upstream’s
- Deferred strict generation read refusal and automatic cross-pool read recovery:
- Goal: When partial overwrite failures left split object generations across drives or pools, we tested refusing divergent reads or auto-recovering across pools during
GET. - Why we deferred it: Fault-injection testing showed that refusing reads after a failed partial overwrite broke MinIO’s established availability contract (where the prior readable version remains accessible). Those strict diagnostics remain gated behind the non-default
consistency_strictbuild tag while production builds preserve existing read-quorum and healing behavior.
- Goal: When partial overwrite failures left split object generations across drives or pools, we tested refusing divergent reads or auto-recovering across pools during
- Closed multi-pool overwrite affinity without merge (#245):
- Goal: In multi-pool clusters, unversioned overwrites can land on a different pool and leave stale copies on the old pool. PR
#245routed overwrites back to the object’s existing pool. - Why we closed it: When the historical pool lost write quorum due to failed drives while another pool remained healthy, forced affinity caused overwrites to fail outright—and two rounds of multi-node
m3benchmarks showed small-object overwrite latency still missed our bar (see Overwrite Pool Affinity).
- Goal: In multi-pool clusters, unversioned overwrites can land on a different pool and leave stale copies on the old pool. PR
- Removed experimental access-frequency intra-cluster tiering (
ilm access_*):- The experimental feature never met production reliability requirements and was removed in Server 20260916 (see Access-Frequency Tiering Removal).
Remaining Operational and Upgrade Boundaries
When evaluating or upgrading to SILO, keep these boundaries in mind:
- No distributed lock fencing: If a node loses its lock lease during a GC pause, network partition, or stalled drive, an already-admitted commit on that node can still land late; partial metadata updates can still briefly expose stale tags or attributes. The release retains existing timeout semantics; a returned request is not proof that underlying I/O has stopped.
- No unverified cluster-wide throughput claims: We verify hot-path microbenchmarks (for example,
silo-pkg v3.14.3reducedContainsAdminActionfrom 24 allocations and742.5 ns/opto zero allocations and5.53 ns/op), but we have not run a head-to-head multi-node benchmark between2025-10-15and2026-10-26on dedicated hardware. - Coordinated upgrades for new IAM and replication state: While on-disk
xl.metav1.3 remains compatible, features such as durable IAM revocations (siloParentRevocation) and site-replication metadata tombstones require all participating nodes to upgrade together rather than running mixed versions indefinitely or performing rolling downgrades (see the Component Matrix and IAM Upgrade Guide). - Known minor items: The repository’s Helm chart (
7.0.3) default server image tag still points toRELEASE.2026-09-16T00-00-00Z(overrideimage.taginvalues.yamlwhen deploying newer releases), and two legacy batch-lifecycle handlers in Console (batch expiration and batch transition creation) have early-return/empty-slice quirks that do not affect single-rule configuration and are queued for the next Console patch.
Closing Note
When we forked MinIO a year ago, the goal was simple: keep a production-grade open-source S3 server alive with timely CVE patches, a complete administration console, honest bug fixes, and clean binary and container distributions.
One year later—604 commits, 7 CVE patches, 18 security boundary hardenings, and dozens of root-cause write-ups in our Design Archive and Security Archive—that is the ledger of what SILO has done.
If you are running open-source MinIO, see the Migration Guide and Downloads page. If you hit a reproducible protocol, replication, or storage bug in production, open an issue at pgsty/silo.
1.2 - MinIO Fork, Promise Kept
Two months ago in “MinIO is Dead, Long Live MinIO,” I promised I’d keep the MinIO fork patched. The recurring objection on HN is fair: can one person actually maintain something like this? The real answer isn’t clicking fork. It’s what happens when CVEs start landing.
Between April 15 and 17, pgsty/minio shipped RELEASE.2026-04-17,
closing four CVEs and a handful of related vulnerabilities disclosed in the same window.
The scope I committed to originally was narrow: no new features, keep the supply chain running, handle reproducible bugs and security issues as they come in. This release is what it looks like when that promise gets tested.
What happened upstream
In December 2025, MinIO moved the open-source repo to maintenance mode. The README said security fixes would be “evaluated case by case.” In February 2026, the repository was archived and the landing page became “this repository is no longer maintained.”
The SECURITY.md in that same archived repo still says: “we will always provide security updates for the latest release.”
Over the past month, four high-severity and two medium-severity vulnerabilities have been disclosed against the final open-source release.
It’s been 184 days since the last upstream release. Vulnerabilities get disclosed; fixes ship only in the commercial build. The guidance for OSS users is a single line: upgrade to AIStor.
AIStor starts around $100k/year for 400 TiB — roughly S3 pricing, for software you install and operate yourself.
It’s a clean arrangement: archive the repo so there’s no obligation to patch, keep publishing CVE advisories for visibility, and route everyone who reads them toward the commercial product.
Someone still has to patch the old one.
What this release fixes
Full write-ups, CVSS arithmetic, and PoCs are in the release notes. The short version:
- CVE-2026-33322 (OIDC JWT algorithm confusion, CVSS 9.8): under certain IdP configurations, an attacker who knows the OIDC ClientSecret can mint a token claiming any identity — including
consoleAdmin— and MinIO will accept it. Vulnerable window: November 2022 through March 2026. About three and a half years. - CVE-2026-33419 (LDAP STS enumeration and brute-force): the login endpoint leaks which usernames are real, and there’s no rate limiting on the subsequent password guessing. End of the chain is an STS credential.
- CVE-2026-34204 (replication-header metadata injection): a regular PUT or COPY with certain
X-Minio-Replication-*headers can write an object into a permanently unreadable state. The data is still on disk; you just can’t read it back out. - CVE-2026-39414 (S3 Select memory exhaustion): one request, one OOM.
- GHSA-hv4r-mvr4-25vw / GHSA-9c4q-hq6p-c237: two signature-verification bypasses on the unsigned-trailer path. Anonymous or forged-signature requests can successfully write objects on certain routes.
Plus the usual dependency cleanup from go-jose, go.opentelemetry.io, and the Go 1.26.2 upgrade itself — about twenty security items in total counting transitive dependencies.

How it got fixed
I said in the earlier post that I’d rely on AI coding agents, and that’s how this round went. My role was closer to “review and decide” than “write code.”
Per-issue flow, roughly:
- Codex drafts first. Given the CVE description and relevant code paths, it produces an initial patch.
- Claude Code reviews adversarially. Picks holes from the attacker’s side.
- Back to Codex. If it agrees with Claude Code’s critique, it reworks. If not, it has to write out why. No silent overrides.
- Another round of review by Claude Code, with both sides’ reasoning on the table. Iterate until they converge.
- Tests. Codex proposes cases, Claude Code adds more, Codex runs them, Claude Code reviews the results.
- I decide. Read the diff, run the tests, merge or send it back with comments.
I didn’t write any of the code in this round. My job was to define the problem, set constraints, pick between approaches, read diffs, run tests, and merge. The GitHub log shows Vonng, Codex, and Claude Code as co-authors — that’s just who did the work.

A few things I noticed about how this runs in practice.
Two heterogeneous agents in opposition catch more than one agent alone. A single agent patching a security bug tends toward confident-sounding fixes that quietly miss a boundary condition. Having a second agent argue against the first filters out most of those.
It forces the tradeoffs into writing. When two implementations diverge, someone has to say why A over B. That exchange is the thing I can actually act on as the person deciding what to merge.
Real maintenance is patch-on-patch, not one-shot. The LDAP STS fix is a good example. The first version landed, and then we realized: successful requests shouldn’t count against the rate limit; X-Forwarded-For shouldn’t be trusted by default; the limiter should key on source IP plus normalized username, not just one. Three follow-up commits before it settled. Iterating through that by hand would have cost a lot more time.
Why this fork exists
Because I use MinIO myself.
MinIO is a production dependency for Pigsty. I need working binaries, a complete console, packages that keep shipping, and someone actually handling CVEs. That keeps the scope narrow. No new features, no turning the repo into a playground. Compatibility, supply chain, fixes when they’re needed.
— Chainguard also ships MinIO container images that track upstream’s post-archive commits, a useful option if you use their images. This fork is a different shape: source tree, RPM/DEB packages, restored console, and doesn’t depend on upstream continuing to push patches somewhere.
The fork is at about 1,300 stars on GitHub and 50,000+ pulls on Docker Hub now. Not remarkable numbers, but enough to tell me I’m not the only one who needed this fork to keep shipping.

2026-10-25 update: The maintained Docker image is now
pgsty/silo. The Docker example below uses the current name; see the migration guide.
If you’re already running OSS MinIO, migration is cheap:
- Docker: swap
minio/minioforpgsty/silo. - RPM / DEB: on GitHub Releases, or via
pig. - Source: pgsty/minio
- Docs: silo.pgsty.com
You don’t need to replace anything around it or relearn the API. In most cases, you’re just pointing a compatible binary at the same deployment. If you want a full HA production setup, Pigsty ships one for free.
Something I use broke; I’m fixing it.
What’s different in 2026 is the cost of “I’m fixing it.” With two coding agents and someone to referee between them, the maintenance load of a mid-sized Go codebase is tractable for one person in a way it wasn’t a year or two ago. That’s about it — not a grand theory about open-source resilience, just the current operating point.
If you’re running OSS MinIO, the migration is cheap and the patches are current. If another CVE drops, I’ll still be here.
1.3 - MinIO Is Dead, Long Live MinIO
MinIO’s open-source repo has been officially archived. No more maintenance. End of an era — but open source doesn’t die that easily.
I created a MinIO fork, restored the admin console, rebuilt the binary distribution pipeline, and brought it back to life.
2026-10-25 update: The maintained Docker image is now
pgsty/silo. The Docker links and examples below use the current name; see the migration guide.
If you’re running MinIO, swap minio/minio for pgsty/silo.
Everything else stays the same. (CVE fixed, and the console GUI is back)
The Death Certificate
On December 3, 2025, MinIO announced “maintenance mode” on GitHub. I wrote about it in MinIO Is Dead.
On February 12, 2026, MinIO updated the repo status from “maintenance mode” to “no longer maintained”, then officially archived the repository. Read-only. No PRs, no issues, no contributions accepted. A project with 60k stars and over a billion Docker pulls became a digital tombstone.

If December was the clinical death, this February commit was the death certificate.
Today (Feb 14), a widely circulated article titled How MinIO went from open source darling to cautionary tale laid out the full timeline.

Percona founder Peter Zaitsev also raised concerns about open-source infrastructure sustainability on LinkedIn. The consensus in the international community is clear:
MinIO is done.
Looking back at the timeline over the past years, this wasn’t a sudden death. It was a slow, deliberate wind-down:
| Date | Event | Nature |
|---|---|---|
| 2021-05 | Apache 2.0 → AGPL v3 | License change |
| 2022-07 | Legal action against Nutanix | License enforcement |
| 2023-03 | Legal action against Weka | License enforcement |
| 2025-05 | Admin console removed from CE | Feature restriction |
| 2025-10 | Binary/Docker distribution stopped | Supply chain cut |
| 2025-12 | Maintenance mode announced | End-of-life signal |
| 2026-02 | Repo archived, no longer maintained | End of project |
A company that raised $126M at a billion-dollar valuation spent five years methodically dismantling the open-source ecosystem it built.
But Open Source Endures
Normally this is where the story ends — a collective sigh, and everyone moves on.
But I want to tell a different story. Not an obituary — a resurrection.
MinIO Inc. can archive a repo, but they can’t archive the rights that the AGPL grants to the community.
Ironically, AGPL was MinIO’s own choice. They switched from Apache 2.0 to AGPL to use it as leverage in their disputes with Nutanix and Weka — keeping the “open source” label while adding enforcement teeth. But open-source licenses cut both ways — the same license now guarantees the community’s right to fork.
Once code is released under AGPL, the license is irrevocable. You can set a repo to read-only, but you can’t claw back a granted license. That’s the beauty of open-source licensing by design: a company can abandon a project, but it can’t take the code with it.
So — MinIO is dead, but MinIO can live again.
That said, forking is the easy part. Anyone can click the Fork button. The real question isn’t “can we fork it” but “can someone actually maintain it as a production component?”
Why would I do that?
I didn’t set out to take this on. But after MinIO entered maintenance mode, I waited a couple of weeks for someone in the community to step up.
But I didn’t find one. So I did it myself.
Some background: I maintain Pigsty — a batteries-included PostgreSQL distribution with 460+ extensions, cross-built for 14 Linux distros. I also maintain build pipelines for 290 PG extensions, several PG forks, and dozens of Go Projects (Victoria, Prometheus, etc.) packaging across all major platforms. Adding one more to the pipeline was a piece of cake.
I’m not new to MinIO either. Back in 2018, we ran an internal MinIO fork at TanTan (back when it was still Apache 2.0), managing ~25 PB of data — one of the earliest and largest MinIO deployments in China at the time.
More importantly, MinIO is an optional module in Pigsty. Many users run it as the default backup repository for PostgreSQL in production. We did consider several alternatives, but none were a drop-in replacement for MinIO-based workflows.
We use MinIO ourselves, so keeping the supply chain alive was not optional — it had to be done. As early as December 2025, when MinIO announced maintenance mode, I had already built CVE-patched binaries and switched to them.
What We’ve Done
As of today, three things.
1. Restored the Admin Console
This was the change that frustrated the community the most.
In May 2025, MinIO stripped the full admin console from the community edition, leaving behind a bare-bones object browser. User management, bucket policies, access control, lifecycle management — all gone overnight. Want them back? Pay for the enterprise edition. (~$100,000)
We brought it back.

The ironic part: this didn’t even require reverse engineering.
You just revert the minio/console submodule to the previous version.
They swapped a dependency version to replace the full console with a stripped-down one. The code was always there.
We put it back.
2. Rebuilt Binary Distribution
In October 2025, MinIO stopped distributing pre-built binaries and Docker images,
leaving only source code. “Use go install to build it yourself” — that was their answer.
For the vast majority of users, the value of open-source software isn’t just a copy of the source — supply chain stability is what matters. You need a stable artifact you can put in a Dockerfile, an Ansible playbook, or a CI/CD pipeline — not a requirement to install a Go compiler before every deployment.
We rebuilt the distribution:
- Docker Images
pgsty/silois live on Docker Hub.docker pull pgsty/siloand you’re good.- RPM / DEB Packages
- Built for major Linux distributions, matching the original package specs.
- CI/CD Pipeline
- Fully automated build workflows on GitHub, ensuring ongoing supply chain stability.
If you’re using Docker, just swap minio/minio for pgsty/silo.
For native Linux installs, grab RPM/DEB packages from the GitHub Release page. You can also use pig (the PG extension package manager) for easy installation, or configure the pigsty-infra APT/DNF repo to install from it:
Just works as usual.
3. Restored Community Edition Docs
MinIO’s official documentation was also at risk — links had started redirecting to their commercial product, AIStor.
We forked minio/docs, fixed broken links, restored removed console documentation, and deployed it as the SILO documentation site.
The docs use the same CC Attribution 4.0 license as the original, with necessary maintenance.

Commitments
Some things worth stating up front to set expectations.
No New Features — Just Supply Chain Continuity
MinIO as an S3-compatible object store is already feature-complete. It’s a finished software. It doesn’t need more bells and whistles — it needs a stable, reliable, continuously available build. (I already have PostgreSQL for these, so I don’t need something like S3 table or S3 vector. A stable S3 core is all I need)
What we’re doing: making sure you can get a working, complete MinIO binary, with the admin console included and CVE fixed. RPM, DEB, Docker images — built automatically via CI/CD, drop-in compatible with your existing minio. We keep the existing minio naming and behavior where legally and technically feasible.
This Is a Production Build, Not an Archive
We run these builds ourselves and have been dogfooding them in production for three months. If something breaks, we detect it early and patch it quickly.
I build this primarily for Pigsty and our own usage, but I hope it helps others too.
I’m willing to Track CVEs and Fix Bugs
If you run into issues, feel free to report them at pgsty/minio.
I’ll do my best to fix these — but please don’t treat this as a commercial SLA.
Given that AI coding tools have made bug fixing dramatically cheaper, and that we’re explicitly not adding any new features, I believe the maintenance workload is manageable. (how often do you see one?)
Trademark Is Tricky, But We’ll Cross That Bridge When We Come to It
Disclaimer
Trademark Notice: MinIO® is a registered trademark of MinIO, Inc. This project (pgsty/minio) is an independently maintained community fork under the AGPL license. It has no affiliation with, endorsement by, or connection to MinIO, Inc. Use of “MinIO” in this post refers solely to the open-source software project itself and implies no commercial association.
AGPLv3 gives us clear rights to fork and distribute, but trademark law is a separate domain. We’ve marked this clearly everywhere as an independent community-maintained build.
If MinIO Inc. raises trademark concerns, we’ll cooperate and rename (probably something like silo or stow).
Until then, we think descriptive use of the original name in an AGPL fork is reasonable — and renaming all the minio references doesn’t serve users.
AI Changed the Game
You might ask: can one person really maintain this?
It’s 2026. Things are different now.
AI coding tools are changing the economics of open-source maintenance.
With tools like Claude Code & Codex, the cost of locating and fixing bugs in a complex Go project has dropped by more than an order of magnitude. What used to require a dedicated team to maintain a complex infra project can now be handled by one experienced engineer with an AI copilot.
Maintaining a MinIO build without adding new features is a manageable task. The key requirement is testing and validation. and we already have that scenario, which lets us verify compatibility, reliability, and security in practice.
Consider: Elon cut X/Twitter’s engineering team down to ~30 people and the system still runs. Maintaining a MinIO fork without new features is considerably less daunting
Just Fork It
MinIO Inc. can archive a GitHub repo, but they can’t archive the demand behind 60k stars, or the dependency graph behind a billion Docker pulls. That demand doesn’t disappear — it just finds its way out.
HashiCorp’s Terraform got forked into OpenTofu, and it’s doing fine. MinIO’s situation is actually more favorable —
AGPL is more permissive for forks than BSL, with no legal gray area for community forks.
A company can abandon a project, but open-source licenses are specifically designed so the code can’t die.
Fork is the most powerful spell in open source. When a company decides to shut the door, the community only needs two words:
Fork it.
Reference
- MinIO Is Dead
- MinIO Is Dead, Are There Alternatives?
- From AGPL to Apache: Reflections on Pigsty’s License Change
- MinIO: Promise made, Promise kept
1.4 - MinIO Is Dead. Which Next?
MinIO announced maintenance mode two days ago. I ranted in “MinIO Is Dead” and immediately got flooded with “so what now?”
The usual suspects: Ceph, RustFS, SeaweedFS, Garage. I packaged all of them for Linux (RPM/DEB) and ran them through the grinder.
Short version: there’s no perfect substitute. Ceph is powerful but overkill; SeaweedFS rocks tiny files but needs an external metadata DB; Garage is cute but too barebones; RustFS targets the MinIO niche but is still alpha.
Quick scan of the field
MinIO is the open-source S3 clone. If all you need is basic object CRUD, any S3-compatible store works. But parity with MinIO means more than APIs—it’s about reliability, operability, tooling, documentation, SOPs. Replacing it cleanly is hard.
Ignoring commercial clouds, here’s the OSS menu:
- Ceph – arguably the best choice for enterprises, but brutally complex. Most folks don’t need block + file + object in one, and it requires extras like Podmon. MinIO’s single binary spoiled us.
- SeaweedFS – optimized for oceans of small files; O(1) disk seeks make it absurdly fast there. But it relies on an external metadata store. If you want a general-purpose object store, that dependency is annoying.
- Garage – built by Deuxfleurs with NGI funding. Delightfully light (10 MB), great for self-hosters and edge nodes. But S3 compatibility is thin: no versioning, no cross-region replication, no IAM. Enterprises will laugh.
- RustFS – the only project explicitly chasing “drop-in MinIO,” but it’s still alpha.
RustFS vs. MinIO
RustFS looked the most promising, so I wired it into Pigsty as a MinIO replacement. Most logic carried over, but a few differences popped up:
- Certificates must follow specific naming rules.
- Health checks differ from MinIO’s endpoints.
mc admindoesn’t work; you can’t push fine-grained IAM policies. That’s a deal-breaker for many teams.
It ran, but I’m not shipping alpha software into production, so I shelved the branch. I’ll revisit when RustFS hits GA.
Will RustFS repeat MinIO’s mistakes?
RustFS has potential, but I worry it’ll retrace MinIO’s path. I asked the AI big three (GPT‑5 Pro, Claude 4 Opus, Gemini 3 Pro) to audit the project. Gemini leveled some serious accusations; Claude corroborated.
The red flags match MinIO’s history: Apache 2.0 license + copyright assignment CLA + single commercial gatekeeper. With that risk profile, I’m downgrading RustFS from “optimistic” to “cautious wait-and-see.”
So what now?
Pigsty bundles MinIO as an optional module for PostgreSQL backups or as an on-prem S3 for apps like Supabase. After surveying the alternatives, I’m not eager to swap it out. I might add a pgBackRest-native backup server option, but ripping out MinIO today feels premature.
Best plan: stay on the latest MinIO release, lock the version, isolate it on the network, and wait a few months. Maybe the community forks it; maybe RustFS matures. Adjust when reality changes.
RustFS still has a golden window to seize MinIO’s niche with a safer, community-friendly fork. That window is measured in months, not years.
If you stick with MinIO
Use the latest build, not the April 22, 2025 edition with the GUI. There’s a serious CVE in the interim:
- CVE-2025-62506 – privilege escalation via session-policy bypass (HIGH). Low-privilege users can mint new accounts and escalate.
In a locked-down intranet the risk is manageable, but you still want the fix, which landed in the 2025‑10‑15 release. MinIO pulled the prebuilt binaries starting with that version, offering source only. Annoying, but it’s Go—go build and you’re done. I forked MinIO, ran their packager, and produced RPM/DEBs for 2025‑12‑03 so I’m not deploying vulnerable bits: https://github.com/pgsty/minio

Security patches still need humans. MinIO claims they’ll fix critical issues, but if the community wants a maintained fork, now’s the moment. Start from 2025‑04‑22, cherry-pick critical bug/security fixes, and keep a community LTS alive.
MinIO is “done” software. It doesn’t need the latest S3 gimmick (Vector/Table); it needs steady bugfixes. That’s perfect for a community branch. Plenty of storage vendors rely on MinIO; maintaining a fork beats writing a new object store from scratch.
2026-02-14 Update: MinIO’s official repo has been fully archived and is no longer maintained. Besides, I’ve personally maintained an oss fork of minio:
pgsty/minio/ Docs: https://silo.pgsty.com. Which based on the last upstream version 2025-12-03 with restored console capabilities.
1.5 - MinIO is Dead
December 3, 2025 was a day to mark in open-source software history. MinIO’s team updated the project status on GitHub, announcing the MinIO open-source project was entering “maintenance mode.” This basically declared the death of MinIO as an open-source project.
MinIO the company has finally completed its transformation from a dragon-slaying hero into the very dragon it once sought to slay.

From Dragon-Slayer to Dragon
Democratization Era (2014–2019): The Apache of Object Storage
MinIO was founded in 2014 with a highly idealistic vision – to be “the Apache of object storage.” In an era dominated by AWS S3, MinIO’s ultra-lightweight design (a single static binary) and 100% S3 API compatibility quickly won developers’ hearts.
During this phase, MinIO was licensed under the liberal Apache 2.0 license, encouraging developers to integrate it into all kinds of applications. Its core pitch: “turn any hardware into AWS S3.” This open strategy was wildly successful. MinIO claimed its Docker image had been pulled over 1 billion times, making it the world’s most widely deployed object storage service. At this point, MinIO was a darling of the cloud-native stack – the default storage backend in many Kubernetes setups.
License Weaponization (2019–2025): The AGPL War
The first major crack in community relations appeared around 2019–2021. MinIO announced it was changing its core license from Apache 2.0 to GNU AGPLv3.
The official explanation was that this move aimed to prevent cloud providers (like AWS, Azure) from “freeloading” the code and repackaging it as proprietary services — a common defensive tactic in open source. During this period, MinIO shifted from being a community guardian to an aggressive defender of its IP. In 2022, MinIO publicly accused Nutanix Objects of violating its license and revoked Nutanix’s right to use MinIO; in 2023, MinIO sued high-performance filesystem vendor Weka on similar grounds. These legal actions, though legally contentious, sent a clear signal: MinIO no longer welcomed commercial use without paying up. This set the legal and psychological stage for the full lockdown that would come in 2025.
Control Plane Neutered (May 2025)
In May 2025, MinIO decided to strip the MinIO Console out of the community edition. The console was a critical GUI for bucket management, IAM, monitoring, and audit logging. After this removal, the open-source MinIO was left with only a basic “object browser” GUI – essentially just a file viewer/downloader.
Meanwhile, key admin features like policy management, site replication configuration, and lifecycle management were moved entirely into the commercial enterprise edition. This change downgraded the open-source MinIO from a full-featured storage management system into a mere data-plane component, robbing it of the control-plane capabilities needed to run as a standalone product in production.
Cutting Off Binary Distribution (Oct 2025)
On October 15, 2025 – right as a critical security vulnerability (CVE-2025-10-15T17-29-55Z / GHSA-jjjj-jwhf-8rgr) was disclosed – MinIO stopped publishing updated Docker images to Docker Hub and Quay.io. The timing of this move was highly strategic. By cutting off binaries during a major security incident, MinIO effectively used security as a bargaining chip.
This decision directly broke the automated deployment pipelines for countless users. Helm charts, Ansible playbooks, and Terraform scripts expecting minio/minio (or Bitnami’s minio) image suddenly failed to find updates.
Auto-scaling groups trying to pull new nodes hung due to missing images. For teams without a Go build environment or an internal container registry, MinIO instantly became unusable.
Maintenance Mode (Dec 2025)
On December 3, 2025, MinIO, Inc. officially updated its channels and GitHub repo to announce that the open-source project is now in “maintenance mode.” The README stated that there will be no further feature additions or improvements, issues and PRs will no longer be reviewed, and even critical security fixes would be provided “as appropriate.” No more RPM/DEB packages or Docker images will be released. Essentially, anyone needing updates or support is advised to switch to the commercial AIStor product.

Technical Impact: Damage to the Open-Source Ecosystem
MinIO’s move to maintenance mode dealt an immediate and far-reaching blow to many tech stacks.
Broken CI/CD Pipelines and an Automation Crisis
Thousands of Helm charts, Ansible playbooks, and Terraform scripts depend on the minio/minio (or Bitnami’s minio) container image.
With official images no longer published, third-party packagers like Bitnami — who can’t get a stable upstream release — also had to stop updates.
- Cascade effect: Deployments in fresh environments started failing outright. Auto-scaling groups, upon launching new instances, would hang or error out when the MinIO image couldn’t be pulled.
- Cost of fixes: Companies now have to rewrite their deployment scripts to point to a self-hosted image, and set up internal build pipelines to compile and package MinIO from source.
Security Vacuum: CVE Patches Go Private
The most lethal consequence of halting binary distribution is delayed security patches. In the October 2025 incident, for example, MinIO effectively withheld the patched binaries for the vulnerability.
- Risk exposure: Companies without dedicated security teams are forced to keep running older, vulnerable versions with known critical flaws.
- Compliance nightmare: For organizations under PCI-DSS, HIPAA, SOC2, etc., not being able to obtain vendor-signed security updates is a compliance disaster. Lacking official patches, they technically fall out of compliance.
Exponentially Higher Ops Complexity
Removing the UI wasn’t just a hit to user experience – it increased operational burden.
Tasks that used to be a few clicks in the Console (configuring bucket policies, setting user permissions) now require ops engineers to master the mc CLI or hand-craft complex JSON policy docs.
This raises the skill floor and makes MinIO far less friendly as a lightweight internal tool.
Underlying Reasons: Pressure from Capital and Commercialization
The driving force behind MinIO’s decisions is the logic of venture capital. By 2025, MinIO had raised a total of $126 million in funding. The most significant was a $103 million Series B in January 2022 led by Intel Capital, SoftBank Vision Fund II, and General Catalyst, which crowned MinIO a unicorn (valued over $1 billion).
In VC terms, a $1B valuation means the company must show a clear path to IPO — typically demanding $100M+ in Annual Recurring Revenue (ARR) and rapid growth. In Feb 2025, MinIO announced its ARR had grown 149% over the past two years businesswire.com. Impressive growth, but to live up to a sky-high valuation, organic conversion alone wasn’t enough.
Cutting off the free open-source offering is the most direct way to force a huge user base into paid customers.
In 2025, MinIO underwent a full rebrand and launched “MinIO AIStor,” styling itself as “the data backbone for enterprise AI.” Management recognized that general-purpose object storage (for backups, file servers, etc.) was a red-ocean market with thin margins, whereas generative AI’s appetite for high-throughput data (the exascale AI era) promised the next big surge. By tuning its product for AI workloads and focusing on Fortune 500 enterprises linkedin.com, MinIO essentially decided to cut loose its low-value open-source user base. The move to maintenance mode signaled MinIO’s official pivot from a broad open-source project into a vertical, high-end AI software vendor.
MinIO isn’t a garage hobby project by a few geeks anymore; it’s a company that took $126M in VC and is valued at over $1B. Backed by Intel Capital and SoftBank, once you take that money, your boss is no longer the users — it’s the investors. And what do investors want? ARR, growth, IPO. You tell them, “We have a billion Docker pulls!” and they’ll ask, “How many dimes did those pulls pay us?”
The reality is brutal. To the VCs, those small businesses and individual devs using free MinIO are low-value assets. They open issues and ask for support — consuming expensive engineer time, bandwidth, and servers — yet will never convert to paying customers. MinIO’s leadership knows their real cash cows are the Fortune 500 firms doing generative AI. The ones training GPT models or running self-driving pipelines need AIStor, ultra-high performance, and 24/7 enterprise SLAs.
So flipping the project into “maintenance mode” is essentially an asset carve-out. MinIO is cutting away the “dead weight” (free users) and concentrating on the milkable “cash cows” (enterprise AI clients). In business strategy this is called focus. To the investors, it’s being responsible. But from the perspective of open source, it’s simply betrayal.
Personal Reflections
I started using MinIO around 2018 (back when it was Apache-licensed). We built a few multi-petabyte object storage clusters for videos, images, backups — probably one of the largest MinIO deployments in China at the time. I wrote deployment/monitoring playbooks for MinIO (still open-sourced in Pigsty).
As an open-source startup founder, I can understand the motivation behind these moves. But as an open-source contributor and user — I also know many folks right now have one phrase in their minds: “I have never seen such shamelessness.”
An open-source license isn’t a shackle, but it is a social contract. Developers contribute code, users contribute testing, feedback, and reputation; together, they make a project successful. MinIO enjoyed a decade of community goodwill and parlayed the bragging rights of “#1 in global downloads” into venture funding. Then it turned around and told the very users who propped it up: “You free-riders, get lost.” This kind of move breaks the fundamental trust that open source is built on.
This “bait and switch” tactic is even more nauseating than a crypto rug pull. A rug pull only takes your money — MinIO is pulling the rug out from under the tech stacks of thousands of companies. Adopting a technology isn’t just picking up a binary; it’s buying into an ecosystem and a design philosophy. They got everyone onboard, let the switching costs pile up sky-high, and then suddenly kicked away the ladder. In fact, as open-source expert Tison thoroughly discussed in his article The Bait-and-Switch Open-Source Strategy, the core issue with this model is deception.
MinIO betrayed the community, so the community may abandon it as well. Alternatives like Garage, SeaweedFS, or the new RustFS are ready to step in.
If I have to sum up my feelings, I’d borrow a line from The Hitchhiker’s Guide to the Galaxy:
—— “So long, and thanks for all the fish.”
2026-02-14 Update: MinIO’s official repo has been fully archived and is no longer maintained. Besides, I’ve personally maintained an oss fork of minio:
pgsty/minio/ Docs: https://silo.pgsty.com. Which based on the last upstream version 2025-12-03 with restored console capabilities.
2 - Release Notes
Release notes track feature updates, security fixes, dependency upgrades, compatibility requirements, and artifact delivery across the SILO stack.
The 20261026 coordinated release consolidates all SILO Server, mcli, and KES changes since 20261018 alongside Console 2.5.1 and silo-pkg 3.15.2 (the October 24–25 maintenance drafts have been folded into the final release notes, with their URLs redirecting automatically).
2.1 - SILO 20261026: Server, mcli and KES
SILO, mcli, and PGSTY KES 20261026 are now released, alongside SILO Console 2.5.1 and silo-pkg 3.15.2. This article consolidates the October 24–25 maintenance records and summarizes all changes since the 20261018 release. All three executables share the same release date and package version; KES remains an optional, independently deployed key encryption service.
Released components
| Component | Version | Source | Delivery |
|---|---|---|---|
| SILO Server | 20261026000000.0.0 |
c3ad836aed0d |
Platform archives, Linux packages, Docker Hub |
| mcli | 20261026000000.0.0 |
0ceb298cdcd0 |
Platform archives, Linux packages, Docker Hub |
| PGSTY KES | 20261026000000.0.0 |
e9b4f1acebc1 |
Platform archives, Linux packages, Docker Hub |
| SILO Console | 2.5.1 |
fff1618d9abb |
Embedded in Server; standalone binaries, packages, and image |
| silo-pkg | 3.15.2 |
0416eb1f4505 |
Go source module |
All five projects build with Go 1.27.3. Server, Console, and mcli select silo-pkg v3.15.2 and the pinned upstream commit minio-go v7.4.2-0.20261025165926-320af050c445 (a pseudo-version rather than an upstream v7.4.2 tag). Server explicitly embeds the released Console and mcli sources; upgrading standalone Console does not replace the UI embedded inside an older Server binary. See the component matrix and downloads.
Server changes since 20261018
- LDAP and IAM (#250): Restores group-based login for LDAP users whose normalized DNs exceed 255 bytes in a single ASCII path component by treating missing optional IAM records on those paths as absent, while continuing to fail closed on unreadable revocation records or storage errors (writing overlong path records remains unsupported on the object backend). Policy reloads no longer hold the authentication-cache lock during storage I/O, swap the cache only after a complete load, and preserve IAM write ordering. Request cancellation now propagates to KMS and SSE-S3 decryption paths without shortening the caller’s policy-query deadline.
- Built-in policy Reset and live bucket inventory: Resetting an overridden built-in policy immediately restores the native policy definition on the next read, while delayed peer/etcd notifications reload committed state without clobbering newer edits (in-use checks, delete revisions, and custom-policy deletion remain unchanged). Console’s
account-infoendpoint now reads the live bucket inventory so newly created or deleted buckets appear immediately on refresh, while object usage totals remain scanner-derived. - Multi-pool deletion safety (#257, #256): Ordinary and bulk deletes clean non-authoritative duplicate pools before removing the authoritative copy, select an available pool when writing new delete markers, and protect remote-tier payloads still referenced by surviving copies.
- Versioned lifecycle expiration (#253, #252): Retains remote tier payloads whenever lifecycle expiration only inserts a delete marker (including non-
nullversions in versioning-suspended buckets), reclaiming remote tier objects only when a historical version is permanently purged and no surviving reference needs its payload. - Multipart replication (#255, #254): Forwards the
FULL_OBJECTchecksum type when replicating CRC multipart objects so the target cluster does not reject the replica withBadDigest. - Storage RPC and timeout semantics: Forwards storage RPC deletion options (including the previous data directory needed for overwrite rollback) while preserving existing drive/RPC deadlines, request cancellation, object locking, read quorum, and healing rules. The final release withdrew the 20261025 draft’s admitted-I/O draining waits and per-object mutation guards (including #258) in
dcc2f20ba; see the multi-pool object consistency record. - Dependencies, packaging, and delivery: Updates SFTP, etcd, NATS, cloud SDKs, and other Go modules; preserves Kafka’s default protocol
2.1, CLI formatting, and NetBSD compatibility; and fixes legacy DARE v1 unsealing while keeping v2 writes. Container images, the verified installer, and Helm client jobs bundle mcli 20261026. Windows archives now ship as.zip, staticcurlis updated to8.23.0with the certificate-trust mitigation described below, and RPM signer verification is fixed during the final SBOM/provenance refresh. - Documentation and migration: Incorporates the single-node MinIO migration guide (#228), reconciles operational defaults, and consolidates upgrade instructions under the migration guide.
This release retains all Object Lock, IAM authorization, durable STS revocation, pool-migration, and LIST recovery repairs shipped in 20261018. The dependency security inventory below separates newly adopted fixes from earlier repairs and scanner findings in unused code paths.
mcli changes since 20261018
- Client identity cache isolation (#50, #55): Isolates cached S3 and admin clients by credentials, TLS trust, proxy, custom resolver, and request headers while preserving credential refresh and token-file rotation.
- Replication resync status and throughput (#48, #49, #52):
admin replicate resync status --jsondrains the status stream through terminalCompleted,Failed, orCanceledstates so completed jobs emit their final JSON record; explicitly reports when replication is disabled; stops waiting onFailedorCanceledin TTY mode; and fixes integer overflow in throughput calculations past ~8.6 GiB. - Honest exit codes (#51):
resync startandresync cancelretain per-bucket error details and return a nonzero exit code when any bucket fails, whileresync statuscontinues to exit0when successfully inspecting an alreadyFailedorCanceledjob.admin info --jsonalso exits nonzero on request failure while preserving its JSON error output. - Batch job observation (#53, #54):
batch statuscontinues watching when live metrics are not yet available, emits apendingstatus instead of treating queued or missing samples as complete, falls back to bounded persisted-status queries when the metric stream ends early, and fixes IOPs integer overflow on large object counts. - Alias regions (#44): Honors persisted alias
regionsettings with precedenceMC_REGION→AWS_REGION→ alias configuration → bucket-location discovery, and isolates signing regions across cached clients (without adding analias set --regionCLI flag). - SDK fixes: Rejects truncated sequential or parallel multipart streams; preserves
ReadAtEOF/negative-offset semantics and post-Statread/seek bounds; deduplicates fallback batch-delete errors; and accepts both HTTP200and204when removing bucket replication configurations (while still reporting403and other real errors). - Packaging and help (#45): Builds with Go 1.27.3 and GoReleaser 2.19.0, switches Windows archives to
.zip, refreshes help URLs and disabled server-update/SUBNET messages, and preserves existing CLI configuration andMC_*environment variables.
Automation scripts should handle the pending batch state and nonzero exit codes on partially failed resync start/cancel commands. Upgrading mcli requires no Server protocol change. Mid-stream download interruptions still fail immediately; SIGINT during a multipart upload exits 130, and incomplete upload parts may require explicit cleanup. See the complete client commit history.
KES changes since 20261018
KES 20261026 rolls up the Go 1.27.3 toolchain and cloud-SDK dependency maintenance and aligns the executable, Linux packages, and container images with the 20261026 release batch. Its HTTP API, configuration schema, and stored-key format remain unchanged.
| Dependency | 20261018 | 20261026 |
|---|---|---|
| Go | 1.27.2 | 1.27.3 |
| AWS SDK v2 core | 1.47.3 | 1.47.4 |
| AWS config / credentials | 1.33.9 / 1.20.9 | 1.33.10 / 1.20.10 |
| AWS Secrets Manager | 1.50.4 | 1.50.6 |
| Google Secret Manager | 1.22.0 | 1.23.0 |
| Google API / IAM | 0.304.0 / 1.14.0 | 0.308.0 / 1.15.0 |
| Google certificate proxy / GAX | 0.3.23 / 2.26.2 | 0.3.24 / 2.27.0 |
| go-rootcerts / procfs | 1.0.2 / 0.22.0 | 1.1.0 / 0.23.0 |
The 25 module updates also cover AWS support packages and Google generated APIs, while keeping Lip Gloss v1 and gopkg.in/yaml.v3. Packaging uses GoReleaser 2.19.0 and nFPM 2.48.0; Linux packages install /usr/local/bin/kes without auto-starting an unconfigured service.
Compatibility checks verified OpenBao KV v2 and local Moto Secrets Manager fixtures across pre-upgrade and post-upgrade encrypted objects, version rollbacks, and cold Server restarts. Preserve persistent keys, certificates, and configuration across upgrades, and verify object decryption after restarting Server. See the complete KES commit history.
Console 2.5 and silo-pkg 3.15
Console 2.5.1 delivers the 2.5 interface refresh, permission-aware navigation, Overview dashboard, bounded object previews, Tasks center, identity management, and high-latency diagnostic stream fixes; the 2.5.1 patch additionally preserves group members during status changes, OIDC string-array group permissions, secure session cookies behind TLS proxies, exact UTF-8 credential exports, streaming ZIP error statuses, and keyboard focus/accessibility labels. Server 20261026 embeds Console 2.5.1 directly.
silo-pkg 3.15.2 combines the 3.15.0 Go 1.27.3 and dependency refresh, 3.15.1 upstream SDK fixes, and 3.15.2 HTTP 204 handling when removing bucket replication, while retaining existing policy semantics and the 3.14.3 admin-action lookup optimization. As a Go source module, silo-pkg is adopted by rebuilding dependent applications.
Upgrade and failure boundaries
- Coordinated upgrade: Back up IAM state, configuration, and encryption keys, and upgrade all Server nodes and replication peers together following the IAM upgrade guide. Running mixed old/new Server versions against a shared IAM backend or performing rolling downgrades remains unsupported.
- Post-upgrade checks: Verify LDAP/OIDC login, built-in policy Reset, bucket list refresh, versioned reads, cross-site replication, and encrypted-object decryption. The
FULL_OBJECTmultipart replication fix requires upgrading the sending Server first and then checking its pending replication queue. - Archive formats and scripts: Windows archives now use
.zip(Linux and macOS retain.tar.gz); update automation scripts formcli batch statuspendingoutputs and nonzero exit codes on partially failedresync start/cancelcommands. Existing environment variables, configuration filenames, and on-disk formats remain unchanged. - Known multipart and issue scope: Public
ListMultipartUploadscontinues to use the node-local in-memory cache and exact-key lookup, ignoring the retiredstrictswitch with a startup warning. Issues #79 and #217 remain open, webhook hot-reload PR #249 is not included, and broader performance symptoms remain tracked in #251.
Storage timeout and consistency boundary: A client-visible request timeout does not guarantee that underlying disk or RPC I/O has stopped or that the write failed to commit. Late commits, delayed rollbacks, and stale values or tags remain possible after partial drive failures, lock lease expiration, or network partitions. This release does not add distributed lock fencing and cannot automatically recover previously lost objects, remote tier payloads, or metadata.
Downloads and verification
See the download matrix for platform archive filenames, SHA-256 checksums, Linux packages, and verification commands. Versioned container images are available on Docker Hub:
- Server standard image (Linux
amd64/arm64manifest):sha256:7bb2aff6b89efe1cb1303c10fc21d579447434ed06649ac9ec05a5e2dbf30ad7(matcheslatest) - Server distroless image (Linux
amd64/arm64manifest):sha256:e6544a902f426f2b4e775b89f3c1cd78310dbffc262d1ed82e9800c1cb1b5885(matchesdistroless, with nativesilo healthcheck readyconfigured on both architectures; see the Server image workflow)
Each component’s artifacts are identified by its release commit, archive checksums, package signatures, and build provenance:
- Server artifact verification: Verified 36 release assets and 34 attested subjects (including six platform binaries and six Linux packages) built from
c3ad836aed0d. Both RPM packages are GPG-signed, and the checksum manifests, SBOMs, and provenance attestations match the final packages (see the Go CI, VulnCheck, Release, and Finalize workflows). - Local multi-node and fault-injection verification: Passed 88 baseline four-node, 88 upgraded four-node, and 136 eight-node multi-pool S3 checks on Linux
arm64, preserving historical objects, tags, encrypted objects, and in-flight multipart uploads across the coordinated upgrade. Rolling restarts,SIGKILL, and a 75-second pause/recovery test passed with all 408 acknowledged writes read back with matching hashes and complete listings; the native trusted-TLSmclisuite and 16 persistence checks also passed. Because health endpoints evaluate reported drive quorum rather than per-node installed peer connections, run a data-path check before resuming traffic after a restart (see Startup Readiness Window). - KES artifact and integration verification: Verified all 19 public assets, 11 attested subjects, and both signed RPMs. OpenBao KV v2 and Moto fixtures exercised the Darwin
arm64KES binary across upgrade, rollback, and cold Server restarts—first against SILO 20261018 and then against the final Server/mcli 20261026 Linuxarm64binaries. Both KES Linux container images match the archive binaries, with multi-architecture manifest digestsha256:ea36fcdd260c606eb89e693b0949aea73b149359605273a98fcf13c8433049cd(Release workflow · Image workflow).
Contributors and change history
Thanks to Denis Babchuk / @botdeniska for lifecycle, replication, and multi-pool deletion repairs; @910patin for long-DN LDAP login; Azat / @azataiot for reproducing high-latency health-report truncation; @MikeVL for reporting resync throughput overflow and JSON status output gaps; @mrjavadseydi for alias-region support and the webhook reload proposal; Gerald Combs / @geraldcombs for Windows archive feedback; @yupoet for the single-node migration guide; and @GhaziTriki, @devTeam-AiReg, and @sargarass for their bug reports and feedback.
The table below summarizes the community Issues and PRs merged, adopted, or tracked across the five component repositories during this release cycle:
| Project | Contributor | Issue / PR record | Summary |
|---|---|---|---|
| SILO | @botdeniska | #252, #253; #254, #255; #256, #257; #245, #248 | Fixed tiered-object lifecycle expiry, multipart FULL_OBJECT replication checksums, and multi-pool deletion safety; the cross-pool migration repair in #248 was adopted in 60c7179dd with co-authorship retained. |
| SILO | @910patin | #250 | Fixed long-DN LDAP login failures. |
| SILO | @yupoet | #228 | Contributed the field-validated single-node migration guide. |
| SILO | @mrjavadseydi | #249 | Webhook hot-reload proposal (under review, not included in this release). |
| SILO | @sargarass | #79 | Multipart upload listing feedback (tracked for a future release). |
| SILO | @GhaziTriki | #251 | Performance report (under investigation). |
| SILO | @devTeam-AiReg | #247 | Release availability feedback. |
| SILO | @Vonng | #217, #258 | Multi-site replication boundary notes (#217) and candidate lock-isolation experiment (#258, reverted before final release). |
| mcli | @mrjavadseydi | #44 | Added persistent custom S3 region support in alias configurations. |
| mcli | @MikeVL | #48, #49 | Reported replication resync throughput overflow and missing --json status output (fixed in this release). |
| mcli | @geraldcombs | #45 | Proposed .zip packaging for Windows release archives (adopted). |
| mcli | @Vonng | #50–#55 | Client identity-switch isolation and replication resync status repairs. |
| Console | @azataiot | #63 | Reported and reproduced high-latency health-report truncation (fixed in Console 2.5.1). |
| Console | @Vonng | #64–#69 | Dependency updates, UI improvements, end-to-end verification, and release maintenance for Console 2.5.1. |
| silo-pkg | @botdeniska | #11 | Policy evaluation fast-path optimizations. |
| KES | @Vonng | 20261018–20261026 release maintenance | Dependency refresh, packaging, and OpenBao / Moto compatibility verification. |
The community contributor roll recognizes every human issue and PR author across all repositories. Full commit comparisons: Server, mcli, KES, Console, and pkg.
The sections below summarize the October 22–25 dependency and security audit records (dependencies shared across multiple components are listed once under the corresponding package):
Additional Go dependency updates on October 24
The Go toolchain remains 1.27.3. Compared with the October 22 candidate dependencies, this update includes 26 newly released third-party module versions, plus Arrow and codescan updates (28 modules in total):
| Dependency | Version change | Components |
|---|---|---|
| pkg/sftp | 1.13.11 → 1.13.12 | SILO |
| etcd api/v3, client/pkg/v3, client/v3 | 3.7.2 → 3.7.3 | silo-pkg, mcli, Console, SILO |
| JWX | 3.3.0 → 3.4.0 | silo-pkg, mcli, Console, SILO |
| nats.go | 1.54.0 → 1.55.0 | SILO |
| procfs | 0.22.0 → 0.23.0 | mcli, Console, SILO, KES |
| AWS SDK v2, 14 modules | core 1.47.3 → 1.47.4; config 1.33.9 → 1.33.10; credentials 1.20.9 → 1.20.10; Secrets Manager 1.50.5 → 1.50.6, with matching supporting modules | KES |
| GCP IAM / Monitoring / Secret Manager / Google API | IAM 1.14.0 → 1.15.0; Monitoring 1.31.0 → 1.32.0; Secret Manager 1.22.0 → 1.23.0; Google API 0.307.0 → 0.308.0 | IAM and Google API: SILO, KES; Monitoring: SILO; Secret Manager: KES |
| go-rootcerts | 1.0.2 → 1.1.0 | KES |
| Arrow | 18.7.0 → 18.8.0 | SILO, through the Azure SDK |
| codescan | 0.36.4 → 0.36.6 | Console generator toolchain |
Public API behavior and output formats remain unchanged; see the component table above for the final pinned module versions.
Included runtime security fixes
Go and certificate trust
CVE-2026-111943 / GO-2026-6809: Go 1.27.3 rejects known untrustworthy intermediate certificates in crypto/x509, protecting TLS verification even when host trust stores have not yet been updated. All five primary projects and the standalone debugging modules are built with Go 1.27.3.
In addition, the UBI container trust stores for SILO, mcli, and KES apply the root certificate blocklist from Red Hat RHSB-2026-012. Because blocking a root rejects every certificate chain anchored under that root, this container-level mitigation is broader than Go 1.27.3’s intermediate-certificate check; external services still using certificates chained to those roots must rotate their certificate chains. See Locally mitigated items below for the UBI CA package status.
SFTP dependency hardening
pkg/sftp 1.13.12 fixes three security issues disclosed on October 23. At the time of this review, none has a CVE identifier. They are recorded separately by GHSA and are not counted as new CVEs:
| Advisory | Upstream issue | Applicability to SILO |
|---|---|---|
| GHSA-xgcg-wgmj-5rw3, Medium | Client.RemoveAll follows a top-level symlink and can delete contents of the target directory |
The SILO server does not call this client API |
| GHSA-47hq-m6q9-3rm6, Medium | ReadOnly mode on sftp.Server permits hardlink creation |
SILO uses NewRequestServer and its own driver, not this read-only Server API |
| GHSA-h959-g3h6-w76c, High | ReadOnly mode permits file truncation or creation | Same as above |
Because the latter two upstream advisories have inconsistent affected-range and patched-version metadata, this batch upgrades directly to 1.13.12 containing the code fixes and verifies them with regression tests (the Go vulnerability database snapshot remains at October 21 and has not yet ingested these October 23 advisories).
Static curl and linked libraries
The static curl binary bundled in SILO container images is upgraded from 8.22.0 to 8.23.0 (built with HTTP/2, SSH, and Brotli enabled over the OpenSSL backend). The table below lists all 21 CVEs from the official curl 8.22.0 vulnerability summary alongside their applicability to SILO’s static build:
| CVE | Issue / applicability |
|---|---|
| CVE-2026-92392 | HTTP/2 upload use-after-free |
| CVE-2026-107929, CVE-2026-94266, CVE-2026-94265, CVE-2026-86413, CVE-2026-86414, CVE-2026-86415, CVE-2026-87861 | Connection reuse and credential/authentication boundaries |
| CVE-2026-92386, CVE-2026-86412, CVE-2026-87862, CVE-2026-87863 | Cookie and HSTS state handling |
| CVE-2026-92387, CVE-2026-92388, CVE-2026-92389, CVE-2026-92390, CVE-2026-92391 | Loops, FTP/TFTP, download filenames, and MIME recursion |
| CVE-2026-86410, CVE-2026-87865 | SSH host verification and known_hosts callbacks |
| CVE-2026-107928, CVE-2026-87866 | GnuTLS / wolfSSL only; neither backend is used by SILO’s OpenSSL build |
The statically linked Alpine libraries are also updated to the following patched revisions (-rN denotes the Alpine security backport revision):
| Library | Version change | CVE |
|---|---|---|
| Brotli | 1.2.0-r1 → 1.2.1-r0 | CVE-2026-111497 |
| musl | 1.2.6-r2 → 1.2.6-r3 | CVE-2026-107891 |
| libssh2 | 1.11.1-r3 → 1.11.1-r4 | CVE-2025-15661, CVE-2026-58050, CVE-2026-58051, CVE-2026-66032, CVE-2026-66033, CVE-2026-66034, CVE-2026-66035 |
The previous SILO arm64 image already linked OpenSSL 3.5.9-r0, nghttp2 1.70.0-r0 (CVE-2026-58055), and zlib 1.3.2-r1 (CVE-2026-85091); this update enforces minimum version floors in the build script and verifies both architectures against the official Alpine security metadata.
Build and development security fixes
This section covers the Node.js, Vite, and code-generation toolchain used during SILO Console frontend compilation and testing (Console runs in production as a compiled Go server binary), as well as the standalone Go debugging modules in the server repository.
Node 24.22.1
The build environment updates Node.js from 24.22.0 to 24.22.1, addressing all eight Node 24.x CVEs in the October 2026 Node.js security release (excluding Node-26-only findings):
| CVE | Fix |
|---|---|
| CVE-2026-105482 | DNS names truncated at NUL |
| CVE-2026-105483, CVE-2026-106069, CVE-2026-106072 | TLS hostname checks for non-canonical IP addresses |
| CVE-2026-111497 | Brotli decoder out-of-bounds write (backported in Node while the embedded version string remains 1.2.0) |
| CVE-2026-105481 | HTTP keep-alive accepts data sent before a request |
| CVE-2026-106074 | Random caches reused across startup snapshots |
| CVE-2026-106073 | Permission-model bypass in fs.realpathSync |
Node 24.22.1 updates bundled Undici from 7.29.1 to 7.30.1, resolving eight CVEs documented in the Undici v7.30.1 release notes:
| CVE | Fix |
|---|---|
| CVE-2026-87018, CVE-2026-109383 | TLS renegotiation crash and lost Agent verification callbacks |
| CVE-2026-109829, CVE-2026-109841 | Cross-origin cache disclosure and SRI bypass |
| CVE-2026-109869, CVE-2026-87029, CVE-2026-109924, CVE-2026-109932 | Denial of service in content decoding, EventSource, and WebSocket handling |
Node 24.22.1 also updates bundled OpenSSL from 3.5.8 to 3.5.9, incorporating 13 fixes across DTLS, QUIC, CRLs, CMP, EC/SM2, and handshake memory safety: CVE-2026-84782, CVE-2026-35189, CVE-2026-35191, CVE-2026-42772, CVE-2026-54872, CVE-2026-54873, CVE-2026-54875, CVE-2026-72897, CVE-2026-75804, CVE-2026-75805, CVE-2026-75806, CVE-2026-77696, CVE-2026-84784.
Node 24.22.1 updates bundled c-ares from 1.34.8 to 1.34.9, resolving 11 upstream GHSAs that do not carry separate CVE IDs (GHSA-wpr3-8q3r-4q8c covers a connection-shutdown path omitted from the earlier CVE-2026-33630 fix and remains listed as No known CVE):
| Advisory | Fix |
|---|---|
| GHSA-wpr3-8q3r-4q8c | Memory corruption when cancelling after connection failure |
| GHSA-fqh5-3x7m-wp44 | Replies accepted from the wrong DNS server |
| GHSA-97vq-4mxv-7vhm | Freed query used after failed connection shutdown |
| GHSA-rf6g-ww74-97qh | Use-after-free during cancellation/shutdown callbacks |
| GHSA-mw23-pfg6-2x9r | Use-after-free after socket-state callbacks |
| GHSA-v33j-65h4-4ch8 | Use-after-free when server-state callbacks cancel queries |
| GHSA-pj7p-v3cw-hq66 | Freed server record used after a callback changes servers |
| GHSA-grv7-fq5q-9vfc | Cached answer freed while being read |
| GHSA-ph6c-3cvx-mq6v | Buffer overflow during concurrent option copying/server updates |
| GHSA-cfcv-8v2j-qj6r | Uninitialized memory in IPv6 addresses with prefix lengths |
| GHSA-wrgc-prmw-27xh | Cache reuses replies obtained with different EDNS options |
Frontend dependencies and code generation
| Dependency | CVE / GHSA | Fix |
|---|---|---|
| fast-uri 3.1.7 → 3.1.9 | CVE-2026-86472 | Inconsistent normalization of encoded hostnames |
| ip-address 10.4.0 → 10.7.4 | CVE-2026-101913 | Incorrect link-local IPv6 classification |
| ip-address 10.4.0 → 10.7.4 | CVE-2026-101910 | Unrecognized local-use NAT64 range |
| ip-address 10.4.0 → 10.7.4 | CVE-2026-101912 | Subnet checks mix address families |
| ip-address 10.4.0 → 10.7.4 | CVE-2026-101911 | Unbounded diagnostics for long addresses |
| moment 2.30.1 → 2.31.0 | CVE-2026-17495 | Path traversal through non-string locale input |
| smol-toml 1.8.0 → 1.9.1 | GHSA-r4xh-jqrq-34v2 | Quadratic TOML parsing |
| vite 8.0.16 → 8.3.5 | CVE-2026-112945 | Cross-origin script execution in development middleware |
| vite 8.0.16 → 8.3.5 | CVE-2026-112946 | Development-server filesystem allowlist bypass |
| vite 8.0.16 → 8.3.5 | CVE-2026-112943 | Insufficient Windows editor-argument escaping |
| vite 8.0.16 → 8.3.5 | CVE-2026-112944 | Editor arguments interpreted as command options |
| vite 8.0.16 → 8.3.5 | CVE-2026-112942 | Windows editor-path check bypass |
In addition, go-swagger is updated from 0.33.2 to 0.36.6, resolving GHSA-8gmg-wf5j-hvf7 and GHSA-rg22-m539-jwvf (code injection during code generation when processing untrusted OpenAPI content types or enum values; first fixed upstream in 0.35.3 without a separate CVE ID). Generated files are synchronized with the updated tool. The generation process explicitly preserves the reviewed configuration-import binder and multipart consumer, retaining the existing file-size, filename and form-encoding behavior under regression tests.
Independent Go debugging modules
The standalone inspect, s3-verify, and xattr module graphs in the server repository previously lagged behind the root module. Updating their dependencies clears 37 CVE-numbered dependency findings and one non-CVE S2 advisory:
| Dependency | Affected independent modules | Selected version |
|---|---|---|
| golang.org/x/crypto | inspect, s3-verify | 0.58.0 |
| golang.org/x/net | s3-verify | 0.61.0 |
| golang.org/x/text | s3-verify | 0.43.0 |
| golang.org/x/sys | inspect, xattr | 0.49.0 |
| klauspost/compress | inspect, s3-verify | 1.20.3 |
The table below links each finding to the Go vulnerability database and records the first upstream fixed version (the unfixed OpenPGP notice is listed separately under Build-stage and non-CVE advisories):
| CVE / advisory | Dependency | First fixed | Issue |
|---|---|---|---|
| GO-2026-5841 | klauspost/compress | v1.18.7 | OOB read in github.com/klauspost/compress/s2 |
| CVE-2026-39824 | x/sys | v0.44.0 | integer overflow in NewNTUnicodeString in golang.org/x/sys/windows |
| CVE-2025-47913 | x/crypto | v0.43.0 | Potential denial of service in golang.org/x/crypto/ssh/agent |
| CVE-2025-58181 | x/crypto | v0.45.0 | Unbounded memory consumption in golang.org/x/crypto/ssh |
| CVE-2025-47914 | x/crypto | v0.45.0 | Malformed constraint may cause denial of service in golang.org/x/crypto/ssh/agent |
| CVE-2026-39833 | x/crypto | v0.52.0 | key constraints not enforced in golang.org/x/crypto/ssh/agent |
| CVE-2026-39832 | x/crypto | v0.52.0 | agent constraints dropped when forwarding keys in golang.org/x/crypto/ssh/agent |
| CVE-2026-46597 | x/crypto | v0.52.0 | byte arithmetic causes underflow and panic in golang.org/x/crypto/ssh |
| CVE-2026-39828 | x/crypto | v0.52.0 | bypass of certificate restrictions in golang.org/x/crypto/ssh |
| CVE-2026-39835 | x/crypto | v0.52.0 | server panic during CheckHostKey/Authenticate in golang.org/x/crypto/ssh |
| CVE-2026-39827 | x/crypto | v0.52.0 | memory leak when rejecting channels can lead to DoS in golang.org/x/crypto/ssh |
| CVE-2026-39830 | x/crypto | v0.52.0 | client can cause server deadlock on unexpected responses in golang.org/x/crypto/ssh |
| CVE-2026-39829 | x/crypto | v0.52.0 | pathological RSA/DSA parameters may cause DoS in golang.org/x/crypto/ssh |
| CVE-2026-39831 | x/crypto | v0.52.0 | bypass of FIDO/U2F security keys physical interaction in golang.org/x/crypto/ssh |
| CVE-2026-39834 | x/crypto | v0.52.0 | infinite loop on large channel writes in golang.org/x/crypto/ssh |
| CVE-2026-42508 | x/crypto | v0.52.0 | auth bypass via unenforced @revoked status in golang.org/x/crypto/ssh/knownhosts |
| CVE-2026-46595 | x/crypto | v0.52.0 | VerifiedPublicKeyCallback permissions skip enforcement in golang.org/x/crypto/ssh |
| CVE-2026-46598 | x/crypto | v0.52.0 | pathological inputs can lead to client panic in golang.org/x/crypto/ssh/agent |
| CVE-2026-56854 | x/crypto | v0.55.0 | Source-address critical option not enforced for non-public-key auth callbacks in golang.org/x/crypto/ssh |
| CVE-2026-78662 | x/crypto | v0.56.0 | Prevent DoS on deadlocked undecided channel in golang.org/x/crypto/ssh |
| CVE-2026-56855 | x/crypto | v0.56.0 | Prevent DoS on deadlocked established channel in golang.org/x/crypto/ssh |
| CVE-2025-47911 | x/net | v0.45.0 | Quadratic parsing complexity in golang.org/x/net/html |
| CVE-2025-58190 | x/net | v0.45.0 | Infinite parsing loop in golang.org/x/net |
| CVE-2026-33814 | x/net | v0.53.0 | Infinite loop in HTTP/2 transport when given bad SETTINGS_MAX_FRAME_SIZE in net/http/internal/http2 in golang.org/x/net |
| CVE-2026-42506 | x/net | v0.55.0 | incorrect handling of namespaced elements in foreign content in golang.org/x/net/html |
| CVE-2026-39821 | x/net | v0.55.0 | failure to reject ASCII-only Punycode-encoded labels in golang.org/x/net/idna |
| CVE-2026-42502 | x/net | v0.55.0 | incorrect handling of HTML elements in foreign content in golang.org/x/net/html |
| CVE-2026-25680 | x/net | v0.55.0 | denial of service when parsing arbitrary HTML in golang.org/x/net/html |
| CVE-2026-25681 | x/net | v0.55.0 | incorrect handling of character references in DOCTYPE nodes in golang.org/x/net/html |
| CVE-2026-27136 | x/net | v0.55.0 | duplicate attributes can cause XSS in golang.org/x/net/html |
| CVE-2026-46600 | x/net | v0.56.0 | Parsing an invalid SVCB or HTTPS RR can panic in golang.org/x/net/dns/dnsmessage |
| CVE-2026-78659 | x/net | v0.60.0 | HTTP/2 server memory exhaustion due to Trailer headers in net/http |
| CVE-2026-78660 | x/net | v0.60.0 | HTTP/2 transport accepts malformed framing-related headers in net/http |
| CVE-2026-78669 | x/net | v0.60.0 | Excessive CPU consumption from repeated initial window changes in net/http |
| CVE-2026-78663 | x/net | v0.60.0 | Double flow control refund on HTTP/2 server streams in net/http |
| CVE-2026-97032 | x/net | v0.60.0 | HTTP/2 server crash due to HPACK encoder race in net/http |
| CVE-2026-56852 | x/text | v0.39.0 | Infinite loop on invalid input in golang.org/x/text |
| CVE-2026-56851 | x/text | v0.41.0 | Panic parsing crafted input in x/text/secure/precis in golang.org/x/text |
Outstanding upstream or distribution issues
As of 2026-10-23, the advisories below do not yet have an upstream or distribution release available. Rather than suppressing scanner rules, we record each finding alongside its local mitigation or runtime reachability.
Locally mitigated, awaiting upstream publication
| Issue | Upstream / distribution status | Action and remaining boundary |
|---|---|---|
| CVE-2026-93687 / braces | Latest npm version remains 3.0.3, with no upstream fixed release | A local Yarn patch limits parsing and AST traversal in [email protected] to depth 100 (13 regression tests pass). The package is only in the frontend build/test graph; version-only scanners continue to report 3.0.3 until an upstream release replaces the patch. |
| CVE-2026-111943 / UBI CA | At acceptance, UBI 9 supplied ca-certificates 2025.2.80_v9.0.305-91.el9 without the updated trust list |
The Red Hat root blocklist is applied during image build and verified on the output PEM bundle; Go 1.27.3 and Node 24.22.1 also enforce their own built-in certificate checks. |
UBI 9.8 findings without an applicable vendor fix
Scans conducted on 2026-10-22 against the SILO, mcli, and KES system-component acceptance images (containing the updated static curl and CA bundle on amd64 and arm64) each report 24 RPM findings across the same 12 unique CVEs (15 Medium and 9 Low per image, zero High/Critical, and no vendor FixedVersion). Red Hat security queries on 2026-10-23 continue to list the RHEL 9 source packages below as Affected or Fix deferred.
Because RPM scanners map advisories by source package (SRPM), defects in uninstalled CLI utilities or other CPU architectures are reported against shared runtime packages such as libgcc and glibc. The applicability of each entry is detailed below:
| CVE | Source package | Red Hat severity | RHEL 9 status | Applicability |
|---|---|---|---|---|
| CVE-2026-56391 | coreutils | Moderate | Fix deferred | uniq multibyte over-read requiring specific CLI options and input |
| CVE-2026-8674 | glibc | Moderate | Fix deferred | Long DNS resolver search domains can abort a process |
| CVE-2026-86805 | glibc | Moderate | Affected | Local dynamic-loader race requiring local setuid, path, and hardlink conditions |
| CVE-2026-89092 | glibc | Moderate | Affected | nscd DNS caching daemon issue; the nscd package is not installed in the images |
| CVE-2026-95818 | glibc | Moderate | Affected | Dynamic-loader stack overflow requiring specific local setuid and path conditions |
| CVE-2026-97399 | glibc | Low | Affected | Power8-specific code path; published images target amd64 and arm64 only |
| CVE-2021-46195 | gcc | Low | Fix deferred | libiberty symbol-demangler recursion (the mingw-gcc update listed in the advisory does not apply to runtime libgcc) |
| CVE-2022-27943 | gcc | Low | Fix deferred | libiberty defect; Red Hat notes that libgcc and libstdc++ do not contain the affected code |
| CVE-2026-95619 | gcc | Moderate | Affected | libstdc++ aligned-allocation overflow mapped to libgcc via the gcc SRPM |
| CVE-2026-102010 | gcc | Moderate | Affected | Optional libstdc++ pb_ds container defect mapped to libgcc via the gcc SRPM |
| CVE-2023-50495 | ncurses | Low | Fix deferred | tic terminal-compiler utility defect; not present in libncurses |
| CVE-2022-41409 | pcre2 | Low | Fix deferred | Affects pcre2-tools CLI utilities only; not shipped in UBI |
Build-stage and non-CVE advisories
- Alpine build-stage
curl/libcurlremains8.22.0-r0: As of 2026-10-23, the Alpine package indexes for amd64 and arm64 have not yet published8.23.0. This host package is used only inside the temporary build stage to fetch SHA-256-pinned source archives; the runtime image ships the newly compiled staticcurl 8.23.0. - GO-2026-5932:
golang.org/x/crypto/openpgpis deprecated upstream with no fixed version and no CVE alias. Whilegolang.org/x/cryptoremains in the module graph, none of the target builds import the affectedopenpgppackages.
Historical repairs, not newly fixed vulnerabilities
Module-path binary scanners continue to match SILO against CVE-2025-27414, CVE-2026-33322, CVE-2026-33419, CVE-2026-34204, CVE-2026-39414, CVE-2026-40344, CVE-2026-41145, and CVE-2026-42600 because SILO retains the github.com/minio/minio module path and those upstream advisories specify open-ended version ranges. All eight issues were patched and verified in earlier SILO releases; see SFTP, OIDC, LDAP, replication metadata, CSV, Snowball, signature validation, and ReadMultiple.
Compatibility and dependency regression evidence
To prevent reproduced regressions in CLI table formatting, character-width calculation, and NetBSD cross-compilation, this batch pins tablewriter 0.0.5, go-humanize 1.0.1, go-runewidth 0.0.29 in the root modules, and go-systemd/v22 22.6.0 (while xattr retains its existing width-library combination). In addition:
- Kafka notification targets explicitly initialize the protocol version default to
2.1before applying user settings; - DARE encryption retains 32-byte keys and v2 writes while fixing buffer handling when unsealing legacy v1 payloads;
pgzipretains the integrity check that returnsunexpected EOFwhen a gzip checksum trailer is missing.
The October 22–23 candidate validation (built with Go 1.27.3 on both the rollback baseline and the candidate, with persisted-data acceptance run on a single-node four-drive Linux arm64 setup) passed:
- Full compilation and applicable
-raceand lint suites; - Encrypted-object read and rollback verification across the old and new dependency stacks;
- Service restart, missing-shard healing recovery, and CLI output checks;
- End-to-end Console object uploads/downloads, ZIP archive previews, WebSocket sessions, and metrics charts.
The candidate sources with the October 24 dependency updates passed these additional checks:
- Full race suites for silo-pkg, mcli, and KES; API / pkg / cmd race suites for Console; the full SILO test suite and focused SFTP, IAM, and Parquet race tests. Lint, module verification, and dependency tidiness checks passed for all five projects.
- SFTP object reads, writes, listing, deletion, Unicode names, transfer block boundaries, and read-only data protection through the SILO driver against HTTP / HTTPS S3 backends. Upstream read-only server and symlink deletion regression tests passed. Both dependency baselines passed the new object-operation tests.
- JWX signature and time-claim boundary comparisons; 68 CLI output comparisons with no new differences, normalizing only existing metadata map ordering.
- Local etcd mTLS, account authentication, transactions, watches, leases, and old → new → old clients reading across server restarts; NATS / STAN message persistence, ACKs, durable subscription cursors, and reconnection.
- Local Azure / GCS protocol fixtures, KES GCP Secret Manager operations and pagination, the Azure SDK’s Arrow decoder tests, and the full upstream procfs parser fixtures. The procfs update corrects bcache units and NVMe sizes, so the corresponding metric values may change to reflect those corrections.
- Native Darwin arm64 and cross-compiled Linux amd64 builds of mcli, Console, SILO, and KES; NetBSD amd64 cross-builds of mcli and SILO.
Linux amd64 package-level Go vulnerability scans of all five projects found no affected imported packages (only the module-level GO-2026-5932 entry remains, with no imports of the affected OpenPGP subpackages).
2.2 - SILO 20261018 Release Notes
Server 20261018 is available with versioned archives, native packages, and standard/distroless Docker Hub images for amd64 and arm64. It supersedes Server 20260916 (20261015 is retained as a superseded historical draft record).
| Identity | Value |
|---|---|
| Release tag | RELEASE.2026-10-18T00-00-00Z |
| Package version | 20261018000000.0.0 |
| Release source | 0cd388c666f8211f552604ae73448a066e9c25ce |
| Previous public source | 2a4d51406b7ed87af5fe6fe0f801f3290f96eb3c (Server 20260916) |
| Complete change set | 20260916 → 20261018 |
See the download page for versioned archives, native packages, and pinned container commands.
Object Lock and IAM security
SN-2026-015 / SN-2026-016 — Object Lock retention authorization and in-place overwrite protection. #234 and #235 repair retention authorization and in-place overwrites:
- Every
PutObjectRetentionrequest evaluatess3:PutObjectRetentionwith the request’s Object Lock condition keys before looking up the target object. Shortening, clearing, or changing the mode of an activeGOVERNANCEretention additionally requires bothx-amz-bypass-governance-retention: trueands3:BypassGovernanceRetention(which supplements rather than replacess3:PutObjectRetention). Clearing retention evaluates remaining retention days as0, and activeCOMPLIANCEretention cannot be shortened or changed. - Caller-selected write
versionIdparameters require trusted replication credentials (s3:ReplicateObject). Directorynullversions and persisted multipart targets are re-verified under the destination write lock, and unreadable retention metadata fails closed while preserving existing legal holds. Upgrade all writing nodes and every replication site accepting S3 requests. See the security advisory and repair design.
SN-2026-017 — Admin credential self-service authorization. #242 ensures credential self-inspection and cleanup respect explicit Deny statements, session-policy restrictions (sessionPolicy), and external authorization plugin decisions. Under built-in authorization, restricted session credentials cannot create or update service accounts broader than their own scope, and non-root administrators cannot mutate root’s derived credentials; unrestricted empty-policy inheritance and cleanup of damaged non-root keys remain supported. Follow-up commit 995439260 normalizes bulk self-listing aliases to the parent identity and preserves self-inspection for ordinary users without attached policies, requiring no new Admin actions or IAM storage migration. See the security note and authorization contract.
Scoped STS revocation
0cd388c66 updates RevokeTokens to enumerate durable STS identity records alongside in-memory caches, allowing nodes that missed an earlier token-issuance notification to revoke matching tokens reliably. Committed token tombstones are broadcast to sibling nodes and site-replication peers even if another token in the batch fails to save, and offline peers converge via IAM deletion healing. Object-backed and etcd-backed cache refreshes also evict revoked tokens after missed notifications or watches.
The revocation scope remains the requested parent identity and optional token type across the cluster’s snapshot prior to the operation cutoff, leaving newly issued tokens, unrelated users, and parent policies untouched. Durable enumeration adds work only to the administrative revocation path, not to per-request authentication. See the STS revocation design and cost analysis.
Rebalance and pool decommission consistency
60c7179dd (incorporating #248) and 1748f62c7 prevent rebalance and pool-decommission workflows from overwriting a newer object copy in another pool with an older migrated copy. Migration re-verifies source versions (including multipart completions and tiered-object metadata) under the shared object lock, preserves null and UUID delete markers, and removes only confirmed migrated versions rather than deleting the entire source prefix. Concurrent writes, deletes, or destination changes safely abort cleanup for a later retry.
Before retiring a pool, verify object versions, tags, and delete-marker state as described in multi-pool object consistency (independent cross-site delete-marker ordering remains tracked in #217).
Ordinary object listing recovery
#239 (incorporating the original #221 work and closing #218) fixes an issue where ListObjects / ListObjectsV2 could succeed while omitting readable keys during eligible cross-pool metadata disagreements. The pool merger now re-verifies uncertain same-name entry groups under the corresponding object read locks: it returns current readable metadata, omits confirmed absent keys, or returns an explicit error such as 503 SlowDownRead when quorum cannot establish either state. A failed pagination producer fails the scan; clients must consume all pages and treat any page error as an incomplete listing.
Resolved entries retain their lock-free fast path. Recovery applies only to eligible latest-object groups without lowering quorum or adding persistent metadata; request cancellation interrupts lock backoff, and identified recovery lock conflicts wait for a single bounded interval. Follow-up commits 3dd53eb81 and 2080b3d03 preserve directory and CommonPrefixes entries when a colliding object name is confirmed absent. Measured latency and lock-cost benchmarks are documented in listing recovery lock cost.
Multipart upload listing and cleanup
Default disk scan rolled back to legacy in-memory listing. The default disk-scan implementation merged on October 17 (#243) was withdrawn by 4733e1423. Public ListMultipartUploads uses the node-local in-memory cache when prefix is empty and performs an exact-key lookup when prefix is nonempty; process restarts or requests routed to different nodes can omit in-flight uploads (scalable persistent enumeration remains tracked in #79).
Compared with 20260916, MINIO_API_MULTIPART_LISTING=strict is retired: setting the environment variable logs a startup warning and is ignored (the legacy shared api multipart_listing config key is likewise tolerated and ignored). However, the 20260916 upload-creation identity metadata, uploadId path validation, abort target matching, and post-abort cache/peer notifications remain active. The read-only multipart-preflight Admin diagnostic still requires admin:StorageInfo and reports mode=legacy, applying its entry, timeout, and concurrency budgets strictly to explicit diagnostic calls. See the multipart listing design, rollback rationale, and capacity benchmarks and the configuration reference.
Background stale-upload cleanup (default 24-hour expiry checked every 6 hours) includes two repairs:
- #236 parses native upload-directory timestamps before falling back to Base64 decoding, avoiding unnecessary disk metadata reads.
0f9199e0cremoves empty upload-parent directories immediately after deleting expired child upload directories, rather than waiting another 24-hour cycle after child deletion updates the parent’smtime(nonempty parents and active uploads remain protected).
Error classification, metadata reuse, and smaller repairs
| Change | User-visible result |
|---|---|
| #232: missing staged commit metadata | PUT and multipart completion return retryable 503 SlowDownWrite when staged metadata disappears before final commit, rather than misreporting IncompleteBody or NoSuchKey; storage RPCs and healing preserve this error class. |
| #233: reused metadata and scanner structs | Clear absent legacy EcM / Signature fields when decoding into pooled xl.meta structs and clear mutually exclusive scanner update/final fields (see stale metadata reuse). |
| #225, #229, #238: checksum error wording | Align Content-MD5 capitalization and algorithm-specific checksum mismatch messages with AWS S3; known algorithms are named in PUT / UploadPart errors while unknown algorithms retain the generic message. |
| #238: embedded Console error unwrapping | Preserve wrapped S3 error codes so attempting to suspend versioning on an Object Lock bucket surfaces InvalidBucketState cleanly in the UI. |
| #220: Lambda target sets | Fix the inverted TargetIDSet.IsEmpty boolean check. |
| Erasure-coding documentation | Clarify parity and drive-failure tolerance per erasure set and stored object layout (see storage classes). |
Coordinated components and dependencies
| Component | Selected identity |
|---|---|
| Embedded Console | v2.4.2 (v0.0.0-20261017112417-ef2dd2c41d21) |
| MC module / image-bundled mcli | 20261018 (v0.0.0-20261017101407-fa3b8bf72458) |
| Server shared package | github.com/pgsty/silo-pkg/v3 v3.14.3 |
| Upstream S3 SDK | github.com/minio/minio-go/v7 v7.4.1 |
| Go toolchain | 1.27.2 |
| Optional external key service | PGSTY KES 20261018 (standalone service at 2f7e741db905) |
Standalone Console v2.4.2 and mcli 20261018 declare silo-pkg v3.14.2, while Server’s module graph selects v3.14.3 (which optimizes admin-action classification without altering authorization semantics) and explicitly requires the maintained Console and MC modules.
minio-go v7.4.1 adds failed-upload cancellation cleanup, UTC retention timestamps, and preserved STS durations. Dependency updates include go-jose v4.1.6, ldap v3.4.15, compress v1.20.3, msgp v1.6.5, etcd v3.7.2, and golang.org/x/*. Note: go-jose v4.1.6 rejects OIDC/JWK RSA keys larger than 4096 bits; verify OIDC discovery, JWKS, and login flows before upgrading if your identity provider uses oversized RSA keys (see the dependency security note).
Release tooling pins GoReleaser v2.18.3, nFPM v2.48.0, golangci-lint v2.15.0, and govulncheck v1.9.0, with Object Lock race regressions gating release builds. make rebrand-guard now uses git grep and reports read errors explicitly without requiring ripgrep (7007c164a). Because the bundled Helm chart still defaults to the 20260916 Server image tag, override the image tag explicitly when deploying 20261018 via Helm.
Upgrade and rollback checklist
- Back up state and pin artifacts: record your installed versions and pin verified binary or image digests. Follow the migration guide and IAM recovery procedure to back up complete IAM storage, configuration, and encryption keys (standard live IAM exports omit revocation tombstones).
- Coordinate cluster cutover: upgrade all Server nodes and replication sites together; do not mix old and new binaries against the same IAM backend or perform rolling cross-version downgrades.
- Verify Object Lock and service-account policies: test retention updates, legal holds, service-account creation/updates, explicit self
Denyrules, and root credential management with the exact credentials used by your automation. Reissue previously damaged expiring keys through an authorized parent identity. - Remove
strictmultipart listing dependencies: ifMINIO_API_MULTIPART_LISTING=strictwas enabled on 20260916, ensure clients track their own activebucket/key/UploadIDtuples for completion or cancellation rather than relying on publicListMultipartUploadsfor full discovery. - Check listing, retry handling, and OIDC/TLS paths: verify ordinary and paginated
LIST, client retry policies for503 SlowDownWriteon PUT/completion, OIDC/JWKS RSA key sizes, and outbound connections across TLS-inspecting middleboxes (see the Go 1.27 TLS and OIDC guide). - Rollback considerations: downgrading to 20260916 reintroduces the October Object Lock and admin credential vulnerabilities, and upgrading does not retroactively restore versions or tags overwritten prior to the upgrade.
Artifact verification and acceptance record
The signed tag RELEASE.2026-10-18T00-00-00Z and all 36 release assets build from 0cd388c666f8211f552604ae73448a066e9c25ce and passed Go CI, VulnCheck, Release, and package finalization.
- Static artifact verification: verified all asset digests, 6 archive version stamps, 6 native package payloads (RPM/DEB/APK), 34 build provenance attestations, 2 Sigstore checksum signatures, 2 RPM package signatures, and 12 SBOMs.
- Functional and fault-recovery acceptance: exercised native Darwin and Linux S3/IAM/STS/Object Lock suites, LDAP/OIDC, in-place upgrade from 20260916, bucket and site replication, 3-site recovery with Console, 3-site and 4-process shared-cluster STS revocation, and 4 multi-pool migration scenarios (2/3 pools × rebalance/decommission). An 8-process, 2-pool cluster completed 2,896 full listing scans under rolling restarts and 1,270 full listing scans under fault-recovery load with zero missing keys, duplicates, or corrupted reads; 3 Linux package lifecycle suites and 8 container runtime variants also passed (Windows and Darwin amd64 artifacts were verified statically, and amd64 containers ran under emulation).
- Published Docker Hub digests:
- Standard image (and
latestat publication):sha256:2636fc4155d9a7e41ff58cdb65ddd13ba9304e67184e13c25bc5cb34016269d7 - Distroless image:
sha256:472f59c7be537571f0f9a2ee6b8fe033a220939d521255bd4bf5339e53db7218
- Standard image (and
Thanks to szopi00 for reporting the Object Lock issues; mr javad seydi (@mrjavadseydi) for the initial LIST and multipart work; Julien Laurenceau (@julienlau) for the commit-metadata classification and struct-reuse fixes; clown15 for upload-directory cleanup; cbornet and yupoet for checksum error message improvements; AleksaMCode for the Lambda target-set fix; and Denis Babchuk (@botdeniska) for the #248 migration overwrite fix and silo-pkg optimization.
2.3 - silo-pkg 3.14.3 Released
Release: v3.14.3 · Source: e9d585ba1e7a. See the v3.14.2 release notes for the October dependency baseline.
github.com/pgsty/silo-pkg/v3 v3.14.3 reduces the cost of classifying admin: actions during IAM policy evaluation and is selected by Server 20261018.
AdminAction.IsValid and HasResource now perform exact map lookups before scanning the action sets: literal misses return immediately, and wildcard patterns whose prefix cannot match the admin: namespace skip the validation scan, while patterns such as adm* and a*d*m*i*n* retain their exact existing behavior.
Performance and validation
Across six alternating single-CPU benchmark runs on an Apple M5 Max (Go 1.27.2), median evaluation time for a policy with 100 non-matching bucket grants followed by a matching grant dropped from 303.2 µs to 54.4 µs (~82% reduction) with zero additional heap allocations.
Validation covered Go 1.26 and 1.27 race suites, lint, LDAP integration, vulnerability scanning, 60 seconds of equivalence fuzzing, and an exhaustive comparison across 50,000 policies and 970,020 policy evaluations against v3.14.2; mcli 20261018 also produced identical results against server builds using v3.14.2 and v3.14.3.
Adoption
Public APIs, policy formats, authorization decisions, dependencies, and the go 1.26 floor are unchanged. Server 20261018 adopts v3.14.3, while standalone mcli 20261018 and Console v2.4.2 remain on v3.14.2 (see the component matrix).
Thanks to Denis Babchuk (@botdeniska) for PR #11, and Klaus Post (@klauspost) for the original optimization.
2.4 - PGSTY KES 20261018 Released
Release: RELEASE.2026-10-18T00-00-00Z · Package: 20261018000000.0.0 · Source: 2f7e741db905.
This is the first maintained PGSTY release of the Key Encryption Service (KES). KES is a standalone stateless service for SILO deployments that require external KMS-backed server-side encryption (see the component matrix).
Changes
- Establish PGSTY maintenance and release pipelines while preserving upstream copyright, attribution, and third-party licenses. The executable and system packages remain
kes, and the Go module path remainsgithub.com/minio/kes. - Build with Go 1.27.2, refresh dependencies, and migrate the AWS Secrets Manager backend from the deprecated AWS SDK for Go v1 to AWS SDK for Go v2. Preserve YAML configuration fields, explicit credential precedence, custom endpoints, and stored key formats (continuing to ignore
~/.aws/configand SDK endpoint environment overrides). - Preserve the KES HTTP API,
MINIO_KMS_KES_*configuration settings, existing root keys, and encrypted objects so existing SILO + KES deployments can upgrade in place without key migration. - Add upgrade, rollback, and restart integration suites pairing SILO with OpenBao KV v2 and the Moto Secrets Manager emulator, restarting SILO at each version transition so in-memory key caches cannot mask compatibility regressions.
- Ship signed Git tags, GitHub build provenance attestations, GPG-signed RPMs, and non-root container images, with the built-in
kes updateself-updater disabled.
Deployment and upgrade
In a server-side encryption deployment, SILO encrypts object contents, KES authenticates mTLS client requests and wraps or unwraps data encryption keys (DEKs), and OpenBao persists the root key encryption keys (KEKs). Point MINIO_KMS_KES_ENDPOINT at KES, not OpenBao (see the KES and OpenBao design note for architecture boundaries and recovery requirements).
Configure TLS certificates, mTLS client identity policies, and a persistent key store before starting KES in production (development mode stores keys only in memory). Official containers run as non-root UID/GID 1000:1000, so mounted configuration files and certificates must be readable by that user; Linux RPM/DEB/APK packages install /usr/local/bin/kes without starting an unconfigured service.
Back up your key store and configuration before upgrading, then verify decryption of existing objects, creation of new encrypted objects, and cold readback after restarting both KES and SILO.
Delivery and validation
All 19 release assets (6 Linux/macOS/Windows amd64/arm64 archives, 6 Linux RPM/DEB/APK packages, an archive checksum manifest, and 6 package checksum sidecars) were verified across downloaded digests, binary source/platform stamps, package payloads, both RPM signatures, and 11 GitHub build attestations (release workflow).
The official multi-arch (linux/amd64 and linux/arm64) container image docker.io/pgsty/kes:RELEASE.2026-10-18T00-00-00Z (Docker publication run) has index digest:
The primary verified integration stack is SILO + PGSTY KES + OpenBao KV v2 (covering mTLS authentication, OpenBao AppRole access, DEK wrap/unwrap, restart persistence, unauthenticated request rejection, and interoperability/rollback matrices across SILO 20260903/20260916 and upstream KES 2025-03-12T09-35-18Z, with AWS Secrets Manager protocol and storage regression coverage via Moto).
Download KES · Configuration and installation · Artifact verification · Changes since upstream.
2.5 - SILO Console 2.4.2 Released
Release: v2.4.2 · Source: ef2dd2c41d21.
Console v2.4.2 ships both as a standalone release and as the embedded Console in Server 20261018 (see also the silo-pkg v3.14.2 release notes).
Changes
- Adopt mcli 20261018, silo-pkg v3.14.2, and upstream minio-go v7.4.1. The SDK cleans up failed multipart uploads after cancellation, formats retention timestamps in UTC, and preserves requested STS durations (parallel GET and ComposeObject remain opt-in).
- Preserve wrapped S3 error codes during bucket-versioning operations so attempting to suspend versioning on an Object Lock bucket surfaces
InvalidBucketStatecleanly. - Build with Go 1.27.2 and Node 24.22.0 LTS. Refresh
golang.org/x/*modules alongside compress v1.20.3, msgp v1.6.5, etcd v3.7.2, fwd v1.2.1, and cpuid v2.4.1 (retaining go-systemd v22.6.0 for NetBSD and tablewriter v0.0.5 for MC API compatibility). - Pin GoReleaser v2.18.3, golangci-lint v2.15.0, and govulncheck v1.9.0, rebuild embedded frontend assets as Console 2.4.2, and update pinned container build stages.
Integration
Because the retained module path has no /v2 suffix, Go consumers reference Console v2.4.2 via pseudo-version v0.0.0-20261017112417-ef2dd2c41d21. An embedding Server must declare both Console and MC replacements explicitly:
Console v2.4.2 requires github.com/pgsty/silo-pkg/v3 v3.14.2 (see the password-permission migration guide). Note that the go-jose v4.1.6 4096-bit RSA JWK limit applies only to Server OIDC paths that import go-jose; neither go-jose nor go-ldap is linked into the standalone Console binary.
Delivery and validation
All 44 release assets (6 platform binaries, 6 archives, 9 Linux packages, checksum manifests, SBOMs, and signed provenance attestations) were verified against tagged commit ef2dd2c41d21. The commit passed the full CI matrix (permissions, SSO, replication, browser tests, source-image equivalence, and reproducible packaging), vulnerability checks, the local Go race suite, 251 frontend unit tests, and live API/browser sharing checks against Server 20261018.
The published docker.io/pgsty/silo-console:v2.4.2 multi-arch (linux/amd64 and linux/arm64) manifest digest is:
Release verification contract · Component matrix · Changes since v2.4.1.
2.6 - mcli 20261018 Released
Release: RELEASE.2026-10-18T00-00-00Z · Package: 20261018000000.0.0 · Source: fa3b8bf72458.
mcli 20261018 ships both as the standalone CLI client and as the coordinated client module in Server 20261018 and Console v2.4.2 (see also the silo-pkg v3.14.2 release notes).
Changes
- Adopt silo-pkg v3.14.2 and upstream minio-go v7.4.1. The SDK cleans up failed multipart uploads after cancellation, formats retention timestamps in UTC, and preserves requested STS durations while retaining existing
CopyObjectembedded-error and checksum handling (parallel GET and ComposeObject remain opt-in). - Build with Go 1.27.2. Refresh
golang.org/x/*modules, compress v1.20.3, msgp v1.6.5, and etcd clients v3.7.2 (with fwd v1.2.1 fixing reads that return bytes alongside an error as well asSkipoffsets, and cpuid v2.4.1 fixing cache detection). - Pull in go-jose v4.1.6 (fixing excessive CPU usage on oversized RSA JWKs in GHSA-mgcc-cww5-fp2q) and LDAP v3.4.15 via
silo-pkg. - Refresh pinned UBI 9 base images and release tooling: GoReleaser v2.18.3 (including log secret redaction and archive fixes), nFPM v2.48.0, golangci-lint v2.15.0, govulncheck v1.9.0, and gofumpt v0.12.0.
Upgrade
CLI commands, aliases, MC_* environment variables, and configuration formats are unchanged (retaining the go-systemd v22.6.0 replacement for NetBSD builds and tablewriter v0.0.5). See the component matrix for version pairings.
Delivery and validation
All 19 release assets (6 Linux/macOS/Windows amd64/arm64 archives, 6 Linux RPM/DEB/APK packages, an archive checksum manifest, and 6 package checksum sidecars) were verified across downloaded digests, 6 binary source stamps, 6 package payloads, PGSTY GPG signatures (9592A7BC7A682E7333376E09E7935D8DB9BD8B20), and 11 GitHub build attestations bound to fa3b8bf72458.
The tagged commit passed Go unit tests, cross-compilation, vulnerability scanning (zero reachable or imported-package vulnerabilities; the unused OpenPGP module-only advisory GO-2026-5932 remains), and live Go/Shell TLS functional suites against SILO fbcb5119782a (release build · Docker publication).
The published docker.io/pgsty/mc:RELEASE.2026-10-18T00-00-00Z (and latest at publication) multi-arch (linux/amd64 and linux/arm64) manifest digest is:
Download and installation · Full changelog · Changes since 20260916.
2.7 - silo-pkg 3.14.2 Released
Release: v3.14.2 · Source: c6835145176f.
github.com/pgsty/silo-pkg/v3 v3.14.2 provides the October dependency baseline for mcli 20261018 and Console v2.4.2; follow-up patch v3.14.3 adds an admin-action classification optimization used by Server 20261018.
Changes
- Update upstream
minio-goto v7.4.1: clean up failed multipart uploads even after caller context cancellation, serialize Object Lock retention timestamps in UTC, preserve requested STS durations, and support projected Kubernetes service-account token files (parallel GET and ComposeObject remain opt-in). - Update
go-joseto v4.1.6: fix excessive CPU consumption when parsing oversized RSA JWKs (GHSA-mgcc-cww5-fp2q); update LDAP to v3.4.15, msgp to v1.6.5, and etcd clients to v3.7.2. - Refresh
golang.org/x/*:x/cryptov0.58.0,x/netv0.61.0,x/sysv0.49.0, andx/textv0.43.0. Built and tested with Go 1.27.2 while retaining thego 1.26module floor and thego-systemd v22.6.0NetBSD replacement.
Compatibility and adoption
Public package signatures and password-policy semantics are unchanged (see the password-permission migration guide when upgrading from before v3.14.0, and the component matrix for version pairings).
Behavioral notes:
go-jose v4.1.6rejects RSA JWKs larger than 4096 bits; verify identity-provider key sizes in OIDC/JWKS consumers.- STS durations below one hour are now sent as requested rather than silently clamped by the client, so invalid short durations will be rejected by the server.
Validation
The release passed full lint and race suites under Go 1.27.2, race tests under Go 1.26.9, OpenLDAP integration tests, module tidiness checks, and govulncheck scanning (zero reachable or imported-package vulnerabilities).
2.8 - SILO 20261015: Superseded Draft Record
Status: Superseded by Server 20261018. RELEASE.2026-10-15T00-00-00Z was briefly published at 2026-10-15 13:52:28 (UTC+8), then reverted to Draft and superseded by Server 20261018, which includes the complete October repair and component stack. This page archives the candidate source scope and acceptance test results; use 20261018 for all deployments and upgrades.
- Historical tag:
RELEASE.2026-10-15T00-00-00Z - Package version:
20261015000000.0.0 - Candidate source:
7a3ca0f82100cce06289a465f54139da2639f642(full diff against Server 20260916)
Object Lock repairs
- SN-2026-015 — Retention authorization (#234): every
PutObjectRetentionrequest requiress3:PutObjectRetention, evaluated with request-body Object Lock condition keys before reading object state. Shortening, clearing, or changing the mode of activeGOVERNANCEretention additionally requires the explicit bypass header ands3:BypassGovernanceRetention(which does not override aDenyons3:PutObjectRetention). Clearing retention evaluates remaining days as0, and distant future dates no longer saturate duration arithmetic. - SN-2026-016 — Protected version writes (#235): caller-selected write
versionIdparameters require trusted replication credentials (s3:ReplicateObject). Directorynullversions and persisted multipart targets are checked under the destination write lock before overwrite (including uploads initiated before upgrading or recovered from legacy staging state), enforcing activeGOVERNANCE,COMPLIANCE, and legal-hold protections; unreadable lock metadata fails closed. - Release gate (#237): the release workflow runs the Object Lock race regression suite as a build prerequisite, covering slow-body directory overwrites, preflight read failures, legacy multipart completions against
COMPLIANCE, and recovered-upload UUID collisions.
See the security advisory and repair design for full details.
Other changes since 20260916
| Change | User-visible effect |
|---|---|
| #233 | Clear absent fields when decoding reused legacy xl.meta headers and keep scanner update/final responses mutually exclusive (see stale metadata reuse). |
| #225 | Align checksum error messages with AWS S3, including Content-MD5 capitalization. |
| #220 | Fix the inverted TargetIDSet.IsEmpty check for Lambda notification target sets and add regression coverage. |
fa565ec9f |
Upgrade to Go 1.27.2, updated golangci-lint, and golang.org/x/net v0.60.0 across module requirements and pinned Docker build stages. |
Component identities
| Component | Selected version |
|---|---|
| Server | RELEASE.2026-10-15T00-00-00Z (package 20261015000000.0.0) |
| Embedded Console | v2.4.1 (source 1360e26d976d) |
| mcli | RELEASE.2026-09-16T00-00-00Z (source e952aa78f10a) |
| Shared package | github.com/pgsty/silo-pkg/v3 v3.14.1 |
| Upstream S3 SDK | github.com/minio/minio-go/v7 v7.3.1-0.20260915093545-32e1f32cb176 |
| Go toolchain | Go 1.27.2 |
Upgrade requirements
- Upgrade all writers and replication peers: upgrade every Server node and all replication sites accepting S3 writes so unpatched peers cannot accept unauthorized retention changes or in-place overwrites.
- Review retention policies: policies granting
s3:BypassGovernanceRetentionwithouts3:PutObjectRetention, or relying on bypass to evade retention condition keys, now receive403 AccessDenied. - Caller-selected
versionIdrestriction: ordinary S3 writes (including root-signed requests) cannot specify a targetversionId; trusted replication preserves version IDs, andmcli batch replicateinto SILO requiress3:ReplicateObjecton the destination credentials. - Fail-closed lock checks: locked directory objects and locked multipart targets cannot be overwritten; if lock metadata is unreadable, restore drive health before retrying.
- Audit historical state: review prior audit logs for unexpected retention changes before the upgrade (when upgrading from releases prior to 20260916, also follow the September upgrade requirements).
Candidate artifact acceptance
The following CI workflows and verification checks passed against 7a3ca0f82100cce06289a465f54139da2639f642:
| Check | Result | Workflow run |
|---|---|---|
| Main Go CI | Passed | Run 38686240587 |
| Vulnerability check (VulnCheck) | Passed | Run 38686240625 |
| Release build and Object Lock gate | Passed | Run 38686425467 |
| Finalize release packages | Passed | Run 38687734395 |
- Static artifact verification: verified all 36 release assets, 34 build provenance attestations, static version/commit identities across all 6 Linux/macOS/Windows
amd64/arm64binaries, payloads across all 6 RPM/DEB/APK packages, and both RPM signatures (see historical archive checksums and package checksums). - macOS arm64 runtime acceptance (110/110 passed):
| Scenario | Passed |
|---|---|
| Fresh installation, full Object Lock API suite | 49/49 |
| In-place upgrade over data created by the official 20260916 binary | 12/12 |
| Full Object Lock API suite after upgrade | 49/49 |
(Across those runs, all 65 rejected operations left the complete target object state unchanged; Windows and macOS amd64 binaries were verified statically.)
Docker images
Docker publication run 38689158028 built and attested the multi-platform (linux/amd64 and linux/arm64) images from the same release commit:
| Variant | Version tag | Multi-platform index digest |
|---|---|---|
| Classic | RELEASE.2026-10-15T00-00-00Z |
sha256:aaac6bce13eb99316b3c88bd20143fc3c38ab607585256a411a095c44ee57f62 |
| Distroless | RELEASE.2026-10-15T00-00-00Z-distroless |
sha256:70cac541c5a928be5cd16d6b8732d85ac06e891a0e3d9956a745381330692fc6 |
All four platform-specific images (whose /usr/bin/silo binaries matched the official Linux archive binaries byte for byte) passed /minio/health/ready probes and the full Object Lock API suite (196/196 passed, with all 120 rejected operations preserving object content, version identity, ETag, retention, legal hold, and version listings):
| Image | Runtime environment | API checks | Binary comparison |
|---|---|---|---|
Classic linux/arm64 |
Native ARM64 on Docker Desktop ARM64 | 49/49 | Byte-identical to Linux arm64 archive |
Classic linux/amd64 |
AMD64 emulation on Docker Desktop ARM64 | 49/49 | Byte-identical to Linux amd64 archive |
Distroless linux/arm64 |
Native ARM64 on Docker Desktop ARM64 | 49/49 | Byte-identical to Linux arm64 archive |
Distroless linux/amd64 |
AMD64 emulation on Docker Desktop ARM64 | 49/49 | Byte-identical to Linux amd64 archive |
Combined with the native macOS arm64 installation and upgrade suite, candidate runtime acceptance passed 306/306 checks.
2.9 - SILO Console 2.4.1 Released
SILO Console v2.4.1
was released on September 16, 2026 from
1360e26d976d.
It integrates pkg v3.14.1 and mcli 20260916, restricts the anonymous sharing proxy,
streams multi-object ZIP downloads and adds signed, traceable release artifacts.
Shared downloads
The fix for #52 restricts the anonymous proxy to object-content GET requests at the configured S3 origin. It rejects redirects, system paths and query-selected operations that are not object downloads. Public objects, presigned links and version-specific downloads continue to work. No new sharing-disable environment variable is needed.
Thanks to Jiri Pejchal (@jiri-pejchal) for reporting the internal-metrics exposure. See the sharing boundary for the supported request forms and reverse-proxy requirements.
Password-policy migration
This patch release includes an authorization change. The Change Password
button and session capability use admin:ChangeMyPassword; creating users and
resetting another user’s password continue to use admin:CreateUser.
With the matching SILO Server, denying CreateUser alone no longer locks the
caller’s own password. To retain that restriction, deny both actions in the
same statement before upgrading, keeping its original scope and conditions.
The updated built-in readonly permits self-service password changes and no
longer overrides a separate CreateUser Allow. Saved policy overrides retain
their stored statements. Upgrade Server, Console and pkg together and review
the migration and rollback guide.
Downloads and browser recovery
- Multi-selection ZIP downloads stream to a file writer when the browser supports it, with backpressure, cancellation and duplicate-transfer protection. Other browsers hand the download to their native download manager. Neither path buffers the complete ZIP in JavaScript.
- Progress uses a percentage only when the response supplies a reliable total. The native fallback identifies the handoff explicitly; cancellation then belongs to the browser’s download manager. Selections over 5 GiB or with an unknown total recommend mcli for long-running transfers.
- Invalid routes and rendering failures offer recovery without clearing preferences. English and Chinese labels, icon controls, sign-out text and keyboard-accessible tooltips are completed.
Dependencies and embedding
| Component | Selected version |
|---|---|
| Shared package | github.com/pgsty/silo-pkg/v3 v3.14.1 |
| MC source | github.com/pgsty/mc v0.0.0-20260916070421-e952aa78f10a — mcli 20260916 |
| Upstream minio-go | v7.3.1-0.20260915093545-32e1f32cb176 |
| JWX / strfmt | v3.3.0 / v0.27.2 |
| React Router | v7.18.4 |
| Build tools | Go 1.27.1, Node 24.21.0, Yarn 4.13.0 |
The SDK handles S3 errors embedded in HTTP 200 CopyObject responses; JWX fixes custom JSON field-name escaping and strfmt updates hostname validation for Go 1.27. The embedded frontend and third-party credits are regenerated from this dependency graph. go-systemd v22.6.0 and tablewriter v0.0.5 remain intentional compatibility pins.
Go embedders select Console as
github.com/pgsty/silo-console v0.0.0-20260916075814-1360e26d976d through the
historical github.com/minio/console module replacement. Go does not inherit
dependency replacements: copy the explicit MC replacement from the
tagged README.
The supported administration target is the coordinated SILO stack; upstream
MinIO/MC compatibility is best effort.
Packages and verification
The GitHub Release contains 44 assets: binaries and bundles, DEB/RPM/APK packages, source, legal notices, SPDX SBOMs, a checksum manifest and its Sigstore bundle. Checksums use Cosign signatures, and provenance records the tagged workflow inputs.
For standalone v2.4.1 deployments, use the GitHub Release binaries or Linux packages; multi-architecture Docker Hub images on docker.io/pgsty/silo-console are available starting with Console v2.4.2. Server-embedded Console is unaffected.
Linux packages retain minio-console.service, console-user and
/etc/default/console. The service now uses /var/lib/silo-console for state,
/etc/silo-console/certs for certificates, and a 90-second shutdown limit.
Before restarting an existing installation, migrate certificates with the
documented ownership or retain the old path through CONSOLE_OPTS. The installer
preserves old certificates and keys and does not restart the service for you.
See the package upgrade instructions.
The exact release source passed the complete CI matrix, vulnerability checks and release verification. Local validation also covered 251 frontend unit tests, 51 browser tests, byte-identical embedded assets and sharing in standalone and embedded deployments. The downloaded Darwin/arm64 executable was independently checked against the signed checksum manifest and exercised against a SILO test server.
2.10 - silo-pkg 3.14.1 Released
v3.14.1 was published from
fa657ef4.
This is a dependency and tooling release of github.com/pgsty/silo-pkg/v3;
the diff from v3.14.0
changes no package Go source or public API.
Dependency changes
- JWX advances from v3.2.0 to v3.3.0. The vendor’s release notes
identify GHSA-4cf7-xm37-g63h: custom JSON field names must be escaped during
encoding. Reachability depends on accepting attacker-controlled custom claim
names; this package’s
envJWT path uses registered claim names. - Upstream minio-go advances to
v7.3.1-0.20260915093545-32e1f32cb176, including minio-go #2306: recognize an S3CopyObjecterror embedded in an HTTP 200 response instead of reporting a successful copy. - Testify advances to v1.12.1 and the lint configuration adopts
gomodguard_v2. The Go 1.26 floor, Go 1.27.1 toolchain and go-systemd NetBSD replacement remain.
Component and migration boundary
The v3.14.0 password-policy change remains
in force; this patch does not revert it. Preserve the paired
admin:CreateUser / admin:ChangeMyPassword Deny where that was the intended
restriction, using the migration guide.
SILO 20260916, mcli 20260916, and Console 2.4.1 select this package; see the component matrix.
Release verification is recorded in the linked release and PR #9.
2.11 - SILO 20260916 Release Notes
Version date: 2026-09-16 · Published: 2026-09-16 UTC (September 17 in China) · Tag: RELEASE.2026-09-16T00-00-00Z · Package version: 20260916000000.0.0. Use the download page for installation.
The source range runs from released commit 9b11dc9469e6 to released 2a4d51406b7e: 210 commits, including 140 non-merge commits. These notes describe the final behavior, including the listing-quorum, version-purge and migration-tag repairs after #214.
Highlights
- Stronger request authentication: reject unsigned operation headers in ordinary and presigned SigV4, verify presigned body hashes, and use authenticated inputs for signature-age and payload-hash policies.
- Durable IAM revocation: retain deletion revisions, parent revocation boundaries and membership grant times across delayed events, same-name recreation and restart/recovery.
- One logical object across pools: coordinate conditional PUT, multipart completion, conditional DELETE, addressed-version deletion and metadata updates under consistent selection and locking rules.
- Preserve tags during pool migration: rebalance and decommission retain tags, their original revision and ordered empty states for ordinary and multipart objects.
- Correct encrypted and federated copies: preserve raw SSE-C replicas, plaintext part sizes, independently ordered lock/tag metadata, and the checksum, version and time of the actual destination write.
- More reliable replication and configuration convergence: repair tag resurrection, marker purge/MRF recovery, resync counters/cancellation, and bucket metadata source timestamps and deletions.
- Opt-in durable multipart discovery: add persistent scanning, global pagination and read-only preflight. Legacy remains the default; capacity and late-creation-write limitations remain explicit.
- Updated administration stack: select Console 2.4.1, mcli 20260916 and pkg 3.14.1, including bounded browsing, streaming ZIPs, reliable failure exits and policy correctness fixes.
- Two upgrade requirements need advance preparation: password permissions change the meaning of existing policies, and durable IAM revocation requires coordinated upgrades and complete backend backups.
Versions and components
| Component | Server 20260903 | Server 20260916 |
|---|---|---|
| Server | RELEASE.2026-09-03T13-18-01Z |
RELEASE.2026-09-16T00-00-00Z; inspected source 2a4d51406b7e |
| Embedded Console | Source 464a59d73ada, v2.3.0 identity |
Released v2.4.1, source 1360e26d976d |
| MC module / image-bundled mcli | 20260903 | Released 20260916, source e952aa78f10a |
| Shared package | github.com/pgsty/silo-pkg/v3 v3.13.2 |
v3.14.1, including the v3.13.3 and v3.14.0 changes |
| Upstream S3 SDK | minio-go/v7, revision 0e78d3f18efe |
v7.3.1-0.20260915093545-32e1f32cb176 |
| Go / madmin-go | Go 1.27.1 / v3.0.110 | Unchanged |
| Helm | 7.0.2 | Chart 7.0.3 in the tagged source; no separate chart package or Helm repository publication |
The release explicitly replaces Console and MC with v0.0.0-20260916075814-1360e26d976d and v0.0.0-20260916070421-e952aa78f10a. Go does not inherit a dependency’s replacements. The shared package uses the PGSTY module path directly; Server, Console and MC retain their historical module identities and protocol names. minio-go/v7 continues to use a verified upstream revision.
The supported release stack is SILO + Console + mcli + silo-pkg; compatibility with unmodified upstream MinIO/MC remains best effort.
Security and authorization
This cycle includes four local security identifiers. SN- identifiers are project tracking IDs, not CVEs. See the signed-header contract and September security notes.
| Identifier | Repair and scope | References |
|---|---|---|
| SN-2026-011 | Ordinary Authorization and presigned SigV4 reject unsigned x-amz-* operation headers, preventing a signed PUT from becoming a CopyObject with the signer’s authority. Empty-first-value multi-value headers cannot bypass the check. |
#173, #177 |
| SN-2026-012 | Generic authentication verifies presigned bodies whose hash is supplied only through a header; mismatches return XAmzContentSHA256Mismatch. The caller must hold the corresponding signed write URL. |
#177 |
| SN-2026-013 | Persistent IAM revisions and parent boundaries prevent older events, child credentials and grants from crossing retained revocation history. | #191, #192 |
| SN-2026-014 | Console’s anonymous sharing proxy accepts only object-content GETs at the configured S3 origin and rejects redirects, system paths and operation-selecting queries. | Console #56, Server #209 |
Related changes remove ambiguous duplicate replication-source headers, derive s3:signatureAge from the signed date rather than a scratch header, align presigned query/header payload-hash precedence, and inject the server-generated PutObjectTagging header after authentication but before authorization. X-Amz-Content-Sha256 may remain outside SignedHeaders because its effective value is separately bound and checked. Explicitly signed UNSIGNED-PAYLOAD retains its meaning.
Streaming SigV4 seed operation headers remain outside this check. Unsigned-header rejection returns 400 AccessDenied. The Console fix adds no global sharing-disable variable and continues to enforce S3 authorization.
Password permission split: a compatibility change
Server #175, pkg #6 and Console #51 separate password self-service from user administration:
- An internal user changing their own password checks
admin:ChangeMyPassword, allowed unless explicitly denied. - Creating users or resetting someone else’s password still requires
admin:CreateUser. Existing root, STS and service-account restrictions remain. - A standalone CreateUser Deny no longer locks the caller’s password. An existing ChangeMyPassword Deny now constrains this endpoint.
- Updated built-in
readonlyno longer overrides a separate CreateUser grant with its own deny; the package also supplies the separateconsolereadonlycapability policy. Saved policy overrides are not rewritten.
To retain the old combined restriction, deny both actions in the same statement before upgrading, preserving its resources and conditions. Keep both denies during rollback: an old Server cannot enforce a password-only deny on this endpoint. See password migration.
Shared policy correctness
pkg #4 fixes statement deduplication that omitted NotResource from hashing or collapsed distinct string-condition sets, potentially discarding Deny clauses. Directly constructed policies report Deny before indexing. Multi-star wildcard matching no longer uses exponential recursion while retaining byte matching and historical MatchSimple boundaries.
Requests that relied on lost denies may now be rejected. If a clause has already disappeared from stored JSON, upgrading cannot reconstruct it: compare with the original policy and reapply it.
IAM replication, revocation and recovery
Deletion revisions remain in the original IAM paths for users, service accounts, groups, policies and policy mappings. Normal listing and authorization hide deleted records. Object and etcd backends serialize revision comparison and persistence, retain source timestamps, and prefer deletion over a live record at the same timestamp.
- Delayed same-backend deletion notifications reload committed storage rather than deleting a subsequently recreated identity.
- Recreated parents retain
RevokedBefore; new service accounts and internal STS credentials carry a signedsiloParentRevocationboundary. Replaying or editing an old child does not renew its authority. - Each group member retains its own grant time. Editing another member or group status does not regrant every member.
- Newer service-account snapshots can replace an older copy of the same key while preserving secrets, disabled state, absolute expiration and revocation boundaries. Credential-type conflicts fail; expired snapshots cannot extend old credential life.
- Healing resumes after leadership is regained, retains acknowledgements and retries bounded batches. Natural expiration of ordinary STS credentials does not automatically create permanent deletion history; reusable service-account keys retain revisions.
- IAM metrics report retained revocations, healing failures, duration and last success. See the design and metrics.
Cross-site revocation propagates asynchronously. A parent deletion commits before secondary cleanup; if cleanup returns HTTP 500, the parent revocation is already in effect. Tombstones have no TTL or automatic compaction, and storage/startup costs grow with retained history. Concurrent independent membership edits, equal-time conflicting live updates, arbitrary clock skew and cross-site individual-token revocation retain documented limits.
S3 conditions and multi-pool consistency
The coordination layer selects a logical object under a shared namespace lock before evaluating conditions or merging copies of the same version. Relevant draining pools participate. Retention, legal hold and tags each use their own revision; object modification time is not a version for every metadata field.
| Operation | Change | References |
|---|---|---|
| Single-object DELETE | A nonempty If-Match compares the client-visible ETag before deletion and rejects a mismatch with 412. * requires an existing non-marker representation; explicit versionId compares that version. |
#145, #178 |
| Ordinary conditional PUT | If-Match/If-None-Match use the logical current object across pools, rather than a stale destination copy. A current delete marker means absence. | #207, fixing #199 |
| Conditional multipart completion | Evaluate once across pools under the object lock, not only beside the upload. | #190 |
| Ordinary addressed-version DELETE | Reconcile copies across pools, covering UUID/null versions, delete markers and applicable directory-marker deletion, preventing an older surviving copy from becoming visible under the supported quorum conditions. | #188 |
| Object and metadata mutation | Coordinate writes, metadata, healing, replication and movement while retaining independent lock/tag revisions and shared remote-tier references. | #178, #189 |
Availability tradeoff: unverifiable metadata in another relevant pool can fail conditional PUT/completion or coordinated deletion even when another pool serves GET/HEAD. Read-quorum failures return 503; restore readability before retrying. Cleanup can fail after partial physical mutation; there is no distributed rollback. Pending outbound delete replication retains versions until purge completes.
A public conditional PUT with a destination versionId compares the current object while preserving the requested write version. It does not clean stale copies, reverse previously accepted overwrites or provide global clock ordering. Multipart #190 and ordinary PUT #207 address separate defects.
Unsupported boundaries: batch DeleteObjects XML ETag still provides no condition protection. Empty If-Match follows ordinary deletion; internal recursive prefix deletion bypasses object conditions. There is no s3:if-match policy key or new GetObject authorization requirement. Use the supported single-object path when conditions matter. See conditional DELETE and multi-pool consistency.
Preserve object tags during pool migration
fced8630365f repairs lost object tags during rebalance/decommission. The read representation stores tags separately in ObjectInfo.UserTags; forwarding UserDefined alone could omit the canonical tag field needed by the destination write.
The four ordinary-PUT and multipart-initialization entry points now rebuild that field in an independent metadata copy. They preserve the original tag revision, including absent/empty revisions and ordered empty states, without generating a new revision or mutating the reader’s map. Writes and multipart completion retain existing rereads under the coordination lock so an old reader does not override a later tag update or removal.
The patch does not recover tags lost during earlier migrations. Audit previously migrated objects and tag-dependent policy and lifecycle rules separately.
Null-version listing quorum
8d06424b1 prevents a newer minority from hiding an older null version that has listing quorum. It recounts per header only when the original selection lacks quorum, every nonempty drive stream has exactly one ordinary null version, and all share the same erasure layout. Mixed version histories retain their previous behavior.
Remaining limitation: #218 tracks successful listings that still omit readable objects during rolling restarts with concurrent overwrites. Do not run synchronization tools that delete destination objects based on such listings; list again after the cluster stabilizes.
Encryption, checksums and object attributes
- Raw SSE-C replicas: store authorized replica ciphertext verbatim, avoiding another encryption/compression pass caused by destination defaults (#109, #122).
- Plaintext multipart sizes: preserve logical part lengths in replicas, repairing part-number/range reads, object size after replacement and attributes (#119, #124).
- No new compressed SSE-C objects: exclude ordinary PUT, multipart initialization, COPY and Snowball SSE-C writes from compression. SSE-S3/KMS retain their encryption-compression options (#118, #126).
- Retransmission repair: update existing or undecodable SSE-C replicas through object retransmission instead of ordinary metadata COPY, preserving newer destination retention/hold (#120, #134).
- Explicit checksums during key rotation: perform the required full rewrite, including requests for the existing algorithm. A multipart source may become a single-part destination and its ETag may change (#113, #123).
- Logical attributes: GetObjectAttributes returns plaintext part sizes for compressed/encrypted objects and terminates pagination correctly with sparse part numbers (#114, #115, #128).
- Honest unreadable-object errors: return 500 for integrity/decryption failures instead of a 206 accepted as a successful read (#110, #121).
- Chunked checksum interoperability: honor and validate the actual checksum header on non-trailer chunked PUTs even when x-amz-trailer is advertised (#107, #143).
These prevent new errors and provide supported repair paths; they do not rewrite historical encrypted data. Retain keys and exact version identities when auditing compressed SSE-C, wrong part sizes or unreadable replicas. See SSE-C integrity.
Federated CopyObject
This section concerns remote forwarding in legacy etcd bucket federation, rather than bucket/site replication. Changes accumulate through #157, #159, #163, #172, #177 and #179.
| Previous problem | Corrected behavior |
|---|---|
| Ignored checksum algorithm and incorrect empty/multipart results | Compute/forward logical full-object checksums with correct empty-body, trailer and inherited-checksum handling; reject missing, malformed or multipart-suffixed required remote results |
| Inline source rejected because internal metadata leaked | Strip the complete reserved internal metadata category case-insensitively; use public fields for metadata and tags |
| Wrong encrypted-source bytes/length or double transformation | Forward logical plaintext and length; let the destination apply explicit SSE or its defaults; send KMS context as a JSON object |
| Proxy auto-encryption overrode destination defaults | Leave the destination in control when no explicit SSE mode was requested |
| Legal hold lost or encoded as user metadata | Use actual Object Lock headers and retain full retention timestamp precision |
| Zero LastModified, missing destination version, empty event identity | Bind responses/events to the actual committed destination write, not proxy time or a later unversioned HEAD |
| Undefined raw SSE-C replica COPY | Return 501 NotImplemented before creating the destination; ordinary COPY with the source customer key is a separate supported operation |
Issues: #99, #100, #158, #166–#171. A remote write may have committed before response validation fails; inspect the actual destination version before retrying a versioned copy. See the federated copy contract.
Object Lock, tags and replication recovery
Independent lock and tag ordering
- An ordinary write supplying only legal hold no longer suppresses bucket default retention; ON and OFF do not substitute for retention selection, and explicit timestamp precision is retained (#165, #177).
- Empty-value and timestamp-only retention/hold removals participate in ordering, preserve newer destination state and no longer cause endless retransmission (#111, #117, #129, #131).
- Replica writes reconcile authoritative Object Lock state across pools, extending the earlier single-set boundary (#133, #178).
- Pool reconciliation carries tags with their revisions; SSE-KMS metadata COPY and SSE-C rotation retain tag timestamps. Local PUT/DELETE tagging advances revisions monotonically, and empty tags are ordered deletions rather than missing state (#189, #193, #196).
Both replication endpoints need the fixes to respect empty-tag tombstones. Explicit resync/heal adds a metadata COPY for objects with recorded tag revisions; destination default KMS can make that operation rewrite data. Tag-filtered rules still evaluate target eligibility using the post-deletion empty tags. Missing or malformed historical revisions are not reconstructed. See Object Lock ordering and replicated tag ordering.
Delete markers, MRF and resync
- Correct single-object marker-purge classification and retain marker identity/retry state through workers, scanner/heal and persistent MRF recovery. Accept a relevant 405 marker response only when bucket, key, version and modification time match. Normalize purge audit status from COMPLETE to COMPLETED (#153, #162, #196).
- Expose bounded MRF queue drops through administration, metrics and deduplicated logs, and align object/delete worker allocation. This does not turn the bounded queue into a lossless unbounded queue (#152).
- Persist final counters before publishing resync Completed; count success by the actual replication outcome rather than the existence of a target object (#136, #139).
- Dispatch only to the resync target ARN. Give each resync ID an independent cancellation context, drain blocked dispatch/walkers/workers, and prevent older work from overwriting terminal state (#141, #137).
Exact-version purges and queued creation
eb4f5e5b3 stops a pure version purge from creating markers on drives that lacked them. Missing-version retries require write-quorum absence; purge results combine removed and reliably absent replies. Receivers resolve the addressed version across pools, and marker healing preserves stored replication and purge metadata.
254b19ac0 rereads queued marker creations under the replication lock. A missing version, non-marker version or version under purge makes the task stale; an uncertain read is retried through MRF. The check covers tasks queued by the DELETE handler, GET/HEAD/LIST healing, scanner and MRF.
358ab38fb reports a delete marker only for a stored marker, preventing a data version awaiting purge from incorrectly returning x-amz-delete-marker: true and producing the wrong removal event.
Remaining limitations: creations already in flight, replays from another site, and minority residue after a crash following a majority-acknowledged purge remain tracked in #217. This is not a persistent receiver-side purge fence.
Replica metadata normalization
Restore only six replication-specific fields after ordinary metadata extraction, preventing transport-only aws-chunked from becoming stored Content-Encoding. Trusted Snowball entries no longer inherit ordinary metadata from the outer archive (#185, #194).
These changes do not scan/rewrite existing objects or recover lost tags, lock history or every unfinished purge. Use the read-only audit tool and runbook to inspect exact versions, mixed encodings, ambiguous findings and controlled recovery rather than running bulk self-COPY.
Bucket configuration, caches and startup
PR #180 completes source-state convergence for #77. Policy, Tags, SSE, Quota, Versioning and Object Lock read/compare/mutate/save under the same metadata.lock. Peer events retain source time; local changes advance monotonically past retained time. Equal-time deletions win, and stable content keys resolve other conflicts.
- Dedicated events, bulk, import, initial synchronization and heal use the same rules. Omitted fields leave existing settings unchanged; empty Versioning/Object Lock events still do not delete those settings.
- Canonical policy JSON allows accepted NotAction/NotResource policies to be retrieved/exported. Statement/set order may change without changing authorization.
- Empty Policy retains deletion semantics. Zero Quota remains a live document imposing no capacity limit, rather than silently becoming a different deletion state.
- Missing historical bucket Created values can be recovered from physical information and persisted. Adoption does not turn empty defaults into deletions, and unknown/invalid/unreachable sources receive bounded diagnostics.
MINIO_SITE_REPLICATION_METADATA_TOMBSTONESis off by default. Upgrade every participating site before enabling it consistently and restarting, exposing empty-payload Tags/SSE/Quota deletion times to heal/initial sync. Missed-deletion convergence remains limited while it is off.
Historical receive-time pollution is not automatically rewritten, and rolling downgrade across the new persistent IAM state remains unsupported. See bucket metadata convergence.
Other fixes close remaining bucket deletion/update/lifecycle/cache publication races, remove the reload guard that mistook the maximum field timestamp for a whole-record version, and preserve per-bucket CORS through metadata export/import (#105, #112, #127, #156, #164).
#116 adds startup-readiness documentation and Linux restart evidence. Verify readiness with signed PUT/GET requests at each coordinator in addition to health probes.
Multipart discovery, pagination and cancellation
PR #198 adds durable discovery; #213 restores conservative defaults and rollback compatibility. The final default in 20260916 is legacy. (SILO 20261018 4733e1423 subsequently retired MINIO_API_MULTIPART_LISTING=strict and kept legacy cache/exact-key listing as the sole public mode; see the latest decision.)
| Behavior | Default legacy | Explicit strict |
|---|---|---|
| Discovery | Released exact-key/node-cache behavior, retaining known defects | Scan durable uploads across pools/sets/drives and verify quorum |
| S3 semantics | No promise of repaired prefix, delimiter or complete pagination | Global prefix, delimiter, ordering and markers; continue after the returned marker’s upload completes or is canceled |
| Legacy uploads | Preserve existing availability | Diagnostic 503 for missing durable bucket/key identity; never fake a successful empty page |
| Abort | Existing read quorum, best-effort cleanup and pool ordering | Majority deletion acknowledgements in relevant sets; retry minority remnants and do not mask observed-remnant deletion failures with empty drives |
New writers record reserved bucket/key identity in the existing quorum-written xl.meta and remove upload-only fields on completion. There is no secondary index or directory migration. The shared maxUploadsList request limit decreases from 10,000 to 1,000 in both modes. This bounds the accepted page-size parameter; the legacy cache path can still return more entries because it ignores pagination.
Enable strict only through process environment MINIO_API_MULTIPART_LISTING=strict and restart; mcli admin config set is not supported. Invalid values diagnose and fall back to legacy while retaining other API settings. First upgrade every writer, drain legacy uploads, run the read-only GET /minio/admin/v3/multipart-preflight endpoint with admin:StorageInfo, and validate capacity. Unidentified legacy uploads can block strict lists for other buckets.
Each process admits two scans; each scan has 16 identity workers, four metadata workers, a 30-second context and a 100,000 aggregate directory-entry budget (counted across drives, so 100,000 directory entries rather than 100,000 unique uploads). Every page rescans persistent state. Exhaustion or uncertain coverage returns 503 rather than a successful partial page. A temporary 10,000-upload/four-APFS-volume experiment took 18.7 seconds for one page, missing the provisional five-second target.
Both modes fix wrong-key/bucket Abort evicting a valid upload’s cache and peer notification cancellation after unlocking. HTTP retains idempotent 204 for an absent upload. Late physical creation writes can still restore an upload after acknowledged cancellation; no durable creation fence was added. #79 therefore remains open. See the enablement and rollback contract.
Cumulative Console 2.4.1 improvements
Beyond sharing and password permissions, the embedded selection includes these post-20260903 changes:
- Cursor browsing: ListObjectsV2 pages of 50/100/250/500/1000 entries, with explicitly page-scoped filtering, sorting and selection; no full-bucket count scan. Failed page changes preserve the current listing, cancellation releases list workers, and rewind discloses row/time limits. Empty/null versioning responses no longer crash the list.
- Exact operation identity: retain bucket/key/version for deletion and constrain directories to their trailing-slash prefix. Lifecycle edits preserve current/noncurrent, date-based and unexposed settings, read legacy prefixes and remove stale filter representations. Replication-rule deletion retains shared targets and uses configuration deletion for the last rule. Governance bypass is explicit and off by default for authorized users, including object-only retention.
- Capabilities and sessions: gate Create Bucket on s3:CreateBucket; preserve sessions after background transport failures; URL-encode Watch filters and preserve structured Inspect errors without consuming bodies twice.
- Embedded compatibility: restore loopback TLS login, apply embedded local-proxy trust only to an explicitly initialized TCP loopback peer, retain standalone trust-none defaults, and forward all four WebSocket limits instead of incorrectly capping public browsing at eight connections per source (Server #108, #147, #148).
- Streaming ZIPs: use a backpressured, cancellable file writer where supported and native download management elsewhere, without buffering whole archives in JavaScript. Native handoff cancellation belongs to the browser. Show percentages only with reliable length and recommend mcli above 5 GiB or when size is unknown.
- Recovery/localization/accessibility: recover from malformed routes, persisted preferences and render errors, and complete English/Chinese labels, icon names, sign-out text and keyboard tooltips. Shared dropdowns no longer select a stale hovered item after a quick click/touch, or select another value when clicking a disabled item.
- Standalone packages: improve reproducibility, immutable tooling/browser inputs, SBOMs, signatures and provenance. Preserve minio-console.service, console-user and /etc/default/console; move certificates to /etc/silo-console/certs with explicit migration instructions. Installation does not automatically restart the service.
References: Console #47, #50, #57, 2.4.1 notes.
Cumulative mcli 20260916 improvements
The bundled client includes both 20260913 and 20260916 changes:
- Pipe honors applied global quiet/JSON settings, avoids progress frames on redirected output, preserves explicit checksums for empty input and streams nonempty input. MC_* booleans accept on/off/enabled/disabled.
- SQL/S3 Select, transfers, mirror permission errors, legalhold, partial recursive retention failures and mv source-deletion failures return nonzero. Mv waits for deletion results without rolling back successfully copied destinations.
- Per-object mirror permission errors do not stop later objects; finite failed jobs exit 1 and do not print an ordinary successful completion summary. Explicit JSON summaries report failure, and very short transfers retain finite speed values. Existing listing/watch cancellation-retry boundaries remain.
- Mirror –remove –watch preserves targets when historical source versions are deleted. Administrative dry runs never fall back to a real restart; non-TTY results remain reliable.
- Empty retention durations and invalid find –regex expressions report errors instead of panicking; JSON usage errors retain valid output contracts.
- The SDK detects CopyObject errors embedded in HTTP 200, preventing mv from deleting its source after false copy success. This applies when the client selects CopyObject, normally below 64 MiB or with –disable-multipart, while the default large-object path uses ComposeObject.
Scripts must inspect the final exit status rather than initial success messages or progress bytes. Skip-errors does not turn an observed failure into exit 0; some retention JSON failures may lack readable error text. Commands, aliases, MC_* variables, configuration format and package naming remain compatible.
HTTP, TLS and observability
- Absolute header deadline: #196 wires –read-header-timeout / MINIO_READ_HEADER_TIMEOUT into the server and prevents small reads from indefinitely extending HTTP/1 header parsing. The default is 30 seconds. Bodies retain rolling idle timeouts. Shorter values constrain TLS handshake reads; HTTP/2 retains its independent behavior. TLS writes and handler CPU/storage waits do not gain a new total deadline. Negative values disable the header-specific limit and are not recommended. The design responds to #183.
- Go TLS defaults: remove eight explicit Server curve overrides so listeners, nodes, identity providers, replication and etcd honor Go 1.27 defaults and applicable GODEBUG controls without weakening certificate/hostname checks (#164). This changes what SILO sends and accepts on the wire. With no GODEBUG set, the offered group list becomes Go’s current default, which additionally includes
SecP256r1MLKEM768andSecP384r1MLKEM1024; withGODEBUG=tlsmlkem=0set, that switch now actually takes effect — process-wide, and on the inbound listener as well as outbound connections. #154 is closed, though the reporter’s 2026-09-17 retest on this release still failed; total discovery/JWKS timeouts remain separate work. See TLS/OIDC and pinned TLS parameters and handshake compatibility. - CPU metrics race: locking around both map levels read by loadCPUMetrics prevents concurrent resource updates and Prometheus scrapes from terminating the process. The repair entered main through #214. Separate #215 closed as superseded, and issue #210 also closed.
- Quota/logger metrics: read the effective quota field, make AddSystemTarget idempotent and serialize logger initialization/console-target publication, avoiding duplicate collectors after log subscriptions (#106, #150, #132, #151).
Dependencies, delivery and maintenance
- Upgrade amqp091-go 1.10.0 to 1.14.0, including bounded malicious-frame allocation; SILO reachability requires an AMQP notification target (#176, #177).
- JWX 3.3.0 escapes custom JSON field names; strfmt 0.27.2 repairs Go 1.27 hostname validation; LZ4 4.1.30 fixes reader, partial-read and concurrency behavior, with Snowball/S3 Select integration regressions.
- Cumulative upstream SDK changes include configurable upload limits, streaming Content-Type signing, caller TLS trust for RDMA, consistent listing checksums/optional restore status, and the HTTP-200 CopyObject repair above. See the upstream adoption record. Console also updates React Router to 7.18.4 and retains js-yaml/smol-toml denial-of-service fixes with rebuilt embedded assets.
- Refresh Go x/* dependencies while retaining Go 1.27.1, the NetBSD go-systemd 22.6.0 replacement and tablewriter compatibility selection.
- Refresh container base digests and build static curl 8.22.0 from verified source on both Linux architectures. Pin published mcli 20260916 archives and SHA-256 values in the installer and image.
- Pin GoReleaser 2.18.1 and action v7.2.3 by full commit; dependency-only changes now trigger Test Release Pipeline. Include Helm Chart 7.0.3 in the tag with the new Server/client defaults; there is no separate chart publication.
- Fix conditional PUT and tag-replication test-fixture races, restore valid integration credentials, and add CPU/tag/multipart/S3 Select race coverage.
- Maintain rebrand guards, AGPL-3.0-or-later/DCO/no-CLA contribution rules and human contribution records; refresh README repository cards daily.
- Consolidate formal product, upgrade, security and design documentation on this site. Retain repository entry points and necessary compatibility/testing documents; move temporary investigations out of public source and retain historical evidence through immutable links.
September 4 Helm image-tag correction, mcli archive pinning and recovery container publication completed the previous release’s delivery.
Withdrawn experiment
GET-frequency-based cross-pool ILM tiering #60 was merged during this cycle and removed by #188. It never shipped in Server 20260903 and is not a feature of this release. The tracker, mover, scanner hooks, dedicated configuration/XML actions and metrics are removed. Ten old ILM keys are accepted but ignored; v8/v9 usage caches are readable and v8 is written without retired fields.
Intermediate development deployments should remove access-only rules and retain ordinary actions in mixed rules. Ordinary expiry, remote tiers, rebalance and decommission remain; independent multi-pool lock, conditional-delete, write-coordination and shared-tier protections remain. See migration.
Upgrade and recovery requirements
- Freeze final identities. Verify Server, Console, mcli, packages, images and Chart separately. Official image names are docker.io/pgsty/silo, docker.io/pgsty/mc and docker.io/pgsty/silo-console.
- Migrate password denies and compare original policies. Preserve CreateUser and ChangeMyPassword in the same Deny when the old combined restriction must remain. Recover previously lost clauses from original sources.
- Use coordinated maintenance. Pause IAM writes and isolate unknown offline sites. Upgrade every participating node/site, including nodes sharing IAM without site replication. Mixed old/new access to one IAM backend and rolling downgrade are unsupported.
- Save complete recovery points. Back up IAM storage and required encryption material; mcli admin cluster iam export omits deletion history. Stop relevant processes before restoration so cached peer acknowledgements are not reused.
- Verify every endpoint. Old revoked credentials must fail; new users, service accounts and newly issued STS credentials must work at every node. Check signed PUT/GET, bytes, versions, tags, retention and replication convergence before reopening traffic.
- Reconcile revocations after rollback. An old snapshot loses later revocations. Reconcile each restored site and test every endpoint; use differently named replacement identities/reissued credentials when needed. Pre-restore STS continuity is not promised. Do not delete tombstones online or convert only live records to force an old binary to start.
- Enable optional behaviors independently. Metadata tombstone export requires all participants upgraded. Strict multipart additionally requires all writers updated, legacy uploads drained, preflight and capacity validation. Ordinary upgrades may keep default legacy mode.
- Handle development-build remnants. If an intermediate build persisted api multipart_listing, back up configuration and use
mcli admin config reset ALIAS api multipart_listingto remove only that key, retaining other settings. Do not reset the entire subsystem or replay history containing the key. Follow the separate access-tiering retirement guide if that experiment was used. - Check TLS-inspecting devices on outbound paths. Handshake contents changed with the Go 1.27 toolchain in 20260903 and again here. An SSL-inspection NGFW, SASE gateway or IDS/IPS — or a JA3/JA4 client-fingerprint allow-list — can reset identity provider, webhook, remote tier or replication connections with
connection reset by peerwhilecurlfrom the same container succeeds. After upgrading, verify OIDC discovery and JWKS, audit and notification targets, remote tiers and replication endpoints. Remedies and their trade-offs: pinned TLS parameters and handshake compatibility. - Audit historical data separately. Inventory unreadable SSE-C, wrong part sizes, tags/locks, aws-chunked pollution, past erroneous overwrites and pending purges. Validate approved repairs on copies.
Runbooks: IAM upgrade/recovery, historical replica audit, password migration and multipart contract. While object and erasure layouts are unchanged, the new persistent IAM state prevents rolling downgrade to an older Server binary.
Known limitations and remaining work
| Item | Boundary / disposition |
|---|---|
| #79, multipart listing | Strict exists as an option; legacy remains default. Production capacity, durable late-creation fencing and full delivery acceptance remain open. |
| Batch conditional deletion | XML ETags provide no protection; empty headers, recursive extensions and policy enforcement are outside the implemented contract. |
| Streaming SigV4 | Seed operation-header coverage remains open beyond the ordinary and presigned SigV4 repairs. |
| IAM/configuration history | Lost revisions cannot be reconstructed automatically; clock, membership and retained-history costs remain. |
| #197, two-drive EC documentation | Open. |
| Other proposals | #20 Renovate, #30 Operator, #40 notification target hot reload, #61 KES/OpenBao planning and #160 image processing remain open outside this release. |
| Historical migration tags | Previously lost tags are not recovered automatically; audit tag-dependent policy and lifecycle rules separately. |
| #217 / #218 | In-flight/cross-site creation, crash residue and restart-time listing omissions remain unresolved; see the scopes above. |
Validation and release status
Released source 2a4d51406b7e passed Go CI and the Test Release Pipeline. The Release build and Finalize completed. The GitHub release provides 36 assets, including six-platform archives, Linux packages, SBOMs, checksums and attestations.
Console v2.4.1, mcli 20260916 and pkg v3.14.1 have independent published releases. At the 2026-09-17 recheck, anonymous token requests for standalone docker.io/pgsty/silo-console returned HTTP 401, so public pulls are not confirmed. Use Console’s GitHub binaries or packages. This standalone image limitation does not affect the source selected for Server embedding.
Issue index and acknowledgements
Fixed or completed within the described scopes: conditions #10/#133/#144/#199; federation #99/#100/#158/#165–#171; bucket metadata #77/#105/#112; encryption/attributes #107/#109–#115/#117–#120; embedded Console #108/#147/#148; replication recovery #136/#137/#139/#141/#152/#153/#185; runtime/metrics #106/#116/#150/#154/#176/#183/#210. The pool-migration tag repair is recorded as a direct main commit; #79 remains partially implemented and open. The sections above link the individual records and limits.
Thanks to reporters, patch authors and reviewers, particularly @mrjavadseydi (quota, multipart and tiering exploration), @nikitapogromsky (logger idempotency), @Aeirx (federated legal hold), @magicxor (early conditional DELETE proposal), @julienlau (purge/MRF), @chodorenko (replica encoding), @orenyomtov (signed headers), @jiri-pejchal (Console sharing), @AEGEGE (HTTP report), @aschyolkin (CPU race), and @vampywiz17, @kh0mka and other reproduction/feedback contributors.
Original PRs #12, #125, #155, #184, #187 and #195 were superseded by later implementations. See the full contribution record.
2.12 - mcli 20260916 Released
Release: RELEASE.2026-09-16T00-00-00Z · Package: 20260916000000.0.0 · Source: e952aa78f10a.
Changes since 20260913
- Adopt silo-pkg v3.14.1, upstream minio-go v7.3.1-0.20260915093545-32e1f32cb176, and JWX v3.3.0. CopyObject retries and propagates S3 errors embedded in HTTP 200 responses, so a failed copy cannot authorize
mvto delete its source. JWX correctly escapes JSON field names. - Report
mirrorpermission failures, including unreadable sources, rejected writes and failed local destination removals. Later objects continue; a finite mirror with failures exits 1. These per-object permission errors do not require--skip-errorsand do not restart an entire watch scan. - Preserve the existing cancellation/retry path for listing and watcher failures. Persistent listing permission denial can still cause repeated scans; this release does not redesign that retry policy.
- Suppress normal final success statistics after a failed mirror. Explicit
--summaryretains statistics: JSON usesstatus: failure; text error diagnostics remain on stderr. Transfers completed within one clock tick no longer produce infinite JSON speeds. - Return failure for legalhold set/clear errors, partial recursive retention failures, and moves whose source deletion fails.
mvwaits for deletion results; successful copies are not rolled back. Retention reports each failed object once instead of duplicate or misleading diagnostics. - Reject empty retention durations and invalid
find --regexexpressions with normal CLI errors instead of panics.
Automation and upgrade notes
Affected failure paths that returned 0 now return 1. Check final exit status and error records. For mv, a deletion failure may leave both the source and the successfully copied destination; inspect both before retrying.
Configuration format, aliases, MC_* variables, commands and package naming remain unchanged. The password-policy split introduced by pkg v3.14.0 is unchanged in v3.14.1. See the component matrix and password-policy migration.
SILO is the maintained administrative integration target. Compatibility with unmodified upstream MinIO and other S3 services remains best effort.
Delivery and validation
The release contains 19 assets: six Linux/macOS/Windows archives for amd64/arm64, six Linux RPM/DEB/APK packages, one archive checksum manifest and six package checksum sidecars. Archives, the manifest and DEB/APK packages have verified Sigstore build attestations bound to this signed tag and source commit. RPMs carry the PGSTY GPG signature (9592A7BC7A682E7333376E09E7935D8DB9BD8B20).
The final release commit passed Go tests on Linux, macOS and Windows with the race detector, Linux 386 tests, cross-compilation, vulnerability scanning and the release packaging pipeline. Pre-release acceptance also exercised both the published SILO 20260903 and fixed Server source 3c26a8b0b5bd, Object Lock and permission failures, CopyObject error injection, and both-architecture RPM/DEB/APK installation and upgrades. No reachable or imported-package vulnerability was found; an unused OpenPGP module-only advisory remains.
CI: VulnCheck · Test Release Pipeline · Crosscompile · Go · Release · Docker
The docker.io/pgsty/mc release and latest tags resolve to the verified Linux amd64/arm64 manifest:
Download and installation · Client compatibility · Changes since 20260913.
The HTTP-200 CopyObject repair applies when mcli selects a single CopyObject: objects smaller than 64 MiB or explicit --disable-multipart. Larger copies use ComposeObject.
See pkg 3.14.1 for the precise JWX/GHSA-4cf7-xm37-g63h scope. A failed --json mirror without --summary emits no trailing statistics record; --skip-errors does not turn recorded failures back into exit 0. Some retention JSON failure records carry status: failure with an error object serialized as {}; consumers must not require a human-readable message.
2.13 - mcli 20260913 Released
Release: RELEASE.2026-09-13T00-00-00Z
· Package: 20260913000000.0.0
· Source: 4f609a4d.
Changes since 20260903
- Preserve historical destination versions under
mirror --remove --watch. - Keep service-restart dry runs from executing a restart and make noninteractive behavior explicit.
- Return failure for transfer and S3 Select errors, honor explicit checksums on
empty uploads, and retain
pipeJSON output when quiet mode is enabled. - Accept on/off boolean environment values and repair CLI/JSON behavior across supported platforms.
- Adopt pkg v3.14.0 and upstream SDK
v7.3.1-0.20260910142817-60bd07042d49, refresh Go x/* and UBI dependencies, and keep Go 1.27.1. NetBSD retains the go-systemd v22.6.0 portability pin.
The package retains the policy Deny/NotResource and bounded wildcard repairs from pkg v3.13.3. Correctly preserved denies may now reject requests that relied on lost clauses. Recover already-lost clauses from the original policy source.
The new pkg also changes password capabilities. Read the password-policy migration before a coordinated Server upgrade. The matching password UI shipped in Console v2.4.1; see the component matrix for the coordinated upgrade.
Delivery and validation
The immutable release contains 19 assets: six archives for Linux, macOS and Windows on amd64/arm64, RPM/DEB/APK packages for both Linux architectures, checksums and package checksum sidecars. Archives and the checksum manifest have verified build attestations; RPMs carry the PGSTY GPG signature. Native macOS arm64 version and Go build metadata were checked against the released pins.
The release and latest tags of docker.io/pgsty/mc resolve to the same verified
amd64/arm64 manifest:
Container publication completed; both architecture images were pulled and their version output checked. Go, cross-compilation, vulnerability and Test Release gates passed before tagging. No reachable or imported vulnerable package was reported; unused OpenPGP retains module-only GO-2026-5932. Compatibility with upstream MinIO and other S3 endpoints is best effort; SILO is the maintained administrative integration target.
Download and installation · Compatibility notes · Complete source diff.
pipe streams nonempty stdin until EOF; empty input uses a regular zero-byte PUT instead of unknown-size multipart. Explicit checksum options reach the upload path. Quiet/non-TTY execution keeps progress disabled while retaining JSON results for both empty and non-empty inputs.
2.14 - silo-pkg 3.14.0 Released
v3.14.0 was published on 2026-09-13 from commit 827f8109 under the module path github.com/pgsty/silo-pkg/v3.
Password-policy migration
This release updates authorization semantics for self-service password changes:
Policy.IsAllowedActionsnow allowsadmin:ChangeMyPasswordby default unless explicitly denied, whileadmin:CreateUserstill requires an explicitAllow. On matching Server and Console builds, denyingadmin:CreateUseralone no longer blocks a caller from changing their own password; denyingadmin:ChangeMyPassworddoes.- The built-in
readonlypolicy drops its previousadmin:CreateUserDenyrule (so an explicitadmin:CreateUserAllowgranted by another attached policy can now take effect). The newconsolereadonlypolicy adds bucket listing and uses the same permission split. Neither built-in policy grants user-management permissions on its own. - Existing custom policies and user-level policy overrides stored in the cluster are not rewritten automatically.
To keep the previous restriction that blocks both user administration and self-service password changes, include both admin:CreateUser and admin:ChangeMyPassword in the same Deny statement (preserving its original resource scope and conditions) before upgrading and throughout your rollback window—older Server builds do not enforce a standalone admin:ChangeMyPassword deny on this endpoint. See the full password-permission migration guide.
September 16 follow-up: Console v2.4.1 ships the matching password UI alongside silo-pkg v3.14.1 and mcli 20260916. See the component version matrix for coordinated upgrade requirements.
Dependencies and verification
- Upstream
minio-goupdate: Pinned tov7.3.1-0.20260910142817-60bd07042d49, bringing configurable upload size limits, streamingContent-Typesigning, caller TLS trust propagation for RDMA, consistent listing checksums, and optional archive-restore status parsing fixes. - Toolchain and dependency refresh: Refreshed
golang.org/x/*modules while keeping the Go 1.26 minimum requirement, Go 1.27.1 build toolchain, and existing exported Go signatures unchanged. Thego-systemd v22.6.0replacement is retained for NetBSD cross-compilation compatibility. - Testing and vulnerability scanning: Passed full
-racetest suites on Go 1.26.8 and 1.27.1, lint checks, LDAP configuration validation, and Go module proxy / checksum database resolution. Scanning withgovulncheck 1.8.0found no imported vulnerable packages (only the module-levelGO-2026-5932entry for unused OpenPGP subpackages remains).
This release retains the Deny/NotResource policy serialization fix and bounded wildcard matcher from v3.13.3 (policies that already lost statements under earlier versions must still be restored from their original source). The github.com/pgsty/silo-pkg/v3 module-path migration introduced in v3.13.0 is complete across all four maintained components.
2.15 - SILO Console 2.4.0 Released
v2.4.0 was published
on 2026-09-08 from c103d08e.
Released behavior
- Object-browser directories load 100 entries per page by default, with choices of 50, 100, 250, 500 and 1000. Each page uses one S3 continuation-token request. Sorting, filtering and selection apply to the displayed page, not a complete directory scan; there is no unbounded “all” mode.
- Version deletion stays bound to the selected object and directory deletion to its prefix. Lifecycle edits preserve independent actions and unexposed settings; replication rule deletion is saved consistently.
- Transient network failures preserve sessions. Empty versioning state, malformed sidebar preferences and stale dropdown selections are handled.
- Embedded Console restores loopback proxy behavior when Server explicitly enables the policy; standalone trust defaults remain unchanged.
- Dependencies include pkg v3.13.3, MC source
v0.0.0-20260908140805-c8aa5d25a63a, and upstream SDK0e78d3f18efe. This release consumed MC as source and did not itself publish a new mcli tag.
Correctly preserved Deny/NotResource clauses may reject requests that relied on old deduplication defects. Restore already-lost clauses from the original policy source. Embedders must copy the MC replacement from the tagged README.
Later release
Console v2.4.1 adds restricted shared downloads, the password-permission split, streaming multi-object ZIPs, browser recovery and verifiable release artifacts. It selects pkg v3.14.1 and mcli 20260916. v2.4.0 uses the earlier browser-memory ZIP path. Read the password migration before upgrading.
The exact v2.4.0 source passed the complete Console CI matrix. See the tagged changelog, object-browser guide, and release assets.
2.16 - Silo Console 2.2.0 Release Notes
Published: 2026-08-26 · Version: v2.2.0 (7dc4258a6) · Repository: pgsty/silo-console
SILO Console 2.2.0 adds a bounded text preview for logs and structured text, tightens download error handling so truncated archives cannot pass as complete, fixes table-action permission gating, splits user status updates from group membership edits, and validates database notification DSNs and IAM policy documents before writing them to the server.
Underneath, the dependency stack moves to Go 1.27.0 and onto the maintained SILO packages: pgsty/silo-pkg 3.12.1 replaces upstream minio/pkg, pgsty/mc replaces upstream mc as a library, minio-go moves to 7.3.0, and the etcd client line moves to 3.7.1, closing CVE-2026-73500.
The final change set since v2.1.1 spans 33 commits touching 176 source files (+8,794/−2,057 lines, excluding regenerated embedded frontend assets).
Why 2.2.0 instead of 2.1.2: this release switches the shared policy/certificate package provider, crosses an etcd client minor version boundary, splits a REST route, and changes folder-download failure semantics.
Text Preview for Logs and Structured Text
The object browser can now preview .log, .txt, .json, and .xml objects as literal text. The implementation is deliberately paranoid, because “render arbitrary bucket content in the admin UI” is an XSS invitation:
- Bounded by construction. The request carries
Range: bytes=0-1048576; the response is streamed into a fixed buffer with a hard cap, andContent-Range/Content-Lengthare strictly validated against what was actually read. An object over 1 MiB reports too large — it is never partially rendered as if complete. - Text or nothing. The bytes must decode as strict UTF-8 (
TextDecoderwithfatal: true) and must not contain NUL; anything else reports not previewable instead of rendering mojibake or binary junk. Eligibility is an exact allowlist — the four extensions plus the precise MIME typestext/plain,application/json,application/xml,text/xml— and.html/.htm/.xhtmlare explicitly excluded even when their metadata claimstext/plain. - No active document. Content renders into a single DOM text node — never
innerHTML, never an iframe, never a JSON-to-HTML transform. A regression suite feeds it HTML, SVG, and XML payloads and asserts they stay inert text. - Race-proof. Switching quickly between objects invalidates in-flight previews by generation, so a slow response for the previous object cannot paint over the current one. Cancellation and retry are first-class states.
- Anonymous-friendly. Public/anonymous object pages get the same preview through the anonymous request path, without triggering any credentialed API call.
Alongside the new preview type, the existing preview plumbing was corrected: empty objects preview as empty instead of erroring, appended logs re-preview at their new length instead of the stale listed size, and metadata races when flipping between objects are gone. The preview and share dialogs now receive the object’s real size (92e8f4e65).
Download Integrity and Progress
The download pipeline — progress reporting, single objects, folders, archives, and byte ranges — enforces strict completion and error reporting.
Indeterminate progress when size is unknown
A missing, invalid, or contradictory total no longer becomes a fabricated percentage: the progress indicator stays indeterminate until the real total is known. Zero-byte objects are normalized deliberately instead of falling into the unknown-total path, and abort/cancel are terminal states — a late progress event cannot resurrect a cancelled download. Download requests settle exactly once, JSON error blobs are parsed safely, and generated object URLs are revoked.
Streaming folder downloads
Single-folder downloads now use the browser’s native streaming download path instead of accumulating the entire ZIP in JavaScript memory — a multi-GB folder no longer risks tab death. One consequence of the handoff model: the console’s transfer manager reports the folder download complete once it hands the stream to the browser (a toast says so), and the browser’s own download UI takes over from there — cancelling in the console after handoff does not stop the browser-side transfer. Multi-selection downloads still ride the existing POST response and therefore remain an in-memory Blob; changing that requires a separate API decision and is out of scope here.
ZIP integrity — a deliberate behavior change
This is the one change most likely to be noticed as “downloads broke”. In 2.1.1 and earlier, a folder/ZIP download that failed to read some objects silently skipped them and delivered an HTTP 200 archive missing files. In 2.2.0 any per-object failure — list, stat, read, entry creation, close, or copy — aborts the archive: a clean error if nothing was sent yet, an aborted connection mid-stream, so a truncated ZIP can never pass for a complete one.
The practical consequence: a user whose policy grants List on a prefix but GetObject on only a subset of it — a common IAM setup — previously received a partial archive; now the download fails at the first denied object. That was silent data omission, and 2.2.0 treats it as the bug. Download exactly what you can read, or scope the folder download to a readable prefix.
Byte ranges and status codes
- Malformed or unsatisfiable non-empty
Rangeheaders now return416withContent-Range: bytes */Ninstead of a 500. Range parsing is stricter than Go’s lenient default: signs, embedded whitespace,bytes=-0, and empty range elements are rejected. - A range request against a zero-byte object returns an empty 200 instead of a 500 — this was the empty-object preview bug.
- Failures from the lazy object
Statnow surface the real S3 status (403, 404, …) instead of a blanket 500. 206responses now setContent-Lengthbefore the header flush, so partial responses carry correct framing.- Object sizes are now always serialized — REST and WebSocket listings report
"size": 0for zero-byte objects instead of omitting the field, and the UI displays0 B. This is the ground truth the honest-progress work stands on.
Version history
S3’s valid null version ID stays visible instead of being dropped, version counting filters prefix matches to the exact object, and history is retained after bucket versioning is suspended or disabled.
Permission Gates That Actually Gate
Row actions were never disabled
Every screen with a data table passed its permission predicate through a prop name (disableButtonFunction) that the table component no longer reads — so view/edit/delete buttons rendered enabled regardless of permission, on every one of the 10 affected screens (Users, Groups, Policies, IDP, webhook settings, bucket access/replication/lifecycle panels). The predicates are now wired to the prop the component actually honors, and a source-guard test fails the build if the dead prop name ever reappears. Server-side authorization was never affected — the buttons produced errors when clicked — but the UI now communicates permissions instead of lying about them.
Independent service-account capabilities
Access-key management previously keyed several UI decisions off one combined permission check. 2.2.0 derives four independent capabilities — List, Create, Update, Remove — in one module and applies them consistently across the self-service Account screen and the admin user-details screen:
- the service-accounts tab shows when you can list, the create button when you can create, row selection and bulk delete only when you can remove, and the edit pencil only when you can update;
- View is now genuinely read-only: all fields disabled, no submit path, Enter is inert — previously “view” opened the editable dialog;
- the session’s advertised Create Access Key capability is computed correctly for request-scoped policies: a
Denyconditioned onsvc:DurationSeconds(an expiry restriction that can only be evaluated once a concrete request exists) keeps the capability visible — including with wildcard admin actions — while unconditional or login-time denies now properly hide it. The server remains the final authority at request time. The previous code kept the capability visible for any conditional deny, which over-advertised it. - an OIDC create/list/get/delete integration regression now runs through the UI’s explicit-credential endpoint, including the expected self-update denial.
Validate IAM policies before writes
Named-policy and service-account policy writes now reject malformed documents and bare S3 resource ARNs with a client error before making an Admin API request. Historical policy reads stay permissive for compatibility, but an incompatible stored policy must be corrected before it can be saved again. The strict parser and resource checks live in the console’s write path instead of importing fork-only policy APIs, preserving the advertised upstream minio/pkg v3.6.1 source-build floor.
Anonymous pages behave anonymously
Anonymous object-browser pages no longer issue protected Object Lock/retention requests that could only produce Access Denied noise, and they gained the language (文/A) and dark-mode controls.
User Status and Groups Are Separate Operations
Toggling a user’s enabled/disabled status and editing their group membership were one combined PUT /user/{name} call that required both payloads and — through the combined permission gate — demanded admin:EnableUser merely to edit groups. 2.2.0 splits them:
PUT /user/{name}/status(new) changes only the status, validates the value againstenabled|disabled, and rejects the signed-in user’s attempt to enable/disable themselves. Failed toggles no longer leave the switch out of sync — the UI reflects the server’s answer, not an optimistic flip.PUT /user/{name}/groups(existing) is now the only thing group editing calls, and no longer requiresadmin:EnableUser.PUT /user/{name}survives unchanged on the wire as a deprecated compatibility endpoint for existing API consumers; an unknown status there is now a 400 instead of a 500.
See Compatibility for the API-contract details, including a caveat for regenerated Swagger clients.
Database Notification Forms That Emit What They Mean
The PostgreSQL and MySQL event-destination forms were rewritten around a shared DSN parser/serializer:
- Structured fields and the raw connection string are one state: the raw string stays authoritative until a structured field is edited, at which point a canonical DSN is rebuilt — libpq keyword/value quoting for PostgreSQL, IPv6-bracket-aware
go-sql-driverformat for MySQL. Mode switches no longer mangle manually entered strings. - Generated previews mask credentials; the mask can never reach the API payload (the payload always uses the raw connection value).
- Saving requires both a connection string and a table, and cleared values actually propagate.
- The server now rejects DSNs its own configuration grammar would corrupt — embedded newlines, values that would parse as sibling config keys (
table=…inside a password), unbalanced quoting — with a 400 before anything is stored, and without echoing the submitted secret back. - Generic password and token fields across notification targets (Kafka, Redis, MQTT, NATS, webhooks) render as password inputs, including environment-overridden values.
Accurate restart-required status
Configuration add, update, delete, and reset now honor the server’s actual restart-required answer instead of hardcoding “restart needed” (or worse, dropping it). The pending-restart flag is monotonic: once any operation requires a restart, later operations that don’t cannot clear it — only an actual service restart does.
Upload advisory
Browser uploads larger than 5 GiB get a non-blocking warning: the console uploads as one non-resumable request, and mcli with multipart upload is the right tool at that size.
Sessions, Metrics Auth, and i18n Hardening
- Sessions fail fast. An anonymous/empty session gets an immediate 401 instead of hanging on an empty-credential admin request. The console accepts the canonical
401and the legacy403invalid-session responses, and expired-session redirects are subpath-aware — a console served under/console/redirects within its base path. - Prometheus Basic auth works everywhere. The health check and the root-fallback probe now send Basic credentials (previously Bearer-only, so a Basic-auth Prometheus disabled every dashboard widget), Bearer tokens keep precedence, and every response body is drained so keep-alive connections are actually reused.
- Placeholder substitution is escape-proof by construction. Translated placeholder filling uses a one-pass literal formatter: values containing
$&,$1,$`, backticks, or braces stay literal, repeated placeholders all fill, and an AST source-guard test bans the unsafeString.replacepattern from ever returning.
Toolchain and Dependencies
Go 1.27 baseline
| Component | 2.1.1 | 2.2.0 |
|---|---|---|
go directive, build image, CI matrix |
1.26.5 / 1.26.x |
1.27.0 / 1.27.x |
golang.org/x/crypto |
v0.54.0 | v0.55.0 |
golang.org/x/net |
v0.57.0 | v0.58.0 |
golang.org/x/text |
v0.40.0 | v0.41.0 |
golang.org/x/mod |
v0.37.0 | v0.40.0 (closes CVE-2026-56864/-56865) |
golang.org/x/tools |
v0.47.0 | v0.49.0 |
The SILO forks
silo-pkg3.12.1 is the release dependency.pgsty/silo-pkgv3.12.1 was published on 2026-08-25 and the console pins it directly. It builds on the 3.12.0 release, which carries the policy resource-boundary hardening, condition-key lookup repair, LDAP and certificate-watcher fixes, and the Go 1.27 / etcd 3.7 baseline.- The
mclibrary moved to thepgsty/mcfork at a date-tagged pseudo-version. Import paths are unchanged. - The
requireline stays on upstreamv3.6.1deliberately. Go ignoresreplacedirectives in dependency modules, so a downstream module that requires this console resolves upstreamminio/pkg— and the SILOv3.12.1tag does not exist upstream. Requiring a real upstream tag keeps the console resolvable downstream; the replace applies the fork for the console’s own builds. The final CI verifies the public source-build surface against upstream v3.6.1. This is compile compatibility: a downstream build without its own top-level replace gets upstream behavior, not the fork’s SILO-specific IAM semantics. go-systemdis pinned back to v22.6.0: v22.7.0 usesCLOCK_MONOTONICon NetBSD, which doesn’t compile there; the pin holds until upstream ships the fix.
etcd 3.7.1 — client libraries only
All three etcd Go modules move together from 3.6.8 to 3.7.1, closing the TLS-listener denial of service GO-2026-6107 / CVE-2026-73500. etcd 3.7 removes legacy protobuf remnants and makes clientv3.New non-blocking — migrations that matter to consumers that construct clients or embed servers. The console does neither: its only etcd path is silo-pkg/quick operating on an already-created client with plain v3 Get/Put. This upgrades compiled client libraries only — it does not touch an operator’s etcd servers, cluster data, or deployment topology; a server upgrade to 3.7 still follows etcd’s own one-minor-at-a-time procedure.
Third-party maintenance stays conservative
The final release updates minio-go to v7.3.0 after a separate compatibility review, migrates its INI import path, and adds lifecycle-filter XML coverage. A console-side compatibility decoder accepts legacy AccountInfo tag payloads returned by older servers. Other accepted maintenance moves include jwx v2→v3 with httprc v3, go-openapi/swag/conv+typeutils 0.28.0, grpc-gateway 2.29.0, cheggaaa/pb 1.0.30, and go.yaml.in/yaml/v3 3.0.5. Larger unrelated go-openapi, pb/v3, compression, and test-library updates remain deferred.
On the frontend, the vulnerability workflow now covers pushes, manual runs, and development dependencies with immutable installs; vulnerable transitive resolutions were refreshed (fast-xml-parser 5.11, nanoid 3.3.18, @babel/core 7.29.7), and dead exports along with the unused http-status-codes dependency were removed. The fast-xml-parser 5.x transitive packages are development-only and are not bundled into the browser build.
Security Review
govulncheckat the release tree: zero vulnerable symbols reached, zero vulnerable imported packages. The Swagger build tool scans clean separately.- Closed by dependency moves: CVE-2026-73500 (etcd TLS listener DoS), CVE-2026-56864 / CVE-2026-56865 (x/mod verification).
- Still reported, still unreachable: the module-level GO-2026-5932 openpgp advisory — the console does not import
x/crypto/openpgp, and no fixed release exists. - The text preview was reviewed as an XSS surface (see above); its regression suite includes active-payload tests.
- The permission-gating fixes are UI-truthfulness fixes: server-side authorization was never bypassed in 2.1.1; the console simply displayed controls it shouldn’t have.
Compatibility
Nothing changes for deployment plumbing: no environment variable, configuration format, command, binary name, systemd unit, port, or embedded data layout changes. The release binary remains self-contained; Go 1.27.0 is a build-time requirement only.
HTTP API contract (console’s own REST API):
| Route | Change |
|---|---|
PUT /user/{name} |
Unchanged on the wire; now deprecated. OperationId renamed UpdateUserInfo → UpdateUserInfoLegacy, body model renamed to legacyUpdateUser (identical schema). Unknown status: 500 → 400. |
PUT /user/{name}/status |
New. Status-only body (enabled/disabled enum, 422 on violation), returns a user object populated with access key and status only — clients must not read group data from it. |
updateUser model |
Now status-only with enum — breaking for generated-spec consumers; the old shape lives on as legacyUpdateUser. |
| Object download | Error semantics changed: bad ranges 500→416 (+Content-Range: bytes */N), range-on-empty 500→200, Stat failures 500→real S3 status, ZIP failures silent-partial-200→visible failure, 206 responses carry Content-Length. |
| Listings & WebSocket | size always serialized, including 0. Additive. |
PUT /configs |
New 400 class for database DSNs the server’s config grammar would corrupt. |
| Config reset/delete | restart in the response now reflects the server’s real answer instead of always true. |
GET /session |
401 for empty-credential principals; the advertised Create-Access-Key capability is computed more strictly (see service accounts). |
If you generate client SDKs from swagger.yml: the UpdateUserInfo operation now points at /user/{name}/status with a status-only body. Code calling the generated UpdateUserInfo symbol keeps compiling but targets the new route; the old combined call is UpdateUserInfoLegacy. Audit call sites when you regenerate.
Behavior changes an operator may notice:
- Folder/ZIP downloads over partially readable prefixes fail instead of silently omitting unreadable objects (details).
- Range parsing is stricter than Go’s lenient default; degenerate range headers (
bytes=-0, empty elements) now get 416 instead of best-effort handling. - Database notification configs that only worked by accident (DSNs the config grammar mangled on the way in) are now rejected up front with a 400.
- The Create-Access-Key control is hidden for sessions whose policy unconditionally denies it (previously any conditional deny kept it visible).
- A pending restart-required indicator persists until an actual restart, instead of being clearable by a later unrelated config change.
- If a reverse proxy in front of the console compresses
/api/v1/…/downloadresponses, the text preview’s strictContent-Lengthverification will reject every preview as an error. The console itself only compresses static assets — leave API responses uncompressed at the proxy.
Known Limitations and Follow-up Items
Wiring the table disable prop exposed two pre-existing UI-guard defects where predicates compare a row object against a string:
- the “cannot delete the Default IDP configuration” row guard compares the row object against
"Default"and therefore does not disable the button (IDPConfigurations.tsx); - the “cannot delete an env-override webhook endpoint” row guard has the same object-vs-string mismatch (
WebhookSettings.tsx).
In both cases the server still enforces the restriction when the action is submitted. Additionally, the PostgreSQL DSN parser populates structured fields only from canonical key=value syntax (non-standard libpq DSNs display empty structured fields and rebuild from those fields if edited), and TestCafe/Playwright suites cover UI gating while live-server deny paths are covered by the integration suites.
Verification
The release decision combines local release-preparation evidence with remote gates run against the exact tagged tree, 7dc4258a6:
Local release-preparation gates — go build ./..., go vet ./... (plus -tags testrunmain), gofmt, golangci-lint, go test -race ./..., go tool swagger validate, govulncheck, TypeScript tsc, Playwright, Prettier, knip, release-tag cross-compiles for linux/amd64 and linux/arm64, and go mod verify.
Embedded-assets determinism — the frontend was rebuilt from source through the full pipeline (yarn build + embed optimization) and the result compared against the committed web-app/build: byte-identical, zero dirty files. The nine commits after the earlier 19047161f candidate changed Go compatibility, workflows, and browser-test timing, but not product frontend source or embedded assets.
CI, exact final tree — Workflow run 32888892876 reported 32 successful jobs and one explicitly disabled React-test placeholder. It covered lint, semgrep, Go and API tests, five cross-compile targets, Swagger drift, latest-MinIO source builds, distributed integration, site replication, hermetic SSO, the complete TestCafe permissions matrix, subpath-nginx, Playwright, and coverage. Vulnerability Check 32888899120 passed both jobs on the same commit.
Release pipeline — goreleaser run 32916237254 passed both jobs against v2.2.0 and published the public GitHub release.
Downstream contract — in a scratch tree with every replace removed, go mod tidy + go build ./... succeed against upstream minio/pkg v3.6.1, proving the fork replacement never leaks fork-only symbols into the public module surface.
Test-infrastructure repairs shipped in this cycle (so the gates above actually gate): the Docker-backed integration/replication/SSO suites are back behind the testrunmain build tag (a bare go test ./... no longer tries to start containers); the SSO gate is hermetic — no sudo /etc/hosts edits, no ad-hoc pip installs, pinned SILO image, own port, real teardown; the integration gate asserts the new 416 range semantics and stopped binding its PostgreSQL fixture to a host port; the browser gates run off-Linux by publishing fixture ports; a stale pre-fork “MinIO administrator” selector was fixed; state-mutating TestCafe suites are serialized; Playwright CI installs the committed lockfile immutably.
Release Artifacts
The v2.2.0 GitHub release publishes:
- six standalone binaries: Linux amd64/arm64/armv6, macOS amd64/arm64, and Windows amd64;
- nine Linux packages: DEB, RPM, and APK for amd64, arm64, and armv6;
silo-console_2.2.0_checksums.txt, plus the SHA-256 digest GitHub records for every asset.
Related Commits
16960f7ab— fix: keep unknown downloads indeterminate5968bb37d— chore(deps): align the SILO Go dependency stack288ab1240— fix: harden sessions, metrics, and translationsecf3bb492— fix: harden object previews and downloads902d9650d— chore: tighten dependency and test gates194c70c7a— build: prepare SILO Console v2.2.0927b44e26— fix: harden database notification formsda2191be9— fix: clarify console upload and secret limits6141c2445— build: refresh SILO Console v2.2.0 assets097e76155— chore(deps): bump the shared package fork to v3.12.099ca523d6— fix: split user status updates out of the combined user routef4097992f— fix: honor the server restart result for configuration changesf1280032a— fix: restore permission-gated table row actions24ce0af97— fix: keep request-scoped access key conditions visible in the console92e8f4e65— fix: pass the object size into the preview and share dialogs8f6fb3c78— test: make the SSO gate hermetic and pin it to a SILO release384a2cb95— build: refresh SILO Console assets and record the changesa73cda376— test: fix the integration gate’s stale range and host portcf5049c1d— test: make the browser gates runnable and fix a stale selector6fa19d857— fix: complete service account permission boundaries7e57771a4— build: refresh SILO Console assets57cfe7aa0— fix: restore downstream and browser release gates19047161f— test: stabilize permissions browser gatese37dec873— fix: validate IAM policies before writes28505ed23— chore: update minio-go to v7.3.016abb971e— ci: harden validation and release gates2ddfcd036— fix: accept legacy AccountInfo tag payloads31332bca9— fix: preserve policy source compatibility3a8251086— ci: allow permission tests to finishc159fff78— ci: serialize shared-role permission tests2e91cdf9a— test: wait for watch controls to become ready7dc4258a6— test: allow asynchronous UI controls to settle
Links:
2.17 - SILO Console 2.5: Interface and Administration Updates
SILO Console v2.5.1 is now released (commit fff1618d9abb) and ships embedded inside SILO 20261026. This note summarizes the full 2.5 release series since v2.4.2, combining the v2.5.0 interface overhaul and the v2.5.1 patch fixes.
GitHub release · Downloads · Full commit comparison
Navigation, overview, and metrics
The 2.5 interface adopts neutral light and dark themes, system typography, consistent information density, Lucide icons, clear keyboard focus states, and prefers-reduced-motion support. A mobile drawer and tablet rail adapt navigation to narrow viewports while respecting saved user preferences; object details open in a focus-restoring bottom sheet on mobile, and dialogs and forms scale cleanly to smaller screens. English and Chinese localization, accessible controls, and license attributions remain built in.
Permission-aware navigation organizes Identity, Monitoring, Settings, and Tools around the active session’s grants, consolidates bucket access under a single Buckets entry with recent-bucket shortcuts, and preserves post-login redirect targets and existing object deep links (/browser redirects to /buckets, and bare bucket-management URLs open the Summary tab). Help continues to link directly to Documentation, Video guides, and Blog/RSS feeds.
The Overview dashboard displays node and drive health, raw physical capacity, active alerts, quick actions, and data freshness. Capacity panels report the exact figures returned by Server rather than estimating usable erasure-coded capacity, marking uncollected statistics as unknown. Metrics V3 panels under Monitoring distinguish missing Prometheus samples from true zero values and handle per-node cluster metric exports accurately. Note that while Server 20261026 returns a live bucket inventory, bucket and account usage totals remain scanner-derived.
Buckets, object browser, and tasks
- Bucket list: Supports sorting, name filtering, URL query-parameter persistence, table/card view toggling, large-list virtualization, and explicit unknown-usage badges. Stable row keys keep selection attached to the right bucket across sorts, and in-flight search updates are discarded on navigation.
- Fine-grained permission handling: Write-only users can enter the object workspace; wildcard bucket-root grants and prefix restrictions are preserved; and rejected actions (such as tag edits) surface the server’s error message directly (listing, read, and mutation permissions remain evaluated independently).
- Bounded previews and CLI hints: Adds bounded 1 MiB previews for plain text, Markdown, and CSV/TSV files alongside source views, sanitized HTML rendering, and explicit truncation notices. Supported configuration forms also display secret-redacted
mclicommand equivalents. - Tasks center: Tracks uploads, downloads, native streaming ZIP downloads, bulk deletes, noncurrent-version cleanup, and object restores with bounded mutation concurrency and failed-item-only retries (canceling a task does not roll back already-completed writes; verify large object or streaming ZIP downloads on disk after browser transfer completes).
- Keyboard navigation and pagination: Adds arrow-key navigation, Shift-range selection, current-page Select All, a keyboard-shortcut reference dialog, and permission-aware context menus. Cursor pagination remains unchanged—sorting, filtering, and Select All operate on the current page.
Identity management and high-latency diagnostics
User detail pages display account status, direct policies, group-inherited policies, and Access Key counts; policy pages distinguish native built-in, overridden built-in, and custom policies while showing assignment counts and affected identities. Native built-in policies remain read-only until explicitly overridden, and modified built-in policies offer a one-click Reset. Failed reloads clear stale editor contents, language switches preserve unsaved drafts, and forms bind labels properly while masking remote secrets.
Note: Built-in policy Reset and live bucket inventory require Server 20261026. Upgrading standalone Console by itself does not update an older Server’s policy cache or replace its embedded frontend.
Across all six live diagnostic streams (Health, Trace, Logs, Watch, Speedtest, and Profile), Console now completes the WebSocket close handshake through the single existing reader goroutine with a shared five-second close budget that never cuts off active streams. This fixes the high-RTT health-report truncation and abnormal 1006 closures reported by Azat / @azataiot in #63 (wrapped timeouts still return close code 1008 without leaking internal details). Health-report downloads also reuse a single JSON decoder across the version probe and payload so buffered bytes are never dropped, reject truncated or canceled streams, and retain signed madmin v3 transport and retry behavior.
Fixes in 2.5.1
- Group status toggles (#67): Enabling or disabling a group now preserves its existing members, while an explicit
membersarray continues to replace membership. - OIDC groups and reverse-proxy cookies (#66): Preserves permissions for OIDC string-array group claims and sets
Securesession cookies only when HTTPS or a trusted TLS proxy header is configured. - Credential exports and streaming ZIP errors (#68): Preserves exact UTF-8 JSON bytes when exporting Access Keys, and propagates the S3 HTTP error status when a streaming ZIP download fails before writing its first response byte.
- Accessibility and focus restoration (#68): Restores keyboard focus across tables and dialogs, adds accessible names and progress-bar value attributes, and keeps the focused tab visible when resizing the viewport.
- Companion dependencies (#69): Updates to
mcli 20261026,silo-pkg v3.15.2, and the upstream SDK fix accepting HTTP204when removing bucket replication.
Dependencies and delivery
Console 2.5.1 builds with Go 1.27.3, Node 24.22.1, Vite 8.3.5, GoReleaser 2.19.0, silo-pkg 3.15.2, mcli 20261026, and upstream minio-go 320af050c445, while keeping tablewriter v0.0.5, go-humanize v1.0.1, go-runewidth v0.0.29, and go-systemd v22.6.0 pinned for output and platform compatibility (see the shared dependency and security inventory).
The GitHub release provides platform binaries and archives, Linux packages, source tarballs, SHA-256 checksums, SPDX SBOMs, and Sigstore/GitHub provenance attestations. Windows archives now use .zip (Linux and macOS retain .tar.gz), and multi-architecture container images are published at docker.io/pgsty/silo-console:v2.5.1 (see the release contract for verification commands).
Frontend assets and embedded bundles build reproducibly in isolated workspaces. The supported administration target is pgsty/silo (with best-effort upstream MinIO compatibility), and all existing CONSOLE_* environment variables, API routes, service names, and Go module paths remain unchanged; see the Console compatibility guide for API and embedding details.
2.18 - silo-pkg 3.15: Go 1.27.3 and S3 SDK Fixes
silo-pkg v3.15.2 is now released (commit 0416eb1f4505) and is adopted directly across the SILO 20261026 stack as github.com/pgsty/silo-pkg/v3. This note summarizes the full 3.15 series (v3.15.0, v3.15.1, and v3.15.2) since v3.14.3.
3.15.0: Toolchain and shared dependencies
The module’s minimum Go version increases from 1.26 to 1.27.3, alongside updates to 13 direct dependencies: JWX 3.4.0, etcd api / client/pkg / client 3.7.3, gRPC 1.84.0, grpc-gateway 2.32.0, go-json 0.11.2, swag/conv 0.29.2, go-colorable 0.1.16, cpuid 2.4.1, fwd 1.2.1, and the two Google generated API/RPC modules pinned at ce908d29299b.
Downstream applications must rebuild with Go 1.27.3 to pick up the Go standard library’s certificate-verification fix. All public package APIs, IAM policy formats, authorization rules, and the v3.14.3 policy evaluation optimizations remain unchanged.
3.15.1: Upstream S3 SDK fixes
v3.15.1 advances upstream minio-go from v7.4.1 to ab93db7ba435, bringing in six fixes:
- Rejects truncated sequential and parallel streamed multipart uploads before transmitting an incomplete part.
- Preserves sequential read state after
ReadAthitsEOFand rejects negativeReadAtoffsets before fetching object metadata. - Fixes read and
Seekbounds after callingObject.Stat. - Deduplicates batch-delete fallback errors so each failed entry is reported once.
3.15.2: Bucket replication removal
v3.15.2 advances upstream minio-go to v7.4.2-0.20261025165926-320af050c445 (a pinned pseudo-version, as upstream has not tagged v7.4.2). Removing bucket replication—including passing an empty configuration to SetBucketReplication—now accepts both HTTP 200 and HTTP 204 while continuing to surface AccessDenied and other real errors (see upstream comparison).
Compatibility and verification
Downstream applications that rely on established CLI formatting and NetBSD builds continue to pin go-humanize v1.0.1, go-runewidth v0.0.29, and go-systemd/v22 v22.6.0. Because Go does not inherit replace directives from dependencies, each consuming application declares its own module replacements.
v3.15.2 passed the full lint and -race test suites (including live LDAP integration), module verification, and all three CI workflows (Go, LDAP, and VulnCheck). HTTP fixtures verified that both replication-removal entry points accept 200/204 while preserving 403 errors, and earlier 3.15 regression suites covered JWT validation boundaries, local etcd auth/watch/lease persistence, and CLI formatting. Scanner finding GO-2026-5932 resides in deprecated OpenPGP packages that silo-pkg does not import (see the shared security inventory).
silo-pkg is a source-only Go module adopted by rebuilding Server, Console, and mcli. The supported dependency graph is the coordinated PGSTY stack, with best-effort upstream MinIO/MC compatibility. GitHub release · Full commit comparison.
2.19 - Silo Console 2.1.0 Released
Published: 2026-08-06 · Version: v2.1.0 · Repository: pgsty/silo-console
SILO Console 2.1.0 is the first feature release after the independent 2.0.0. It delivers three main changes:
- Bilingual interface — every console screen, help topic, and documentation link now renders in English or Chinese behind a toggle on every page, with zero new runtime dependencies;
- Metrics V3 dashboard — the dashboard moves from MinIO Metrics V2 names onto V3, with explicit handling for the semantics V3 changed underneath it;
- Edge-case correctness fixes — a select-all that matches what a bulk action will delete, placeholders that handle object names containing
$&, timestamps with an explicit timezone, and empty metrics that display “no data” rather than a fabricated0.
This is a minor release. No environment variable, module path, API contract, binary name, or data layout changes. Upgrading is a binary or image swap.
A 2.1.1 patch follows this release
v2.1.1, published the same day, completes the legend hardening described below: a label placeholder the legend builder cannot resolve is now removed instead of leaking literal braces into the Traffic chart legends, the one remaining substitution branch is escape-proofed against label values containing $& or $1, and the License page reports the actual release version instead of 2.0.0. Nothing else changes — upgrade straight to 2.1.1, and everything in this note applies unchanged.
Rebuild your embedded assets if you vendor this console
2.1.0 fixes a packaging defect present on the main branch after 2.0.0: the go:embed payload still carried the 2.0.0 frontend build, so a binary built from an intermediate commit would serve the old UI. The released 2.1.0 artifacts are built from the regenerated payload and are unaffected.
A Bilingual Console
The console is an administration surface for an object store, and a large share of its operators read Chinese first. 2.1.0 makes the interface bilingual without importing an i18n framework — the embedded delivery model means every kilobyte is paid for in the binary. This is issue #6, which proposed i18next; the dependency-free substitution is the one deliberate deviation from it.
How it works
The design constraint was: no new dependency, no build step, no extraction pipeline, and partial coverage must never break the page.
- English source strings are the dictionary keys.
t("Create Bucket")looks up the Chinese entry; a missing key returns the English string unchanged. Coverage can therefore grow incrementally, and a typo degrades to English rather than to a raw key likeconsole.bucket.create. - Three dictionaries, one merge.
zh.ts(165 chrome entries),zhHelp.ts(247 help-topic entries), andzhScreens.ts(1,373 screen entries) merge with chrome taking precedence — about 1,785 entries in total. - The language preference mirrors dark mode:
localStorage→systemSlice→setLanguage. There is no browser-locale detection; the default is English, and the choice is explicit. - Central interception points rather than per-callsite edits: the page-header wrapper, confirm dialogs, help items, route definitions, and the dashboard’s panel renderer each translate on the way out. This is why 220 screen files could be localized without touching their business logic.
- Module split matters.
i18n/lang.tsholds pure primitives (translate,localizeUrl) and imports no store —systemSlicedepends on it, so importing the store back would form a cycle. The hooks (useT,useLanguage,useLocalizedLink) andinterpolate()live ini18n/index.tsx.
The toggle is a stroke-drawn 文/A icon mounted in the page header on every page and reused on the login page.
What it covers
Login and SSO flows, navigation and the command palette, the dashboard and every metrics panel, buckets and the full object browser (uploads, previews, sharing, versioning, rewind), users/groups/policies/access keys, configuration and event destinations, IDP and KMS, logs, health reports, speedtest, profiling, inspect, trace, watch, and the license page.
Beyond visible strings:
- Documentation links localize.
silo.pgsty.comlinks gain a/zhprefix in Chinese; the Pigsty site swaps domains (pigsty.io↔pigsty.cc). GitHub, MinIO, AWS, and YouTube links are left alone. - The help blog feed is per-language, fetching
/zh/blog/index.xmlin Chinese, with an independent cache per language. - The command palette stays searchable in both languages. Menu entries translate for display but keep their English originals as keywords, so “桶” and “buckets” both match.
- Chart legends translate only their static prefix.
translateLegendpreserves instance suffixes like[server:drive], and the data layer keeps raw legends so components that match on them for arithmetic (capacity summing) keep working. - Timestamps are unified, not merely translated — see below.
What it costs
Roughly +61 KB on the embedded payload (2.79 MB → 2.85 MB, +2.2%), zero new dependencies, and the dictionaries land in their own lazily-loaded chunk. The English rendering path is byte-stable: with the default language, output is identical to 2.0.0.
What stays English
Backend error strings (182 of them) are produced by the Go server and are not translatable from the frontend. A handful of strings hardcoded inside the vendored mds component library — the collapsed-menu “Sign Out” tooltip, and the data table’s “Columns”, “Loading…”, and ON/OFF toggles — remain English; two of them (“Sign Out”, “Actions:”) are swapped via a scoped CSS rule, but the rest would require patching the vendor.
Metrics V3 Migration
The dashboard queried MinIO Metrics V2 names. SILO deployments scrape V3 (/minio/metrics/v3), so the dashboard depended on an endpoint the monitoring pipeline no longer collected. 2.1.0 rewrites all 26 widgets onto the V3 catalog — 31 queries over 29 distinct metric names — and drops three widgets (51/61/62) that no layout ever referenced. This is issue #7; the Info-page half is #8.
The decision is V3-only: no runtime fallback, no probing, no version-selection knob. SILO Console targets SILO deployments, where the server, the scrape pipeline, and the console ship together. The SILO server keeps serving V2 endpoints for external consumers; the console simply stopped using them. A fallback would have been actively harmful — a metrics store retaining 15 days of V2 series would let an or-fallback silently read stale data.
The semantics V3 changed
Three properties of V3 break a naive name-for-name rewrite, and each needed a deliberate answer:
- Cluster groups are exported identically by every node.
/cluster/*metrics carry no server label and are not leader-gated, so an N-node scrape yields N duplicate series. Queries aggregate withmax()/min()— neversum(), which would multiply cluster totals by the node count. - Zero values are not exported at all. Any metric whose value is ≤ 0 is skipped. Offline drive counts, healing-drive counts, and erasure-set health simply vanish rather than reporting
0, which a stat card renders as an empty panel. Every affected query carries a companion guard so the panel reads a real0. - There is no
minio_heal_*namespace. The V2 heal activity signal was in-memory anyway — it reset on restart and bumped on any scan. It is replaced by two cards with defensible semantics: Erasure Health (baselined on write quorum) and Usage Data Age (how stale the scanner’s usage snapshot is).
Zero-state semantics
The migration distinguishes zero, no data, and not yet scanned across eight edge cases:
- Capacity free/used baselines on the always-present total, so a full cluster reads
0 freeinstead of vanishing. - Online Drives is guarded against the all-offline case, where the zero-skip would erase the panel when all drives are down.
- Bucket and object counts guard on the usage group’s own freshness gauge, so a cluster that has not completed its first scan reads no data rather than a fabricated
0. - Empty single-value results render as
—, not0. - An empty size distribution no longer fabricates seven zero-height bins.
- Fractional rates stay visible (
parseFloataxis domain, two-decimal CPU formatter) instead of collapsing to0. - Sub-second Usage Data Age clamps to “1 second” instead of rendering blank.
A regression suite (api/admin_info_metrics_test.go) now pins every widget query to the V3 catalog, asserts widget-ID uniqueness, and enforces the per-widget guard taxonomy: health and traffic widgets need a nodes-online companion, usage counts need the usage-group freshness companion, and capacity needs the total baseline. The full mapping is documented in docs/metrics-v3.md.
Also fixed
- Widget 17 queried
sent_bytestwice and widget 11 queriedsyscall_readtwice — both internode/syscall pairs were transposed into duplicates. - Label-less matrices (the result of
max()aggregation) serialize with nometricfield at all, which crashed the frontend’s label extraction and produced a0 Bcapacity donut and an empty usage-growth chart. Guarded. - An unused per-widget Prometheus label-values prefetch stalled every widget request by up to a second. Deleted.
- The dashboard’s usage cards, chart controls, and dense Traffic/Resources panels were rebuilt on one grammar and now reflow through tablet widths.
Two server-side bugs were identified during this work and are tracked upstream rather than worked around here: minio_cluster_usage_buckets_since_last_update_seconds emits nanoseconds (the objects variant is correct), and V3 bucket-level sent/received traffic are transposed.
Correctness Fixes
Placeholders that survive real object names
String.prototype.replace interprets $&, $', $`, and $1 in the replacement value as directives. S3 keys legally contain $. So an object named report$&.csv did not render as itself — it re-injected the matched placeholder text into the output and corrupted the message. All 37 dictionary placeholder substitutions now pass the value through a function replacement, where no such interpretation happens. This bug existed in the original English UI and surfaced during the localization audit.
A select-all that matches bulk actions
The vendored data table renders a plain untranslatable “Select” header whenever onSelectAll is absent — which was the case on all seven selectable tables. Worse, the naive fix is wrong: a select-all that replaces the whole selection drops rows hidden by an active filter, so the header checkbox and a subsequent bulk action can target different sets. The implementation toggles only the currently visible rows and preserves filter-hidden selections, so the header state can no longer imply a different set than the action would touch.
Timestamps with a timezone
Bucket, object, version, rewind, and access-key timestamps rendered as a mix of verbose English forms and — in several places — a 12-hour clock without AM/PM, which is simply ambiguous. All of them now render as yyyy-MM-dd HH:mm[:ss] (ZZZZ) in both languages.
A translation runtime that survives live data
t() also receives runtime strings: user agents, RSS titles, object names. Two hardening changes followed:
- misses return unchanged, unconditionally — the implicit
@contextsuffix stripping is gone, because it silently mutated live data that happened to contain@; - dictionary lookups are guarded with
hasOwnProperty, so a hostile input naming an inheritedObject.prototypemember (constructor,toString) cannot leak a function into the UI.
Interaction and accessibility
- An expired session opening a deep link bounced through
/loginand back, accumulating a redirect chain instead of landing on the form once (#1). - Collapsed sidebar buttons carried no accessible name; screen readers announced them as unlabelled (#4). Access Key inputs now declare their autocomplete intent instead of letting password managers guess (#5).
- Mobile metrics and bucket panels scroll instead of clipping (#3).
- The speedtest control row wraps instead of overflowing its card, its duration accepts seconds or minutes, and its size defaults to MiB to match its own unit list.
- Sidebar bucket rows use a virtual row pitch matching the 44px item, so selected and hovered highlights no longer overlap.
- Unit chips render the selected unit’s label rather than its raw value.
No SUBNET, No Telemetry
Upstream removed Subnet, Registration, and Call Home; this fork inherited that state but still carried three traces. 2.1.0 removes them:
- the health websocket’s
subnetResponsefield never addressed a subnet — it is a sentinel meaning “the report was assembled” — and is nowreportStatus: "ok"; - two help topics claimed the health report “uploads automatically to SUBNET” and that inspect output is “transmitted to SILO SUBNET”. Neither was true. They now describe what happens: the report is generated on the deployment and downloaded by the browser;
- the unreferenced
CONSOLE_SUBNET_PROXYconstant is deleted.
For the record, 2.1.0’s outbound network posture is unchanged and remains: no analytics, no telemetry, no beacons, no external scripts or fonts. silo-console update is still disabled. The release catalog is contacted only if SILO_RELEASE_SERVICE_HOST (or RELEASE_SERVICE_HOST) is explicitly set — there is no default. The only automatic outbound request the browser makes is the help panel’s blog feed, and only after a user opens the Blog tab.
Upgrade Guide
There is nothing to migrate. No environment variable, module path, protocol field, systemd unit, binary name, or data layout changes between 2.0.0 and 2.1.0.
Two things are worth knowing:
- The dashboard now requires Metrics V3. If your Prometheus scrapes only the V2 endpoints, dashboard panels will read no-data. Point the scrape at
/minio/metrics/v3; Pigsty-managed deployments already do. - The language default is English, chosen per browser and stored in
localStorage. There is no server-side default and no browser-locale detection, so no existing deployment changes appearance on upgrade.
Verification Scope
Before tagging, the following verification checks ran against the final tree: go build, go vet, golangci-lint (0 issues), the Go unit suite across all packages, gofmt, TypeScript type checking, the frontend production build, Prettier across all sources, dictionary duplicate-key checks, and a debug-leftover scan of the complete diff.
The embedded payload was rebuilt twice from a clean directory and confirmed byte-identical for the release pipeline’s zero-diff gate. Metrics V3 queries were validated against a live metrics store with real cluster data.
Known Limitations
- The SSO end-to-end suite requires an external OpenLDAP/Dex/MinIO topology and was not run in that environment this cycle; the OIDC code paths are covered by unit tests.
- Backend error strings and several vendored
mdscomponent strings remain English (see What stays English). - Chinese translation covers the console’s own surfaces; help-topic bodies are translated, but the documentation pages they link to follow the docs site’s own language coverage.
- Two server-side V3 metric bugs (nanosecond bucket-usage age, transposed bucket traffic) are tracked upstream and are not worked around in the console.
- Automatic self-update remains disabled; upgrades are explicit.
Issues Closed
2.1.0 closes every issue filed against 2.0.0. Each carries a comment on the tracker describing the fix, the commits, and the coverage added.
| Issue | Resolution |
|---|---|
#1 — unauthenticated deep routes recurse /login |
Absolute, base-path-aware login destination; deep-link and subpath test coverage |
| #2 — stale Uptime, malformed legends, cramped menus | Uptime derived from real server state, legends resolve on the V3 name label, 32 px chart controls, popup width floors |
| #3 — 390 px viewport clips content | Scrollable metrics tab strip; bucket table with a deliberate mobile column budget |
| #4 — unnamed collapsed sidebar buttons | Labels visually hidden rather than removed from the accessibility tree; named, keyboard-operable collapse toggle |
| #5 — Access Key fields lack autocomplete metadata | Field-level username / new-password tokens in a dedicated autofill section |
| #6 — English/Chinese localization | Hand-rolled bilingual layer, zero new dependencies, English-as-key fallback |
| #7 — migrate monitoring queries to Metrics V3 | V3-only; 26 widgets, 31 queries, 29 metric names, guard taxonomy, regression suite |
| #8 — replace N/A Info metrics | Erasure Health and Usage Data Age, sharing the advanced dashboard’s widget results |
Three acceptance criteria are recorded as unmet rather than quietly ticked: web-app has no unit-test runner, so the i18n test suite (#6) and the focused constructLabelNames test (#2) would require introducing test tooling first, and #6’s contributor documentation for adding translation keys is not yet written.
Related Commits and Links
The complete v2.1.0 change set consists of 20 logical commits. The v2.1.0 tag additionally carries three later documentation commits that rewrote the repository README; they change no shipped behavior.
8764f5d— fix(web): stop recursive login redirects437c56c— fix(ui): make the dashboard and bucket list usable on narrow screens85fc0c6— fix(a11y): name collapsed sidebar controls and credential fieldse3fed07— fix(metrics): rebuild dashboard cards, chart controls, and layoutfa11576— feat(login): polish controls and legal attribution9fc17c1— feat(i18n): add hand-rolled EN/ZH core, dictionaries, and language toggle622c02e— feat(i18n): localize login, navigation, and the help system6a03719— feat(i18n): localize dashboard and metrics screens14b1c2d— feat(i18n): localize bucket and object browser screens0298062— feat(i18n): localize identity, configuration, and event destinations41094f6— feat(i18n): localize observability, admin tools, and shared componentse964992— feat(metrics): migrate the dashboard to MinIO Metrics V30b2251f— fix(i18n): harden the translation runtime for live data and chart legends9b60148— fix(console): unify timestamps on a timezone-carrying standard formatbf110ae— fix(console): give selectable tables a visible-rows select-all5fc8f22— fix(i18n): escape-proof all placeholder substitutionsfef8fab— fix(console): polish speedtest, sidebar, and help chromec4911e8— chore(console): drop SUBNET remnants from health reporting1d631c4— docs: record the SILO Console v2.1.0 changelog912d847— build: regenerate optimized embedded web assets
Links:
2.20 - Silo Console 2.0.0 Released
Published: 2026-08-04 · Version: v2.0.0 · Repository: pgsty/silo-console
SILO Console 2.0.0 is the first major release of this object-storage administration console as an independent project. Continuing from the georgmangold/console v1.9.1 maintenance line, it accomplishes three things:
- An independent identity — product name, visual system, documentation entry points, source attribution, and the release pipeline all move into the SILO project, while the Go module path, environment variables, and other compatibility contracts are deliberately retained;
- A redesigned interface — the login page, theme system, dashboard, and console details are reworked under one design language, backed by a regenerated brand icon set;
- Hardened engineering — the embedded frontend payload shrinks from roughly 10MB to 3.5MB, known dependency vulnerabilities drop to zero, and a batch of inherited defects — including a runtime data race — is fixed.
Read the compatibility boundary before upgrading
Version 2.0.0 updates public identity and delivery contracts while keeping the object data format and S3 protocol unchanged. Installation scripts that reference the old repository, binary name, or container image must be updated; existing integrations that use CONSOLE_MINIO_SERVER, CONSOLE_MINIO_REGION, github.com/minio/console, or the MinIO-compatible Admin API remain unchanged and must not be search-and-replaced.
Why 2.0.0
This console originated as MinIO Console and was carried forward by the Alevsk/console and georgmangold/console community maintenance lines. SILO Console continues from there, maintained by the Pigsty community as the browser-based administration interface for SILO.
The version jumps from v1.9.1 to v2.0.0 because these public contracts change together:
- the product is now uniformly SILO Console, with the primary repository at
pgsty/silo-console; - the release binary changes from
consoletosilo-console; - release assets, checksums, package metadata, CLI descriptions, and project links all switch to SILO;
- in-product identity, help entry points, copyright attribution, source offers, and trademark notices are re-established.
The migration strategy is “clear external identity, restrained internal compatibility”: operators must take notice, but the underlying compatibility interfaces are not mechanically renamed.
Naming and Delivery Contracts
| Scope | Previous name or location | 2.0.0 contract |
|---|---|---|
| Product | Console / legacy MinIO Console | SILO Console |
| Repository | georgmangold/console |
pgsty/silo-console |
| Release binary | console |
silo-console |
| Binary assets | console-<os>-<arch> |
silo-console-<os>-<arch> |
| Checksums | console_<version>_checksums.txt |
silo-console_<version>_checksums.txt |
| Website and docs | upstream / previous maintainer | silo.pgsty.com and silo.pgsty.com/docs/ |
For container deployments, use pgsty/silo-console on Docker Hub. See the Console v2.4.1 release notes for the current version and pull command.
CLI authorship, usage text, and project descriptions now identify Pigsty and SILO Console. DEB/RPM/APK vendor, maintainer, homepage, description, and license metadata are updated accordingly; the executable installs to /usr/local/bin/silo-console.
Deliberately Retained Compatibility Identifiers
The following names retain minio or console as part of interface, protocol, or installation compatibility:
| Surface | State in 2.0.0 | Reason |
|---|---|---|
| Go module | github.com/minio/console retained |
changing it breaks every Go import |
| Server endpoint | CONSOLE_MINIO_SERVER retained |
widely used by existing deployments |
| Server region | CONSOLE_MINIO_REGION retained |
existing compatibility contract |
| Other configuration | existing CONSOLE_* variables remain valid |
avoids migration with no benefit |
| S3/Admin API names | MinIO-compatible fields and enums retained | they describe the actual protocol |
| Development build | make console still produces ./console |
keeps developer workflows working |
| Package systemd unit | minio-console.service retained |
avoids duplicate services on upgrade |
| systemd user and config | console-user and /etc/default/console |
avoids unnecessary account/config migration |
Upgrade scripts therefore must not run repository-wide minio → silo or console → silo-console replacements. Migrating these compatibility interfaces in the future will require aliases, deprecation windows, and an explicit dual-read strategy.
A Redesigned Interface
Version 2.0.0 redesigns the console interface end to end.
Login page
The login page is rewritten from scratch. The left brand panel renders a slowly drifting sine-mesh animation generated purely on Canvas (zero external dependencies, honors prefers-reduced-motion, pauses in background tabs), states the project’s proposition — “Keep the S3 Interface / Own the Object Store” — and keeps the full MinIO trademark notice at the bottom. The right-hand form is functionally untouched, preserving every existing automation selector. The Chakra Petch typeface used by the SILO wordmark ships as a ~20KB locally bundled subset with no external requests.
A unified theme system
All console colors converge into one light/dark theme layer: neutral greys for text and borders, the brand steel blue for primary actions and selection, and a sidebar that uses the same night palette as the login panel in both modes. Controls and cards share consistent radii and transitions, inputs get a keyboard focus ring, and modals animate in (also honoring reduced motion). Server-provided customStyles keep full precedence.
Console polish
- Dashboard (Metrics): stat cards rebuilt under one grammar — muted labels, tabular numerals, aligned status dots; charts and info strips are theme-driven; the upstream absolute-positioning layout is gone.
- Unified empty states: placeholder text in Watch, Trace, bucket Events/Replication/Lifecycle, and every other data panel is now centered and de-emphasized instead of raw top-left text.
- Vertical tabs: detail-page tabs change from bordered grey blocks to a quiet pill list, eliminating the stray empty cell at the bottom of the rail.
- License page: a new VERSION section shows both the connected server’s release and the Console’s own version; accounts without
admin:ServerInfonever issue the request and the row stays hidden. The page also consolidates AGPLv3 licensing, the AGPL section-13 source offer, lineage, and trademark boundaries. - A batch of interaction fixes: the sidebar now collapses on initial load at mobile widths (previously it waited for a resize event); the bottom navigation no longer lags window-height changes; the bucket accordion highlight spans the full row; the dashboard no longer overflows horizontally on narrow screens; and the help panel is now truly lazy — the login page makes no external requests at all.
Brand icon set
The favicon, PWA, and Apple Touch icons still carried a previous-generation hand-drawn emblem. 2.0.0 re-rasterizes every size (ico 16+32, favicon 16/32/96, apple 180, manifest 192/512) from the official silo.svg vector emblem, with safe-area margins on home-screen sizes, and trims the Web App Manifest to the modern icon set, dropping the 2014-era legacy density entries. The icon payload drops from 473KB to 160KB, and the browser tab icon finally matches the in-product brand.
Smaller and Faster
Embedded delivery is this console’s core form factor — the frontend ships inside the binary via go:embed. 2.0.0 optimizes that path systematically:
- Embedded payload: ~9.6MB → 3.5MB. Text assets (JS/CSS/SVG/…) are precompressed at build time with deterministic gzip and embedded compressed-only; legacy WOFF fonts (~1.25MB that no supported browser ever downloads) and a set of entirely unreferenced orphan images are removed.
- First-load transfer: ~5.7MB → ~1.7MB. Static assets previously shipped uncompressed on the wire; they are now emitted directly with
Content-Encoding: gzipat zero runtime cost, with on-the-fly decompression for the rare client that does not accept gzip. - Correct HTTP semantics. Accept-Encoding is parsed with full RFC 9110 q-values (
gzip;q=0gets identity bytes), responses carryVary: Accept-Encoding, and non-GET/HEAD requests to static paths and the SPA entry receive 405 with anAllowheader. - Reproducible builds. Compression uses a pure-JS implementation (fflate) for byte-identical output across platforms, and the release pipeline enforces a hard gate: rebuilding the embedded assets in a clean environment must produce zero diff against the commit.
Release binaries (all frontend assets included, stripped) weigh roughly 35–40MB; for the downstream SILO server, embedding this console now costs about 3.5MB instead of about 10MB.
Security and Dependencies
Go: the build baseline moves to Go 1.26.5 and the golang.org/x family is fully refreshed. Every reachable vulnerability reported by govulncheck is resolved:
| Dependency | Fixed version | Advisories |
|---|---|---|
google.golang.org/grpc |
v1.82.1 | GO-2026-6061 |
github.com/prometheus/prometheus |
v0.311.3 | GO-2026-5710 / -5662 / -5381 / -5264 (incl. remote-read DoS) |
github.com/klauspost/compress |
v1.18.7 | GO-2026-5841 |
The single remaining advisory sits in golang.org/x/crypto, has no upstream fix yet, and is unreachable from this codebase; it is tracked as a known item.
Frontend: the full dependency-tree audit (production and tooling) is clean, covering the high-severity form-data CRLF injection and the DOMPurify and qs advisories; React Router is migrated to 7.18.2 (keeping the v6-compatible declarative API, with full routing regression). The only explicitly ignored advisory affects an unstable API this project does not use.
Runtime correctness: a real data race between HTTP log-target initialization and shutdown is fixed, along with shared-mock races in the test suite; supported Go packages pass -race across the board. As a side benefit, the go-m1cpu upgrade fixes the local go run cgo crash on recent macOS.
Update Checks and Default Network Behavior
This release keeps conservative defaults for upgrade tooling:
- automatic self-update in
silo-console updateis disabled — the command prints guidance and never downloads or replaces the binary; - the release catalog gains
SILO_RELEASE_SERVICE_HOST, with the previousRELEASE_SERVICE_HOSTas a compatibility fallback; with neither set, no remote release service is contacted; - the help panel’s blog content loads only when opened, and its links accept
https://silo.pgsty.comexclusively.
Automatic updates will be reconsidered once signed release assets and a tested rollback path are in place.
Release Artifacts and Platform Matrix
The release ships 16 assets:
| Type | Coverage |
|---|---|
| Standalone binary | Linux amd64/arm64/arm, macOS amd64/arm64, Windows amd64 |
| System packages | DEB / RPM / APK × amd64/arm64/armv6 |
| Checksums | silo-console_2.0.0_checksums.txt (SHA-256) |
The pipeline triggers on tag pushes, pins third-party Actions to commit SHAs, and enforces the clean-checkout and zero-diff asset-rebuild gates before GoReleaser runs.
Upgrade Guide
Standalone binary
When building from source, make console still produces ./console; install it under the release name before wiring it into a production service.
DEB/RPM/APK and systemd
Packages continue to install /etc/systemd/system/minio-console.service, whose unit starts /usr/local/bin/silo-console. EnvironmentFile=/etc/default/console, console-user, and existing CONSOLE_* variables are unchanged. This retention lets package upgrades keep acting on the existing service instead of creating a parallel one.
Configuration and integrations
- do not rename
CONSOLE_MINIO_SERVERorCONSOLE_MINIO_REGION; - do not touch
github.com/minio/consolein Go imports; - prefer
SILO_RELEASE_SERVICE_HOSTfor self-hosted release catalogs; - replace any reliance on
console updatewith explicit download, verification, and deployment; - update process-path-based monitoring to
/usr/local/bin/silo-console.
This release does not change the object data layout and requires no bucket or object migration.
Validation Scope
Release verification covered Go -race across supported packages, go vet, golangci-lint, govulncheck, frontend type checks, production builds, Prettier, dead-code checks, full dependency audits, and the following integration checks:
- all 184 embedded files fetched three ways each (gzip client, identity client, HEAD) with per-file hash comparison against the embedded sources;
- RFC semantics probes (including combined q-values such as
gzip;q=0, *;q=0.5), method restrictions, the OIDC callback, and SPA deep links; - full React Router 7 regression: deep links, client-side navigation, bucket-detail tab switching, and browser history back;
- mobile first-load sidebar behavior, login-page external-request monitoring, and light/dark full-site tours;
- downloaded release assets verified byte-for-byte against checksums, binary self-reported version confirmed, and a smoke test of the published binary against a live server;
- zero-diff asset rebuilds confirmed on both macOS and Linux.
Known Limitations
- automatic self-update is disabled; upgrades are explicit;
- the SSO end-to-end suite requires an external OpenLDAP/Dex/MinIO topology and was not run in that environment this cycle (the OIDC code paths are covered by unit tests and HTTP-level checks);
- one
golang.org/x/cryptoadvisory has no upstream fix yet and is unreachable from this codebase; - SILO does not yet maintain its own video library; videos in the help panel are clearly labeled upstream compatibility material;
- administrative features depend on the MinIO-compatible Admin API — SILO Console is not a generic browser for arbitrary S3 services;
- retained Go module paths, environment variables, protocol fields, and the systemd unit name still appear in code, configuration, and process listings.
Related Commits and Links
The complete v2.0.0 change set consists of 8 logical commits:
50797de— feat: establish SILO Console identity and compatibility23ae6e8— feat: redesign and harden the SILO Console web app7a83a77— build: update Go toolchain and dependencies1330d25— fix: eliminate logger shutdown and test mock races06b3a34— docs: publish the SILO Console v2.0.0 guide4b24372— build: regenerate optimized embedded web assetsc38eb64— ci: package and publish SILO Console v2 releasesb952a12— brand: regenerate the icon set from the official silo.svg emblem
Links:
2.21 - silo-pkg 3.13.0 Released
Release date: 2026-08-30 · Version: v3.13.0 · Commit: 215f116 · Repository: pgsty/silo-pkg
Version 3.13.0 is a breaking release: the module adopts github.com/pgsty/silo-pkg/v3 as its own import path and retires the pgsty/silo-go fork in favour of upstream minio-go. All maintained SILO components (silo, silo-console, and mcli) migrated to this module path in the 2026-09-03 / v2.4.0 release wave; see the component matrix.
Release at a Glance
- Published: the
silo-pkg v3.13.0tag and GitHub Release, declaringmodule github.com/pgsty/silo-pkg/v3. - Retired: the
github.com/minio/pkg/v3module identity and thereplacedirective every consumer had to repeat. - Retired: the
github.com/pgsty/silo-go/v7fork; this release requires upstreamminio-godirectly. - Unchanged: every package, symbol, and behaviour. Only the path they are imported from moved.
- Downstream adoption: completed across SILO 20260903, Console v2.4.0, and mcli 20260903.
Why the Path Moved
The fork kept upstream’s github.com/minio/pkg/v3 path so it stayed a drop-in replacement, selectable with one replace directive. Go does not inherit replace directives from dependency modules, and three costs came due while preparing the next SILO server release.
Every consumer had to repeat the redirect, and one that forgot built against upstream silently. A module that requires this package but omits the replacement resolves the real minio/pkg, compiles, and quietly loses the fork’s behaviour.
The require line had to name a version the source no longer matched. ParseConfigStrict first appears upstream in v3.11.0, and Resource.IsBareARN exists in no upstream version at all. A require github.com/minio/pkg/v3 v3.6.1 next to source that needs v3.12 APIs is metadata that cannot be made true by editing the number.
Working around that propagated. pgsty/mc carried a compile-time sentinel — a reference to Resource.IsBareARN — purely to turn the silent downgrade into a build failure. Raising mc’s floor to make its metadata honest pushed the requirement through the module graph and forced silo-console’s deliberately low floor upward, which is exactly what that floor existed to prevent.
Upstream’s path had little to offer in return here. minio/pkg is a small internal library, and this module’s consumers are silo, silo-console, and mcli. The Silo Go SDK keeps upstream’s github.com/minio/minio-go/v7 path, where drop-in compatibility is worth having and upstream is actively maintained.
Upstream minio-go Replaces the Silo Go SDK
pgsty/silo-go no longer carried any functional divergence. Its one unique change, Return CopyObject checksums in UploadInfo, was merged upstream on 2026-08-24. Everything else in the fork was a version string, a logo, a README, and a lint-tool block.
Upstream’s newest tag v7.3.0 predates that merge by 14 commits, among them a data race fix in parallel multipart checksum hashing, so pinning the tag would be a regression. This release therefore requires the pseudo-version v7.3.1-0.20260828014306-0e78d3f18efe and will move to a tag when upstream cuts one.
Compatibility
Every package, exported symbol, and behaviour is unchanged. This release moves where they are imported from, nothing else.
Take it together with the matching silo, silo-console, and mcli changes. A consumer that upgrades alone will not build, because the old and new paths are different modules and types do not cross between them.
Before:
After:
…and rewrite github.com/minio/pkg/v3/... imports to github.com/pgsty/silo-pkg/v3/.... Consumers that stay on the old arrangement keep building against v3.12.2 and earlier, which remain published.
One consequence is worth expecting. Because the two paths are now separate modules, a build can contain both — a third-party dependency that imports github.com/minio/pkg/v3 no longer has its import redirected. In the SILO stack this happens once, through minio/colorjson and minio/dperf, both of which reach only pkg/v3/console. That package’s colour switch is fatih/color’s process-wide NoColor, which every copy shares, so disabling colour still disables it everywhere.
Verification
go build ./...,go vet ./...,go mod tidy -diff, andgofmt -l .are clean;go test ./...passes 23 packages.- CI on the release commit: Go, LDAP Config Validator, VulnCheck.
- The new path was resolved from the module proxy by a fresh module with no
replacedirective, which importedpolicyand ran. - The downstream migration was validated end to end before this release:
mcli(194 files),silo-console(36 files), andsilo(181 files) all build, vet, and pass their suites.silo’s rebrand guard passes with a compatibility-baseline diff of exactly 19 deleted import entries and no change to environment variables, metrics, headers, routes, policy values, or exported symbols.
2.22 - silo-pkg 3.12.0 Released
Release date: 2026-08-24 · Version: v3.12.0 · Commit: 2b087a1 · Repository: pgsty/silo-pkg
Version 3.12.0 is a main-line minor release with two themes: a policy-write guard for ARN prefixes that name no resource, and the maintained Go 1.27 / etcd 3.7 dependency baseline. It adds two exported inspection/validation methods, raises the verified consumer floor to Go 1.26, and is the first silo-pkg release whose strict validation path is enabled by the SILO server (eee05a17c, SN-2026-005, shipped in SILO 20260903) for named-policy and service-account policy writes.
Release at a Glance
- Published: the
silo-pkg v3.12.0tag, GitHub Release, and strict library validation for bare ARN prefixes. - Server integration: named-policy and service-account policy write validation in
eee05a17c(SILO 20260903). - Deliberately permissive: existing IAM policy loading, IAM import, and site-replication receive paths.
- Deferred: STS inline-policy strict validation and Console-side early validation; the server remains authoritative.
Bare ARN Prefixes Were Policy No-ops
An S3 resource ARN needs a resource after its namespace prefix:
The policy parser also accepted the prefix by itself:
That string names no bucket or object. The existing parser normalized it to the wildcard resource type while retaining arn:aws:s3::: as the match pattern. Real S3 authorization candidates look like bucket or bucket/object, so the pattern normally matched nothing even though the policy validated successfully.
For statements that actually perform resource matching, the impact depends on Effect and on whether the prefix appears in Resource or NotResource:
| Statement shape | Existing runtime result |
|---|---|
Allow + bare Resource |
Grants nothing |
Deny + bare Resource |
The intended denial does not fire |
Allow + bare NotResource |
Excludes nothing and can grant far more than intended |
Deny + bare NotResource |
Can deny far more than intended |
The dangerous cases are policy-dependent fail-open behavior, not an unauthenticated remote exploit and not a CVE. A policy author, template, or automation must first submit the malformed resource. The same issue applies to registered S3 Tables and KMS ARN prefixes.
Resource-less admin actions, sts:* action statements, and the first phase of two-step KMS authorization bypass resource matching; a bare prefix does not change their existing runtime decision. Strict writes still reject the deceptive field so a policy cannot look scoped when that scope is ignored.
The Historical *arn:... Spelling
On the permissive compatibility path inherited from earlier releases, serializing a parsed bare prefix adds the wildcard type marker:
Re-parsing either spelling produces the same internal resource value. The 3.12 guard therefore recognizes both the exact prefix and its historical star-prefixed serialization. This matters for stored/exported policies and for clients that parse and marshal a document before sending it to the server.
The accepted wildcard corpus remains unchanged: *, **, ***, *foo, and explicit resources such as arn:aws:s3:::* still parse as before.
A Strict Write Path, Not a Storage Migration
The fix deliberately separates policy loading from policy creation:
ParseConfigandValidateremain permissive. Existing stored policies keep loading and evaluating with the same matching and serialization behavior.ParseConfigStrictandValidateStrictreject a registered ARN prefix that names no resource, in bothResourceandNotResource.Resource.IsBareARN()detects the normalized exact/historical form without changing the exportedResourcestructure,ParseResource, matching, or JSON representation.ResourceSet.ValidateStrict()exposes the strict resource-set check to consumers.
Keeping the existing resource representation is important for mixed-version sites: v3.11 and v3.12 nodes continue to compare and serialize stored policies the same way, so this fix does not create a site-replication mismatch or require a storage migration.
SILO Enables the Guard on Three Boundaries
SILO commit eee05a17c selects silo-pkg v3.12.0 and uses strict parsing when:
- creating or replacing a named IAM policy;
- creating a service account with an inline session policy; and
- updating a service account’s inline session policy.
Compatibility-sensitive paths stay permissive in this rollout:
- loading named policies and embedded policies already at rest;
- IAM import/restore;
- site-replication receive and apply paths;
- STS inline session policies; and
- bucket policies, whose existing bucket/action validation already rejects these forms.
Enabling ParseConfigStrict also activates two pre-existing admin-policy checks: one admin statement may not contain both Resource and NotResource, and a bucket-scoped admin action may not use a non-S3 resource. These are intentional authorization tightenings and are documented in the advisory ledger under SN-2026-005.
In this article, bare ARN prefix means an ARN namespace with no resource after it, such as arn:aws:s3:::. It is different from the valid bare bucket ARN used in the 3.11 bucket/object-boundary fix, such as arn:aws:s3:::my-bucket.
Operator Action
Existing policies are not rewritten automatically because the intended resource cannot be inferred. Before deploying a SILO server build that contains the strict integration:
- inspect named IAM policies for exact or historical bare prefixes;
- inspect service-account inline policies;
- replace each finding with the intended concrete resource; or use a suffix wildcard only if all resources in that namespace are truly intended; and
- repeat the audit after all sites have completed the rolling upgrade.
Do not blindly turn every finding into arn:aws:s3:::*: that could replace an inert statement with a cluster-wide grant or denial. A legacy policy containing a bare prefix still loads, but it cannot be submitted unchanged through the three strict write endpoints; correct it before editing another property on the same policy or service account.
The safer audit path uses policy-info and access-key-info APIs rather than a full IAM export, because a complete export contains user and service-account secrets. STS strict validation remains deferred until live machine clients and their session-policy templates can be audited separately.
Why This Is a Minor Release
This is v3.12.0, not v3.11.1, because the release combines three compatibility-relevant changes:
- the etcd client crosses from the 3.6 minor line to 3.7;
- the module’s verified
gofloor rises from 1.25.0 to 1.26.0; and - the package adds exported bare-ARN inspection and strict resource-set validation APIs.
The module path remains github.com/minio/pkg/v3; the /v3 import suffix and every existing import site stay unchanged.
Go and Dependency Baseline
The go and toolchain directives have separate purposes:
go 1.26.0is the supported consumer floor required by the selected etcd 3.7 modules.toolchain go1.27.0is the maintained development and CI baseline.- CI actions move to the Node 24 runtime.
Key selected versions change as follows:
| Module | 3.11.0 | 3.12.0 |
|---|---|---|
go.etcd.io/etcd/{api,client/pkg,client}/v3 |
3.6.6 |
3.7.1 |
golang.org/x/crypto |
0.54.0 |
0.55.0 |
golang.org/x/net |
0.57.0 |
0.58.0 |
golang.org/x/text |
0.40.0 |
0.41.0 |
github.com/minio/minio-go/v7 |
7.0.97 |
7.0.99 |
github.com/minio/mux |
1.8.2 |
1.9.2 |
github.com/cheggaaa/pb |
1.0.29 |
1.0.30 |
github.com/lestrrat-go/jwx/v3 |
3.0.12 |
3.0.13 |
github.com/lestrrat-go/httprc/v3 |
3.0.1 |
3.0.6 |
github.com/grpc-ecosystem/grpc-gateway/v2 |
2.27.3 |
2.29.0 |
go.uber.org/zap |
1.27.1 |
1.28.0 |
Smaller selected updates include uax29 2.3.1, fastjson 1.6.10, secp256k1 4.4.1, and goccy/go-json 0.10.6. The old lestrrat-go/option v1, gogo/protobuf, and stale test-only requirements leave the selected graph.
etcd 3.7.1 is the first fixed release on the 3.7 line for GO-2026-6107 / CVE-2026-73500, an unauthenticated TLS-listener denial of service. Updating these Go client modules does not upgrade an operator’s external etcd server. SILO does not use the removed grpc.WithBlock behavior, and client compatibility with an etcd 3.6.14 server was exercised during release validation.
The dependency graph declares coreos/go-systemd 22.7.0, but the module replaces it with 22.6.0 because 22.7.0 does not compile on NetBSD. The SILO server carries the same portability override.
Compatibility
- Import paths and the module major remain unchanged.
- Existing policies keep loading and evaluating unchanged; only strict create/update calls reject the malformed prefixes.
- No policy, IAM database, wire protocol, or etcd data migration is performed.
- Downstream users of
grpc.WithBlockin etcd client dial options must migrate to a supported readiness check; SILO does not use it. - A real external etcd cluster upgrade remains a separate operational procedure.
- Upstream AIStor Memory and new AIStor-only S3 Tables action vocabularies are not imported by this fork.
Consumers select the release with:
Verification
The tagged package passed:
- the repository’s complete
make testgate: pinned lint plusgo test -race -tags kqueue ./...; - targeted bare-ARN tests repeated to disturb Go map iteration order; and
go mod verify,go vet,git diff --check, andgovulncheckwith zero reachable vulnerabilities.
The SILO integration passed:
- the complete IAM server suite, including exact/historical named-policy rejection and service-account create/update rejection;
go test ./cmd -count=1,go vet ./cmd, andgo test ./...;- golangci-lint 2.13.1 with zero findings,
go mod verify, andmake check-gen; and - mutation tests proving all three strict call sites and their integration assertions are load-bearing.
Direct VCS module resolution verified v3.12.0 at commit 2b087a1 with module checksum:
The Git tag, GitHub Release, direct module archive, origin commit, and checksum were verified.
Related Changes
2bc3a91: move CI actions onto Node 24c8c6872: align the SILO Go dependency stack2b087a1: reject bare ARN prefixes on strict policy writes; taggedv3.12.030c49bd: update the README dependency example after the tageee05a17c: enable strict named-policy and service-account writes in SILO56c67dacf: recordSN-2026-005
Server Release Scope
The server-side strict policy-write integration (eee05a17c) shipped in SILO 20260903. STS inline-policy strict validation remains deferred so machine clients and session-policy templates can be audited separately.
2.23 - Silo Pkg 3.11.0 Released
Release date: 2026-08-04 · Version: v3.11.0 · Commit: d8b1fa7 · Repository: pgsty/silo-pkg
This is the fork’s first pinned release. It restores the IAM bucket/object resource boundary reported as upstream minio/minio#20449: a policy condition-key bypass fix, three LDAP connection defects, a certificate watcher leak, a seeded-RNG defect, and the module’s real minimum Go version.
Two things to check before upgrading
- This release tightens authorization. Twelve bucket-level write actions are no longer reachable through an object-only resource pattern such as
arn:aws:s3:::bucket/*. If you write your own bucket-scoped policies, read The IAM bucket/object boundary — the fix is one line of policy for anyone affected, andMINIO_API_LEGACY_BUCKET_RESOURCE_MATCH=onrestores the previous behaviour in full. - Pair with the server condition-key update. The policy lookup change and the server changes that reserve internal condition-key names each cover one half of the condition-key fix. The companion server work (
2f55347f7) ships in SILO 20260804.
What This Repository Is
silo-pkg is a maintained fork of minio/pkg, carrying fixes needed by community MinIO forks that the now commercially driven upstream no longer accepts. The repository was renamed from pgsty/minio-pkg on 2026-08-02.
The module path intentionally remains unchanged as github.com/minio/pkg/v3. Existing import "github.com/minio/pkg/v3/..." statements do not change; only the right-hand side of the replace directive does:
The /v3 suffix is the module’s major version, not a directory name, and must not be omitted. It is also why this release is numbered v3.11.0 rather than v4.0.0: Go requires the major version of a tag to match the major-version suffix declared in go.mod, so a v4.0.0 tag on a .../v3 module is rejected by the toolchain. Publishing a real v4 would mean changing the module path and rewriting roughly 395 import sites across the server, mc and Console — abandoning the drop-in property that is the point of keeping upstream’s path.
The IAM Bucket/Object Boundary
Every bucket-level S3 operation authorizes with an empty object name. The IAM matcher turned that into a resource string and, for the empty-object case, appended a trailing slash:
"bucket/" is matched by the wildcard pattern "bucket/*", because * matches the empty string. A policy granting s3:* on arn:aws:s3:::bucket/* — which reads as “anything, but only on the objects in this bucket” — therefore also authorized bucket-level actions. In a multi-tenant cluster, a tenant holding only that grant could call PutBucketPolicy and install {"Principal":"*"}, making the bucket publicly readable or writable, or grant itself bucket-level control. It could also delete the bucket outright, which is the reproduction in the upstream issue.
The bucket-policy evaluation path used for anonymous access never had this slash and was already reference-correct. Only the IAM path was wrong, in exactly one place.
Why not correct the whole boundary
Removing the slash for every bucket-level request is the obvious fix, and upstream tried it: the change was reverted the same day for breaking policies that relied on the old behaviour. Two properties make the full correction a migration rather than a patch.
It revokes grants real deployments depend on. It does not only revoke the dangerous bucket writes — it also revokes ListBucket, GetBucketLocation and ListBucketMultipartUploads when granted through bucket/*. The evidence is upstream’s own test suite: eleven STS integration tests grant s3:ListBucket on bucket/* and then assert that listing works. If the project that wrote the server writes it that way, production policies do too.
It cuts both directions. The matcher builds the same resource string for Allow and Deny, so removing the slash tightens over-granting Allow statements and simultaneously loosens over-blocking Deny statements. An administrator who locked a bucket with Deny s3:* on bucket/* would silently lose that protection.
How the protected set was chosen
The scope was decided by one question: does reaching this action give the caller something its object-scoped grant does not already provide?
That question is the right one because of how the defect fires. Resource matching runs after action matching, so the bug only bites when the statement already grants the bucket-level action — which in practice means s3:*. The affected principal therefore already holds full read, write and delete over every object in the bucket. The relevant question is what reaching a bucket action adds to a caller that already controls all object data in the bucket.
Withheld from object-only grants (twelve actions):
| Action | Why it qualifies |
|---|---|
PutBucketPolicy, DeleteBucketPolicy |
Hand access to other principals, anonymous included, and can grant the caller bucket-level actions it was never given. Self-escalation and public exposure. |
PutBucketObjectLockConfiguration, PutBucketVersioning |
Defeat protections that exist precisely to stop a holder of write access from destroying data. |
PutReplicationConfiguration, PutLifecycleConfiguration |
Act under server credentials and keep acting after the caller’s access is revoked. |
DeleteBucket, ForceDeleteBucket |
Destroy the bucket entity and its configuration irreversibly. The reproduction in the upstream issue. |
PutBucketCors, DeleteBucketCors, PutBucketQOS, PutInventoryConfiguration |
No server behaviour is attached to these today — no handler at all, or a handler that returns NotImplemented after the authorization check. Withholding them costs nothing and covers them in advance. |
Deliberately not withheld, and asserted by a test so that adding one is a deliberate act with a visible cost rather than an edit to a list:
PutBucketTagging,PutBucketEncryption,PutBucketNotification. These are bucket-level writes and an earlier draft did withhold them. None gives the caller access it does not already hold — the harm is to the owner’s posture, not to the access boundary — while a tenant handeds3:*onbucket/*and told the bucket is theirs may quite reasonably tag it, set default encryption, or wire up event notifications. Low security gain against a real compatibility cost is the wrong trade for a maintenance release.CreateBucket. It targets a bucket that does not exist yet, so there is nothing to mutate or destroy, and provisioning flows commonly create a tenant’s bucket with that tenant’s ownbucket/*credentials.- The read/list family (
ListBucket,GetBucketLocation, the configuration reads). Breaking these is what got upstream’s own attempt reverted. They wait for a migration-gated release.
Only Allow statements are affected. Deny statements keep the historical resource string, so no bucket lock is ever weakened, and NotResource exclusions keep their full reach.
Monotonicity, and the claim that was wrong twice
All of the above rests on one property: this change may remove permissions and must never add one. That property was asserted twice from reasoning rather than from tests, and was false both times. Recording how is more useful than recording only the final state.
The first attempt let the withheld slash reach the NotResource match as well — and NotResource is an exclusion. An Allow s3:* NotResource bucket/* statement historically did not apply to bucket-level requests on that bucket; matching the exclusion against the bare bucket name made it stop matching, so the Allow it qualified grew, for exactly the writes being protected.
The second attempt fixed that and shipped saying the result was provably monotone. Testing against fixed-width wildcard patterns exposed a counterexample. Withholding the slash changes which string patterns are matched against, and a pattern can match "mybucket" without ever having matched "mybucket/". A fixed-width wildcard is the clean case:
? matches exactly one character. Against the nine-character "mybucket/" it does not match, so this statement never authorized the bucket-level write. Against the new eight-character "mybucket" it does, so the hardening granted something the buggy matcher refused.
On the protected path the matcher now requires both forms to match — the bare bucket name and the historical "bucket/". The result is an intersection with the historical decision, so it is monotone by construction: there is no pattern it can newly satisfy. mybucket* still grants (it matched both all along), mybucket/* is still withheld, and mybucke? is refused as before.
Evidence
The property is verified rather than argued. A decision corpus of 27,000 authorization outcomes — 15 resource patterns × 3 buckets × 5 object names × 20 actions × 6 statement forms — was generated against both the pre-hardening baseline and this release and compared entry by entry:
| Transition | Count |
|---|---|
false → true (broadening) |
0 |
true → false (narrowing) |
144 |
| unchanged | 26,856 |
Every one of the 144 narrowed outcomes falls inside the design intent, with nothing outside it: exactly the twelve protected actions; only the three Allow statement forms, with zero transitions for Deny, NotResource-excluded or deny-NotResource forms; only four object-only resource patterns; and only bucket-level requests, with object-level requests entirely untouched. 12 × 4 × 3 = 144, fully accounted for.
Regression coverage exists at both layers. In this repository, twelve matcher tests pin each direction, including an invariant test that every protected action really is bucket-only — ResetBucketReplicationState, despite its name, is an object action and stays out. In the server, three end-to-end tests drive the real handlers at the client, inline-session-policy and S3-router levels; all three fail against the pre-fix build and pass against this one.
What to change
You are affected only if a stored policy grants one of the twelve actions — or s3:* — on a resource pattern containing /, with no bare bucket ARN for the same bucket. The fix is to add the bare ARN alongside the object pattern:
That pairing is the conventional form, is what upstream’s own tests use, and worked before this release as well. Built-in canned policies are unaffected — readwrite, readonly, writeonly and diagnostics all use Resource: "*".
MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH=on, read once at startup, restores the historical matching in full — both the over-granting and the over-blocking. It is a single global switch; per-action scoping is deferred.
Policy Condition-Key Lookup Order
getValuesByKey() previously looked up a policy condition key by its canonical MIME spelling (http.CanonicalHeaderKey) before trying the original name. The map it reads mixes values calculated by the server for the current request (SourceIp, SecureTransport, CurrentTime, username and others, stored under condition-key spellings) with HTTP headers supplied by the request (stored under canonical MIME spellings).
Checking the canonical spelling first allowed a client header to override a value calculated by the server.
For a MinIO server this is a policy bypass. The simplest example is s3:prefix: a Prefix request header could satisfy a home-directory prefix condition while the real ?prefix= query parameter still listed the entire bucket. The same path reached aws:SourceIp, aws:SecureTransport, aws:CurrentTime, aws:EpochTime, aws:username, aws:userid, aws:principaltype, aws:UserAgent, aws:groups, ldap:username, ldap:groups, jwt:groups, s3:versionid, s3:signatureversion, s3:signatureAge, s3:authType and s3:LocationConstraint. Anonymous bucket policies were directly exposed. SigV4 did not prevent the attack because a client can add a header that is not listed in SignedHeaders.
There was a second consequence: when the server stored a value under one spelling and the policy key resolved another, the wrong entry won. s3:object-lock-mode could resolve to the caller’s X-Amz-Object-Lock-Mode header rather than the retention mode the server would actually apply.
The fix reverses the lookup order: match the condition key’s exact name first, then use the canonical spelling only as a fallback for condition keys that genuinely name request headers, such as the s3:x-amz-* family. This ports minio/pkg#226 and adds regression coverage the upstream change did not carry.
At the library’s raw-map layer, if a producer stores one logical field under both the exact condition name and its canonical MIME name, the exact name now wins. This is a library lookup rule, not an S3 wire-protocol rule that says query parameters take precedence. The Silo server first normalizes condition values by their real source. For storage class and upload tagging, where both Header and query forms remain compatible, Header presence wins, including an empty value; query is only the fallback.
LDAP Connection Path
Three defects in connect(). Two were introduced by this fork in b0c08a7 and shipped in v3.6.2 and v3.6.3. Users of either release should upgrade promptly.
StartTLS was skipped when ServerInsecure was enabled. Upstream called StartTLS in an outer block controlled only by ServerStartTLS, so enabling both options created a plaintext connection and then upgraded it. b0c08a7 moved the call into an else branch, making StartTLS unreachable whenever ServerInsecure was true. The connection stayed plaintext and the following bind sent credentials over it. MinIO exposes MINIO_IDENTITY_LDAP_SERVER_INSECURE and MINIO_IDENTITY_LDAP_SERVER_STARTTLS independently and Validate() rejects no combination, so this state was reachable.
This release restores the upstream semantics: the two switches are additive, not mutually exclusive. ServerInsecure disables implicit ldaps://; ServerStartTLS still performs the upgrade. The exposure window is limited to v3.6.2 and v3.6.3.
A Config without a TLS section could panic on the ldaps:// path. After l.TLS.Clone() moved outside the StartTLS branch, ordinary ldaps:// connections also called it. Clone() returns nil for a nil receiver, but the next line assigned ServerName. The MinIO server always supplies TLS settings, but this is a library and mc also consumes it. The code now falls back to an empty tls.Config, matching what DialURL would have built.
StartTLS had no deadline. go-ldap only starts its request timer when requestTimeout > 0, while StartTLS itself has no timeout. A server that completed TCP setup and then stopped responding to the extension request could hold the connect goroutine forever. The timer is now armed before StartTLS.
A failed StartTLS leaked the connection. Inherited from upstream. Dial failures do not return a connection, making StartTLS failure the only connect() path that could return both a connection and an error. Callers only took ownership when the error was nil, leaving a socket behind for every login attempt against a server with a broken upgrade. The failure path now closes the connection and returns nil.
Other Fixes
- certs: file watchers were never stopped.
Manager.AddCertificate()registered twonotify.Watch()calls and stopped neither: if the second failed, the first leaked, and both survived until process exit after the manager closed.Certificate.Watch()andwatchFile()had the same problem. All four paths now usewatchDirSafe(), which returns a stop function invoked on errors andctx.Done(). This ports thecerts/part of minio/pkg#228. On Windows the function replaces filesystem notification with polling rather than using polling only as a failure fallback, so certificate reload can lag by onesymlinkReloadInterval(10 seconds). This fork has no Windows CI; that platform was only cross-compiled. - rng: reader subkeys came from a zeroed local variable.
init()read 32 bytes of entropy intor.tmpbut derived four subkeys from a same-named zeroed local, collapsing four per-block streams into one.Reset()andResetSize()then replayed the previous stream byte for byte. MinIO creates a new reader for eachrandreader.New()call and never resets it, so the practical server impact is limited; warp exposed the defect. This ports minio/pkg#230. - xtime:
DurationimplementedUnmarshalJSONbut notMarshalJSON. Encoding produced an integer number of nanoseconds while decoding unconditionally stripped the first and last byte and expected a quoted string, so neither direction could round-trip. It now encodes usingtime.Duration’s string form. This ports minio/pkg#242.
Compatibility Impact
- Twelve bucket-level write actions are no longer authorized through an object-only resource pattern. See What to change. Object access,
ListBucket,CreateBucket, bucket tagging, default encryption and event notification are all unaffected, as areDenystatements andNotResourceexclusions. - The minimum Go version moves from
1.26.1down to1.25.0. A patch number in thegodirective is a hard minimum for every consumer, not a record of the toolchain used to build the module. The conventional split is a language version on thegoline and a development version on a separatetoolchainline.1.25.0is what the dependency graph actually requires and what upstream declares. CI builds the complete test suite with Go 1.25 underGOTOOLCHAIN=local, so the minimum is proven rather than aspirational. - The JSON wire format of
xtime.Durationchanges from a nanosecond integer to a duration string such as"2h"or"30m". Persisted numeric values can no longer be read back. No such use was found in MinIO ormc: batch job definitions persist as YAML and the msgp path remains int64. - Deployments with both
ServerInsecureandServerStartTLSenabled whose LDAP server does not support StartTLS connected successfully in plaintext on v3.6.2/v3.6.3 and now fail to connect. That is the correct result, but it surfaces during connection rather than configuration validation. DisableServerStartTLSfor such a server. Policy.IsAllowedActionscan disagree with a direct decision for the twelve protected actions. It enumeratesSupportedActions, which includes thes3:*pattern itself, so the returned set can contains3:*— and therefore appear to permit a protected action — while the direct evaluation denies it. Nothing in the server calls it, and Console calls it with an empty bucket name, which never reaches the hardened branch. Recorded rather than changed, because altering a public API’s output in a maintenance release is the larger risk.
Divergence from Upstream v3.11.0
The version number follows upstream’s line and makes no claim of identical content. The measured delta, comparing action-string constants across policy/:
| Count | |
|---|---|
Upstream minio/pkg v3.11.0 |
291 |
silo-pkg v3.11.0 |
270 |
24 actions exist only upstream: six s3:*ObjectAnnotation* actions, five admin: actions (DistJobStatus, Get/SetBucketCompression, two TablesReplication*), and thirteen s3tables: actions covering function CRUD and tagging. These belong to the AIStor vocabulary this fork deliberately does not carry, because the community server does not implement them.
Three actions are named differently on each side. Upstream renamed and split these; this fork retains the earlier names:
silo-pkg v3.11.0 |
upstream minio/pkg v3.11.0 |
|---|---|
s3tables:TagResource |
s3tables:TagTable, s3tables:TagWarehouse |
s3tables:UntagResource |
s3tables:UntagTable, s3tables:UntagWarehouse |
s3tables:ListTagsForResource |
s3tables:ListTagsForTable, s3tables:ListTagsForWarehouse |
A policy naming any of these six action strings therefore validates on exactly one of the two. Nothing in the Silo server, mc or Console references them, so there is no impact inside this ecosystem — but a consumer swapping upstream v3.11.0 for this release should know the vocabulary is not interchangeable.
rng has no arm64 assembly. Upstream added rng/xor_arm64.{go,s} after this fork’s divergence point; this release falls back to the pure-Go xor_noasm.go path on arm64. The result is correct and cross-compiles cleanly, but slower than upstream on that architecture. It is a clean candidate for a future sync, being a pure performance change with no vocabulary entanglement.
Companion Server Behavior
- The condition-key change in this release must be paired with the server changes that reserve internal condition-key names and populate values by semantic source, as noted at the top.
s3:signatureAgeis exposed only after the SigV4 presigned-request verifier calculates it. A client-suppliedx-amz-signature-ageHeader is ignored on every other request type.s3:prefix,s3:delimiterands3:max-keyscome only from query parameters. Content hash, copy source, metadata directive, SSE and object-lock conditions come only from the corresponding headers. TheX-Amz-Content-Sha256query value consumed while verifying a presigned request does not become a policy condition.s3:x-amz-storage-classretains its compatible query form, as do request tags onPutObjectandCreateMultipartUpload. For both fields, Header presence wins and query is used only when the Header is absent.s3:ExistingObjectTag/*comes only from tags loaded from the stored object, so a request’s ownX-Amz-Taggingcan no longer impersonate existing object state.PutObject,CreateMultipartUploadandPutObjectTaggingbinds3:RequestObjectTag/*to the tag input those handlers consume. Other action paths retain the historicalX-Amz-TaggingHeader fallback for compatibility, so treat request-tag conditions as constraints only where the API actually consumes tags.aws:SourceIpis calculated from forwarding headers. Whether it is enforceable depends on the server’s trusted-proxy configuration; see the server’s own release notes forMINIO_API_TRUSTED_PROXIES.
Verification
Everything below was run against the tagged commit, with the working tree clean and the tag pointing at HEAD:
make test— golangci-lint plusgo test -race -tags kqueue ./..., all packages passing.go mod tidy -diffclean;gofmt -lempty;go vet ./...clean.- Cross-compilation for
linux/amd64,linux/arm64,darwin/arm64andwindows/amd64. govulncheck ./...— zero reachable vulnerabilities. One module-level notice remains, GO-2026-5932 inx/crypto/openpgp; that package is unmaintained, has no fixed version, and this repository does not import it.- Resolution from an empty module cache through the public proxy, confirming the release is fetchable as published.
- The 27,000-outcome authorization corpus described above.
Dependencies and Tooling
Dependency updates clear nine reachable findings previously reported by govulncheck: seven x/crypto/ssh issues reached through sftp, GO-2026-6061 in gRPC reached through etcd, and GO-2026-4945 in go-jose reached through oidc.
Five dependencies — minio-go, minio/mux, etcd client/v3, go-oidc and lestrrat-go/jwx — were deliberately not upgraded. MinIO consumes this module through replace, and Minimal Version Selection chooses the highest version in the entire graph, so upgrading them here would also pull the server forward. None has a reported vulnerability requiring that change.
All three workflows previously asked setup-go for a Go version lower than go.mod required and failed on the first Go command; they are now aligned. The linter also fetched an installer from the master branch and reinstalled it on every run. The URL and version are now pinned to v2.11.3, and a matching installed version skips the download.
Changes Deliberately Not Taken from Upstream
- AIStor policy vocabulary (Memory/cortex, Tables/Iceberg, KMS, compression and annotations) and the typed action-constant refactor, none of which the community server implements. This is the source of the action-vocabulary delta.
securityAuditAdmin, which grantsadmin:ExportIAMand therefore exposes every secret key despite what the name suggests.- rng AVX2/NEON assembly. Revisiting the arm64 half is noted above as a future sync candidate.
net.BandwidthBytesPerSec(declared but never read upstream),replicationAdminandDistJobStatusAction.- Two changes initially taken and removed after review: the
consolereadonlybuilt-in policy andGetAllGlobalCertificates. Neither has a consumer. Once operators bind a built-in policy name to users, withdrawing it is particularly unsafe: policy mappings persist by name, and an unresolved name merges into an empty policy that denies everything. Its inheritedadmin:CreateUserDeny also cannot be combined withiamAdmin. The certificate helper inventoried a cache the community server never populates. - Upstream’s golangci-lint
tooldirective, which would add roughly 200 linter dependencies to every downstream consumer’s module graph.
Deliberately Deferred
The general problem in minio/minio#20449 — that bucket/* still reaches ListBucket, GetBucketLocation, the configuration reads, CreateBucket and the three tenant-plausible writes — is not closed here. Closing it means revoking grants real deployments depend on, so it belongs to a release that carries a migration path.
What that release owes operators is more than a longer action list, because no one can enumerate every deployment’s stored policies — which puts a hard ceiling on any approach that picks the protected set by guessing. Three things raise it:
- A startup policy audit that walks stored policies and names each one whose meaning changes, in both the grant and the deny direction. It is read-only and can ship before the enforcement change rather than with it, turning an upgrade surprise into a pre-upgrade checklist.
- A denial that explains itself. When a request is refused because only an object-scoped grant matched, say so and name the compatibility switch. A break an operator can diagnose in thirty seconds costs an order of magnitude less than a silent one.
- A switch with a scope.
MINIO_API_LEGACY_BUCKET_RESOURCE_MATCHis all-or-nothing today, so an operator who needs one action back must reopen the self-escalation path along with it.
Related Commits
- d8b1fa7: fix(policy): settle the bucket-write hardening’s scope and monotonicity
- 1f97549: fix(policy): extend the bucket-write hardening to every bucket-only write
- 3c24ad1: fix(policy): withhold object-only grants from sensitive bucket writes
- da6a22a: docs: say what this fork is and how to depend on it
- 4055b2f: fix(xtime): marshal Duration as a duration string
- 13c26cd: fix(rng): initialize the reader subkeys from the seeded entropy
- 88b37ac: fix(certs): stop file watchers on every exit path
- 74dd36e: fix(ldap): keep StartTLS when ServerInsecure is also set
- 424c3d0: fix(ldap): close the connection when StartTLS fails
- 045d10f: fix(ldap): guard a nil TLS config and arm the StartTLS deadline
- 5c4bf50: fix(policy): prefer the exact key name over the canonical header form
- 802539f: chore(deps): refresh the dependency set and declare the real minimum Go
- e4ec64a: ci: build on the Go version go.mod requires, and prove the declared minimum
- 747d8b8: build: pin the golangci-lint installer and skip a matching install
2.24 - Silo 20260903 Released
Version date: 2026-09-03 · Published: 2026-09-04 · Version: RELEASE.2026-09-03T13-18-01Z · Previous release: RELEASE.2026-08-06T00-00-00Z
SILO 20260903 is the security and correctness release following the first fully rebranded Silo release. It tightens core server invariants: client headers do not create replication authority, browser origins cannot trigger bucket-metadata I/O before authentication, concurrent bucket-configuration writers cannot silently overwrite one another, and authorization and checksum behavior more closely match AWS S3.
This note consolidates the complete server, Console, shared-package, and client baseline delivered on 2026-09-03 relative to 20260806. Later fixes—including SN-2026-011—shipped in SILO 20260916, SILO 20261015, and SILO 20261018.
Highlights
- Replication trust is authorization-derived.
X-Minio-Source-Replication-Requestand related internal fields gain replication semantics only after signature verification, an exact marker check, ands3:ReplicateObjectors3:ReplicateDeleteauthorization. Untrusted fields are stripped after authentication so SigV4 remains valid. - Per-bucket CORS is complete.
PUT,GET, andDELETE ?corspersist a real bucket policy, override the server-wide fallback, and converge through site replication. The pre-authentication lookup is resident-only and remains fail-closed during startup and after a known metadata-load failure. - Bucket metadata updates are serialized. A shared
metadata.lockcovers all whole-record configuration writers, migrations, imports, site adoption, and healing.ForceCreateno longer erases existing configuration, and an Object Lock document always implies plainEnabledversioning. - Authorization matches the requested operation. Explicit version deletion requires
s3:DeleteObjectVersion; user and group enable/disable calls require the action matching the requested status; policy writes reject empty ARN namespaces. - Checksum behavior moves toward AWS S3. Upload parts can receive server-computed checksums, federated
UploadPartCopyreturns its checksum, multipart completion reportsChecksumType, invalid assertions use AWS-compatible failures, andCRC64NVMEwithCOMPOSITEis rejected. - SSE-C reads and copies authenticate the key consistently. Zero-byte objects and
GetObjectAttributesno longer bypass key validation; null-version rewrites and in-place key rotation no longer leave unreadable ciphertext or inconsistent checksum metadata. - The release was simplified before publication. Dead code and obsolete lint exemptions were removed, compatibility snapshots now track served surfaces instead of source-level noise, dependencies were pinned to reviewed revisions, and flaky timeout tests received a private random source.
- The client release is part of the same baseline. mcli 20260903 adds read-only checksum verification, fail-closed credential redaction, repaired JSON metrics and S3 Select behavior, strict policy writes, signed packages, provenance, and verified amd64/arm64 images.
Security fixes
The repository security ledger assigns stable local identifiers to five findings without CVEs. An SN- identifier is not a CVE. The complete threat models, compatibility effects, and operator actions are collected in SILO 20260903 Security Notes.
| ID | Affected surface | Fixed invariant | Operator impact |
|---|---|---|---|
SN-2026-006 |
SSE-C reads and copies of zero-byte objects | The supplied customer key is authenticated even when there is no payload block to decrypt | Wrong keys now return 403 AccessDenied; no change for correct keys |
SN-2026-007 |
GetObjectAttributes on SSE-C objects |
Object attributes require the customer key, except for an actually authorized replication peer | A bare replication marker cannot skip key authentication |
SN-2026-008 |
Internal replication fields across reads, writes, multipart, deletes, Snowball, and events | Replication semantics require authentication plus the matching replication permission | Ordinary clients carrying internal-looking headers keep ordinary S3 semantics |
SN-2026-009 |
Admin user and group status changes | Enable and disable are authorized against different actions | Split custom admin policies that previously granted only one action but relied on both |
SN-2026-010 |
DeleteObject / DeleteObjects with versionId |
Explicit version deletion requires s3:DeleteObjectVersion |
Review grants and add an explicit Deny where Deny s3:DeleteObject was intended to block permanent deletion |
All five defects were inherited from the archived upstream server lineage and affect every earlier SILO release. SN-2026-008 completes the receiver-wide repair begun by CVE-2026-34204.
The embedded Console v2.3.0 line also contains a trusted-proxy boundary, TLS verification repair, response and log redaction, and bounded WebSocket forwarding.
S3 correctness and interoperability
Multipart checksums
This release closes the multipart interoperability group reported in #46, #47, #48, and #50:
UploadPartcomputes the selected checksum when the client selects an algorithm at upload creation but omits a per-part checksum header.UploadPartCopyreturns the remote checksum through a legacy federated backend instead of dropping it.CompleteMultipartUploadincludesChecksumTypeand uses AWS-compatible error codes when a checksum is absent, malformed, contradictory, or unsupported.- Unknown checksum algorithms and the invalid
CRC64NVME+COMPOSITEcombination are rejected rather than silently canonicalized.
The fixes are deliberately layered. The wire parser rejects invalid declarations, the multipart state records the selected algorithm and type, and completion validates the final claim against that state. Ordinary clients that do not opt into additional checksums are unchanged.
The individual evidence records are server-computed UploadPart checksums, multipart completion error semantics, and ChecksumType propagation.
CopyObject and SSE-C
CopyObject now:
- calculates checksums over the logical object before optional compression;
- returns the checksum fields in its response;
- preserves transform state on metadata-only copies;
- decrypts source checksum metadata with the source key and rewrites it for the destination key;
- handles null-version copies in either pool direction without losing current-version metadata;
- re-encrypts an in-place SSE-C key rotation when the object layer must rewrite the data.
The zero-byte and attributes fixes remove two independent places where “no data was decrypted” had accidentally been treated as “the key was authenticated.” The new rule is explicit: access to SSE-C metadata or ciphertext requires a successfully unsealed object key, unless the request has already crossed the authorized replica boundary.
The CopyObject state matrix and live interoperability evidence are preserved in CopyObject SSE-C, checksum, and response correctness.
Listing and federation
- ListObjects shortcuts now return
NoSuchBucketfor a missing bucket even when a prefix selects an early-return path (#32, PR #37). - Federated
UploadPartCopypreserves the backend checksum in the public response (#72). - Site-replication status counts metadata operations per site instead of multiplying a group-wide result, and validates CORS as an independent configuration field.
The missing-bucket shortcut is documented in ListObjects must prove the bucket before it optimizes.
Per-bucket CORS and request trust
Bucket CORS is implemented end to end:
- the S3 XML type and rule matcher enforce the protocol grammar and multi-rule preflight behavior;
- bucket metadata stores the raw configuration and returns it byte-for-byte on
GET ?cors; - the outer middleware uses the bucket policy when present and the global origin setting only as fallback;
- site replication carries CORS in its own tombstone-aware last-writer-wins register;
- pre-authentication lookup never loads metadata or creates cache entries;
- startup, a known bucket-metadata load failure, deletion, refresh, and on-demand reload all update the fail-closed state explicitly.
The load-failure state is intentionally retained. A pre-signed URL authenticates itself without consulting bucket policy; if the server forgets that a real bucket’s CORS document failed to load, falling back to a permissive global policy would discard the only browser-origin boundary around that otherwise authorized request.
Replication-request hardening shares the same principle. The server authenticates the original request first because internal headers may be covered by SigV4, derives one private trust decision from the authenticated identity and permission, and only then sanitizes an untrusted request clone. The clone shares the original trailer map so streaming checksums remain visible. Snowball workers derive trust per entry rather than inheriting a request-wide privilege bit.
The full design and rejected alternatives are preserved in No I/O Before Auth, No Privilege From Headers.
Bucket metadata and Object Lock
Bucket configuration is stored as one .metadata.bin record but was previously updated through independent locks for policy, lifecycle, encryption, tags, quota, replication, Object Lock, and CORS. Two valid read-modify-write operations could therefore each succeed while the later save silently discarded the earlier field.
This release introduces one bounded metadata.lock transaction around every whole-record mutation. The same lock covers:
- normal configuration saves and deletes;
- legacy metadata migration and import;
- bucket creation,
ForceCreate, and site adoption; - healing and inconsistent-version repair;
- replication receive paths that merge changed fields.
The implementation does not turn the metadata system into a generic transaction framework. It keeps the existing record and parsers, centralizes only the serialization boundary, and narrows replicated updates to fields that actually changed.
Object Lock receives a second invariant. Once a valid lock configuration parses as enabled, versioning is normalized to a plain Enabled document; prefix exclusion and suspended states cannot survive. The decision is based on the parsed configuration, not byte equality with the smallest possible XML document, so a valid Default Retention rule cannot bypass it. Update, read-back, and disk-reload tests cover both malformed prior versioning shapes.
IAM, policy, and configuration behavior
- Explicit version deletion follows AWS
s3:DeleteObjectVersionauthorization while replication deletes retains3:ReplicateDelete. SetUserStatusandSetGroupStatusauthorize the requested target state instead of always checking the enable action.- New and updated named policies and service-account policies reject bare S3, S3 Tables, and KMS ARN namespace prefixes, their historical serialized forms, and statements that combine
ResourcewithNotResource. Stored policies continue to load. - Enabled legacy PostgreSQL or MySQL notification targets must supply a connection string. Startup now fails with a credential-free diagnostic instead of silently discarding all targets.
MINIO_CONFIG_ENV_FILEparsing preserves named targets, quoted values, and supported shell-like assignments without evaluating the file as shell code.
These changes are compatibility tightenings, not format migrations. Review the upgrade checklist before deploying.
Detailed records cover target-state user/group authorization, safe configuration environment-file parsing, and fail-fast legacy database notification migration.
Components
The 20260903 dependency and companion-release line is pinned as follows:
- Go 1.27.1
minio-go/v7: upstream-compatible pseudo-version ending in0e78d3f18efe; the temporarysilo-gofork is retired from the server graphmadmin-go/v3:v3.0.110silo-pkg/v3:v3.13.2, under its owngithub.com/pgsty/silo-pkg/v3module path- embedded Console:
v2.3.0, including the trusted-proxy, TLS, redaction, and WebSocket security work - bundled mcli:
RELEASE.2026-09-03T07-13-05Z, with themccompatibility alias preserved - Helm chart:
7.0.2; use the published server imagepgsty/silo:RELEASE.2026-09-03T13-18-01Zand the mcli imagepgsty/mc:RELEASE.2026-09-03T07-13-05Z
Tagged-source Helm note: the signed Server tag retains the provisional
RELEASE.2026-09-03T00-00-00Zvalue inhelm/silo. Post-release commit4c164907fcorrects the currentmainchart to the published tag. If you render Chart 7.0.2 directly from the tagged source archive, setimage.tag=RELEASE.2026-09-03T13-18-01Zexplicitly; binary and package artifacts are unaffected.
The server deliberately preserves the MinIO-compatible Go module and wire identifiers. Replacing an import target does not rename the public S3/Admin API, MINIO_* configuration, x-minio-* headers, /minio/* routes, or on-disk metadata.
Engineering cleanup
The final release pass removed complexity where it did not protect a compatibility or correctness boundary:
- removed dead encryption helpers, obsolete handler paths, and unused event-target functions;
- replaced a source-level exported-symbol inventory with the smaller served-route compatibility baseline;
- removed the
wait_pipelint exclusion and migrated togomodguard_v2; - consolidated CORS load-failure lifecycle changes behind two helpers instead of open-coding set operations in six paths;
- narrowed site-replication imports to changed metadata fields;
- gave dynamic-timeout tests their own random source so parallel tests cannot mutate a package-global seed;
- retained the shared metadata lock, CORS tombstones, two-level replication trust, and regression tests covering each reproduced failure.
The pre-release verification record, including fixes identified during release testing, is in SILO Server 20260903 Pre-release Review.
Upgrade checklist
Read Upgrading from RELEASE.2026-08-06 before changing production. At minimum:
- grant
s3:DeleteObjectVersionto identities that intentionally delete explicit versions; add its Deny next to anyDeny s3:DeleteObjectintended to prevent permanent deletion; - split custom admin grants for enable and disable operations;
- correct bare ARN prefixes before re-submitting policies;
- add connection strings to enabled legacy PostgreSQL/MySQL notification targets;
- verify applications that explicitly choose checksum algorithms or types;
- upgrade every member of a site-replication group before creating or changing per-bucket CORS;
- snapshot CORS configuration before rollback, because 20260806 does not understand it;
- switch every node of a distributed cluster to the same binary in one maintenance operation; mixed Silo binaries do not form a cluster.
The on-disk object and erasure formats are unchanged; distributed clusters require all nodes to run the same release.
Known issues and explicit deferrals
These items are not fixed by this release:
- Conditional delete (#10, PR #12).
DeleteObjectignores HTTPIf-MatchandDeleteObjectsignores each<Object><ETag>. Both delete unconditionally; see the design analysis (subsequently implemented in SILO 20261015). - Non-CORS multi-site configuration deletion (#77). A peer can restore a policy, SSE, tag, or quota configuration deleted on another site. Single-site deployments are unaffected. CORS is not affected because it uses a tombstone-aware register. Deployments relying on these replicated deletions must verify every site (subsequently fixed in SILO 20260916).
ListMultipartUploadsfilters (#79).prefixbehaves like an exact key match, whilemax-uploads,key-marker, anddelimiterare not fully honored; see the conformance analysis and 2026-10-18 decision (commit4733e1423).- Legacy federation
CopyObject(#99, #100). The legacy federated backend can ignore a requested checksum algorithm and rejects inline source objects. Deployments not using that backend are unaffected (subsequently fixed in SILO 20261015). - Mixed-version site replication. A 20260806 peer accepts but ignores bucket CORS and continues reporting a mismatch. Do not configure CORS until all sites are upgraded.
- Rollback of bucket CORS. 20260806 drops the unknown CORS field if it rewrites the bucket record. Export the configuration first and recreate it after upgrading again.
Open enhancement work such as ILM relocation (PR #60), broader SSE support (#61), operator discovery (#30), NATS target hot reload (#40), and Renovate (#20) is outside this release’s scope.
Verification
The complete local acceptance was run on ebac0ca73bbf251b070bb6df4d8005015841f901:
- full
cmdandinternaltest suites; go test -race ./cmd, passing in 365.448 seconds;- lint with 0 issues, the rebrand/compatibility guard, generated-file checks, and
govulncheck; make verifyacross FS, erasure, distributed erasure, erasure sets, multi-pool, and IPv6 multi-pool: 174 PASS / 0 FAIL.
One subsequent one-line fix, 84e1580a4, clears the CORS load-failure bit after a successful on-demand GetConfig reload. Later dependency work moves the toolchain to Go 1.27.1 and x/crypto 0.56.0. On exact release source 9b11dc9469e650815b775cb47b039610644f5da4, git diff --check, full cmd and internal tests, the complete cmd race suite, rebrand and generated-file guards, lint, govulncheck, and remote Go CI, VulnCheck, Test Release, and Release workflows pass.
The final functional acceptance used four Ubuntu 24.04 arm64 VMs with four XFS data disks each. It covered a coordinated 20260806 → 20260903 upgrade, rollback, and second upgrade; the complete mcli functional script over TLS; 1,004-object upload/download checksum comparison; one-node and quorum-loss behavior; single-drive reconstruction; and two-site replication of objects, versions, CORS, tags, quota, ILM, Object Lock, IAM, delete markers, and offline catch-up. The cluster finished with all 16 drives online and no unrecovered data mismatch.
Container publication exposed one delivery-only pin error after the binary/package Release was public: the first Docker workflow stopped before pushing any image because the tagged Dockerfile selected mcli 20260903 but retained the two 20260901 archive digests. Commits ce2326c94 and 9936a69d8 corrected those pins and added a narrow recovery mode that resolves them from the immutable, attested mcli checksum manifest. Recovery run 33847571968 then built, ran, SBOM-attested, and published both architectures from the original Server release archives. The classic release/latest manifest is sha256:b616a0cf8cb281e7e6bb3c9b1fb53875b4016a2878223925541c18f82d6c5ca3; distroless is sha256:2711cdc5bce0ada353a4871dcda60f48f46414fdd69a45c36c71f6b7ebe1aead.
The published server GitHub release identifies the exact tagged source and verified archives, packages, checksums, SBOMs, and provenance; its classic and distroless images publish matching amd64/arm64 manifests. The separately published mcli release is immutable, contains 19 verified assets, and publishes matching amd64/arm64 release and latest images. Shared package v3.13.2 and Console v2.3.0 retain their own release evidence.
Resolved issue ledger
The tracker was reconciled during release preparation. Core issues are closed; deliberately excluded work remains in its own open issue:
| Tracker | Resolution |
|---|---|
| #102 | Closed by merged PR #103: shared metadata.lock and complete writer/migration/adoption coverage; reproduction-first residual audit split to #105 |
| #58 | Closed by merged PR #104: explicit-version authorization with multi-delete context preservation and least-privilege replication tests; PR #59 closed as superseded |
| #46, #47, #48, #50 | Multipart checksum selection, computation, response, validation, and errors |
| #32, PR #37 | Missing-bucket listing shortcut returns NoSuchBucket |
| PR #57 | CompleteMultipartUploadResult.ChecksumType is present in the selected client/server stack |
Selected changes
1c9a2431fthrough13e6458d9: implement per-bucket CORS and site replication938603458through04b097fd9, plus Snowball follow-ups: authenticate replication semantics and remove pre-authentication metadata I/Of9f9fa6c9through32a1b81e4: reproduce and serialize cross-type bucket metadata updates3b5de82f5,21646eebd: enforce Object Lock’s versioning invariant, including retention-rule documentsf8b598f1dthroughd2d47a41f: align explicit-version delete authorizationb73581b05,474cd5801,74c97d005: authenticate SSE-C zero-byte and attributes reads7fea6d5a5,5d152416d,7e079ff05,d28885d0e: align multipart checksum behaviorc0e715977,e73436c99,ffb70eb37: repair CopyObject transforms, checksums, and SSE-C rotation84e1580a4: clear the load-failure guard after an on-demand metadata reload succeeds
Acknowledgments
This release incorporates reports and proposed fixes from community contributors as well as the fork’s security and compatibility audit. In particular, the checksum, missing-bucket listing, and explicit-version authorization work started from public issues and pull requests.
The complete authorship record remains in CONTRIBUTORS.md.
2.25 - Explicit Version Deletes Now Require DeleteObjectVersion
Starting in Server 20260903 (pgsty/silo#104, tracking issue #58), SILO maps object-delete authorization to the operation the request actually performs:
| Request | Required action |
|---|---|
No versionId |
s3:DeleteObject |
| Version UUID | s3:DeleteObjectVersion |
Explicit versionId=null |
s3:DeleteObjectVersion |
DeleteObjects |
Applied independently to each XML entry |
Previously, SILO required s3:DeleteObject for every delete operation and evaluated s3:DeleteObjectVersion only as an explicit-deny check. A principal holding only DeleteObject could permanently remove a specific historical version, whereas a least-privilege purge principal holding only DeleteObjectVersion was denied.
With this change, a DeleteObject-only principal can perform an unversioned delete or create a delete marker, but receives AccessDenied when targeting a version UUID or null version. A DeleteObjectVersion-only principal can permanently remove a named version, but cannot create a delete marker. Explicit denies and s3:versionid conditions retain normal policy precedence. Multi-object delete (DeleteObjects) evaluates each entry’s own VersionId so a query-string parameter cannot alter another entry’s condition value.
Replication compatibility
Bucket and site replication target policies do not need to add s3:DeleteObjectVersion. Authenticated replication requests establish delete trust through the exact internal replication marker and s3:ReplicateDelete, preserving the deployed s3:DeleteObject + s3:ReplicateDelete receiver contract. An explicit deny on s3:DeleteObjectVersion continues to block replicated version purges.
This separation keeps upgraded replication targets compatible with existing least-privilege policies without granting new standalone delete capabilities to ReplicateDelete-only credentials. Two-site regression tests verified that a target user holding the documented minimal policy (without DeleteObjectVersion) converges both permanent version deletes and delete markers.
Upgrade impact
- Review user, service-account, OPA, and external authorization policies that currently grant only
s3:DeleteObjectfor principals that runmc rm --version-id,mc rm --versions, Console delete-all-versions, or SDK deletes with an explicitversionId, and grants3:DeleteObjectVersionwhere needed. - External authorization plugins now receive a single
s3:DeleteObjectVersioncheck for ordinary named-version deletes instead of the former deny-only check followed bys3:DeleteObject. X-Minio-Force-Deleteprefix cleanup remains gated bys3:DeleteObject.- Wire and storage formats are unchanged.
See Object Deletion for the full permission matrix.
2.26 - mcli 20260903 Released
Published: 2026-09-03 · Version: RELEASE.2026-09-03T07-13-05Z · Source: a2ef95c0
Twenty-eight days after mcli 20260806, this release carries everything the client accumulated since. The toolchain moves to Go 1.27.1 and the shared dependency stack is aligned with the SILO family, with silo-pkg imported under its own module path so go.mod says what the binary is built from. The first genuinely new command since the fork arrives — mcli checksum verify, a read-only audit of stored S3 additional checksums against the bytes an endpoint actually returns. The panic that made --json admin prometheus metrics unusable is fixed, as is the response double-close that could panic after mcli sql had printed valid rows. Every path that can print a credential — --debug traces, admin trace, error messages, JSON error documents, checksum reports — was rewritten fail-closed. The release process changed as much as the code: artifacts come from a GPG-signed tag at the exact tip of main with green CI for that commit, ship LICENSE, NOTICE, and a generated CREDITS in every format, carry Sigstore build provenance, and land in an immutable GitHub release.
The release was built from a2ef95c0, the merge of pgsty/mc#27: 58 reachable commits, 50 of them non-merge commits, across 262 files (+12,742 / −6,566 lines) since RELEASE.2026-08-06T00-00-00Z. The intermediate 20260901 build remains available on GitHub, but this page treats 20260806 as the public documentation baseline and records the complete 20260903 result.
Behavior changes
- macOS 13 is the new floor. Go 1.27 links darwin binaries with a minimum OS version of 13.0; the darwin archives no longer run on macOS 12 and earlier.
mcli --json admin prometheus metricsworks again and returns metric families in deterministic name order. Plain-text metrics output is unchanged.pipewith empty standard input now performs a regular zero-byte PUT instead of a one-part multipart upload. The resulting ETag isd41d8cd98f00b204e9800998ecf8427erather than the<md5>-1form. Non-empty input is unchanged.- Policy writes are validated strictly on the client —
admin policy create,admin user svcacct add/set,idp ldap accesskey create/edit. A bare ARN such asarn:aws:s3:::is rejected (writearn:aws:s3:::*to mean every resource under that prefix); a named policy withoutVersionis rejected; an empty policy is rejected; mixing S3 and admin actions in one statement is rejected. The last three were already refused by the server, so only bare ARNs change what can be stored. Reads of existing policies remain permissive. Session policies for service accounts may still omitVersion, andadmin user svcacct set --policystill accepts an empty document — that is the way to clear an inline policy and return the account to its inherited one. - A malformed global flag before the command name is now an error.
mcli -H "bad" ls,--resolveand--limit-*values that fail to parse at the app level used to be silently ignored and the command ran without them. They now fail with the same message the command-level form produces, reported once on stderr — or as a JSON error document under--json— where the CLI library used to echo the text to stdout first. - Error messages no longer echo secrets.
alias setreports only the length requirement for a bad secret key;MC_CONFIG_ENV_FILEparse errors name a line number instead of repeating the line; SSE-C key errors withhold the key;admin config setmasks secret-bearingkey=valuepairs; remote replication targets are printed without credentials; a malformed--custom-headerentry is reported by position, never by value. - Support artifacts are private. Files written by
support diag,support profile,support perf,support inspect,admin cluster bucket export, andadmin cluster iam exportare created with mode0600, including the rotated backup of a previous file. They were0644or0666before. --debugshows less. AnAuthorizationheader keeps only its scheme (AWS4-HMAC-SHA256 **REDACTED**); tokens, cookies, API keys, SSE-C keys and every--custom-headervalue are redacted; 2xx response bodies are no longer dumped.admin traceoutput is redacted the same way in both its default and its verbose rendering: header values by name, query strings, bodies, error text and annotations.- The User-Agent reports
minio-go/v7.3.0(the SDK’s version constant) instead ofminio-go/v7.0.98.
Major Changes
- Go 1.27.1 across the module, container build stage, Linux/macOS/Windows CI, cross-builds, tidy checks, and vulnerability analysis. The Go-maintained module family is current:
x/net0.58.0,x/sys0.47.0,x/term0.45.0,x/text0.41.0,x/crypto0.56.0,x/mod0.40.0,x/sync0.22.0, andx/tools0.49.0.golangci-lintstays outside the product module graph and is installed independently at the pinned v2.13.1. - The SILO package under its own module path.
go.modnow requiresgithub.com/pgsty/silo-pkg/v3v3.13.2 directly. The client’s policy validation depends on helpers that exist only there —policy.ParseConfigStrictandpolicy.Resource.IsBareARN— and areplaceof upstreamminio/pkgwould not be inherited by projects consuming this module (pgsty/mc#12), so the requirement is declared honestly. Upstreamminio/pkgsurvives only as an indirect dependency ofcolorjson, held at v3.6.1 by the dependency-floor check, andgomodguardblocks the upstream import path from returning.minio-gomoves from v7.0.99 to the latest upstreamminio/minio-gomaster commit, represented byv7.3.1-0.20260828014306-0e78d3f18efe; the interimpgsty/silo-gofork is retired. The final refresh includes etcd 3.7.1,go-pretty6.8.3,klauspost/compress1.20.0, Prometheus client/common/procfs 1.24.1/0.71.0/0.22.0,grpc-gateway2.30.0, gRPC 1.83.2, Protobuf 1.36.12, and JWX 3.2.0.go-systemddeliberately stays at 22.6.0 through SILO’s portability pin — 22.7.0 does not compile on NetBSD. - Runtime version floors are guarded. A first dependency pass let the removal of development tools silently lower a dozen runtime selections — among them
shoenig/go-m1cpu, whose older releases crash during initialization on newer Apple Silicon when CGO is enabled — and was rejected.buildscripts/check-dependency-floorsnow compares every requirement ingo.modagainst the previousRELEASE.*tag and fails the build on a regression; the only permitted exception is thego-systemdpin. mcli checksum verify(design). Verifies stored S3 additional checksums — CRC32, CRC32C, CRC64NVME, SHA1, SHA256, of typeFULL_OBJECT— against the logical object bytes returned by the object API, for one object, a prefix with--recursive, all versions with--versionsor an exact--version-id, or a JSON Lines candidate manifest. It reads withIf-Matchon unversioned objects and re-stats afterwards, so a write that races the verification is reported asUNKNOWN_OBJECT_CHANGEDrather than as a mismatch. Each object yields one ofMATCH,MISMATCH,NO_CHECKSUM,WOULD_VERIFY(dry run), one of tenUNKNOWN_*classifications, or one of threeSKIPPED_*results; the summary carriesobjects, averifiedcount (MATCH+MISMATCH, the only results that actually recomputed a checksum), per-status counts andincomplete.--fail-onacceptsmismatch,unknown,no-checksum(fails when any object lacks a checksum or when nothing was verified at all),any(default: mismatches, unknowns, and objects skipped by--max-size), ornone.--reportwrites the same records to a new0600JSON Lines file. Results go to stdout whether or not stdout is a terminal, and with--jsonthey are compact JSON Lines outside a terminal — the non-TTY silence reported as pgsty/mc#5 is fixed. Dry runs,--max-workers,--max-size, time filters, and SSE-C through--enc-care supported. The command is verified against a real SILO server in CI.- Credential redaction, fail-closed. Earlier pattern-based redaction only caught expected credential shapes and let edge cases through: access keys containing a slash, an
Authorizationvalue whose first field is empty so the second one hides, non-hex signatures, secrets tucked into the credential scope orSignedHeaders,Proxy-Authorization, cookies,X-Api-Key, response trailers, a redirectLocationwith userinfo, a server that reflects request headers into an error body, a JSON-escaped secret inside--jsonerror output, or a short token that appears twice. The final design has three rules and no exceptions. AnAuthorization-class value keeps only its scheme. A header whose name looks like a secret, or that the caller supplied through--custom-header, is redacted without inspecting its content. Text the client did not generate itself — response bodies, server error messages, trace notes — is scanned for credential shapes and then scrubbed of every secret literal the process has learned. That last layer is backed by a process-wide secret registry: alias secret keys and session tokens,MC_HOST_*values, SSE-C keys in every encoding, tier and service-principal secrets, GCS credential files, LDAP passwords and the STS credentials they return, SUBNET tokens,--secret-key/--account-key/--api-keyflags, and secret-bearingkey=valuearguments are registered as they are read, before validation and before any network call. Occurrences are collected as intervals and merged before replacement; JSON error documents are scrubbed on the decoded value tree, string leaves only, so keys and compact non-terminal formatting survive. Over forty edge-case test vectors, a black-box test that runs the real client against an endpoint that reflects everything it receives, and live captures against a SILO server pin the result. admin traceredaction. Server-supplied header maps for other clients’ requests take a path the S3 tracer never touched. Both the human-readable and the JSON rendering, in the default and the verbose form, are now produced from redacted copies — the event is shared between them and never mutated. Query strings (a presigned URL’s signature, a session token in the first parameter), bodies, error text and annotations are scrubbed with the event’s own header values, so a token another client sent cannot reach privileged trace output. Headers are redacted by name: a--custom-headeranother client sent under a name that does not look like a secret is shown as the server saw it.
Fixes
- JSON metrics no longer panic.
mcli --json admin prometheus metricsfailed for both the v2 and v3 metrics APIs withpanic: Invalid name validation scheme requested: unsetin the 20260806 client.madmin-gobuilt the zero value of the Prometheus text parser, whichprometheus/common0.66 and newer reject. The client now constructs the parser with UTF-8 validation, returns ordinary parse errors instead of panicking, and emits families in name order. pipewith empty input performs a regular PUT; the one-part multipart upload it used to issue for zero bytes was needless and produced a multipart-style ETag.- RPM payload validation aligned with the signing script, so an RPM whose contents do not match the release metadata is refused before it is signed.
sqlno longer panics after printing its rows. The SDK’s select reader closes the HTTP response itself on every terminal event, and the client closed it a second time; on a zstd-compressed response the two drains raced and about two in five queries against a SILO server ended inpanic: invalid memory addresswith exit status2after the rows had already been written (pgsty/mc#23). The client now cancels the request and drains the result pipe instead of closing the response again; 820 consecutive queries on a live server ran without a panic where the previous client failed 21 of 50.
Hardening
- Operator-produced files are no longer world-readable.
support diagwarned that its archive may contain sensitive environment information and then wrote it0666; the sharedmoveFilehelper relaxed every0600temporary file to0644on its way to the final name, and a0644left by an earlier client was carried into the rotated backup even when source permissions were preserved. Every support artifact is now created0600and chmodded before the first byte is written. Test_FullSuitefails on panic. The functional suite recovered from panics to run cleanup and then reported success; a panic anywhere in the suite, including in the binary build, was invisible to CI.- A leftover
play/alias in the new command’s help — the only file in the repository still advertising MinIO’s operated demo alias — is replaced, and the brand gate now rejects bare aliases in command examples, not only hostnames. - AGPL notices completed on six fork-authored files that carried only the first half of the license text.
Engineering and Delivery
- Third-party notices in every artifact.
CREDITSis generated bybuildscripts/gen-credits.shfrom the union of the runtime modules linked into all six GoReleaser targets and is compared in CI so a dependency change cannot leave it stale.LICENSE,NOTICE, andCREDITSship in the GoReleaser archives, the DEB/RPM/APK packages, and every container image under/licenses/. - CI runs on
main.GoandCrosscompileran only on pull requests, so the merge commit — the one that gets tagged — was never tested; both now run on every push tomain. Their concurrency group was keyed ongithub.head_ref, which is empty outside pull requests, so every manual run cancelled every other; it is now keyed on the pull request number with the ref as fallback.VulnCheckalso runs weekly, because a fork of an archived upstream can go weeks without a commit. The functional suite in CI now targets SILORELEASE.2026-08-06T00-00-00Z. - Release only from signed tags. The artifact workflow has no manual trigger; artifacts come from a pushed
RELEASE.*tag, and the event SHA, the commit the tag points to, and the checkout must all agree.verify-release-tag.shrequires an annotated tag with a valid OpenPGP signature from the expected key — fingerprint and key location come from repository variables rather than from the tree the tag points at, and the imported key must match the expected fingerprint.check-release-commit.shrequires the tagged commit to be the exact tip oforigin/mainand demands a successful push-to-mainrun of four workflows for that SHA, matched by workflow file path; pull request and manual runs are not evidence.check-release-state.sh(pgsty/mc#6) runs serially per tag, replaces exactly one existing Draft from scratch, refuses a published release, and re-checks for a single Draft before packages are uploaded. Every one of these decisions lives in a shell script with fixture coverage; the fakeghin the tests pins both the verdicts and the arguments passed, verbatim. - Provenance and immutability. Archives, checksums, DEB and APK packages carry Sigstore build provenance (
gh attestation verify <file> --repo pgsty/mc). RPMs are excluded on purpose:sign-release-rpms.shrewrites their bytes with a detached GPG signature from key9592A7BC7A682E7333376E09E7935D8DB9BD8B20while the release is still a Draft, and an attestation made earlier would name a digest no published RPM has. GitHub releases are immutable once published; fixing an asset means a new tag. Thepgsty/mccontainer image is built only from attested archives, aftergh attestation verifybinds each one torelease.yml@refs/tags/<tag>, the release commit digest, GitHub-hosted runners, its own filename and its SHA-256; the unpacked binary’s ELF machine type must match the archive’s architecture; each per-architecture image is pulled, run, and checked for the tag before the manifest is assembled, andlatestis written only from the two-architecture manifest. The UBI base image is pinned by digest and kept current by Dependabot. - Repository governance matches a single-maintainer project. Direct pushes to
mainare allowed; the stale pull-request ruleset and its exact-name status-check coupling were removed after this release. Release integrity does not depend on that branch rule: the artifact gate still requires successful push-to-mainGo, Crosscompile, VulnCheck, and Test Release runs for the exact tagged commit.RELEASE.*tags remain administrator-created and immutable; secret scanning, push protection, Dependabot alerts, and security updates remain enabled. - Verified before and after tagging. On exact source
a2ef95c035d9ae7cc01469a63926900f1786f9e2, tidy/verify, dependency floors, unit tests, the full race suite, lint, actionlint, source-awaregovulncheck, nine standard cross-builds, a clean-cache Go proxy/sumdb build, 172 functional checks against a temporary SILO server, a 47-command black-box comparison with the published 20260901 binary, and seven additional cross targets all passed. All four exact-SHA main workflows and both publication workflows passed. The immutable public release contains 19 assets; archives and sidecars were checksum-verified, all 11 attested subjects were bound to the tag and source commit, both RPMs were GPG-verified, and thepgsty/mcrelease andlatestimages were independently confirmed as matching amd64/arm64 manifests. The source scan reports no reachable vulnerability.
Compatibility
The full --help tree of this release was diffed against the 20260806 binary: the only difference is the new checksum command. Side by side on the same server, head, cat, get, cp, mirror, find, share, sql, stat, and ls produce identical output and JSON field sets; twenty real-world policy shapes are accepted or rejected identically except for bare ARNs; admin policy info --json is byte-identical. The mc command name and the mcli package name; the ~/.mc / ~/.mcli configuration directories; the github.com/minio/mc module path and all import paths; MC_* environment variables; protocol headers (x-minio-*) and the minio-go user-agent prefix; the minio tier type; the .part.minio transfer suffix; the minio-job scrape job name; and the package formats, asset naming, and YYYYMMDDHHMMSS.0.0 version scheme are unchanged. The client remains compatible with MinIO servers and other S3-compatible endpoints. Projects that consume this module as a Go dependency now resolve github.com/pgsty/silo-pkg/v3 directly and need no replace of their own for it.
Known issues and boundaries
checksum verifyreportsUNKNOWN_READ_ERROR, notMISMATCH, when the server’s own integrity checks refuse a read — SILO’s bitrot protection rejects a corrupted shard before the object API returns it. AMISMATCHmeans the stored checksum describes different bytes than the object API returns, which is the historical write-path defect the command exists to find; anUNKNOWNis never reported as aMATCH.- Access key IDs are redacted wherever they appear inside a credential structure, but a server that reflects a bare access key into an error message will still have it printed: it doubles as the user identifier in legitimate messages. Secrets shorter than three characters are not registered. When the secret key equals the access key — the default
minioadmin:minioadmin— the identifier is redacted from error text too, because the string is the secret. --jsonerror documents list the keys ofcause.erroralphabetically rather than in struct order; the keys and values are unchanged.mcli sqlreports a failed query on stderr but still exits0(pgsty/mc#25); per-object errors are reported without failing the run so a recursive query can continue. Unchanged from upstream.govulnchecktraces GO-2026-5932 in the indirect dependencyx/crypto/openpgp; no fixed version exists and the client does not call it.- JWX 3.2 enforces JWT
critprocessing more strictly, as required by the standard. The terminal-width refresh can also adjust alignment for Indic or ZWJ-heavy text; neither change affected the tested command surface. - minio/mc#5139 continues to apply:
mirror --remove --watchcan delete a live object from the target when a non-current version of it is removed from the source. - Quality debt that does not block the release — archive reproducibility, test-tool portability, deferred coverage of KMS / SSE-S3 / multi-site replication — is tracked in pgsty/mc#21.
Related Commits
- 98a8b0f: build: prepare the 20260903 dependency release
- 6686b64: chore(deps): align the SILO Go dependency stack
- 423490d: fix: stop JSON metrics from panicking
- 14b05e6: build: generate and package third-party notices
- e938cea: fix: align RPM payload signing validation
- 8158605: release: adopt Silo Go SDK v7.3.1
- d6ba236: ci: allow manual release workflow runs
- b6b2655: ci: release an existing tag on dispatch
- 70a2950: fix: preserve embeddable pkg compatibility
- aa07b1c: feat: add read-only object checksum verification
- 1b39423: fix: preserve checksum output in pipelines
- a31a42b: docs: record checksum verify output contract
- 2121081: fix: validate policy writes strictly
- 712bf3a: fix: use regular PUT for empty pipe input
- 9a3805d: ci: make release retries tag-idempotent
- 7bc9843: test: make report permissions portable
- 2d5abc2: chore: complete the AGPL notice on fork-authored files
- f5c5dc9: build: import the SILO package under its own module path
- a8e7eb4: ci: test main on push and stop dispatch runs cancelling each other
- b75d28b: fix: stop checksum verify help from advertising the play alias
- c05da7b: chore: drop configuration for bots that are not installed
- a29b455: fix: stop –debug from printing credentials
- 08244df: fix: keep secrets out of error messages and traces
- 30a7ea0: fix: stop writing support artifacts world-readable
- 14bd724: test: fail the full suite when it panics
- 4abbb3a: feat: expose how much checksum verify actually verified
- f484d0d: ci: make a release prove its commit, and pin what it builds from
- 2c33a4e: chore: ignore the local consumer-module review scratch directory
- bdc742a: fix: close the remaining –debug credential leaks
- 3998392: fix: withhold SSE-C keys and credential URLs the parser rejects
- c1358c4: fix: force 0600 on rotated support artifacts too
- 45e5144: fix: fail –fail-on no-checksum when nothing was verified
- 3674933: ci: make the release gate provable and its claims true
- 8b7b2e9: test: exercise checksum verify against a real server
- 4b36b0a: fix: make credential redaction fail-closed and scrub final output
- a6c14da: ci: verify build provenance before repackaging archives into images
- 915cfd9: fix: reject a malformed app-level global flag instead of ignoring it
- 81e2767: fix: rebuild SigV4 traces from known fields and widen secret scrubbing
- 625713c: fix: keep only the scheme of a signed header and scrub JSON structurally
- ec12b64: fix: redact admin trace output and register every credential ingress
- 1445e88: ci: release only from a signed tag at the tip of main, verify exact provenance
- f188939: fix: close final credential and release verification gaps
- f5b88dd: fix: redact the default admin trace and stop the shape sweep mangling prose
- 59989f8: fix: let svcacct set clear a policy again, report bad global flags once
- d38412a: fix: classify an SSE-C refusal with a key as a read error, keep policy and size messages clear
- b291b53: fix: register a URL password as written, keep listing parameters visible in traces
- 9dfca8f: test: drive the SSE-C refusal classification through the verify path
- 76b2e89: fix: keep an unquoted DSN password whole and match auth schemes case-insensitively
- 328efee: fix: register the password before the first separator of an unquoted DSN payload too
- 7057705: fix: avoid double-closing S3 Select responses
2.27 - mcli 20260806 Released
Published: 2026-08-06 · Version: RELEASE.2026-08-06T00-00-00Z
Two days after mcli 20260804, this release completes the client’s transition to the Silo identity. It is deliberately a pure rebranding and lockdown release: --version and --help now present the Silo client, every remaining path to MinIO’s SUBNET service is disabled at build time, the embedded vendor encryption key is removed from the diagnostics tooling, and the contribution policy moves to no-CLA with a mandatory DCO sign-off. There are no dependency changes and no protocol changes in this cycle — go.mod is byte-for-byte identical to 20260804 — so the regression surface is confined to text, command gating, and CI.
Behavior changes
Every path that previously reached MinIO SUBNET is now disabled at build time and cannot be re-enabled at runtime:
mcli license register,mcli support upload,mcli support proxy set,mcli support callhome enable, and the online-renewal form ofmcli license update ALIASprint a stable notice — “MinIO SUBNET services (registration, licensing, uploads) are disabled in this Silo build of mc; diagnostics remain available locally.” — and always exit1. Drop these calls from scripts. The file-basedmcli license update ALIAS license.keystill works, with the license parsed offline against the bundled public key.mcli support diag/perf/profile/inspectalways operate in local (airgap) mode: reports, profiles, and inspect archives are written to local files and nothing is uploaded anywhere. The--airgapflag is still accepted for compatibility and is effectively always on. SUBNET registration is no longer a prerequisite for any of them.mcli support callhome disable|status,mcli support proxy show|remove,mcli license info, andmcli license unregisterkeep working — they only read or clear local and server-side configuration.- Fresh configurations no longer seed the
playalias pointing at MinIO’s public demo cluster; the defaults are nowlocal,s3, andgcs. Existing configuration files are never modified, and legacy-config migration still recognizes the historical entries. mcli --versiongains an identity line (“Silo object storage client, based on MinIO technology”) and a second copyright line. The first line’s machine-readable format is unchanged, so scripts parsing it are unaffected.
Major Changes
- Silo identity across the CLI: the client introduces itself as “Silo client for object storage and filesystems”. Roughly 220 help texts were reworked: usage lines that refer to the managed server now say “Silo/MinIO server”, example aliases moved from
myminio/playtomysilo, example LDAP DNs moved todc=example,dc=com, and example tier names toSILOTIER-*. Factual references stay factual: theminiotier type, protocol headers, and third-party interop mentions are untouched. - SUBNET disabled at build time: connectivity is compiled out behind a single guard, and the one HTTP choke point that every SUBNET request funnels through refuses with the stable error above. Command entry points gate early, diagnostics force local mode, and the AGPL license notice shown by
mcli license infono longer carries a commercial-subscription pitch. A dedicated regression suite (cmd/subnet-disabled_test.go) pins all of this, so an upstream merge cannot silently reconnect anything. - Governance — no CLA, DCO required: contributions are accepted inbound=outbound under AGPL-3.0-or-later; contributors keep their copyright, and the maintainers collect no rights beyond the project license. Every commit must carry a
Signed-off-bytrailer, enforced by a new CI workflow that matches the trailer against the commit author’s email and exempts only GitHub-issued bot addresses.CONTRIBUTING.md, the PR template, and both READMEs document the policy, and the code-of-conduct contact now points at the fork’s maintainer. - Dual copyright attribution: runtime output and help now credit both lineages —
Copyright (c) 2015-2025 MinIO, Inc.andCopyright (c) 2025-2026 PGSTY— with source builds computing the end year dynamically.NOTICEstates the fork relationship, and the non-affiliation with MinIO, Inc., explicitly. - Release line renamed to
main: workflow branch filters, documentation, and contributor instructions now targetmain; the legacymasterreferences are gone.
Hardening
- Vendor encryption key removed:
mcli support inspectused to fall back to encrypting its output with an embedded MinIO RSA public key whenever no key was supplied — producing archives only the vendor could decrypt. The embedded key is gone: inspect now relies on the server-generated per-request key that is printed to the caller (or an operator-supplied key), and any encrypted-upload path with no configured recipient fails closed instead of silently borrowing a third-party key. Diagnostic output an operator produces is now always decryptable by that operator. - Brand-policy gate:
buildscripts/check-branding.shruns inmake verifiersand in CI. It fails the build if MinIO-operated endpoints, commercial upsell URLs, the upstream product identity, or any embeddedMII…public key reappear in the command tree — while explicitly allowlisting the preserved compatibility identifiers (environment variables, protocol headers, module path, legacy-migration defaults, and original copyright headers).
Engineering and Delivery
- CI moved to the Node 24 Actions line:
actions/checkoutv7,actions/setup-gov7,goreleaser-actionv7, and the Docker action family — all still pinned to commit SHAs, with dependabot keeping the pins current. - Functional tests run against controlled servers only: the suite defaults to a local server (
localhost:9000) instead of MinIO’s public demo cluster, and CI downloads the pinned SILO server releaseRELEASE.2026-08-04T00-00-00Zfrompgsty/silo, verified by SHA-256, before running the suite. - Zero dependency changes: no module updates this cycle; the 20260804 security baseline (Go 1.26.5, zero known reachable vulnerabilities) carries over unchanged.
- Pre-release verification: verification covered the 245-file diff, brand/compatibility grep sweeps, call-graph checks confirming that no command or flag combination can reach
subnet.min.io/play.min.io/dl.min.io, and smoke tests confirming every disabled path returns its stable error with exit status1.
Compatibility
Everything scripts and integrations depend on is deliberately unchanged: the mc command name and the mcli package/binary name; the ~/.mc / ~/.mcli configuration directories (derived from the invoked name); the github.com/minio/mc module path and all import paths; MC_* environment variables; protocol headers (x-minio-*) and the minio-go SDK user-agent prefix; the minio tier type; the .part.minio transfer suffix; the minio-job Prometheus scrape job name; and the package formats, asset naming, and YYYYMMDDHHMMSS.0.0 version scheme. The client remains fully compatible with MinIO servers and other S3-compatible endpoints.
Known issues
The mcli watch regression flagged in the 20260804 notes is resolved on the server side: the fix shipped in SILO 20260804, and this client’s CI now runs the functional suite — including watch — against exactly that release. Pair mcli with SILO server 20260804 or newer to receive bucket events; older published servers remain affected.
Unfixed upstream defects continue to apply, most seriously minio/mc#5139: mirror --remove --watch can delete a live object from the target when a non-current version of it is removed from the source. Exercise caution combining --remove --watch on versioned buckets.
Related Commits
- 8a883ca: ci: move the branch filters to main and fetch the server from pgsty/silo
- 8c304dd: ci: move the pinned actions onto the Node 24 runtime
- 02b1c11: docs: name the release line main, not master
- 810bbd2: ci: pin the functional-test server to a release whose watch API works
- d145647: fix: disable SUBNET connectivity and licensing upsell paths
- 5061c4f: rebrand: adopt Silo identity in CLI help and examples
- c7f7706: docs: align governance files and package metadata with the fork
- 65c71b2: test: default functional tests to a local server and add brand gate
- c62a64d: fix: credit both MinIO and PGSTY in copyright notices
- d205f88: docs: adopt no-CLA plus DCO contribution policy
- 95326ce: docs: add related-projects table and polish contribution wording
- d2c0db7: fix: remove the vendor encryption key and close the proxy-set path
- 0c6704d: fix: repair a link and help text damaged by the brand sweep
2.28 - Silo 20260806 Released
Version: RELEASE.2026-08-06T00-00-00Z · Commit: 3be10fcc1a44f6620ded0bd303461f9d688cca23
SILO 20260806 is the first release published under the Silo name. The previous release, 20260804, was the last one delivered as pgsty/minio; this release completes the cutover to github.com/pgsty/silo and renames every delivery surface — binary, packages, container images, systemd unit, Helm chart — while deliberately preserving every wire and configuration surface a MinIO deployment depends on. On top of the rename it adds native health checking (silo healthcheck), a single-binary distroless container image pilot, complete license-compliance materials in every artifact, and a release pipeline gated on compatibility snapshots and build provenance.
The release covers 28 commits after RELEASE.2026-08-04T00-00-00Z, changing 396 files with 27,188 insertions and 19,561 deletions. It passed a six-phase pre-release acceptance, including a real four-node TLS cluster migration from MinIO to Silo — with byte-verified data integrity, maintenance-gated rolling restarts, fault injection, and a full rollback rehearsal.
Highlights
- The rebrand is complete, and compatibility is the contract. Repository, binary (
/usr/bin/silo), packages (silorpm/deb/apk), images (docker.io/pgsty/silo), and service (silo.service) are renamed; the S3 and admin APIs,/minio/*routes,MINIO_*environment variables,x-minio-*headers, on-disk.minio.sysformat, and Go module paths are all preserved and frozen by a CI compatibility guard. - Native health checking:
silo healthcheck [live|ready|cluster|cluster-read]probes the server’s own health API with correct exit codes, decoded quorum diagnostics, TLS auto-detection, and a--maintenancepre-drain gate — no shell,curl, ormcrequired in the container. - Distroless image pilot:
pgsty/silo:<RELEASE>-distrolessships exactly one program — thesilobinary — ongcr.io/distroless/static, with an exec-formHEALTHCHECKbaked in and/datacreated writable in the image layer. - The classic image does not change behavior: same entrypoint, same bundled tools,
mc ready localkeeps working, and noHEALTHCHECKwas added to it. It now bundlesmcli20260806. - Compliance completed: LICENSE and NOTICE ship in every package and image, CREDITS is regenerated from the actually-linked module set (291 modules) and guarded in CI, and the project adopts a no-CLA, DCO-based contribution policy.
- Components refreshed: embedded SILO Console 2.1.1,
silo-pkg3.11.0,mcli20260806, Go 1.26.5. - Provenance-gated releases: container images are built only from published, checksum- and attestation-verified release archives; image SBOMs and provenance attestations now cover the distroless variant too.
The rename
What changed, and what deliberately did not:
| Renamed (delivery surface) | Preserved (compatibility surface) |
|---|---|
Repository: github.com/pgsty/silo (main branch) |
S3 API, admin API, and request signing behavior |
Binary: /usr/bin/silo |
/minio/* routes, including /minio/health/* and metrics |
Packages: silo-*.rpm, silo_*.deb, silo_*.apk |
MINIO_* environment variables and x-minio-* headers |
Images: docker.io/pgsty/silo (+ -distroless) |
On-disk format (.minio.sys), erasure coding, versioning |
Unit: silo.service (conflicts with, and supersedes, minio.service) |
Go module and import paths (github.com/minio/...) |
Default config dir: ~/.silo (falls back to an existing ~/.minio) |
mc compatibility alias for the bundled mcli |
The server presents its own identity — silo --version reports the AGPL-3.0 license, MinIO’s 2015-2025 copyright, PGSTY’s modification copyright, and the “based on MinIO technology” attribution — and every inherited connection to MinIO-operated services (the update feed and its signing key, SUBNET, telemetry) is severed rather than redirected. The container entrypoint translates the legacy minio argv token, so docker run pgsty/silo minio server /data keeps working.
A snapshot-based rebrand guard runs in CI: it fails on any drift, in either direction, across 334 route literals, 437 environment tokens, 84 headers, and 9,014 exported symbols.
Native health checking
The server binary can now probe its own health endpoints, which makes container health checks possible without any second binary — and is what the distroless image relies on:
- The check vocabulary maps 1:1 onto
/minio/health/<path>;live(the default) answers “is this process serving,”readyadds KMS/etcd reachability when configured, and theclusterpair evaluates write/read quorum across every erasure set. - Exit codes are
0(healthy) and1(anything else) — never the Docker-reserved2. One diagnostic line decodes the server’sx-minio-server-statusand quorum headers fordocker inspect;--jsonemits a machine-readable verdict. - The probe target is derived the way the server derives its own listen address:
--address/MINIO_ADDRESS, with HTTPS auto-detected frompublic.crt+private.keyin the certs directory, or overridden wholesale with--url/MINIO_HEALTHCHECK_URL. The environment form exists because a probe process cannot see the server’s command line — if the server’s address or TLS comes from CLI arguments, one environment variable redirects the baked-in probe. silo healthcheck --maintenance clusteranswers the pre-drain question: exit0means the node can be taken down without losing HA; HTTP 412 (exit1) means it cannot.- Certificate verification is skipped, matching the kubelet’s documented behavior for HTTPS probes, and the transport ignores
HTTP_PROXYso loopback probes never route through a proxy.
Kubernetes needs none of this — kubelet httpGet probes hit /minio/health/live and /minio/health/ready from outside the container — and the cluster checks should stay out of per-container probes: they reflect cluster-wide quorum, not one process. The full design rationale, including verified endpoint semantics, is recorded in the health-check design note.
Distroless image pilot
Alongside the classic image, this release publishes a distroless variant: pgsty/silo:RELEASE.2026-08-06T00-00-00Z-distroless, plus a rolling distroless tag.
- Base is
gcr.io/distroless/static-debian12: CA certificates, tzdata,/tmp, and an/etc/passwdwith anonroot(65532) entry — no shell, no package manager, no libc. On top of it, exactly one program:/usr/bin/silo(plus the license set under/licenses/). The image is 128 MB versus the classic 199 MB. - The binary is the
ENTRYPOINT; an exec-formHEALTHCHECKrunningsilo healthcheck readyis baked in (interval 30s, timeout 10s, start-period 2m, retries 3), so Compose users get workingdepends_on: condition: service_healthywith zero configuration. /datais created in the image layer, world-writable — there is no entrypoint left to repair volume ownership at runtime, and this is what makes every privilege mode work,--userincluded. This fixes, for the distroless variant, the non-root failure documented in #55.- Not supported in this variant: the deprecated
MINIO_USERNAME/MINIO_GROUPNAMEprivilege-drop path (use--useror KubernetesrunAsUser),docker exec <c> shdebugging (use ephemeral-container tooling), and in-imagemc(use the releasedmclior the client image). - TLS: mount certificates at
/tmp/.silo/certs(the container’s default certs directory) and both the server and the baked-in probe derive HTTPS from the same location; for CLI-configured servers, setMINIO_HEALTHCHECK_URL.
The classic image remains the default and is unchanged. If the pilot proves out, the distroless variant becomes the recommended image later; the decision record lives in the design note above.
Container images
The classic image was diffed field by field against pgsty/minio:RELEASE.2026-08-04T00-00-00Z: entrypoint, exposed ports, volumes, working directory, user, and (absent) health-check configuration are identical. Exactly three differences exist, all deliberate: Cmd is ["silo"] instead of ["minio"], the upstream update-verification key variable MINIO_UPDATE_MINISIGN_PUBKEY is removed (updates through upstream channels are permanently disabled), and HOME=/tmp is declared to match the entrypoint’s writable-home guarantee.
The bundled client is upgraded to mcli RELEASE.2026-08-06T00-00-00Z (with the mc alias preserved), pinned by per-architecture SHA-256 digests and verified against the published checksums at build time. Interoperability of the released mcli 20260806 against this server — multipart, versioning, presigned URLs, metadata/tags, user and policy administration — was verified as part of release acceptance.
Helm chart
The chart ships as silo 7.0.1, preserving rendered resource identity with the legacy chart across a simulated upgrade (verified by the migration guard over 7 rendered resources). Its default image tag now points at this release — docker.io/pgsty/silo is a fresh repository, so the inherited default could never have pulled. The chart still ships no liveness/readiness/startup probes; adding them is planned, and documented, in the design note’s follow-up phase.
Packaging and migration
RPM, DEB, and APK packages install exactly six files: /usr/bin/silo, silo.service, a sysusers definition (creating the silo system user), /etc/default/silo (marked config/noreplace), LICENSE, and NOTICE. RPMs are GPG-signed with the PGSTY maintainer key (9592A7BC 7A682E73 33376E09 E7935D8D B9BD8B20). RPM and DEB now carry a unified, PGDG-style 1PGSTY release segment — silo-<version>-1PGSTY.<arch>.rpm and silo_<version>-1PGSTY_<arch>.deb — replacing the inherited bare -1 on RPM and the missing revision on DEB; APK names stay bare because Alpine pkgrel admits only -r<integer>.
silo.service is designed for takeover: Type=notify (readiness is signaled by the server itself), Conflicts=minio.service + After=minio.service (starting Silo stops a running MinIO unit), and two environment files — /etc/default/minio is read first and /etc/default/silo overrides it — so an existing MinIO configuration is inherited without editing. For existing deployments whose data is owned by the minio user, the documented drop-in keeps ownership untouched:
Distributed migrations must switch all nodes together.
Cluster bootstrap verifies that every node runs the same binary (by checksum). A mixed cluster — some nodes on Silo, some still on MinIO — does not form: the new node stays in activating, logging Expected Silo binary checksum ... seen: ... and Waiting for at least 1 remote servers with valid configuration, indefinitely. Stop MinIO on all nodes, then start Silo on all nodes (near-simultaneously). Once every node runs Silo, rolling restarts work normally — gate each one with silo healthcheck --maintenance cluster.
Migration troubleshooting, from the acceptance run: if Silo starts as the packaged silo user against a deployment whose TLS certificates live under the minio user’s home, it fails with HTTPS specified in endpoints, but no TLS certificate is found and restart-loops until the systemd start limit — the legacy-user drop-in above is the fix. Keep the MinIO package and unit installed (disabled) during the migration window: the rollback path — stop Silo, start MinIO — was rehearsed and reads all data written during the Silo window, because the migration touches neither data ownership nor format.
Components and dependencies
- SILO Console 2.1.1 — the embedded console, selected from
pgsty/silo-consolewhile preserving thegithub.com/minio/consoleimport path. silo-pkg3.11.0 — retains the policy/LDAP/certificate fixes including the LDAP-over-TLS repair tracked in #15.mcli20260806 — bundled in the image and released separately; see its release notes.- Go 1.26.5 — toolchain unchanged from 20260804.
Build, CI, and release pipeline
- Compatibility as a CI gate: the rebrand guard snapshots routes, environment tokens, headers, metrics, storage/policy identifiers, and exported symbols, and fails on any unreviewed drift; companion scripts assert the delivery surface (binary path, unit contents, image layout) and that no live upstream endpoint remains in runtime code.
- Release-image gate: every release-pipeline run builds both container images and asserts, among others: the distroless
HEALTHCHECKsurvives into the image config (it is a Docker extension outside the OCI spec),/dataships world-writable, no shell and no/usr/bin/minioexist, Docker’s health state turns healthy from the baked probe alone, and SIGTERM still stops the server gracefully as root and as--user 1001:1001. - Provenance chain: images are built from the published release archives after checksum verification and
gh attestation verifyagainst the exact tag; per-architecture SBOMs and provenance attestations are pushed for the classic and distroless images; the distroless health-check gate runs before the multi-arch manifests are promoted. - Workflow runtime moved to Node 24 across CI actions.
Compatibility and upgrade notes
- Package upgrades are a takeover, not an in-place update. Install
silo, keep/etc/default/minioas is (it is inherited), enablesilo.service; starting it stopsminio.servicevia the conflict relation. Data is untouched. - Keep data ownership stable with the legacy-user drop-in above; do not chown storage or move certificates during migration.
- Distributed clusters: full-stop switchover only. See the warning above — mixed Silo/MinIO nodes do not form a cluster.
- Container users: the image is now
docker.io/pgsty/silo;docker.io/pgsty/miniostays frozen at 20260804 as an archive. The classic image’s behavior is unchanged — includingmc ready localhealth checks — and the distroless variant is strictly opt-in. - Distroless differences are deliberate: no shell, no in-image
mc, noMINIO_USERNAMEpath; health is native; servers configured via CLI arguments needMINIO_HEALTHCHECK_URLfor the baked-in probe. - Helm users: chart 7.0.1’s defaults now pull this release; override
image.tagexplicitly if you pin versions. - Known and unchanged: the classic image still does not create
/datain the layer, so fully non-rootdocker runagainst a Docker-managed volume fails as before (#55, fixed in the distroless variant); the inherited Postgres/MySQL legacy notification-migration limitation from the 20260804 notes still applies (#53). - Pair with
mcli20260806 for the client side; older clients continue to work over the unchanged wire protocol.
Verification
This release was verified in stages, each with recorded evidence:
- unit and end-to-end matrices for the health-check command: target derivation and precedence (flag/env/derived), real-TLS auto-detection, exit-code contract, JSON schema, timeout bounds, usage errors;
- cluster-semantics verification on a four-node cluster: with 2 of 4 nodes stopped,
clusterreports 503 withwrite-quorum=5whilecluster-readandlivestay 200 — the write/read quorum split observed live, matching the erasure math; - image acceptance: the classic image diffed field-by-field against the 20260804 baseline; the distroless image asserted down to file inventory, exact health-check configuration, and root/non-root/TLS/env-override runtime scenarios;
- a six-phase pre-release acceptance concluding in a real migration: a Pigsty-deployed four-node TLS MinIO 20260804 cluster (16 drives, EC:4) was migrated to Silo via the packaged takeover path — reference data (multipart, versioned, tagged objects) read back byte-identical, four maintenance-gated rolling restarts,
kill -9fault injection with the load balancer serving 23/24 continuous IO rounds (the only failure in the kill second), Prometheus metrics continuity, and a full rollback to MinIO and back, proving the migration reversible.
Validation boundaries
External LDAP/OIDC/KMS/etcd services were not tested against live endpoints in this release run; amd64 packages were cross-built and payload-checked; Windows and Intel macOS targets were cross-compiled only.
Artifacts
- GitHub release
RELEASE.2026-08-06T00-00-00Zatpgsty/silo, with checksummed platform archives, provenance attestations, and RPM/DEB/APK packages (GPG-signed RPMs); docker.io/pgsty/silo:RELEASE.2026-08-06T00-00-00Zandlatest;docker.io/pgsty/silo:RELEASE.2026-08-06T00-00-00Z-distrolessanddistroless— published on demand from the finished release;- companion releases:
mcli20260806,silo-pkg3.11.0, embedded SILO Console 2.1.1; - design record: Native Health Checks and the Distroless Image.
Selected changes
15def34dc,77bdc4c0c: drop upstream delivery residue; present Silo identity and close inherited upstream services15ab10833: rename the delivery artifacts to silo and complete the package payload30749911b: ship the silo binary in the image and translate the legacy argv commande071bb77e: replace the minio chart with a silo chart that preserves identitybd8df5166: gate the rebrand on compatibility, packaging, and provenance evidence6613c2a3c: pin the external test fixtures and run the suites against the silo binaryfd2ca1c6d,c46b16ec6,c47733abc,f1c77d5a2: cut over to pgsty/silo and main; document the archived branch6740e6978: move the workflow actions onto the Node 24 runtimeb57275be3: adopt the no-CLA plus DCO policy and fix copyright terms62717d7bf,a6d6d9b02: update the embedded Console to 2.1.0, then 2.1.16bd9cf77e: regenerate CREDITS from the linked module set and guard it in CI219670d31: ship LICENSE and NOTICE in every package and image2ff594f4b: add the nativesilo healthchecksubcommand4c34d2309: add the distroless image variant as a pilotb6d47b739,9462cce16: healthcheck and distroless hardening; lint cleanup16b78eb4e: bundle mcli 20260806 and point the Helm defaults at this release062a91bee: pin the CREDITS module closure to the shipped linux target467931455: unify the rpm and deb release segment as1PGSTYb14ea22aa: match checksum manifest entries exactly in the image publish lane3be10fcc1: add a manual finalize lane refreshing SBOMs and checksums for signed Draft packages
Acknowledgments
Four contributors have code merged into this fork, and the Git history carries their authorship: @ZouhairCharef patched CVE-2026-34986 in go-jose (#18), @mfredenhagen patched CVE-2026-39883 in OpenTelemetry (#19), @pinginfo implemented Flush on trackingResponseWriter to repair bucket notification streaming (#34), and @waterkip repointed the documentation links to the Silo portal (#41).
A first release under a new name is also the right moment to thank everyone who has filed issues against this fork — bug reports, compatibility findings, and proposals alike, resolved and still open:
@mosesdd (#1), @Xavier-777 (#2, #17), @jiadzh (#3), @TLINDEN (#4), @AntonOfTheWoods (#5), @zylpsrs (#6), @nsanitate (#7), @makinikm (#9), @magicxor (#10), @spaceg00se-r (#11, #14), @heroes1412 (#13), @vampywiz17 (#15), @davinkevin (#20), @chalukyaj (#30), @cbornet (#31, #32), @jvasile (#33), @Kesavaambati (#35), @redfoxfox (#38), @kuldeep-link11 (#39, #40), @meesudzu (#42), @pmezhuev (#43), and @kh0mka (#51).
Several of this release’s headline items trace directly back to those reports: the bundled-client guarantee to #4 and #9, the LDAP-over-TLS repair to #15, the completed package payload to #33, GPG-signed RPMs to #43, the migration guide to #42, and the distroless /data fix to #55.
Pull requests still in flight deserve a mention too. @davinkevin’s distroless image PR (#21) anticipated this release’s pilot months in advance — the shipped variant supersedes that PR with the native health check built in, but the direction was proposed there first. Conformance PRs from @magicxor (#12) and @ycjlin (#37) are queued for review immediately after this release.
Everyone who has contributed to this fork is recorded in CONTRIBUTORS.md, which is now the project’s attribution record — GitHub generates no contributor graph for forks.
2.29 - Silo 20260804 Released
Version: RELEASE.2026-08-04T00-00-00Z · Commit: d88f46ccee345a9c2fabe2d221d9a9e56bc11aec
SILO 20260804 is a security, correctness, and release-engineering update to the pgsty/minio community fork. It completes the internode storage-containment work begun with CVE-2026-42600, prevents request-controlled values from impersonating server-calculated S3/IAM policy conditions, restores streaming flush behavior, fixes several multipart and versioning edge cases, hardens notification configuration migration, moves the build baseline to Go 1.26.5, and connects the server to the SILO-maintained Console, shared package, and mcli releases. The release pipeline was rebuilt to produce reproducible binaries and GPG-signed packages.
The release covers 50 commits after the pre-2026-06-18 baseline, changing 155 files with 9,241 insertions and 981 deletions. Every change was reviewed against the tagged commit and verified on macOS ARM64 and Linux AMD64, with GitHub CI green on the released HEAD.
Highlights
- Internode containment completed: validates storage-REST message bodies, storage Grid frames, and peer-S3 Grid requests at the storage boundary, closing the remaining path, volume, erasure-metadata, panic, and unbounded-allocation defects left after removing
ReadMultiple. - S3/IAM decisions now use effective values: client input can no longer shadow internal condition values; request tags and existing-object tags are separated;
s3:signatureAgeis confined to verified presigned requests; ands3:versionidfollows the version the server actually acts on. - Bucket and object resources are separated: twelve sensitive bucket-level writes are no longer authorized through an object-only
bucket/*resource pattern. A documented compatibility switch is available for migration. - Multipart compatibility and correctness improved: full-object checksum completion works without per-part checksums when the protocol permits it, zero-length multipart checksums are preserved, and duplicate part numbers are rejected instead of assembling duplicated data.
- Streaming reliability restored:
trackingResponseWriternow implementsFlushcorrectly and records implicit HTTP 200 responses, repairingmcli watch, bucket-notification listeners, and S3 Select keep-alives affected by the inherited regression documented in the 20260618 release. - Notification configuration hardened: NATS and AMQP keys used by parsers and legacy migration are registered and round-trip correctly; libpq connection parameters are quoted safely; invalid-key errors no longer echo secret values.
- Reproducible, signed release pipeline: binaries no longer embed the build machine’s paths, packages install under the canonical systemd path, and RPMs are GPG-signed. The container entrypoint now shuts down gracefully on every privilege path.
- Release baseline refreshed: Go 1.26.5,
klauspost/compress1.18.7, Apache Thrift 0.24.0, SILO Console 2.0.0,silo-pkg3.11.0, andmcli20260804.
Security Hardening
Internode storage and Grid containment — SN-2026-002
Removing the obsolete ReadMultiple endpoint in 20260618 closed one reachable path but did not close the underlying defect class. Storage-REST request bodies and Grid RPC frames do not pass through the HTTP query-validation middleware, and peer-S3 RPCs can bypass the storage-REST wrapper entirely.
This release moves containment to the storage boundary and validates every caller-controlled path, volume, erasure parameter, part size, shard length, and allocation length before use. The fixes include:
- reject traversal on both path and volume axes, including Windows volume-root aliases;
- cover peer-S3 bucket RPCs that reach drives without the storage-REST wrapper;
- reject zero or unusable data/parity/block-size combinations before shard arithmetic;
- reject negative part sizes and truncated shards instead of reporting them healthy;
- cap storage-REST
ReadFileallocations at 5 GiB; - bound other allocations derived from internode declarations;
- contain panics in deadline-bounded storage work without blocking the caller;
- preserve
ReadPartserrors across keep-alive responses and avoid the empty-part trace panic.
These routes require cluster-root or internode credentials and are registered only in distributed-erasure deployments. Single-node S3 behavior is unchanged. See Internode Path Containment Audit for the protocol-surface analysis.
Effective policy-condition values — SN-2026-003
The policy condition map historically mixed values calculated by the server with raw request entries. A client-controlled spelling could therefore shadow or synthesize an internal condition value. SILO 20260804 pairs silo-pkg 3.11.0’s exact-key lookup rule with server-side source normalization:
- internal condition names cannot be supplied as arbitrary client values;
s3:prefix,s3:delimiter, ands3:max-keyscome from their effective query inputs;- header-backed
x-amz-*conditions do not accept unrelated query substitutes; - when storage class or upload tagging supports both forms, an explicitly present Header wins, including an empty Header;
s3:ExistingObjectTag/*comes only from stored object metadata;s3:RequestObjectTag/*is bound to the tag input consumed by the relevant operation;s3:signatureAgeis exposed only after verified SigV4 presigned authentication calculates it;s3:versionidis absent when no version is named and is rebound perDeleteObjectsentry to the effective resolved version.
The version-ID behavior closes the fail-open trap that a superficial “omit empty values” fix would have created for Multi-Delete. See Absent Is Not Empty.
Bucket/object resource boundary — SN-2026-004
The IAM matcher used to append a slash to a bucket-level request, allowing an object-only resource such as arn:aws:s3:::bucket/* to authorize selected bucket-level operations. This release withholds twelve sensitive writes from that pattern on Allow statements:
PutBucketPolicy, DeleteBucketPolicy, PutBucketObjectLockConfiguration, PutBucketVersioning, PutReplicationConfiguration, PutBucketLifecycle, DeleteBucket, ForceDeleteBucket, PutBucketCors, DeleteBucketCors, PutBucketQOS, and PutInventoryConfiguration.
Deny and NotResource behavior is unchanged. Read/list operations, CreateBucket, bucket tagging, default encryption, and notification configuration remain compatible. Built-in policies use Resource: "*" and are not affected.
Policy migration required for custom bucket grants
If a custom policy grants one of the twelve actions — often through s3:* — using only arn:aws:s3:::bucket/*, add the bare bucket ARN:
MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH=on restores the historical matcher while policies are migrated. It also restores the historical over-grant, so use it only as a temporary rollback control.
Trusted client-address boundary
MINIO_API_TRUSTED_PROXIES provides an enforceable, opt-in boundary for aws:SourceIp, audit remotehost, event notification Host, and the client address shown by mcli admin trace:
- set it to an address/CIDR list to trust forwarding headers only from those peers and walk forwarding chains from right to left;
- set it to
noneto ignore all forwarding headers; - leave it unset to preserve historical behavior exactly.
The old _MINIO_API_XFF_HEADER=off switch still suppresses only X-Forwarded-For; it does not protect against X-Real-IP or RFC 7239 Forwarded. If IP-based policy is part of your security boundary, configure trusted proxies explicitly and prevent direct access to the S3 API port. Multi-node deployments should allow their own node addresses. See Client Source Address Trust.
S3 and Storage Correctness
Multipart upload
- CompleteMultipartUpload accepts the S3 full-object checksum mode when the completed request supplies no per-part checksums and the upload metadata does not require them.
- The checksum of a zero-length multipart object is retained instead of being discarded as empty metadata.
- Part numbers must be strictly increasing. Duplicate entries such as
[1,1]now returnInvalidPartOrderinstead of consuming the upload and assembling the same part twice. Legal part lists with gaps or a non-1 start remain accepted. See Duplicate Part Numbers.
Object reads and buffer ownership
- erasure reads again pool buffers only where ownership permits reuse;
- update downloads return caller-owned buffers instead of exposing data that can be overwritten after return;
- the old HTTP streaming helpers orphaned by
ReadMultipleremoval are deleted after reference and platform-tag checks.
HTTP response tracking and S3 Select
trackingResponseWriter.Flush()delegates to the underlying flusher and commits the response state correctly;- the first implicit write records HTTP 200, preserving audit and metric accuracy;
- S3 Select tests no longer race a client parser against response-body ownership;
- CSV, JSON, and Parquet selection paths remain covered, including range/error and keep-alive behavior.
The inherited silent-flush regression called out in SILO 20260618 is therefore fixed in this release.
IAM, Versioning, and Audit Behavior
- DeleteObject and each entry in DeleteObjects evaluate
s3:versionidagainst the effective version selected by the server. - Request tags can no longer impersonate existing-object tags during policy evaluation.
- The
merrstag is restored when dangling-object deletion records are emitted, preserving the intended audit classification. - Bucket-policy and IAM paths share the hardened condition-source rules while retaining their established S3 routing and error behavior.
Notification Configuration
- registers the NATS
user_credentials,nkey_seed, andtls_handshake_firstkeys read by the parser; - separates the legacy NATS environment-variable spelling from the stored config key;
- repairs NATS migration round trips and the AMQP
immediate/internalmapping; - adds a mechanical audit that compares keys read and written by notification code with each subsystem’s registered schema;
- quotes libpq connection-string parameters so whitespace, quotes, and backslashes retain their intended value;
- prevents invalid-key diagnostics from echoing secret values.
See Notify Keyspace Registration.
Known legacy migration limitation
The inherited Postgres and MySQL legacy migration functions still write the unregistered host, port, username, password, and database fields. A migrated configuration can therefore fail validation on the next load, and notification target loading is fail-fast across subsystems. This predates 20260804, but upgrades from pre-connection-string database notification configurations must be reviewed and converted before restart. The stored password field may contain a plaintext database password.
Components and Dependencies
- Go 1.26.5: includes security fixes in
crypto/tlsandosplus compiler, runtime, networking, and syscall corrections. klauspost/compress1.18.7: refreshes the compression stack used by object and archive paths.- Apache Thrift 0.24.0: updates the dependency compiled through Parquet support.
- go-systemd 22.6.0: deliberately retained instead of 22.7.0 because the later version introduced a NetBSD clock dependency incompatible with the supported cross-build matrix.
- SILO Console 2.0.0: the embedded console is selected from
pgsty/silo-consolewhile preserving the compatiblegithub.com/minio/consoleimport path. silo-pkg3.11.0: provides the companion policy, LDAP, certificate, RNG, and time-format fixes while preserving thegithub.com/minio/pkg/v3module path.mcli20260804: the embedded client comes frompgsty/mc; release images expose it asmcliand keep themccompatibility alias.
See the companion release notes for silo-pkg 3.11.0, mcli 20260804, and SILO Console 2.0.0.
Build, CI, and Packaging
This release rebuilt the release pipeline for reproducibility and supply-chain integrity:
- Graceful container shutdown on every path. The entrypoint’s custom UID/GID branches now
execinto the server so it runs as PID 1 and receivesSIGTERMdirectly; previously those branches left an intermediate shell as PID 1 and the server was killed at the container stop timeout. A CI smoke test builds the release runtime image and asserts graceful shutdown on both the default and drop-privilege paths. - Reproducible binaries. Release binaries no longer embed the build machine’s
GOPATH/GOROOT, so-trimpathholds and a third party rebuilding the tag gets matching bytes. The published Linux binary contains no build-host path. - Hardened release workflow. The release tag is passed through the environment and whitelisted rather than spliced into the shell, the build is checked out at the tag being released, and an untracked shadow GoReleaser config that could publish or move
latestout of band was removed. - Honest gates. CI gates build, vet, unit tests, lint, generation drift, race tests, and cross-compilation; the cross-compile matrix is aligned to the exact set of published targets; and the lint and dependency-install steps now fail on real errors instead of masking them.
- Signed, canonical packages. RPM, DEB, and APK packages are produced with nFPM under the PGSTY identity, the systemd unit installs at
/usr/lib/systemd/system/minio.servicewithType=notify, and RPMs are GPG-signed offline with the PGSTY maintainer key (fingerprint9592A7BC 7A682E73 33376E09 E7935D8D B9BD8B20). - Separate archive and image publication steps. GoReleaser produces the platform archives, checksums, and packages; the multi-architecture container image is built and published from the finished release artifacts.
Compatibility and Upgrade Notes
- Keep every node on one release during a cluster rollout. Internode validation changed across storage-REST and Grid surfaces; mixed binaries were not production-tested.
- Audit custom IAM policies. Add the bare bucket ARN for the twelve protected bucket writes. Use
MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH=ononly as a temporary migration control. - Configure client-address trust deliberately. If
aws:SourceIpor audit attribution matters, setMINIO_API_TRUSTED_PROXIESand close direct network paths around the proxy. - Review legacy database notification settings. Convert Postgres/MySQL host/user/password fields to the supported connection-string format before restart.
- Expect duplicate multipart completion entries to fail. Clients sending the same part number more than once now receive
InvalidPartOrderinstead of a corrupted successful object. - Use the matching
mcli. The 20260804 client disables self-update and must be upgraded through packages or GitHub Releases;mcli updateremains as a compatibility command but exits non-zero. - RPM users can enable signature verification. Packages are signed with the maintainer key above; import it before enabling
gpgcheckfor the SILO packages.
Verification
Verification against the tagged commit covered:
git diff --check, gofmt, module verification, and YAML/shell syntax;go build ./...,go vet ./..., project lint, andgovulncheck ./...;- full
go test ./..., the complete race suite, and repeated race tests over storage, policy, notification, HTTP tracking, and S3 Select changes; - generator idempotence plus deliberate stale-source and untracked-output counterexamples;
- cross-compilation across every published target;
- Linux AMD64 native full tests, targeted race tests, live S3/
mclismoke tests (create/upload/download/copy, range, versioning, delete markers, health checks, graceful shutdown, restart persistence), and systemd notify behavior; - release-artifact verification: GitHub CI green on the released HEAD, reproducible binaries with no build-host path, the systemd unit installed at
/usr/lib/systemd/system, and RPM signatures validated withrpmkeys --checksig.
govulncheck found no vulnerability reachable from the Server or mcli code. One module-level notice remains for the unmaintained golang.org/x/crypto/openpgp package (GO-2026-5932); that package is not imported into these binaries.
Validation boundaries
Native execution was run on Linux AMD64 and macOS ARM64; Windows, Intel macOS, and Linux ARM64 targets were cross-compiled. External LDAP, OIDC, KMS, STS, Postgres, MySQL, NATS, and AMQP services were not exercised against live endpoints in this test run.
Artifacts
- GitHub release
RELEASE.2026-08-04T00-00-00Zwith checksummed platform archives for Linux, Darwin, and Windows on amd64 and arm64; - RPM, DEB, and APK packages under the PGSTY identity, with GPG-signed RPMs;
docker.io/pgsty/minio:RELEASE.2026-08-04T00-00-00Zand the release-selectedlatesttag, published on demand from the release;- matching SILO Console 2.0.0,
silo-pkg3.11.0, andmcli20260804 references.
Selected Changes
ca7baa670,80e8eaa42,b6f70ab08: validate internode paths, erasure metadata, and allocation sizesa36fd8fff: contain panics in deadline-bounded storage work2f55347f7: bind S3/IAM policy conditions to effective request values744a9dcd7: binds3:versionidto the effective object version97b7d2804: enforce the bucket/object resource boundaryfe6dc4780: add the trusted-proxy client-address boundary22c1e41fd: reject duplicate multipart part numbersc8590413f,3e14733f1: restore full-object and zero-length multipart checksum behavior8069a32ac,65795ee1f: restore response commit and streaming flush semantics162ded343,0c14d8151: repair notification key registration and libpq quoting924717926,89d346bf5: restore safe buffer pooling and returned-buffer ownership3b8a55dee: exec into the dropped-privilege process so signals reach the server2ca4971d9: stop stamping the build machine’s paths into the binary4c185d5a6,e064b5555: harden the release workflow and remove the shadow configaa5139369: install the systemd unit under/usr/lib11d79fddc,ca674a696,021110b45,d88f46cce: gate build, vet, tests, lint, generation, race, and cross-compilation, and smoke-test the release image
2.30 - mcli 20260804 Released
Published: 2026-08-04 · Version: RELEASE.2026-08-04T00-00-00Z
This is the first release of the pgsty/mc community fork since 20260417. It fixes a credential leak in debug logging, severs every remaining connection between the client and upstream release channels, moves containers and packages onto artifacts this fork builds itself, and migrates packaging from MinIO’s pkger to standard nFPM — with GPG-signed RPMs for the first time.
Upstream minio/mc was archived in July 2026. Its final commit, 77f82e18, is exactly this fork’s base, and upstream never cut a release containing it — so this build is strictly newer than any official mc binary ever published.
Behavior change
mcli update self-update is disabled in this fork. The command remains for script compatibility and still accepts its original arguments, but it no longer contacts the network or replaces its own binary; it prints an explicit notice and always exits with status 1. Upstream mc update exited 0 when already up to date, so drop the call from any script that treats a non-zero exit as failure. Upgrade through the Pigsty package repository or GitHub Releases.
The automatic version check that ran against upstream release feeds on every invocation has also been removed entirely. The MC_UPDATE and MINIO_UPDATE environment variables are no longer consulted.
Major Changes
- Self-update disabled, upstream release channels severed: the
minio/selfupdateandaead.dev/minisigndependencies and all binary-replacement logic are gone, along with the update notifier and the FIPS/non-FIPS update paths. Theupdatecommand survives as a compatibility shell, and the runtime helpers (Docker / DCOS / Kubernetes / source-build detection) moved to a dedicatedcmd/runtime-info.go. The client previously reached out to upstream release feeds on every invocation to print an upgrade hint; there is now no outbound release probing at all. - Containers and artifacts fully localized: the default image is built from the checked-out fork source, and hotfix binaries are copied from the local build context — no upstream prebuilt binaries are downloaded. The upstream publishing files
Dockerfile.release,Dockerfile.release.old_cpu, anddocker-buildx.shwere removed, and the obsolete MinIO hotfix upload target is disabled. - Packaging migrated to nFPM: replaced MinIO’s
pkgerwith standard nFPM. Artifact layout and install path are unchanged (/usr/local/bin/mcli, package namemcli,YYYYMMDDHHMMSS.0.0version scheme), but the vendor is now PGSTY, the license uses the SPDX identifierAGPL-3.0-or-later, and the DebianSectionmoved from empty toutils. - RPMs are now GPG-signed: RPMs are signed offline with the maintainer key (fingerprint
9592A7BC7A682E7333376E09E7935D8DB9BD8B20). All package metadata is asserted before signing, and the signature is re-verified with checksums regenerated afterwards. DEB and APK packages carry no package-level signature; their trust anchor lives at the repository layer. - Build provenance hardened: every previously published binary was stamped by the Go toolchain as built from a modified working tree (
vcs.modified=true), which broke the link between an artifact and its Git tag. This release fixes that and adds an enforcing check to both the release and test pipelines, so every binary is traceable to an exact commit.
Security Fixes
- SUBNET credentials redacted in debug logs: with
--debugenabled, SUBNET HTTP exchanges are printed in full. Previously theapi-key/api_keyquery parameters, authentication headers, and response bodies all reached the log in clear text — and SUBNET’s authentication and registration endpoints return API keys, licenses, and tokens in their responses. Both parameter spellings and duplicate values are now masked uniformly, sensitive response headers are redacted, and SUBNET response bodies are excluded from debug dumps entirely. The leak is inherited from upstream and present in every previous release, upstreammcincluded: if you have ever shared--debugoutput of SUBNET commands (mcli license .../mcli support ...), treat the API keys and licenses in it as exposed and rotate them. - Redaction isolated from caller state: debug tracing now dumps copies of the request and response, so redaction cannot mutate objects the caller still holds. Zero-length, fixed-length, and unknown-length response bodies are all covered, and callers can still read the response normally.
Dependency Updates
Every dependency bump below except the term / mod / sync / tools refresh closes at least one published advisory in the Go vulnerability database, and govulncheck reports zero known vulnerabilities reachable from this release’s code. No security advisory has ever been published for minio/mc, minio-go, madmin-go, or minio/pkg themselves.
- Go build baseline upgraded from
1.26.2to1.26.5(the newest 1.26.x at release time), picking up the 1.26.3–1.26.5 security batches — including GO-2026-4970 (symlink-based root escape inos) and GO-2026-5856 (Encrypted Client Hello privacy leak incrypto/tls), the two most relevant to an S3 client that writes local files and speaks TLS. github.com/klauspost/compressfromv1.18.5tov1.18.7(closes GO-2026-5841).github.com/prometheus/prometheusfromv0.310.0tov0.311.3(closes GO-2026-5264, GO-2026-5381, GO-2026-5710).google.golang.org/grpcfromv1.79.3tov1.82.1(closes GO-2026-6061), with thegenprotofamily refreshed alongside.- The
golang.org/x/*family refreshed across the board:cryptov0.49.0→v0.53.0(the 14-advisory GO-2026-5005…5033 batch),netv0.52.0→v0.56.0(GO-2026-5025…5030 and GO-2026-5942),sysv0.42.0→v0.46.0(GO-2026-5024),textv0.35.0→v0.39.0(GO-2026-5970), plusterm,mod,sync, andtools. - Removed
aead.dev/minisignandgithub.com/minio/selfupdate, and synchronized the third-party credits file.
Engineering and Delivery
- Integration test dependencies pinned: CI no longer downloads the MinIO server from a mutable upstream URL. It now uses a versioned
pgsty/miniorelease archive verified by its SHA-256 digest, with Go pinned to1.26.5. - Release pipeline verification: a packaging validation workflow compares the binary inside all three package formats byte-for-byte against the build output, and checks package names, checksums, architecture fields, and every metadata field. The expected RPM metadata is sourced from the signing script itself, so configuration drift cannot strand a release part-way through signing.
- CI supply-chain hardening: every GitHub Action is pinned to a commit SHA with dependabot keeping them current, workflow permissions are narrowed to read-only, and a stale workflow pointing at the upstream organization’s project board was removed.
- Documentation: the English and Chinese READMEs now state this fork’s distribution channels and self-update policy explicitly, and installation instructions that would silently install upstream
mcwere removed.
Known issue
mcli watch (bucket event notification) receives no events against any published pgsty/minio server release. The cause is a silent streaming-flush regression on the server side, inherited from upstream — it is not a client problem, and the previous mcli release is affected identically. The fix was merged to the server’s master on 2026-07-29 but has not shipped in a published server release. See the SILO 20260618 release notes and PR #34.
Separately, this fork inherits upstream’s unfixed defects, and with the upstream repository archived they can only ever be fixed here. The most serious is minio/mc#5139: mirror --remove --watch can delete a live object from the target when a non-current version of it is removed from the source. Exercise caution combining --remove --watch on versioned buckets.
Related Commits
- 9603ee3: fix: redact SUBNET secrets in HTTP debug logs
- f6ae2b0: fix: disable self-update in Pigsty builds
- c05a6e4: build: update Go deps and toolchain to 1.26.5
- 1f105aa: build: use local fork artifacts for containers
- 1182da5: ci: pin fork integration test dependencies
- 9ee207f: docs: clarify Pigsty fork distribution channels
- 0686cd8: fix: isolate SUBNET debug redaction
- ad10a2a: build: complete local Docker context isolation
- 5f54221: docs: update mc README and cn version
- 02c0305: build: migrate release packaging to nFPM
- 4c4dcc4: build: harden release provenance and package metadata
2.31 - Silo 20260618 Released
Published: 2026-06-18 · Version: RELEASE.2026-06-18T00-00-00Z
This release is a security and dependency-maintenance update for the pgsty/minio fork. It hardens LDAP STS throttling, completes S3 Select oversized-record enforcement, removes the obsolete ReadMultiple internode storage-REST API, upgrades the Go build baseline to 1.26.4, and refreshes Go module dependencies to pick up additional third-party security fixes.
Note
Known issue: this release — like every earlier community release since RELEASE.2025-12-03T12-00-00Z — carries a silent streaming-flush regression inherited from upstream that breaks mc watch / bucket-notification listeners and S3 Select keep-alives. The fix (PR #34) shipped in SILO 20260804.
Major Changes
- Remove the obsolete
ReadMultiplestorage-REST API: the legacy/rmplinternode endpoint is removed rather than patched in place, including its route, handler, client wrapper, storage interfaces, xlStorage methods, generated datatypes, and related metric. No production caller is expected after upstream multipart handling moved toReadParts, but clusters should still run a consistent release during rolling upgrades. - Complete S3 Select oversized-record enforcement: JSON Lines input now uses the bounded reader path, so oversized records are rejected consistently instead of bypassing limits on SIMD-capable CPUs. S3 Select stream errors now preserve the intended error code and wrap JSON parser failures as
JSONParsingError. - Harden LDAP STS rate-limit source bucketing: throttling is now keyed only by source IP, avoiding username-shared buckets that could be drained by one client to lock out a legitimate user. Trusted-proxy handling now resolves
X-Forwarded-Forfrom right to left, rejects catch-all trusted-proxy CIDRs, ignores RFC 7239Forwarded, and documents theX-Real-IPdeployment contract. - Refresh the Go runtime and module baseline: release, hotfix, goreleaser, and old-CPU Docker builds now use
golang:1.26.4-alpine;go.modis updated to Go1.26.4; and dependencies are refreshed across NATS, Prometheus, Azure SDK, Apache Thrift, gRPC, OpenTelemetry, Google API/auth, Gox/*, and related transitive libraries.
Direct Security Fixes
- CVE-2026-42600: remove the obsolete
ReadMultiplestorage-REST API to close the legacy internode file-read path exposed through/rmpl. - CVE-2026-39414: complete oversized S3 Select record enforcement for JSON Lines inputs and preserve correct S3 Select error semantics.
- CVE-2026-33419: further harden LDAP STS rate-limit accounting and trusted-proxy source-IP handling.
Dependency Security Updates
- Update
github.com/Azure/go-ntlmsspfromv0.1.0tov0.1.1, fixing CVE-2026-32952, where malformed NTLM challenges could panic a Go process. - Update
github.com/apache/thriftfromv0.22.0tov0.23.0, fixing CVE-2026-41602 in the GoTFramedTransportimplementation. - Update
github.com/nats-io/nats-server/v2fromv2.11.1tov2.11.15, absorbing the NATS 2.11.x security patch line. Notable fixes include pre-auth WebSocket and leafnode denial-of-service issues, MQTT authorization issues, JetStream management API authorization hardening, credential exposure fixes, and request identity-spoofing fixes, including CVE-2026-27889, CVE-2026-29785, CVE-2026-33217, CVE-2026-33218, CVE-2026-33222, and CVE-2026-33247. - Update
github.com/prometheus/prometheusfromv0.310.0tov0.311.3, absorbing Prometheus security fixes for remote-read denial of service, stored XSS in UI surfaces, and remote-write configuration secret exposure, including CVE-2026-42154, CVE-2026-44903, CVE-2026-42151, and CVE-2026-40179. - Upgrade the release build baseline through Go
1.26.4and refresh supporting Go module families, includinggolang.org/x/crypto,golang.org/x/net,golang.org/x/sys,golang.org/x/text,google.golang.org/grpc, and OpenTelemetry. These updates keep the fork aligned with patched upstream dependency baselines even where the previously pinned version was already past the specific public advisory range.
Related Commits
2.32 - Silo 20260417 Released
Published: 2026-04-17 · Version: RELEASE.2026-04-17T00-00-00Z
This release focuses on security hardening and compatibility tightening. It bundles fixes across OIDC, LDAP STS, S3 Select, replication metadata handling, unsigned-trailer flows, the Snowball upload path, and multiple dependency- and Go toolchain-related security issues, while also incorporating the LDAP TLS regression fix and a cleanup of community-fork documentation.
Major Changes
- Tighten the identity-authentication flow: OIDC / WebIdentity now accepts only asymmetrically signed
ID Tokenvalues backed by the IdPJWKS; symmetrically signed tokens such asHS256are no longer accepted. LDAP STS also now hides the distinction between unknown-user and bad-password failures to reduce username-enumeration risk. - Update LDAP STS rate limiting: limits now apply to both source IP and normalized username, and successful requests no longer consume quota incorrectly. By default MinIO now uses only the socket peer address as the source and no longer trusts
X-Forwarded-For,X-Real-IP, orForwarded; to rate-limit by real client IP, configureMINIO_IDENTITY_LDAP_STS_TRUSTED_PROXIESexplicitly. - Make upload and write paths stricter: presigned query parameters can no longer be combined with
unsigned-trailerPUTor multipart uploads. Snowball auto-extract now also performs full signature validation on theunsigned-trailerpath and rejects anonymous or forged-signature requests. - Prevent replication metadata spoofing: internal
X-Minio-Replication-*headers attached to ordinaryPUT/COPYrequests are now rejected or ignored, and only trusted replication flows may write the related internal metadata. - Clarify S3 Select error semantics: oversized CSV and line-delimited JSON records now return
OverMaxRecordSizedirectly instead of the genericInternalError; clients or alerting rules that depend on the old error code should be adjusted. - Upgrade the runtime and dependency baseline: fix the regression where
ldaps://did not correctly apply TLS settings, replaceminio/pkg/v3withpgsty/minio-pkg/v3, and pin several critical dependencies that are prone to breaking changes. The release also upgradesgo-jose,go.opentelemetry.io, and Go1.26.2to unify the build and release baseline. - Refresh documentation and security guidance: update
SECURITY.md,VULNERABILITY_REPORT.md,docs/sts/ldap.md, and related documents, add a security advisory index, and switch upstreamminio/minioreferences in the security guidance over topgsty/minio.
Fixed CVEs
- CVE-2026-34986: upgrade
go-josetov4.1.4and fix known security issues in the JWT / JOSE dependency chain. - CVE-2026-39883: upgrade the
go.opentelemetry.iodependency stack to fix the PATH-hijacking risk. - CVE-2026-33322: restore the strict JWKS-only OIDC JWT verification path to block keyring injection and algorithm-confusion risk.
- CVE-2026-33419: systematically harden LDAP STS authentication, rate limiting, source-address identification, and accounting logic across four follow-up fixes.
- CVE-2026-34204: reject injection of
X-Minio-Replication-*metadata by untrusted requests to prevent objects from being written with invalid replication state. - CVE-2026-39414: reject oversized S3 Select records early to avoid continued buffering and parsing of abnormal inputs.
- GHSA-hv4r-mvr4-25vw: close the unsigned-trailer query-auth bypass.
- GHSA-9c4q-hq6p-c237: harden unsigned-trailer authentication and signature validation in Snowball auto-extract scenarios.
- CVE-2026-32280, CVE-2026-32281, and CVE-2026-32283: upgrade Go to
1.26.2and absorb the upstream toolchain and stdlib security fixes.
Related Commits
- ce1c537: fix: pin deps with breaking changes and fix LDAP TLS regression (#15)
- 68e0ba9: fix: upgrade go-jose to v4.1.4 to patch CVE-2026-34986
- 1869bd3: fix: CVE-2026-39883 upgrade go.opentelemetry.io
- ff58df9: merge: PR #18 upgrade go-jose to v4.1.4 for CVE-2026-34986
- e4fa063: merge: PR #19 upgrade go.opentelemetry.io for CVE-2026-39883
- d24f449: fix: CVE-2026-33322 restore JWKS-only OIDC JWT verification
- 3b950f8: fix: CVE-2026-33419 harden LDAP STS auth
- 56fa63b: fix: CVE-2026-34204 reject untrusted replication metadata
- 3252d5b: fix: CVE-2026-39414 reject oversized S3 Select records
- f444b6f37: fix: fake CVE-2026-40027 block unsigned-trailer query auth bypass
- efb6e5b00: fix: fake CVE-2026-40028 harden snowball unsigned-trailer auth
- db4c0fd: fix: CVE-2026-32280/CVE-2026-32281/CVE-2026-32283 upgrade Go to 1.26.2
- 18b712d: fix: CVE-2026-33419 preserve LDAP STS rate limits on success
- 9e10f6d: fix: CVE-2026-33419 harden LDAP STS rate-limit source IP
- f441108: fix: CVE-2026-33419 tighten LDAP STS rate-limit accounting
- f48dbe7: docs: refresh security guidance and fork references
2.33 - Silo 20260325 Released
Published: 2026-03-25 · Version: RELEASE.2026-03-25T00-00-00Z
This maintenance release bundles mcli/mc into the Docker image, fixes an LDAP TLS configuration regression on ldaps:// connections, and pins security-patched versions of key dependencies.
Major Changes
- Bundle
mcli(and itsmccompatibility alias) into the Docker image with checksum verification so administrative CLI tools are available out of the box. - Fix the LDAP TLS regression affecting
ldaps://deployments so custom root CAs andMINIO_IDENTITY_LDAP_TLS_SKIP_VERIFYare properly honored. - Remove inherited upstream CI/CD workflows that are no longer used in the community fork.
- Pin key dependencies to isolate upstream breaking changes and incorporate the security fixes below.
Fixed CVEs
- CVE-2026-24051: Pin
go.opentelemetry.io/otel/sdk v1.42.0to address the macOSPATH-hijacking arbitrary code execution vulnerability. - CVE-2025-10543: Upgrade to
github.com/eclipse/paho.mqtt.golang v1.5.1, fixing incorrect MQTT packet encoding for oversized UTF-8 strings. - CVE-2025-58181: Upgrade to
golang.org/x/crypto v0.49.0, fixing unbounded memory consumption insshGSSAPI authentication handling.
Related Commits
2.34 - Silo 20260321 Released
Published: 2026-03-21 · Version: RELEASE.2026-03-21T00-00-00Z
This maintenance release is built around the Go 1.26.1 upgrade and a broad dependency refresh. Beyond stricter compiler and linter compatibility fixes, it also delivers the most substantial security dependency refresh in the current release line.
Major Changes
- Upgrade the build environment from Go
1.26.0to Go1.26.1. - Refresh direct and indirect dependencies to converge on the newer toolchain.
- Fix linter and test issues exposed by the stricter Go 1.26.1 checks.
Fixed CVEs
- CVE-2026-27137: Go stdlib
1.26.0->1.26.1fixes incomplete email-constraint enforcement incrypto/x509. - CVE-2026-27138: Go stdlib
1.26.0->1.26.1fixes acrypto/x509panic triggered by malformed certificates. - CVE-2026-25679: Go stdlib
1.26.0->1.26.1fixes insufficient validation of IPv6 host literals innet/url. - CVE-2026-27139: Go stdlib
1.26.0->1.26.1fixesFileInfometadata escaping theRootboundary inos. - CVE-2026-27142: Go stdlib
1.26.0->1.26.1fixes missing URL escaping inhtml/templateformeta refreshcontent. - CVE-2026-26958:
filippo.io/edwards25519v1.1.0->v1.2.0fixes incorrect or undefinedMultiScalarMultbehavior. - CVE-2025-10543:
github.com/eclipse/paho.mqtt.golangv1.5.0->v1.5.1fixes incorrect MQTT packet encoding for oversized UTF-8 strings. - CVE-2026-24051:
go.opentelemetry.io/otel/sdkv1.38.0->v1.42.0fixes the macOS PATH-hijacking arbitrary code execution issue. - CVE-2026-33186:
google.golang.org/grpcv1.77.0->v1.79.3fixes authorization bypass caused by a missing leading slash in the HTTP/2:pathpseudo-header.
Related Commits
2.35 - Silo 20260314 Released
Published: 2026-03-14 · Version: RELEASE.2026-03-14T12-00-00Z
This release switches the project to the community-maintained Console fork and performs a sizeable dependency refresh to establish the base for the later Go 1.26.x maintenance releases.
Major Changes
- Switch to the community-maintained
georgmangold/console v1.9.1fork in place of the unmaintainable upstream Console dependency. - Refresh a large portion of the direct and indirect dependency graph so the new Console and toolchain combination builds cleanly.
- Fix the
go vetformat directive issue ingrid_test.goand adjust tests for the HTTP behavior changes in Go 1.26.
Fixed CVEs
- CVE-2025-47913:
golang.org/x/cryptov0.37.0->v0.46.0fixes a panic inssh/agentwhen handling malformed responses. - CVE-2025-58181:
golang.org/x/cryptov0.37.0->v0.46.0fixes unbounded memory consumption insshGSSAPI authentication parsing. - CVE-2025-47914:
golang.org/x/cryptov0.37.0->v0.46.0fixes a panic inssh/agentcaused by malformed identity messages. - CVE-2025-47911:
golang.org/x/netv0.39.0->v0.48.0fixes quadratic parsing complexity inhtml.Parsefor crafted inputs. - CVE-2025-58190:
golang.org/x/netv0.39.0->v0.48.0fixes an infinite parsing loop ingolang.org/x/net/html.
Related Commits
2.36 - Silo 20260214 Released
Published: 2026-02-14 · Version: RELEASE.2026-02-14T12-00-00Z
This early infrastructure-focused community release restores the embedded Console, introduces GitHub CI/CD, and lifts the Go baseline to 1.26.0, which also absorbs a batch of security fixes from the older toolchain generation.
Major Changes
- Restore the embedded Console and refresh the README to clarify the community fork position.
- Add GitHub CI/CD workflows as the base for automated builds and multi-platform delivery.
- Add quick links for docs, Docker, the GitHub repository, and installation through the
pigpackage manager.
Fixed CVEs
These issues were absorbed as part of the Go 1.25.5 -> 1.26.0 upgrade:
- CVE-2025-68121:
crypto/tlscould incorrectly accept mutated CA configuration during session resumption. - CVE-2025-61730: TLS 1.3 could process handshake messages incorrectly across encryption-level boundaries.
- CVE-2025-61726:
net/urlquery parsing could be abused for memory exhaustion. - CVE-2025-61728:
archive/zipcould consume excessive CPU while building archive indexes. - CVE-2025-68119:
cmd/gocould trigger unexpected code execution when invoking external VCS tooling. - CVE-2025-61731: the
#cgo pkg-config:directive could be abused for arbitrary file writes. - CVE-2025-61732:
cmd/cgocomment parsing discrepancies could enable code smuggling.
Related Commits
- 8630937: Restore embedded console and update README for community fork
- 68521b3: add github ci/cd pipeline
Historical note: CI/CD pipeline evidence links maintained-history commit 68521b37f, which preserves the pipeline files and includes a README adjustment.
2.37 - Silo 20251203 Released
Published: 2025-12-15 · Version: RELEASE.2025-12-03T12-00-00Z
This is the earliest traceable community release, establishing the community packaging and distribution baseline.
Major Changes
- Build the community packaging flow around
minio/pkger. - Choose a maintenance-mode upstream MinIO baseline as the starting point for the community-maintained fork.
- Produce the first
apk,deb, andrpmartifacts for ongoing community releases.
Fixed CVEs
- Initial community release baseline; no delta CVE list against an earlier community version applies.
Related Commits
- d4cd4b4: RELEASE.2025-12-03T12-00-00Z with go 1.25.5
3 - SILO Security Chronicle
This section publishes technical security advisories for PGSTY SILO, ordered from newest to oldest. Each advisory documents the affected versions, trigger conditions, root cause, code fix, and upgrade requirements.
For a single-page index of all CVE-* and SN-2026-* identifiers, fix commits, and dependency security updates, see the Security Advisory Ledger.
3.1 - October Security Update: Object Lock, Admin Credentials and JWK Parsing
SILO Server 20261018 delivers the fixes for SN-2026-015, SN-2026-016, and SN-2026-017, scoped STS token revocation, and Go toolchain and dependency updates across silo, mcli, and silo-console. Server 20260916 and 20260903 are affected (and 20261015 was a superseded pre-release draft).
Object Lock: SN-2026-015 and SN-2026-016
The SN-2026-015 & SN-2026-016 advisory and repair design note document the affected PutObjectRetention and in-place version overwrite paths, reproductions, and mitigations. PR #234 (14042b5aa) and PR #235 (0d3ba5d4c) enforce s3:PutObjectRetention before object lookup, require both x-amz-bypass-governance-retention: true and s3:BypassGovernanceRetention when weakening active GOVERNANCE retention, and reject caller-addressed ?versionId= overwrites while enforcing WORM checks under the object write lock for null-version overwrites.
Every cluster and replication site that accepts S3 writes must be upgraded to 20261018.
SN-2026-017: Admin credential authorization boundaries
SN-2026-017 groups the authenticated Admin API handler authorization defects fixed in PR #242 (commit 2616d43fb) and its bulk access-key listing follow-up 995439260.
| Boundary | Before 20261018 |
20261018 Behavior |
|---|---|---|
| Self inspection and cleanup | Ownership could override an explicit policy or plugin denial; self service-account listing could skip authorization | Preserves self-service access for ordinary users (DenyOnly), while enforcing explicit user/group Deny rules, inline session policies, and external plugin denials |
| Restricted credential delegation | A derived credential allowed the create/update action could mint a key inheriting broader parent permissions or strip a target service account’s session policy | Under built-in authorization, callers constrained by an active inline session policy cannot create or update service accounts (including LDAP creation and metadata-only updates) |
| Root-derived credentials | A non-root administrator with broad admin:* grants could inspect, modify, or delete root’s derived service accounts |
Only an authorized root credential or an unrestricted root-derived identity can manage root-owned credentials (the internal site-replication account is excluded) |
The actor passed to IAM policy evaluation is always the authenticated access key (cred.AccessKey), while parent resolution (ParentUser) is used only for comparing ownership. Full handler contracts and benchmark measurements are recorded in IAM Admin API Authorization Boundaries.
Compatibility repairs retained alongside the security fix
- Semantic empty policy normalization on issuance: Creating a service account filters reserved parent policy/expiration claims and normalizes semantic empty policy documents (
null,{},{"Statement":null}) to parent inheritance, while preserving explicit empty statement arrays ({"Version":"2012-10-17","Statement":[]}). - Damaged non-root key cleanup: Deleting or inspecting a damaged non-root service account falls back to stored identity metadata so operators can clean up corrupted or expired keys without allowing a damaged token to mask a root parent.
- Bulk access-key listing (
995439260): Applies the same self-service authorization helper to bulk key listing (ListAccessKeysBulk), resolving omitted users, the caller’s own key, or its parent to the caller’s parent inventory while enforcing explicit denies and cross-user restrictions. - Error codes: Denied self-service operations return
403 AccessDenied; unauthorized attempts to delete another user’s or root’s service account retain their404response.
Operator action
- Verify administrative automation using your production user, service-account, STS, and external-plugin identities. Any script that relied on a session-restricted credential to create further child service accounts, or a non-root admin to manage root service accounts, must switch to an unrestricted parent or root identity.
- Audit child service accounts and policy updates created prior to upgrading, revoke unexpected keys, and follow the IAM upgrade and recovery guide.
Follow-up: scoped STS revocation
Release commit 0cd388c66 fixes cache-only RevokeTokens enumeration and incomplete propagation across peer nodes and sites. It persists deletion tombstones for tokens durably known to the cluster—scoped by parent identity, token type, and operation cutoff timestamp—and propagates them through peer notification and deletion healing. See Scoped STS Revocation and the Server 20261018 release notes.
Dependency security and login compatibility
Server 20261018 builds with Go 1.27.2, golang.org/x/net v0.61.0, golang.org/x/crypto v0.58.0, golang.org/x/text v0.43.0, silo-pkg v3.14.3, and go-jose v4.1.6 in go.mod:
go-josev4.1.6 (GHSA-mgcc-cww5-fp2q): Bounds CPU consumption when parsing crafted RSA JWKs on the OIDC path and rejects RSA JWKs larger than 4096 bits. Verify OIDC discovery, JWKS endpoints, and login flows before upgrading if your identity provider uses oversized RSA keys.- Standalone Console: See Console v2.4.2 for standalone Console dependency updates.
Verification and scope
Handler regressions in cmd/ cover explicit self denials, session-restricted delegation, root-credential protection, empty-policy issuance, damaged-key cleanup, decrypted bulk-list identity contents, and Object Lock WORM invariants. See the IAM authorization design note, Object Lock repair design note, and component version matrix.
3.2 - SN-2026-015 & SN-2026-016: S3 Object Lock Governance Retention Bypass and Version-Addressed Overwrites
SN-2026-015 and SN-2026-016 cover two S3 Object Lock retention and immutability vulnerabilities fixed in SILO Server 20261018 via PR #234 (14042b5aa) and PR #235 (0d3ba5d4c), as detailed in the companion repair design note. While the delete path already enforced s3:BypassGovernanceRetention (CVE-2023-25812), the retention-write path (PutObjectRetention) and four in-place version overwrite paths remained bypassable.
Advisory Scope: SN-2026-015 and SN-2026-016
SN-2026-015: PutObjectRetention Authorization and Governance Bypass (#231)
In PutObjectRetentionHandler (cmd/object-handlers.go), three interacting flaws in enforceRetentionBypassForPut and isPutRetentionAllowed broke S3 Object Lock authorization:
- F1 (Unconditional
s3:PutObjectRetentionbypass via header): Passingx-amz-bypass-governance-retention: truecausedisPutRetentionAllowedto check onlys3:BypassGovernanceRetention(!byPassSet), allowing an authenticated principal with zeros3:PutObjectRetentionand zeros3:BypassGovernanceRetentionpermissions to place or extend retention on any unlocked object! - F2 (Governance retention shortening/clearing without
s3:BypassGovernanceRetention): BecauseisPutRetentionAllowedevaluatedbyPassSet || retSet, a caller who helds3:PutObjectRetention(retSet == true) but lackeds3:BypassGovernanceRetention(byPassSet == false) could passx-amz-bypass-governance-retention: trueto clear (<Retention/>) or shorten an activeGOVERNANCEretention lock, and then delete the object (GHSA-67mv-7hc9-wj88, reported byszopi00). - F3 (
s3:object-lock-remaining-retention-dayscondition bypass on governance bypass): Whenx-amz-bypass-governance-retention: truewas set on an existing governance-locked object,s3:PutObjectRetentionwas skipped whenevers3:BypassGovernanceRetentionsucceeded, bypassing IAM policy conditions such asNumericGreaterThanEquals: {"s3:object-lock-remaining-retention-days": 30}attached tos3:PutObjectRetention.
SN-2026-016: Version-Addressed and In-Place Object Overwrites (#234)
While auditing every write path that can mutate an existing object version in place, we found and closed four write-path bypasses (14042b5aa):
- Caller-supplied
?versionId=onPutObjectandCopyObject: Non-replication callers could pass?versionId=<existing-uuid>onPutObjectorCopyObject(orminio.versionIdin SnowballPutObjectExtractPAX headers) to overwrite an existing version in place without checking WORM retention or legal hold on that target version. - Directory object (
key/)null-version overwrite: Creating or overwriting a directory object (strings.HasSuffix(object, "/")) wrote tonullVersionIDeven in versioned/Object-Lock buckets without checking existing retention or legal hold on thenullversion. CompleteMultipartUploadin-placenull-version overwrite: In an Object-Lock-enabled bucket with versioning suspended (or targeting an existingnullversion),CompleteMultipartUploaddid not enforce retention/legal-hold checks on an existingnullversion before replacing it.- Replication header trust: Unprivileged callers with
s3:PutObjectcould spoofX-Minio-Source-Replication-Requestheaders unless gated bys3:ReplicateObject.
Expected S3 Object Lock Semantics
Under the AWS S3 Object Lock specification, PutObjectRetention must enforce two independent orthogonal boundaries:
| Target Object State | Proposed <Retention> Change |
Required IAM Permissions | Required Request Header |
|---|---|---|---|
| Unlocked, or retention expired | Set GOVERNANCE or COMPLIANCE with future date |
s3:PutObjectRetention |
None |
Active GOVERNANCE lock |
Extend RetainUntilDate to a strictly later date (same mode) |
s3:PutObjectRetention |
None |
Active GOVERNANCE lock |
Clear (<Retention/>), shorten RetainUntilDate, or switch GOVERNANCE -> COMPLIANCE |
s3:PutObjectRetention AND s3:BypassGovernanceRetention |
x-amz-bypass-governance-retention: true |
Active COMPLIANCE lock |
Extend RetainUntilDate to a strictly later date (COMPLIANCE mode) |
s3:PutObjectRetention |
None |
Active COMPLIANCE lock |
Clear, shorten, or switch COMPLIANCE -> GOVERNANCE |
Forbidden for all principals | Forbidden (AccessDenied) |
Reproduction and Verification
The three SN-2026-015 failures can be reproduced directly against pre-20261018 builds (verified against baseline b0a540190):
Reproduction 1: Clearing an Active Governance Lock Without s3:BypassGovernanceRetention (F2)
Assume user nobypass has Allow on s3:PutObjectRetention and s3:DeleteObjectVersion, and no s3:BypassGovernanceRetention grant:
| Step | Action by nobypass |
Pre-20261018 Result |
Server 20261018 Result |
|---|---|---|---|
| 1 | Object doc.txt (version v1) is locked in GOVERNANCE mode until 2030-01-01 |
Locked | Locked |
| 2 | DELETE /bucket/doc.txt?versionId=v1 |
403 AccessDenied |
403 AccessDenied |
| 3 | PUT /bucket/doc.txt?retention&versionId=v1 with header x-amz-bypass-governance-retention: true and body <Retention/> |
200 OK (lock cleared!) |
403 AccessDenied |
| 4 | DELETE /bucket/doc.txt?versionId=v1 |
204 No Content (version permanently deleted!) |
Object remains locked and protected |
Reproduction 2: Locking an Object with Zero Retention Permissions (F1)
Assume user zeroperm has an empty/unrelated policy with neither s3:PutObjectRetention nor s3:BypassGovernanceRetention:
| Step | Action by zeroperm |
Pre-20261018 Result |
Server 20261018 Result |
|---|---|---|---|
| 1 | PUT /bucket/unlocked.txt?retention (without bypass header), body <Mode>COMPLIANCE</Mode><RetainUntilDate>2099-01-01T00:00:00Z</RetainUntilDate> |
403 AccessDenied |
403 AccessDenied |
| 2 | Repeat the exact same PUT /bucket/unlocked.txt?retention adding header x-amz-bypass-governance-retention: true |
200 OK (irreversible 2099 COMPLIANCE lock applied!) |
403 AccessDenied |
Reproduction 3: Bypassing s3:object-lock-remaining-retention-days Conditions (F3)
Assume user govadmin holds s3:BypassGovernanceRetention unconditionally, and holds s3:PutObjectRetention conditioned on NumericGreaterThanEquals: {"s3:object-lock-remaining-retention-days": 30}. On an object already under GOVERNANCE lock, sending x-amz-bypass-governance-retention: true with a 5-day retention period previously succeeded (byPassSet == true satisfied byPassSet || retSet even though retSet was false), bypassing the 30-day minimum policy condition.
Root Cause Analysis
In cmd/object-handlers.go, PutObjectRetentionHandler authenticated the caller (authenticateRequest) and delegated all authorization to enforceRetentionBypassForPut, which called isPutRetentionAllowed:
And in enforceRetentionBypassForPut:
- When the existing object had no active retention (
opts.Mode == ""), the function checkedif !retSet && !govBypassand returnednilwheneverbyPassSet(x-amz-bypass-governance-retention: true) was true—because on the unlocked path it never even calledisPutRetentionAllowedproperly, or trusted the header boolean directly (F1)! - When the existing object was in
GOVERNANCEmode and the caller passedx-amz-bypass-governance-retention: true,isPutRetentionAllowedreturnedbyPass || retSetas the second return value (govBypass). ThusretSet == true(s3:PutObjectRetentionalone) madegovBypass == true(F2), and converselybyPass == true(s3:BypassGovernanceRetentionalone) madegovBypass == trueeven whenretSet == false(F3).
Upstream History
Tracing git log shows how three separate changes over five years combined to create F1–F3:
- 2020-04 (
43a3778b4,minio/minio#9259): IntroducedenforceRetentionBypassForPutandisPutRetentionAllowedwith thebyPass || retSetboolean conflation. - 2025-02 (
437dd4e32,minio/minio#20929): Refactored retention checks inPutObjectRetentionHandler. - 2025-04 (
8c7097528,minio/minio#21103): Replaced the entrycheckRequestAuthTypecall inPutObjectRetentionHandlerwithauthenticateRequest(which only checks SigV4 signature validity, nots3:PutObjectRetention), leavingenforceRetentionBypassForPutas the sole authorization gate and exposing F1 to any authenticated user with zero permissions.
Repair Design
Five Security Invariants
Our repair enforces five explicit invariants across PutObjectRetention (0d3ba5d4c) and object write paths (14042b5aa):
- Invariant 1 (Mandatory
s3:PutObjectRetention): EveryPutObjectRetentionrequest must satisfys3:PutObjectRetention(including all policy conditions such ass3:object-lock-remaining-retention-days,s3:object-lock-retention-mode, ands3:object-lock-retain-until-date), regardless of whetherx-amz-bypass-governance-retentionis present. - Invariant 2 (Additional
s3:BypassGovernanceRetentionon governance weakening): If the target version has an active (unexpired)GOVERNANCElock and the request clears retention (Mode == "") or shortensRetainUntilDate(!RetainUntilDate.After(existing.RetainUntilDate)), the request must additionally carryx-amz-bypass-governance-retention: trueand satisfys3:BypassGovernanceRetention(including itss3:ExistingObjectTag/<key>conditions). - Invariant 3 (
COMPLIANCEimmutability): An active (unexpired)COMPLIANCElock can never be cleared, shortened, or downgraded toGOVERNANCEby any principal. - Invariant 4 (No caller-specified version overwrite): Non-replication S3 clients cannot supply
?versionId=onPutObject,CopyObject, orPutObjectExtractto overwrite an existing version ID. - Invariant 5 (In-place
null-version write-lock enforcement): Any write path that replaces an existingnullversion in place (PutObjecton a suspended-versioning bucket, directory objectkey/, orCompleteMultipartUpload) must enforce existing WORM retention and legal-hold checks on thatnullversion.
Where Each Check Lives
In PutObjectRetentionHandler and enforceRetentionBypassForPut:
s3:PutObjectRetention(retOK) is always evaluated against the proposed retention parameters (retDays,retDate,retMode) and existing object tags.s3:BypassGovernanceRetention(bypassOK) is evaluated independently and required whenever an activeGOVERNANCElock is cleared or shortened (or when switching an activeGOVERNANCElock toCOMPLIANCEwith a shorter date).
Complete Decision Table for PutObjectRetention
| Row | Existing Lock State | Requested Mutation | x-amz-bypass-governance-retention |
retOK (s3:PutObjectRetention) |
bypassOK (s3:BypassGovernanceRetention) |
Outcome |
|---|---|---|---|---|---|---|
| 1 | Any | Any | false or true |
false |
Any | 403 AccessDenied (closes F1 & F3) |
| 2 | Unlocked or expired | Set future retention | false or true |
true |
Not required | 200 OK |
| 3 | Active GOVERNANCE |
Extend RetainUntilDate (same mode or upgrade to COMPLIANCE with >= date) |
false or true |
true |
Not required | 200 OK |
| 4 | Active GOVERNANCE |
Clear (<Retention/>) or shorten RetainUntilDate |
false |
true |
Any | 403 AccessDenied |
| 5 | Active GOVERNANCE |
Clear (<Retention/>) or shorten RetainUntilDate |
true |
true |
false |
403 AccessDenied (closes F2) |
| 6 | Active GOVERNANCE |
Clear (<Retention/>) or shorten RetainUntilDate |
true |
true |
true |
200 OK |
| 7 | Active COMPLIANCE |
Clear, shorten, or downgrade to GOVERNANCE |
false or true |
Any | Any | 403 AccessDenied |
Five Rejected Alternatives
During design review we evaluated and rejected five simpler-looking patches:
- Changing only
byPass || retSettobyPass && retSetinsideisPutRetentionAllowed: Would fix F2 and F3, but would still leave F1 open on unlocked objects (whereisPutRetentionAllowedwas bypassed) and would wrongly requires3:BypassGovernanceRetentionwhen a user withs3:PutObjectRetentionextends a lock while happening to passx-amz-bypass-governance-retention: true. - Adding a
checkRequestAuthType(..., PutObjectRetentionAction)call at the top ofPutObjectRetentionHandler: At handler entry, the XML body (<Retention>) has not been parsed yet and the existing object’s tags have not been read from disk, so policy conditions ons3:object-lock-remaining-retention-days,s3:object-lock-retain-until-date,s3:object-lock-retention-mode, ands3:ExistingObjectTag/<key>would all fail! - Requiring
s3:BypassGovernanceRetentionwheneverx-amz-bypass-governance-retention: trueis present on the request: S3 only requiress3:BypassGovernanceRetentionwhen the operation actually needs governance bypass (clearing or shortening an activeGOVERNANCElock), not when locking an unlocked object or extending an existing lock. - Blocking
<Retention/>empty XML payloads: Clearing an activeGOVERNANCElock via<Retention/>is a documented MinIO/SILO extension used bymcli retention clear; it must remain supported when the caller holds boths3:PutObjectRetentionands3:BypassGovernanceRetentionand sendsx-amz-bypass-governance-retention: true. - Enforcing
SN-2026-016?versionId=rejection only on Object-Lock-enabled buckets: Allowing arbitrary clients to overwrite historical versions in place via?versionId=violates S3 versioning immutability on all versioned buckets, not just Object Lock buckets.
Implementation Notes
ParseObjectRetentionand empty<Retention/>(s3:object-lock-remaining-retention-days): When a caller sends<Retention/>to clear governance retention,retDateis the zerotime.Time{}. Previously,getRetentionDayscomputed a negative integer fromtime.Until(zeroTime)(~-106751days due totime.Duration290-year saturation!). Commit0d3ba5d4cnormalizes zeroretDateto0days and computes future remaining days using non-saturating calendar/seconds arithmetic so dates beyond year 2316 do not overflowtime.Duration.s3:ExistingObjectTag/<key>propagation: Both thes3:PutObjectRetentionands3:BypassGovernanceRetentionchecks receive the target version’s stored tags so tag-based IAM policies evaluate accurately.
Compatibility
- Standard S3 and
mcli retentionworkflows: Fully compatible. Callers that clear or shorten governance retention viamcli retention clear --governancemust hold boths3:PutObjectRetentionands3:BypassGovernanceRetention. - Active-active / site replication: Legitimate replication requests that preserve source version IDs must hold
s3:ReplicateObject(included in standard replication policies).
Regression Test Matrix
PR #234 and PR #235 lock down the full matrix in cmd/object-handlers-lock_test.go and cmd/bucket-versioning_test.go:
- All 7 rows of the
PutObjectRetentiondecision table (F1 zero-permission caller, F2PutObjectRetention-only caller, F3BypassGovernanceRetention-only caller withs3:object-lock-remaining-retention-daysconditions), - Far-future retention dates (
2400-01-01T00:00:00Z) exceedingtime.Duration’s 290-year limit, PutObject,CopyObject, andPutObjectExtractwith caller-supplied?versionId=/minio.versionId,- Directory object (
key/) andCompleteMultipartUploadin-place overwrites against lockednullversions.
Mitigation and Upgrade
Delivery Status
Both SN-2026-015 (0d3ba5d4c, PR #235) and SN-2026-016 (14042b5aa, PR #234) are shipped in SILO Server 20261018. Operators relying on S3 Object Lock WORM governance or compliance retention should upgrade to 20261018 immediately.
3.3 - September Security Repairs: Payload Integrity, IAM Revocation and Console Sharing
This bulletin covers three security repairs (SN-2026-012 through SN-2026-014) and adjacent dependency updates shipped in Server 20260916 and Console v2.4.1. See the security advisory ledger for the complete index.
| Finding | Fixed source | Delivery status |
|---|---|---|
SN-2026-012: presigned payload integrity |
Server c4b5e1cb4, PR #177 |
Shipped in Server 20260916 (affected in RELEASE.2026-09-03T13-18-01Z and earlier) |
SN-2026-013: durable IAM revocation |
Server PR #191 / PR #192 | Shipped in Server 20260916; requires coordinated multi-node/multi-site upgrade |
SN-2026-014: anonymous Console share proxy |
Console PR #56; Server PR #209 | Shipped in standalone Console v2.4.1 and embedded in Server 20260916 |
Presigned payload integrity — SN-2026-012
The generic authenticated handler path could verify a presigned request using a payload digest supplied via the X-Amz-Content-Sha256 header without verifying the actual request body bytes against that digest. A caller holding a presigned write URL could therefore substitute a different request body (the regression suite reproduces this against presigned PutBucketPolicy).
Commit c4b5e1cb4 in PR #177 binds body digest verification to the effective signed payload hash; any mismatch fails with XAmzContentSHA256Mismatch (400 Bad Request). An explicitly signed UNSIGNED-PAYLOAD token retains its standard meaning. The same PR also ensures IAM policy condition evaluation reads the authenticated signed values (with signed query parameters taking precedence over fallback headers) and removes the internal X-Amz-Signature-Age scratch header, complementing SN-2026-011.
Before upgrading, verify that custom SigV4 presigning clients send body payloads matching any signed X-Amz-Content-Sha256 digest.
Durable IAM revocation — SN-2026-013
Previously, delayed site-replication events, parent account recreation, or missing deletion history could resurrect identities or policy grants that an administrator had already deleted or revoked. PR #191 and PR #192 persist monotonic source revisions, deletion tombstones, and parent-revocation epochs embedded in child credentials so stale replication events and pre-revocation child tokens are rejected.
Because this change introduces persistent IAM revision and tombstone records:
- Upgrade all nodes sharing an IAM backend and all participating replication sites together; mixed-version operation and rolling downgrades are not supported.
mcli admin cluster iam exportexports active records only and omits deletion tombstones; maintain a full backend snapshot and follow the IAM upgrade and recovery guide.- See the IAM revocation design note for revision ordering, metrics, and group-membership scope.
Anonymous Console share proxy — SN-2026-014
Jiri Pejchal reported that the unauthenticated Console shared-download proxy accepted URLs pointing to the configured Server origin outside the intended presigned object-download surface.
Console PR #56 restricts scheme, host, and port to the configured S3 origin, requires a valid /{bucket}/{object} path, rejects reserved system prefixes, path traversal segments, and operation-changing query parameters, disables upstream HTTP redirects, and forwards caller context cancellation while preserving the exact signed URL bytes for valid object downloads.
Upgrade standalone Console deployments to Console v2.4.1, and upgrade SILO Server deployments (which embed Console via Server PR #209) to Server 20260916.
Adjacent dependency hardening
Server 20260916 upgrades github.com/rabbitmq/amqp091-go to v1.14.0 to address GHSA-6c5v-hqjr-5xxp (fixed upstream in v1.13.0), where a malicious AMQP broker could trigger excessive memory allocation when an AMQP bucket notification target is configured. It also adopts silo-pkg v3.14.1 with the updated JWX JSON encoding dependency.
3.4 - SN-2026-011: Fix and Release Status
SN-2026-011 was fixed on main starting with 123325430 on 2026-09-13 and shipped in Server 20260916. Server RELEASE.2026-09-03T13-18-01Z and earlier releases are affected (silo-pkg v3.14.0 and mcli 20260913 are client/library releases and do not patch existing Server deployments).
A holder of a signed PUT request could append an unsigned x-amz-copy-source header and turn the authorized upload into a server-side CopyObject of another object readable by the signing key. If the destination permits anonymous reads, the copied object’s contents become readable without holding the signing credentials. Both presigned URLs and Authorization-header requests were affected.
The fix verifies received x-amz-* headers against the request’s signed-header set (retaining only explicit protocol exceptions) before dispatching the operation. Follow-up request-signing and checksum coverage is documented in the signed-header design note.
Remediation
Upgrade SILO Server to Server 20260916 or later. Until all Server nodes are upgraded, scope write-signing credentials to the minimum required object prefixes and avoid granting broad read permissions to principals that issue presigned uploads into publicly readable buckets.
Reported by Oren Yomtov. See the security advisory ledger, the September 16 release hardening summary, and the component version matrix.
3.5 - CVE-2025-62506: Session-Policy Privilege Escalation
Status: Inherited and released
First SILO community release: RELEASE.2025-12-03T12-00-00Z
Upstream fixed release: RELEASE.2025-10-15T17-29-55Z
GitHub advisory: GHSA-jjjj-jwhf-8rgr
Upstream fix: minio/minio#21642
A service account or STS account constrained by a restricted session policy could invoke an “own account” Admin API operation to create a child service account without the session-policy restriction. Because the policy check set DenyOnly = true for own-account operations, any session policy that lacked an explicit Deny rule allowed the call, and the newly created child service account inherited the broader permissions of the parent user.
SILO did not require a separate backport. The upstream fix and its two regression test groups were already present in the Git history from which the first SILO community release (RELEASE.2025-12-03T12-00-00Z) was cut.
Commit mapping
| Role | Repository | Commit | Evidence |
|---|---|---|---|
| Upstream remediation | minio/minio |
c1a49490 |
Merge commit for minio/minio#21642 |
| SILO inheritance | pgsty/silo |
c1a49490 |
Preserves the exact Git commit object and SHA; RELEASE.2025-12-03T12-00-00Z and all subsequent SILO releases descend from it |
The identical commit SHA establishes direct source inheritance from upstream history. The Security Advisories ledger indexes the same commit mapping.
Regression evidence
Upstream PR #21642 added two regression test groups, each exercising both root and non-root parent accounts:
TestServiceAccountPrivilegeEscalationBug2_2025_10_15(cmd/admin-handlers-users_test.go) verifies that restricted service accounts cannot escalate privileges by creating unrestricted child service accounts.TestSTSPrivilegeEscalationBug2_2025_10_15(cmd/sts-handlers_test.go) verifies the same boundary for restricted STS temporary credentials.
Both test groups run as part of the standard go test ./cmd suite.
Fix and operator boundary
Whenever a session policy (sub-policy) is attached to the caller’s credential, the policy evaluator clears the DenyOnly flag before evaluating the session policy. An own-account operation can no longer pass simply because the session policy lacks an explicit Deny; the restricted session policy must explicitly grant Allow for the action.
Every SILO release contains c1a49490. Operators migrating from an upstream MinIO build older than RELEASE.2025-10-15T17-29-55Z should upgrade to SILO, audit service accounts created by restricted service or STS identities prior to the upgrade, and revoke any unauthorized child accounts.
Related advisory: CVE-2026-32285
Several vulnerability feeds later published CVE-2026-32285 with text describing github.com/buger/jsonparser while pointing to commit c1a49490. For the dependency reachability investigation and why no code change was made for that advisory, see CVE-2026-32285: The jsonparser Advisory That Required No Patch.
3.6 - SILO 20260903 Security Notes: SN-2026-006 through 010
This bulletin covers five security findings fixed in SILO Server 20260903 (RELEASE.2026-09-03T13-18-01Z), spanning customer-provided encryption keys (SSE-C), client-supplied replication headers, administrative status transitions, and explicit object-version deletion. SN-2026-006 through SN-2026-010 are SILO security-note identifiers grouping the pre-release audit findings inherited from the upstream minio/minio codebase.
Summary
| ID | Required attacker position | Security failure | Released behavior |
|---|---|---|---|
SN-2026-006 |
Read access to a zero-byte SSE-C object | A wrong customer key was accepted because no data block was decrypted | The object key is authenticated; wrong keys return 403 AccessDenied |
SN-2026-007 |
Read access to an SSE-C object, or a client-supplied replication marker | GetObjectAttributes returned protected metadata without authenticating the customer key |
Ordinary requests need the correct key; only an authorized replication peer uses the replica exception |
SN-2026-008 |
Any authenticated principal able to invoke the affected read/write/delete operation | Internal-looking headers granted replication-only effects without replication authorization | Exact marker plus s3:ReplicateObject or s3:ReplicateDelete is required |
SN-2026-009 |
Authenticated admin API principal holding only one status action | The enable action authorized disable, and vice versa | The requested target state selects the required admin action |
SN-2026-010 |
Authenticated S3 principal with s3:DeleteObject but not s3:DeleteObjectVersion |
A caller could permanently delete an explicit version without the AWS-required action | Explicit version deletion requires s3:DeleteObjectVersion |
SN-2026-006: zero-byte SSE-C key authentication
Affected operations: GetObject, HeadObject, CopyObject source processing, and GetObjectAttributes for zero-byte objects encrypted with SSE-C.
Tracking: issue #82.
Fixes: b73581b05 and c4fd97d0b.
SSE-C stores a sealed object key and requires the caller to supply the matching customer key on reads. Normal reads authenticate that key while preparing the encrypted payload stream. Because a zero-byte object has no payload block, the legacy fast path returned before unsealing the stored object key.
As a result, an authenticated caller with read access to a zero-byte SSE-C object could receive a successful response while supplying an incorrect customer key, or copy the object under a newly chosen key without proving knowledge of the original key.
The fix authenticates the stored SSE-C key regardless of payload length. Valid keys succeed as before; wrong keys fail with 403 AccessDenied, including null-version and key-rotation paths.
SN-2026-007: SSE-C GetObjectAttributes
Affected operation: GetObjectAttributes on SSE-C objects.
Tracking: issue #84.
Fixes: 474cd5801, 74c97d005, and 21870fa2e.
GetObjectAttributes exposes object size, ETag, checksums, storage class, and multipart part metadata. The inherited handler returned those attributes without authenticating the supplied SSE-C key, and also treated the raw X-Minio-Source-Replication-Request header as permission to skip key verification.
The fix enforces two distinct rules:
- an ordinary S3 caller must supply the valid SSE-C customer key (wrong or missing keys return
403 AccessDenied); - a replication peer may use the replica metadata path only after passing authentication and
s3:ReplicateObjectauthorization.
SN-2026-008: replication headers are not authority
Affected surface: object reads and writes, multipart completion, object and version deletion, Snowball extraction, Object Lock timestamps, checksum metadata, and bucket events.
Tracking: PR #101.
Primary fix range: 938603458 through 04b097fd9, followed by Snowball and rule-prefix hardening.
While CVE-2026-34204 blocked ordinary PUT and COPY requests from importing a subset of replication SSE metadata, a wider audit showed that several downstream handlers still treated the presence of internal replication headers as proof of a trusted replication request.
Depending on the operation, a caller holding ordinary read or write permissions could:
- trigger an SSE-C no-decryption read path without supplying the customer key;
- preserve a client-supplied source ETag or
MTime; - inject source checksum, actual-size, retention, legal-hold, or
REPLICAstatus metadata; - invoke replication-only delete semantics;
- suppress bucket notification events;
- carry replication trust across Snowball archive entries.
The fix establishes a single receiver-wide trust calculation:
- authenticate the incoming request in its original signed form;
- require a single
X-Minio-Source-Replication-Requestheader with the exact lowercase valuetrue; - authorize
s3:ReplicateObjectors3:ReplicateDeleteon the target resource; - grant
replicaTrustedonly when the authenticated replication status also equalsREPLICA; - bind the trust decision to a private
req.Context()key; - strip untrusted internal replication headers on a post-authentication request clone while preserving the trailer map for streaming checksum verification.
See No I/O Before Auth, No Privilege From Headers for the full protocol matrix, CORS interaction, black-box two-site replication verification, and rejected alternatives.
SN-2026-009: user and group status authorization
Affected operations: admin SetUserStatus and SetGroupStatus.
Tracking: PR #73 / PR #85.
Fixes: 58735ee38 and 229fe2b3c.
The inherited handlers checked admin:EnableUser or admin:EnableGroup before inspecting the requested status parameter. A delegated administrator permitted only to enable accounts could therefore disable them as well, while an administrator granted only admin:DisableUser or admin:DisableGroup was rejected.
The requested target state now selects the required admin action:
| Target | Required action |
|---|---|
| Enable user | admin:EnableUser |
| Disable user | admin:DisableUser |
| Enable group | admin:EnableGroup |
| Disable group | admin:DisableGroup |
admin:* and the built-in consoleAdmin policy remain unchanged; only custom least-privilege policies that relied on EnableUser/EnableGroup to perform both transitions require updating.
SN-2026-010: explicit object-version deletion
Affected operations: DeleteObject and each DeleteObjects entry carrying an explicit versionId.
Tracking: issue #58 and PR #104.
Fix range: 75a6734e4 through d2d47a41f.
Previously, the S3 delete handlers authorized explicit-version deletion under s3:DeleteObject and checked s3:DeleteObjectVersion only as a secondary deny filter, allowing callers with s3:DeleteObject alone to permanently delete historical versions.
Server 20260903 requires s3:DeleteObjectVersion whenever a versionId is specified. Unversioned DELETE requests continue to check s3:DeleteObject, and replication targets continue to check s3:ReplicateDelete. Callers granted only s3:DeleteObject can no longer delete explicit versions, and policies that pair Allow s3:* with Deny s3:DeleteObject to prevent permanent deletion must also deny s3:DeleteObjectVersion.
Operator actions
Before upgrading:
- identify policies and applications that delete explicit object versions and grant or deny
s3:DeleteObjectVersionexplicitly; - inspect custom admin policies for user and group enable/disable operations;
- confirm that replication service accounts hold
s3:ReplicateObject/s3:ReplicateDeleteand that ordinary application identities do not; - verify that SSE-C clients send the matching customer key on
GET,HEAD,GetObjectAttributes, andCopyObjectsource reads, including zero-byte objects.
After upgrading:
- verify both wrong-key (
403 AccessDenied) and valid-key responses on a test SSE-C object; - test object and delete replication across paired sites;
- test delegated user/group enable and disable permissions independently;
- verify explicit-version deletion with principals holding
s3:DeleteObjectalone vs.s3:DeleteObjectVersion.
Verification boundary
All five fixes are covered by unit, handler, authorization, streaming-trailer, Snowball, and site-replication regression tests. Full local acceptance on ebac0ca73 passed the cmd race suite and 174 test cases across six deployment topologies (174 PASS / 0 FAIL), followed by the RELEASE.2026-09-03T13-18-01Z release gates documented in the pre-release review.
Residual risks outside this bulletin
- Conditional deletion (
If-MatchonDeleteObjectand per-object ETag onDeleteObjects, #10) and bucket-metadata deletion convergence across site-replication peers (#77) are tracked separately; see the Server 20260903 release notes and Server 20260916 release notes.
3.7 - CVE-2026-32285: The jsonparser Advisory That Required No Patch
Status: Closed without a code change
GitHub issue: pgsty/silo#26
Referenced upstream commit: c1a49490 (CVE-2025-62506)
The initial assessment of CVE-2026-32285 assumed that the repository might still carry a vulnerable version of github.com/buger/jsonparser, and the team briefly considered replacing the package or maintaining a local fork. Checking the advisory’s referenced upstream commit, the resolved Go module version, and actual symbol reachability changed the conclusion:
- Public vulnerability feeds mislabeled
CVE-2026-32285as ajsonparserissue while pointing to upstream commitc1a49490, which is the session-policy privilege-escalation fix for CVE-2025-62506 already included in every SILO release sinceRELEASE.2025-12-03T12-00-00Z. - For
github.com/buger/jsonparseritself, the repository already resolvedv1.1.2, which contains the upstreamjsonparserfixes, andgovulncheckfound no reachable vulnerable symbol.
Instead of manufacturing a cosmetic dependency bump, we recorded the verification evidence and closed pgsty/silo#26.
What was wrong with the initial assumption
The issue was initially framed around the premise that jsonparser had no fixed upstream version in the tree. Acting on that premise without verifying the module graph and commit provenance would have caused real engineering harm:
- altering the dependency graph for zero security benefit;
- risking subtle behavioral regressions in JSON parsing paths for a nonexistent vulnerability;
- taking on long-term maintenance of an unnecessary dependency fork;
- misleading operators into believing that prior SILO releases were exposed when they were not.
Leaving already-remediated code unchanged is a deliberate security decision when backed by version and reachability evidence.
Investigation
The investigation verified four distinct layers:
- Commit provenance: Checked the commit linked by the CVE feed (
c1a49490) and confirmed it is the inherited session-policy fix (CVE-2025-62506). - Resolved module version: Verified
go.modandgo.sumto confirm thatgithub.com/buger/jsonparserresolves tov1.1.2, which already includes the upstreamjsonparserpatch. - Symbol reachability: Ran
govulncheckacross the tree and confirmed that no vulnerable symbol is reachable. - Feed discrepancy attribution: Traced the remaining alarm to mismatched metadata in external vulnerability feeds rather than unpatched code in the SILO tree.
When evaluating any dependency advisory, four questions must be kept separate: whether a package version was once listed as affected, whether the module is imported, whether a vulnerable symbol is reachable in the compiled call graph, and whether an external entry point can trigger that path with untrusted input.
Why there was no “just in case” upgrade
When the selected version in go.mod already contains the fix, bumping to an arbitrary commit or swapping libraries does not improve security. In a large Go module graph, unmotivated dependency churn expands the regression surface and obscures the cause of later behavioral changes.
We therefore took four concrete steps:
- closed the issue without a synthetic commit;
- recorded the commit provenance,
v1.1.2selection, andgovulncheckoutput in pgsty/silo#26; - retained automated
govulncheckandgo.modversion gates in CI to prevent accidental dependency downgrades; - indexed
CVE-2026-32285alongsideCVE-2025-62506in the Security Advisories ledger so operators auditing scanner reports can find an authoritative answer.
Verification boundary
The investigation on 2026-04-15 established that SILO’s module graph (github.com/buger/jsonparser v1.1.2) and commit history (c1a49490) required no source change for CVE-2026-32285. Any future change to module selection or jsonparser imports remains subject to the standard CI govulncheck gate.
Engineering principle
For dependency CVEs, verify in order: what commit or version the advisory actually points to, which version go.mod selects, whether any vulnerable symbol enters the call graph, and whether a runtime entry point exposes it. Only when those checks reveal an unpatched or reachable flaw should the investigation produce a code diff.
3.8 - CVE-2026-33322: OIDC JWT Algorithm Confusion
Status: Released
First containing release: RELEASE.2026-04-17T00-00-00Z
Affected entry points: AssumeRoleWithWebIdentity, AssumeRoleWithClientGrants
Fix commit: d24f449
GitHub issue: pgsty/silo#22
The previous OIDC implementation in internal/config/identity/openid placed the relying-party client_secret into the JWT verification keyring while also accepting HMAC signing algorithms (HS256, HS384, HS512). An attacker who obtained that client_secret could therefore sign a forged ID token with HMAC and exchange it through STS (AssumeRoleWithWebIdentity or AssumeRoleWithClientGrants) for temporary S3 credentials.
The final fix (d24f449) enforces asymmetric, JWKS-only verification and deliberately drops HS256/HS384/HS512 token support rather than keeping a configuration toggle that would reintroduce ambiguous trust semantics.
Why the vulnerability was key-role confusion, not missing signature verification
The vulnerable code did verify JWT signatures before issuing STS credentials. The failure lay in which keys the verifier accepted and whether the token’s alg header could select a signing algorithm whose trust semantics did not match the key’s intended role.
In OIDC, the client_secret authenticates the relying party (SILO) to the identity provider’s Token Endpoint. It is a shared credential between the relying party and the IdP—not an asymmetric signing key used by the IdP to assert user identities. By loading client_secret into the local token.Parse keyring alongside JWKS public keys, the verifier allowed a single credential leak to escalate into full identity forgery:
- The attacker obtained the configured OIDC
client_secret. - The attacker constructed a JWT claiming arbitrary user or role attributes and signed it with
HS256,HS384, orHS512using theclient_secret. - The verifier matched the HMAC algorithm against the
client_secretin its keyring and accepted the signature. - The forged token passed
AssumeRoleWithWebIdentityorAssumeRoleWithClientGrantsand returned valid temporary STS credentials.
A compatibility path was implemented, then deliberately removed
During development, we first implemented an explicit algorithm-allowlist and allow_hmac-style compatibility prototype (fc47fa0). On the surface, an opt-in flag appeared reasonable: enforce asymmetric verification by default while letting deployments with HMAC-based tokens opt in explicitly.
Design review overturned that approach. Keeping client_secret anywhere inside the local JWT verification keyring meant that enabling the option would silently widen the entire STS trust boundary: anyone holding the relying-party secret could again mint arbitrary user tokens. Furthermore, any future drift in the parser’s algorithm allowlist or JWKS retry path could reopen the vulnerability by default.
| Option | Benefit | Risk | Decision |
|---|---|---|---|
Keep client_secret in keyring and filter algorithms |
Minimal code diff; preserves HMAC IdPs | Still conflates Token Endpoint client auth with local JWT signature verification | Rejected |
Add an allow_hmac configuration option (fc47fa0) |
Makes HMAC compatibility explicit | Difficult for operators to reason about safely; expands parser and retry test surface | Implemented, then removed |
Enforce JWKS-only asymmetric verification (d24f449) |
Clean separation of key roles; primary parse and JWKS refresh share one parser policy | Identity providers issuing HS256/384/512 tokens must migrate |
Adopted |
Removing the completed compatibility implementation produced a much simpler and more auditable security boundary.
Final invariants
Commit d24f449 establishes four invariants in internal/config/identity/openid:
- Verifier keys are loaded exclusively from the identity provider’s JWKS endpoint (
jwks_uri). - The OIDC
client_secretnever enters the local JWT verification keyring. HS256,HS384, andHS512tokens are rejected unconditionally.- Both the initial token verification pass and the JWKS cache-refresh/retry path use the same asymmetric algorithm allowlist (
RS256,RS384,RS512,ES256,ES384,ES512).
We also kept the algorithm scope tight rather than expanding JOSE support during a security fix: PS256 and EdDSA remain unsupported.
Verification and release
The test suite in internal/config/identity/openid covers HS256 token rejection, RS256 token acceptance, and JWKS refresh/retry behavior when a key rotation occurs. All temporary compatibility helpers, configuration flags, and HMAC opt-in tests from the initial prototype were stripped from the final commit (d24f449), which shipped in RELEASE.2026-04-17T00-00-00Z.
Compatibility trade-offs
This fix introduces an intentional breaking change. Identity providers configured to issue HS256, HS384, or HS512 ID tokens must migrate to JWKS-backed RSA or ECDSA signing keys before upgrading SILO. If you suspect the OIDC client_secret was exposed prior to upgrading, rotate the secret in both the identity provider and SILO configuration.
3.9 - CVE-2026-33419: LDAP STS Enumeration and the Throttling Chain
Status: Released, followed by two rounds of hardening
First containing release: RELEASE.2026-04-17T00-00-00Z
Complete correction: RELEASE.2026-06-18T00-00-00Z
Fix commits: 3b950f8, 18b712d, 9e10f6d, f441108, 5e40665
GitHub issue: pgsty/silo#23
The original defect in AssumeRoleWithLDAPIdentity was straightforward: LDAP STS returned distinct errors for “user does not exist” and “password is wrong,” creating an unauthenticated username oracle. Our first patch unified the external authentication failure response and added rate-limit buckets keyed by both source IP and username.
Subsequent design reviews uncovered a deeper problem: naive success refunds, spoofable proxy headers, reservation accounting, and the shared username bucket itself turned the rate limiter into a targeted denial-of-service primitive. The final June design (5e40665) removed the shared username bucket that enabled cross-origin account lockouts, retained the per-source-IP bucket, and formalized proxy IP attribution as an explicit deployment contract.
Initial threat model
The vulnerable entry point is AssumeRoleWithLDAPIdentity. An attacker requires no existing S3 or LDAP credentials; network access to the STS endpoint is sufficient to compare the HTTP status code and error message returned for an unknown username versus a wrong password, enumerate valid directory accounts, and follow up with password spraying.
At the same time, remediation could not flatten every LDAP error into “invalid credentials.” Directory connection failures, lookup-bind credential errors, and LDAP server timeouts must still surface as HTTP 500 infrastructure errors so operators can detect and diagnose directory outages.
First round: uniform responses and a dual-key limiter
The initial patch on 2026-04-15 (3b950f8) introduced three changes:
- unknown-user lookups and bad-password binds both returned the same external
400 InvalidParameterValueSTS authentication error; - LDAP infrastructure failures continued to return HTTP
500, logging the underlying directory error on the server; - a new in-memory limiter throttled authentication attempts using buckets for both client source IP and normalized
LDAPUsername.
While this closed the response-content oracle and raised the cost of brute-force guessing, reviewing the limiter’s state machine and IP extraction immediately exposed flaws in how tokens were consumed and attributed.
Second round: success refunds, proxy attribution, and token accounting
Follow-up commits on April 16 (18b712d, 9e10f6d, f441108) fixed three defects in the initial limiter:
- Explicit reserve/commit/cancel/refund lifecycle: Successful authentication and LDAP infrastructure errors must refund their reserved token so legitimate high-frequency logins do not exhaust the failure allowance.
- Untrusted proxy headers: The limiter must default to the TCP socket peer (
RemoteAddr) and ignore client-suppliedX-Forwarded-For,X-Real-IP, andForwardedheaders unless the socket peer matches an explicit trusted-proxy allowlist. - Bounded token refunds: Cancellation and refund paths must clamp token counts to bucket capacity so concurrent cancellations cannot mint extra capacity.
Third round: why we deleted the username bucket
A second design review in June overturned our initial assumption that “throttling by source IP plus username is strictly stronger than throttling by source IP alone.”
Because the per-username bucket was shared globally across all client origins on a node, an attacker who knew a victim’s LDAPUsername could exhaust that user’s bucket from any IP address (or across rotating IPs). With only a handful of failed requests per minute, an attacker could keep a targeted user’s AssumeRoleWithLDAPIdentity calls permanently throttled before the legitimate user’s request ever reached an LDAP bind.
Commit 5e40665 (shipped in RELEASE.2026-06-18T00-00-00Z) therefore made five adjustments:
- deleted the shared per-username bucket so one origin cannot lock out a specific
LDAPUsernamefor other origins; - peeled trusted hops from
X-Forwarded-Forright-to-left and selected the first untrusted client address; - rejected catch-all trusted-proxy CIDRs (
0.0.0.0/0and::/0) that would allow any client to spoofX-Forwarded-For; - dropped RFC 7239
Forwardedfrom security-sensitive rate-limit bucketing; - accepted
X-Real-IPonly when the immediate peer is a configured trusted proxy that overwrites the header.
This proxy trust model was subsequently generalized across all S3 and Admin handlers as SN-2026-018 (MINIO_API_TRUSTED_PROXIES).
Rejected alternatives
| Option | Why it was rejected |
|---|---|
| Perform a dummy LDAP bind for unknown users | Amplifies load on the LDAP directory and adds a second bind path after the response-content oracle is already closed |
Bucket all IPv6 clients by /64 prefix |
Causes unrelated users behind the same enterprise or carrier /64 prefix to throttle one another |
Take the leftmost X-Forwarded-For value |
The leftmost entry is controlled by the client and trivially spoofed |
Fall back to the socket peer when X-Forwarded-For and X-Real-IP disagree |
An attacker can intentionally inject a conflicting header to force fallback and collapse all users behind a proxy into a single bucket |
Parse RFC 7239 Forwarded for rate limiting |
High parser complexity and ambiguity with no practical security advantage over X-Forwarded-For and X-Real-IP |
Verification and release
The unit test suite verifies the limiter’s reserve/commit/cancel/refund state machine, concurrent access, token refunds on successful binds and LDAP infrastructure errors, external response equivalence between unknown users and bad passwords, and source-IP extraction across direct RemoteAddr, spoofed headers, single- and multi-hop trusted proxies, and rejected catch-all CIDRs (0.0.0.0/0, ::/0).
Note that the end-to-end LDAP security test skips live directory binds when _MINIO_LDAP_TEST_SERVER is unset; full end-to-end verification requires pointing _MINIO_LDAP_TEST_SERVER at a live LDAP instance in addition to running the package unit tests.
Final trade-offs and operational boundaries
- Source-IP-only throttling: Removing the username bucket prevents targeted cross-IP account lockouts, at the cost of no longer enforcing a single-account failure cap across distributed botnet origins.
- Per-node memory state: The limiter is per-node and in-memory rather than a cluster-wide password defense; distributed spraying and LDAP bind timing differences still require directory-side policies or edge rate limiting.
- Proxy configuration: Deployments behind a reverse proxy must configure trusted proxy CIDRs (
MINIO_API_TRUSTED_PROXIES) so legitimate clients behind the proxy are bucketed by their client IP rather than the proxy’s socket address. - Uniform responses are the primary defense: Rate limiting slows down guessing from a single source, while the unified
400 InvalidParameterValueresponse is what closes the username enumeration oracle.
3.10 - CVE-2026-34204: Replication Metadata Injection
Status: Released
First containing release: RELEASE.2026-04-17T00-00-00Z
Fix commit: 56fa63b
GitHub issue: pgsty/silo#24
Before 56fa63b, ordinary PutObject and CopyObject requests could smuggle client-supplied X-Minio-Replication-* headers into internal X-Minio-Internal-* SSE and replication metadata. That mismatch between the stored metadata and the actual write path could leave the resulting object unreadable.
The fix stops accepting replication-only metadata in the default extraction path, restores those fields solely inside flows explicitly authorized for ReplicateObjectAction (s3:ReplicateObject), and sanitizes CopyObject request headers before any downstream header consumer runs.
Threat model
Exploiting this flaw required only ordinary object-write permission (s3:PutObject), not internode or replication credentials. Although the input arrived in client-controlled X-Minio-Replication-* HTTP headers, metadata extraction converted those headers into internal replication and SSE state without checking whether the caller was authorized to replicate objects.
When a subsequent GetObject or HeadObject request read the object, the server interpreted the stored object according to that forged internal state and failed to read or decrypt it. Because any authenticated writer could trigger this integrity and availability failure, every server accepting untrusted S3 writes was affected.
The underlying flaw was that client-controlled headers acquired privileged internal semantics before passing a replication authorization check.
Why we chose surgical stripping over rejecting the whole request
The most obvious fix would have been to reject any non-replication request carrying an X-Minio-Replication-* header. However, returning 400 or 403 whenever an extraneous header appears would turn previously ignored headers into hard client failures.
Instead, we designed a strip-by-default, restore-after-auth model:
- the default metadata extraction helper in
cmd/handler-utils.gonever accepts replication-only metadata; - ordinary
PutObjectandCopyObjectrequests strip those fields before building object metadata; - only code paths that have passed
ReplicateObjectActionauthorization explicitly restore replication-only metadata; - replica-status writes require the same
ReplicateObjectActionauthorization; - legitimate multipart and Snowball replication flows explicitly restore the SSE metadata they require after authorization.
This design closes the injection path while keeping wire behavior unchanged for ordinary clients that happen to pass unrecognized headers.
Why CopyObject had to sanitize headers before cleanMetadata
In CopyObject, request headers are not used solely to construct the final destination metadata map. Earlier stages in the handler inspect request headers for precondition evaluation and SSE-C source handling.
If untrusted X-Minio-Replication-* headers were stripped only inside cleanMetadata right before writing the destination object, those earlier consumers would already have acted on the forged headers. Moving header sanitization to the very beginning of CopyObject—before precondition and SSE-C helpers run—enforces a single invariant across the entire handler: untrusted replication headers never enter internal request semantics.
Implementation and verification
Commit 56fa63b updated cmd/handler-utils.go, object handlers, and multipart handlers, backed by regression tests across multiple layers:
- helper-level unit tests for trusted versus untrusted metadata extraction;
- handler-level tests for malicious
PutObjectandCopyObjectrequests carryingX-Minio-Replication-*headers; CopyObjectearly-sanitization tests verifying that precondition and SSE-C paths do not observe stripped headers;- end-to-end verification confirming that forged replication headers no longer produce unreadable objects, while authorized replication, multipart, and Snowball replication flows continue to succeed.
Trade-offs and subsequent hardening
- Default stripping: Internal replication headers sent by ordinary S3 clients are silently stripped rather than honored.
- Fail-safe default for new endpoints: Because replication-only metadata must be explicitly restored inside an authorized branch, forgetting to call the restore helper in a future replication path causes a functional test failure rather than silently reopening an untrusted injection path.
- Follow-up receiver-wide hardening:
- SN-2026-008 (
RELEASE.2026-09-03T13-18-01Z) and the CORS and replication trust design record generalized this pattern into a single post-authentication trust boundary across all internal replication headers (X-Minio-Source-Etag,X-Minio-Source-Mtime,X-Minio-Source-Replication-Request, replication SSE key headers, andX-Amz-Bucket-Replication-Status), covering SSE-C reads, source timestamps, multipart completion, object lock, deletes, and events. - Replica Metadata Normalization (
RELEASE.2026-09-16T00-00-00Z, PR #194) subsequently ensured that even authorized replicationCopyObjectrequests normalize transport-onlyaws-chunkedContent-Encodingvalues and strip GHSA-redacted metadata keys rather than re-injecting raw request headers into stored objects.
- SN-2026-008 (
3.11 - CVE-2026-39414: Oversized S3 Select Records and a SIMD Bypass
Status: Released; second-round fix completed in June
Initial fix release: RELEASE.2026-04-17T00-00-00Z
Complete fix release: RELEASE.2026-06-18T00-00-00Z
Fix commits: 3252d5b, 5dc705936 (fd69c89)
GitHub issue: pgsty/silo#25
Our initial April patch (3252d5b) reused the existing 1 MiB maxCharsPerRecord limit for both CSV and standard JSON Lines in S3 Select, stopping unbounded buffering while waiting for a record delimiter and returning the S3 OverMaxRecordSize error code to clients.
A follow-up call-chain audit in June uncovered a hardware-dependent gap: on x86_64 CPUs with AVX2/AVX-512 support where simdjson.SupportedCPU() returned true, JSON Lines took a separate simdjson fast path that bypassed the bounded json.PReader completely. The June fix (5dc705936 / fd69c89) routed JSON Lines through the bounded reader on every CPU architecture, while also correcting streaming error codes, parser error wrapping, and pre-error output queue flushing.
Threat model
An authenticated caller with s3:GetObject permission can issue an S3 Select (SelectObjectContent) query against an object containing an arbitrarily long single record without a newline delimiter. If the record reader buffers bytes without an upper bound until it encounters a delimiter, a single query can exhaust server memory and CPU.
Hardware-dependent dispatch makes this class of bug harder to catch: the same S3 Select query can execute different parser implementations depending on the host CPU’s instruction-set flags. Passing a test on a non-SIMD machine does not guarantee that production hosts with AVX2/AVX-512 are protected.
First round: reusing the existing 1 MiB invariant
Rather than introducing a new configuration knob, the April patch (3252d5b) applied the existing maxCharsPerRecord = 1 MiB (1048576 bytes) constant across CSV and standard line-delimited JSON:
- the CSV splitter and line-delimited JSON reader rejected records exceeding 1 MiB before buffering or parsing them further;
- the reader preserved the earliest splitter error instead of letting a subsequent partial decode overwrite it;
- the error propagated to the client as
OverMaxRecordSizerather than collapsing into a genericInternalErrorso clients and monitoring can distinguish an enforced record-size limit from a server fault.
Second round: the hardware-dependent SIMD bypass
Tracing the full S3 Select JSON dispatch chain in June revealed a second path for JSON Lines (DOCUMENT type LINES):
Whenever simdjson.SupportedCPU() returned true, JSON Lines bypassed the bounded json.PReader and called simdj.NewReader. Inside simdjson.ParseNDStream, the third-party parser continued reading across chunk boundaries until it found a newline. Wrapping the underlying io.Reader in a byte-counting adapter could not solve the problem cleanly: because ParseNDStream reads ahead in chunks, a generic wrapper cannot simultaneously preserve already-completed records in the buffer and enforce an exact 1 MiB cap on the next record.
Rather than layering a brittle byte-counting wrapper around simdjson.ParseNDStream, commits 5dc705936 and fd69c89 disabled the SIMD NDJSON path and routed JSON Lines through the bounded json.PReader on all CPUs. If a SIMD fast path is reintroduced in the future, the SIMD parser itself must enforce the 1 MiB per-record limit and pass the same CPU-independent regression suite.
Stream semantics corrected in the same round
Reviewing the S3 Select streaming pipeline during the June fix also resolved four adjacent protocol issues:
- Interface-based error matching: Used
errors.Asto pass through any error implementingSelectError, rather than matching only a single concrete struct type. - Typed JSON parser errors: Wrapped syntax failures in the JSON worker as
JSONParsingErrorinstead of returning untyped errors. - Pre-error batch flushing: Flushed completed records still waiting below the batch threshold in the output queue before emitting a terminal error event, matching S3 Select streaming semantics.
- Input-order error priority: Preserved the first error encountered in input order so a later oversized record cannot overwrite an earlier parse failure on a preceding line.
Deliberately kept out of scope
To keep the security patch focused and avoid speculative behavior changes, three adjacent topics were left out of scope:
- the historical interaction between CSV
AllowQuotedRecordDelimiterand the outer physical-newline splitter; - whether
\rinCRLFcounts toward the 1 MiB record length limit; - re-implementing a bounded SIMD NDJSON parser before having a design that enforces record boundaries inside
simdjson.
Verification and release
The regression suite in internal/s3select tests oversized CSV and JSON Lines records, OverMaxRecordSize and JSONParsingError propagation, pre-error record flushing, and parser behavior across both SIMD and non-SIMD code paths.
The initial fix shipped in RELEASE.2026-04-17T00-00-00Z (3252d5b), and the complete SIMD-independent fix shipped in RELEASE.2026-06-18T00-00-00Z (5dc705936 / fd69c89).
Final trade-offs
- Throughput vs. predictable bounds: Routing JSON Lines through
json.PReadergives up SIMD NDJSON acceleration on AVX2/AVX-512 hosts in exchange for a strict 1 MiB per-record memory bound on every CPU. - 1 MiB record ceiling: S3 Select queries against CSV or JSON Lines objects with individual records larger than 1 MiB are rejected with
OverMaxRecordSize; callers must split records exceeding 1 MiB. - Quoted multiline CSV: Quoted multiline CSV delimiter semantics remain tracked separately from this CVE.
3.12 - CVE-2026-40344: Snowball Auto-Extract Authentication Bypass
Status: Released
First containing release: RELEASE.2026-04-17T00-00-00Z
Fix commit: efb6e5b00
GitHub advisory: GHSA-9c4q-hq6p-c237
Snowball’s PutObjectExtractHandler (X-Amz-Meta-Snowball-Auto-Extract: true) previously omitted the authTypeStreamingUnsignedTrailer authentication case. A tar archive sent with X-Amz-Content-Sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER and a forged signature could enter untar() before authentication completed, allowing a single unauthenticated request to fan out into multiple object writes.
Commit efb6e5b00 initializes newUnsignedV4ChunkedReader(), validates the decoded content length, and completes SigV4 verification before any tar byte reaches the extractor.
Why the commit subject used a temporary identifier
When the fix was committed on 2026-04-16, the official CVE number had not yet been published by GitHub Security Advisories, so the Git commit subject used the temporary placeholder fake CVE-2026-40028. Once GHSA-9c4q-hq6p-c237 was published, the official identifier was assigned as CVE-2026-40344. We kept the immutable Git commit SHA efb6e5b00 intact and mapped it to CVE-2026-40344 in the advisory ledger.
From one missing authentication branch to bulk object writes
The affected entry point is PutObjectExtractHandler, which handles Snowball batch archive uploads (X-Amz-Meta-Snowball-Auto-Extract: true). When the client selected STREAMING-UNSIGNED-PAYLOAD-TRAILER (authTypeStreamingUnsignedTrailer), PutObjectExtractHandler lacked the switch branch that standard PutObjectHandler used to authenticate the request and wrap the chunked stream.
The impact exceeded a single unauthorized object write: as soon as the unauthenticated tar stream entered untar(), the server extracted each archive entry as a separate object write. A single missing authentication branch was therefore amplified into an arbitrary multi-object write primitive.
Final invariant: the extractor sees zero bytes on authentication failure
Design review established a strict requirement for the fix:
If authentication fails,
untar()must have read zero bytes.
That invariant rules out any design that begins streaming or extracting archive entries and attempts to roll back created objects if a later check fails. Because Snowball extraction writes objects across multiple internal paths, proving a clean rollback is far more error-prone than proving that unauthenticated bytes never cross into untar(). Authentication had to close before the extractor was invoked.
Implementation
Commit efb6e5b00 aligns PutObjectExtractHandler with standard PutObjectHandler for authTypeStreamingUnsignedTrailer:
- adds the
authTypeStreamingUnsignedTrailercase to the handler’s authentication switch; - parses and validates
X-Amz-Decoded-Content-Length; - wraps the request body with
newUnsignedV4ChunkedReader(), which enforces SigV4 signature verification before yielding payload bytes (sharing the reader boundary hardened in CVE-2026-41145); - preserves valid signed Snowball auto-extract uploads and trailing CRC32 checksum verification.
Verification
Regression tests in cmd verify the full boundary:
- rejection of a Snowball unsigned-trailer request carrying a forged SigV4 signature with zero objects created;
- rejection of anonymous Snowball auto-extract writes to non-public buckets;
- successful extraction of valid signed Snowball unsigned-trailer archives with trailing CRC32 checksums.
Compatibility and operational boundaries
- Rejected unverified combinations: Clients that sent unsigned-trailer Snowball requests with invalid or unverified query-string signatures will now fail with
AccessDeniedbefore extraction begins. - Shared reader boundary: Because Snowball auto-extract and standard
PutObjectnow sharenewUnsignedV4ChunkedReader()for unsigned-trailer streams, any future change to chunked trailer parsing must run regression tests for both paths. - Orthogonal archive surfaces: Closing pre-extraction authentication does not alter how
untar()processes archive entry paths, sizes, or object counts once an authenticated caller is inside the extractor.
3.13 - CVE-2026-41145: Unsigned-Trailer Query Authentication Bypass
Status: Released
First containing release: RELEASE.2026-04-17T00-00-00Z
Fix commit: f444b6f37
GitHub advisory: GHSA-hv4r-mvr4-25vw
Before f444b6f37, query-string SigV4 credentials could enter a STREAMING-UNSIGNED-PAYLOAD-TRAILER upload while the handlers verified the request signature only when an HTTP Authorization header was present. An attacker who knew a valid access-key identifier could therefore complete an object or multipart write using a forged query-string signature.
The fix moves presigned-credential rejection and SigV4 verification directly into newUnsignedV4ChunkedReader() (cmd/streaming-v4-unsigned.go), ensuring every handler that consumes an unsigned-trailer chunked stream shares a single authentication boundary.
Commit identifier note
When the patch was written on 2026-04-16, the official CVE had not yet been assigned, so the commit subject used the temporary placeholder fake CVE-2026-40027. Upon publication of GHSA-hv4r-mvr4-25vw, the vulnerability was assigned CVE-2026-41145. The historical commit SHA f444b6f37 remains unchanged.
Root cause: coupling signature verification to credential transport
The vulnerable entry points included PutObject and PutObjectPart (UploadPart). When a client sent X-Amz-Content-Sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER and supplied its SigV4 credentials (X-Amz-Credential, X-Amz-Signature, etc.) in the URL query string rather than the Authorization header, two assumptions collided:
- The handler checked whether the
Authorizationheader was non-empty to decide whether to runreqSignatureV4Verifybefore reading the stream, assuming an empty header meant an anonymous request. - Identity resolution and
newUnsignedV4ChunkedReader()still consumed the request normally while attributing the write to the access key named inX-Amz-Credential.
As a result, query-authenticated STREAMING-UNSIGNED-PAYLOAD-TRAILER requests bypassed signature verification altogether. An attacker only needed to know a valid access-key ID; the signature value in the query string was never checked.
Why we placed the check inside the shared reader instead of individual handlers
| Option | Risk | Decision |
|---|---|---|
Add header/query checks separately to PutObject and PutObjectPart |
Closes the two known handlers, but any sibling or future caller of newUnsignedV4ChunkedReader() can omit the check again (as CVE-2026-40344 demonstrated) |
Rejected |
Invent a custom presigned STREAMING-UNSIGNED-PAYLOAD-TRAILER verification flow |
Expands the SigV4 streaming state machine without an existing S3 compatibility contract | Rejected |
Reject presigned credentials and verify SigV4 inside newUnsignedV4ChunkedReader() |
Forces every caller consuming the stream through one authentication gate | Adopted |
Note that genuinely anonymous STREAMING-UNSIGNED-PAYLOAD-TRAILER uploads were not disabled. When a request carries neither header nor query credentials and the bucket policy explicitly grants anonymous s3:PutObject, the request still follows the anonymous authorization path. What is forbidden is carrying unverified credentials into the stream.
Implementation and verification
Commit f444b6f37 enforces presigned rejection and SigV4 verification at the entry of newUnsignedV4ChunkedReader() in cmd/streaming-v4-unsigned.go and removes the Authorization-header presence gates from PutObject and multipart upload handlers.
Regression tests in cmd verify:
- rejection of forged query-string
PutObjectandPutObjectPartrequests inSTREAMING-UNSIGNED-PAYLOAD-TRAILERmode; - rejection of requests mixing header and query-string SigV4 credentials;
- continued operation of valid header-authenticated unsigned-trailer uploads and anonymous uploads permitted by bucket policy.
Compatibility and operational boundaries
- Presigned unsigned-trailer is unsupported: Combining query-string SigV4 credentials with
STREAMING-UNSIGNED-PAYLOAD-TRAILERis rejected unconditionally. Clients using unsigned-trailer chunked uploads must pass SigV4 credentials via theAuthorizationheader. - Reader-level enforcement: Placing the authentication gate inside
newUnsignedV4ChunkedReader()ensures thatPutObject,PutObjectPart, andPutObjectExtract(CVE-2026-40344) cannot construct an unsigned-trailer reader without completing SigV4 verification.
3.14 - CVE-2026-42600: ReadMultiple Storage-REST Path Traversal
Status: Released
First containing release: RELEASE.2026-06-18T00-00-00Z
Fix commit: 73ac524
GitHub advisory: GHSA-xh8f-g2qw-gcm7
Affected scope: Distributed erasure deployments only; cluster-root / internode JWT required
The internode storage-REST route /rmpl (ReadMultiple) decoded a msgpack request body carrying Bucket, Prefix, and Files, and joined those fields directly into filesystem paths without a drive-root containment check.
Our initial fix retained the endpoint and added preflight path validation. A subsequent call-chain audit then revealed that ReadMultiple had had zero production callers since September 2024. We therefore discarded the in-place validation patch and deleted the route, handler, client wrapper, StorageAPI method, and generated msgpack types (73ac524)—eliminating roughly a thousand lines of unused privileged RPC surface.
Threat model
The /rmpl storage-REST route was registered only in distributed erasure mode; single-node deployments do not register storage-REST routes and were unaffected. Exploiting the endpoint required an internode JWT derived from the cluster root credentials, control of a peer node, or interception of unencrypted internode traffic.
Because the attacker-controlled Bucket, Prefix, and Files fields arrived inside the msgpack request body rather than in the URL or query string (r.Form), upper HTTP path-validation middleware never saw them. xlStorage.ReadMultiple joined the raw strings directly onto the drive path and read the resulting files, allowing a caller with internode credentials to escape the drive root and read any file accessible to the SILO server process.
First design: retain the API and validate every path
Our first patch hardened xlStorage.ReadMultiple in place:
- rejected absolute paths,
.and..segments, backslashes, Windows drive prefixes, and NUL bytes; - verified drive-root containment across
drive -> volume -> prefix -> file; - preserved historical edge cases for an empty
Bucketand.minio.sys/multipart; - returned an error before opening files or streaming responses.
That first draft closed the obvious traversal path, but code review immediately caught a flaw in how the loop interacted with MaxResults.
How MaxResults exposed the flaw of validate-as-you-go
The initial draft validated each entry of Files inside the file-reading loop right before opening it. When a request passed Files = [good, "../../etc/passwd"] together with MaxResults = 1, the loop read the first file, hit MaxResults == 1, and returned early without ever inspecting the second entry.
Although that specific call did not read the second file, it violated a core input-validation rule: a malformed msgpack request containing a path-traversal payload must be rejected before any partial execution or streaming begins. We moved path and length validation into a complete preflight pass over the entire request before touching the filesystem.
Final decision: why we deleted ReadMultiple instead
Before merging the preflight-validation patch, we audited every caller of ReadMultiple across the repository and git history:
- upstream had removed the last production caller in September 2024 when multipart reads migrated to
ReadParts; - the current SILO tree had zero in-tree production callers;
- upstream’s own remediation for GHSA-xh8f-g2qw-gcm7 also deleted
ReadMultiple.
| Option | Short-term diff | Long-term attack and maintenance surface | Decision |
|---|---|---|---|
| Validate paths in place | Smaller diff; preserves /rmpl |
Permanently maintains an unused, privileged multi-file read API | Abandoned |
Delete ReadMultiple |
Larger diff across interface and msgpack codegen | Removes the unused endpoint completely while preserving storageRESTVersion wire compatibility |
Adopted |
Commit 73ac524 removes the /rmpl route, storage-REST server handler, client wrapper, StorageAPI and xlStorage methods, metric definitions, and generated msgpack request/response types.
Verification and release
During the in-place validation phase, we ran unit tests covering xlStorage, the storage-REST client, msgpack encoding, path traversal vectors, and the MaxResults preflight edge case. After switching to full deletion in 73ac524 (shipped in RELEASE.2026-06-18T00-00-00Z), verification confirmed that all routes, interfaces, and generated msgpack surfaces compiled and passed go test ./cmd cleanly.
Compatibility and follow-up audit (SN-2026-002)
- External S3 API: Unchanged.
- Rolling upgrades: Keep all nodes in a distributed cluster on the same SILO release during upgrades.
- Follow-up audit across all internode endpoints: All remaining internode endpoints carrying filesystem paths inside msgpack bodies or Grid RPC frames were audited and hardened in SN-2026-002: Internode Path and Allocation Containment (shipped in
RELEASE.2026-08-04T00-00-00Z), which enforced volume and file-path containment across storage-REST, storage Grid RPC, and peer-S3 Grid RPC endpoints.
3.15 - SN-2026-002: Internode Path Containment Audit — Paying Off What CVE-2026-42600 Left Owing
Status: Released in Server 20260804 (release notes)
Identifier: SN-2026-002 (Security Advisories)
Fix commits: ca7baa670, b6f70ab08, 80e8eaa42, 1af351a70, a36fd8fff
Affected scope: Distributed erasure deployments only; cluster-root / internode JWT required
Prerequisite reading: CVE-2026-42600 · ReadMultiple
Following the removal of /rmpl (ReadMultiple) in CVE-2026-42600, SN-2026-002 audits every remaining internode endpoint carrying filesystem paths inside msgpack request bodies or Grid RPC frames. This advisory records that audit across all four internode protocol surfaces, the twelve inherited defects it uncovered, and the four regressions we introduced and caught while building the fix.
Conclusions first
- Twelve defects across three unprotected protocol surfaces, all inherited unchanged from upstream (verified by per-function checksum comparison across the affected storage files).
- Same architectural root cause as CVE-2026-42600:
ReadMultiplewas one endpoint on one surface; three protocol surfaces remained exposed to the same gap. - Four regressions introduced and caught during development: every regression occurred in a rule we invented from scratch (
isVolumeRootAliasandguardErasureParams), while reusing the object layer’s existinghasBadPathComponentrule produced zero regressions.
Why counting endpoints was the wrong frame
Earlier spot checks kept counting individual handlers—arriving at 19, then 21, then 22—while missing an entire protocol surface each time. Grouping internode calls by how their arguments reach xlStorage reveals four distinct surfaces:
| Protocol surface | Entries | Covered by global HTTP query middleware |
|---|---|---|
| Storage-REST HTTP query arguments | 9 | Yes — previously misclassified as unprotected |
| Storage-REST HTTP msgpack body | 4 | No; r.Form is never populated from a msgpack body |
Storage Grid RPC (/minio/grid/v1) |
18 | No; after the initial WebSocket upgrade, frames never re-enter the HTTP middleware chain |
| Peer-S3 Grid RPC | 5 | No; and it bypasses getStorage() to call xlStorage drives directly |
The first row corrects an earlier misconception: audits that grepped exclusively inside handler functions missed that the 9 query-parameter endpoints are already guarded by the global HTTP middleware (url.ParseQuery(RawQuery) in r.Form). Conversely, the fourth row (peer-S3 Grid RPC) bypasses storageRESTServer.getStorage() completely, meaning no amount of hardening limited to storage-rest-server.go could protect DeleteBucket or peer bucket metadata calls.
Three stacked layers in the storage path
Three independent design choices combined to create the vulnerability class:
- Path validation ran only on HTTP query strings: The HTTP middleware inspects
r.Form, which is populated fromurl.ParseQuery(RawQuery)and never from a request body (introduced in 2017). - Grid RPC bypassed the HTTP middleware chain:
/minio/grid/v1performs a single HTTP upgrade; all subsequent msgpack RPC frames dispatch directly to registered handlers without re-entering HTTP middleware (introduced in 2023). xlStorageassumed callers had already validated paths:getVolDirrejected a volume only when it equaled exact"",., or.., andpathJoinranpath.Cleanagainst the joined path (settled in 2018).
Each layer relied on another layer to validate paths, and none of them inspected the msgpack or Grid RPC channels in between.
One timeline interaction escalated a DoS bug from a failed request into a full process crash: ShardFileSize has contained an unguarded division by DataBlocks since 2020, but when CheckParts was wrapped inside xioutil.WithDeadline in October 2024 to bound slow disk operations, that division moved onto a bare goroutine where the HTTP server’s recover() middleware could no longer catch a divide-by-zero panic.
Vectors confirmed by execution
Every vector below was reproduced against a live xlStorage instance through the storage-REST and Grid RPC clients using planted sentinel files:
| Vector | Surface | Observed effect |
|---|---|---|
WriteAll("vol", "../../x") |
Storage Grid RPC | Arbitrary file write outside the drive root |
RenameFile(".minio.sys", "", "bucket", "x") |
Storage Grid RPC | Relocates the entire .minio.sys system volume (IAM, cluster config) into a readable S3 bucket with no .. sequence |
DeleteBucket("../victim", force) |
Peer-S3 Grid RPC | Recursive deletion of a directory tree outside the drive root |
DeleteBulk("vol", "") |
Storage-REST body | Moves an entire volume into the drive trash directory |
ReadAll(volume: "../") |
Any surface | Trailing slash bypasses getVolDir’s exact ".." check |
CheckParts with zero Erasure (DataBlocks = 0) |
Storage Grid RPC | Divide-by-zero panic in WithDeadline terminates the server process |
AppendFile declaring Content-Length: 64 GiB |
Storage-REST HTTP | Allocates 68,719,574,840 bytes for an empty request body |
DeleteVersions declaring 100,000,000 entries |
Storage-REST body | A 10-byte msgpack header allocates a 10.4 GB slice |
Part metadata with Size = -2 |
Storage Grid / AddVersion |
Truncated shard is reported healthy; healing silently skips repair |
Two vectors deserve specific attention:
RenameFilewith an empty source path (""): An emptysrcPathresolves to the volume directory itself. When pointed at.minio.sys,RenameFile(".minio.sys", "", "bucket", "x")moves the entire system metadata volume into a normal bucket, where an ordinaryGetObjectcan read cluster IAM and configuration files—without using..anywhere in the payload.- Negative part size (
Size = -2): A negative part size causesShardFileSizeto floor both terms to0. BecausecheckPartchecks onlyst.Size() < expectedSize, any existing shard file (even a 0-byte truncated file) satisfies0 < 0 == falseand is reported intact. BecauseFileInfo.IsValid()did not reject negative part sizes, poisonedxl.metametadata caused background healing to skip corrupted shards silently.
The fix: two chokepoints across 36 endpoints
The security invariant has two parts: every file path in an internode payload must resolve strictly inside its target volume, and every volume must resolve strictly inside the drive root.
Because pathJoin folds leading slashes and backslashes under volumeDir via path.Clean, only .. segments can escape a volume directory, and only volume-root aliases ("", "/", ".", and Windows trailing-space/period components) can target the volume directory itself. Commit ca7baa670 enforces those two rules at two chokepoints:
| Chokepoint | Location | Coverage |
|---|---|---|
| Volume axis | getVolDir + explicit guard in NSScanner |
Every xlStorage caller, including peer-S3 Grid RPC |
| Path axis | getStorage() decorator (guardPaths, isVolumeRootAlias) |
All 31 remote storage-REST and Storage Grid RPC methods plus nested struct slices |
Two implementation details are critical to preserve during future refactors:
- Checks must run on the raw argument before
pathJoin:pathJoincallspath.Cleanon the concatenated absolute path (/drive+volume+filePath), which silently collapses/drive/../../etcinto/etc. A check placed afterpathJoinsees a clean path with no..and allows the escape. NSScannerrequires its own volume guard:NSScanneris the onlyxlStoragemethod that constructs a volume path without callinggetVolDir, making its explicitcheckPathguard load-bearing.
We evaluated placing containment checks inside all 33 pathJoin(volumeDir, ...) call sites in xl-storage.go, which would require deciding at each of the 33 sinks whether the volume root itself is a valid target. Enforcing the invariant at getVolDir and the getStorage() decorator—backed by reflection guardrail tests on StorageAPI—covers all remote entry points without touching local erasure hot paths.
Construction log: four regressions we caused ourselves
During development and review, our own drafts introduced four regressions before shipping:
- Rejecting whitespace object keys (
" "and" "): Our first draft of the path guard treated whitespace as an invalid path character, rejecting" "and" "—which are valid S3 object keys. BecausePutObjectcommits writes across drives viaRenameData, uploading an object named" "failed on every remote drive and broke write quorum. - Rejecting backslash-only keys (
"\\") on Unix: The same draft rejected"\\". On Unix,path.Cleantreats\as an ordinary filename character rather than a directory separator. Rejecting"\\"in the internode decorator caused distributed clusters to reject a valid S3 key that single-node deployments accepted. - Windows trailing spaces and periods (
" "and"..."): The Win32 path normalizer strips trailing spaces and periods from path components, meaning a component composed solely of spaces or periods disappears and resolves to its parent directory. On Windows, both" "and"..."alias the volume root—and"..."was sitting inside our own positive test list of “valid object names” until review caught it. - Negative part sizes (
Size < 0): Our initialguardErasureParamscheck rejected zeroDataBlocksonly whenpart.Size > 0, equating “non-zero” with “positive” and missing that negative sizes floorShardFileSizeto zero even whenDataBlocksis valid.
Two false-green tests and a fuzzer blind spot
Writing the reproduction and regression tests exposed three testing pitfalls:
storageRESTClientsilently overrodeContent-Length: When testingAppendFilewith a forgedContent-Length: 64 GiBover an empty*bytes.Reader, the client library inspected the*bytes.Readertype and overwroteContent-Lengthwith0.- Go’s
http.Clientrefused mismatched body lengths: Switching to an opaqueio.Readercaused Go’s standard HTTP client transport to abort the request locally before sending it because the body closed before 64 GiB. Reproducing the server-side 64 GiB allocation required driving the handler directly viahttptest(or a raw socket). - An early return in the fuzzer hid the backslash bug: Our first fuzz property test skipped strings composed solely of
/and\via an earlyreturn. That early return blinded the fuzzer to"\\"across millions of executions until we rewrote the property test to assert strict inclusion againstIsValidObjectName.
Reused semantics vs. invented rules
| Guard component | Source of semantics | Regressions introduced during development |
|---|---|---|
guardPaths |
Reused existing hasBadPathComponent (IsValidObjectPrefix) |
0 |
getVolDir guard |
Reused existing hasBadPathComponent |
0 |
isVolumeRootAlias |
Newly written predicate | 3 |
guardErasureParams |
Newly written predicate | 1 |
hasBadPathComponent had already been exercised by production S3 traffic via IsValidObjectPrefix for years and structurally cannot reject any object key creatable through the S3 API. Every regression came from newly invented predicates where edge cases (" ", "\\", "...", Size = -2) diverged from S3 or OS semantics.
Guardrail tests
To ensure future changes to StorageAPI cannot silently reintroduce unguarded paths (avoiding the point-fix regression pattern previously seen with CVE-2026-39414), the test suite enforces three guardrails:
- Falsifiability (mutation verification): Temporarily removing the path and volume guards causes 191 subtests to fail; removing the allocation guards reproduces the 64 GiB and 10.4 GB allocations.
- Legal-traffic fuzz equivalence: Fuzz tests verify that every object key accepted by
IsValidObjectNamepassesguardPaths(1.96M executions) and every valid bucket name passesgetVolDir(810K executions). StorageAPIreflection guardrail: A reflection test inspects every method onStorageAPIand fails CI by method name if a new method accepting path or volume arguments is added without an explicit entry in the decorator guard table.
Independent review rounds
Five successive rounds of independent code review each uncovered one real flaw before release: whitespace keys -> backslash keys -> the AppendFile 64 GiB allocation -> Windows trailing spaces/periods -> negative part sizes. Separating implementation from review caught all four S3 compatibility and integrity regressions before release.
Follow-up fixes in Server 20260804
Five companion commits shipped alongside ca7baa670 in Server 20260804:
ReadFileHandlerread-length cap (b6f70ab08): Commitb6f70ab08rejects declared read lengths above 5 GiB (the maximum size of a single S3 part on this legacy whole-file bitrot path), preventing arbitrary caller-declared multi-tens-of-GiB allocations.- Negative part size rejection (
80e8eaa42): Commit80e8eaa42rejects negative part sizes at theAddVersionwrite funnel and again inCheckParts,VerifyFile, and internodeguardErasureParams, protecting both new writes and already-stored metadata. - Non-positive erasure block size rejection (
80e8eaa42): Commit80e8eaa42validatesblockSize > 0inNewErasureand guards rebalance division boundaries. ReadPartsstorage error propagation (1af351a70): Commit1af351a70fixes a pre-existing bug whereReadPartsHandlercalleddone(nil)before the storage call completed, which had caused storage errors (errFileNotFoundas well as new guard rejections) to be written as plain text into a msgpack keep-alive stream (msgp: attempted to decode type "int" with method for "map").- Panic recovery in
xioutil.WithDeadline(a36fd8fff): Commita36fd8fffrecovers panics insideWithDeadline, logspanic in deadline-bounded work: ...with a stack trace tostderr(rate-limited to 10 stack dumps per process lifetime), and fails only the in-flight request. Operators who monitor node health should add a log alert forpanic in deadline-bounded work:.
Known limitations
Two platform boundaries remain:
- Windows test environment: SILO publishes Windows binaries while automated CI runs on Linux; the Windows volume-root alias rule (
" "and"...") is derived from documented Win32 path normalization behavior. - Lexical symlink handling: Path containment is lexical (
hasBadPathComponentandisVolumeRootAlias), matching upstreamxlStoragesemantics, and does not resolve OS symlinks placed inside a drive root.
3.16 - SN-2026-003: s3:versionid Condition Evaluation and Request Condition Hardening
SN-2026-003 and SN-2026-005 cover a set of S3 IAM policy evaluation fixes shipped in Server 20260804 (80c2c9d03, 2f55347f7, eee05a17c, and silo-pkg v3.12.0 commit 2b087a1; pgsty/silo#40, pgsty/silo#53):
SN-2026-003(80c2c9d03):getConditionValuespopulatedargs.ConditionValues["versionid"] = []string{""}even when a request carried noversionId, causing IAMNullconditions ons3:versionidto evaluate as if the key were always present; fixing that alone would have opened aDeleteObjectsXML<VersionId>bypass without per-entry rebinding inauthorizeRequestWithTags.- Condition-source isolation (
2f55347f7): Separated query parameters, HTTP headers, internal server keys, ands3:ExistingObjectTag/<key>vs.s3:RequestObjectTag/<key>so untrusted query strings or headers cannot spoof internal condition keys. SN-2026-005(2b087a1/eee05a17c): AddedParseConfigStrictto reject degenerate bare resource prefixes ("arn:aws:s3:::"and"*arn:aws:s3:::") when writing IAM and bucket policies.
Conclusions first
- Unconditional empty slice inverted
Nullchecks: Insilo-pkg,nullFunc.evaluatetests whether a condition key is absent by checkinglen(rvalues) == 0. BecausegetConditionValuesunconditionally assignedargs.ConditionValues["versionid"] = []string{vid}(producing[]string{""}whenvid == ""),len(rvalues)was always1—makingNull: {"s3:versionid": "true"}alwaysfalseandNull: {"s3:versionid": "false"}alwaystrue. - Fixing
getConditionValuesalone breaksDeleteObjects: In multi-objectDeleteObjects(POST /?delete), the target<VersionId>is inside each XML<Object>element, not in the URL query string. IfgetConditionValuessimply omits"versionid"when the URL query has noversionId, a policy denying versioned deletes (Deny s3:DeleteObjectVersionorCondition: {"Null": {"s3:versionid": "false"}}) would fail to matchDeleteObjectsXML entries unlessauthorizeRequestWithTagsrebindsConditionValues["versionid"]fromReqInfo.VersionID. - Whitespace trimming closes a
Denybypass: Version-reading handlers calledstrings.TrimSpace(r.URL.Query().Get("versionId")), whereasgetConditionValuesdid not trim whitespace. A request with?versionId=V%20bypassedDeny StringEquals {"s3:versionid": "V"}while still deleting or reading versionV.
Why Null on s3:versionid Was Inverted
In silo-pkg/bucket/policy/condition/nullfunc.go, the IAM Null condition operator checks whether a condition key is missing from the request context:
However, in cmd/signature-v4-utils.go, getConditionValues populated "versionid" at the end of the function without checking whether vid was non-empty:
When a caller sent an unversioned request (DELETE /bucket/key with no versionId query parameter), vid was "", and args["versionid"] became []string{""} (len == 1). Consequently:
- A policy allowing only unversioned operations via
"Null": {"s3:versionid": "true"}evaluatedlen(rvalues) == 0asfalseand denied valid unversioned requests. - A policy restricting version-specific operations via
"Null": {"s3:versionid": "false"}evaluated!f.v && len(rvalues) > 0astrueand matched even when noversionIdwas supplied.
The Trap: Why Omitting Empty vid Alone Opens a DeleteObjects Bypass
The obvious one-line fix—only writing args["versionid"] = []string{vid} when vid != ""—turned out to hide a dangerous authorization bypass in DeleteMultipleObjectsHandler (POST /{bucket}?delete).
Unlike single-object DELETE /{bucket}/{object}?versionId=V, a multi-object DeleteObjects request carries no versionId in the URL query string. Instead, the request body contains an XML list of <Object><Key>...</Key><VersionId>V</VersionId></Object> entries. Inside DeleteMultipleObjectsHandler, the server loops over each XML entry, sets r.WithContext (ReqInfo.VersionID = obj.VersionID), and calls authorizeRequestWithTags for DeleteObjectAction.
Consider what happens if an administrator configures a policy that allows normal delete-marker creation (s3:DeleteObject when s3:versionid is null) or denies permanent version deletion via:
Because authorizeRequestWithTags calls getConditionValues(r, ...)—which inspects r.Form.Get("versionId") on the HTTP request URL rather than ReqInfo.VersionID—omitting empty vid in getConditionValues without propagating ReqInfo.VersionID means args["versionid"] is absent during every per-object check in DeleteObjects. Worse still, an attacker could pass a decoy query parameter on the batch URL (POST /mybucket?delete&versionId=decoy) while putting the protected <VersionId>target-v1</VersionId> inside the XML body!
The Two-Part Fix (80c2c9d03)
Commit 80c2c9d03 fixes both halves together:
- Omit
"versionid"whenvid == ""ingetConditionValues: - Rebind
"versionid"fromReqInfo.VersionIDforDeleteObjectActioninauthorizeRequestWithTags(cmd/auth-handler.go):
Why rebind conditionValues["versionid"] only when action == iampolicy.DeleteObjectAction rather than for every action? Because in CopyObjectHandler, ReqInfo.VersionID holds the destination query parameter (if any), whereas the GetObjectAction authorization check on the copy source must evaluate s3:versionid from x-amz-copy-source (?versionId=... on the source URL). Restricting the ReqInfo.VersionID override to DeleteObjectAction ensures both single-object DeleteObjectHandler and per-entry DeleteMultipleObjectsHandler evaluate the exact version being deleted—and explicitly delete(conditionValues, "versionid") when reqInfo.VersionID is empty so that a URL query decoy (POST /bucket?delete&versionId=query-level-decoy) cannot spoof an unversioned XML entry.
Closing the Whitespace Normalization Gap
While auditing every caller that reads versionId, we noticed that object handlers (GetObjectHandler, HeadObjectHandler, DeleteObjectHandler, PutObjectRetentionHandler, etc.) extract the query parameter with:
In contrast, getConditionValues read r.Form.Get("versionId") (and u.Query().Get("versionId") from x-amz-copy-source) without strings.TrimSpace.
If a policy denied access to a specific version via "StringEquals": {"s3:versionid": "v1"}, a caller requesting ?versionId=v1%20 (with a trailing space) had "versionid": ["v1 "] during policy evaluation—evading the StringEquals deny rule—after which the handler trimmed "v1 " to "v1" and operated on the protected version. Applying strings.TrimSpace to both query versionId and x-amz-copy-source versionId inside getConditionValues closes that mismatch.
Call-Chain Impact and Condition-Source Isolation (2f55347f7)
| Operation | Source of s3:versionid |
Pre-20260804 Behavior |
Server 20260804 Behavior |
|---|---|---|---|
GetObject / DeleteObject (no versionId) |
Absent | ["versionid"] = [""] (Null:true fails) |
Key omitted (Null:true matches) |
GetObject / DeleteObject (?versionId=v1%20) |
URL query versionId |
Policy sees "v1 ", handler uses "v1" |
Both policy and handler see "v1" |
CopyObject (x-amz-copy-source: /b/k?versionId=v1) |
x-amz-copy-source query |
Untrimmed vid |
Trimmed vid |
DeleteObjects (POST /?delete with <VersionId>) |
Per-entry ReqInfo.VersionID |
Ignored XML <VersionId>; read URL query |
Rebound per XML entry; URL decoy ignored |
In companion commit 2f55347f7 (pgsty/silo#53), we also hardened getConditionValues across all other request condition keys:
- Reserved internal keys: Client query parameters and HTTP headers can no longer overwrite server-derived condition keys (
username,userid,principaltype,authtype,signatureversion,tlsVersion,cipherSuite). s3:signatureAge: Computed directly from the validated SigV4 timestamp rather than trusting client-supplied values.- Tag condition separation:
s3:ExistingObjectTag/<key>is populated exclusively from stored object tags, ands3:RequestObjectTag/<key>exclusively fromx-amz-taggingon the incoming request.
SN-2026-005: Bare ARN Prefix Rejection
During the same IAM condition audit, we investigated SN-2026-005 (silo-pkg v3.12.0 commit 2b087a1, wired into Server 20260804 in eee05a17c): what happens when a policy contains a degenerate resource string with an empty bucket and object suffix, such as "arn:aws:s3:::" or "*arn:aws:s3:::"?
Because the legacy parser stripped the arn:aws:s3::: prefix and accepted an empty resource pattern "", that pattern could never match any real bucket or object. In an Allow statement, it was merely a dead rule—but in a Deny statement on Resource or an Allow statement using NotResource, a degenerate arn:aws:s3::: pattern failed open!
To prevent administrators from saving policies that silently fail open without risking startup failures on existing clusters:
silo-pkg v3.12.0addspolicy.ParseConfigStrict, which rejects degenerate bare ARN prefixes (arn:aws:s3:::and*arn:aws:s3:::).- Server
20260804(eee05a17c) callsParseConfigStricton all policy write paths (bucket policies, named IAM policies, and service-account/STS inline policies), while keeping stored policy load paths on permissiveParseConfigso existing deployments never fail to boot.
Why SN-2026-003 Is Tracked as an Advisory Rather Than a CVE
Exploiting the s3:versionid evaluation bugs required an authenticated caller operating against a policy that explicitly used s3:versionid conditions (Null, StringEquals, or StringNotEquals). Standard s3:DeleteObjectVersion action checks were unaffected unless conditioned on s3:versionid. We track the condition evaluation and DeleteObjects rebinding fixes under SN-2026-003 and the bare ARN prefix rejection under SN-2026-005.
Scope Boundaries and Related Fixes
- Multi-Delete Object Lock governance bypass (
75a6734e4/#104): While auditingDeleteMultipleObjectsHandler, we noted how per-entry headers and retention checks interact; Server 20260903 (75a6734e4,SN-2026-008) subsequently tightenedDeleteMultipleObjectsHandlersox-amz-bypass-governance-retentionis only evaluated against the specific locked entry rather than across the batch. - Snowball
PutObjectExtractminio.versionIdPAX header: Noted during ourversionIdaudit and closed in Server 20261018 (SN-2026-016,14042b5aa). - Other condition keys (
username,userid,signatureversion,authType): We intentionally scoped80c2c9d03tos3:versionidrather than changing empty-slice semantics for identity keys that are always defined on authenticated requests.
Verification
Regression tests in cmd/signature-v4-utils_test.go, cmd/auth-handler_test.go, and cmd/bucket-policy_test.go verify:
Null: {"s3:versionid": "true"}andNull: {"s3:versionid": "false"}across versioned and unversionedGetObject,DeleteObject,CopyObject, andDeleteObjectscalls,?versionId=v1%20whitespace trimming againstDeny StringEquals {"s3:versionid": "v1"},POST /bucket?delete&versionId=query-level-decoywith both versioned and unversioned XML<Object>entries,- Rejection of
arn:aws:s3:::and*arn:aws:s3:::viaParseConfigStrict.
Source References
80c2c9d03—s3:versionidcondition omission, whitespace trimming, andDeleteObjectsper-entry rebinding (SN-2026-003)2f55347f7— request condition-source isolation (#53)eee05a17c/silo-pkg@2b087a1—ParseConfigStrictbare ARN prefix rejection (SN-2026-005)
Summary
Shipped in Server 20260804, 80c2c9d03 and eee05a17c (silo-pkg v3.12.0) ensure accurate s3:versionid condition evaluation across single-object and batch DeleteObjects operations while rejecting degenerate bare S3 ARN prefixes on policy writes.
3.17 - SN-2026-004: Object-Scoped Policy Grants Reaching Bucket Mutations
SN-2026-004 fixes an IAM resource-matching flaw in silo-pkg v3.11.0 (3c24ad1, 1f97549, d8b1fa7) and Server 20260804 (97b7d2804; upstream reference minio/minio#20449): when evaluating bucket-level S3 requests (args.ObjectName == ""), Statement.IsAllowed synthesized "bucket/" as the target resource string, allowing object-scoped wildcard grants such as Allow s3:* on arn:aws:s3:::bucket/* to authorize sensitive bucket-level mutations including s3:PutBucketPolicy and s3:DeleteBucket.
Conclusions first
- Root cause: When
args.ObjectNamewas empty,Statement.IsAllowedappended'/'to the bucket name ("bucket/"), which matched the glob"arn:aws:s3:::bucket/*". - Privilege escalation: A tenant principal given
Allow s3:* on arn:aws:s3:::tenant/*(a common shorthand for “all object operations inside this bucket”) could callPutBucketPolicyto grantPrincipal: "*"on the bucket, or callDeleteBucket,PutReplicationConfiguration,PutBucketLifecycle,PutBucketVersioning, orPutBucketObjectLockConfiguration. - Why a targeted fix was required: Simply dropping the trailing
'/'for all empty-object checks broke 11 existing STSListBuckettests and weakenedDeny s3:* on arn:aws:s3:::bucket/*policies. - Targeted boundary: For the 12 sensitive bucket-mutation actions on
Allowstatements (!legacyBucketResourceMatch && statement.Effect == Allow && isSensitiveBucketMutation(args.Action)), the resource pattern must match both"bucket"and"bucket/". SettingMINIO_API_LEGACY_BUCKET_RESOURCE_MATCH=onrestores pre-20260804matching if a deployment temporarily relies on legacy policies.
Why arn:aws:s3:::bucket/* Matched Bucket Requests
In silo-pkg/bucket/policy/statement.go (and iam/policy/statement.go), Statement.IsAllowed(args Args) constructed the resource string to test against Statement.Resources as follows:
Because the else branch unconditionally appended "/", every bucket-level request (args.ObjectName == "") was evaluated as resource = "mybucket/". When matched against a resource pattern of "arn:aws:s3:::mybucket/*", the glob wildcard * matched the empty suffix after "/" and returned true.
Multi-Tenant Escalation Impact
In AWS IAM and S3 bucket policies, arn:aws:s3:::mybucket identifies the bucket itself, whereas arn:aws:s3:::mybucket/* identifies objects inside the bucket. Administrators routinely write policies that grant a tenant or application team full object access using a wildcard action on the object ARN:
Because s3:* includes both object actions and bucket actions, and "arn:aws:s3:::tenant-a/*" matched resource = "tenant-a/", a caller holding this object-scoped policy could execute bucket-control mutations on tenant-a:
s3:PutBucketPolicy/s3:DeleteBucketPolicy: Replace the bucket policy with"Principal": "*"to expose the bucket publicly or grant themselves explicit bucket-level permissions.s3:DeleteBucket/s3:ForceDeleteBucket: Delete the bucket outright.s3:PutReplicationConfiguration: Point bucket replication at an attacker-controlled destination to exfiltrate objects.s3:PutBucketLifecycle: Expire all objects and historical versions in the bucket.s3:PutBucketVersioning/s3:PutBucketObjectLockConfiguration: Suspend versioning or alter retention defaults.
Why We Did Not Drop the Trailing Slash Globally
Our first prototype simply removed the else { resource += "/" } branch so that args.ObjectName == "" always evaluated against "mybucket". Running the test suite and code review exposed two major regressions with a blanket change:
- Historical
ListBucketcompatibility: Eleven upstream STS and IAM tests incmd/sts-handlers_test.gointentionally granteds3:ListBucket(ors3:*) onarn:aws:s3:::mybucket/*without listing the bare bucket ARNarn:aws:s3:::mybucket. Dropping"/"for read/list actions would break existing deployments whose application policies only listarn:aws:s3:::mybucket/*. Denystatement weakening: If an administrator wroteDeny s3:* on arn:aws:s3:::mybucket/*expecting it to block all operations on that bucket, evaluatingargs.ObjectName == ""against"mybucket"instead of"mybucket/"would cause theDenystatement to stop matching bucket-level calls—turning a security tightening patch into aDenybypass!
The Fix: Action-Scoped Dual Matching
To close the escalation path without breaking ListBucket or weakening Deny statements, Statement.IsAllowed restricts the stricter check to Allow statements on sensitive bucket-level mutations (silo-pkg v3.11.0, wired into the server in 97b7d2804):
For those actions on an Allow statement, the resource set must match "mybucket" (so arn:aws:s3:::mybucket/* alone is rejected, while arn:aws:s3:::mybucket or arn:aws:s3:::* is accepted) and match "mybucket/" (see Why Monotonicity Required Matching Both).
Second-Pass Action Audit: Which Bucket Writes Are Covered
Our initial draft only gated PutBucketPolicy and DeleteBucketPolicy, overlooking DeleteBucket and lifecycle/replication mutations. In the second pass, we audited every bucket-level write action (args.ObjectName == "") and divided them into two explicit groups.
The 12 protected sensitive bucket mutations (isSensitiveBucketMutation) are:
s3:PutBucketPolicys3:DeleteBucketPolicys3:DeleteBuckets3:ForceDeleteBuckets3:PutBucketLifecycles3:PutReplicationConfigurations3:PutBucketVersionings3:PutBucketObjectLockConfigurations3:PutBucketQuotaConfigs3:PutBucketRetentionConfigs3:PutBucketCorss3:DeleteBucketCors
(Note: when Server 20260804 shipped, per-bucket CORS handlers were not yet active; when Server 20260903 implemented per-bucket CORS in #71 / #80, s3:PutBucketCors and s3:DeleteBucketCors were already protected in isSensitiveBucketMutation.)
Four lower-risk or provisioning bucket actions were intentionally kept on historical matching to avoid breaking existing automation pipelines that scope credentials to arn:aws:s3:::app-bucket/*:
s3:CreateBucket(creating the bucket itself when provisioned with a prefix-scoped policy),s3:PutBucketTagging/s3:DeleteBucketTagging,s3:PutBucketEncryption,s3:PutBucketNotification.
Why Monotonicity Required Matching Both "bucket" and "bucket/"
We wanted a strict monotonicity guarantee: when MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH=off (the default), no Allow statement may authorize a request that would have been denied under legacy matching.
Proving that invariant took two iterations during code review:
- Attempt 1 (
NotResourcefailure): When we simply testedstatement.Resources.Match(args.BucketName, ...)instead of"mybucket/", a policy usingNotResource: ["arn:aws:s3:::mybucket/*"]flipped from denied (because"mybucket/"was inside the excludedNotResourceset) to allowed (because"mybucket"did not match the excluded pattern). - Attempt 2 (single-character
?wildcard failure): Even withoutNotResource, testing onlyargs.BucketNamefailed monotonicity on fixed-width wildcards! ConsiderAllow s3:PutBucketPolicy on Resource: ["arn:aws:s3:::mybucke?"]. Under legacy matching,"mybucket/"is 9 characters long, somybucke?(8 characters) did not match and the request was denied. If the new check tested only"mybucket"(8 characters),mybucke?would suddenly match and allow a request that legacy mode denied (1f97549).
Requiring statement.Resources.Match(args.BucketName, ...) and statement.Resources.Match(resource, ...) ("mybucket" and "mybucket/") guarantees mathematical monotonicity for all Resource and NotResource patterns: the new check is a pure subset (NewAllow = LegacyAllow ∧ MatchesBareBucket) of legacy Allow evaluation (d8b1fa7). Note that this means a synthetic pattern like arn:aws:s3:::mybucke? (which matches "mybucket" but not "mybucket/") will not authorize sensitive bucket mutations; administrators should write arn:aws:s3:::mybucket or arn:aws:s3:::mybucket*.
Operator Impact and Compatibility Escape Hatch
- Standard policies need no changes: Any policy that grants bucket administration on
arn:aws:s3:::mybucket,arn:aws:s3:::mybucket*, orarn:aws:s3:::*continues to work unchanged. - Object-only policies (
arn:aws:s3:::mybucket/*): Can still read/write objects and callListBucket,CreateBucket,PutBucketEncryption,PutBucketTagging, andPutBucketNotification, but will now receiveAccessDeniedon the 12 sensitive bucket mutations unlessarn:aws:s3:::mybucketis explicitly added toResource. - Escape hatch: If an existing automation workflow requires
arn:aws:s3:::mybucket/*to authorize one of the 12 sensitive bucket mutations during migration, setMINIO_API_LEGACY_BUCKET_RESOURCE_MATCH=onuntil the policy is updated.
Follow-Ups Left Open
- Startup policy audit: SILO does not rewrite stored IAM or bucket policies on upgrade; operators can inspect policies with
mcli admin policy infoto check whether any administrative role relied onarn:aws:s3:::bucket/*for bucket mutations. - Self-explaining denial diagnostics: S3
AccessDeniedresponses intentionally do not disclose which statement or ARN shape failed to match.
Summary
Shipped in Server 20260804 (5ff00073a and silo-pkg d8b1fa7 / v3.12.0), the 12 sensitive bucket-mutation actions require the policy Resource to match the bucket ARN itself unless MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH=on is explicitly set.
3.18 - SN-2026-018: Client Source Address Trust, Forwarded Headers, and aws:SourceIp
SN-2026-018 addresses client source-IP spoofing across aws:SourceIp IAM conditions, audit logs, bucket notification events, and admin HTTP traces in silo-pkg v3.12.0 (62f326d) and Server 20260804 (00a8171cb, pgsty/silo#41): handlers.GetSourceIP previously trusted X-Forwarded-For, X-Real-IP, and RFC 7239 Forwarded from any TCP peer by default, and setting _MINIO_API_XFF_HEADER=off only disabled X-Forwarded-For while continuing to trust client-supplied X-Real-IP and Forwarded headers.
Conclusions first
_MINIO_API_XFF_HEADER=offleft two headers wide open: Setting_MINIO_API_XFF_HEADER=offskippedX-Forwarded-For, buthandlers.GetSourceIPimmediately fell through toX-Real-IPandForwarded. Any direct client could forgeX-Real-IP: 10.0.0.1to satisfyaws:SourceIppolicies.- “Put a reverse proxy in front” is often insufficient: Even behind an ingress proxy, if the proxy appends to
X-Forwarded-For(proxy_add_x_forwarded_for) while the server reads the left-most IP, or if internal pods can reach the server’s ClusterIP directly, an attacker can still spoofaws:SourceIp. MINIO_API_TRUSTED_PROXIESprovides explicit right-to-left chain verification: Server 20260804 introducesMINIO_API_TRUSTED_PROXIES(off/none, CIDR/IP list, orall/*for legacy behavior) and walksX-Forwarded-For/Forwardedfrom right to left, stopping at the first hop not in the trusted proxy set.
Every Consumer of handlers.GetSourceIP
In SILO, handlers.GetSourceIP(r) (silo-pkg/handlers/proxy.go) is the single function responsible for extracting the client IP address from an incoming *http.Request. Four security- and observability-sensitive subsystems depend on its return value:
| Consumer | Location | Security / Operational Role |
|---|---|---|
aws:SourceIp IAM condition |
cmd/signature-v4-utils.go (getConditionValues) |
Evaluates IpAddress and NotIpAddress policy conditions on bucket and IAM policies. |
Audit logging (remotehost) |
cmd/logger/audit.go |
Records the client IP in structured compliance audit logs. |
Bucket notification events (Host) |
cmd/notification.go |
Populates requestParameters.sourceIPAddress in S3 event notifications. |
Admin HTTP trace (mc admin trace) |
cmd/http-tracer.go |
Displays the client IP in live diagnostic traces. |
Note that handlers.GetSourceIPRaw returned the first matched header token without validating whether it was even a syntactically valid IP address (a unit test in upstream silo-pkg literally asserted that X-Forwarded-For: _gazonk returned "_gazonk"!). While IpAddress condition evaluation parses the string with net.ParseIP, audit logs and notification payloads received the raw string.
Why _MINIO_API_XFF_HEADER=off Did Not Stop Spoofing
In silo-pkg/handlers/proxy.go, GetSourceIPRaw inspected three groups of headers in order:
When PR minio/minio#20977 (and minio/pkg#306) added _MINIO_API_XFF_HEADER=off in February 2025, it gated only the xffHeaders loop:
- With
_MINIO_API_XFF_HEADER=off,GetSourceIPRawskippedX-Forwarded-For, - Then immediately checked
X-Real-IP(realIPHeaders) andForwarded(forwardedHeaders) unconditionally!
Thus, an operator who set _MINIO_API_XFF_HEADER=off specifically to prevent aws:SourceIp spoofing on a directly exposed SILO endpoint remained completely vulnerable to curl -H "X-Real-IP: 192.0.2.10" or curl -H "Forwarded: for=192.0.2.10".
Upstream History since 2017
This area had a long back-and-forth history:
- 2017-07 (
minio/minio#4736): Upstream briefly removedX-Forwarded-ForfromGetSourceIPafter recognizing that clients could spoofaws:SourceIp. - Subsequent refactors: Reverse-proxy deployments complained that
aws:SourceIpand audit logs only saw the load balancer’s IP, soX-Forwarded-For,X-Real-IP, andForwardedwere re-admitted—taking the first (left-most) comma-separated entry inX-Forwarded-For. - 2023-08 (
minio/minio#17878) & 2025-02 (minio/minio#20977): Operators requested a way to disableX-Forwarded-Fortrust, resulting in_MINIO_API_XFF_HEADER, which leftX-Real-IPandForwardedactive.
Why “Just Put a Reverse Proxy in Front” Is Not Enough
A common misconception is that forwarded-header spoofing only matters when SILO is exposed directly to the internet without a reverse proxy. In practice, two standard production topologies remain vulnerable even when a reverse proxy is present:
- Left-most
X-Forwarded-Forvs. proxy append (proxy_add_x_forwarded_for): Standard nginx configurations useproxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;, which appends the client’s real IP to whateverX-Forwarded-Forheader the client sent. If the client sendsX-Forwarded-For: 10.0.0.5, nginx forwardsX-Forwarded-For: 10.0.0.5, 203.0.113.99. Because legacyGetSourceIPunconditionally took the first (left-most) address (10.0.0.5), the attacker’s spoofed IP won even behind nginx! - Coexisting ingress and direct paths (Kubernetes ClusterIP): In Kubernetes, external traffic arrives through an Ingress controller, while in-cluster workloads connect to the
siloService (ClusterIP) directly. IfGetSourceIPblindly trusts forwarded headers whenever an Ingress exists, any pod inside the cluster can connect to theClusterIPdirectly and forgeX-Forwarded-FororX-Real-IPto impersonate a trusted subnet.
Design of MINIO_API_TRUSTED_PROXIES
To solve both problems cleanly, silo-pkg v3.12.0 (62f326d) and Server 20260804 (00a8171cb) introduce MINIO_API_TRUSTED_PROXIES:
| Setting | Peer Trust | Header Evaluation |
|---|---|---|
MINIO_API_TRUSTED_PROXIES=off (or none) |
Trust no proxy hops | Always uses the direct TCP peer (r.RemoteAddr) for external peers; ignores X-Forwarded-For, X-Real-IP, and Forwarded. |
MINIO_API_TRUSTED_PROXIES=10.0.1.0/24,192.168.1.10 |
Trust only listed IPs/CIDRs | If r.RemoteAddr is not in the trusted set, returns r.RemoteAddr immediately. If r.RemoteAddr is trusted, walks X-Forwarded-For (or Forwarded) from right to left and returns the first IP not in the trusted set. |
Unset, all, or * |
Legacy compatibility mode | Preserves pre-20260804 behavior (unless _MINIO_API_XFF_HEADER=off is set). |
Note that handlers.GetSourceScheme (X-Forwarded-Proto) is intentionally left untouched so TLS redirect and presigned URL scheme detection behind reverse proxies continue to work regardless of MINIO_API_TRUSTED_PROXIES.
Seven Engineering Trade-Offs
1. Default Compatibility vs. Secure-by-Configuration
Changing the unset default to off in a minor release would immediately break every reverse-proxy deployment that uses aws:SourceIp policies without having set MINIO_API_TRUSTED_PROXIES. We kept the unset default compatible and made MINIO_API_TRUSTED_PROXIES=off and CIDR lists first-class, documented controls.
2. Why We Reversed Widening _MINIO_API_XFF_HEADER=off
In our first draft, we widened _MINIO_API_XFF_HEADER=off so that it also disabled X-Real-IP and Forwarded. Running the server test suite immediately failed TestXFFDisabled in cmd/generic-handlers_test.go!
Inspecting TestXFFDisabled showed why: the test specifically modelled an nginx reverse proxy that overwrites X-Real-IP: $remote_addr (trusted single IP) while appending to X-Forwarded-For: $proxy_add_x_forwarded_for (untrusted left-most IP). Operators who set _MINIO_API_XFF_HEADER=off in that exact topology relied on SILO ignoring X-Forwarded-For so that it would fall through to X-Real-IP. We therefore preserved _MINIO_API_XFF_HEADER=off semantics for TestXFFDisabled when MINIO_API_TRUSTED_PROXIES is unset, and provided MINIO_API_TRUSTED_PROXIES=off to disable all forwarded IP headers (X-Forwarded-For, X-Real-IP, and Forwarded).
3. Why MINIO_API_TRUSTED_PROXIES Is Env-Only (No mc admin config Key)
In SILO’s startup sequence, the HTTP listener and initial request routing come up before the distributed api subsystem configuration is loaded from erasure storage. If trusted proxies were configured via mc admin config set api, there would be a startup window before config load where GetSourceIP either failed open or failed closed, and updating the setting at runtime would require atomic reloads across silo-pkg. Making MINIO_API_TRUSTED_PROXIES an environment variable configured before listener startup eliminates any race window.
4. Loopback Exemption: Peer vs. Chain Hop
SILO’s built-in FTP and SFTP servers (cmd/ftp-server.go, cmd/sftp-server.go) and local inter-node forwarding connect to the S3 layer over loopback (127.0.0.1 / ::1) and pass the external client’s IP in X-Forwarded-For. Therefore, when the immediate TCP peer (r.RemoteAddr) is loopback, SILO treats that local peer hop as trusted so FTP/SFTP audit logs and aws:SourceIp continue to see the real FTP/SFTP client IP.
Crucially, however, loopback is not implicitly trusted as an intermediate hop inside the X-Forwarded-For header chain! Otherwise, an external client behind one trusted proxy could send X-Forwarded-For: 10.0.0.1, 127.0.0.1, causing the right-to-left walker to skip the proxy (r.RemoteAddr), skip 127.0.0.1 in the header, and trust the attacker’s 10.0.0.1.
5. Header Precedence: X-Forwarded-For Before X-Real-IP
When MINIO_API_TRUSTED_PROXIES is configured with a CIDR/IP list, GetSourceIP evaluates X-Forwarded-For (right-to-left), then X-Real-IP (validated with parseIP), then Forwarded (right-to-left). Note that this differs intentionally from getSTSLDAPTrustedProxySourceIP (cmd/sts-handlers.go), which checks X-Real-IP first for single-hop LDAP STS proxies; in general multi-hop S3 proxy chains, X-Forwarded-For carries the hop-by-hop chain needed for right-to-left verification while X-Real-IP may remain frozen at an outer hop.
6. Broad CIDR Warning
If an operator sets MINIO_API_TRUSTED_PROXIES=10.0.0.0/8 when ordinary application pods also run in 10.0.0.0/8, any pod in that /8 is treated as a trusted proxy and can forge X-Forwarded-For. Operators should list only the dedicated ingress/load-balancer IPs or subnet.
7. Multi-Node Distributed Clusters
In a multi-node SILO cluster (silo server http://node{1..4}/data), a load balancer may route a client request to Node 1, which internally proxies or forwards sub-requests to Nodes 2–4:
| Topology | What MINIO_API_TRUSTED_PROXIES Must Include |
|---|---|
| Single node behind reverse proxy | Reverse proxy IPs/CIDRs only |
| Multi-node cluster behind reverse proxy | Reverse proxy IPs/CIDRs and all SILO cluster node IPs/CIDRs |
| Direct client access to multi-node cluster (no external proxy) | All SILO cluster node IPs/CIDRs (or off if clients always hit nodes directly without peer forwarding) |
Compatibility Details
- No non-IP garbage returned in trusted-proxy mode: When
MINIO_API_TRUSTED_PROXIESis active (offor CIDR list), every candidate header token is validated as a real IP address; malformed tokens like_gazonkare rejected rather than returned to callers. - Removal of unreachable IPv4-mapped prefix string hack: Legacy
proxy.gocontained a string check for"::ffff:"that was unreachable or mishandled hex forms;netip.Addr.Unmap()now normalizes IPv4-mapped IPv6 addresses (::ffff:192.0.2.1->192.0.2.1) cleanly.
Bugs Caught During Multi-Pass Code Review
Iterative code review of the patch across silo-pkg and silo caught nine subtle edge cases before release:
Second-Pass Review Findings
init()vs.MINIO_CONFIG_ENV_FILEstartup ordering: ReadingMINIO_API_TRUSTED_PROXIESand_MINIO_API_XFF_HEADERonly inside a packageinit()function meant values loaded later fromMINIO_CONFIG_ENV_FILEinserverHandleEnvVarswere ignored! Server20260804explicitly re-applies both settings inserverHandleEnvVars.- Loopback in-chain spoofing (
X-Forwarded-For: spoofed, 127.0.0.1): Caught the distinction between trusting a loopbackr.RemoteAddrpeer vs. trusting127.0.0.1inside anX-Forwarded-Forheader list (fixed as described in Trade-Off 4). - IPv6 zone identifiers (
fe80::1%eth0): Link-local IPv6 peers include a%zonesuffix inr.RemoteAddr. Usingnetip.ParseAddrand stripping the zone before CIDR matching prevents link-local peers from failing IP parsing. - Multiple
X-Forwarded-Forheader lines (r.Header.Values): HTTP proxies such as HAProxy or Envoy may append a secondX-Forwarded-Forheader line rather than comma-joining onto the first line. Callingr.Header.Get("X-Forwarded-For")read only the first (client-controlled!) header line. We switched to iteratingr.Header.Values("X-Forwarded-For")from last header line to first. - Internal
forwarder.goheader hygiene: WhenMINIO_API_TRUSTED_PROXIESrejected an untrusted peer’sX-Real-IPorForwardedheader, internal HTTP forwarding needed to avoid re-injecting untrusted headers alongside a trusted peer hop. ::ffff:0:0/96CIDR unmapping: Unmapping the prefix address of::ffff:0:0/96without checking prefix length would turn a/96IPv6 prefix into0.0.0.0and match all IPv4 addresses; we preserve CIDR prefix semantics accurately.
Third-Pass Review Findings
- Fail-closed on empty CIDR list (
MINIO_API_TRUSTED_PROXIES=","): If an operator setsMINIO_API_TRUSTED_PROXIES=","(or whitespace-only), silently treating the empty slice as “unset (trust all)” would fail open! Startup now returns an explicit configuration error ifMINIO_API_TRUSTED_PROXIESis set to an empty or unparseable value. os.LookupEnvvs.env.Get(""): Usingenv.Get(EnvAPITrustedProxies, "")could not distinguish betweenMINIO_API_TRUSTED_PROXIESbeing unset and being set to""; we check explicit presence so empty settings fail closed.- Zero-allocation backwards scan and 100-hop cap: Allocating
strings.Split(header, ",")on every request allowed a client sending a 1 MBX-Forwarded-Forheader full of commas to trigger a ~33 MB slice allocation per request. We replacedstrings.Splitwith an in-place right-to-left scanner and capped inspection at 100 hops.
Operator Recommendations
- Direct-attached SILO (no reverse proxy): Set
MINIO_API_TRUSTED_PROXIES=off. - SILO behind nginx / HAProxy / Envoy / Ingress: Set
MINIO_API_TRUSTED_PROXIESto the CIDR or IP list of the reverse proxies (and SILO node IPs in a multi-node cluster).
Out of Scope
- PROXY protocol v1/v2: TCP-layer PROXY protocol support at the listener is separate from HTTP
X-Forwarded-For/Forwardedparsing. GetSourceScheme: Left unchanged so HTTPS scheme detection behind TLS-terminating proxies continues to work independently ofMINIO_API_TRUSTED_PROXIES.ExistingObjectTag/*condition isolation: Fixed separately in2f55347f7(also shipped in20260804) so only stored object tags—never client request headers—populateExistingObjectTag/*conditions.
Severity Classification
Tracked as SN-2026-018 in the SILO advisory ledger (as with CVE-2026-42600, minio/minio is archived; severity is Moderate-to-High depending on whether policies rely on aws:SourceIp). Upgrading to Server 20260804 and setting MINIO_API_TRUSTED_PROXIES closes the spoofing vector across IAM, audit logs, notifications, and traces.
3.19 - SN-2026-019: Notification Keyspace Registration and Validation Redaction
SN-2026-019 covers a class of notification configuration defects fixed in Server 20260804 (162ded343 and 672c07a0f, pgsty/silo#39): notification parser and legacy-migration paths read or wrote KVS configuration keys that were missing from Default*KVS, while CheckValidKeys echoed raw key values in validation errors and FetchEnabledTargets aborted all active notification targets on the first validation failure. In Server 20260903 (f1ba68358), the discrete PostgreSQL and MySQL host/credential keys uncovered by our AST audit were retired in favor of connection_string and dsn_string.
Conclusions first
- Four uncoordinated surfaces drifted apart: In the notification configuration subsystem,
Default*KVS(valid key schema),Help*(help text and secret/sensitive classification),GetNotify*(parser), andSetNotify*(legacy config migration) were maintained by hand without a cross-check. - One invalid target disabled all notification targets:
FetchEnabledTargetsvalidates every notification subsystem sequentially and returns on the first error, leaving the global notification target listnilacross the server. - Validation errors echoed secret values: When
CheckValidKeysrejected an unknown or mistyped key, it formatted the error usingkv.String(), printingkey=value—including credentials or passwords—into logs and admin CLI responses. - AST-enforced invariant: Server
20260804registers the missing NATS and AMQP keys, redacts values fromCheckValidKeyserrors, expandsHasSensitiveKeys, and enforces a compile-time AST test ensuring every key read or written bycmd/config/notifyis registered inDefault*KVS.
A Puzzling Bug Report
The investigation started from a bizarre symptom reported by kuldeep-link11: an operator ran mc admin config set us notify_nats:FITCHECK address="..." subject="..." and received an error complaining about a completely different target (notify_nats:ONE) and two keys (nkey_seed= tls_handshake_first=off) that the operator had never passed on the command line:
Two separate mechanisms combined to produce that confusing error:
- Tokenizer absorption on
mc admin config set:kvFieldsininternal/config/parser.gosplits input tokens by matching only keys registered inDefaultKVS. When a user passes an unregistered key on the command line,kvFieldsdoes not recognize it as a key delimiter and silently absorbs"nkey_seed=..."as part of the preceding key’s value (such assubject). - Legacy migration poisoning on
notify_nats:ONE: Earlier in the deployment’s life, the legacy configuration migrator (SetNotifyNATSincmd/config/notify/legacy.go) had serializednotify_nats:ONEdirectly into the KVS struct—bypassingkvFields—and writtennkey_seedandtls_handshake_firstas discrete KVS keys. When the operator later editednotify_nats:FITCHECK,validateConfigre-validated all stored targets and tripped overnotify_nats:ONE.
One Constant Used for Two Different Namespaces
Inspecting internal/config/notify/target/nats.go immediately revealed a second bug right next to nkey_seed:
When PR minio/minio#19139 added NATS user credentials support in February 2024, NATSUserCredentials was placed inside the Env* constant block with the value "MINIO_NOTIFY_NATS_USER_CREDENTIALS", and then used as both the environment variable name and the KVS key name in DefaultNATSKVS. As a result, setting user_credentials=/path/to/creds via mc admin config set notify_nats was rejected as an unknown key, whereas the literal string MINIO_NOTIFY_NATS_USER_CREDENTIALS=/path/to/creds was what DefaultNATSKVS expected on disk.
Four Uncoordinated Surfaces
Auditing all notification targets in cmd/config/notify/ showed that every target defines four separate surfaces:
Default*KVS(cmd/config/notify/config.go): the canonical list of valid KVS keys enforced byCheckValidKeys,Help*(cmd/config/notify/help.go): user-facing help metadata andType(Sensitive/Secret),GetNotify*(cmd/config/notify/parse.go): reads KVS keys andMINIO_NOTIFY_*environment variables,SetNotify*(cmd/config/notify/legacy.go): migrates older JSON config structs into KVS pairs.
Across NATS and AMQP, four keys had drifted across those four surfaces:
| Target | Key | Introduced In | Default*KVS |
Help* |
GetNotify* |
SetNotify* |
Failure Mode |
|---|---|---|---|---|---|---|---|
notify_nats |
user_credentials |
#19139 (2024-02) |
Registered as MINIO_NOTIFY_NATS_USER_CREDENTIALS |
Missing | Read MINIO_NOTIFY_NATS_USER_CREDENTIALS |
Wrote MINIO_NOTIFY_NATS_USER_CREDENTIALS |
user_credentials rejected on mc admin config set |
notify_nats |
nkey_seed |
#21231 (2025-05) |
Missing | Missing | Read nkey_seed |
Wrote nkey_seed |
Stored config fails CheckValidKeys |
notify_nats |
tls_handshake_first |
#21008 (2025-02) |
Missing | Present | Read tls_handshake_first |
Wrote tls_handshake_first |
Stored config fails CheckValidKeys |
notify_amqp |
immediate |
Initial KVS migration | Missing | Missing | Read immediate |
Wrote cfg.Immediate into AmqpInternal! |
immediate rejected; SetNotifyAMQP overwrote internal with cfg.Immediate |
Notice the AMQP bug in the last row: SetNotifyAMQP in cmd/config/notify/legacy.go had a copy-paste typo that wrote strconv.FormatBool(cfg.Immediate) under the key target.AmqpInternal and never serialized cfg.Internal at all.
Blast-Radius Amplifier: All-or-Nothing Target Loading
Why did a broken notify_nats:ONE target matter so much? In cmd/config/notify/parse.go, FetchEnabledTargets loads all ten notification subsystems (amqp, elasticsearch, kafka, mqtt, mysql, nats, nsq, postgres, redis, webhook) in sequence. If any subsystem returns a validation error from CheckValidKeys, FetchEnabledTargets returns early before populating the active target map.
Consequently, a single invalid key in an unused notify_nats target causes startup or config reload to leave the global notification target list empty, silently disabling all bucket notifications (such as active Kafka or Webhook targets) until an administrator removes the offending config entry.
We considered changing FetchEnabledTargets to isolate errors per subsystem and keep other targets running, but declined that change: across SILO, configuration validation is intentionally all-or-nothing so that mc admin config set rejects broken configurations rather than partially applying them. Fixing key registration removes the false-positive failures without weakening config transaction semantics.
Fixing Key Registration and Classification
Commit 162ded343 fixes the NATS and AMQP keyspace defects:
- Split key constant from env constant:
target.NATSUserCredentialsis now"user_credentials", andtarget.EnvNATSUserCredentialsis"MINIO_NOTIFY_NATS_USER_CREDENTIALS". - Register all four keys in
DefaultNATSKVSandDefaultAMQPKVS:user_credentials,nkey_seed, andtls_handshake_firstare registered inDefaultNATSKVS, andimmediateis registered inDefaultAMQPKVS. - Complete
HelpNATSandHelpAMQP: Added help entries foruser_credentials,nkey_seed, andimmediate. Bothuser_credentialsandnkey_seedtake local file paths on the server filesystem (just likeclient_key), so we classified them asconfig.Sensitiverather thanconfig.Secret. - Fix
SetNotifyAMQP:SetNotifyAMQPnow writescfg.Internaltotarget.AmqpInternalandcfg.Immediatetotarget.AmqpImmediate.
Narrow Backward Tolerance for Stored NATS Configs
What if a deployment prior to 20260804 had worked around #19139 by storing the literal key MINIO_NOTIFY_NATS_USER_CREDENTIALS=/path/to/creds in its persisted cluster configuration?
Simply renaming NATSUserCredentials to "user_credentials" would cause CheckValidKeys to reject that persisted configuration on upgrade—triggering the exact FetchEnabledTargets outage described above.
While designing backward tolerance, we uncovered a subtle trap between two identically named functions in internal/config:
- The free function
config.CheckValidKeys(subSys, kvs, defaultKVS, deprecatedKeys...)accepts an optional variadic list of deprecated keys and permits them during load validation. - The method
kvs.CheckValidKeys(subSys, defaultKVS)callsCheckValidKeys(subSys, kvs, defaultKVS)without passing any deprecated keys.
We updated the NATS load path (GetNotifyNATS) to tolerate "MINIO_NOTIFY_NATS_USER_CREDENTIALS" via the deprecated-key list on config.CheckValidKeys, and implemented deterministic precedence when populating UserCredentials:
MINIO_NOTIFY_NATS_USER_CREDENTIALSenvironment variable (highest precedence),user_credentialsKVS key,- Legacy
MINIO_NOTIFY_NATS_USER_CREDENTIALSKVS key (fallback whenuser_credentialsis empty).
Redacting Values in Validation Errors (SN-2026-019)
Before 20260804, both config.CheckValidKeys and KVS.CheckValidKeys built their error string by appending kv.String() (key=value) for every unrecognized key. If an administrator mistyped a secret key name—for example, pasword=Hunter2 or auth_tokn=eyJ...—or if a legacy migrator wrote an unregistered credential key, the error message echoed the cleartext secret into the console response and server error logs.
Commit 162ded343 changes both CheckValidKeys functions to report only the offending key names (kv.Key), never kv.Value.
In commit 672c07a0f, we also audited config.HasSensitiveKeys (internal/config/config.go), which masks secrets when exporting configurations via mc admin config get or generating health-diagnostics bundles. We expanded HasSensitiveKeys to cover all credential key names used across notification, identity, and KMS subsystems (password, auth_token, secret_key, client_secret, nkey_seed, user_credentials, sasl_password, connection_string, and dsn_string).
Enforcing the Invariant with an AST Audit Test
To make sure Default*KVS, parse.go, and legacy.go can never drift apart again, we wrote an AST-based verification test in cmd/config/notify/config_audit_test.go. Using go/ast, the test parses parse.go and legacy.go, resolves every kvs.Get(...) and kvs.Set(...) call back to its string constant definition in internal/config/notify/target, and verifies that every referenced key exists in the corresponding Default*KVS.
While reviewing the audit test itself, we found and closed two blind spots in our initial test implementation:
- Reverse floor assertion: Checking only that
parse.go ⊆ Default*KVSwould pass even if the AST walker broke and extracted zero keys! We added the reverse check requiring every key registered inDefault*KVS(except documented universal keys likeenableandcomment) to be read inparse.goand asserting a minimum key-count floor per subsystem. - Untyped composite literals: In
legacy.go, helper tables constructedconfig.KVS{...}using untyped{Key, Value}composite literals rather than callingkvs.Set(). Extending the AST visitor to inspectconfig.KVScomposite literals immediately surfaced a second bug in the codebase.
What the AST Audit Caught: PostgreSQL and MySQL Discrete Keys
As soon as the AST visitor inspected config.KVS composite literals in legacy.go, it flagged SetNotifyPostgres and SetNotifyMySQL: both functions wrote five discrete keys—host, port, username, password, and database—that were read by GetNotifyPostgres / GetNotifyMySQL (parse.go) when connection_string / dsn_string was empty, yet were completely absent from DefaultPostgresKVS and DefaultMySQLKVS!
In addition, because password was not registered in DefaultPostgresKVS or DefaultMySQLKVS, passing password=... or migrating a legacy PostgreSQL/MySQL config triggered CheckValidKeys and printed the database password in the error message prior to 162ded343.
We handled this in two steps across two releases:
- Short-term fix in Server 20260804 (
672c07a0f): Registeredhost,port,username,password, anddatabaseinDefaultPostgresKVSandDefaultMySQLKVS(and markedpasswordasSecretinHelpPostgres/HelpMySQL) so that existing configurations and the new AST invariant were immediately consistent. - Architectural cleanup in Server 20260903 (
f1ba68358): Having two parallel ways to configure database connections (connection_stringvs. discretehost/port/username/password/databasekeys) causedSetNotifyPostgresandSetNotifyMySQLto overwriteconnection_stringwith empty discrete fields during config round-trips. Server20260903retired the five discrete keys acrossDefault*KVS,Help*,parse.go, andlegacy.go, standardizing exclusively onconnection_string(PostgreSQL) anddsn_string(MySQL).
Declined Changes
- Do not change
kvFieldstokenizer framing in this patch: ChangingkvFieldsto split on arbitrarykey=valuetokens rather than knownDefaultKVSkeys would alter how unquoted spaces inside values (such as SQL queries or subjects) are parsed. WithDefault*KVSnow verified by the AST audit, valid keys are always recognized bykvFields. - Do not weaken
FetchEnabledTargetsfail-fast validation: As explained in Blast-Radius Amplifier, configuration validation remains atomic across all subsystems.
Review Record
Code review verified the AST audit’s reverse floor assertions, untyped config.KVS composite literal inspection, CheckValidKeys value redaction (162ded343), and HasSensitiveKeys credential coverage (672c07a0f).
Summary
Shipped in Server 20260804 (162ded343 and 672c07a0f) with follow-up cleanup in Server 20260903 (f1ba68358), all notification KVS keys are verified against Default*KVS by AST tests and unrecognized-key errors redact values.
3.20 - Duplicate Part Numbers in CompleteMultipartUpload
In CompleteMultipartUpload, the S3 API requires part numbers in the completion XML to be strictly ascending. Up through Server 20260725, SILO enforced that invariant with sort.SliceIsSorted and a strict < comparator—which accepts non-decreasing sequences such as [1, 1] and duplicates the referenced part in the assembled object. Server 20260804 (pgsty/silo#49) replaces that check with an explicit adjacent-pair comparison (Parts[i-1].PartNumber >= Parts[i].PartNumber) and fixes an adjacent empty-part panic in ReadParts (2602177ef).
Conclusions first
- The bug is real and small:
CompleteMultipartUploadusedsort.SliceIsSortedwith<, which tests non-decreasing order (a[0] <= a[1] <= ...) rather than strictly increasing order (a[0] < a[1] < ...). Passing duplicate part numbers such as[1, 1]succeeded and duplicated the underlying part in the final object. - This is a data-integrity defect, not a CVE: Triggering it requires a caller who already holds
s3:PutObjectpermission on the target key and a validuploadIdfor an in-progress multipart upload. It does not cross tenant, bucket, or credential boundaries. - The fix stays at the HTTP handler boundary:
cmd/object-handlers.gorejectsParts[i-1].PartNumber >= Parts[i].PartNumberwithInvalidPartOrder(HTTP 400). Sparse part numbers ([1, 3],[5, 9],[3]) remain valid.
Why sort.SliceIsSorted Accepted Duplicates
In cmd/object-handlers.go, CompleteMultipartUploadHandler validated the parsed <CompleteMultipartUpload> part list with:
In Go’s standard library, sort.SliceIsSorted iterates i from n-1 down to 1 and calls less(i, i-1). If less(i, i-1) returns true, the slice is out of order. When two adjacent elements are equal (Parts[i].PartNumber == Parts[i-1].PartNumber), Parts[i].PartNumber < Parts[i-1].PartNumber evaluates to false, so SliceIsSorted treats them as sorted.
Replacing sort.SliceIsSorted with a direct adjacent check closes the gap:
Two Refactors Preserved the Defect
Tracing git log -S across the handler history shows that the non-decreasing check survived two separate cleanups over nine years:
| Date | PR / Commit | What Changed | Why [1, 1] Survived |
|---|---|---|---|
| 2016-08 | minio/minio#2490 |
Introduced sort.IsSorted(completedParts(complMultipartUpload.Parts)) with Less(i, j) { return a[i].PartNumber < a[j].PartNumber }. |
sort.IsSorted only checks that no element is smaller than its predecessor. |
| 2017-11 | minio/minio#5170 |
Replaced the named completedParts type with sort.SliceIsSorted using the same < closure. |
Mechanical refactor of the same comparator. |
| 2023-04 | minio/minio#17066 |
Moved ErrInvalidPartOrder validation earlier in CompleteMultipartUploadHandler before hitting the object layer. |
Moved the check block verbatim without testing duplicate part numbers. |
Every unit and mint test for ErrInvalidPartOrder tested inversions such as [2, 1], never adjacent duplicates such as [1, 1].
Observed Impact
Before the fix, uploading a single 5 MiB part (partNumber=1) and completing the upload with two <Part> entries both referencing PartNumber=1 and its ETag produced the following behavior:
| Step | Request | Pre-Fix Result | Post-Fix Result |
|---|---|---|---|
| 1 | UploadPart(partNumber=1) (5 MiB) |
200 OK, returns ETag: "e1" |
200 OK, returns ETag: "e1" |
| 2 | CompleteMultipartUpload with [(1, "e1"), (1, "e1")] |
200 OK, ETag: "...-2" |
400 Bad Request (InvalidPartOrder) |
| 3 | HeadObject / GetObject |
Content-Length: 10485760 (part 1 repeated twice) |
Object not created; upload remains open |
| 4 | Retry CompleteMultipartUpload with [(1, "e1")] |
404 NoSuchUpload (upload already consumed) |
200 OK, Content-Length: 5242880 |
Because CompleteMultipartUpload consumes the upload state on success, a buggy client that accidentally duplicated an entry in its completion XML (for example, when retrying an UploadPart call and appending to an in-memory slice instead of deduplicating by part number) silently wrote a corrupted object of the wrong size and could not retry completion without re-uploading every part.
Why This Is Not a CVE
We evaluated whether a malicious caller could weaponize duplicate part numbers for storage amplification:
- No physical disk amplification: In erasure-coded storage,
CompleteMultipartUploadrecords part metadata inxl.metareferencing the existingpart.1data files on disk rather than copying part bytes per entry. Reading the object streamspart.1twice, and background healing or copying materializes the logical size, but the caller could achieve the same result legitimately by uploading two parts or callingUploadPartCopy. - Same-caller scope: Executing
CompleteMultipartUploadrequires write access to the object and knowledge of theuploadId. No unauthenticated or read-only principal can trigger it.
The issue is therefore tracked as a correctness and S3 protocol compliance bug (#49) rather than a security advisory (SN-2026-*).
Handler vs. Object-Layer Boundary
During investigation we checked whether erasureObjects.CompleteMultipartUpload (cmd/erasure-multipart.go) also validates part ordering. It does not: the object layer iterates for i, part := range parts and matches each entry against uploadID part metadata.
We kept the check in CompleteMultipartUploadHandler (cmd/object-handlers.go) rather than duplicating it inside every ObjectLayer implementation for two reasons:
- Single untrusted entry point: Client-supplied completion lists enter only through
CompleteMultipartUploadHandler. Internal callers (batch-replicate,restore,decommission,rebalance, andPutObjectPartpathways) construct part lists directly from existingObjectInfo.Partsor sequential1..nloops. - Fail-fast before lock acquisition: Rejecting malformed XML in the HTTP handler avoids acquiring the object’s namespace lock or reading upload metadata from disk.
Preserving Sparse Part Numbers
The S3 specification requires part numbers in CompleteMultipartUpload to be strictly increasing (1 <= PartNumber <= 10,000), not contiguous and not starting at 1. A client that uploads parts 1, 2, 3 and completes using only [1, 3], or completes using [5, 9] or [3], is conforming to S3.
To make sure our fix did not over-constrain valid clients, we added explicit regression cases for:
[1, 3](sparse gap),[3](single part not starting at1),[1, 1]and[1, 2, 2, 3](adjacent duplicates, rejected withErrInvalidPartOrder),[1, 3, 2]and[2, 1](decreasing order, rejected withErrInvalidPartOrder),[1, 2, 1](non-adjacent duplicate, rejected withErrInvalidPartOrder).
Error Code Precedence for Out-of-Range Duplicates
One subtle edge case is an input that is both out of range (PartNumber < 1) and duplicated, such as [0, 0] or [-1, -1].
Because CompleteMultipartUploadHandler validates part ordering before iterating into the object layer (where individual part numbers are looked up and ErrInvalidPart is returned), inputs like [0, 0] previously passed sort.SliceIsSorted and failed later in the object layer with InvalidPart (HTTP 400). With strict ordering enforced up front, [0, 0] and [-1, -1] now fail immediately at the handler with InvalidPartOrder (HTTP 400). Both error codes map to HTTP 400 Bad Request and neither is retriable; a single out-of-range part ([0] or [-1]) still reaches the part lookup and returns InvalidPart.
Mutation Testing and a Misleading Comment
While verifying the unit tests in cmd/object-handlers-multipart_test.go, we ran two manual mutations against the handler:
- Mutating the check to require
Parts[0].PartNumber == 1: Immediately caught by the sparse-part regression tests ([3]and[5, 9]). - Deleting the
len(complMultipartUpload.Parts) == 0guard: Our draft test comment originally claimed that removing the empty-parts guard in the handler would allow an empty multipart completion to succeed. Running that mutation revealed something much more interesting: an emptyPartsslice bypassed the handler check, entered the object layer, and triggered an index-out-of-range panic inReadParts(cmd/storage-rest-client.go) inside an errgroup goroutine.
That second mutation served two purposes: we corrected the misleading comment in cmd/object-handlers-multipart_test.go, and we fixed ReadParts in commit 2602177ef (also shipped in Server 20260804) so that an empty partNumbers slice returns early instead of indexing partNumbers[0].
Declined Changes and Follow-Ups
- No retrospective repair tool: Existing objects assembled from duplicate part lists before
20260804have valid erasure metadata for their logical size; automatic truncation or deduplication would risk corrupting objects intentionally assembled from identical chunks. - No duplicate object-layer check: As noted in Handler vs. Object-Layer Boundary, internal callers construct part lists from validated metadata, and
2602177efcloses theReadPartsempty-slice boundary found during mutation testing.
Review Record
Three independent review passes verified the handler predicate, error-code precedence (InvalidPartOrder vs. InvalidPart), sparse part numbering ([3] and [5, 9]), and the ReadParts empty-slice guard (2602177ef).
Summary
Shipped in Server 20260804 (5df7b5466 and 2602177ef), CompleteMultipartUploadHandler enforces strict part-number monotonicity (PartNumber[i-1] < PartNumber[i]) and ReadParts safely rejects empty part-number slices.
4 - Design Records
Design records document the engineering decisions behind PGSTY SILO: root causes, S3 and IAM protocol invariants, trade-offs against rejected alternatives, operational limits, and regression coverage.
October 24: FULL_OBJECT Checksum Replication
- FULL_OBJECT Checksum Replication: Preserve the Type at Upload Initiation (
#254,#255): Why dropping the checksum type makes valid multipart CRC objects fail replication, how one shared header fix also restores the existing CopyObject path, and the source-upgrade and acceptance boundaries.
October 24: Multi-Pool Deletion and Placement
-
Multi-Pool Deletion: Preserve the Latest Copy on Failure (
#256,#257): Why an online-drive preflight cannot protect against later I/O failures, how staged cleanup preserves each item’s authoritative copy, and the retry, tier-reference, performance, and compatibility boundaries of the merged fix. -
Overwrite Pool Affinity: Why #245 Was Not Adopted (
#245): The original placement problem, request-aware quorum and health improvements, unversioned-only scope, two rounds of m3 performance measurements, and the decision to stop after small-object overwrite regressions remained. The experimental changes were not merged.
October 23 IAM & KMS Design Notes
- LDAP Long DNs: Restoring Login Without Hiding Revocation Errors (
#250): How\xxhex escaping inNormalizeDNtriggers the 255-byte single-component filename limit on the object IAM backend, and whyloadIAMConfigBytesWithMetadataclassifies only impossible pure-ASCII path components (> 255bytes) aserrConfigNotFound. - IAM Policy Reloads: Lock Scope, Cancellation, and Time Budgets (
#251): Moving policy reload and KMS decryption outside the authentication-cache lock while retainingwriteMu, threading request cancellation end-to-end toKES.Decrypt, and keepingetcd30-second RPC deadlines separate from total decryption budgets.
Server 20261018 Key Design Notes
For the Server 20261018 release, see these core architecture notes:
- Object Lock Governance & Overwrite Protection (
SN-2026-015,SN-2026-016) - IAM Admin Credential Authorization (
SN-2026-017) - Durable IAM & Scoped STS Revocations (
SN-2026-013) - LIST Recovery Lock Cost
- ListMultipartUploads Rollback & Capacity Boundaries
- Multi-Pool Consistency & Rebalance Protection
- KES & OpenBao Integration
4.1 - SILO, KES and OpenBao: Key Ownership and Recovery
SILO integrates with external key-management systems through PGSTY KES (MINIO_KMS_KES_ENDPOINT). The current release is KES 20261026 (built with Go 1.27.3 and updated cloud dependencies while retaining the API, configuration, and key formats described below); the earlier PGSTY KES 20261018 release (source 2f7e741db905) migrated the KES AWS Secrets Manager backend to AWS SDK for Go v2 while preserving existing SILO/KES key formats and OpenBao KV v2 compatibility.
Three services, three responsibilities
| Service | Owns | Calls |
|---|---|---|
| SILO | Object bytes and object encryption metadata; encrypts/decrypts object contents | KES over HTTPS (mTLS) to generate/decrypt data keys |
| PGSTY KES | Application identities, key-access policies, and the key API; wraps/unwraps data keys | OpenBao KV v2 to store/retrieve root keys |
| OpenBao | Persistent KES root-key records, backend access control, and its own recovery material | Its configured storage backend |
MINIO_KMS_KES_ENDPOINT must point to the KES endpoint because OpenBao does not implement the KES key-wrapping API. KES accesses OpenBao through its keystore.vault driver (version: v2 for OpenBao KV v2 mounts). The same kes executable provides both the server and the administration CLI, and multiple stateless KES instances can share a single OpenBao backend.
SILO stores the encrypted data encryption key (DEK) alongside the object metadata. Because the object bucket does not contain the external master key, losing the root key in OpenBao renders encrypted objects unrecoverable even when every object shard on disk is intact.
Minimal KES configuration for OpenBao KV v2:
Minimal OpenBao ACL policy for the KES AppRole on a KV v2 mount named secret with prefix kes:
Authentication and failure behavior
Configure TLS trust on SILO → KES and KES → OpenBao separately. KES authorizes the SILO client certificate identity (kes identity of <cert>) for the required key names; OpenBao authorizes the KES AppRole backend identity.
| Layer / Configuration | Failure mode | Operational resolution |
|---|---|---|
| OpenBao sealed or unreachable | Warm caches can allow some operations to continue temporarily, but uncached key lookups and fresh SILO restarts fail closed without falling back to plaintext storage. | Unseal OpenBao before starting or rolling SILO nodes, and verify recovery with a fresh process restart and actual object readback. |
| KV v1 vs KV v2 mismatch | Configuring version: v1 against a KV v2 mount (or omitting secret/data/... and secret/metadata/... capabilities in the OpenBao ACL policy) returns 403 permission denied or 404 on key creation and listing. |
Set engine: <mount>, version: v2, and grant ACL capabilities on both <mount>/data/<prefix>/* and <mount>/metadata/<prefix>/*. |
| Client certificate rotation | Rotating MINIO_KMS_KES_CERT_FILE changes the SHA-256 certificate identity. If the new identity is not authorized in the KES policy before rollout, SILO fails KMS startup checks. |
Authorize both old and new certificate identities in KES before rotating SILO client certificates, then remove the old identity after rollout completes. |
Keep persistent key-store backups, KES configuration, TLS trust roots, identity/policy definitions, and OpenBao recovery material under strict access control, and verify that backup restores preserve both key names and key material. Development mode keeps keys only in memory and must not be used for durable encrypted storage.
SDK migration and compatibility
PGSTY KES 20261018 (2f7e741db905) migrates keystore/aws/secretsmanager from the archived github.com/aws/aws-sdk-go (v1) to AWS SDK for Go v2 (github.com/aws/aws-sdk-go-v2, config, credentials, service/secretsmanager, and github.com/aws/smithy-go).
The migration preserves stored key formats and YAML configuration fields:
- Explicit credentials in the KES YAML configuration retain precedence; empty credential fields delegate to the standard SDK credential provider chain.
- KES does not load
~/.aws/configimplicitly, and the configuredendpointin KES configuration remains authoritative over SDK endpoint environment overrides. - AWS Secrets Manager
ResourceNotFoundException,ResourceExistsException, and invalid-parameter responses map to the same KES keystore errors (errKeyNotFound,errKeyExists).
The compatibility test matrix (testing/kms-compat.sh) exercises OpenBao KV v2 and Moto Secrets Manager across SDK-v1 -> SDK-v2 -> rollback transitions, restarting SILO on each transition so cached keys are discarded before readback verification.
Upgrade and recovery checks
- Record the SILO and KES versions, key names, backend endpoint, and client certificate identities. Retain existing keys and backend backups until recovery has been verified.
- Read an existing encrypted object before upgrading and record its SHA-256 digest. Upgrade KES to the target PGSTY KES 20261026 binary against the same persistent backend.
- Restart KES and SILO (clearing warm key caches), read the existing encrypted object, write a new encrypted object, and verify both digests using the deployment’s production TLS identities and access policies.
- In an isolated rehearsal, test restoring the prior KES binary against the same key store, restart both services, and repeat encrypted object readback.
For step-by-step configuration commands, see Configure KES and OpenBao for Server-Side Encryption, the KES reference configuration, and the KES compatibility test suite.
4.2 - Pinned TLS Parameters and Handshake Compatibility
After the Go TLS default repair 48e184652 shipped in Server 20260916, the reporter of issue #154 retested on that release and still saw connection reset by peer on OIDC discovery. Meanwhile, upstream Go issues reported the exact same symptom and traced it to TLS-inspecting middleboxes resetting connections over the signature_algorithms_cert extension (see The Known Upstream Instance). This article extends Go 1.27 TLS Defaults and OIDC Discovery Failure Modes with the post-release mechanism analysis, the scope of the 20260916 repair, the remediation options for affected deployments, and the resulting release and issue-closure gates.
In Plain Terms
A TLS connection opens with a ClientHello listing the algorithms the client supports. Many corporate networks place a middlebox in the path — a WAF, a TLS inspection appliance, or a load balancer — that parses that message. Some of those devices reset the connection when they encounter an unrecognized algorithm identifier.
Go 1.27 added three new identifiers to that message — the ML-DSA post-quantum signature schemes — for a total of 12 extra bytes. Rebuilt on Go 1.27, SILO sends those extra 12 bytes in the TLS 1.3 ClientHello. A middlebox in front of the reporter’s identity provider rejects the message and sends a TCP reset. curl from the same container succeeds because curl uses OpenSSL, which does not offer those identifiers.
Go provides a compatibility switch for hybrid key exchange, GODEBUG=tlsmlkem=0, which only applies when the application has not specified its own curve list (Config.CurvePreferences == nil). SILO carried an explicit curve list inherited from upstream that included a post-quantum entry, rendering GODEBUG=tlsmlkem=0 ineffective. Commit 48e184652 removes those explicit curve lists so GODEBUG=tlsmlkem=0 works as designed.
However, GODEBUG=tlsmlkem=0 disables post-quantum key exchange (ML-KEM); it does not disable post-quantum signature algorithms (ML-DSA), and the 12 extra bytes added in Go 1.27 are ML-DSA identifiers. Go exposes no GODEBUG switch for ML-DSA. When a middlebox resets on the ML-DSA offer, removing CurvePreferences alone cannot suppress those 12 bytes — matching the retest result.
What Go Is Actually Solving
Post-quantum key exchange requires advance migration. Under a harvest-now-decrypt-later threat model, an adversary records encrypted traffic today and decrypts it years later once a cryptographically relevant quantum computer exists. Confidentiality therefore has to migrate before such a machine exists. NIST finalized ML-KEM (FIPS 203) and ML-DSA (FIPS 204) in 2024, and the deployed form is hybrid: X25519MLKEM768 runs classical X25519 and ML-KEM together and combines both secrets, so a flaw in ML-KEM leaves the connection at least as strong as X25519 alone. Browsers, CDNs, and SSH implementations have enabled hybrid key exchange by default since 2024; Go followed in 1.24.
Handshake signatures have a different migration timeline. Because handshake signatures cannot be forged retroactively to break past sessions, ML-DSA in TLS 1.3 today simply advertises client support for ML-DSA certificates. Go 1.27 added MLDSA44, MLDSA65, and MLDSA87 to the default signature algorithm list, which accounts for the 12 extra bytes:
Those identifiers are defined for TLS 1.3 only. isDisabledSignatureAlgorithm drops them when the configuration cannot reach TLS 1.3, which — as noted below — is the only lever an application has over them.
The Switch That Was Never Meant to Cover Explicit Lists
The change that exposed SILO’s pinned curve list is documented in the Go 1.27 release notes:
Post-quantum hybrid key exchanges can now be explicitly enabled in
Config.CurvePreferenceseven if thetlsmlkem=0ortlssecpmlkem=0GODEBUG options are used. Those options were always meant to only apply to the default set used whenConfig.CurvePreferencesis nil.
The standard library has documented the two paths as alternatives since Go 1.24:
In code it is one branch: an explicit list is used verbatim; only the empty case consults the defaults, and the GODEBUG lives in the default path.
The principle is that explicit configuration outranks an environment variable — otherwise an operator could silently override a security policy the developer wrote in code. Go 1.26 and earlier filtered explicit lists through the GODEBUG as well; SILO’s behavior on Go 1.26 was therefore relying on something Go considers to have been a bug.
Go also ties GODEBUG defaults to the main module’s go directive, so a module that still declares an older version keeps the older behavior until it moves. SILO’s go.mod declares go 1.27.1, which explicitly adopts the 1.27 behavior.
Where SILO’s Debt Was — and Where It Still Is
The list in question was {X25519MLKEM768, CurveP256, X25519, CurveP384, CurveP521}, inherited from upstream and later extended with the post-quantum entry. Pinning it produced the worst of both worlds: the deployment could not benefit from the library’s evolving defaults, and could not use the library’s compatibility switch.
48e184652 removes that assignment at all eight Server TLS configuration points — the general external HTTP transport, the replication transport, the client-certificate cloud transport, the internode transport, both grid links, etcd, and the inbound S3/Console listener — retires the TLSCurveIDs helper, and adds wire-level regression tests across five outbound constructors, two peer TLS versions, and both GODEBUG settings. Certificate and hostname verification, cipher-suite policy, proxy handling, and HTTP/2 selection are unchanged, and no post-failure downgrade or retry was introduced. The earlier OIDC-only candidate was withdrawn because the same transport also serves identity plugins, notification and Lambda reachability checks, audit webhooks, and S3 cloud tiers.
The same debt remains one layer down. Cipher suites are still pinned: crypto.TLSCiphers() and crypto.TLSCiphersBackwardCompatible() are assigned in the outbound transports, LDAP, etcd, the grid links, and the inbound listener. The listener at least has MINIO_API_SECURE_CIPHERS to choose between the two sets; no outbound path has any switch at all.
The general rule this design establishes: set MinVersion and leave the rest to the standard library. Where an incompatible endpoint genuinely requires a narrower profile, expose it as product configuration that is visible in mcli admin config and in a support bundle — never as a value pinned in source. Protocol ossification is an old problem: TLS 1.3 had to disguise itself as TLS 1.2 on the wire, and GREASE exists precisely to force middleboxes to tolerate identifiers they do not know.
What Is Actually Different on the Wire
Measured against the same synthetic IdP fixture, with default settings:
| Build | ClientHello | ML-KEM (4588) | ML-DSA 0x0904-6 | User-Agent |
|---|---|---|---|---|
| 20260804, Go 1.26.5 — works | 1497 B | present | absent | MinIO (…) |
| 20260903, Go 1.27.1 — fails | 1509 B | present | present | Silo (…) |
| 20260916, Go 1.27.1 — repaired | 1509 B | present (default) | present | Silo (…) |
20260804 with tlsmlkem=0 |
275 B | absent | absent | MinIO (…) |
20260903 with tlsmlkem=0 |
1509 B | still present | present | Silo (…) |
Two readings matter and are easy to miss:
- ML-KEM was already being offered by the working release. Unless the operator had set
tlsmlkem=0before the upgrade, ML-KEM is not a variable between the working and failing versions, and restoring the switch cannot by itself explain or repair their failure. - Under default settings, exactly two things changed between the two releases: the three ML-DSA identifiers, and the HTTP User-Agent, which the rebrand changed from
MinIO (…)toSilo (…). The second only matters if the reset happens after the handshake completes.
Blast Radius and Why the Symptom Misleads
The pinned list covered both directions of every TLS path, but the consequences differ sharply:
| Path | Consequence of an incompatible ingress | Visibility |
|---|---|---|
| OIDC discovery / JWKS | Identity init blocks, Console never starts, the node serves nothing | Immediate, fatal |
| KMS/KES, LDAP(S) | Encryption or directory authentication unavailable. Both already used default curves, so the GODEBUG always worked for them | Immediate |
| Replication targets | Replication silently backs up; visible only in metrics | Easy to miss |
| Audit/notification webhooks, Lambda checks | Audit records lost, events undelivered | Easy to miss |
| S3 tiers / cloud backends | Transition failures, remote objects unreadable | Easy to miss |
| etcd, internode grid | Internal traffic, rarely crosses such a device | Low |
| Inbound S3/Console listener | Clients cannot connect; determined by the peer’s hello | Depends on client |
Only the identity path blocks startup, which is why it was reported first. The others degrade quietly, so a deployment can be affected without anyone filing an issue.
Which deployments are exposed at all:
| Situation | Why |
|---|---|
| An SSL-inspection NGFW, SASE, SWG, or IDS/IPS sits on the outbound path | The reported case, and the one with a confirmed vendor defect |
| Egress through an enterprise VPN that inspects TLS | Same mechanism, applied to the whole egress |
| The environment allow-lists JA3/JA4 client fingerprints | The fingerprint changes whenever the handshake changes, so every toolchain upgrade can trip it, with the same silent reset |
| An old TLS terminator or load balancer in front of the endpoint | Intolerant of unknown extensions, or mishandles a hello split across segments |
| Migrating from an upstream MinIO release built with Go ≤ 1.22 | That handshake carried neither post-quantum key exchange nor ML-DSA — roughly 275 bytes against roughly 1509. This cohort makes the largest single jump and is the most exposed |
| Regulated environments that disallow post-quantum algorithms | They need the classical profile for policy reasons, not interoperability ones |
Deployments whose outbound paths carry no inspection device are unaffected and should not set any of the compatibility options below.
Four properties combine to make the symptom nearly undiagnosable in the field: identity initialization retries at randomized 0–3 second intervals; the discovery and JWKS fetches have no total timeout, so startup can hang indefinitely; /minio/health/live and /minio/health/ready both stay 200 while identity is offline, so a Kubernetes readiness probe reports success (only /minio/health/cluster returns 503 with X-Minio-Server-Status: iam-offline); and the error text — read: connection reset by peer — does not say whether the TLS handshake completed. A curl control test from the same image then succeeds because curl uses a different TLS stack and negotiates HTTP/2.
What 20260916 Itself Changed
Removing a pinned list does not restore the previous wire format; it adopts the current Go default, which is larger:
| Release | supported_groups actually offered |
|---|---|
| 20260903 and earlier (pinned) | X25519MLKEM768, X25519, P256, P384, P521 |
| 20260916 (Go defaults) | X25519MLKEM768, SecP256r1MLKEM768, SecP384r1MLKEM1024, X25519, P256, P384, P521 |
For an operator who sets nothing, 20260916 therefore offers two additional post-quantum identifiers at every TLS endpoint, inbound and outbound. This is harmless for almost every deployment, but an ingress that allow-lists group identifiers could in principle be broken by 20260916 where 20260903 worked. GODEBUG=tlssecpmlkem=0 disables those two while retaining X25519MLKEM768. For an operator who does set tlsmlkem=0, 20260916 produces a smaller, more conservative handshake than any prior release — which is the point of the repair.
Three Mechanisms, One Discriminating Fact
The investigation reproduced three mechanisms that each produce the reported error text. The 20260916 repair addresses one of them.
| Branch | Mechanism | Covered by 20260916 |
|---|---|---|
| A | The ingress rejects ML-KEM, and the operator had tlsmlkem=0 set before the upgrade |
Yes — this is what the repair restores |
| B | The ingress rejects the ML-DSA signature identifiers | No. Only a TLS 1.2 cap suppresses the offer, and no such setting is exposed |
| C | An HTTP-layer rule rejects the changed Silo User-Agent after a successful handshake |
No, and no TLS change is relevant to it |
A fourth observation explains why the curl control test misleads: the discovery transport disables HTTP/2 and sends no ALPN, while curl negotiated h2. That difference exists in both the working and failing releases.
One fact separates A/B from C, and the fixture results separate B from A: whether the reset arrives immediately after the ClientHello or only after the GET is written. A single packet capture in the failing network position, or the ingress’s own reject reason for the same second, settles it.
The Known Upstream Instance
Branch B is not hypothetical. Two upstream Go issues describe it:
- golang/go#81199, “add a GODEBUG to disable advertising ML-DSA signature algorithms in the ClientHello (Go 1.27 regression against TLS-inspecting middleboxes)”, open since 2026-08-28. The reported symptoms are identical to #154:
read: connection reset by peeron every TLS 1.3 connection through a TLS-inspecting firewall, whilecurl, OpenSSL 3.6, and Node.js 24 succeed from the same host, a Go 1.26 build of the same program succeeds, andMaxVersion: tls.VersionTLS12works. The product named there is Palo Alto Prisma Access with PAN-OS 10.2.10-h37, and the Go team’s reply notes that a patched version is available. - golang/go#79626, closed, carries the diagnosis. A network administrator traced it with their own firewall team: a Palo Alto IDS/IPS threat signature for OpenSSL CVE-2020-1967 — reported there as PA threat ID 58033, last updated 2022-07-12 — inspects the
signature_algorithms_certextension and resets connections whose algorithm list it does not expect. Palo Alto shipped a content update disabling that signature on 2026-06-23.
CVE-2020-1967 was a null-pointer dereference reachable through a malformed signature_algorithms_cert; the vendor’s detector for it has been firing on legitimate modern handshakes. This also explains the 12-byte delta precisely: Go 1.27 adds three ML-DSA identifiers to both signature_algorithms and signature_algorithms_cert, which is 3 × 2 bytes in each of two extensions.
Go will not provide a switch. The Go security lead’s position on #81199 is that they cannot GODEBUG every low-level ClientHello change, that GODEBUGs are reserved for changes that alter negotiated parameters, and that advertising a signature algorithm is not supposed to change anything. The proposed tlsmldsa setting (golang/go#81307) has not landed, and Chrome is expected to start GREASEing signature_algorithms_cert, which will expose the remaining unpatched middleboxes even faster.
Two consequences follow for SILO:
- SILO cannot rely on an upstream
GODEBUGknob for ML-DSA: providing an explicit outbound TLS compatibility setting in SILO is required for deployments blocked on Branch B. - Identify the TLS-inspecting middlebox on the outbound path: applying the vendor’s threat-signature content update resolves the root cause directly at the firewall.
The Option Space
Grouped by who has to act. Each row solves specific branches:
Ingress side — the only complete fix.
| Action | Branches | Cost |
|---|---|---|
| Update the middlebox — threat-signature content as well as firmware — so it tolerates unknown algorithm identifiers | A, B, C | Requires the device’s owner. For the identified product a content update already exists, making this the shortest path |
Route around it: point MINIO_IDENTITY_OPENID_CONFIG_URL at an endpoint that does not traverse the device, or use split-horizon DNS. The discovery document’s issuer must not change |
A, B, C | Certificate hostname and issuer consistency must hold |
| Terminate TLS in a sidecar (stunnel, Envoy) that connects to the IdP itself | A, B, C | An extra component and its own trust chain |
Send the request through HTTPS_PROXY |
none | A CONNECT tunnel forwards the same ClientHello; only a proxy that terminates TLS changes anything |
Deployment side — available today.
| Action | Branches | Cost |
|---|---|---|
GODEBUG=tlsmlkem=0 |
A | Disables hybrid key exchange process-wide, including internode links and the inbound listener; certificate verification unaffected |
GODEBUG=tlssecpmlkem=0 |
the two groups added by 20260916 | Narrower; retains X25519MLKEM768 |
GODEBUG=fips140=on |
A and B | Not recommended. The FIPS allow-lists exclude ML-KEM and ML-DSA, but also replace cipher suites and curves and exclude Ed25519/X25519 for the entire process |
| Remain on 20260804 | A, B, C | Forfeits every subsequent security fix; emergency measure only |
Allow the Silo User-Agent at the ingress |
C | One rule change |
There is no way to disable ML-DSA via GODEBUG. Go provides no GODEBUG for signature algorithms — the crypto/tls entries in internal/godebugs/table.go are fips140ems, tlsmaxrsasize, tlsmlkem, tlssecpmlkem, and tlssha1 — and tls.Config has no signature-algorithm field. The only lever is the version gate: the identifiers are TLS 1.3-only, so capping MaxVersion at TLS 1.2 suppresses them.
Why the Toolchain Is Not Rolled Back
Rebuilding on Go 1.26 would restore the exact handshake of the working release, and upstream reports confirm that downgrading works. It is still the wrong instrument, for five reasons:
- It pays for a middlebox defect with everyone’s toolchain. The device at fault has a vendor fix available.
- It expires. Go supports the two most recent releases. Once Go 1.28 ships, a 1.26 build runs on a standard library that no longer receives security fixes, and the same decision returns with worse options.
- Upstream is keeping the new defaults. Go has declined to add a switch, and Chrome plans to GREASE the same extension.
- All four repositories require Go 1.27. Server, Console,
mcli, andsilo-pkgall declarego 1.27.1; building the current source with Go 1.26.7 is rejected by the module requirement, and transitive dependencies already require 1.27. - It only covers one branch. If the reset is caused by the HTTP-layer rule of branch C, the Go version is irrelevant.
Lowering the go directive in go.mod while compiling with Go 1.27 does not help either: that mechanism only governs behaviors controlled by a GODEBUG setting, and ML-DSA advertisement has none.
For an affected operator waiting on a firewall signature update, staying on the 20260804 release image is a valid temporary mitigation while the middlebox is patched.
Requirements Taken from This
- Phase-labeled diagnostics on the discovery and JWKS fetches. Record, via
httptrace, connection established → handshake started → handshake completed (with negotiated version, suite, and group) → request written → first response byte, and name the last phase reached in the returned error. No configuration, no protocol change, and it separates branches A, B, and C from a log line instead of a packet capture. - An explicit outbound TLS compatibility setting, covering at least the identity provider: classical curves, and optionally a TLS 1.2 cap. It must be opt-in, warn loudly at startup, and be visible in
mcli admin configand support bundles.MINIO_API_SECURE_CIPHERSis the existing precedent. This is the only in-process remedy for branch B. - Startup robustness: a total deadline and context cancellation for the discovery and JWKS fetches, and a decision on whether readiness should reflect identity state.
- A supported diagnostic subcommand derived from the investigation probe: handshake phases, negotiated parameters, and classical / TLS 1.2 / User-Agent controls, so an operator can localize this class of failure directly.
- Retire the remaining pinned cipher suites, or at minimum give outbound paths the switch the listener already has.
Two options are explicitly rejected:
- Shipping a lowered default — a
godebugdirective ingo.mod, orENV GODEBUG=…in the image — silently weakens post-quantum protection for every connection and still does not address branch B. - Automatic downgrade-and-retry after a reset hands an active attacker a trivial way to force TLS 1.2 with classical curves by injecting an RST; browsers removed this kind of fallback for that reason.
Release and Communication Gates
The repair’s compatibility requirement was documented in the Server README’s TLS section, in Go 1.27 TLS and OIDC, and in the 20260916 release notes. However, three things were missed during issue handling:
- The reporter was not told in the issue that the repair requires setting
GODEBUG=tlsmlkem=0to take effect, nor which branch it addresses. - The
GODEBUGrequirement was documented as a change note without being highlighted as an explicit operator action on the upgrade checklist. - The issue was closed when the PR merged, before the reporter confirmed a retest.
When a fix requires an operator-configured environment variable to take effect, it must be tracked through the same release-note and upgrade-checklist gates as a compatibility change:
- Any repair whose effect depends on an operator action is listed as a required action in the release notes and upgrade checklist, with the same treatment as a coordinated-upgrade item.
- An issue reported by an external user is closed only after that reporter confirms a retest; a merged patch changes status labels, not the issue state.
- When a repair does not cover every branch of a reported symptom, the issue explicitly states which branch was addressed, which branches remain, and what discriminating evidence is needed.
Attribution
This record extends the #154 investigation and the September Go 1.27 stack review with the post-release retest analysis, the upstream Go issue findings, the option space, and the process gates. Go behavior is grounded in the Go 1.27.1 standard library source and the official release notes.
4.3 - Bucket Configuration Replication: Source Times, Deletions, and Deterministic Convergence
#77 is a site-replication correctness defect repaired in Server 20260916 (PR #180, commit 48ec10312). Before this repair, a receiving site overwrote the peer’s source timestamp with local arrival time (UTCNow) and could then reject a genuinely newer deletion. In addition, several configuration types stopped exporting their timestamp once deleted, preventing heal from recovering a deletion missed during an outage. Adding a DELETE branch alone cannot solve both problems; convergence requires preserved source times, durable deletion tombstones, canonical comparison keys, and whole-record locking.
Existing Work and Scope
Earlier release notes, the 20260903 hardening record, and the Server compatibility page noted the #77 deletion limitation. This page records the state model, design trade-offs, and verification evidence (archived in docs/investigations/issue-77.md).
| Prior work | What it repaired | Remaining gap addressed by #77 |
|---|---|---|
| PR #91 | Per-site configuration counts, Policy/Quota reporting, malformed-field isolation | Accurate counts do not establish convergence of field values and source times |
| PR #103 | Serialized writes to the whole .metadata.bin record |
A write lock cannot validate an ordering decision made before acquiring the lock |
| #76, #78 | Object Lock replication payload and existing-bucket adoption protection | Does not provide a unified source-state comparison across all six configuration types |
| CORS replication repair | Dedicated per-bucket CORS deletion register and replication trust boundary | Specific to CORS; its semantics cannot be applied blindly to every metadata type |
This repair covers Policy, Tags, SSE, Quota, Versioning, and Object Lock. Lifecycle/expiry has its own merged-payload time semantics, and CORS retains its separate mechanism. Notification, IAM, object replication reliability, MRF, resync, and public counters are separate subsystems.
What the Reproduction Established
The original regression reproduced on both ErasureSD and 16-disk Erasure ObjectLayers. A peer PUT originated at a past time $T$, but the receiving disk stored its local arrival time. A subsequent DELETE carrying $T + 1\text{ minute}$ was rejected as older than the arrival timestamp. RPC success alone therefore did not produce the intended final state.
| Configuration | Pre-fix PUT preserves source time | Empty-event behavior | Pre-fix timestamp export when empty |
|---|---|---|---|
| Policy | No | Delete | Yes |
| Tags | No | Delete | No |
| SSE | No | Delete | No |
| Quota | No | Delete; zero-value JSON has separate semantics | No |
| Versioning | No | No operation | Not used as deletion |
| Object Lock | No | No operation | Not used as deletion |
Three further entry-point defects surfaced during investigation:
- An older bulk event could overwrite a newer individual field.
- A request that checked state before queuing for
metadata.lockcould act on an obsolete decision once it acquired the lock. - Remote Tag heal omitted
UpdatedAton outbound repair calls.
Each defect required a repair at its entry point in addition to fixing source selection in heal.
The Facts a Field Needs
The state model reuses the existing payload, field UpdatedAt, and bucket Created fields in .metadata.bin without adding new disk formats, SDK fields, or persisted deployment IDs.
| State | Conditions | Eligible source |
|---|---|---|
| Unknown or invalid | Unknown creation time, invalid payload, or field time before creation | No; missing information cannot mean deletion |
| Empty baseline | Empty payload at zero time or Created |
No |
| Live baseline | Valid nonempty payload at zero time or Created |
Yes, for historical configuration initialization |
| Real update | Valid nonempty payload later than Created |
Yes |
| Real deletion | Empty payload for a deletable field, later than Created |
Yes; this timestamped empty value is a tombstone |
Empty Versioning and Object Lock events remain no-ops; sharing an ordering helper must not give those two types deletion semantics. Zero field times use Created as their comparison baseline without turning historical emptiness into a deletion.
Ordering applies four deterministic rules:
- Real states (
UpdatedAt > Created) outrank baselines. - Between real states, the later source
UpdatedAtwins. - At equal timestamps, a deletion tombstone outranks a live value.
- Conflicting live values at the same rank and timestamp resolve by comparing a canonical content key, with the lexicographically greater key winning.
For timestamped peer events and heal, an identical effective state triggers neither a disk save nor an event notification. A local write always allocates a new monotonic revision even when the visible payload is unchanged. Live baselines also use content-key ordering so a later creation default cannot outrank a real modification. Lexicographic tie-breaking guarantees deterministic convergence across sites; after conflicting concurrent edits, operators should still verify and resubmit the intended business configuration.
Comparison Must Match Persistence
Replication status, heal source selection, and peer apply must compare the exact representation that persistence writes:
- Policy: Policy sets are map-backed in Go, so ordinary JSON encoding depends on map iteration order. The server recursively sorts the complete JSON tree of a validated policy—including
Statement,Action/NotAction,Resource/NotResource,Principal, andCondition—while retainingSidand numeric precision. Because the policy parser acceptsNotActionandNotResourcewhile the legacy struct’s requiredAction/Resourcemarshaling could fail on empty sets, the server uses explicit field encoding acrossGetBucketPolicy, admin export, and peer replication. This prevents accepted negative-set policies from failing on subsequent reads or replication and ensures replication status and heal compare identical keys. - Quota: Quota keys use the parsed quota configuration encoded as JSON.
{}, JSONnull, and valid zero-quota documents remain live documents rather than implicit deletions. Older senders converted a zero-quotaPUTinto a peer deletion while retaining a local document; withmcli quota clearsending a zero quota, upgraded peers retain that live zero document with capacity enforcement disabled on all sites. An empty Policy payload instead follows the established peer deletion rule: a valid empty-policyPUTclears the policy, and a subsequentGETreturnsNoSuchBucketPolicy. - XML configurations (Tags, SSE, Versioning, Object Lock): XML types compare the validated document bytes directly without a general XML canonicalizer. Versioning requires one specific rule: apply the existing Object Lock constraint (
Enabledrequired when Object Lock is active) before comparing the effective document. Otherwise, the comparator would accept a Versioning payload thatSaveimmediately rewrites, causing heal to resend it on every cycle.
Lock the Decision as Well as the Save
All six fields share a single .metadata.bin record. Loading raw state, validating payloads, comparing timestamps, mutating fields, and saving to disk must all execute inside the existing metadata.lock. Comparing outside the lock permits stale decisions under concurrent writes, while splitting metadata.lock into per-field locks would allow whole-record read-modify-write operations to overwrite one another.
A local write allocates its revision timestamp inside metadata.lock:
This monotonic rule lets a local operator correction advance beyond a previously stored future field timestamp. Timestamped peer events retain their originating UpdatedAt time, and identical or older timestamped peer states return early inside the lock without writing to disk.
Bulk processing validates all explicitly supplied fields and saves at most once. Omitting a field preserves its current state; an explicit null follows that type’s deletion semantics, and any invalid field aborts the entire bulk update before persistence. Import assigns a single shared commit timestamp for the selected fields under the final per-bucket commit lock. Both disk persistence and outbound replication events use the normalized snapshot returned by that commit rather than re-reading storage after unlocking. Normalized empty Policies emit the dedicated policy deletion event so bulk JSON omitempty tags cannot drop them.
Adoption Must Not Manufacture a Deletion
Bucket adoption can change Created. Moving Created earlier while retaining an empty field’s old default timestamp would make an empty baseline look like a real deletion (UpdatedAt > Created). Moving Created later could invalidate historical initial values (UpdatedAt < Created).
Under the existing adoption lock, the server rebases only those six fields whose previous timestamp was zero or equal to the old Created timestamp. Real update and deletion timestamps remain untouched.
One Rule Across Entry Points
| Entry point | Required behavior |
|---|---|
| Local S3/Admin writes | Allocate monotonic UpdatedAt under metadata.lock; build outbound events from the committed snapshot |
| Typed peer events | Compare and persist original source UpdatedAt under metadata.lock; retain legacy Object Lock payload compatibility |
| Bulk/import | Track explicit field presence, validate all fields, and allocate the import timestamp under the final commit lock |
| Initial synchronization | Recover missing Created from physical disk probe; preserve live baselines; include real deletions when tombstones are enabled |
| Local/remote heal | Filter valid candidates before selecting the maximum; compare canonical keys; isolate failed peers |
| Status export | Pair value and UpdatedAt from one snapshot; gate newly exposed deletion timestamps behind the rollout switch |
Initial synchronization retains its existing five-type send path, while Versioning is initialized through MakeBucketHook and aligned through heal. When a bucket has no recorded Created time in .metadata.bin, GetBucketInfo returns the physical directory probe (matching ListBuckets), and initial synchronization passes that recovered timestamp to the bucket creation hook so configuration writes do not fail on metadata-free buckets.
Heal filters valid candidates before selecting the maximum state rather than seeding the winner from the first map entry and filtering defaults afterward. Public mismatch counters are not used as the sole heal gate: identical content with different source timestamps still requires synchronization, whereas identical effective states require neither a disk write nor an RPC.
Why Rollout Needs a Default-Off Switch
The startup environment variable MINIO_SITE_REPLICATION_METADATA_TOMBSTONES defaults to off. It controls whether newly exposed tombstone timestamps for absent Tags, SSE, and Quota payloads are exported in status and initial sync.
| Behavior | off (default) |
on |
|---|---|---|
Source-time ordering and atomic apply under metadata.lock |
Active | Active |
| Live deletion RPC events | Replicated | Replicated |
| Existing Policy deletion-time export | Preserved | Preserved |
| Real deletion-time export for absent Tags/SSE/Quota | Hidden | Exported |
| Real deletions included in initial synchronization | Policy only | Policy, Tags, SSE, and Quota |
Unupgraded peers cannot safely consume all newly exposed deletion timestamps. For example, legacy Quota heal clears the raw payload while retaining an already parsed in-memory cache value. Keeping the switch off by default isolates rolling upgrades from that legacy behavior.
Coordinated Upgrade and Rollback
- Upgrade every node at every participating site to Server 20260916 or later, verify consistent settings within each site, and drain in-flight requests.
- Set
MINIO_SITE_REPLICATION_METADATA_TOMBSTONES=onacross all sites and restart. - Before rolling back to an earlier release, set
MINIO_SITE_REPLICATION_METADATA_TOMBSTONES=offon every node, restart, and only then downgrade the binary.
While the switch is off, hidden Tags, SSE, and Quota tombstones can cause peers to send stale heal RPCs that upgraded receivers reject. Steady-state heal with zero metadata RPCs is reached once on is enabled across all sites.
Kept and Rejected Alternatives
| Decision | Engineering reason |
|---|---|
| Retain one internal six-field ordering helper | All six types suffered the same arrival-time defect; one shared comparator prevents entry-point drift while preserving per-type deletion rules |
| Keep the whole-bucket lock, persistence fields, and heal interval | Provides atomicity, durable tombstone state, and missed-event recovery without introducing an external coordination service |
Do more than replace UTCNow with source time |
Replacing the timestamp alone leaves lock-external decision races, invisible deletes, equal-time ties, and initial-sync gaps unresolved |
| Gate newly exported tombstone times behind a switch | Legacy receivers and legacy Quota cache handling remain unsafe during a mixed-version rolling upgrade |
| Do not use deployment ID as a tie-breaker or add an HLC/schema | Existing timestamps and canonical content keys suffice for deterministic convergence without changing on-disk schemas |
| Do not reject all zero-time typed events | Legacy Tag heal omits UpdatedAt; accepting zero-time events with local monotonic time preserves backward wire compatibility |
| Do not impose one deletion rule across all bucket metadata | Versioning and Object Lock cannot be deleted, while Lifecycle and CORS have distinct merge and deletion structures |
The production change adds 729 lines and removes 692 lines of Go (+37 net lines), replacing duplicated per-type apply and heal branches with the shared state comparator. Every retained mechanism maps to a reproduced failure mode: removing whole-record locking reopens read-modify-write races, removing canonical Policy keys reintroduces permanent status/heal mismatches on negative-set policies, and removing the rollout switch exposes unupgraded Quota caches during rolling upgrades.
Validation and Test Evidence
Regression suites in cmd/site-replication-metadata_test.go, cmd/site-replication-metadata-heal_test.go, and cmd/site-replication-metadata-gate_test.go verify the contract across both ErasureSD and 16-disk Erasure backends:
| Test scope | Verified behavior |
|---|---|
| Source time, four deletable types, duplicate/reordered events, pre-lock queuing, cross-field writes | Regressions and -race checks pass on both ObjectLayers |
Equal-time arrival in both orders, deletion priority over live values, canonical Policy keys, negative-set GET, bulk/import atomicity |
Deterministic convergence regardless of event arrival order |
| Creation-time recovery and legacy-order Policy status | Real-drive ObjectLayer tests stamp bucket directory mtimes and verify that metadata-free buckets recover Created and converge without false status mismatches |
| Two-site process experiments (four drives per site) | Initial sync preserves Created across all six types; dropping an outbound PUT RPC and injecting reordered events converges on the next 30-second heal cycle |
| Missed deletions across source restart | Tombstones for Policy, Tags, SSE, and Quota survive source process restart and converge after reconnection |
| Steady-state quiescence and diagnostic deduplication | Two consecutive 65-second windows emit zero metadata RPCs after convergence; exceptional warnings deduplicate by bucket, field, and reason |
During test review, an early creation-time recovery test that used a stubbed ObjectLayer probe was replaced with a real-drive test that stamps bucket directories on disk—ensuring the test fails against unfixed GetBucketInfo code and passes only with the production repair.
Implementation Review Progression
Across the review sequence from baseline 5c5765816 and initial implementation 1ee64a8d8, three review findings (creation-time recovery on disk, legacy-order Policy status keys, and heal diagnostic deduplication) were fixed in 62cf066ff and fcbb93e89, with final test formatting and lint validation completed at 461e9a721.
Operational Boundaries That Remain
- Historical timestamp pollution cannot be reconstructed automatically. Arrival-time overwrites and untimestamped legacy events have already lost their original source timestamps. After upgrading, inspect divergent buckets and resubmit the intended configuration or deletion at the authoritative site.
- Zero-time typed events remain compatible. Untimestamped typed peer events allocate monotonic local time and emit a rate-limited
legacy-zerowarning; bulk imports still require valid timestamps. - Bucket generation conflicts are not merged automatically. Events with
UpdatedAt < Createdare skipped (before-created). Missing recordedCreatedtimestamps are approximated from the bucket directory modification time on disk during configuration writes or initial sync. If physical creation time is still zero for an existing bucket, writes return500 InternalError(404 NoSuchBucketwhen the bucket is absent). Until a write or initial sync persists the recoveredCreatedvalue, periodic heal skips the bucket in both directions. - Physical clocks are not causal clocks. Monotonic local allocation allows an operator correction to advance past a future field timestamp, but concurrent conflicting writes across sites still require operator review if business intent differs from lexicographic tie-breaking.
- Diagnostics are bounded per bucket, field, and reason.
legacy-zero,before-created,indeterminate,unreachable, andpeer-errorreuseLogOnceIfat warning level with hourly cleanup. Empty baselines and peers that do not yet have the bucket stay silent, while invalid stored payloads emitindeterminatewithout triggering outbound RPCs.
Related records: Replicated Tag Ordering · Replica Metadata Normalization · Request Header Timeouts · Replica Metadata Audit
4.4 - SigV4 Signed-Header Coverage: Rejecting Unsigned x-amz-* Headers
This article records the unsigned-header coverage repair committed to SILO as 123325430 in PR #173 (tracked as SN-2026-011) and the follow-up signing and payload-verification fixes in PR #177 (tracked as SN-2026-012), which shipped in Server 20260916. The vulnerability was reported by Oren Yomtov and reproduced locally on both presigned and Authorization-header SigV4 paths.
Summary (TL;DR)
A presigned PUT URL can sign only the host header. SILO confirmed that each header listed in SignedHeaders had arrived, but never inspected the full set of headers present on the incoming request, allowing x-amz-* headers outside SignedHeaders to pass through untouched. Meanwhile, cmd/api-router.go routes any PUT carrying x-amz-copy-source to CopyObjectHandler based on that header alone. Combined, these two behaviors allowed a caller to turn a single-object write grant into a server-side copy that reads any object reachable by the signing key, executed under the signer’s authority — a confused deputy. The Authorization-header path behaved the same way whenever x-amz-copy-source was omitted from SignedHeaders.
The repair enforces one invariant:
This matches AWS S3, which rejects the same request with AccessDenied (“There were headers present in the request which were not signed”).
Failure: The One-Way Coverage Gap
extractSignedHeaders in cmd/signature-v4-utils.go iterates over the signed-headers list and pulls each named header from the request (or query string). That verifies that every promised header is present, but omits the converse check: whether every arriving x-amz-* header is included in SignedHeaders.
The only routine that inspected arriving headers, checkMetaHeaders, matched only the X-Amz-Meta- prefix and ran only on the presigned path (doesPresignedSignatureMatch). The Authorization-header verifier (doesSignatureMatch) had no equivalent check. As a result, an unsigned x-amz-copy-source — or any other operation-shaping x-amz-* header — passed through on both paths:
In local reproduction, the control PUT returns 200 with an empty body; the same presigned URL with an appended unsigned x-amz-copy-source header also returns 200, returning a CopyObjectResult whose ETag matches the victim object’s MD5 and writing the victim’s bytes into the destination object. If the destination bucket permits anonymous GetObject, the copied private object becomes readable without credentials.
Provenance
This gap was inherited from upstream MinIO. The SigV4 verifier in cmd/signature-v4-utils.go and doesSignatureMatch in cmd/signature-v4.go date back to 2016, and the header-driven CopyObject router dispatch in cmd/api-router.go dates to 2019. Upstream added checkMetaHeaders on 2023-07-27 in minio/minio#17737 (535f97ba6), recognizing that arriving headers must be checked against the signed set, but scoped that check to the X-Amz-Meta- prefix on the presigned path — leaving x-amz-copy-source and the entire Authorization-header path uncovered.
Prior to this fix, SILO’s only modification to cmd/signature-v4-utils.go was a single-line import path update in 9b11dc946 (policy moved to pgsty/silo-pkg/v3). Commit 123325430 and the follow-ups in PR #177 close the boundary across both paths.
The Repair
checkMetaHeaders is renamed to checkUnsignedHeaders, widened from X-Amz-Meta- to the full X-Amz- prefix, and invoked on both the presigned and Authorization-header paths. Any arriving x-amz-* header missing from the signed set is rejected with ErrUnsignedHeaders before handler logic runs.
Four design decisions define the exact boundary:
Set Membership Lookup
The inherited checkMetaHeaders function evaluated signedHeadersMap.Get(k) == val[0]. For a header absent from signedHeadersMap, Get returns "", so an unsigned multi-value header whose first element is empty — such as X-Amz-Copy-Source: ["", "/src/secret"] — evaluated equal and bypassed the check. checkUnsignedHeaders checks key membership in signedHeadersMap directly; once a header name is in the signed set, its values are already bound by the signature digest.
Exempt X-Amz-Content-Sha256
X-Amz-Content-Sha256 may be omitted from SignedHeaders because the effective payload hash is bound separately in the SigV4 canonical request. On presigned requests, the query parameter takes precedence and falls back to the header only when the query parameter is absent; an explicit UNSIGNED-PAYLOAD remains valid. PR #177 aligns policy-condition evaluation with this effective payload hash while preserving header-presence semantics, and enforces body-hash verification for header-only presigned requests in the generic authentication path as well as upload paths. Signature verification, policy evaluation, and payload verification all consume the same bound value.
Derive Signature Age from the Signed Date
The initial fix exempted an internal x-amz-signature-age scratch header written after signature verification. However, PUT and UploadPart evaluate authorization before signature verification, before that scratch header existed. PR #177 computes s3:signatureAge directly from the signed X-Amz-Date value and removes the scratch header, its constant, and its exemption. A tampered date fails signature verification, an unsigned client-supplied x-amz-signature-age header is rejected, and verification remains idempotent without mutating request headers.
Inject X-Amz-Tagging After Signature Verification
PutObjectTaggingHandler synthesizes an X-Amz-Tagging header from the XML request body so that IAM policy conditions can inspect object tags, and previously injected that header before authenticateRequest. Under the widened X-Amz- check, that server-synthesized header would be rejected as unsigned. Moving the injection to run after signature verification and before policy authorization preserves tag condition evaluation while keeping signature verification strict. We rejected the alternative of blanket-exempting X-Amz-Tagging, which would have allowed clients to attach unsigned X-Amz-Tagging headers to arbitrary signed or presigned write requests.
Scope Across Signature Modes
Authorization-header (signed) and presigned SigV4: both enforcecheckUnsignedHeaders.- Streaming SigV4: the seed verifier does not call
checkUnsignedHeaders. Copy handlers reject streaming authentication through their standard authentication dispatch, while streamingPUTandUploadPartheader coverage remains a separate boundary. - SigV2: unaffected. V2 canonicalization folds all
x-amz-*headers into the string-to-sign by construction, so any addedx-amz-*header alters the expected signature and fails verification.
Status Code: 400 vs. 403
AWS S3 returns 403 Forbidden for an unsigned header, whereas SILO returns 400 AccessDenied (ErrUnsignedHeaders), inherited from upstream. Both reject the request with the AccessDenied S3 error code; only the HTTP status code differs. Changing ErrUnsignedHeaders to 403 in cmd/api-errors.go is a one-line change, but it would also alter the HTTP status for the existing unsigned x-amz-meta-* rejection path, so 400 AccessDenied is retained for compatibility.
Tests
Several existing unit tests constructed a signed request and then attached x-amz-copy-source, x-amz-copy-source-range, or x-amz-metadata-directive after signing. Those tests now re-sign with signRequestV4 after setting operation-shaping headers (signRequestV4 excludes Authorization from its own signed set, making re-signing safe). Unit tests cover checkUnsignedHeaders against empty first values, the X-Amz-Content-Sha256 exemption, rejection of unsigned x-amz-signature-age headers, and TestPresignedVerifyIdempotent, which verifies the same presigned request twice.
Evidence
- Against a built server instance, the reproduction harness confirmed the confused-deputy copy on both presigned and
Authorization-header paths before the fix, and confirmed that both are rejected after the fix while controlPUTrequests,minio-goCopyObject,PutObjectwith user metadata and tags, and body-basedPutObjectTaggingcontinue to succeed. go test ./cmd/passes cleanly, along withgofmt,gofumpt, andgo vet.- First-round code review identified three issues in the initial draft — non-idempotent verification via the scratch header, the empty-first-value bypass, and over-rejection of unsigned
x-amz-content-sha256— all of which are resolved and covered by regression tests. - Second-round review confirmed that the re-signed tests preserve their original assertions: an invalid access key still returns
InvalidAccessKeyId, and a mismatched SSE-C key still returns403after signature verification.
Compatibility and Operations
- Standard clients: unaffected. Conforming SigV4 signers include all non-exempt
x-amz-*headers they send inSignedHeaders. - Unsigned
x-amz-*headers: rejected withAccessDenied, matching AWS S3. - Rolling upgrade: wire and on-disk storage formats are unchanged. Upgraded nodes enforce the check immediately, while unupgraded nodes remain exposed until replaced during a rolling upgrade.
- Rollback: data written under the patched version remains readable by older binaries, though rolling back reopens the confused-deputy vulnerability.
Related Fixes and General Principle
- Advisory identifiers: the unsigned-header repair is tracked as
SN-2026-011, and the companion payload-verification and signing fixes in PR #177 are tracked asSN-2026-012. - Status code alignment: the
400-versus-403HTTP status question remains open for a future major release. - Adjacent signing fixes: PR #177 resolves duplicate
x-amz-copy-sourceheader ambiguity, fixes signature-age authorization ordering and effective payload-hash policy evaluation, and closes the header-only presigned body-checksum gap. Regression tests cover signed and presigned requests, policy evaluation before upload verification, and live HTTP bucket-policy tampering. - General principle: any request element that selects or shapes an operation must be covered by the signature before routing or handler logic acts on it, and the value bound by the signature must be identical to the value consumed by the handler.
Conclusion
SigV4 verification must verify both that every header listed in SignedHeaders is present and that every arriving x-amz-* header is listed in SignedHeaders. Rejecting unsigned x-amz-* headers before handler dispatch closes the confused-deputy boundary on both signed and presigned SigV4 paths. While this signing fix does not alter storage formats, upgrading to Server 20260916 also includes IAM changes requiring a coordinated upgrade.
4.5 - Durable IAM Revocations
SILO retains the revision of every deleted IAM record so an offline site cannot resurrect a deleted identity or policy grant when it reconnects. This mechanism covers built-in users, service accounts, groups, policy documents, and policy mappings across the user, STS-parent, and group namespaces. When a built-in user is deleted, older service accounts, STS temporary credentials, and group memberships remain revoked even if an administrator later recreates a built-in user with the same name.
The persistent IAM revocation architecture (PR #191, #192, commit f99ed829b5eba549160725f035156c9e020b6a07) shipped in Server 20260916 to close SN-2026-013; the scoped STS revocation extension for direct RevokeTokens operations (0cd388c66) shipped in Server 20261018. For multi-site rollout procedures, see the IAM upgrade and recovery runbook.
October 18: Scoped STS Revocation
Commit 0cd388c66 in Server 20261018 extends the September durable revocation mechanism to direct RevokeTokens operations.
In earlier releases, a node could miss an issuance notification and revoke only the tokens present in its local memory cache. With this fix, RevokeTokens reads persisted STS identity records, merges them with in-memory entries, filters by the target parent identity and optional token type at the operation cutoff, and commits a dedicated tombstone for each matching immutable token key. Even if one key in the batch fails to save, all committed keys are immediately notified to sibling nodes and replicated to peer sites while returning the save error to the caller for retry.
Private IAM replication messages carry the token deletion marker and expiration time. A revocation message that arrives before its corresponding creation message still blocks subsequent replay, and a creation message with a newer event timestamp cannot resurrect an immutable token. Existing deletion healing retries offline delivery. Full IAM refreshes on both object storage and etcd rebuild STS revision records and evict revoked entries from the active cache even if concurrent IAM writes defer a full cache swap. Tombstones are retained until the token’s natural expiration plus clock-skew allowance, or retained conservatively when expiration is unknown.
A single revocation snapshot covers STS identities known to the requesting cluster prior to the cutoff; tokens issued at an isolated site and never replicated into this cluster fall outside that snapshot. Keys issued after the cutoff, unrelated users, and reusable parent-policy mappings are unaffected. All participating nodes must be upgraded together because older binaries do not implement the private STS replication message.
Execution cost: An administrative revocation enumerates persisted STS identities and filters by parent and token type, so read volume and temporary map allocations scale linearly with active token count. The scan releases the authentication-cache lock while holding the IAM write serialization lock (writeMu), so concurrent IAM admin writes may briefly wait. etcd’s periodic refresh also reads STS records. Ordinary cached S3 authentication incurs zero extra network calls, and no new background timers or external dependencies are introduced.
Regression tests in iam-sts-revocation_test.go cover three-site outage/restart recovery and four processes sharing IAM storage. After upgrading, verify old-token rejection across endpoints, continued validity of newly issued tokens, and state recovery across restarts as detailed in the upgrade procedure.
Ordering and Persistence
Deletion records are stored at the original IAM configuration paths and contain the originating timestamp, Deleted, and any required RevokedBefore boundary; identity deletion records never store secret keys or session tokens. Normal IAM listings and policy evaluation hide deleted records automatically. Both object storage and etcd serialize version comparisons and writes per path using a distributed lock, persisting the originating source timestamp directly rather than overwriting it with the receiving node’s local clock.
Revision ordering follows five deterministic rules:
- Older events never overwrite a newer revision; at an identical timestamp, a deletion wins over a live record.
- A deliberate local same-name recreation receives a revision newer than the stored deletion tombstone.
- If an older user or group deletion message arrives late after a same-name recreation, the server updates and retains its
RevokedBeforeboundary while keeping the newer recreated live record intact. - Receiving an already-applied tombstone is a no-op that neither rewrites the record nor advances its revision.
- Cold-loading persisted IAM state on startup applies the exact same merge rules.
Writing the user or group revision is the authoritative commit point of a deletion; cleanup of policy mappings and child credentials follows afterward. A failure during dependent cleanup never rolls back the committed parent revocation: the API returns the cleanup error to the caller while still notifying sibling nodes on the same backend to reload the committed revocation state. Consequently, an HTTP 500 cleanup error on admin delete does not mean the old identity remains active. Administrators can safely retry the deletion while keeping IAM writes and same-name recreation paused. When dependent cleanup fails after commit, the admin handler skips the immediate cross-site push hook and lets the background deletion-healing loop propagate the tombstone across sites. Sibling nodes reload current shared state upon notification, so a delayed notification cannot delete a subsequently recreated identity; etcd siblings also receive persisted changes via watches.
Operators must keep node clocks synchronized via NTP and monitor clock skew. Revision ordering relies on source wall-clock timestamps with monotonic advancement for consecutive local writes to the same path, without establishing cross-site causal ordering for concurrent writes.
Users, Child Credentials, and Groups
A recreated user retains its historical RevokedBefore boundary. Newly created service accounts and built-in STS credentials carry a cryptographically signed siloParentRevocation claim recording the parent revocation boundary known at issuance. Editing or replaying an old child credential never updates this signed claim; thus old children remain invalid even if their own UpdatedAt timestamp is newer than the parent deletion, whereas children minted for the recreated parent work normally. The claim is extracted from verified tokens during credential load and write.
In site replication, a service-account snapshot with a newer revision can replace an older service account with the same AccessKey (including deliberate owner changes or secret rotations), while local duplicate AccessKey creation remains rejected. Replicated snapshots preserve the Disabled status and the service account’s own revocation boundary so older policy mappings cannot reattach to the replacement service account. Equal-revision retries reload the committed identity without rewriting storage. Periodic live healing compares source revisions directly—even when the public status summary is unchanged—and includes disabled identities as healing sources.
Service-account snapshots also preserve their absolute expiration timestamp instead of reapplying the receiving site’s minimum lifetime for newly minted credentials. A newer but already-expired snapshot still supersedes the old key, is denied by authentication, and is collected into a durable service-account tombstone during normal loading. If cache or claim loading fails, the receiver returns an error to trigger retry rather than acknowledging prematurely; once a replacement commits, the stale cached secret is evicted immediately. If a replicated snapshot collides with an existing built-in user or cached STS identity of a different credential kind, the server rejects the type conversion and logs an error for explicit administrative resolution.
Each group member carries an individual MemberGrants timestamp. Adding or removing other members or toggling the group’s enabled status does not refresh existing members’ grant timestamps. A membership is effective only when its MemberGrants timestamp is newer than both the user’s and the group’s retained RevokedBefore boundaries; both group listings and IAM policy evaluation enforce the same effective-membership check. Peer replication snapshots preserve original grant timestamps (including unknown legacy timestamps) rather than treating a recent snapshot sync time as a fresh grant to a revoked user; only a new explicit administrative group grant restores group access.
The inherited live-group snapshot merge adds members incrementally and does not automatically reconcile an ordinary member removal missed during a site outage; the mechanism here governs user/group deletion boundaries and isolation across same-name recreation.
Retention and Expiration Reclamation
Tombstones for permanent identities (built-in users and service accounts), groups, policy documents, and policy mappings have no automatic TTL. Disconnected remote sites or cold backups may reconnect after arbitrary intervals; peer replay acknowledgements suppress redundant network sends but never trigger physical garbage collection of permanent tombstones.
When an immutable STS temporary token expires naturally, the server physically removes its token-key record and any legacy mapping without creating a permanent tombstone. Conversely, an explicitly revoked STS token retains its tombstone until the token’s original expiration plus clock-skew allowance, blocking any replay carrying a later event timestamp. The etcd backend uses native expiration leases, while the object-storage backend reclaims expired STS tombstones during its existing credential load and purge cycles. Records with unknown expiration are retained conservatively.
Optional expiration-reclamation writes run best-effort under a short lock timeout budget: if a cleanup write fails, that load cycle halts optional reclamation, logs the error, and continues loading active users while leaving expired credentials denied under their existing durable revision until the next cycle retries. External IdP identity disablement is treated as an early revocation rather than natural expiration, so its cached STS tokens and service accounts are included in cleanup. Expiring service accounts retain a durable revision record because their AccessKey can be updated and reused and an earlier version might have had no expiration.
In the September 20260916 release, deleting a built-in parent user invalidated all older child credentials via the persisted parent boundary (including children absent from the deleting node’s cache); cross-site per-token sync for direct RevokeTokens calls was completed by the October 18 scoped STS extension.
Healing Loop, Failure Handling, and Operational Cost
Each server process starts a single background healing loop. Losing its distributed leader lease pauses the loop until leadership is reacquired rather than terminating it permanently, and configuration hot-reloads do not spawn duplicate loops. A 30-second interval separates leadership acquisition and each completed healing pass.
Normal IAM loaders maintain an in-memory index of deletion records and revocation boundaries (without secrets). The healing loop reads this in-memory index directly, avoiding a repeated full disk walk of config/iam/ every 30 seconds; persistent storage records (including tombstones) are scanned only on process startup and full IAM refresh.
Healing sends batches of at most 128 records per peer. The sender tracks the highest revision acknowledged per path and peer, and unrelated mutations on either site do not reset existing acknowledgement progress. If a batch fails, it stays pending for retry while subsequent independent batches proceed; a lost response simply triggers a safe idempotent replay. Each pass has a bounded total duration, preserving acknowledgements received before timeout. The sync protocol reports both node name and process instance ID: load-balancer switching across known instances preserves acknowledgements, whereas a newly started process instance conservatively invalidates prior acknowledgements once to re-verify state. Always stop affected server processes before restoring persistent IAM state from backup so a restored node cannot reuse stale in-memory acknowledgements.
In steady state, healing iterates and sorts the retained in-memory set, checks peer protocol status, and suppresses repeated deletion PUT requests once acknowledged. Memory usage scales linearly with retained path count and peer count, while cold-start storage reads scale with total historical records. On first upgrade, existing live records that previously received different local timestamps on different receiving sites will trigger a one-time revision-alignment wave; factor those storage writes and sibling notifications into maintenance window planning.
Cached credential lookups check only the in-memory parent-revision index with zero extra storage I/O, while STS issuance and cold credential loads check the persisted parent revision. Revision I/O and distributed lock waits release the IAM cache read/write lock while a dedicated local writer mutex (writeMu) preserves per-node write ordering under bounded timeout contexts (see IAM Policy Reloads: Lock Scope, Cancellation, and Time Budgets).
Replication Protocol and Coordinated Upgrade
The versioned inter-site replication endpoint /minio/admin/v3/site-replication/peer/iam-revisions transports source revisions, member grant timestamps, and user/group revocation items without altering public Admin client SDKs or S3 APIs. Older server binaries reject this route, and the sender reports the sync failure directly rather than falling back to legacy routes that would strip revision metadata. Existing legacy IAM routes remain readable for backward compatibility.
All participating nodes—including nodes that share an IAM storage backend without site replication—must be upgraded together; mixed old/new binaries on the same backend and rolling downgrades are unsupported because older binaries do not understand tombstones or signed parent boundaries. Execute coordinated upgrades during a planned maintenance window:
- Pause all IAM mutations and isolate any offline site or backup whose state is unknown.
- Back up each site’s underlying IAM storage and required KMS/encryption keys; online IAM admin exports omit deletion tombstones and cannot substitute for a storage-level backup.
- Stop all nodes sharing the same IAM backend, replace the binaries, and start them on the new version. Resume IAM writes only after all participating sites are upgraded.
- Inspect IAM load logs, site-replication status, and revocation convergence metrics; verify that representative revoked credentials are rejected and newly issued credentials succeed.
- Explicitly reconcile any revocations lost prior to upgrade: delete surviving stale records on sites that still hold them, and rebuild long-offline peers from approved state before reconnecting them rather than joining an unknown old snapshot to the cluster.
Child credentials issued by pre-upgrade binaries for a recreated parent lack the signed siloParentRevocation claim and must be reissued after upgrading; parent identities that have never undergone deletion and same-name recreation keep their existing credential behavior unchanged.
20261026 built-in policy Reset update: Unmodified built-in policies remain protected against local deletion. Starting with Server 20261026, deleting an explicit built-in policy override immediately restores the native built-in definition while retaining the deletion revision to reject stale overrides (replacing the suppression behavior in Server 20260916/20261018). Peer and etcd notifications reload committed state so edits made after a delayed deletion event are preserved; in-use checks and custom-policy deletion remain unchanged. Deleting a non-existent policy locally remains idempotent without creating a tombstone, whereas a replicated deletion for an unknown policy retains its revision record.
To roll back, stop and isolate the affected sites and audit permission changes since the backup before restoring compatible state. Restoring an older backup discards revocations committed after the backup point and requires permission reconciliation and key rotation before reopening traffic. Never manually delete tombstone files online or convert only live records to force an older binary to start.
Regression and Performance Verification
Focused regression suites reside in iam-revocation_test.go, iam-revision_test.go, iam-revision-lock_test.go, iam-revision-boundary_test.go, iam-replication-protocol_test.go, iam-credential-retention_test.go, iam-peer-reload_test.go, and iam-replay_test.go. Point SILO_TEST_IAM_REVOCATION_ETCD at a disposable etcd instance to run the etcd lifecycle, distributed lock, and boundary suites under isolated key namespaces.
Benchmarks cover:
BenchmarkIAMCachedCredentialandBenchmarkIAMSetTempUser: baseline latency for cached authentication and temporary user issuance.BenchmarkIAMRevisionConvergedHealing: steady-state in-memory index traversal and zero-duplicate-PUTverification at 1,000 and 10,000 retained records.BenchmarkIAMColdLoadExpiredServices: cold-start loading and cleanup latency with 100 or 1,000 expired reusable credentials (run with-benchtime=1x).
Error Semantics and Observability
October 23 update (pgsty/silo#250): LDAP Long DNs distinguishes a pure-ASCII path component that exceeds 255 bytes (and therefore could never be persisted on the object IAM backend) from an existing record that cannot currently be read. loadIAMConfigBytesWithMetadata translates only the former to errConfigNotFound; full-path length errors (PATH_MAX) and genuine storage failures still fail closed so an existing revocation record can never be masked. Writing direct user policies or parent-revision tombstones for overlong DNs remains unsupported on the object backend.
When an authoritative revocation revision commits to storage but a subsequent dependent cleanup step fails, the Admin delete endpoint may return HTTP 500; the old identity is already revoked. Pause IAM mutations and same-name recreation, retry the delete operation, and verify old-credential rejection across sites. Whenever a persisted parent revision cannot be read or token issuance cannot be committed, STS fails closed via STSInternalError rather than minting a credential with a guessed boundary.
Scrape each process’s authenticated /minio/metrics/v3/cluster/iam endpoint for observability:
| Metric Name | Description |
|---|---|
minio_cluster_iam_revocation_records |
Total deletion tombstones and parent revocation boundaries in this process’s in-memory index |
minio_cluster_iam_revocation_heal_failures |
Failed healing convergence passes since process startup |
minio_cluster_iam_revocation_heal_duration_millis |
Duration of the most recent healing pass in milliseconds |
minio_cluster_iam_revocation_heal_last_success_timestamp_seconds |
Unix timestamp (seconds) of the last successful healing pass |
Index metrics are reported per process; summing across sibling nodes sharing a backend double-counts shared records. Healing metrics advance only when site replication is enabled and the node holds the leader lease; in shared-backend deployments without site replication, zero values on the last three metrics are expected.
Why Simpler Alternatives Fail
In multi-site deployments with offline reconnection, simpler designs fail closed-world security invariants:
- Physical deletion without tombstones: Deleting configuration files outright destroys the ordering evidence needed when an offline node or delayed replication message reconnects, allowing old snapshots to resurrect deleted identities.
- Comparing only the child credential’s
UpdatedAttimestamp: After a parent user is deleted and recreated, a metadata edit or delayed replay on an old child credential would give it anUpdatedAttimestamp newer than the parent deletion and restore access; embedding the signedsiloParentRevocationboundary at issuance prevents any post-issuance edit from crossing the parent deletion boundary. - Overwriting source event timestamps with receiver local time: Destroys source causality and turns an idempotent network retry into a new mutation with a newer timestamp. SILO preserves source timestamps and advances revisions monotonically only on deliberate local writes.
- Assigning a fixed TTL to permanent identity tombstones or exporting only live records: Offline sites or cold backups can remain disconnected longer than any fixed TTL; expiring permanent identity tombstones or restoring live-only exports erases the historical boundary that blocks credential replay.
For implementation details, see cmd/iam-revision.go and cmd/metrics-v3-cluster-iam.go.
4.6 - Federated CopyObject: Preserve the Destination Contract
Server 20260916 repairs the legacy etcd bucket-federation CopyObject path across PR #157, PR #159, PR #163, PR #177, and PR #179. When CopyObject forwards a copy across federated deployments as a remote PutObject, the proxy streams logical source bytes, delegates destination encryption and Object Lock enforcement to the destination deployment, validates full-object checksums, and returns the exact committed version metadata.
Logical Byte Streaming and Destination Encryption
The proxy reads logical source bytes by decrypting and decompressing the source object locally and declaring the logical content length on the outbound PutObject request:
- No double transformation: The proxy never encrypts or compresses payload bytes locally before forwarding them to the remote deployment.
- SSE-C and SSE-KMS handling: Reading an SSE-C source requires the source customer key over TLS. Destination encryption is governed by the request’s explicit destination SSE headers or the destination bucket’s default encryption configuration; the proxy never injects its own local encryption defaults when the caller omits SSE headers. SSE-KMS context headers are forwarded as valid JSON objects.
- Raw SSE-C replica exclusion: Trusted raw SSE-C replica
CopyObjectrequests are rejected with501 NotImplementedon the federated path before creating any destination object.
Checksum and Metadata Forwarding
The proxy strips all reserved internal metadata prefixes (case-insensitively) from the outbound PutObject call and forwards only public user metadata and tags through standard S3 headers. Checksums on the forwarded write must describe the logical full object written at the destination:
- Algorithm and composite conversion: When the request specifies a checksum algorithm or the source carries a multipart
COMPOSITEchecksum, the destination write computes and stores aFULL_OBJECTchecksum. - Trailer vs. header delivery: Nonempty streams deliver computed digests in an
x-amz-trailerchecksum trailer, which the destination validates. Zero-byte objects use a standardx-amz-checksum-*request header, and existingFULL_OBJECTsource checksums are forwarded via standard checksum headers. - Remote response verification: If a required checksum in the remote
PutObjectresponse is missing, malformed, or carries a multipart-Nsuffix, the proxy returns an error. Because the remote write may already have committed before response validation fails, callers copying into a versioned bucket should inspect the destination before retrying.
Object Lock Precision and Typed Headers
The proxy forwards legal hold state through the typed LegalHold client option so it is sent as x-amz-object-lock-legal-hold rather than x-amz-meta-* user metadata. Retention timestamps retain full sub-second precision instead of being truncated to whole seconds. The destination deployment enforces authentication and bucket Object Lock rules.
Committed Write Identity
Both the CopyObjectResult XML response and the ObjectCreated:Copy notification event report the committed destination key, logical size, ETag, exact VersionID, and modification time returned in the authenticated write-time response from the destination deployment. The proxy neither fabricates a local timestamp nor issues a follow-up unversioned HEAD request that could observe a concurrent writer. When the caller specifies a source versionId, source-version response headers continue to identify the selected source version.
Source and Test Index
- Implementation:
cmd/object-handlers.go,cmd/object-handlers-common.go(PR #157, PR #159, PR #163, PR #177, PR #179) - Regression tests:
cmd/object-copy-federation*_test.go,cmd/object-federation-time_test.go - Related records: SSE-C Replica Integrity · Object Lock Replication Ordering · Component Version Matrix
4.7 - Multi-Pool Object Consistency
Shipped in SILO Server 20260916, PR #178 repairs multi-pool mutation ordering, PR #207 extends current-object selection to conditional PUT (closing #133 and #144), and commit fced86303 preserves object tags across pool rebalance and decommission. SILO Server 20261018 further adds 60c7179dd (incorporating PR #248) and 1748f62c7 to guard pool migration against stale payload overwrites and delete-marker loss.
October 26: Storage Timeouts and Rollback Boundaries
Server 20261026 retains the multi-pool deletion protections described below and forwards the storage RPC deletion options (including the previous data directory) needed for overwrite rollback. Prior to final release, commit dcc2f20ba withdrew the October draft’s admitted-I/O draining waits, per-object mutation guards, rollback-ownership checks, and backup-serialization changes, keeping the established drive/RPC timeouts, cancellation, object locking, read quorum, and healing behavior.
Note that a client-visible timeout does not guarantee that underlying disk or RPC I/O has stopped or that the write failed to commit. Late commits, delayed rollbacks, and stale values or tags after partial metadata updates remain possible; object locks do not provide distributed fencing across lease loss or network partitions, and upgrading does not automatically repair pre-existing inconsistent objects (stricter diagnostic checks remain gated behind the non-default consistency_strict build tag).
October 24: Protecting the Latest Copy During Deletion
PR #257, merged as aafe4f8886, extends multi-pool deletion safety to ordinary unconditional and bulk deletion paths. Bulk data deletion reuses the existing shared locks and resolver to retire non-authoritative copies before deleting each item’s authoritative copy, while preserving request response identity, supporting retry after partial physical deletion, and protecting shared remote tier references. It operates directly on the retained capacity-weighted allocator without depending on #245. See Multi-Pool Deletion: Preserve the Latest Copy on Failure for the full design and benchmark results.
October 18: Preserving Data and Delete Markers During Pool Migration
Commits 60c7179dd (PR #248) and 1748f62c7 in SILO Server 20261018 close two rebalance and decommission race windows that remained after the September fixes: stale payload overwrite and delete-marker loss.
Before committing a moved version to the destination pool, the pool layer inspects current copies across all pools under the common namespace object lock. It aborts the move if the source snapshot has disappeared or been replaced, or if another pool already holds a newer copy of the key. Equal-timestamp ties use the same deterministic pool-index tie-breaker as reads. The verified metadata snapshot is reused for field reconciliation; multipart completion re-verifies the snapshot after copying all parts, and tiered-object movement applies the same guard.
Delete markers are now migrated even when a delete marker is the only version present in the source pool, because older live versions of that key may still reside in other pools. Both null and UUID-versioned delete markers preserve their replacement and retry semantics. During source cleanup, the migration worker re-verifies and deletes only the specific versions observed by the walker while holding the object write lock. Cleanup stops immediately if the source version changed or if no retained destination copy preserves the externally visible read result, rather than deleting an entire key prefix that might now contain a concurrent write or a newer delete marker.
Runtime cost and boundaries: Migration now performs per-pool metadata reads and per-version source cleanup. Across $P$ pools and $V$ source versions, cleanup executes at least $\mathcal{O}(P \times V)$ metadata operations, and decoding/rewriting xl.meta version histories scales with version count. Ordinary resolved reads do not pay this migration check cost. No new on-disk format, S3 parameter, or external dependency is introduced. Data, tags, or delete markers lost during migrations on older releases cannot be reconstructed automatically, and cross-site late-arriving delete markers (#217) remain a separate topic.
Release verification exercised live 2-pool and 3-pool rebalance and decommission runs covering single-part and multipart objects, null and UUID versions and delete markers, concurrent writes and deletes, and interrupted/resumed two-pool migrations (see the 20261018 release verification record). Before retiring a decommissioned source pool in production, compare acknowledged version IDs, byte counts, ETags, tags, and deletion state through the S3 API.
Why One Key Can Have Multiple Physical Copies
Pool expansion, decommissioning, or an interrupted cleanup can leave physical copies of the same object key in more than one storage pool. Taking a lock on a single erasure set does not serialize against a concurrent writer that hashes or routes to another pool, and reading the first responding pool does not prove that its ETag, tags, or Object Lock retention state represent the current logical object.
Without cross-pool coordination, three classes of anomalies occur:
- A stale ETag in one pool falsely satisfies a conditional write (
If-Match/If-None-Match) or conditional delete. - A newer Object Lock retention date or legal hold in one pool is shadowed by older object metadata in another pool.
- Deleting the winning copy in one pool uncovers an older copy in another pool, resurrecting deleted data.
Operating on each pool independently and combining only their success statuses cannot prevent these races.
Shared Selection and Lock Boundary
In cmd/erasure-server-pool-consistency.go and cmd/erasure-server-pool.go, the pool coordinator holds a single namespace-level object lock across both candidate selection and mutation. objectPoolInfos queries the target version across every pool, including draining pools. An unreadable or quorum-failed pool is treated as an error rather than evidence of absence. Candidate copies are ordered by ModTime, with the pool index breaking exact timestamp ties.
An unversioned metadata request first resolves the logical current version across pools and then gathers physical copies of that exact version, never mixing metadata from distinct object versions. Because Object Lock retention, legal hold, and user tags each carry their own revision timestamps, mergedPoolObjectInfo merges each field by its own revision timestamp rather than using the object’s payload ModTime as a blanket version for all metadata fields. A timestamped tag or hold removal is preserved as an explicit state transition.
Metadata updates, object writes, and background healing share this coordinating lock. Note that the lock serializes concurrent callers but does not act as a multi-pool rollback transaction if a disk failure occurs midway through multi-pool cleanup.
Conditional Writes and Deletions
For conditional DELETE, the coordinator evaluates the precondition once against the resolved logical object before starting multi-pool cleanup, and clears the precondition callback before invoking individual pool deletes. When an explicit versionId is supplied, the condition is evaluated against that specific version.
For conditional PUT, PR #207 selects the latest logical representation across all eligible pools—including delete-marker state—while holding the outer namespace lock before writing the new version. This prevents a stale copy in the destination pool from causing If-Match or If-None-Match to disagree with the cluster’s current logical object.
If a storage failure occurs partway through multi-pool cleanup or metadata propagation, operators should re-verify object state and retry after drive recovery. Batch DeleteObjects XML ETag conditions remain unsupported (see Conditional DELETE).
Object Lock and Bucket Replication
Incoming replica writes reconcile destination Object Lock retention and legal-hold state against authoritative same-version copies across all pools, ensuring that an out-of-order replica write cannot shorten a newer retention period or clear a newer legal hold. Replicated object tags likewise travel with their revision timestamps, including empty-string deletion markers. See Object Lock Replication Ordering and Replicated Tag Ordering for details.
Preserving Object Tags Across Pool Migration
Pool rebalance and decommission read each version from disk as a FileInfo, convert it into an ObjectInfo—which moves X-Amz-Meta-Minio-Internal-Tagging out of the general UserDefined metadata map into the dedicated ObjectInfo.UserTags field—and then write the object into the destination pool. Because the migration writers introduced in 2022 copied only the UserDefined map when constructing the destination write, both rebalance and decommission silently dropped user tags on single-part and multipart objects.
Commit fced86303 routes all four migration write paths through a shared metadata assembly helper that clones the metadata map, restores UserTags, and preserves existing tag revision timestamps verbatim under the coordination lock. Regression tests cover single-part and multipart objects, initial tags, updated tags, cleared tags, versioned objects, and legacy metadata records. Tags dropped by migrations on earlier releases must be restored from source buckets or backups.
Verification and Operational Guidance
The multi-pool consistency implementation in cmd/erasure-server-pool-consistency.go and cmd/erasure-server-pool.go is covered by multi-pool unit and functional tests exercising stale pool copies, delete markers, explicit version IDs, concurrent writes, and partial failure paths.
All nodes in a multi-pool cluster must be upgraded together so every peer participates in the shared namespace lock. Clusters that previously ran multi-pool migrations or concurrent multi-pool writes should audit existing copies, tags, and Object Lock state using the Replica Metadata Audit runbook and check the component version matrix.
4.8 - Object Lock Replication Ordering
Initial Object Lock replication timestamp ordering (f4c1286c9) shipped in SILO Server 20260903 (recorded in the security advisory ledger). Subsequent fixes for timestamp-only removals, SSE-C replica retransmission, and multi-pool Object Lock reconciliation (PR #129, PR #134, and PR #178) shipped in SILO Server 20260916.
Preserve Stored State Before Rebuilding Destination Metadata
Previously, a replica COPY rebuilt the destination metadata map from the incoming request before comparing Object Lock retention and legal-hold revision timestamps. Rebuilding the map first wiped out the stored timestamps, causing stale incoming replication requests to overwrite newer destination state. In addition, the handler wrote the legal-hold revision timestamp into the retention timestamp key.
Commit f4c1286c9 captures the stored Object Lock state before rebuilding the destination metadata map and tracks each field’s revision timestamp under its own dedicated metadata key.
Retention (mode + retain-until-date) and legal hold (status) act as independent state registers:
- A newer retention timestamp never makes an older legal-hold value authoritative (and vice versa).
- The object’s payload modification time (
ModTime) never substitutes for either field’s explicit revision timestamp. - An incoming field update is applied only when its source revision timestamp is strictly newer than the stored timestamp for that field; stale or equal-timestamp updates preserve the stored destination value.
Treating Removals as Timestamped Tombstones
Clearing an object’s retention period or legal hold leaves an empty field value on the object while still carrying a source revision timestamp. If that revision timestamp is dropped when the value becomes empty, a delayed replication request carrying an older non-empty retention or legal hold will re-apply the cleared lock.
PR #129 and PR #134 preserve and compare timestamp-only removals across replication send, receive, and resync paths, distinguishing an uninitialized field (no value and no revision timestamp) from an explicit removal (empty value with a revision timestamp).
The receiver preserves these per-field timestamps across both metadata-only replica COPY requests and full SSE-C replica object retransmissions. Rewriting an SSE-C replica’s encrypted payload during retransmission cannot clear a newer destination legal hold or overwrite a newer destination retention timestamp.
Cross-Pool Authority
When an object version has physical copies in multiple storage pools (for example, during or after pool expansion or decommissioning), reading a single local erasure set cannot establish the latest Object Lock state. The multi-pool consistency coordinator (PR #178, closing #133) gathers all same-version copies across pools under the shared namespace object lock and merges retention and legal-hold fields independently by their respective revision timestamps. An unreadable or quorum-failed pool returns an error rather than being treated as an empty lock state.
Authorization and Operational Boundaries
Timestamp ordering governs conflict resolution on authorized replica writes; it does not grant permission to modify Object Lock settings. Standard replication trust checks and S3/admin permissions (s3:ReplicateObject, s3:BypassGovernanceRetention, s3:PutObjectRetention, s3:PutObjectLegalHold) remain enforced. Timestamp comparison establishes per-field ordering across replication hops; it does not compensate for unsynchronized wall clocks across sites or provide distributed rollback across partial drive failures.
Upgrading prevents stale replication events from overwriting newer Object Lock state going forward, but does not automatically reconstruct retention or legal-hold values that were overwritten prior to the upgrade. Operators should verify exact-version retention and legal-hold state against authoritative records before declaring recovery.
Verification
The Object Lock merge logic in cmd/erasure-server-pool-consistency.go and the replica write paths in cmd/erasure-object.go are covered by regression tests exercising stale vs. newer field updates, timestamped empty-value removals, independent retention and legal-hold timestamps, SSE-C full retransmission, and multi-pool reconciliation.
See also the Replica Metadata Audit runbook, SSE-C Replica Integrity, and the component version matrix. All participating nodes in a replication topology should be upgraded together.
4.9 - SSE-C Replica Integrity
This article records the SSE-C replica integrity contract formed by #122, #123, #124, #126, and #134, which shipped in Server 20260916, building on the earlier zero-byte/read-attribute authentication and destination-key checksum repairs from Server 20260903.
Ciphertext and Trust
An ordinary SSE-C request supplies the customer key and operates on plaintext. An authorized replica transfer can carry raw ciphertext with the sealed object-key metadata needed to preserve the original object. The destination must store those bytes verbatim; encrypting them again produces an unreadable double-encrypted object. The internal marker alone is not authorization: the exact replication marker and the required replication permission remain mandatory.
This path does not reveal the customer key or make a normal keyless read permissible. Ordinary GET/HEAD and GetObjectAttributes keep their key/permission checks. Federated raw SSE-C replica COPY is explicitly unsupported.
Multipart Sizes and Checksums
Encrypted part size and logical plaintext part size are different. Replica multipart records preserve the actual logical size of each part so partNumber reads, ranges, and GetObjectAttributes agree with the source after overwrite. Checksums must retain the correct encryption context and logical meaning; a checksum response cannot be decrypted with the source key after a copy committed under a different destination key.
For ordinary SSE-C key rotation, any explicitly requested checksum algorithm forces a complete rewrite, even if it names the existing algorithm. A multipart source becomes a single-part destination; the ETag can change and replication retransmits object bytes. An eligible metadata-only rotation without that request preserves the prior checksum state, including absence. See the operator procedure.
Retransmission and Object Lock
A keyless target HEAD can report an existing SSE-C object as inaccessible rather than absent. The sender must distinguish this from NoSuchKey; it cannot assume an ordinary metadata-only COPY will repair the replica. Existing SSE-C replicas use object retransmission. A destination with undecodable old replica state can be replaced through the repaired retransmit path, while preserving the newer destination Object Lock state and ordering removal timestamps correctly.
Upgrading the server does not automatically repair damaged historical replicas. To audit an existing replica, re-read the exact source and replica versions with approved key access, compare the logical bytes and part boundaries, and check retention and legal hold separately rather than relying on a metadata-only HEAD.
Compression and Historical Objects
SILO excludes every new SSE-C write from compression, including normal PUT, multipart initiation, COPY, and Snowball. SSE-S3 and SSE-KMS continue to follow their encrypted-compression setting. This avoids a replica format that transports ciphertext without the required compression metadata.
Historical compressed SSE-C objects are not automatically rewritten. Preserve their keys and exact version identities, inventory affected data, and rehearse a supported rewrite/recovery path.
Verification Boundary
The compression decision, replication-trust-ssec-replica_test.go, erasure-multipart-ssec-replica_test.go, replication-ssec-retransmit_test.go, and compression-ssec_test.go enforce the source and regression contracts, including incorrect-key/unauthorized controls, multipart byte comparisons, and retransmission cases.
See Object Lock ordering, multi-pool consistency, and replica recovery. Upgrade every participating Server before relying on the combined behavior.
4.10 - Replication Reliability: Delete Completion, MRF Visibility, and Resync Cancellation
This article records the root-cause analysis, design tradeoffs, and validation across three rounds of replication reliability repairs shipped in Server 20260916:
- Round 1 (PR #162, merged as
d1105bbb): addresses #153, #152, and #137, covering operation classification at the HTTP handler entry, MRF drop observability, and the resync cancellation lifecycle. - Round 2 (PR #196, commit
0c61128d2): repairs the replication worker’s delete exits, outer status aggregation, and the persisted MRF marker-recovery path (see Round 2). - Round 3 (commits
eb4f5e5b3,254b19ac0, and358ab38fb): repairs exact-version purge drive writes and quorum checks, delete-marker healing metadata retention, and pre-send revalidation for queued marker creations (see Round 3).
Overall Decision and the Surrounding Series
Each fix repairs the reproduced invariant at the smallest boundary that owns it, preserves existing background recovery mechanisms, and proves convergence with deterministic tests.
| Issue | Confirmed defect in SILO | Selected repair |
|---|---|---|
| #153: delete-marker purge | Single-object DELETE classified a permanent deletion as marker replication; the remote marker disappeared while source purge remained PENDING |
Classify by purge status, matching bulk delete, scanner/heal, and resync |
| #152: invisible MRF drops | The queue was already bounded, but internal drop counters did not reach administration or monitoring; object and delete worker arguments were reversed | Expose existing counters, warn at actual drop sites with deduplication, and align worker routing |
| #137: unreliable resync cancellation | One shared token could not stop multiple runs; blocking phases missed cancellation; stale runs could overwrite terminal state | Give each run an owned context, cancel by resync ID, and constrain registration, finalization, and state updates |
Two earlier fixes in the same series addressed adjacent invariants:
- #136 / PR #138 repaired counter completeness: receive and apply the final result before persisting terminal state, rather than waiting for the one-minute periodic flush.
- #139 repaired outcome accuracy: record success and failure from the target’s actual replication response rather than assuming success because an object already exists at the destination.
- #137 repairs cancellation and resource lifecycle: canceling a run stops its walker, workers, and result consumer, and prevents stale runs from overwriting a newer run’s state.
Authorization to use internal replication semantics is covered in No I/O Before Auth, No Privilege From Headers. Authoritative Object Lock state across pools is tracked in #133 and repaired by #178; see Multi-Pool Object Consistency.
#153: Distinguishing Delete-Marker Replication from Version Purge
Why the External Report and Local Reproduction Differ
The original issue described a sustained HTTP 405 storm involving ILM and replication, and proposed treating every delete-marker 405 probe as completion. Source inspection and local reproduction showed why that diagnosis was incomplete and why the proposed patch would break deletion guarantees.
Starting from baseline 450dcb848, the reproduction used two locally built SILO servers, separate data directories, and mcli / minio-go clients. A key finding was that mcli uses bulk DELETE even for a single key, and the bulk-delete path was already classified properly. Exercising the faulty entry point required issuing a direct single-object S3 DELETE.
| Observation | Single-object DELETE before the fix | After the fix |
|---|---|---|
| Matching delete-marker version at the target | Removed | Removed |
Purge state for that version in source xl.meta |
Still PENDING, although ordinary replication status was complete |
Cleanup completed on the first replication attempt |
| Original data version | Retained | Retained |
| Scanner needed to finish this cleanup | Required later recovery | No scanner assistance needed |
Checking only that the remote version disappeared would miss the lingering PENDING state in source xl.meta. The reproduction confirmed the misclassified initial purge, the wrong completion state, and an unnecessary HEAD probe, without triggering a sustained 405 storm because the scanner/heal and resync paths already selected the purge branch.
Why One Classification Condition Is Sufficient
When permanently deleting an existing delete marker, the object layer returns both DeleteMarker=true and a non-empty VersionPurgeStatus. The former describes the version being acted on; the latter describes the permanent purge now in progress.
The old single-delete handler checked only DeleteMarker and populated DeleteMarkerVersionID. Completion then updated ordinary ReplicationStatus instead of VersionPurgeStatus. The repaired condition in the DeleteObject handler is:
This matches the condition already used by bulk delete, scanner/heal, and resync. It requires no on-disk format change and no new recovery loop; existing PENDING entries continue to be cleaned up by the scanner/heal purge path.
Why HTTP 405 Is Not Unconditional Success
A 405 Method Not Allowed from a versioned HEAD indicates that a delete marker exists at the target. For replicating a delete marker, that proves idempotent completion. For permanently purging that version, it proves the marker is still present at the target.
Marking VersionPurgeComplete on a HEAD 405 would let the source erase its local tracking metadata while leaving the target version behind. Remote 403, 405, and 503 responses during a purge are genuine failures that must be retried.
Git blame traces the DeleteMarker-only check to an upstream 2020-11-19 commit. A 2023-07-10 optimization changed scheduling from dsc.ReplicateAny() to the returned object’s replication or PENDING purge state while keeping the DeleteMarker-only branch, and a 2025-04-02 commit renamed Pending to replication.VersionPurgePending at that site.
#152: Exposing Drops from the Bounded MRF Queue
MRF (Most Recent Failures) buffers recent failed replication operations for background retry. It enforces two bounds: mrfSaveCh has capacity 100000, and mrfRetryLimit is 3. Entries are dropped from the queue when RetryCount > mrfRetryLimit or when mrfSaveCh is full. The source object and its pending replication state remain on disk, and dropped queue entries are picked up later by the background scanner.
The defect was that TotalDroppedCount and TotalDroppedBytes incremented internally but were omitted from both administration statistics snapshots and from Prometheus v2/v3 metrics. A deterministic capacity-one test fixture reproduces the gap: one entry is queued, a 20-byte overflow entry and a 30-byte retry-exhausted entry are dropped, and internal totals reach 2 / 50 while administration endpoints report 0 / 0.
The Selected Minimum Change
- Atomically populate the existing drop counters into both administration snapshots.
- Export cumulative drop counters in Prometheus metrics v2 and v3.
- Log deduplicated warnings at the two actual drop sites (retry exhaustion and a full MRF channel) using static messages and deduplication keys so changing counter values do not defeat log deduplication. Normal worker-queue overflow into MRF does not drop the entry and does not log a drop warning.
- Route ordinary object replication, healing, and deletion consistently by
(bucket, objectName)to preserve per-object worker affinity.
| Interface | New metric |
|---|---|
| Prometheus v2 | minio_node_replication_mrf_dropped_operations_total |
| Prometheus v2 | minio_node_replication_mrf_dropped_bytes_total |
| Prometheus v3 | minio_replication_mrf_dropped_operations_total |
| Prometheus v3 | minio_replication_mrf_dropped_bytes_total |
These counters accumulate since Server startup and reset on restart. Operations count dropped queue entries, so repeated failures on the same object increment the counter multiple times; bytes sum known object sizes, with delete entries contributing zero bytes. Operators should monitor the rate of increase alongside replication backlog and target health.
The queue capacity and retry limit remain unchanged, keeping memory bounded while relying on the background scanner for eventual recovery.
#137: Making Cancellation Part of a Run’s Lifecycle
Why One Shared Token Cannot Cancel a Multi-Bucket Group
Previously, cancellation pushed a single unkeyed token into a shared channel. Because a site resync can span multiple buckets (up to 10 concurrent bucket runs) with dispatchers and workers competing to read from the same channel, a cancel request could stop only one bucket, be consumed by an unrelated run, or remain buffered and abort a later run.
Two blocking stages also ignored cancellation: reading from Walk output and sending to a full worker channel. If workers exited first, the dispatcher could block indefinitely on a full channel. In addition, Walk inherited the parent context directly, so an early return left its walker goroutine running.
State tracking had three accompanying bugs: site-level updateState mutated a value copy without writing it back to the map; bucket-level Canceled state handling was incomplete; and a late finalizer could overwrite Canceled with Completed.
Registration, Cancellation, and Status Share a Write Boundary
Each resyncBucket run now creates an owned context.WithCancelCause and registers it before waiting for a concurrency slot. Registration and cancelResyncID share the resyncer’s state lock:
- Registration verifies that the target still exists and the resync ID still matches, and checks whether the run has already been canceled.
- Cancellation marks matching
PendingandStartedentriesCanceledbefore canceling every registered context for that resync ID. - Runs waiting for a concurrency slot are canceled immediately, and any run registering after cancellation observes the
Canceledstate on entry. Walk, dispatch, workers, and result sends all select on the run’s owned context. Unrelated resync IDs are unaffected, and no shared token remains buffered.
A dedicated operation mutex serializes site start/cancel configuration updates, and in-flight runs are still canceled even if part of the target-configuration loop fails. Bucket finalizers and progress updates verify both the target and the resync ID, discarding late results from removed, replaced, or canceled runs. Site state writes back to the map, and Canceled is never overwritten by a late Completed.
Success and Failure Require Different Finalization Order
Finalization preserves the #136 ordering contract — terminal state is persisted only after the result consumer drains the channel:
Canceling the context before joining workers on a normal completion path would abort in-flight work and turn a completing run into Failed. Calling finish from a single defer sequences cleanup deterministically without an extra sync.Once.
WithCancelCause separates explicit user cancellation (Canceled) from parent-context shutdown. A run interrupted by server shutdown is not marked Completed, and a run still waiting for a concurrency slot during shutdown keeps its Pending state so it can resume on restart.
Protecting Terminal State and Resumed Work
Because a cancel request can arrive between the context check and the disk write, markStatus re-checks the in-memory state under lock and preserves Canceled even if the caller computed Completed, while rejecting writes from superseded resync IDs. If an individual bucket finished and persisted before a site-wide cancel arrived, its bucket record remains Completed while the site record is marked Canceled.
On the restart recovery path, loadResync previously spawned recovery goroutines and immediately ran defer cancel(). In SILO’s shared-lock implementation, that cancel() terminates the merged leader context, aborting the newly launched recovery goroutines. A WaitGroup now keeps the leader context alive until all resumed runs exit, while still canceling them if leadership is lost. Loading on-disk state also avoids overwriting newer in-memory start or cancel state.
Design Review and Complexity Decisions
| Proposal or review point | Final decision |
|---|---|
| Treat a purge’s 405 probe as completion | Rejected: marker existence does not prove permanent deletion |
| Add bounded retries, backoff, and a persistent MRF scheduler | Not introduced: the existing queue and scanner already provide recovery; the demonstrated gap is visibility |
| Send more shared cancellation tokens | Rejected: this still cannot guarantee identity routing, broadcast, or isolation from later tasks |
| Add a separate cancellation tombstone registry | Unnecessary: existing target status and resync ID under one lock close the registration race |
| Give each run an owned context and cancelable blocking operations | Retained: full-queue deadlock and walker leaks have deterministic reproductions |
Protect finish with sync.Once |
Narrowed: a single deferred call site makes sync.Once unnecessary |
| Wait for resumed runs before releasing leader context | Retained: SILO’s leader context cancel is active, so the WaitGroup is required and covered by leadership-loss tests |
Two additional details were kept simple: the infrequent resync start operation reads and writes configuration under the status lock (matching terminal saves), and the active registry keys on resyncOpts (including resyncBefore).
Validation and Reproducible Evidence
Regression tests were written against the unmodified baseline and verified to fail before applying the fixes. The test suite exercises the production HTTP handlers, erasure storage, Prometheus metric registries, and resyncBucket.
| Validation area | Result and evidence |
|---|---|
| Single-delete purge | ErasureSD and Erasure source/target cleanup; legacy PENDING recovery; marker idempotency; real 403/405/503 failure semantics |
| MRF | Actual capacity-one overflow and retry exhaustion; administration JSON; registered v2/v3 counter types and values; object/delete worker affinity |
| Resync | testing/synctest coverage for blocked Walk, full worker queues, user and queued cancellation, unrelated IDs, later tasks, terminal races, stale IDs, slot release, and leader recovery |
| Complete local suite | go test ./... -count=1 -timeout=30m passed across all 50 tested packages |
| Concurrency and repetition | Focused replication race tests passed; cancellation regressions passed 100 repetitions |
| Tooling and contracts | Local build, vet, lint, generated-file checks, rebrand compatibility guard, and git diff --check passed |
| Two native servers | Built from local source; direct single-object DELETE removed the target marker and source xl.meta entry while retaining the original data version |
| Remote integration | All eight PR checks passed, followed by all six main Go CI jobs and VulnCheck |
Regression test files at the merged commit: delete markers, MRF visibility, and cancellation lifecycle. Reproducible test commands:
| Evidence point | Exact identity |
|---|---|
| Baseline | 450dcb8484bc1337deba0cf608cc893a6691d794 |
| Final PR head | 66fe61ff65c83d68b74baa637a11623015c7aa21 |
| Merged main | d1105bbb3d4a0afa33b3a4ac11b821235038ed0e |
| PR Go CI | 34320440012 |
| Main Go CI | 34321319278 |
| Main VulnCheck | 34321319274 |
Maintenance Principles
Replication and resync regression tests must independently assert five invariants: accurate operation classification, observable queue drops, per-object replication outcomes derived from target responses, complete terminal counters prior to state persistence, and full goroutine and slot cleanup on cancellation.
Second Round (2026-09-16): Worker-Side Purge Classification and Persisted MRF Recovery
Round 1 classified deletes at the HTTP handler entry (#153), exposed MRF queue drops (#152), and completed the resync cancellation lifecycle (#137). Round 2 — PR #196, commit 0c61128d2, integration verification aea3882c9 — repairs the background replication pipeline: the worker’s own delete exits, outer status aggregation, and the persisted MRF recovery path for delete markers.
What Was Still Broken
- Legacy-shape tasks were skipped entirely. A task with an empty
VersionID, a non-emptyDeleteMarkerVersionID, aCOMPLETEDcreation target, and aPENDINGpurge target never issued its DELETE because the per-target creation early-return suppressed it. - Fixing only the per-target function caused outer aggregation to misreport. Outer status selection keyed on
VersionIDand creation state, so a failed legacy-shape purge aggregated asCOMPLETED, emittedObjectReplicationComplete, and skipped persisted MRF queueing. - Persisted MRF dropped every delete-marker 405. Replaying an on-disk MRF entry for a delete-marker version returned the marker metadata alongside
MethodNotAllowed, and the error branch discarded the entry instead of recovering it. - Status clobbering and duplicate requests. Failed purges overwrote successful resync markers, completed purges were re-sent, and an unready
HEADprobe unconditionally overwrote creation state. - Multi-target empty-state regex misparses corrupted creation metadata. Replaying a disk marker that carried only creation metadata against a task carrying purge state parsed two empty target states as
Pending; the deleted-flag guard then rewrote the entire creation block as empty entries with fresh timestamps.
Three root causes drove these failures: the worker lacked task-level purge classification (the per-target predicate in community PR #184 prevented composite purge statuses from ever reaching COMPLETE, though Julien Laurenceau’s investigation and proposed fix informed this round); MRF recovery lacked a valid-405 identity gate; and delete tasks carried no retry counter, making mrfRetryLimit unreachable on the delete path.
The Repair
- Unified purge classification across all exits.
isVersionPurge()(true whenVersionIDis non-empty, or whenDeleteMarkerVersionIDis non-empty alongside a composite purge status) drives both the inner per-target function and outer aggregation. Purge exits write onlyVersionPurgeStatusand leaveReplicationStatusempty, which signals the storage layer not to modify creation status. - Three-field clear guard. On the purge path, all three creation-status fields are explicitly cleared so the multi-target regex misparse cannot occur during disk writes.
- Canonical permanent-delete requests. Purges send an explicit
versionIdwithReplicationDeleteMarker=falseand skip the pre-flightHEADreadiness probe, preventing a lost-response retry from re-creating a delete marker via an unversioned fallback. - Valid-405 gate for MRF recovery.
MethodNotAllowedschedules recovery only when the returned object is a delete marker, the bucket/object/version identity matches, and the modification time is non-zero. - Bounded retry budget. Delete tasks carry a retry counter that increments across all three persisted-MRF entry points (aggregation failure, lock failure, and queue-full fallback) up to
mrfRetryLimit; after exhaustion, the background scanner can still schedule healing. - Boundary-only status mapping.
COMPLETEis mapped toCOMPLETEDonly at the statistics and event-notification boundary, reusing the existing legacy constant.
What 405 Means, Precisely
For creation (replicating a delete marker), a HEAD 405 on the target marker version means the marker already exists at the target — idempotent completion. For purge (permanently deleting a version), a 405 during local MRF identity probing with a matching marker identity means the marker still exists and needs purging; purge completion is decided solely by the remote DELETE response, where 403, 405, and 503 are always real failures. Empty or null version markers return ObjectNotFound rather than 405 and never enter the 405 gate.
Boundaries That Remain
The legacy in-memory task shape does not serialize across restarts and has no active producer; handling it provides defensive coverage. Targets without a configured client still log only, and target-level resync replacing a purge subset remains unchanged. For the tag-ordering and replica-metadata fixes in the same series, see Replicated Tag Ordering and Replica Metadata Normalization.
Third Round (2026-09-16): Exact-Version Purges, Marker Healing, and Stale Creations
Round 2 made the worker’s purge exits truthful. Round 3 — commits eb4f5e5b3, 254b19ac0, and 358ab38fb, shipped in Server 20260916 (40220bd836cbd066ca424fa4dc5dbb90057fb55a, alongside silo-pkg 6f27ee6) — repairs how an exact-version purge updates drives and how queued marker creations behave after a purge completes. It was prompted by three-site container runs (three sites, four processes and sixteen drives per site, EC 12+4) in which a purged delete marker reappeared on both target sites while the source site still returned the data version.
What Was Still Broken
- A purge could write the delete marker it was removing. Lookup marked a version under purge as deleted for visibility, and the purge write-back treated that flag as an instruction to write a delete marker to disk. Drives that lacked the marker had it written to them, causing a partially purged erasure set to flip-flop on retry rather than converge.
- Missing versions were acknowledged without write quorum. Because the read path reported a version absent once half of the drives said so, a retried purge returned 204 while up to half of the drives still held the delete marker.
- Healing a delete marker stripped its replication metadata. The heal path reconstructed delete markers with an empty metadata map. Healed markers lost their replication and purge state, appeared pending again, and could be re-replicated as fresh marker creations.
- Queued marker creations executed after the purge finished. Every GET, HEAD, or LIST on a pending delete marker, along with the scanner and MRF, queued a creation task carrying a snapshot of the marker; tasks from multiple frontends serialized on the per-object replication lock. In the reproduced timeline, both targets acknowledged the user’s purge, and ~85 ms later a queued creation carrying the same
VersionIDand original modification time arrived at both targets. Because nothing in the wire request distinguished an initial creation from a stale replay, both targets recreated 16/16 marker copies while the source held 0/16.
The Repair
- Classify purge intent once (
isVersionPurge). A request is a version purge when it carries a valid explicitVersionID, sets no marker-creation, prefix, data-movement, free-version, expiry, or transition options, and either carries a completed purge status, has no replication state, or is a trusted replica request. A purge never sets marker-creation flags, and its vote tally combines drives that physically removed the version with drives that confirmed it absent. The response reports a delete marker only when a stored marker was actually removed; a data version with an earlier pending purge is reported as a data version. - Require a write-quorum majority (
N/2+1) to confirm absence. When lookup reports a version missing on a retried purge, the server re-reads every drive. Fewer thanN/2+1absent replies fail with insufficient write quorum, and any erasure set that still holds a copy is scheduled for MRF healing. Offline, corrupt, or unreadable drives never count as absent. The multi-pool layer applies the same check to pools skipped by lookup, and replica receivers resolve purges by version across pools. - Preserve stored metadata when healing markers. Healing copies the delete marker’s persisted metadata map and strips only transient healing, data-movement, and tiering keys. Typed state generation is restricted to fresh creation writes that have no stored metadata, and never fabricates timestamps.
- Revalidate marker creations under the replication lock before sending. Before transmitting a marker creation, the delete replication worker re-reads the source version under the per-object replication lock. If the version is missing, is no longer a delete marker, or is undergoing a purge, the task is stale and is dropped without modifying the source; inconclusive reads are retried via MRF rather than treated as absence.
Evidence and Boundaries
Two three-site batched-recovery runs (with and without a source restart, returning one target site in two halves) converged within ~15 s and remained consistent for the rest of a 450 s test window. The stale-creation regression fails on unmodified main and passes with the fix on both single-drive and 16-drive erasure fixtures.
Source-side revalidation under the lock cannot recall a creation request already in flight on the network or replayed by another site; closing that cross-site window requires a receiver-side persistent purge record or sequence number, tracked in #217. If all server processes crash simultaneously right after a majority-acknowledged purge, minority copies on drives that were offline during the purge can persist for up to seven scanner cycles, during which all frontends continue to serve the data version.
4.11 - FULL_OBJECT Checksum Replication: Preserve the Type at Upload Initiation
Issue #254 reported a replication failure affecting multipart objects uploaded with a FULL_OBJECT CRC32 or CRC32C checksum: the object uploads and reads normally on the source, but the replication target rejects CompleteMultipartUpload with 400 BadDigest. PR #255, contributed by Denis Babchuk (@botdeniska), restores the omitted checksum-type header in outgoing replication requests.
The patch was merged on October 24, 2026 as 6d890fc86 and shipped in Server 20261026, closing #254. It changes five production lines (including two comment lines) and adds 257 lines of regression tests.
Observable Failure
When an object has a multipart checksum recorded as FULL_OBJECT with algorithm CRC32 or CRC32C and bucket or site replication is enabled:
- The initial multipart upload succeeds and the source version remains readable.
- The replication sender transfers all parts to the target, but the target’s
CompleteMultipartUploadreturns400 BadDigest. - The source version remains in
FAILEDreplication status and no replica version is created on the target. - Every subsequent replication retry (including retries enqueued by
HeadObjecton theFAILEDversion) re-transfers the object parts and fails again at completion.
Multipart CRC64NVME objects replicate normally because the receiver defaults CRC64NVME to FULL_OBJECT; ordinary single-part PUT uploads and multipart COMPOSITE uploads are also unaffected. Performing a server-side CopyObject of an affected multipart FULL_OBJECT object triggers a second failure mode described below.
Root Cause: An Algorithm Does Not Define the Checksum Type
For multipart CRC32 and CRC32C, the same algorithm supports two distinct object-level checksum calculations. Let P1 ... Pn be the part bodies and || denote byte concatenation:
| Type | Object checksum calculation |
|---|---|
FULL_OBJECT |
CRC computed across the concatenated bytes `P1 |
COMPOSITE |
CRC computed across the concatenated raw per-part CRC digests, encoded as Base64 with the -n part-count suffix |
These two calculations produce different digests even for a single-part multipart upload. Forwarding only the algorithm name and digest value leaves the checksum semantics ambiguous.
The failure chain in the replication sender proceeds in four steps:
putReplicationOpts()callsgetCRCMeta()to build object checksum headers. The previous implementation copied the digest value andx-amz-checksum-algorithm, but droppedx-amz-checksum-type.replicateObjectWithMultipart()passes those options toNewMultipartUploadon the target. When CRC32 or CRC32C is initiated without an explicit checksum type, the target initializes and persists the upload asCOMPOSITE.- After uploading all parts, the sender calls
getCRCMeta()again for the completion request, sending the source’s whole-object (FULL_OBJECT) CRC digest. - The target’s completion validator reads the
COMPOSITEtype persisted at initiation, compares the supplied whole-object CRC against the composite CRC of the part checksums, and rejects the mismatch withBadDigest.
| Stage | Before the fix | After the fix |
|---|---|---|
| Source checksum metadata | CRC32/CRC32C + FULL_OBJECT |
Unchanged |
| Target initiation headers | Algorithm only; type omitted | Algorithm + FULL_OBJECT |
| Target persisted upload type | COMPOSITE |
FULL_OBJECT |
| Completion checksum | Source whole-object CRC | Source whole-object CRC + explicit FULL_OBJECT type |
| Target validation result | 400 BadDigest |
Valid completion |
The core invariant is that replication must preserve the checksum algorithm, digest value, and type interpretation across both upload initiation and completion so the target validates the transferred data under the same contract as the source.
Upstream Regression and Earlier Repairs
Upstream MinIO commit 4ea6f3b06, merged on August 22, 2025 in PR #21535, removed checksum-type forwarding from getCRCMeta(). Prior to that commit, getCRCMeta() forwarded the checksum type unconditionally for every checksum—a behavior that cannot be restored wholesale without breaking single-PUT SHA replication (see below).
Three earlier SILO checksum fixes addressed separate parts of the lifecycle:
| Earlier change | What it fixed | Why replication initiation still failed |
|---|---|---|
c8590413f (#31) |
Allowed FULL_OBJECT completion requests to omit per-part checksums while validating the whole-object CRC |
Requires the target upload to be initialized as FULL_OBJECT first |
7e079ff05 |
Verified that an explicitly supplied completion checksum type matches the type persisted at initiation | The replication sender omitted the type at both initiation and completion |
| CompleteMultipartUpload ChecksumType response fix | Returned ChecksumType in CompleteMultipartUploadResult XML to client SDKs |
Governs client response serialization after completion, not outgoing replication headers |
The Minimal Fix
Inside getCRCMeta(), immediately after copying a recognized checksum digest and algorithm, the patch adds:
Each condition narrows forwarding to the exact domain required:
isMP: Decoded from the stored checksum’s multipart flag (which does not require the current object to carry a-nmultipart ETag).cktype.CanMerge(): Selects the algorithms that support merging per-part CRCs into a full-object CRC (CRC32, CRC32C, and CRC64NVME).cs[xhttp.AmzChecksumType] == xhttp.AmzChecksumTypeFullObject: Forwards the type only when the stored checksum is actuallyFULL_OBJECT, never relabelingCOMPOSITEchecksums.
Because both upload initiation (NewMultipartUpload) and completion (CompleteMultipartUpload) call getCRCMeta(), this single guard restores the header across both phases. Individual UploadPart requests read per-part checksum maps that carry no object-level checksum type, leaving part headers unchanged, while CRC64NVME receives an explicit FULL_OBJECT header matching its existing default.
Why Not Forward Every Stored Type?
An ordinary single-PUT SHA1 or SHA256 object also records FULL_OBJECT in its stored and read checksum metadata. However, on the request parser side, getContentChecksum() rejects an explicit x-amz-checksum-type: FULL_OBJECT header for non-mergeable algorithms such as SHA1 and SHA256.
Forwarding stored checksum types unconditionally would therefore break single-PUT SHA1/SHA256 replication. Gating on isMP && cktype.CanMerge() repairs multipart CRC replication while keeping single-PUT and COMPOSITE request headers unchanged.
CopyObject Automatically Reuses the Single-PUT Path
When a server-side CopyObject copies a multipart FULL_OBJECT object, the resulting object has a plain non-multipart ETag (!objInfo.isMultipart()) while retaining the multipart FULL_OBJECT checksum metadata. Because a FULL_OBJECT digest covers the entire logical payload, it remains valid when replicated via a single PutObject.
putReplicationOpts() already contained the routing check for this case:
When getCRCMeta() dropped x-amz-checksum-type, that condition never matched, forcing the copied object down a one-part multipart replication path that failed with BadDigest. Restoring the type in getCRCMeta() re-enables this existing branch automatically: the copied object replicates through a single PutObject without any Copy-specific code or checksum recalculation.
Rejected Alternatives
| Alternative | Decision and rationale |
|---|---|
| Forward every checksum type unconditionally (pre-PR #21535 behavior) | Rejected: explicit FULL_OBJECT headers on SHA1/SHA256 requests are rejected by getContentChecksum() |
Send the checksum type only on CompleteMultipartUpload |
Rejected: the target upload is already initialized as COMPOSITE, and completion rejects a mismatched type |
Default all CRC32/CRC32C multipart uploads to FULL_OBJECT |
Rejected: breaks standard S3 COMPOSITE semantics |
Drop the object checksum or bypass BadDigest validation |
Rejected: sacrifices end-to-end integrity verification |
Recalculate a COMPOSITE digest on the sender to match the target |
Rejected: alters the replica’s checksum type and digest value and adds unnecessary work |
Add a separate CopyObject replication branch or CRC64NVME special case |
Unnecessary: the existing !objInfo.isMultipart() branch and CanMerge() already cover both |
| Adjust replication retry budgets, queues, or resync scheduling | Ineffective: retrying an unchanged malformed request repeats the same BadDigest failure |
Validation and Live Acceptance
Running the new regression tests against the unmodified baseline aafe4f888 reproduced BadDigest on FULL_OBJECT CRC32/CRC32C replication and failed the CopyObject routing assertion (got true, want false), while passing all control cases. With the patch applied, both test suites passed.
The committed regression suite includes 8 table-driven header tests (checking both object-level and per-part headers) and 8 API integration cases across two versioned buckets, verifying routing decisions and matching checksum metadata.
Live acceptance testing ran two single-drive SILO server instances configured for site replication with mcli across 12 end-to-end cases:
| Case family | Algorithms / variants | Cases |
|---|---|---|
Single PUT |
SHA256, SHA1, CRC32 | 3 |
Multipart COMPOSITE |
CRC32, CRC32C, SHA256 | 3 |
Multipart FULL_OBJECT |
CRC32, CRC32C | 2 |
Multipart default FULL_OBJECT |
CRC64NVME | 1 |
CopyObject of multipart FULL_OBJECT |
CRC32, CRC32C | 2 |
Single-part multipart FULL_OBJECT |
CRC32 | 1 |
All 12 cases reached COMPLETED on the source and REPLICA on the target, with independently computed checksum digests and types, version IDs, ETags, object sizes, and downloaded payload bytes matching across both endpoints.
Reproducible test commands:
Both PR CI (Go CI, DCO, VulnCheck) and merged-commit CI (Go CI, VulnCheck) passed in full.
Upgrade and Operational Guidance
- Upgrade the replication source: Because the defect is an omitted header in outgoing replication requests, upgrading only the target cannot restore the missing
x-amz-checksum-type. In active-active bidirectional replication, every site acts as a sender and must be upgraded. Mixed-version testing reported in PR #255 confirmed that a patched source replicates cleanly to an unpatched SILO target, whereas an unpatched source still fails against a patched target. - Replication paths covered: Bucket replication, site replication, and batch replication (
batchReplicationOpts()) all delegate header construction toputReplicationOpts()and inherit the fix. SSE-C encrypted objects continue to pass encrypted checksum metadata throughX-Minio-Replication-Ssec-Crc. - Recovering existing
FAILEDversions and rollback: No configuration setting, on-disk metadata format, or public API changes. ExistingFAILEDversions retain valid checksum metadata on the source and complete replication on retry or resync once the sending node is upgraded; rolling back affects only subsequent outgoing replication requests and requires no metadata migration for already replicated objects.
4.12 - IAM Policy Reloads: Lock Scope, Cancellation, and Time Budgets
Commit 1b5477f3d (pgsty/silo#251) resolves two coupled failure modes under slow KMS/KES responses:
- Authentication-cache lock contention:
ListPoliciesandListPolicyDocsnow load and decrypt policy replacements outside the in-memory authentication-cache lock while retaining IAM write serialization (writeMu). - End-to-end request cancellation: IAM configuration decryption and SSE-S3 list key unsealing now propagate the caller’s request context all the way to outbound
KES.Decryptcalls, while keeping backend RPC timeouts scoped strictly to their storage operations rather than truncating legitimate long-running queries at 30 seconds.
Two Symptoms, Two Mechanisms
During incident investigation of pgsty/silo#251, controlled reproduction isolated two distinct server behaviors when KES responses are slow:
- When an administrator lists IAM policies while KES is slow, the policy listing takes several seconds, and ordinary users’ unrelated
GET,HEAD, andPUTobject requests stall behind it. - When a client abandons an SSE-S3 object listing (
ListObjectswith encrypted metadata), the server continues sendingDecryptrequests to KES for subsequent objects that the disconnected client will never receive.
| Observation | Root-Cause Mechanism | Fix Component |
|---|---|---|
Ordinary S3 GET, HEAD, or PUT requests stall behind a slow policy list |
ListPolicies / ListPolicyDocs hold the in-memory authentication-cache write lock while reading and decrypting every stored policy document |
A: Move slow backend loading and decryption outside the authentication-cache lock |
| KMS/KES decryption continues after the HTTP client cancels the request | Intermediate helpers substitute context.Background(), severing cancellation propagation before the outbound KES request |
B: Thread the caller’s context.Context through the entire decryption call chain |
| A legitimate slow policy query (> 30 s) fails with HTTP 503 in the first repair candidate | Reusing withIAMStorage and reusing etcd’s 30-second Get child context inadvertently imposed a 30-second ceiling on total decryption work |
R1: Scope each timeout strictly to the storage operation it originally bounded |
Shrinkage of the lock scope (A) and cancellation propagation (B) solve complementary problems: releasing the cache lock keeps healthy S3 traffic flowing during a slow admin query, while context propagation stops wasted KMS calls after a client disconnects.
How the Coupling Arose
ListPolicies and ListPolicyDocs reload persisted policies from the backing store and replace the in-memory policy map. In the original implementation, store.lock() (the write lock on IAMCache) remained held from the start of the backend scan until every policy document was read and decrypted. On the object-storage IAM backend, each KMS-encrypted policy document requires a separate KES decryption call. Because every ordinary S3 request acquires the read lock on the same IAMCache to authenticate the caller and evaluate permissions, multi-second KES latency during an admin policy listing directly blocked data-plane S3 requests.
Separately, the request cancellation chain was broken at three layers:
- IAM configuration decryption invoked KMS with a background context unrelated to the incoming HTTP request.
- SSE-S3 object listing (
UnsealObjectKeys) called a single-object helper (UnsealObjectKey) that did not accept acontext.Context. - The KES client adapter invoked
Decryptwithcontext.Background().
Because the lowest layer replaced the context with context.Background(), passing a request context into a top-level handler had no effect on outbound KES traffic.
A: Release the Cache Lock, Retain Write Ordering
The IAM subsystem uses two locks with distinct responsibilities:
- The cache lock (
store.lock()/store.rlock()): protects the in-memory maps read by S3 authentication on every request. - The IAM write mutex (
writeMu): serializes administrative IAM mutations and full cache replacements so concurrent updates cannot interleave.
With fix A, policy reloading keeps writeMu held to preserve mutation ordering, but releases the cache lock while performing slow storage I/O and KMS decryption:
While the private map is being populated and decrypted, ordinary S3 requests continue reading the existing committed cache without waiting on KES. Concurrent IAM administrative writes still wait on writeMu. Releasing both locks during loading was rejected because a slow policy reload that started before an IAM policy update or deletion could finish afterward and overwrite the newer state with a stale snapshot.
The new policy map is published only if the backend load succeeds and ctx.Err() is still nil after reacquiring the cache lock. Checking for cancellation only before reacquiring the lock would miss a cancellation that occurred while waiting for the lock. If any backend error, partial load, or cancellation occurs prior to publication, both the existing in-memory policy map and its updatedAt timestamp remain untouched.
Background IAM refresh follows the same merge and revocation invariants. Regression tests exercise the concrete IAMObjectStore refresh path directly alongside concurrent policy updates, deletions, and durable IAM revocations.
B: Propagate Cancellation to KMS and KES
Both affected decryption chains now thread the caller’s context.Context end-to-end into the KMS and KES layers:
To avoid churning unrelated call sites that do not carry a request context, the existing single-object UnsealObjectKey function keeps its original signature, while UnsealObjectKeys calls an internal context-aware helper.
Cancellation is checked at each loop iteration, before fallback branches, and before cache publication. In particular, when an IAM decryption call fails due to context cancellation, the loader immediately aborts rather than triggering the legacy associated-data (AAD) compatibility fallback. For active (non-canceled) requests, the legacy AAD fallback, ciphertext format handling, key ordering, and non-cancellation KES error mappings remain unchanged.
R1: Keep Timeouts Scoped to Their Original Operations
The first candidate implementation of A and B reused withIAMStorage to release the cache lock around slow backend work. withIAMStorage was introduced in the September 15 revocation-persistence commit (709d50a916) to bound single-record storage I/O, distributed lock acquisition, and cleanup to 30 seconds. Reusing that helper for full policy listing inadvertently attached a 30-second total deadline to an operation that previously had no whole-query time limit.
Adversarial testing exposed the regression immediately:
- On the baseline server, a slow policy query across 61 encrypted policies completed in 32.74 seconds with
HTTP 200. - On the first repair candidate, the same query failed after 31.22 seconds with
HTTP 503because the 30-secondwithIAMStoragedeadline expired before all policies finished decrypting.
Simply removing the outer withIAMStorage wrapper was still not sufficient for the etcd backend. In the etcd IAM store, Get operations create a 30-second child context to bound the etcd RPC. Previously, post-read KMS decryption ignored the context parameter altogether; once fix B wired context propagation through decryption, passing the etcd read child context into decryption caused the 30-second etcd RPC deadline to govern subsequent KMS decryption as well. A fast etcd read followed by 32 seconds of KES decryption therefore failed with a deadline error.
Correction R1 separates the caller’s request budget from individual etcd RPC budgets across both policy documents and policy mappings:
| Time Budget | Scope and Behavior |
|---|---|
| Caller deadline or cancellation | Governs the entire HTTP request and all associated KMS/KES decryption calls |
Existing etcd read budget (30 s) |
Bounds only the etcd Get RPC; post-read decryption switches back to the caller’s parent request context |
| Existing revision-write, revocation, lock, and cleanup budgets | Retain withIAMStorage and their existing bounded timeouts |
| Whole-policy-list 30 s timeout (from first candidate) | Removed from both ListPolicies and ListPolicyDocs |
Because the etcd read context is derived as a child of the caller’s request context, a shorter caller deadline still preempts etcd reads immediately. Once the etcd read succeeds, decryption proceeds under the caller’s remaining time budget—allowing legitimate slow queries beyond 30 seconds to succeed when the caller permits.
Alternatives Considered
| Candidate Approach | Why It Was Rejected |
|---|---|
| Keep the cache lock held and shorten the list timeout | Healthy S3 requests still stall until the timeout expires, and legitimate slow policy queries fail |
Release both writeMu and the cache lock during loading |
A slow list can finish after a concurrent IAM write and overwrite newer state with a stale snapshot |
| Return cached policies without reloading from storage | Breaks the freshness contract of administrative policy listing and masks backend issues |
| Raise the whole-list timeout to a larger fixed constant | Merely shifts the regression threshold to a larger policy count or higher KES latency |
Remove only the outer withIAMStorage timeout |
Leaves the etcd post-read decryption deadline regression intact |
| Remove all backend storage timeouts | Weakens existing bounds on single-record reads, writes, distributed lock acquisition, and cleanup |
| Parallelize policy decryptions or add a second ciphertext cache | Introduces new KES concurrency spikes and cache invalidation complexity not required to fix the lock and cancellation bugs |
| Rewrite KES SDK internal retry behavior | Expands the blast radius into third-party SDK semantics |
Relative to baseline a07d655dc, the final implementation touches 8 production files (+83 / -29 lines) and 5 test files, with no wire-protocol, ciphertext-format, or permission-model changes.
Verification and Regression Benchmarks
Process-level benchmarks against baseline a07d655dc and final commit 1b5477f3d demonstrate the exact behavioral shift:
| Scenario | Before / First Candidate | Final Implementation (1b5477f3d) |
|---|---|---|
| 4 ordinary S3 object requests during a 5.23 s policy list | Mean 5,284 ms (blocked behind policy list) | Mean 2.25 ms (policy list still takes 5.23 s in parallel) |
| Same 4 ordinary S3 requests with a 200 ms client timeout | All 4 time out | All 4 return 200 OK in 1.11–1.82 ms |
| SSE-S3 object list abandoned by client at 200 ms | 21 additional KES decryptions start after client exit | 0 additional decryptions start after exit (4 total before cancel) |
| Object-backed policy query taking > 30 s (61 policies) | Baseline: 200 OK in 32.74 s; First candidate: 503 at 31.22 s |
200 OK in 32.75 s with all 61 policies returned |
Fast etcd read + 32 s KES decryption (60 s caller budget) |
First candidate fails during post-read decryption | Both policy-list entry points and policy-mapping loader succeed |
| 51 normal and denied HTTP control cases | Baseline status codes, error codes, and item counts | All 51 controls match (including pagination, KMS denial, and SigV4 checks) |
During test construction, two subtleties in the KES client surfaced:
- Non-cancelable SDK retry sleeps: The KES SDK can perform up to two backoff sleeps of up to
999 msthat do not select onctx.Done(). A 2-second test window was too tight to cover a300 msrequest deadline plus two backoff sleeps; widening the post-cancel observation window to 3 seconds confirmed that no new outbound decryptions are initiated after cancellation. - Periodic
DescribeKeyhealth probes: Running a real 32-second test required the mock KES fixture to answer periodic backgroundDescribeKeyrequests alongsideDecrypt.
Regression tests are committed in:
cmd/iam-policy-reload_test.gocmd/iam-policy-refresh_test.gocmd/iam-policy-budget_test.gocmd/kms-cancellation_test.go
Operational Boundaries
- Concurrent IAM admin writes: Because
writeMuremains held during a policy reload to preserve write serialization, administrative IAM mutations (such as creating or deleting a policy) still wait for an in-progress policy reload to finish or be canceled. Ordinary S3 authentication does not wait onwriteMu. - KES SDK backoff return latency: Cancellation immediately aborts in-flight HTTP requests to KES and prevents subsequent keys or AAD fallback branches from starting, but if the KES SDK is mid-backoff sleep (
<= 999 ms), the local Go function returns once that sleep finishes.
Engineering Takeaways
- Audit every side effect of a reused helper: Reusing
withIAMStoragefor its lock-release behavior silently imported a 30-second timeout into an operation that previously had no total-duration cap. - Trace context deadlines across layer boundaries: Wiring context propagation into previously context-free helpers can activate child deadlines (such as
etcdRPC timeouts) that were previously ignored by downstream calls. - Pair cancellation tests with slow-success tests: Verifying that canceled requests stop early must be paired with verifying that uncanceled slow requests beyond 30 seconds still succeed.
Related design records:
4.13 - LDAP Long DNs: Restoring Login Without Hiding Revocation Errors
Server PR #250 (merged at d43adc0f2 and shipped in Server 20261026) fixes a login failure affecting LDAP users whose normalized Distinguished Name (DN) exceeds the 255-byte single-component filename limit of the object-storage IAM backend, while preserving read errors that protect the durable IAM revocation boundary.
The initial bug report, patch (9faeb7a11), and long-DN regression tests were contributed by @910patin. During design review prior to merge, maintainers narrowed the error-classification guard to ensure that full-path length errors or unreadable multibyte records on disk can never mask an existing revocation record.
Problem and Impact
An enterprise LDAP user can have a long Distinguished Name (DN), especially when user names and multi-level organizational units (ou=) contain non-ASCII UTF-8 characters (such as Chinese or Cyrillic text). Even when a user derives all permissions from LDAP group memberships and has no directly attached user policy or local parent-identity record, SILO’s login flow still probes those optional IAM records.
On the object-storage IAM backend, the normalized DN is embedded directly as a directory or filename component. When that component exceeds 255 bytes, the filesystem returns a filename-length error (ENAMETOOLONG / ObjectNameInvalid) that aborts an otherwise valid LDAP login, STS token request, or service-account lookup.
Why DN Normalization Triples Non-ASCII Byte Length
An LDAP DN identifies a full directory entry (for example, cn=张三,ou=研发,dc=example,dc=com). Under RFC 4514 §2.4, syntax characters must be escaped, and arbitrary bytes may be escaped using \xx (a backslash followed by two hexadecimal digits per byte). While RFC 4514 allows UTF-8 text to remain literal, the upstream go-ldap library’s DN.String() serializer hex-escapes every non-ASCII UTF-8 byte.
SILO’s NormalizeDN parses the DN returned by the LDAP server and formats it via DN.String() so attribute-type names and escape sequences have a canonical string representation:
| Representation | Byte Length |
|---|---|
Single UTF-8 character 张 |
3 bytes |
Serialized by DN.String() as \e5\bc\a0 |
9 ASCII bytes |
cn= + 40 copies of 张 + ,dc=example,dc=com (raw UTF-8) |
141 bytes |
The same complete DN after NormalizeDN |
381 ASCII bytes |
Because each 3-byte UTF-8 character expands into 9 ASCII characters (\xx\xx\xx), a 141-byte UTF-8 DN expands to 381 ASCII bytes on disk.
This lookup-result normalization was introduced upstream on March 27, 2024 in minio/pkg efb73fa5 (MinIO PR #19358, by Aditya Manthramurthy / @donatello) so that the server uses the authoritative DN returned by the LDAP directory rather than trusting the user’s raw login input spelling (for example, when a user inputs a Unicode fullwidth period that the directory resolves to an ASCII period).
How the Login Regression Arose
The object-storage IAM backend stores per-user records at paths constructed from the normalized DN:
POSIX filesystems enforce a 255-byte limit on each individual /‑delimited path component (NAME_MAX). When the normalized DN is used as the filename in sts-users/<normalized-DN>.json, the 5-byte .json suffix shares the same component budget: a 250-byte escaped DN (250 + 5 = 255) fits, whereas a 251-byte escaped DN (251 + 5 = 256) fails with ENAMETOOLONG.
Two historical behaviors shaped how this failure surfaced:
- Unix backslash omission in
checkPathLength: SILO’s pre-flightcheckPathLengthhelper does not count backslash (\) characters on Unix. In reproduction testing, a 271-byte hex-escaped DN was counted as 197 bytes bycheckPathLength, bypassing pre-flight validation and failing only when the OS filesystem attempted the lookup. - Ancestor directory existence sensitivity: When the parent prefix directory (
config/iam/users/) did not exist yet on disk, the kernel returnedENOENTwhile resolving the missing parent directory—before reaching the overlong<normalized-DN>component—allowing login to succeed (HTTP 200). Once any user record created theconfig/iam/users/prefix directory, the kernel reached the 271-byte component and returnedENAMETOOLONG, causing login to fail withHTTP 500.
| Commit | Role in the Failure Chain |
|---|---|
MinIO ee9077db (May 2020) |
Introduced Unix backslash skipping in checkPathLength. |
MinIO 280526ca (Mar 2024) |
Propagated policy lookup read errors after a cache miss instead of swallowing them. |
MinIO efb73fa5 & 7e45d84a (Mar 2024) |
Adopted NormalizeDN (with \xx hex expansion) for LDAP identity and policy-mapping keys. |
SILO 709d50a9 (Server 20260916) |
Added persistent parent-revision reads (config/iam/users/<DN>/identity.json) during login for revocation and replay protection, triggering the read on every LDAP login even when the policy cache hit. |
The Fix: Narrowly Classify Impossible ASCII Components as Absent
The fix is implemented in the shared object-storage IAM reader, loadIAMConfigBytesWithMetadata. It attempts the normal storage read first. Only if the read fails does it translate the error to errConfigNotFound, and only when all three conditions hold:
- The returned error is
errFileNameTooLongorObjectNameInvalid(including wrapped errors). - The complete object key passes standard bucket and object-name validation (
isValidBucketNameandisValidObjectName). - At least one
/-delimited path component consists entirely of ASCII characters (utf8.RuneCountInString(component) == len(component)) and exceeds 255 bytes (len(component) > 255).
Because a pure-ASCII path component longer than 255 bytes can never be written to the object-storage IAM backend in the first place, no persisted user policy or parent-revision record can exist at that key. Returning errConfigNotFound allows the caller to follow its normal record-absent path—for example, authenticating the LDAP user and evaluating permissions strictly from their LDAP group policies—without granting any default permissions or bypassing revocation checks for storable keys.
Writes, deletes, the etcd IAM backend, and global object path validation remain unchanged. The production change adds 25 lines in cmd/iam-object-store.go accompanied by 318 lines of regression tests, with zero extra I/O on normal reads.
Why the Initial Error Classification Required Narrowing
The initial patch treated any ObjectNameInvalid / errFileNameTooLong error on a syntactically valid key as errConfigNotFound. Design review identified two concrete counterexamples where a length error does not prove that the record never existed on disk:
- Full-path length overflow (
PATH_MAX) hiding an existing revocation: Suppose an administrator revokes an identity whose individual path components are $\le 255$ bytes (so the revocation tombstone is successfully written to disk), and the server’s data directory is later mounted or moved under a deeper parent path on macOS. A read can then fail due to the total path length limit even though the revocation record exists on disk. Treating that error aserrConfigNotFoundwould hide the revocation tombstone and allow a revoked parent identity to pass verification. Checkinglen(component) > 255on individual/-delimited components ensures full-path overflow errors still fail closed. - Multibyte UTF-8 filenames on character-counted filesystems (macOS APFS/HFS+): On macOS, filesystem component limits are enforced in UTF-16 code units / characters (up to 255 characters) rather than raw UTF-8 bytes. In filesystem boundary testing on macOS, a single component consisting of 100 copies of
界occupies 300 UTF-8 bytes (len(component) == 300,RuneCount == 100) and can be created and persisted on disk. If the guard checked onlylen(component) > 255, an existing 300-byte Unicode record could be misclassified as impossible. Requiringutf8.RuneCountInString(component) == len(component)restricts the absence classification to pure-ASCII components (which includes all\xx-escaped DNs), where 1 character is always 1 byte on every platform.
The resulting safety invariant is exact: only a path component that is provably uncreatable may take the errConfigNotFound branch; any potentially existing record that cannot be read must fail closed.
Alternatives Considered
| Candidate Approach | Decision and Rationale |
|---|---|
Narrow ASCII component (> 255 B) classification in loadIAMConfigBytesWithMetadata |
Selected. Restores LDAP login and STS lookup across all callers using the shared IAM reader while failing closed on full-path errors and storable Unicode names. |
Treat every ObjectNameInvalid on a valid key as errConfigNotFound |
Rejected. Full-path (PATH_MAX) errors on deep mount points could hide an existing revocation tombstone on disk. |
Count decoded UTF-8 characters or ignore \xx escape overhead |
Rejected. The OS filesystem stores the literal \xx-expanded ASCII string, not the unescaped UTF-8 runes. |
Fix only Unix backslash counting in checkPathLength |
Insufficient. Counting backslashes in checkPathLength would merely reject the same long DN earlier (with ObjectNameInvalid) without restoring login. |
Change NormalizeDN to preserve raw UTF-8 instead of \xx hex escapes |
Deferred. Would incompatibly alter existing persisted LDAP identity and policy-mapping keys on disk, and still would not eliminate component limits for genuinely long DNs. |
Skip parent-revision checks for LDAP or revert 709d50a9 |
Rejected. Would sacrifice durable revocation and replay protection while leaving the older March 2024 policy-mapping read failure (sts-users/<DN>.json) unfixed. |
| Hash overlong DNs into fixed-length storage keys | Separate feature scope. Full write support for arbitrary-length DNs requires a new storage schema across lookup, listing, import/export, site replication, and mixed-version upgrades. |
Verification Matrix
| Test Suite / Probe | Coverage and Results |
|---|---|
Unit & boundary tests (cmd/iam-revision_test.go) |
Covers escaped 271-byte and 384-byte DNs, exact 255/256-byte component boundaries, .json suffix accounting, multibyte Unicode components, wrapped errors, invalid object names, and existing revocation records moved under deep directory paths. |
End-to-end LDAP STS login (cmd/sts-handlers_test.go) |
Verifies STS login, signed S3 access, and service-account operations for a group-authorized user with a 384-byte normalized DN across 8 object/etcd configurations. |
| Cross-platform filesystem probes | Tested 72 physical filename shapes on macOS (49 creatable) and Linux tmpfs (32 creatable): zero storable filenames were classified as impossible. Verified 105 error-propagation assertions (5 parent-name classes $\times$ 7 storage errors $\times$ 3 entry points) per platform. |
| CI & Toolchain VulnCheck | All 11 PR CI jobs on 3df51549bbbd6e57a6a5f11ac77535390517db65 (merged as a07d655dc69cf04b130c2634d9fa5301078af546) and 7 post-merge main jobs passed on d43adc0f2 (Go CI, VulnCheck), alongside raising the minimum toolchain to Go 1.27.3 for GO-2026-6809. |
Operational Boundaries and Guidance
- Read vs. Write asymmetry: This fix restores read/login paths when optional per-user IAM records do not exist. It does not enable the object-storage IAM backend to write files whose component length exceeds 255 bytes. Attaching a policy directly to a >250-byte normalized user DN (
sts-users/<DN>.json) or writing a persistent parent-revision tombstone (users/<DN>/identity.json) on the object backend will still fail with a filename-length error. - Recommended LDAP authorization pattern: For LDAP users with long or non-ASCII DNs on object-backed IAM deployments, grant permissions through LDAP group mappings (where the group DN itself fits within 250 ASCII bytes after normalization) rather than direct per-user policy attachments.
- No migration required: Upgrading requires no on-disk data migration. Never delete
.minio.sys/config/iam/directories manually to work around path-length errors; follow the standard IAM upgrade and recovery guide.
4.14 - Replicated Tag Ordering: Revision Timestamps, Tombstones, and Resurrection
Shipped in SILO Server 20260916 (40220bd836cbd066ca424fa4dc5dbb90057fb55a), PR #193 (commit 03027727d) and PR #196 (commit 680eac66e) fix two defects in how object tags preserve ordering across bucket replication: SSE-KMS destination copies silently dropping tag revision timestamps (R4), and empty tag values lacking a revision timestamp so delayed replication events could resurrect deleted tags (R5).
The Shared Model: A Tag Value and Its Revision Form One State
Object tags replicate alongside an internal revision timestamp (transmitted over the wire as X-Minio-Source-Tagging-Timestamp and stored in xl.meta as x-minio-internal-tagging-timestamp). On the receiving cluster, reconcileStoredObjectTags enforces a simple ordering rule under the object write lock: an incoming replicated tag state wins only when its revision timestamp is strictly newer than the stored revision timestamp.
Both defects broke that invariant by dropping the revision timestamp rather than the tag value:
- R4 (PR #193): Dropped
ReplicationSourceTaggingTimestampwhen constructingObjectOptionson an SSE-KMS destination, causing newer incoming tag updates to lose to older stored state during storage-layer reconciliation. - R5 (PR #196): Treated an empty tag string (a tag deletion) as having no revision timestamp at all, leaving the replication protocol unable to express “tags deleted at time $T$” and allowing delayed pre-delete replication events to resurrect deleted tags.
R4: SSE-KMS Destination Copies Dropped Tag Revision Timestamps
Failure mode: A trusted replication COPY to an SSE-KMS encrypted destination returned HTTP 200 OK, encrypted the object properly, and served plaintext GET requests—yet the destination object’s tags and tag revision timestamp remained stuck at their previous values. Because the HTTP request succeeded, no replication error or MRF retry was triggered. Inside reconcileStoredObjectTags, the missing incoming timestamp caused the older stored tag state to win every comparison.
Trigger surface: This path is entered not only when a request carries explicit SSE-KMS headers, but also whenever the destination bucket has default SSE-KMS configured or the cluster enables global auto-encryption.
Root cause: In putOptsFromReq, the option builder parsed the trusted X-Minio-Source-Tagging-Timestamp header into ObjectOptions. Immediately afterward, the SSE-KMS branch constructed and returned a fresh ObjectOptions struct literal copying MTime, ETag, replication trust flags, and the two Object Lock timestamps—while omitting ReplicationSourceTaggingTimestamp. That omission originated in upstream commit c4373ef290 (2021) and survived a 2026 Object Lock patch that added two retention/legal-hold timestamps to the same struct literal.
Fix (03027727d): Forwards ReplicationSourceTaggingTimestamp: opts.ReplicationSourceTaggingTimestamp in the SSE-KMS ObjectOptions literal. Regression tests verify the full cartesian matrix of destination encryption modes (none, SSE-S3, SSE-KMS with and without key context, SSE-C), trusted vs. untrusted callers, missing/valid/malformed timestamps, and 50 signed ordered COPY+GET cycles with 1–3 ns spacing across single-drive and 16-drive erasure backends.
Note on adjacent SSE-KMS struct fields: The same SSE-KMS ObjectOptions literal also omits the ProxyRequest and Speedtest fields; that separate observation is tracked independently outside this replication fix.
R5: Empty Tag Values Carried No Revision, Allowing Deleted Tags to Resurrect
Failure mode: Baseline testing uncovered nine distinct failure cases across single-pool and multi-pool storage backends:
DeleteObjectTaggingnever minted a new revision timestamp, so a delayed metadataCOPYcarrying an older tag snapshot overwrote the deletion and restored the old tags.- An incoming replication
COPYcarrying an empty tag set was treated as “no tag update supplied” rather than “tags cleared at timestamp $T$.” - An initial replica
PUTparsedX-Minio-Source-Tagging-Timestampfrom the wire but never persisted it toxl.meta. - When source and target visible tag strings were both empty (or equal), the replication sender skipped metadata replication because
HEADdoes not expose the internal tag revision timestamp, leaving newer deletions or re-additions unreplicated. - When a queued replication event completed, its local status callback wrote the snapshot’s old tags back into local metadata over a newer committed
DeleteObjectTagging—and because the deletion had minted a timestamp in early prototypes or lacked one, the resurrected tags inherited the wrong revision state.
Root cause: A tag set and its revision timestamp (including the timestamp of an empty tag set) form a single versioned register. The legacy implementation only stamped, transmitted, and reconciled timestamps when the tag string was non-empty.
Fix (680eac66e):
- Every tagging mutation mints a monotonic revision:
PutObjectTaggingandDeleteObjectTaggingunconditionally stamp a UTCRFC3339Nanorevision timestamp regardless of whether replication is currently active on the bucket. Under the object write lock, the storage layer enforces strict monotonicity: if a new local timestamp is $\le$ the stored revision, it advances to $\text{stored} + 1\text{ ns}$; on multi-pool clusters, the coordinator computes a unified timestamp strictly greater than the maximum revision across all pools. - Sender transmits tombstones: Empty tag sets with a recorded revision timestamp transmit
X-Minio-Source-Tagging-Timestampas a deletion tombstone. Objects that never had tags (empty value and no recorded revision) do not fabricate a tombstone. If a stored timestamp is malformed on disk, the sender fails closed and retries via MRF until an explicit tag write replaces it. - Receiver reconciles tombstones: Replication
COPYcaptures the existing(tags, timestamp)pair before rebuilding metadata so a timestamped empty value participates inreconcileStoredObjectTagsand clears older stored tags. Internal duplicate-version suppression (412 PreconditionFailed) is relaxed when the incoming trusted tag revision is strictly newer than the stored revision (while clientIf-MatchandIf-None-Matchheaders remain strictly enforced). Multipart completion orders tags using the persisted upload metadata, the replication completion callback no longer writes snapshot tags back over local state, and ordinary SSE-C key rotation strips old tag timestamps from copied encryption metadata so a rotation mints a clean local revision.
Operator-Visible Changes and Upgrade Requirements
- Equal-timestamp ties resolve to stored-wins: Both unversioned and explicit
null-versionCOPYrequests now keep the stored destination state when incoming and stored tag timestamps are identical. - Ordinary
COPYwithX-Amz-Tagging-Directive: REPLACEand empty tags: Now clears destination tags as required by S3, instead of falling back to copying source tags. - Coordinated two-ended upgrade required: Both source and target clusters in a replication pair must be upgraded to SILO Server 20260916 (see also the IAM and replication upgrade guide). An unupgraded peer still ignores empty-value revision headers.
- One additional metadata
COPYper tagged object during resync/heal: BecauseHEADdoes not expose internal tag revision headers throughminio-go, explicit resync/heal issues one metadataCOPYfor objects that carry a recorded tag revision. On destination buckets with default or global SSE-KMS encryption, that metadataCOPYrewrites the object payload—account for this I/O when scheduling bulk resyncs. - Scope boundaries: Historical missing or overwritten deletion timestamps are not reconstructed automatically; repair affected objects by issuing a fresh
PutObjectTaggingorDeleteObjectTaggingon the authoritative source. Tag-filtered replication rules continue to evaluate target eligibility against the post-mutation tag set, and per-hop timestamp comparison relies on synchronized wall clocks (NTP) across sites.
Rejected Alternatives
- Synthesize a tombstone from
ModTimefor untagged objects: Would assign a synthetic tag revision to every object ever written, forcing every untagged object through an extra metadataCOPYon every replication hop. - Transmit the revision header only when the tag value is empty: Withdrawn during review because the sequence “set tag
Xat $T_1$ $\to$ delete at $T_2$ $\to$ re-add identical tagXat $T_3$” loses the $T_3$ revision if non-empty updates omit the timestamp when visible values match (TestTaggingRepeatedValueNeedsRevisionDelivery). - Extend the
HEADwire protocol to return internal tag revisions: The pinnedminio-gometadata reader strips internalX-Minio-*response headers, so exposing revisions onHEADwould require a cross-SDK wire change while still needing a metadataCOPYwhenever revisions differ. - Introduce vector clocks or regenerate timestamps at storage commit: A wall-clock
RFC3339Nanotimestamp paired with the+1 nsmonotonic floor under the object write lock achieves per-key monotonicity without changing on-disk formats. - Enforce monotonicity per pool instead of across all pools: Because ordinary reads can return metadata from a single pool, per-pool monotonic counters could still emit a timestamp older than a sibling pool’s copy; computing the maximum across all pools under the namespace lock prevents cross-pool regression.
Verification
R4 and R5 regression suites cover the full encryption $\times$ trust $\times$ timestamp matrix across single-drive, 16-drive erasure, and multi-pool erasure backends (with all 11 CI checks passing on PR #196).
See the component version matrix for release alignment and Replica Metadata Normalization for the companion metadata extraction fix. Related records: Replicated Tag Ordering · Replica Metadata Normalization · Request Header Timeouts · Replica Metadata Audit.
4.15 - SILO Server 20260903 Pre-release Review
This is the pre-release engineering review behind SILO 20260903 (RELEASE.2026-09-03T13-18-01Z at 9b11dc9469e650815b775cb47b039610644f5da4, reviewed candidate 6e112d1856d4f3655f30fc81ee47e9f43d50d8f3, against baseline RELEASE.2026-08-06T00-00-00Z at 3be10fcc1a44f6620ded0bd303461f9d688cca23). It records what the second-pass review found after the initial implementation pass, how the fixes were narrowed, and how the release gates were structured. Subsequent September and October releases are covered in Replication Reliability, Server 20260916, and Server 20261018.
Why the Second Review Was Necessary
The first implementation pass resolved most reported defects and passed the existing test suite, but passing existing tests did not prove that every security invariant held across all valid wire representations and edge states.
The second-pass review tested each change against seven questions:
- Can the same invariant be bypassed by a different valid wire representation?
- Does a pre-authentication fast path still perform I/O or acquire state?
- What happens when metadata exists but cannot be loaded?
- Do two individually correct read-modify-write paths share the same serialization boundary?
- Does request sanitization preserve all SigV4 streaming state?
- Does a validation record describe the final tree or an earlier commit?
- Is a complex mechanism protecting a reproduced failure, or only a hypothetical future?
That pass uncovered real defects after the first pass and tied every conclusion to a specific invariant and commit.
Review Result by Area
| Area | Second-pass finding | Final resolution | Status |
|---|---|---|---|
| Bucket metadata | Independent config locks could lose updates to the shared .metadata.bin record (#102) |
One bounded metadata.lock surrounds every whole-record writer, migration, import, adoption, and healing path; changed-field replication avoids stale whole-record replacement |
Closed in candidate |
| Bucket creation | ForceCreate and site adoption could replace existing config with defaults |
Preserve existing records and update only creation/adoption state; add regression tests for clobbering | Closed in candidate |
| Object Lock | Comparing lock-document bytes with one canonical XML document missed valid configurations carrying a Default Retention rule | Parse Object Lock first, then derive the versioning invariant from the parsed enabled state; verify update, read-back, and disk reload | Closed in candidate |
| Pre-auth CORS | Arbitrary path segments could cause metadata reads and cache growth | CORS lookup reads resident metadata only and does no object-layer I/O | Closed in candidate |
| CORS startup | A nonresident name could fall back to global CORS before metadata initialization | Preserve an explicit fail-closed startup state | Closed in candidate |
| CORS load failure | Forgetting that a real bucket failed to load made it indistinguishable from a nonexistent bucket and exposed the global fallback to pre-signed requests | Maintain a bounded failed-bucket set, clear it on every successful load/remove/refresh path, and keep those buckets fail-closed | Closed in candidate |
| CORS recovery | A successful on-demand GetConfig reload did not initially clear the load-failure bit |
One-line final fix 84e1580a4 plus targeted race coverage |
Closed in candidate |
| Replication trust | Presence of client-controlled internal headers enabled privileged behavior in multiple handlers | Authenticate first; require an exact marker plus s3:ReplicateObject or s3:ReplicateDelete; carry a private context decision; sanitize untrusted headers afterward |
Closed in candidate |
| Streaming uploads | The sanitized request clone did not initially share the original trailer map | Preserve the trailer map so late-arriving streaming checksums remain visible | Closed in candidate |
| Snowball | A request-wide trust bit could leak between extracted entries | Derive and isolate trust per entry; preserve request defaults across workers | Closed in candidate |
| SSE-C | Zero-byte reads and GetObjectAttributes could skip customer-key authentication |
Require a successfully unsealed key, with a separate authorized-replica exception | Closed in candidate |
| Delete authorization | Explicit version deletes checked the ordinary delete action instead of requiring s3:DeleteObjectVersion |
Align single and multi-delete authorization, keep replication deletes on s3:ReplicateDelete, and preserve auth/audit context |
Closed in candidate |
| Admin authorization | User/group status changes always checked the enable action | Check the action that matches the target state | Closed in candidate |
| Checksums | Multipart and copy paths omitted fields, accepted invalid combinations, or computed over the wrong representation | Complete algorithm/type validation, server-side part calculation, federated propagation, AWS errors, and CopyObject transform ordering | Closed in candidate |
| Release evidence | Full acceptance initially described a tree that changed afterward | Record full acceptance at ebac0ca73 and current-tree targeted gates separately |
Closed as an evidence defect |
The Invariants That Define the Candidate
Trust Is Derived Once, After Authentication
An internal-looking header is still client input. The request must first pass the existing authentication path in its original signed form. Only then can the handler combine:
- an exact, single replication marker;
- a non-anonymous authenticated identity;
s3:ReplicateObjectors3:ReplicateDeleteon the addressed resource;- replica status where the narrower replica-only semantics require it.
The result lives in private request context. Header stripping is defense in depth for legacy consumers, not the source of authority.
This ordering matters because SigV4 may sign the headers. Sanitizing first would reject legitimate replication with SignatureDoesNotMatch. The sanitized clone also has to share the request trailer: trailers arrive after the initial header parse and carry streaming checksums.
The complete receiver-wide model is in No I/O Before Auth, No Privilege From Headers.
A Shared Record Has One Write Boundary
Policy, lifecycle, SSE, tags, quota, replication, Object Lock, versioning, and CORS are logical fields but physical members of one bucket record. A per-field mutex cannot protect a whole-record read-modify-write.
Rather than introducing a separate database or transaction layer, the fix serializes bucket-record updates under one bounded lock:
The lock does not cover object data I/O and is bounded to a bucket-metadata operation. Migration and healing participate because they also replace the whole record. Replication receivers merge only changed fields so an older remote snapshot cannot erase unrelated local state.
Distinguishing CORS Load Failure from Absence
The CORS hot path must distinguish four states:
| State | Result |
|---|---|
| Metadata system not initialized | No CORS headers |
| Known real bucket whose metadata load failed | No CORS headers |
| Resident bucket with a bucket CORS document | Evaluate that document |
| No resident metadata and no known failure | Use the server-wide fallback |
The second row is why the failed-bucket set is required. A pre-signed URL is already authorized by its signature and may access a private object without bucket-policy evaluation. In that case the bucket CORS document is the browser-origin boundary. Losing the failure bit and using a permissive global fallback would weaken that boundary.
The set remains bounded by real bucket load attempts and is maintained through noteLoadFailure and clearLoadFailure. Successful load, removal, stale-bucket cleanup, refresh, reset, and concurrent load all have tests.
Semantic Object Lock Normalization
Any valid enabled Object Lock configuration implies versioning. XML whitespace, element order, and the presence of a Default Retention rule do not change that meaning. Normalization therefore follows XML parsing rather than a byte comparison against one canonical document.
The resulting versioning record is plain Enabled. A suspended state and an exclude-prefix extension are incompatible with the lock invariant and are removed on update, read-back, and reload.
Complexity Audit
The pre-release pass audited the diff for over-design, duplication, defensive code without a threat model, and stale compatibility machinery.
Complexity Retained Because It Protects a Reproduced Failure
- One metadata lock: a deterministic cross-type lost-update test reproduced data loss.
- CORS tombstones: site replication cannot distinguish deletion from “never observed” without them.
- CORS load-failure state: a pre-signed URL provides an authenticated, policy-independent counterexample.
- Two replication trust levels: ordinary replication and replica-ciphertext/SSE semantics do not use identical wire shapes.
- Post-authentication sanitization: sanitizing before SigV4 verification breaks legitimate signed requests.
- Multi-pool/null-version edge-case tests: single-pool happy paths do not exercise the state-selection failures they caught.
Complexity Removed or Narrowed
- CORS failure-set mutations were centralized in
noteLoadFailureandclearLoadFailure. - The replication import path applies changed fields rather than copying an entire possibly stale record.
- Obsolete encryption helpers, dead event-target functions, and abandoned handler branches were deleted.
- The compatibility guard stopped inventorying every exported source symbol and now protects the actual served routes and frozen wire/configuration surfaces.
- The old
wait_pipelint exemption was removed;gomodguard_v2replaced deprecated configuration. - Dynamic timeout tests no longer call global
rand.Seedfrom a parallel package. - The server returned from the temporary
silo-gofork to the reviewed upstream-compatibleminio-gorevision.
Changes Deliberately Not Introduced
- no generic metadata transaction framework;
- no second CORS cache or unbounded negative cache;
- no new public “trusted replication” request header;
- no cross-repository release gate that makes the server depend on a later Console or documentation release;
- no partial conditional-delete contract in the release candidate;
- no broad rewrite of inherited site-replication registers without dedicated convergence tests.
Deferrals and Why They Do Not All Have the Same Severity
| Item | Classification | 20260903 release decision (and subsequent status) |
|---|---|---|
| Conditional delete #10 | Inherited missing S3 feature; affects callers that assume If-Match / per-object ETag is enforced |
Documented for 0903; single-object conditional delete later shipped in 20260916 (#145), while batch ETag deletion remains open |
| Multi-site config deletion #77 | Inherited convergence defect for policy/SSE/tags/quota; CORS has its own fixed register | Documented deployment condition for 0903; later repaired in 20260916 |
ListMultipartUploads #79 |
Inherited listing-conformance gap | Known issue; not a data-integrity blocker for ordinary multipart workflows |
Federated CopyObject #99, #100 |
Legacy-backend checksum/inline-object gaps | Scoped to affected federated features in 0903; later repaired in 20260916 |
| ILM relocation PR #60 and broad SSE issue #61 | New capability requests | Outside the release safety boundary |
Evidence
Full Acceptance Tree
The full local acceptance run targeted ebac0ca73bbf251b070bb6df4d8005015841f901:
- full
cmdandinternalsuites; - complete
cmdrace suite: 365.448 seconds, pass; - lint: 0 issues;
- rebrand/compatibility and generated-file guards;
govulncheckwith no reachable vulnerability;- six
make verifydeployment shapes: 174 PASS / 0 FAIL.
The first two make verify runs hit environment/setup issues while fetching mcli, not test failures. The passing run used the locally checksum-pinned mcli, retained the outbound proxy for GitHub downloads, bypassed it for localhost, and placed GNU userland tools first in PATH.
Post-Acceptance Candidate
The only code change after that full run was 84e1580a4, which clears one CORS failure-state bit after a successful on-demand metadata reload. The candidate merge added no code; 6e112d185 changed only Helm release metadata and documentation. On the final candidate, the following checks passed:
git diff --check;- targeted CORS and Object Lock
go test -race; - rebrand guard;
- generated-file check;
- lint with 0 issues;
- Helm lint, default and optional renders, chart packaging, and the seven-resource legacy-upgrade identity guard.
Go, No-Go, and Ownership of the Release Gates
Code Decision: GO
All confirmed code defects from both review rounds were resolved in the candidate. Each fix was verified at the layer that owns its invariant, and every retained mechanism corresponds to a reproduced failure case.
Production Release Gates
Before publishing RELEASE.2026-09-03T13-18-01Z, the candidate cleared seven release gates:
- candidate commits pushed and reviewed;
- remote CI and Test Release passing on the pushed head;
- the release tag pointing at the reviewed chart 7.0.2 / server 0903 / client 0903 release tree;
- draft artifacts, checksums, SBOMs, attestations, and signed RPMs verified;
- finalize and Docker release publishing both classic and distroless variants;
- anonymous download and pull tests passing;
- release notes updated from the tagged facts and documentation deployed.
Deployment Considerations for 20260903
Operators upgrading to 20260903 must account for three requirements:
- update every node in a distributed cluster within one coordinated maintenance window;
- update every member of a site-replication group before configuring bucket CORS;
- update IAM policies for
s3:DeleteObjectVersionand user/group status-action separation.
Restart and Readback Verification (2026-09-11, #116)
A follow-up acceptance test (#116, run on 2026-09-11) measured bounded restart and readback behavior across four dimensions:
- Acknowledgement ledger. Every acknowledged PUT writes to a unique versioned key, and the acknowledgement records the
VersionId, byte count, and SHA-256 immediately. Readback fetches by exactVersionIdand asserts all three, ensuring an early confirmation cannot be silently replaced by a later write. - Readback timing. Periodic re-reads run at 15, 30, and 60 seconds after the data canary succeeds, across every peer, and the final check covers writes made both during a single-node outage and after its rejoin.
- No retry masking. Each canary carries a hard 60-second deadline covering setup, request, response-body read, and sleep, with SDK retries disabled so a recovery window cannot be masked by client-side retries.
- Driver topology. Four Linux/arm64 containers on one host with independent network identities, one drive each (EC 2+2), and
tmpfsvolumes kept mounted by a holder container through the full shutdown; nodes stop in parallel with a 10-second grace period, then start in parallel.
Admin-endpoint readiness is not data readiness. The historical test record compared 0806, 0903, and the main repair build. The main build passed the admin readiness check 2.461 s after restart, then required another 14.489 s to complete the first data canary. During that startup window, admin/health does not reflect data-plane readiness; upgrade automation must verify actual data-plane operations rather than relying on a fixed sleep.
See the current component/source matrix for the status of all subsequent repairs.
4.16 - Overwrite Pool Affinity: Why #245 Was Not Adopted
Decision, October 24, 2026: do not merge the tested pool-affinity changes (PR #245). The experiment addressed a real space-efficiency problem, but neither the broad policy nor the narrower unversioned-only policy achieved an acceptable performance trade-off. Both experimental branches (codex/healthy-pool-affinity and codex/unversioned-pool-affinity) were retained for reference while keeping main unchanged.
Denis Babchuk proposed keeping overwrites in the pool that already holds the object. Review identified and fixed a request-specific write-quorum bug, tightened the health policy, and validated the behavior. Performance measurements then changed the recommendation: the broad policy made deep version histories substantially more expensive, and a second iteration restricted to ordinary unversioned overwrites still regressed small-object writes.
The subsequent multi-pool deletion fix (#257) was merged independently on October 24: while retaining capacity-weighted allocation, it cleans up non-authoritative copies before deleting the authoritative copy so a mid-delete failure cannot expose an overwritten value.
The problem: an overwrite can leave its old copy in another pool
Consider an unversioned key, reports/latest.json, in pool A. A subsequent PUT writes its replacement to pool B. Reads select the newest logical object, while the old physical copy in A continues to occupy space because ordinary overwrites do not clean up obsolete copies across other pools.
The baseline looks up the existing object and then sends a readable result through capacity-weighted pool selection again. Depending on pool usage, repeated writes can eventually leave a pool-local record (xl.meta) in both pools.
This behavior has a specific history:
- Upstream MinIO issue #21087 reported that single-pool uploads could bypass the minimum free-space threshold. PR #21115, merged on July 19, 2025 as
83b2ad418b, enforced capacity checks by reselecting a pool for readable existing objects, which dropped their earlier placement affinity. - SILO issue #199 and PR #207 evaluate conditional writes against the current logical object across pools while keeping the capacity-weighted allocation policy.
- Pool migration uses separate safeguards against stale payloads and delete-marker loss; see Multi-Pool Object Consistency.
The space benefit of overwrite affinity applies specifically to multi-pool deployments with repeated overwrites of the same unversioned keys. Single-pool deployments have no cross-pool duplicates, new keys still require capacity-weighted allocation across pools, and versioned buckets must retain their historical versions regardless of which pool stores them.
What the original PR changed
The proposal reused the existing object lookup and lock boundary. If the latest readable object was already present, its pool became the preferred destination, subject to write quorum and the existing hard free-space check. New keys retained capacity-weighted allocation. When the existing object’s set could not accept a write, allocation excluded that pool before considering the remaining pools.
The production change in cmd/erasure-server-pool.go added 73 lines and removed 3 lines, accompanied by 331 lines of placement tests, covering PUT and multipart initialization on the shared allocation path.
Two capacity rules apply during selection:
- Soft allocation reserve (15%): when alternative pools exist, the allocator filters out pools above roughly 85% usage so emptier pools fill first.
- Hard space check (
hasSpaceFor): checks write size, per-drive capacity, and other constraints againstdiskFillFraction(0.99).
Affinity bypasses the soft reserve when the existing object’s pool is healthy and has room, while preserving the hard capacity check. After a pool expansion, existing keys continue updating in their older, fuller pool rather than migrating to the new pool on overwrite.
Simply deleting the condition that reselects existing objects would lose the capacity checks and fallback introduced in 83b2ad418b. Checking space alone is also insufficient: an erasure set can still have read quorum for an old object while lacking enough online drives to reach write quorum for its replacement. Because pool selection runs before consuming the request body, an unreplayable upload stream cannot be transparently retried in another pool after a write failure.
Correctness review and implementation improvements
Write quorum must match the request
The original helper inferred quorum from the STANDARD storage class for every write. A controlled two-pool test exposed a regression when overwriting an object with REDUCED_REDUNDANCY (RRS):
| Condition | Test setup |
|---|---|
| Storage policy | optimize=capacity; STANDARD=EC:4, REDUCED_REDUNDANCY=EC:2 |
| Existing object | Created with ordinary STANDARD parity |
| Existing object’s set | 13 of 16 drives online; reported usage 90% |
| Other pool | 16 drives online, ample space; an actual RRS probe write succeeded |
| New request | Overwrite using RRS, through PUT or new multipart initialization |
The helper required 12 writers for EC:4 and selected the original pool, whereas the actual EC:2 write required 14 online drives and failed with InsufficientWriteQuorum (HTTP 503). Because lower parity leaves more data shards, RRS requires more online drives for a write. The baseline’s reserve filter skipped the 90%-full pool, selected the healthy second pool, and succeeded across all four pool/entry-point combinations (12/12 attempts failed on the unpatched PR).
The inverse mismatch occurred with PUT’s internal MaxParity option: a write eligible for EC:8 was evaluated against EC:4 and unnecessarily moved to another pool.
A healthy preference, with explicit fallback
The updated implementation passed the request’s storage class and PUT MaxParity into placement, matching the writer’s availability/capacity parity rules (including the extra drive required when data and parity shard counts are equal) while preserving multipart initialization parity semantics:
| Existing object’s target set | Action |
|---|---|
| Fully healthy, writable for this request, and enough space | Prefer that pool |
| Degraded but still writable, with space | Use the existing capacity allocator; the original pool remains a candidate |
| Known to be unwritable or lacking space | Exclude it before reserve filtering, then select another pool |
| No alternative has space, but the original set has space and lacks write quorum | Let the original set report InsufficientWriteQuorum rather than misreporting StorageFull |
| No usable space | Preserve the StorageFull result |
Health and capacity checks shared a single set-level DiskInfo snapshot. A drive counted as a usable writer only if it was online and returned a positive capacity result; drives with a failed DiskInfo call were treated as unhealthy.
Regression tests were added for request parity, known disk faults, degraded sets, scan counts, heterogeneous set sizes, latest-object selection, and existing multipart sessions. The multipart fallback test was also updated to seed its own initial object rather than reusing a prior PUT target that had already moved pools.
The second design: only ordinary unversioned overwrites
After the first performance round showed high version-history rewrite costs, a second branch (codex/unversioned-pool-affinity) narrowed the policy to ordinary unversioned overwrites:
| Request or bucket state | Narrowed behavior |
|---|---|
Ordinary unversioned PUT replacing a readable object |
Apply the healthy-pool preference above |
| New multipart upload over an ordinary unversioned object | Apply it only when initialization reaches pool selection; preserve existing upload binding |
| Versioning enabled or suspended | Retain baseline allocation |
Explicit version ID, including null |
Exclude from affinity |
| Replication, replica-lock reconciliation, or data movement | Exclude from affinity |
| Internal metadata buckets | Exclude from affinity |
| New keys or a single pool | Retain the original path and capacity behavior |
| Existing multipart sessions, ordinary COPY routing, latest delete markers, or read-quorum-error routing | Preserve their existing paths |
Eligibility checked both the bucket’s versioning configuration and the request options. A version-enabled bucket with prefix-excluded unversioned writes, or a version-suspended bucket with mixed history, remained on the baseline allocator.
Functional validation, separated by revision
| Stage | Evidence | Scope |
|---|---|---|
| Initial PR review | Base/head comparison reproduced normal affinity, STANDARD fallback, the RRS failure, and the MaxParity mismatch |
Object-layer unit tests on macOS |
| Improved full implementation | Full Go test suite: 52 packages, 1,823 top-level tests (8,233 with subtests) passed, 12 skipped; targeted -race suite: 35 top-level tests (201 with subtests) passed |
Full-affinity branch on an isolated APFS test volume |
| Full implementation through mcli | mcli 20261018 verified ordinary overwrite/readback, 4 distinct retained versions with historical reads, and multipart overwrite/checksum |
Single-host local-directory deployment |
| Narrowed implementation | Linux targeted suite: 37 top-level tests (209 with subtests) passed across placement, parity, health/capacity fallback, version-state exclusions, PUT/MPU, and conditional writes |
Targeted placement and overwrite suite |
| Build and static checks | Linux amd64 build, make verifiers, lint (0 issues), and whitespace checks passed |
Repository static verification |
The targeted regression filter was:
Benchmark method and what the numbers mean
Both performance rounds ran on m3 (AMD Ryzen AI MAX+ 395, 32 logical CPUs, 61 GiB RAM, Linux 7.0.0-38, CT2000P310SSD8 NVMe with XFS). Two pools each used 16 directories and one erasure set (EC 12+4), all backed by the same XFS filesystem on the shared NVMe SSD.
Both variants were built with Go 1.27.3 (linux/amd64, CGO_ENABLED=0). The server ran on CPUs 4–11 (GOMAXPROCS=8), and the minio-go v7.4.1 benchmark client ran on CPUs 12–15 (GOMAXPROCS=4) over loopback HTTP.
- Throughput: completed object operations per second (for multipart, one operation includes initialization, two part uploads, and completion).
- P50 / P99: medians across runs of each run’s 50th and 99th percentile request latencies.
- CPU cost: server CPU time per completed operation.
- Block I/O: process-accounted reads and writes from
/proc/PID/io. - Active-bucket space: object files, metadata, and directory overhead excluding temporary files.
Round one: the full policy cost more than expected
Round one compared the full candidate (0aa870115d7d) against the baseline (789a2327ba8c) across 54 measured runs (243,718 completed operations, 0 failures). Basic workloads ran 3 pairs of 15-second runs, with 2 additional 60-second pairs for new-key and 4 KiB overwrite workloads, 3 fixed-count runs of 4,000 versions on a single seeded object, and a 4-run ABBA binary-switch sequence on existing dual-copy data:
| Workload | Baseline → full candidate, operations/s | Change | P50, ms | P99, ms | Sample |
|---|---|---|---|---|---|
| Two pools, 4 KiB overwrite; 256 keys, concurrency 16 | 236.9 → 174.5 | −26.3% | 30.4 → 74.4 | 277.9 → 307.4 | 2 × 60 s per variant |
| Two pools, 1 MiB overwrite; 64 keys, concurrency 8 | 140.9 → 132.2 | −6.2% | 34.2 → 41.3 | 169.3 → 176.2 | 3 × 15 s |
| Two pools, 16 MiB multipart overwrite; 16 keys, concurrency 4 | 25.8 → 26.4 | +2.1% | 144.0 → 141.7 | 265.3 → 272.7 | 3 × 15 s |
| Two pools, one 4 KiB key; concurrency 1 | 135.2 → 136.4 | +1.0% | 6.9 → 6.8 | 12.0 → 10.6 | 3 × 15 s |
| Two pools, 4 KiB new keys; concurrency 16 | 290.7 → 296.8 | +2.1% | 46.9 → 48.7 | 170.2 → 165.9 | 2 × 60 s |
| Single pool, 4 KiB overwrite; concurrency 16 | 166.2 → 159.2 | −4.2% | 83.9 → 89.4 | 296.8 → 293.2 | 3 × 15 s |
| Two pools, one 4 KiB key, 4,000 new versions | 76.6 → 53.0 | −30.8% | 11.4 → 17.8 | 47.7 → 59.5 | 3 fixed-count runs |
| Existing dual-copy data, binary-switch ABBA | 198.6 → 202.0 | +1.7% | — | 291.2 → 290.9 | 2 × 20 s |
For ordinary 4 KiB overwrites, both 60-second pairs were slower (−30.7% and −21.8%), as were all three 15-second pairs, even though CPU per operation fell from 5.85 to 5.33 ms and block writes stayed near 128 KiB per operation.
The −30.8% version-history regression had a direct structural cause. A pool-local xl.meta stores the object’s version metadata and inline payload for small objects, and adding a version reserializes and rewrites the entire file. Capacity allocation split the 4,000 versions roughly evenly across both pools (~2,000 versions per xl.meta), whereas affinity concentrated all 4,001 versions into a single pool, doubling the xl.meta rewrite size on every PUT:
- CPU per write: 26.61 ms → 39.29 ms
- Block reads per write: 12.20 MiB → 24.42 MiB
- Block writes per write: 12.27 MiB → 24.49 MiB (cumulative block writes over 4,000 operations rose from 47.9 GiB to 95.7 GiB for ~15.6 MiB of uploaded payload)
- Active data at 4,001 versions: ~48.90 MiB in both variants
- Mean latency over the final 1,000 writes: 18.4 ms → 28.5 ms
Because versioned buckets must keep all historical versions, concentrating them in one pool doubled the metadata rewrite amplification without saving disk space. This motivated narrowing the policy to unversioned overwrites.
Round two: narrowing fixed one cost, but not the acceptance problem
Before running round two (codex/unversioned-pool-affinity vs. 789a2327ba8c), the acceptance threshold for the experiment was set at ≤ 5% ordinary-overwrite throughput loss and ≤ 10% P99 latency growth, with no material P50 or excluded-scope regression.
Four workloads ran 5 pairs each in alternating AB/BA order (5-second warmup and 60-second measurement for timed workloads; 4,000 operations for versioned control; fresh server and data directories per run, no profilers attached):
| Workload | Baseline → narrowed, operations/s | Change | P50, ms | P99, ms |
|---|---|---|---|---|
| Fresh 4 KiB overwrite; 256 keys, concurrency 16 | 167.9 → 158.2 | −5.7% | 66.2 → 75.8 | 470.2 → 428.7 |
| Fresh 1 MiB overwrite; 64 keys, concurrency 8 | 79.45 → 79.58 | +0.2% | 94.0 → 95.6 | 521.5 → 483.2 |
| One 4 KiB key, 4,000 new versions; control | 48.5 → 59.6 | +22.8% | 12.7 → 12.4 | 180.9 → 135.4 |
| New 4 KiB keys; concurrency 16; control | 289.8 → 243.1 | −16.1% | 40.4 → 49.2 | 314.9 → 170.0 |
On the primary 4 KiB overwrite workload, pairwise throughput changes across the five pairs were +11.9%, −14.9%, −1.4%, −10.3%, and −12.8% (median paired change −10.3%; ratio of variant medians −5.7%; P50 +14.4%). Baseline throughput ranged from 141.4 to 202.6 ops/s (CV 13.0%) and narrowed throughput from 146.3 to 176.6 ops/s.
Excluding versioned buckets eliminated the round-one xl.meta write amplification: block writes per version were 12.272 → 12.268 MiB, block reads 12.201 → 12.197 MiB, and CPU 26.98 → 27.26 ms (in the versioned control, the second run of each pair was faster regardless of binary order, and new-key control pairs varied from −44.3% to +15.3% due to shared-SSD order effects).
Restore the same starting data before each run
Because fresh runs let the two placement policies build different on-disk layouts over time, an additional 8-run fixture test restored an identical snapshot before each run—either one pool-local copy per key (256 records) or copies in both pools (512 records)—followed by a 30-second settle period, 5-second warmup, and 60-second measurement:
| Starting state, 256 keys | Baseline → narrowed, operations/s | Change | P50, ms | P99, ms | Ending pool-local records, baseline → narrowed |
|---|---|---|---|---|---|
| One copy per key | 188.0 → 160.1 | −14.9% | 51.9 → 68.0 | 521.0 → 466.4 | 512 → 256 |
| Two copies per key | 185.7 → 140.1 | −24.5% | 37.6 → 79.7 | 460.0 → 629.9 | 512 → 512 |
The single-copy pairs changed by −0.9% and −23.5%. The dual-copy pairs declined by −22.6% and −26.5%, with P50 rising +111.9% and P99 rising +36.9%, while leaving all 512 existing records on disk. On clusters that already have cross-pool duplicates, the narrowed policy paid a steep write penalty without reclaiming any existing space.
Diagnosis and the I/O optimization attempt
Phase timing measured the new health check at 0.014–0.016 ms per call, compared with milliseconds for existing-object lookup and tens of milliseconds for erasure writes. In round two, 4 KiB overwrite CPU time fell from 5.75 to 5.36 ms per operation while throughput dropped, pointing to I/O and scheduling wait rather than CPU overhead.
Go execution traces showed syscall waits concentrated in RenameData → writeAllInternal → write, alongside read and goroutine scheduling waits.
A follow-up experiment routed small metadata writes (~791 bytes, below the 4 KiB alignment threshold where CopyAligned disables O_DIRECT) from the Linux O_DSYNC path to writeAllDirect, turning the write into a buffered write followed by fdatasync. Across five runs, narrowed affinity plus the I/O change averaged 179.4 ops/s versus the baseline’s 167.9 ops/s (+6.9%, P99 −1.5%, pairwise range −19.2% to +35.9%). However, when the same I/O change was applied to both baseline and candidate in two exploratory pairs, baseline-plus-I/O reached 212.5 ops/s versus 185.2 ops/s for affinity-plus-I/O (−12.8%, pairwise +37.7% and −49.1%). Because changing small-metadata sync paths affects the entire storage engine and did not isolate the affinity regression, the overlay was discarded.
What it buys, and what it costs
In fresh two-pool tests, 256 ordinary 4 KiB keys ended with 512 → 256 pool-local records and active logical size of 6.18 → 3.09 MiB. The 1 MiB workload ended with 128 → 64 records and 171.56 → 85.78 MiB.
At the cluster level, if a fraction $f$ of logical data would otherwise accumulate two copies across two pools, avoiding the second copy reduces steady-state space by $f / (1 + f)$ (for example, ~9.1% at $f = 10%$). Per-request write volume is unchanged: ordinary 4 KiB overwrites generated ~128 KiB of block writes per operation in both variants.
| Aspect | Benefit | Cost or limit |
|---|---|---|
| Capacity | Prevents new stale copies during eligible unversioned overwrites | Does not reclaim existing copies; fallback can still leave copies across pools |
| Write performance | A healthy preferred pool skips a full capacity-selection pass | Placement alters I/O contention; small-object overwrite regressions remained (−5.7% fresh, −24.5% dual-copy fixture) |
| Expansion and hot keys | Existing objects keep a stable pool location | Older pools retain growing/hot keys after expansion instead of shedding overwrites to newer pools |
| API and format compatibility | No wire, configuration, dependency, or disk-format change | Pool distribution changes operational balance under the same S3 interface |
| Availability and correctness | Request-aware quorum and fault checks avoid unsafe pool preference | Preflight state can change before write completion |
| Maintenance | Narrow scope avoids versioned xl.meta amplification |
Parity, bucket state, internal writes, reserve rules, PUT, and multipart routing must stay synchronized |
| Rollback | Returning to baseline requires no format conversion | Stale copies created before or during rollout remain on disk |
Final decision and conditions for revisiting it
- Full affinity (
codex/healthy-pool-affinity): rejected because concentrating version histories in one pool doubledxl.metarewrite I/O and cut 4,000-version throughput by 30.8% without saving space. - Narrowed unversioned affinity (
codex/unversioned-pool-affinity): rejected because ordinary 4 KiB overwrites still regressed by 5.7% on fresh data (median paired change −10.3%) and by 24.5% (P50 +111.9%) on restored dual-copy fixtures.
We therefore kept the mainline allocator unchanged and archived the benchmark records (54 runs / 243,718 operations in round one; 53 runs / 495,745 operations in round two). Any future revisit should start from production duplicate-space measurements and multi-disk hardware benchmarks, and evaluate whether background stale-copy cleanup or an opt-in placement policy better addresses the workload.
Source and evidence identification
Public references: PR #245, upstream PR #21115, Multi-Pool Object Consistency, and the baseline storage implementation in cmd/erasure-server-pool.go, cmd/xl-storage-format-v2.go, and cmd/xl-storage.go.
| Artifact | Exact identity |
|---|---|
| Initial independently reviewed base / PR head | 856f8f4c5580d1c5adacbd8ef2aedf3a4fbc507c / 3a9cdc731528e31bc4f9e61256025a1baae863e8 |
| PR head observed when documenting, after rebase | 0a6cad318409451b52dfcef18efb476c3927b6f8 |
| Baseline for both m3 rounds | 789a2327ba8c2283cda7192eb65f8374a7f919ba |
| Full local branch | codex/healthy-pool-affinity at 0aa870115d7d0904791104b25ad66f3430579de4 |
| Narrowed local branch | codex/unversioned-pool-affinity at 76db3626eb9068b29e4565e78ad22f7720012e65 |
| Narrowed benchmark’s production semantics | bb4b987825985481e5047f94c60f560be85b9c19 |
Round-two results/analysis.json SHA-256 |
68eec465749b625c691abcace90fc71f103b5826ccab4c40a120f39e29ded368 |
| Complete candidate patch SHA-256 | af833ec4b7bbbc22fca8539a6da40f4fe5fa93d61febabbd5b08a63ae1ecbe1c |
4.17 - Replica Metadata Normalization: What a Trusted Copy May Not Re-Inject
Shipped in SILO Server 20260916 (40220bd836cbd066ca424fa4dc5dbb90057fb55a), PR #194 (commit 4fcdf37ce, merged as 9f3037e941, retaining Mikhail Khadarenka’s original implementation from PR #187) fixes a regression where a trusted replication receiver re-extracted raw request headers through the permissive metadata extractor, re-injecting transport-only aws-chunked Content-Encoding tokens and GHSA-redacted user metadata into stored replica objects.
What Went Wrong
The ordinary S3 PUT path normalizes incoming request metadata before storing an object: it strips the transport-only aws-chunked token from Content-Encoding, and deletes the X-Amz-Meta-X-Amz-Unencrypted-Content-Length and X-Amz-Meta-X-Amz-Unencrypted-Content-Md5 user-metadata keys removed by the GHSA-76wf-9vgp-pj7w mitigation.
However, when the trusted replication receiver restored internal replication metadata on an incoming replica write, it re-ran the same general-purpose header extractor with the allowReplication boolean flag enabled. Re-running that extractor replayed every supported standard header and all X-Amz-Meta-* user metadata from the raw HTTP request over the already-normalized metadata map. As a consequence, on a trusted replica write the server could persist and later return via GET or HEAD:
Content-Encoding: aws-chunked(a SigV4 streaming transport framing token that must never be stored as object metadata) or the unsplitaws-chunked,gzipstring instead ofgzip;- the two GHSA-redacted
X-Amz-Meta-X-Amz-Unencrypted-Content-*keys on trusted replica writes; - for Snowball archive entries lacking their own PAX metadata header, the outer archive request’s
Content-Type,Cache-Control,Expires, and user metadata.
The object payload bytes on disk were unaffected, but the stored xl.meta headers were polluted. This regression was introduced in commit 56fa63bfd (2026-04-15, the replication header trust boundary hardening for CVE-2026-34204), whose authentication and trust-boundary enforcement remains intact.
The Fix
In cmd/handler-utils.go, commit 4fcdf37ce deletes the boolean dual-mode helper and separates standard metadata extraction from replication-specific restoration:
- The ordinary metadata extractor unconditionally skips internal replication-only header keys.
- A dedicated replica metadata extractor (
extractReplicationMetadata) walks only the replication-to-internal header mapping table and restores only the six replication-scoped fields:- SSE-C sealed key material (
x-minio-internal-server-side-encryption-ssec-key), - SSE-C sealed algorithm (
x-minio-internal-server-side-encryption-ssec-seal-algorithm), - SSE-C initialization vector (
x-minio-internal-server-side-encryption-ssec-iv), - Encrypted multipart marker (
x-minio-internal-encrypted-multipart, honored by key presence even with an empty value), - Actual logical object size (
x-minio-internal-actual-size), - SSE-C checksum identity mapping.
- SSE-C sealed key material (
extractReplicationMetadatanever re-reads standard S3 headers (Content-Encoding,Content-Type,Cache-Control,Expires) orX-Amz-Meta-*user metadata.
Stored Content-Encoding values after the fix:
Requested Content-Encoding |
Stored Content-Encoding |
|---|---|
aws-chunked |
(none) |
aws-chunked,gzip |
gzip |
gzip |
gzip |
Note that aws-chunked, gzip (with a space after the comma) continues to store gzip with a leading space, and gzip, aws-chunked continues to store the full string; both match pre-existing parser behavior on the ordinary PUT path and are pinned as status-quo test assertions.
Operator-Visible Behavior Changes
- Trusted Snowball archive entries: Entries inside a trusted Snowball archive no longer inherit the outer HTTP request’s ordinary metadata (
Content-Type,Cache-Control,Expires, and user metadata) when PAX records are absent or omit those fields. Per-entryminio.metadata.*PAX records still apply, the six replication-scoped fields still apply to authorized entries, and the outer archive’s storage class is still inherited. (The in-tree batch producer callsPutObjectsSnowballwith the auto-extract marker without marking the outer request as a trusted replica, so it was unaffected.) - GHSA-redacted keys:
X-Amz-Meta-X-Amz-Unencrypted-Content-Lengthand-Md5are never re-inserted during replica metadata restoration, matching ordinaryPUTrequests. - Unchanged boundaries: Authentication,
s3:ReplicateObjectpermission gating, and replication trust semantics are unchanged, and the ordinaryPUTextraction path is byte-for-byte identical.
Why Upgrading Does Not Rewrite Existing Polluted Objects
Upgrading to SILO Server 20260916 stops new metadata pollution on incoming writes, but does not scan or rewrite existing objects on disk. Two operational consequences follow:
- Authoritative source vs. normalized replica divergence: If an authoritative source object still carries polluted metadata (for example, from an earlier bidirectional replication cycle) while its replica has been normalized, resync or healing comparisons may detect a metadata mismatch and repeatedly schedule metadata replication. Always remediate the authoritative source version first so replicas converge from the clean source.
- Limits of ordinary S3 self-
COPY: Issuing a standard S3CopyObjectto self is not a universal in-place repair tool for versioned buckets because it creates a new object version (or shifts modification timestamps) rather than rewriting a specific historicalversionId’s metadata in place.
The Replica Metadata Audit runbook provides a read-only inventory scanner and classification rules to identify affected object versions across buckets.
Stored-Metadata Remediation Design Guidelines
When planning remediation for historical objects flagged by the audit runbook:
- Build a complete version inventory (including non-current versions, not just the latest version).
- Verify payload encoding by comparing against the trusted source version or an independent checksum—never guess from the polluted
Content-Encodingheader alone, and never gunzip an object merely because a header label saysgzip. - Remediate the authoritative source’s exact versions first and allow replication to converge downstream copies.
- Retain immutable inventories and metadata backups, validate on a local clone first, and preserve version identity, Object Lock retention and legal hold, object tags, replication status, and encryption context. Never hand-edit
xl.metafiles on disk.
Known Scope Limits
- The browser
POSTform upload handler (cmd/bucket-handlers.go) calls the low-level header extractor directly without normalizingaws-chunkedencodings; that pre-existing path is outside the replication receiver and tracked separately.
Verification
Regression tests (TestExtractReplicationMetadata*, TestAPIReplicaContentEncoding, TestAPISnowballReplicaContentEncoding, and -race replication trust / SSE-C round-trip suites) verify the header mapping table, all six restored replication fields, and equivalence with the ordinary PUT path across single-drive and 16-drive erasure backends (64 HTTP leaf cases: 44 controls and 20 regression cases that failed prior to 4fcdf37ce, plus 16 helper unit cases).
For the coordinated upgrade matrix, see the component version matrix. Related September replication and HTTP hardening designs: Replicated Tag Ordering · Request Header Timeouts · Replica Metadata Audit.
4.18 - Request-Header Deadlines: Absolute Header Limits and Rolling Body Idle Timeouts
This article records the repair of Server’s HTTP read deadlines, merged in PR #196 (fix 055030ea5) and shipped in Server 20260916 (40220bd836cbd066ca424fa4dc5dbb90057fb55a).
The investigation was prompted by a slow-HTTP DoS scanner report (#183) and verified via a direct TCP comparison (with the header limited to 100 ms and arriving over 400 ms, standard Go refuses where the pre-fix server returned 204) and a single-drive process probe where both flag and environment settings rejected a 400 ms slow header while healthy requests continued.
Two Timeout Classes, One Connection
- Request-header absolute deadline.
ReadHeaderTimeoutbounds the total time from the start of header reading to its completion. Trickle-feeding bytes cannot extend it. On HTTP/1 keep-alive connections, Go first usesIdleTimeoutwhile waiting for the next request’s initial bytes, then starts a fresh header deadline.ReadHeaderTimeoutalso participates in the separate TLS-handshake timeout calculation. - Body rolling idle timeout. Once headers parse and the connection enters the active phase, rolling idle semantics take over: every successful read extends the deadline, and only a stall between bytes exceeding the configured idle timeout closes the connection. Long uploads and downloads that continue transferring bytes run without a total-duration cap at the HTTP/1 connection layer.
Before this repair, the first class did not take effect: the connection-layer wrapper replaced the socket deadline with now + idle + 250 ms before every partial read, overwriting whatever absolute deadline net/http had set — so a slow reader could hold a connection open indefinitely by sending one byte per idle window.
Two Independent Defects
- The connection wrapper overwrote the absolute deadline. The
DeadlineConnwrapper’s read path reset the socket deadline on every partial read, defeating the read-header deadline Go’s server sets. The direct-TCP baseline proved this in isolation: with a 100 ms header limit and a 2 s idle window, a header finishing at 400 ms was accepted. - The CLI setting never reached the server context. The CLI accepted
--read-header-timeoutandMINIO_READ_HEADER_TIMEOUT, including default parsing, but the server-context builder copiedIdleTimeoutwhile omittingReadHeaderTimeout, so the running server always saw zero. Fixing defect 1 alone left the server accepting slow headers; the second fix wiresReadHeaderTimeoutalongside theIdleTimeoutbinding.
This went unnoticed because the flag’s default (30 s) matches the idle timeout’s default, and with the flag unwired net/http fell back to that same 30 s read timeout — making default configurations appear to match the documented setting.
Configuration
- Flag:
--read-header-timeout(Hidden: true, absent from ordinary CLI help) - Environment:
MINIO_READ_HEADER_TIMEOUT - Default: 30 s (equal to the idle timeout default)
- Neither timeout has a YAML configuration field; the value binds once at startup in order of flag > environment > default.
| Setting | Effect |
|---|---|
header > 0 |
Absolute cap on HTTP/1 header phases; participates in Go’s TLS-handshake read window (including the HTTP/2 handshake) |
header = 0 (explicit) |
Falls back to Go’s rule: the read timeout (= idle timeout) applies; the CLI default is 30 s |
header < 0 |
Disables the header-specific cap; positive read/write timeouts still bound TLS handshake reads, and positive IdleTimeout still bounds the keep-alive wait |
idle shortened, header unset |
Header phase uses its own 30 s default — the one combination looser than a naive expectation, while still enforcing a fixed absolute cap |
Setting a negative value reopens unbounded slow-header trickling and should not be used in production. An incomplete header cut off by the deadline closes the connection directly rather than guaranteeing an HTTP error response. The initial report came from @AEGEGE’s scanner issue #183, and PR #195 was integrated through PR #196.
Behavior Across Protocols
- HTTP/1: headers and keep-alive waits use absolute deadlines; the request body uses the rolling idle timeout. The connection-state hook composes with any caller hook.
- TLS: handshake reads take the minimum of the positive header deadline and the existing read/write timeouts, and a fresh header deadline begins once the handshake completes; handshake writes retain rolling timeout semantics.
- HTTP/2: unchanged. When
h2is negotiated, phase switching is skipped;h2retains its native absolute per-stream read timeout, andReadHeaderTimeoutis not passed into theh2configuration. - Internode connections: Linux internode dialing manages its own rolling timeouts, and grid-hijacked connections unwrap to the underlying TCP connection before phase switching applies.
Rejected Alternatives
- Clamp all future deadlines globally. Go 1.27 sets a whole-request deadline in some paths; with the read timeout equal to the idle timeout, clamping globally would hard-cap entire HTTP/1 requests (header plus body) and abort large uploads.
- Drop the read timeout and reinterpret zero as rolling idle. Zero is
net/http’s sentinel for “never time out” on background reads and hijacked connections; reinterpreting it would break long-running handlers and striph2of its per-stream timeout. - Wrap the body reader or use
http.ResponseController. Accounting for chunked encoding, connection draining, EOF transitions, andh2special cases requires a much larger surface area than fixing the connection deadline wrapper. - Reconstruct the standard library’s deadline arithmetic in the hook. That would duplicate
net/httpinternals that shift across Go releases; recording the deadline the standard library actually requested avoids version drift. - Infer the phase by comparing timeout durations. Because all three defaults are 30 s, “shorter than the idle window” cannot distinguish header deadlines from idle timeouts under default production settings.
Verification and Limits
Unit tests verify the connection wrapper across three consecutive refresh intervals (confirming the absolute cap never slides forward on partial reads), phase transitions across HTTP/1 keep-alive, TLS, and HTTP/2-only negotiation, and flag/env binding on the CLI context; a live process probe with a 100 ms header limit rejected a 400 ms slow header while serving normal traffic. Scope notes: TLS handshake writes remain rolling; handler CPU and storage waits are not bounded by socket read deadlines; and the absolute header cap applies without slack whereas rolling idle refreshes retain their ~250 ms slack window.
Upgrade notes (reducing the header timeout also narrows the TLS handshake read window) are documented in the component matrix.
4.19 - Conditional DELETE: One Condition, One Logical Object
Single-object conditional deletion (If-Match on DeleteObject) landed in PR #145, and PR #178 repaired multi-pool serialization and reconciliation. Both changes shipped in Server 20260916.
An earlier August proposal around PR #12 sketched a broader scope—including batch DeleteObjects rejection, an extra s3:GetObject authorization check, and a current-version-only rule—that was not adopted in the merged implementation. This note documents the shipped behavior in f99ed829b and why precondition evaluation must sit above individual storage pools.
Implemented Contract
| Request | Shipped Behavior |
|---|---|
Single DeleteObject, non-empty If-Match: <ETag> |
Evaluates the client-visible ETag under the outer deletion lock; mismatch returns 412 PreconditionFailed before any deletion |
If-Match: * |
Requires an existing, non-delete-marker representation |
Explicit versionId |
Evaluates the addressed version, not an unrelated current version |
No If-Match, or an empty header value |
Installs no conditional callback; ordinary deletion applies |
Batch DeleteObjects with per-item <ETag> |
ObjectToDelete has no ETag field; the XML element supplies no deletion protection |
Internal recursive x-minio-force-delete |
Prefix deletion returns before the object-condition path; do not use it as conditional deletion |
These preconditions do not add an s3:GetObject authorization check. Ordinary deletion authorization applies—including s3:DeleteObjectVersion for an explicitly addressed version, Object Lock checks, and the separate trusted-replication path.
Why the Condition Belongs Above Individual Pools
In a multi-pool deployment, an object can have copies of different ages in more than one pool. A request precondition applies to the single logical object selected for the operation. Evaluating and mutating independently inside each pool can delete one copy and then return 412 PreconditionFailed from another, or delete the newest copy and uncover a stale one.
The multi-pool coordinator (erasureServerPools.DeleteObject) therefore acquires the shared namespace write lock, gathers the relevant pool states, evaluates the condition once, clears the callback for lower layers, and reconciles the selected deletion across pools. The same lock boundary covers concurrent object writes and metadata updates.
Counterexamples from the Early Design
Suppose Pool 1 holds an older copy with ETag A and Pool 2 holds the current copy with ETag B:
If-Match: A(partial delete on412): Per-pool evaluation deletes copyAin Pool 1 and only then fails with412 PreconditionFailedagainstBin Pool 2, mutating storage on a rejected request.If-Match: B(stale copy exposure after204): Per-pool evaluation deletesBin Pool 2 while failing or skipping Pool 1, leaving stale copyAvisible as the current object.
Executing an HTTP precondition callback inside per-pool workers also risks concurrent writes to a single http.ResponseWriter. Consuming the request condition once at the coordinating layer eliminates both hazards.
Failure Boundaries
The reconciliation path reads pool state before evaluating the condition and surfaces read quorum failures instead of treating an unreachable pool as empty. Physical cleanup errors can still occur after a partial mutation: a failed request is not a distributed multi-pool rollback transaction. Retrying and verifying object state remain necessary after a storage failure; see Multi-Pool Object Consistency.
Selection and Evaluation
Evaluate Once
erasureServerPools.DeleteObject holds the outer namespace lock. Multi-pool deployments call deleteObjectReconciled; single-pool deployments read the selected representation and invoke CheckPrecondFn before taking the delete-marker shortcut. Lower storage layers never reinterpret the precondition per copy.
Wildcards and Delete Markers
The DELETE precondition helper treats If-Match: * as requiring an active object representation. A delete marker does not satisfy *. A missing key or missing addressed version follows the normal not-found path and is never coerced into an empty-ETag match.
Authorization
The handler authorizes deletion against the effective target version (s3:DeleteObject or s3:DeleteObjectVersion). It does not require s3:GetObject when a specific ETag is supplied in If-Match.
SSE-C ETags
Conditional deletion compares the established client-visible ETag projection stored in metadata; it does not read or decrypt the object payload. SSE-C read-key authentication remains separate from object deletion.
Explicit Versions
Supplying an explicit versionId selects that exact version for both ETag comparison and deletion. A matching historical ETag allows deleting that historical version even when the current version has a different ETag.
Fail-Closed and Unsupported Edges
An empty If-Match header installs no precondition. The internal recursive prefix-delete extension (x-minio-force-delete) bypasses object preconditions. Batch XML <ETag> elements in DeleteObjects are not parsed by ObjectToDelete, and there is no request-wide NotImplemented rejection guard. Callers requiring compare-and-delete semantics must use single-object DeleteObject with a non-empty If-Match header.
Rejected Alternatives and Scope
- Modifying a shared ETag comparator: Changing low-level string comparison cannot fix multi-pool winner selection or prevent mutations before a precondition check.
- Aggregating errors after per-pool callbacks: Returning
412after one pool has already deleted its copy cannot undo the physical deletion. - Introducing a new distributed transaction framework: Consuming one precondition callback under the existing outer namespace lock is sufficient for single-object
DeleteObject; batch execution and policy keys require separate surface changes.
Source Evidence
The implementation lives in cmd/object-handlers.go, cmd/erasure-server-pool.go, and cmd/api-datatypes.go, with regression tests covering ETag mismatches, wildcard and delete-marker handling, explicit versionId targeting, quorum failures, and multi-pool reconciliation.
Follow-Up Work
Batch Conditions
Supporting per-item <ETag> preconditions in DeleteObjects requires unmarshaling the XML field, evaluating each logical object under its outer lock, honoring Quiet mode, and reporting per-item precondition failures in the XML result. Until that path is implemented, batch <ETag> elements are ignored and must not be used as a concurrency guard.
Policy Enforcement
The IAM policy package does not yet define the s3:if-match condition key. Supporting policy-enforced conditional deletes requires adding the condition key in silo-pkg, populating request condition values in the server handler, and adding policy evaluation tests.
Design Trade-Off
Single-object conditional DELETE reuses the existing multi-pool object-selection and locking boundary without inventing a new transaction layer. Full batch preconditions and IAM policy condition keys cross additional API and package boundaries and remain separate work. The core invariant is strict: a false single-object If-Match precondition is decided once against the selected logical representation before any pool deletes data.
4.20 - DSN-Only Database Notifications: A Compatibility Boundary for #53
Shipped in SILO Server 20260903 via commit f1ba68358, this design resolves SILO issue #53 by establishing the compatibility boundary, legacy-migration behavior, and startup error handling for PostgreSQL and MySQL bucket-notification targets.
Decision
SILO retains PostgreSQL and MySQL notification targets and standardizes each on a single canonical configuration key:
- PostgreSQL requires a complete
connection_string. - MySQL requires a complete
dsn_string.
The pre-2020 five-field form—host, port, username, password, and database—remains unsupported by the KV configuration system. SILO does not register those five keys and does not synthesize a DSN from them during legacy JSON migration.
The legacy migration contract handles three input states:
| Legacy Target State | Migration Result |
|---|---|
| Disabled | Ignored; no KV target is emitted. |
Enabled with a non-empty connection_string or dsn_string |
Migrates only the canonical connection-string key and registered target settings. |
| Enabled with only discrete connection fields | Rejects migration and aborts server startup before activating the new configuration, logging an actionable error that names the subsystem and target without printing credentials. |
Context
SILO inherited two generations of database-notification configuration from upstream MinIO. The pre-KV JSON configuration could represent a database connection either as a single connection string or as five discrete fields:
The current KV configuration system exposes only the driver-native connection string:
Upstream deprecated the five discrete fields in RELEASE.2020-04-10T03-34-42Z and directed operators to migrate to connection_string or dsn_string. SILO’s CLI help tables, environment-variable documentation, and Console UI already use the complete connection string as the supported interface.
The Defect
Before f1ba68358, the legacy migration helpers SetNotifyPostgres and SetNotifyMySQL wrote both representations into the new KV configuration. Even when the legacy JSON target already specified a complete connection_string or dsn_string, the helpers also emitted all five discrete keys (typically with empty values).
The KV parser rejected those migrated entries because neither DefaultPostgresKVS nor DefaultMySQLKVS registers the five discrete keys, and CheckValidKeys validates key names regardless of whether their values are empty. Both legacy JSON forms therefore failed KV validation:
Notification initialization amplified the impact of that parser error. FetchEnabledTargets evaluates notification subsystems fail-fast: the first invalid subsystem returns an error and a nil target list. The caller logged the error and continued starting the object server, leaving healthy Webhook, Kafka, NATS, and other notification targets disabled as well.
Simply returning an error from SetNotifyPostgres and SetNotifyMySQL was not sufficient on its own. While that error propagates through readConfigWithoutMigrate and initConfig, initConfigSubsystem previously logged non-retriable configuration errors as "some features may be missing" and returned success. The server then started without assigning globalServerConfig, leaving not only notifications broken but also region, storage class, compression, identity, and other persisted settings unapplied. Conversely, marking the error as retriable would cause the server to loop indefinitely on a static configuration error. The migration failure therefore has to propagate as a typed, non-retriable, startup-fatal error (LegacyDatabaseTargetError).
This failure mode was particularly hazardous because S3 object reads and writes continued normally while every configured event pipeline was offline. In addition, the unregistered password key lacked secret-redaction metadata and could be included in plain text inside diagnostic bundles, whereas connection_string and dsn_string are registered as sensitive keys.
Why the First Fix Was Reverted
An initial fix in 20260804 (SN-2026-019) registered the five discrete keys in DefaultPostgresKVS and DefaultMySQLKVS and wired the parser to read them. While that allowed migrated targets to pass CheckValidKeys, it broke the documented connection_string interface.
The shared mcli admin config set KV tokenizer identifies key/value boundaries by scanning for registered key names and is not quote-aware. Once port was registered as a top-level key in notify_postgres, a valid libpq key/value connection string inside quotes looked like a second top-level KV parameter:
The tokenizer split the input at port= inside the quoted string, truncated connection_string to "host=db, and passed 5432 dbname=events user=app" to the integer port parser, failing with invalid port.
Under the shared KV tokenizer grammar, registering common words such as host, port, and password creates an irreconcilable grammar collision with key/value connection strings. That registration change was therefore reverted in favor of enforcing connection_string and dsn_string exclusively.
Product Judgment
Database notification targets provide a direct database-backed namespace view or access journal without requiring an external message broker, which remains valuable for deployments already operating PostgreSQL or MySQL.
By contrast, the five-field parameter model cannot express standard driver options such as TLS modes and CA paths, connect timeouts, application names, Unix domain sockets, multi-host PostgreSQL failover, or MySQL driver flags. Maintaining both forms also creates precedence, merging, and secret-redaction ambiguity. Keeping SILO responsible for notification delivery semantics while delegating connection syntax to the database driver’s native DSN string provides a clean boundary.
Goals
- Establish
connection_stringanddsn_stringas the sole supported configuration keys for PostgreSQL and MySQL notifications. - Allow legacy JSON targets that already contain a non-empty canonical connection string to migrate cleanly.
- Reject enabled discrete-only legacy targets before activating or persisting a partial KV configuration.
- Replace silent runtime notification failure with an explicit startup-time failure (
LegacyDatabaseTargetError). - Ensure no migration error, log line, health report, or diagnostic bundle leaks a database password.
- Remove all ten Postgres/MySQL exceptions from the
knownUnregisteredWritesaudit ratchet. - Document operator remediation paths clearly.
Non-Goals
- Supporting both DSN strings and discrete database fields in the KV interface.
- Synthesizing a DSN string from legacy discrete fields during migration.
- Rewriting the shared KV tokenizer grammar.
- Altering
FetchEnabledTargetsfail-fast semantics across notification subsystems. - Silently skipping an enabled invalid database target and starting with partial notification coverage.
- Removing PostgreSQL or MySQL notification support.
- Deleting the legacy Go struct fields used to unmarshal old JSON files and detect unsupported discrete-only configurations.
- Changing error handling for the other eight legacy notification setters outside PostgreSQL and MySQL.
Functional Requirements
Current Configuration
notify_postgresacceptsconnection_string;notify_mysqlacceptsdsn_string.- The five discrete keys (
host,port,username,password,database) remain unregistered and are rejected byadmin config set. - Complete connection strings continue to support native driver syntax, including embedded
host=,port=,user=,password=, anddbname=tokens. - Legacy environment variables
MINIO_NOTIFY_POSTGRES_HOST/PORT/USERNAME/PASSWORD/DATABASEand their MySQL equivalents remain unwired and unsupported.
Legacy Migration
SetNotifyPostgresandSetNotifyMySQLreturnnilwithout emitting a target when the legacy target is disabled (Enable == false).- When enabled,
SetNotifyPostgresrequires a non-emptyConnectionStringand writes only registered Postgres keys. If bothConnectionStringand discrete fields are present in the legacy JSON,ConnectionStringwins and the discrete values are discarded. SetNotifyMySQLapplies the identical rule toDSN.- Neither helper ever emits
host,port,username,password, ordatabase. - An enabled target with an empty canonical connection string returns a
LegacyDatabaseTargetErroridentifying the subsystem (notify_postgresornotify_mysql) and target ID without echoing field values:
cmd/config-migrate.gochecks and propagates errors from both database migration helpers before persisting or activating the migrated configuration.initConfigSubsystempropagatesLegacyDatabaseTargetErroras a non-retriable fatal error toserverMain.
Operator Remediation
Because legacy JSON migration output is only persisted after a successful migration pass, a deployment that still boots from an old JSON configuration file will re-enter migration on startup. Operators with enabled discrete-only database targets should follow one of four paths:
- Before migrating to SILO: On the prior release, replace the discrete fields with
connection_stringordsn_string, verify event delivery, and then upgrade. - Disable and recreate: Disable or remove the legacy database target, upgrade the server, and recreate the target using
connection_stringordsn_string. - Fresh installation: Configure
notify_postgresornotify_mysqldirectly withconnection_stringordsn_string. - Existing deployment reading legacy JSON: Back up the source configuration, then convert, disable, or remove the discrete database target before starting SILO Server 20260903 or later.
Availability Trade-Off
This design intentionally converts an unsupported legacy database target from a degraded startup (where S3 I/O works while all bucket notifications are silently offline) into an immediate startup abort. An object store that appears healthy while dropping every configured audit or pipeline event creates unrecoverable downstream data loss; failing fast at startup forces the configuration error to be resolved before traffic is served.
Security Requirements
LegacyDatabaseTargetErrornever formats the legacy struct or its field values into the error message.- Regression tests inject sentinel passwords and assert that neither returned errors nor captured logs contain the sentinel.
- Migrated KV output uses only the registered sensitive keys (
connection_stringanddsn_string). - Operators who exported diagnostic bundles from an affected pre-20260903 deployment with discrete database targets should rotate that database password.
Rejected Alternatives
Register and Parse the Five Discrete Fields
Benefit: Preserves the pre-2020 JSON field names and reuses existing struct fields.
Reason rejected: Registering host, port, and password causes the shared KV tokenizer to split quoted connection_string values containing port= or host=, breaking the primary documented interface.
Synthesize a Canonical Connection String During Migration
Benefit: Automatically converts discrete-only legacy configurations.
Reason rejected: Requires permanent code and test maintenance for PostgreSQL quoting rules, MySQL DSN escaping, Unix sockets, IPv6 brackets, and driver defaults for a format deprecated since 2020.
Skip Only the Unsupported Target
Benefit: Keeps S3 object service and other notification targets online.
Reason rejected: Silently dropping an enabled notification target causes unannounced event loss.
Change Global FetchEnabledTargets Fail-Fast Behavior
Benefit: Isolates failures between independent notification subsystems.
Reason rejected: Does not fix the broken database migration helper or the unredacted password key, and alters system-wide notification startup semantics outside the scope of #53.
Remove Database Notification Targets Entirely
Benefit: Eliminates all database-specific notification code.
Reason rejected: PostgreSQL and MySQL notification targets remain useful and self-contained; the defect lies solely in the obsolete five-field configuration format.
Implementation Scope
The implementation in f1ba68358 touches six files across internal/config/notify/legacy.go, cmd/config-migrate.go, and cmd/server-main.go:
- Update
internal/config/notify/legacy.goso the two database setters emit only canonical registered keys and reject enabled targets lacking a canonical string. - Update
cmd/config-migrate.goto propagate the two database helper errors with subsystem and target context. - Define
LegacyDatabaseTargetErrorand updatecmd/server-main.gosoinitConfigSubsystemreturns it as a non-retriable fatal error instead of logging and continuing. - Remove all ten Postgres/MySQL entries from
knownUnregisteredWrites.
Acceptance Criteria
- Legacy PostgreSQL targets with a non-empty
connection_stringmigrate cleanly, passCheckValidKeys, and round-trip throughGetNotifyPostgres. - Legacy MySQL targets with a non-empty
dsn_stringmigrate cleanly, passCheckValidKeys, and round-trip throughGetNotifyMySQL. - Enabled discrete-only targets for either database fail before target initialization with
LegacyDatabaseTargetErrornaming the subsystem and target ID, aborting server startup. - Missing-string and malformed-string errors never include sentinel host, username, password, database, or DSN values.
- Disabled discrete legacy targets emit no KVS entries and do not block startup.
- Migrated KVS output never contains any of the ten discrete keys, even with empty values.
- When a legacy target contains both a canonical string and conflicting discrete values, only the canonical string is migrated.
SetKVSregression tests againstDefaultPostgresKVSandDefaultMySQLKVSaccept quoted connection strings containingport=,host=, andpassword=.- Configurations pairing healthy Webhook, Kafka, or NATS targets with an invalid legacy database target fail in
readConfigWithoutMigratewithout persisting or activating partial state. initConfigSubsystemreturnsLegacyDatabaseTargetErrordirectly without logging-and-continuing or entering the retry loop.knownUnregisteredWritesremoves all ten Postgres/MySQL exceptions.- All unit, race, and startup suites pass:
Implementation Result
Commit f1ba68358 implements the design without expanding the public configuration surface. Cross-repository verification confirmed that pgsty/mc, pgsty/silo-pkg, and pgsty/silo-console require no changes because mcli forwards raw KV text and Console already serializes database forms into connection_string and dsn_string.
Release Compatibility Statement
SILO database notification targets require connection_string for PostgreSQL and dsn_string for MySQL. The pre-2020 discrete host/port/username/password/database form is not migrated; convert, disable, or remove such targets before upgrading to SILO Server 20260903 or later.
Design Review Calibration
During design review of the initial specification, four calibrations were incorporated before landing the commit:
- extending fatal error propagation through
initConfigSubsystemtoExitFunc(1)soserverMainterminates vialogger.FatalIfinstead of starting withoutglobalServerConfig; - documenting that existing deployments still booting from unmigrated legacy JSON files also hit the startup gate;
- specifying canonical-string precedence when a legacy JSON record contains both
ConnectionString/DSNand leftover discrete fields; - leaving error handling of the other eight non-database legacy notification setters unchanged to keep
#53tightly scoped.
4.21 - Preview Text, Never Execute It: SILO Console Text Preview Design
SILO Console previews images, PDFs, audio, and video, but in v2.1.1 it could not preview the small logs, text files, JSON documents, and XML documents that operators inspect every day. Even when an object stored an accurate Content-Type, the frontend classified it as unsupported before selecting a preview renderer.
Restoring a browser-native iframe viewer would have been easy, and it would have been the wrong fix. An object in storage is controlled by the client that uploaded it. Loading that object as a same-origin HTML or XML document turns a convenience viewer into a script execution boundary.
The design shipped in Console 2.2.0 (tracking #17) enforces a strict invariant:
SILO Console previews eligible text objects as bounded UTF-8 text. It never asks the browser to interpret their markup, MIME type, or file contents as an executable document.
Decision
Console adds a dedicated text preview type and a PreviewText component governed by ten rules:
- Preserve every existing
image,pdf,audio, andvideoclassification. - Consider a text fallback only when the existing media classifier returns
none. - Admit the four target extensions (
.log,.txt,.json,.xml) or four exact passive text MIME types (text/plain,application/json,application/xml,text/xml). - Fetch bytes through the ordinary authenticated download path without
preview=true. - Enforce a hard application read limit of 1 MiB (
1,048,576bytes). - Decode only strict UTF-8 (
fatal: true) and reject payloads containing NUL bytes. - Render a single React text node inside a scrollable
<pre>element. - Never use an
<iframe>, HTML parser, XML parser, or HTML injection API. - Show the complete object or no object at all; never render a truncated JSON or XML fragment.
- Keep Download available for files that exceed the size limit, fail UTF-8 decoding, or cannot be previewed.
No new S3 operation or backend preview route is added, and the backend inline MIME allowlist remains unchanged. Delivering this feature also tightened Console download responses: zero-byte Range requests return an empty 200 OK, unsatisfiable ranges return 416 Range Not Satisfiable with Content-Range: bytes */N, and size is always emitted in object JSON metadata (see Console 2.2.0).
Previous Behavior in Console 2.1.1
In SILO Console v2.1.1, the frontend preview type union contained only:
Its extension table listed media formats but omitted .log, .txt, .json, and .xml. Its MIME classifier likewise ignored text/plain, application/json, application/xml, and text/xml. Runtime verification showed the resulting split between frontend classification and backend response headers:
| Object | Frontend Classification | Console Download Response |
|---|---|---|
.log / text/plain |
none |
inline, X-Frame-Options: SAMEORIGIN |
.txt / text/plain |
none |
inline, X-Frame-Options: SAMEORIGIN |
.json / application/json |
“Preview unavailable” | inline, X-Frame-Options: SAMEORIGIN |
.xml / application/xml |
none |
attachment, X-Frame-Options: DENY |
In addition, the object-detail view used the wrong boolean conjunction when deciding whether Preview should be disabled: an authorized user could click Preview on an unsupported object only to receive an unavailable dialog, or see an enabled action that the server would reject.
Most importantly, the preview container still held a legacy generic same-origin <iframe> fallback. Because that branch was unreachable under the closed five-member type union, v2.1.1 did not have an exploitable text-preview XSS—but adding text to the union and letting it fall through to the <iframe> branch would have reactivated same-origin document execution.
Root Cause
The defect came from contract drift across three layers:
Classification Drift
The browser frontend determines preview eligibility from the object’s filename and Content-Type metadata, but its closed type union had no text variant. Even accurate object metadata could not select a renderer that did not exist.
Response-Policy Drift
The Console backend independently decides whether a download response may be served inline. It treated text/plain and application/json as passive inline types while forcing application/xml and text/html to attachment. That server policy was disconnected from the frontend classifier.
Renderer Drift
The dead <iframe> fallback remained in the component tree after reachable preview types were narrowed to media and PDF. Realigning the three layers required introducing a dedicated inert text renderer and deleting the dead <iframe> fallback rather than relying on MIME headers as a security boundary.
Why Same-Origin Iframe Preview Was Rejected
X-Frame-Options: SAMEORIGIN is not a sandbox. It restricts which parent origins may embed a response, not what script inside a same-origin frame can execute.
If uploader-controlled HTML, XHTML, SVG, or active XML is ever loaded as an inline same-origin document inside an <iframe>, script inside that frame runs in the Console origin. Even with HttpOnly session cookies blocking direct document.cookie reads, same-origin execution can still issue authenticated state-changing API requests against the Console backend.
X-Content-Type-Options: nosniff, Content Security Policy (CSP), and Content-Disposition provide defense in depth, but the primary security invariant must live in the data flow itself:
Product Contract
The feature is a read-only source-text viewer, not a web document renderer or an online code editor.
Users can:
- Open an eligible small object from either the bucket object list or the object-detail drawer.
- Read whitespace-preserving source text inside the preview modal.
- Select and copy text using native browser selection.
- Distinguish whether a preview refusal is caused by file size, non-UTF-8/binary encoding, missing permissions, object replacement, or network failure.
- Download the original bytes at any time.
Users are never misled by silent transformations:
- JSON is never parsed and re-serialized (which would alter large integers, duplicate keys, or whitespace).
- Large XML or JSON objects are never silently truncated into invalid prefix fragments.
- Invalid UTF-8 sequences are never masked with
U+FFFDreplacement characters.
Goals and Non-Goals
Goals
- Preview small
.log,.txt,.json, and.xmlobjects without requiring a local file download. - Keep object bytes strictly inert regardless of filename extension,
Content-Type, or payload contents. - Bound retained response bytes and rendered text to 1 MiB (
1,048,576bytes). - Preserve exact stored UTF-8 text rather than reformatting it.
- Unify preview eligibility and permission checks across the list and object-detail views.
- Support both current objects and explicitly selected historical
versionIdtargets. - Preserve anonymous-access (
X-Anonymous: 1) and subpath-hosting behavior.
Non-Goals
- HTML, XHTML, or Markdown rendering.
- XML parsing, XSLT evaluation, external entity resolution, or schema validation.
- Automatic JSON pretty-printing (
JSON.parse/JSON.stringify). - In-browser editing, syntax highlighting, line-number gutters, folding, or ANSI escape rendering.
- Head/tail or truncated prefix previews for objects larger than 1 MiB.
- Lossy decoding or charset guessing for GBK, UTF-16, or Latin-1 files.
An object such as notes.md stored with Content-Type: text/plain is eligible for raw UTF-8 text preview, without Markdown rendering.
Eligibility Contract
Eligibility is evaluated in two stages so existing media behavior never changes.
Stage 1: Preserve Legacy Media Classification
Run the existing image, pdf, audio, and video classifier first. If it returns anything other than none, use that media classification.
Stage 2: Apply Text Fallback
Only when the legacy media classifier returns none:
- Explicitly reject final filename extensions
.html,.htm, and.xhtml. - Match the final filename extension case-insensitively against
.log,.txt,.json, and.xml. - Normalize
Content-Typeby stripping parameters (; charset=...), trimming whitespace, and lowercasing. - Match the normalized MIME type against
text/plain,application/json,application/xml, andtext/xml.
Matching either an allowed extension (when not excluded by rule 1) or an allowed exact MIME type selects text:
| Filename and MIME | Classification | Reason |
|---|---|---|
report.txt + image/png |
image |
Existing media classification wins |
report.json + application/pdf |
pdf |
Existing media classification wins |
server.LOG + application/octet-stream |
text |
Allowed extension, case-insensitive |
No extension + application/json; charset=utf-8 |
text |
Exact normalized MIME match |
page.html + text/plain |
none |
Explicit HTML extension exclusion |
page.txt + text/html |
text |
Extension admits it; HTML source renders as inert text |
notes.md + text/plain |
text |
Exact MIME admits raw text |
image.svg + image/svg+xml |
image |
Handled by existing image path |
Resource Contract
The hard binary limit is:
An object of exactly 1,048,576 bytes is eligible; 1,048,577 bytes is rejected as too large.
Known Sizes
- If the selected object version has a known size exceeding
1,048,576bytes, opening the modal goes directly to the Too Large state without issuing a body download request. Clicking Retry re-probes with a boundedRangerequest in case the listed size was stale. - A known zero-byte size still issues the bounded request to verify authorization and existence, rendering the explicit File is empty state on an empty
200 OK. - Missing size metadata (
undefined) is preserved as unknown rather than coerced to0, entering the bounded stream reader.
Bounded Request and Sentinel Byte
For objects with a size within 1 MiB or an unknown size, the client sends:
Requesting 1,048,576 + 1 bytes uses the 1,048,577th byte as an over-limit sentinel. The client:
- Checks
Content-RangeandContent-Lengthheaders immediately on response arrival. - Reads the response body as a
ReadableStreamrather than callingresponse.text()or buffering an unboundedBlob. - Accumulates at most
1,048,577bytes in memory. - Aborts the stream immediately if the sentinel byte arrives (including when an upstream proxy ignores
Rangeand streams a200 OK). - Renders text only after end-of-stream proves the complete object fits within
1,048,576bytes.
Request Identity and Cancellation
Each preview request is keyed by bucket + object name + versionId and preserves same-origin credentials, Console subpath routing, version_id query parameters, and X-Anonymous: 1 headers. Closing the modal, switching objects or versions, or unmounting the component triggers both an AbortController signal and a generation-token invalidation check so a late stream completion can never overwrite a newer preview. Aborted requests clean up silently without firing error toasts.
Encoding and Fidelity
Text decoding uses strict UTF-8:
- A leading UTF-8 BOM (
EF BB BF) is stripped automatically without displaying a zero-width character. - Multi-byte Unicode, emoji, tabs,
LF, andCRLFline endings are preserved verbatim. - Invalid UTF-8 byte sequences throw a decode error and transition to the Unsupported Encoding / Binary state rather than substituting
U+FFFD. - Any decoded string containing a
NUL(\0) character is rejected as binary data.
Safe Renderer and UI States
The success view renders a single React text node inside <pre>{content}</pre> with a monospace font, preserved whitespace, horizontal and vertical scrolling (white-space: pre by default to keep aligned logs readable and avoid expensive browser line-wrapping layout on long single-line JSON payloads), and native keyboard focus. The dead <iframe> fallback in the preview router is removed.
UI States and Permissions
The Preview action is enabled in both list and detail views if and only if:
The modal presents distinct states for Loading, Success, Empty (0 bytes), Too Large (> 1 MiB, showing object size, policy limit, Download, and bounded Retry), Invalid UTF-8 / Binary, Forbidden (403), Not Found (404), and Network / Server Error. HTTP error response bodies are never decoded into the preview pane.
Functional and Security Requirements
Functional Requirements
- FR1–FR3: Preserve legacy media/PDF routing, classify text fallback via the normative extension/MIME matrix, and render complete eligible objects up to 1 MiB as strict UTF-8 source.
- FR4–FR6: Render no partial content for over-limit objects, provide an explicit empty-file state, and bind metadata, size, and body requests to the selected
versionId. - FR7–FR9: Preserve anonymous access, subpath routing, unified list/detail action eligibility, and existing download/share/media contracts.
Security Requirements
- SR1–SR4: Render object bytes solely through
<pre>text content without any HTML/XML/SVG/Markdown parser, retain at most1,048,577bytes, and invalidate in-flight requests on close or identity change. - SR5–SR8: Reject invalid UTF-8 and
NULbytes, keep preview text out of Redux/storage/logs/error bodies, rely on server S3 authorization, and leave CSP andsafeMimeTypesunchanged.
Implementation Scope
- Refactor preview classification so legacy media precedence is preserved and
textis an explicit fallback. - Add the
PreviewTextcomponent with streaming byte bounds, strict UTF-8 decoding,AbortControllercancellation, and explicit modal states. - Remove the unreachable generic
<iframe>fallback, align object-detail and listPrevieweligibility, and preserveundefinedunknown sizes.
Rejected Alternatives
Keep Text Preview Disabled
Rejected because inspecting small logs, status files, and JSON/XML configurations is a daily operational workflow, and forcing a local file download for every check is unnecessary friction.
Reuse the Legacy Same-Origin Iframe
Rejected because serving uploader-controlled content inside a same-origin <iframe> turns mutable MIME metadata into a script execution boundary and leaves browser memory use unbounded.
Add a Dedicated Backend Preview Endpoint
Rejected because the authenticated download endpoint already enforces GetObject / GetObjectVersion permissions and supports HTTP Range requests (bytes=0-1048576). Duplicating that logic in a second endpoint would widen the API surface without adding a new security boundary.
Truncate Large Files to the First 1 MiB
Rejected because displaying a 1 MiB prefix of a 50 MiB JSON or XML file presents structurally broken markup as if it were the object, and requires extra multi-byte UTF-8 boundary slicing. Complete-or-nothing preview keeps the contract unambiguous.
Decode Invalid UTF-8 with Replacement Characters
Rejected because silent U+FFFD substitution corrupts text that operators copy out of the preview modal. Legacy single-byte or UTF-16 encodings should be inspected via Download.
Auto-Format JSON with Parse and Stringify
Rejected because JSON.parse followed by JSON.stringify silently mutates 64-bit integers exceeding Number.MAX_SAFE_INTEGER, collapses duplicate keys, and alters source whitespace.
Embed Monaco or a Full Code Editor
Rejected because web workers, syntax grammars, and editor bundle weight are unjustified for a bounded 1 MiB read-only viewer. A single <pre> text node is fast, predictable, and trivial to audit.
Verification and Acceptance Plan
Classification Matrix
Automated tests lock every row of the two-stage extension/MIME matrix, case insensitivity, parameter stripping, .html/.htm/.xhtml exclusion, and media precedence.
Resource Bounds
Tests cover 0 bytes, 1 byte, 1,048,576 bytes, 1,048,577 bytes, known over-limit skip, bounded Retry, 206 Partial Content, 200 OK when Range is ignored, and stream cancellation on close.
Encoding and Fidelity
Covers UTF-8 CJK and emoji, BOM stripping, invalid UTF-8 sequences, embedded NUL bytes, 64-bit JSON integers, duplicate JSON keys, and XML DOCTYPE/CDATA/stylesheet directives.
Security Payloads
Verifies that <script>, inline event handlers, <iframe>, SVG onload, and XML XXE/XSLT payloads appear literally in <pre>.textContent, create zero child DOM elements, and trigger zero network requests.
Permissions and Version Races
Confirms that 401/403/404/416/5xx bodies never render as preview content and that late responses from an older request generation cannot overwrite a newly selected object or versionId.
Browser Regression
End-to-end Playwright checks against a live SILO + Console instance verify English and Chinese locales, light and dark themes, and desktop and narrow viewports alongside media, PDF, download, share, and version workflows.
Delivery and Completion Gates
Shipped in pgsty/silo-console v2.2.0 (PR #30) and embedded in SILO Server 20260903, resolving pgsty/silo#17.
Trade-Off Summary
This design chooses explicit scope over a generic browser viewer, complete small files over truncated fragments, source fidelity over automatic reformatting, strict UTF-8 over lossy decoding, and a single inert text node over an embedded code editor. Large logs (> 1 MiB) and non-UTF-8 files still require clicking Download—a deliberate boundary that keeps Console text preview small enough to audit and strong enough to trust.
Design Review Record
Reviewed across product scope, frontend security/CSP isolation, and S3 Range/versioning semantics prior to merging silo-console#30.
4.22 - Multi-Pool Deletion: Preserve the Latest Copy on Failure
Decision, October 24, 2026: merge the corrected PR #257 (shipped in Server 20261026). Commit aafe4f8886 fixes the multi-pool deletion failures reported in issue #256 and closes the in-flight failure window after preflight checks. The core invariant is direct: finish cleaning up non-authoritative copies in other pools before deleting the authoritative copy that determines current reads.
Denis Babchuk reported the issue and contributed the initial patch. Code review reproduced the failures, identified a remaining execution-time failure window in bulk deletion, and extended the fix with two-stage execution, per-item response identity preservation, partial-deletion retry handling, and shared tier-object protection.
Background and Earlier Work
SILO uses capacity-weighted pool allocation. Overwriting an unversioned key can place the new value in pool B while an older physical copy remains in pool A. Reads resolve the latest logical object across pools, keeping the stale copy invisible until a subsequent delete encounters an error partway through execution.
Earlier multi-pool consistency fixes relate to this change as follows:
| Earlier work | What it protects | Relationship to this change |
|---|---|---|
| #178 | Shared object locking and cross-pool reconciliation for conditional deletion and addressed versions | Provides the locking and reconciliation foundation reused here |
| #199, fixed by #207 | Conditional PUT evaluates the current logical object across pools |
Retains capacity-weighted placement |
| #248, shipped in Server 20261018 | Pool migration cannot overwrite a newer payload or drop a delete marker | Protects rebalance and decommission paths |
| #245, closed without merging | Proposed keeping overwrites in their existing pool | Neither experiment met its performance target; deletion safety does not depend on overwrite affinity |
Those earlier changes did not cover ordinary unconditional deletion or bulk DeleteObjects. See Multi-Pool Object Consistency for the shared locking and migration design, and Overwrite Pool Affinity for the evaluation that retained the existing allocator.
The Failure: A Delete Can Expose an Overwritten Value
Consider this initial state:
If both pools execute a delete in parallel, pool B can finish removing "latest" before pool A fails with an I/O error. A single DELETE then returns an error to the caller, but the latest acknowledged value is already gone and the next GET returns "old".
Issue #256 documented four symptoms:
| Operation | Original failure behavior |
|---|---|
Ordinary unversioned DELETE |
Returns an error after removing the latest copy, exposing an older overwritten value |
Bulk DeleteObjects |
A successful pool result masks another pool’s deletion failure |
| Bulk deletion with an explicit version ID | Reports success even though the addressed version remains in another pool |
Versioned single DELETE |
Attempts to write the delete marker in an unwritable holding pool even when another readable holding pool can accept it |
Running the author’s 11 regression subtests showed 10 failures on baseline main and 11 passes on the initial PR. However, review found that bulk deletion could still lose the latest copy when a drive failed after the preflight check.
Why Preflight Was Not Enough
The initial patch checked whether affected erasure sets had enough online drives before starting bulk deletion. While that rejects known offline quorum loss early, online drive counts cannot guarantee that subsequent metadata reads, writes, or shard deletions will succeed.
Review tested a 45-case fault-injection matrix that started with all drives online, allowed the authoritative pool to finish deleting the latest copy, and then injected a failure inside DeleteVersions on the non-authoritative pool (covering simulated EIO, simulated storage disconnects, and real filesystem permission errors across every authoritative-pool position in two- and three-pool layouts, for unversioned, explicit null, and UUID-versioned objects):
- Baseline
main(000e653038): Masked the per-item failure in all 45 cases. - Initial PR (
a30270d9ba): Reported the error, but lost the latest value in all 30 unversioned and explicit-nullcases. The 15 UUID cases used identical payloads across copies of the same version and retained a copy while reporting the error.
Review also uncovered a separate response-identity bug: an early-error DeletedObject omitted ReplicationState, which the HTTP handler requires to reconstruct the original request entry’s identity. In a signed bulk request with two blocked replicated version purges, the response returned one error entry and one empty deleted entry. Preserving only the key and version ID was insufficient.
Per-Item Deletion Contract
Under the existing shared object write lock, the repair enforces the following per-item rules:
- Resolve the addressed object or version across pools and identify its authoritative copy.
- Complete cleanup in every non-authoritative pool (Stage 0) before attempting to delete the authoritative copy (Stage 1).
- If non-authoritative cleanup fails, return an error for that exact request item and leave its authoritative copy untouched.
- Allow independent items in the same batch to complete normally.
Key operational boundaries:
- Per-item protection rather than whole-batch rollback: If an object has copies in three pools, Stage 0 may remove one older copy before failing on the second older copy. Stage 1 is then skipped so the healthy authoritative copy remains intact, and unrelated items that succeeded in the same batch are not rolled back.
- In-flight errors on the final authoritative delete: Once Stage 0 succeeds across all non-authoritative pools, Stage 1 deletes the authoritative copy. If a disk failure occurs midway through Stage 1 itself, the caller should inspect the object state and retry after recovery. The two-stage ordering specifically prevents removing the authoritative copy when another pool’s cleanup has failed.
- Explicit version or marker deletion: Deleting an explicitly addressed historical version or delete marker legitimately makes the next older retained version current under standard S3 versioning rules.
The Implemented Design
Single-Object Deletion
Ordinary unversioned deletion retires older copies before the newest copy and stops at the first failure. Conditional deletion and explicit-version deletion continue using deleteObjectReconciled. If the selected unversioned object has completed a tier transition, it also routes through deleteObjectReconciled so local cleanup preserves shared remote references.
When creating a delete marker for a single object held in multiple pools, pool selection skips pools known to lack delete quorum and chooses a readable holding pool that passes the check.
Bulk Data Deletion in Two Stages
The new deleteObjectsReconciled reuses the existing namespace write locks and objectPoolInfos resolver without introducing a second locking domain.
Lookup concurrency is capped at 32 objects. Each lookup inspects all pools for the addressed version, orders candidates by ModTime, and breaks exact timestamp ties by the lowest pool index to match read selection. Unreadable metadata is treated as an error rather than evidence of absence.
Pools still execute in parallel within each stage using the existing erasure-set and disk-level bulk interfaces. Because two objects in the same batch can have their authoritative copies in different pools, a fixed global pool order cannot work; stages must be partitioned per item by authoritative pool.
Stage 0 also includes non-authoritative pools where lookup reported the object absent: a read-quorum absence does not prove that a temporarily unavailable drive holds no stale shard. If that pool cannot complete the delete, the item conservatively returns an error.
Every result entry initializes with the original key, version ID, and full ReplicationState, and explicit index mappings preserve request order across filtering and re-batching. The online-drive preflight remains in place to reject known quorum loss early, while two-stage execution enforces runtime safety.
Versioning and Marker Ordering
The batch entry point separates physical deletion from delete-marker creation using the same versioning rules as the lower layer:
| Request | Action |
|---|---|
Explicit versionId=null or UUID |
Physically remove the addressed version using two-stage cleanup in multi-pool batches |
| No version ID, ordinary unversioned key | Physically remove data using two-stage cleanup |
| No version ID, versioning enabled | Create a delete marker and retain historical versions |
| No version ID, versioning suspended | Create or replace the null delete marker |
| Prefix-specific versioning configuration | Evaluate PrefixEnabledFn for the key alongside the suspended state |
Physical deletion runs before marker creation, preserving execution order when a single batch explicitly purges a null version and creates a new delete marker for the same key. Marker-only batches continue using their existing path (see Compatibility, Retry, and Operational Boundaries for degraded-mode marker edge cases).
Retry After Partial Disk Cleanup
A second failure mode arises when a pool deletes a version from some drives before failing on the remaining drives. On retry, drives that just deleted the version return success while drives that already deleted it on the first attempt return errFileNotFound or errFileVersionNotFound. Counting those two response groups separately can prevent either group from reaching quorum.
The fix in erasureObjects.DeleteObjects combines successful removals with errFileNotFound and errFileVersionNotFound only for pure physical deletion (unknown I/O errors never count toward quorum). Delete-marker creation and replication-state updates still require actual successful writes to reach quorum so an absent version cannot falsely acknowledge a marker or metadata transition that was never written.
Both single-version and bulk deletion paths (including marker writes on this path) use a delete quorum of floor(N/2) + 1, where N is the erasure set’s drive count.
Shared Tier Objects
Physical copies in different pools can reference the same remote tier object, including when one copy has been restored locally. Retiring a non-authoritative local record must not enqueue that remote tier object for garbage collection while the retained authoritative copy still references it.
Bulk items with completed tier transitions therefore route through the existing per-object reconciler, preserving SkipFreeVersion, replication state, and prefix versioning options. That subset executes sequentially, which also prevents concurrent metadata updates for duplicate entries in the same batch, while ordinary local objects stay on the two-stage bulk path. Added regression tests cover shared and distinct remote references, restored copies, pre-authoritative failure, and explicit SkipFreeVersion requests.
Verification at the Final Revision
The commits evaluated during review and verification are:
| Revision | Role |
|---|---|
000e653038 |
Baseline main (without #245 overwrite affinity) |
a30270d9ba |
Initial PR head; reproduced the post-preflight failure window |
3c82748e14 |
Added two-stage bulk cleanup, response identity preservation, and partial-deletion retry |
a2073d2514 |
Final PR head, adding shared-tier protection |
aafe4f8886 |
Merged main (source tree identical to a2073d2514) |
The committed fault tests start with all drives online, inject failures inside DeleteVersions, and assert both an accurate per-item error and zero deletion attempts against that item’s authoritative copy, verifying GET during failure, GET and version listing after recovery, retry completion, and normal completion of unrelated items:
| Coverage | Evidence in the merged tree |
|---|---|
Post-preflight I/O failures, two/three pools, every authoritative position, unversioned/null/UUID |
TestPoolsBulkDeleteKeepsLatestOnIOFailure |
| Signed HTTP response identity with replication state | TestPoolsBulkDeleteReplicationIdentity |
| Mixed null purge/marker, partial cleanup/retry, different authoritative pools | Bulk ordering and retry tests |
Equal timestamps, unreadable metadata, shared lock/concurrent PUT, object-layer cancellation |
Selection and concurrency tests |
| Physical-delete quorum versus marker/replication writes | TestBulkDeleteQuorumEvidence |
| Shared tier references and restored objects | TestPoolsRetiringCopyPreservesSharedTierObject |
| Original single-delete, bulk-error, known-quorum-loss, and marker-placement regressions | Original contribution’s tests |
The new regressions fail deterministically against the initial PR’s production code. The combined deletion, tier, and quorum test suite passed three consecutive -race runs before merge and an additional -race run after merge. Supplemental checks covered single-pool controls, prefix versioning configurations, directory and duplicate keys, Object Lock, version purges, and migration regressions.
Both final-head Go CI and merged-main Go CI passed full cmd and internal suites, race suites, static analysis, and cross-compilation for Linux, macOS, and Windows on amd64 and arm64, alongside DCO and VulnCheck.
Regression test command:
Live mcli verification ran against a two-pool server process (four directory drives per pool): populate data on the baseline binary, restart onto the fixed binary to verify reads, bulk deletes, UUID version deletes, enabled/suspended delete markers, and new writes, and then restart the baseline binary over the same data directories.
Performance: Measured Deletion Cost
Benchmarks compare baseline 000e653038 against final PR head a2073d2514 (identical to merged main).
Method
- Hardware: Apple M5 Max (18 physical cores, 128 GiB RAM) with one 8 TB Apple SSD (APFS, ~95% full during testing; all logical directory drives shared this physical device).
- Configuration: Four directory drives per pool (EC 2+2), 4 KiB objects, 8 concurrent workers,
GOMAXPROCS=8, exercising the object layer directly. - Matrix: Five
pools / copieslayouts (where “copies” denotes physical copies of one logical object across pools rather than erasure shards), testing both singleDELETE(1,024 requests per run) and 64-object bulk deletes (64 requests / 4,096 objects per run). - Sampling: Three alternating pairs per layout and operation (baseline→fixed, fixed→baseline, baseline→fixed), recreating the identical initial state before timing each run. Total: 60 run samples, 30,720 single-delete requests, 1,920 bulk requests, and 153,600 logical objects deleted with zero errors.
Results and Variation
Throughput is measured in logical objects per second, showing the three-run median and observed range; paired change is the median of the three paired percentage changes.
| Pools / copies; operation | Baseline median [range] | Fixed median [range] | Paired change |
|---|---|---|---|
1 / 1; DELETE |
1,286 [1,193–1,292] | 1,347 [1,269–1,371] | +6.1% |
| 1 / 1; bulk 64 | 1,786 [1,739–1,809] | 1,818 [1,680–1,982] | +4.5% |
2 / 1; DELETE |
1,350 [1,272–1,363] | 1,304 [1,270–1,327] | −1.7% |
| 2 / 1; bulk 64 | 1,477 [1,468–1,740] | 1,453 [1,149–1,604] | −7.8% |
2 / 2; DELETE |
560 [546–641] | 534 [469–534] | −16.3% |
| 2 / 2; bulk 64 | 606 [528–695] | 620 [546–707] | +2.2% |
3 / 1; DELETE |
988 [722–994] | 1,053 [974–1,130] | +13.6% |
| 3 / 1; bulk 64 | 1,338 [1,269–1,383] | 1,120 [1,055–1,163] | −16.9% |
3 / 3; DELETE |
364 [323–371] | 354 [335–364] | −2.0% |
| 3 / 3; bulk 64 | 404 [367–450] | 365 [313–378] | −14.5% |
Per-request latency in milliseconds (covering the full 64-object batch for bulk calls, shown as baseline→fixed):
| Pools / copies; operation | P50, ms | P99, ms |
|---|---|---|
1 / 1; DELETE |
5.69 → 5.50 | 15.06 → 14.37 |
| 1 / 1; bulk 64 | 277.28 → 280.00 | 406.23 → 385.23 |
2 / 1; DELETE |
5.46 → 5.47 | 14.09 → 14.88 |
| 2 / 1; bulk 64 | 317.10 → 346.26 | 635.16 → 463.78 |
2 / 2; DELETE |
12.91 → 13.76 | 35.86 → 45.54 |
| 2 / 2; bulk 64 | 823.19 → 752.78 | 1,315.81 → 1,297.48 |
3 / 1; DELETE |
7.23 → 6.42 | 22.09 → 20.20 |
| 3 / 1; bulk 64 | 361.73 → 444.34 | 669.84 → 732.27 |
3 / 3; DELETE |
20.35 → 21.00 | 50.75 → 48.92 |
| 3 / 3; bulk 64 | 1,236.42 → 1,392.63 | 1,697.38 → 2,079.27 |
P99 uses the nearest-rank method (with 64 requests per bulk run, P99 is close to the per-run maximum). Single-pool controls retained their scheduling path and showed no regression. Three-pool bulk throughput decreased by 15–17%, with a 22.5% P99 increase on the 3-pool / 3-copy layout; two-pool / two-copy single deletion decreased by 16.3% while bulk deletion in that layout stayed within run-to-run variation.
Where the Cost Comes From
Single deletion serializes cleanup of older copies before the newest copy. Bulk deletion adds cross-pool metadata resolution and orders Stage 0 before Stage 1 while preserving cross-pool parallelism and disk batching within each stage. For a batch of $B$ items across $P$ pools with $N$ drives per erasure set, resolution inspects $B \times P$ pool states and the online preflight checks up to $B \times P \times N$ drive states. Because the shared key write locks span this work, the added latency can delay concurrent writes to the same keys. Accepting this multi-pool deletion cost is necessary to guarantee that failed deletes never drop the latest copy.
Compatibility, Retry, and Operational Boundaries
- Error codes and client retries: A single-delete write-quorum failure typically returns HTTP 503, whereas bulk
DeleteObjectscan return HTTP 200 with per-item errors such asSlowDownWritein the response XML. WithMaxRetries=4, the tested S3 SDK did not automatically retry an HTTP 200 bulk response containing per-itemSlowDownWrite, whilemcli rmsurfaces the per-item failure. Callers must inspect bulk response entries and retry failed items after storage recovery, preserving the targetversionIdwhere applicable. - Concurrency and cancellation: The shared object write locks serialize multi-pool deletion against participating pooled writes. Because the HTTP bulk-delete handler uses a detached background context, a client disconnect does not cancel an in-flight batch.
- Suspended-bucket degraded bulk markers: Supplemental testing noted an existing edge case already present on baseline
main: in two- and three-pool layouts, when a versioning-suspended bucket with UUID-only history receives a bulk delete marker during drive degradation, explicitly purging thenullmarker after drive recovery can still return a quorum error (one run exposed the retained UUID history and another kept the marker). The marker-only bulk path remains unchanged here. - Object Lock and conditional delete: Object Lock and retention authorization checks are unchanged. Batch XML ETag conditions remain unsupported; see Conditional DELETE.
- Upgrade and rollback: No S3 parameter, RPC wire format, on-disk format, dependency, allocator rule, or configuration flag changes. During a mixed-version rolling upgrade, an unupgraded coordinating node still applies the older deletion path; rolling back to an older release cannot reconstruct a latest copy that was already lost.
Rejected Alternatives
| Alternative | Why it was rejected |
|---|---|
| Error aggregation alone | Prevents false-success replies, but cannot recover an authoritative copy that was already deleted |
| Online-drive preflight alone | Cannot prevent an I/O failure that occurs between the preflight check and the actual disk mutation |
| One fixed global pool order | Different objects in the same batch can have their authoritative copies in different pools |
| Serial per-object deletion for all bulk items | Unnecessarily sacrifices erasure-set and disk-level batching for ordinary local objects; the sequential per-object path is reserved only for items with shared tier references |
| Overwrite pool affinity (#245) | Regressed small-object writes and still left existing cross-pool duplicates, migrated copies, and interrupted cleanups to handle during deletion |
| Cross-pool rollback transactions or config flags | Two-stage per-item cleanup protects the authoritative copy without introducing distributed rollback state or extra configuration switches |
4.23 - Go 1.27 TLS Defaults and OIDC Discovery Failure Modes
After SILO’s toolchain moved to Go 1.27, the Server TLS repair 48e184652 (“fix(tls): honor Go key exchange defaults across transports”, shipped in Server 20260916) removed explicit curve overrides across eight Server TLS configuration points. This article records the Go 1.27 TLS changes, the phase-based diagnostic method for OIDC discovery failures developed during the investigation of issue #154, and the health-endpoint behavior when an identity provider is unreachable at startup.
For the post-release retest analysis — why GODEBUG=tlsmlkem=0 disables hybrid key exchange but does not suppress the 12-byte ML-DSA signature algorithm offer, and the configuration options for TLS-inspecting middleboxes — see Pinned TLS Parameters and Handshake Compatibility.
What Go 1.27 Changed
- Explicit
CurvePreferencesnow override the ML-KEM compatibility switches.GODEBUG=tlsmlkem=0removes all ML-KEM hybrids from the default curve set;tlssecpmlkem=0removes only the P-256/P-384 hybrids introduced in Go 1.26 while retainingX25519MLKEM768. In Go 1.27, any application that setsCurvePreferencesexplicitly retains ML-KEM in that list regardless of theGODEBUGswitches. SILO Server had eight TLS configuration points setting an explicit list that includedX25519MLKEM768; commit48e184652removes all eight assignments and retires the helper so those configuration points use Go’s default curve set and honor theGODEBUGswitches. Stack-wide review confirmed thatsilo-pkg,mcli, and Console clients already used the standard library defaults, while Console’s HTTPS listener retains its separate P-256-only policy. ClientHellonow advertises ML-DSA signature algorithms (code points0x0904–0x0906). Because ML-DSA is a signature scheme rather than a key-exchange mechanism, disabling hybrid key exchange leaves the ML-DSA offer intact, and middleboxes that reject unknown signature-algorithm code points are unaffected by ML-KEM switches.ClientHellosize increased. Measured on identical source and dependencies: Go 1.26.5 defaults produce a 1497-byteClientHello; Go 1.27.1 defaults produce 1509 bytes; undertlsmlkem=0, the old explicit curve list produced a 275-byteClientHellowithout ML-KEM on Go 1.26.5, whereas Go 1.27.1 with that same explicit list still produced a 1509-byteClientHellocontaining ML-KEM. Upgrading the compiler alone altered the outbound handshake payload.- macOS root-CA resolution follows the main module’s
godirective. Whether a process honorsSSL_CERT_FILE/SSL_CERT_DIRor queries the macOS Keychain is controlled by thex509sslcertoverrideplatformGODEBUG setting, whose default tracks the top-level main module’sgodirective:go 1.26main modules ignore both variables on macOS in favor of the platform Keychain, whereasgo 1.27main modules honor them (regardless of oldergodirectives in library dependencies). On macOS, setting either environment variable replaces Keychain trust entirely with the specified file or directory; if that path points to a stale or incomplete bundle, certificate chains that the Keychain would have trusted fail verification until the variable is unset. - Unchanged TLS and platform baselines. TLS protocol versions, cipher suites, certificate and hostname verification, proxy resolution, and HTTP/2 negotiation remain unchanged, and audited Go 1.27 runtime changes such as the standard-library response-body drain cap (256 KiB / 50 ms) do not affect SILO. Go 1.27 binaries require macOS 13 or later, and the module graph across Server, Console, and
mcrequires Go ≥ 1.27.1.
Why the OIDC-Only Patch Was Withdrawn
An initial candidate patch cleared CurvePreferences only on the OIDC discovery transport. We discarded that approach because the same transport constructor backs identity plugins, notification and Lambda reachability probes, audit webhooks, and S3 cloud-storage tiering — patching only the two OIDC call sites would have left every other outbound TLS consumer on the hardcoded curve list. The merged fix removes the explicit curve overrides across all eight Server TLS configuration points while keeping certificate verification strict and avoiding protocol downgrades or silent fallbacks.
Diagnosing a Discovery Failure by Phase
The startup sequence runs: server start → identity subsystem initialization → fetch .well-known/openid-configuration (discovery) → fetch jwks_uri keys → IAM store ready → Console initialization. Console’s OIDC configuration dialog validates through the same server-side transport. A failure at any step leaves IAM offline; even if the discovery document loads, a 503 or timeout when fetching jwks_uri still prevents IAM from coming online.
Isolate the failure by the phase where the connection terminates:
- TCP reset immediately after
ClientHello(reset right aftertls_start): inspect ingress or middlebox handling of theClientHellopacket — proxyCONNECTpolicies, TLS terminators, or middleboxes filtering on packet size or extension code points. All Go 1.27 handshake changes surface at this step. - Reset or error after the TLS handshake completes (reset after
wrote_request): the TLS handshake succeeded; check HTTP-layer policies such as WAF rules,User-Agentallowlists (the server’s default User-Agent changed fromMinIOtoSilo), and layer-7 routing. Changing certificates or key-exchange settings will not affect failures at this stage. - x509 verification errors: inspect the certificate chain returned by the peer, the SNI server name sent, and the trust store selected by the process (on macOS, check whether
SSL_CERT_FILEorSSL_CERT_DIRis overriding the Keychain). - Run diagnostic probes from the exact network namespace and egress path of the failing process, matching source IP and proxy settings.
Health Endpoints and IAM Status
While IAM is offline, /minio/health/live and /minio/health/ready both continue to return HTTP 200, as neither probe gates on the identity subsystem. /minio/health/cluster checks identity initialization and returns 503 Service Unavailable with the response header X-Minio-Server-Status: iam-offline. Monitoring designed to catch broken IdP integrations should probe /minio/health/cluster alongside an authenticated request.
Identity initialization retries automatically in the background at randomized 0–3 s intervals; once the IdP becomes reachable, IAM comes online without a process restart (measured at sub-second to ~1.4 s in local tests). Background retries only recover from transient reachability failures — a ClientHello rejected by a middlebox will fail on every retry until the TLS configuration or middlebox policy is updated.
Discovery and JWKS Transport Parameters
The OIDC discovery and JWKS client constructs a dedicated http.Transport with the following parameters: HTTP/2 disabled (no h2 ALPN, HTTP/1.1 only); proxy settings read exclusively from HTTPS_PROXY and NO_PROXY (uppercase preferred; ALL_PROXY is ignored); DNS cache refresh defaulting to 30 s in Kubernetes/Docker environments and 10 min elsewhere (overridable via the DNS cache TTL setting); sequential address dialing without shuffling; and per-phase timeouts of 5 s per TCP dial, 10 s for the TLS handshake, and 1 min for response headers. No overall request timeout is set on the discovery or JWKS fetch itself, so a stalling IdP can hold startup initialization open indefinitely.
Attribution
This note distills the issue #154 investigation and the September Go 1.27 stack review. Each behavior described above was verified with synthetic experiments (ClientHello packet captures, fresh-process root-CA probes, and local fixture reproductions) and cross-checked against the Go 1.27 release notes and the crypto/tls and crypto/x509 standard library implementations.
4.24 - No I/O Before Auth, No Privilege From Headers
This note documents the CORS hot-path and replication-request trust repair merged in PR #101 (938603458 through 04b097fd9), tracked as SN-2026-008, and shipped in Server 20260903.
Overview
Two defects that appeared unrelated shared a single architectural root cause:
- Any request carrying an
Originheader caused the outermost CORS middleware to treat the first URL path segment as a bucket name and synchronously load its metadata from disk before authentication. - Any request carrying
X-Minio-Source-Replication-Requestcaused downstream handlers to treat the operation as internal replication merely because the header was present.
In both cases, untrusted request syntax acquired expensive or privileged internal meaning before crossing an authorization boundary.
The repair enforces two invariants:
For CORS, the outer middleware reads only bucket metadata already resident in memory. For replication, handlers authenticate the original signed request first, authorize the required replication action (s3:ReplicateObject or s3:ReplicateDelete), and attach a private trust decision to the request context. Untrusted internal headers are stripped only after SigV4 verification completes. Downstream option builders, encryption paths, Object Lock checks, event emitters, and metadata writers consume that context decision rather than inspecting raw headers.
Failure A: Pre-Authentication CORS Amplification
corsHandler wraps the entire HTTP router. Every request carrying an Origin header reaches it before SigV4 authentication, request validation, and API concurrency limiters.
Previously, per-bucket CORS evaluation called the general bucket metadata loader:
When .metadata.bin did not exist on disk, the loader probed ten legacy configuration file paths. When none existed, it returned a newly allocated empty BucketMetadata struct rather than NoSuchBucket, and GetConfig inserted that synthetic entry into the in-memory metadataMap.
An unauthenticated client could therefore send Origin-bearing requests for an unbounded sequence of syntactically valid, nonexistent bucket names and trigger two effects per distinct name:
- Unthrottled erasure/storage metadata reads (
.metadata.binplus ten legacy probes) before authentication. - Unbounded growth of the in-memory bucket
metadataMap(pruned only every 15 minutes on distributed clusters, and never pruned until restart on single-node deployments).
Failure B: A Marker Header Treated as Authority
SILO and MinIO-compatible replication engines exchange internal headers to preserve source state across sites, anchored by:
Before this repair, multiple code paths treated the presence or string value of that header as proof that the caller was a trusted replication peer:
GetObject,HeadObject, andGetObjectAttributeson an SSE-C object could setNoDecryptionand return raw ciphertext without requiring the customer key to any caller holding ordinarys3:GetObjectpermission.- Caller-supplied source ETag and modification time (
MTime) could overwrite server-generated values. - Source tagging, retention, and legal-hold timestamps could influence last-writer-wins conflict resolution.
- Past Object Lock
RetainUntilDatetimestamps were accepted via a raw header check. - Delete-marker version IDs and timestamps could be injected by the caller.
- Object creation and access notifications could be suppressed.
- Multipart actual object size and encrypted checksum metadata could be injected during
CompleteMultipartUpload. X-Amz-Replication-Status: REPLICAcould be persisted from ordinaryPutObject,CopyObject, Snowball/PAX archives, or browserPOSTpolicy uploads.
The earlier CVE-2026-34204 repair prevented ordinary PutObject and CopyObject calls from importing replication SSE metadata, but it did not establish a single post-authorization authority shared by all consumers of replication headers.
Replication Trust Design
One Exact Marker, Two Trust Levels
The marker header X-Minio-Source-Replication-Request is considered present only when it appears as a single header line with the exact lowercase value "true". Multiple header values, mixed case, or any other string are treated as untrusted.
After authenticating the caller, the handler derives two distinct trust flags:
| Decision | Requirements | Enabled Semantics |
|---|---|---|
trusted |
Authenticated non-anonymous principal + exact marker + s3:ReplicateObject (or s3:ReplicateDelete) on the target resource |
Source ETag/MTime and timestamps; multipart actual size and encrypted checksum transfer; notification and re-replication suppression; delete version/pool pinning |
replicaTrusted |
trusted + raw request status REPLICA (or a multipart upload whose stored initiation status is REPLICA) |
REPLICA status persistence; replication SSE sealed-key import; SSE-C raw ciphertext (NoDecryption) path; replica Object Lock exceptions |
Separating trusted from replicaTrusted is required by the wire protocol because legitimate multipart part uploads, multipart completions, and batch replication writes send X-Minio-Source-Replication-Request: true without repeating X-Amz-Replication-Status: REPLICA on every request.
Incoming requests are classified as follows:
| Incoming Request Shape | Receiver Outcome |
|---|---|
| No marker | Ordinary S3 operation |
| Marker present, caller lacks replication permission | Internal replication headers stripped and ignored; request proceeds as an ordinary S3 operation |
X-Amz-Replication-Status: REPLICA without replication permission |
Rejected with 403 AccessDenied |
Exact marker + replication permission, no REPLICA |
trusted only |
Exact marker + replication permission + REPLICA |
Both trusted and replicaTrusted |
Explicitly rejecting unauthorized REPLICA claims with 403 AccessDenied prevents a misconfigured replica write from silently downgrading into a new PENDING source object that would trigger a reverse replication loop.
Authenticate Original Request, Then Sanitize
Because AWS SigV4 signs x-amz-* and x-minio-* headers, stripping internal headers before signature verification would alter the canonical request and fail valid requests with SignatureDoesNotMatch. The execution order is therefore strict:
The header-stripped request clone shares the original http.Request.Trailer map so aws-chunked streaming-checksum uploads continue working for ordinary clients. Snowball/PAX batch archives isolate trust per archive entry (ff44527a3) while preserving legitimate request defaults across worker goroutines (ab3ae99ca).
Replica Status Removed from Generic Metadata
X-Amz-Replication-Status is removed from the generic user-metadata extraction allowlist. Ordinary PutObject, CopyObject, CreateMultipartUpload, Snowball extraction, and HTML POST policy forms can no longer persist REPLICA status by submitting the header or form field; the server sets REPLICA only inside a verified replicaTrusted branch.
Object Lock Receives an Explicit Decision
The Object Lock parser no longer inspects X-Minio-Source-Replication-Request directly. Instead, handlers pass allowPastRetainDate explicitly from replicaTrusted, and use the same replicaTrusted decision when checking whether an existing compliance or legal-hold version may be overwritten by an incoming replica.
Replication Wire Matrix
| Operation | Marker | REPLICA Header on Request |
Receiver Decision |
|---|---|---|---|
Replicated PutObject |
Yes | Yes | replicaTrusted |
Replicated CreateMultipartUpload |
Yes | Yes | Persist trusted multipart REPLICA provenance |
Replicated UploadPart |
Yes | No | trusted; promoted to replicaTrusted only when stored MPU status is REPLICA |
Replicated CompleteMultipartUpload |
Yes | No | trusted; preserves source ETag/MTime, actual size, and encrypted checksum |
CopyObject metadata replication |
Yes | Yes | replicaTrusted |
Replicated DeleteObject |
Yes | Yes | replicaTrusted (requires s3:ReplicateDelete plus s3:DeleteObject, with no explicit deny on s3:DeleteObjectVersion) |
Batch replication PUT / Complete |
Yes | No | trusted (target credentials must hold s3:ReplicateObject) |
| Proxy / readiness / validity probes | Probe headers | No | Probe headers are preserved; validity probes verify replication permissions under the rule prefix (c9ad74673, 5db7be4ee) |
CORS Resident-Only State Machine
The outer CORS middleware calls a dedicated resident-only accessor that acquires a single in-memory read lock and performs zero storage I/O:
| In-Memory Bucket Metadata State | CORS Decision | Storage I/O |
|---|---|---|
| Resident, valid per-bucket CORS config | Apply per-bucket CORS rule (a failed background refresh retains the last valid config) | None |
| Resident, no per-bucket CORS config | Use global CORS fallback | None |
| Resident, corrupted stored CORS XML | Fail closed (omit CORS headers and log once) | None |
| Non-resident while initial startup load is still running | Fail closed | None |
| Non-resident after startup: known disk bucket whose metadata failed to load | Fail closed | None |
Non-resident after startup: nonexistent, reserved, invalid, or .minio.sys name |
Use global CORS fallback | None |
Both startup-in-progress and known load-failure states fail closed: because a presigned URL carries its own credentials in the query string, the bucket’s custom CORS policy is the only browser origin boundary protecting it, and falling back to the permissive global CORS policy during a metadata load failure would expose the presigned URL to arbitrary origins.
Rejected Alternatives
| Rejected Alternative | Reason for Rejection |
|---|---|
| Validate bucket syntax before calling the old CORS loader | Attackers can generate an unbounded stream of syntactically valid, nonexistent bucket names that still trigger pre-auth storage I/O |
Call GetBucketInfo before loading CORS |
Replaces 11 metadata probes with at least 1 unthrottled storage stat call per attacker-supplied bucket name |
| Cache negative bucket lookups with a TTL | Bounds cache lifetime, not attacker key cardinality or initial storage I/O amplification |
| Strip replication headers before SigV4 authentication | Breaks SigV4 canonical request verification (SignatureDoesNotMatch) |
Reject every request carrying X-Minio-Source-Replication-Request |
Breaks legitimate marker-only batch replication and multipart replication calls |
Require REPLICA on every trusted replication call |
Breaks replicated UploadPart, CompleteMultipartUpload, and batch replication wire compatibility |
Store a boolean in ObjectOptions while leaving events and Object Lock on raw headers |
Leaves two authorities that can disagree |
Implementation and Verification
The implementation is structured into seven boundaries:
- A dedicated request-trust helper parses the exact marker, evaluates replication IAM permissions, binds the private trust struct to
req.Context(), and returns the post-auth sanitized request clone. putOpts,copyDstOpts,getOpts, anddelOptsbuilders extract source replication fields only when the context decision istrustedorreplicaTrusted.DecryptObjectInfo, notification event builders, multipart completion, delete options, and Object Lock consume the same context decision.- S3 handlers compute replication trust immediately after their authentication check.
PutObjectPartcombines current-requesttrustedstate with the multipart upload’s persistedREPLICAprovenance.- Generic metadata extraction excludes
X-Amz-Replication-Status. corsHandlercalls the resident-only metadata accessor and never invokes the load-on-miss getter.
Verification Matrix
Automated and black-box two-site TLS replication tests verify:
- Hundreds of distinct nonexistent bucket names across actual and preflight CORS requests with zero storage metadata reads and zero
metadataMapgrowth. - Least-privilege SSE-C
GetObject,HeadObject, andGetObjectAttributescalls rejecting forged, wrong-case, or unauthorized replication markers. - Marker-only batch
PutObjectpreserving source ETag/MTime only when the caller holdss3:ReplicateObject. - Unauthorized
REPLICAPutObjectandDeleteObjectreturning403 AccessDenied. - Browser
POSTpolicy uploads unable to injectREPLICAstatus. - Object Lock past retention dates rejected without
replicaTrustedand accepted for authorized replicas. - End-to-end SSE-C single-part (
4 KiB), multipart (12 MiB, 3 parts),CopyObject, and delete-marker replication across two live TLS-enabled SILO nodes with matching ETag, version ID, SSE-C key MD5, and decrypted SHA-256 digests.
Compatibility and Follow-Up Work
- Ordinary clients: Unchanged; untrusted internal replication headers are ignored rather than acquiring privileged internal behavior.
- Batch replication: Target credentials must include
s3:ReplicateObjectas documented in Batch Replication; without it, marker-only writes fall back to ordinary writes and generate fresh ETag andMTimevalues. - Rolling upgrades: Wire and on-disk formats are unchanged. Upgraded nodes enforce the resident-only CORS lookup and replication trust boundary immediately.
Subsequent Follow-Ups
- Replication execution reliability (2026-09-09): Replication Reliability: Delete Completion, MRF Visibility, and Resync Cancellation documents the fixes in PR #162 for #153, #152, and #137 after trusted requests enter the replication pipeline.
- Rate-limited diagnostics: Receivers log a rate-limited warning when a marker-bearing request lacks replication permission so operators can spot misconfigured batch replication policies without log flooding.
Conclusion
Merged in 0079723d3 (PR #159) and shipped in Server 20260916, the resident-only CORS lookup and post-authentication replication trust boundary close both the unauthenticated bucket-metadata probe path and untrusted replication-header injection.
4.25 - One Endpoint, Two Privileges: Separating User and Group Status
This article records the authorization design and verification for upstream issue minio/minio#21478, SILO PR #73 (commits 58735ee38 / 2e2377d1c), and the set-group-status follow-up (229fe2b3c), which shipped in Server 20260903.
Too Long; Didn’t Read (TL;DR)
SILO exposes both admin:EnableUser and admin:DisableUser, but the shared set-user-status handler historically authorized every request with admin:EnableUser. A policy granting only admin:DisableUser could not disable an account; operators had to grant admin:EnableUser as well, breaking the least-privilege boundary between the two actions.
The fix selects a single required action from the requested target state before authorization:
| Requested status | Required action |
|---|---|
enabled |
admin:EnableUser |
disabled |
admin:DisableUser |
| invalid or unknown | admin:EnableUser, preserving the previous authorization-before-validation default |
The handler then calls validateAdminReq once. A four-way IAM integration test verifies both allowed operations and both denied cross-action operations. Enable-only policies can no longer disable accounts.
The same rule applies to group status:
| Requested group status | Required action |
|---|---|
enabled |
admin:EnableGroup |
disabled |
admin:DisableGroup |
| invalid or unknown | admin:EnableGroup, preserving the previous authorization-before-validation default |
Before the follow-up, an EnableGroup-only principal could disable a group, while a DisableGroup-only principal received AccessDenied for that operation. The group repair uses the same single-selector, single-authorization structure.
The Reported Defect
The Admin API uses one route for both state transitions:
Before the repair, the handler checked one fixed action before reading the requested status:
The subsequent call to SetUserStatus accepted either enabled or disabled, but authorization had already treated both requests as Enable operations. admin:DisableUser existed in the policy vocabulary and documentation but could not authorize this endpoint on its own.
Issue #21478 demonstrated the operational impact: an operator wanted an incident-response policy that could disable compromised accounts without being able to re-enable them. A policy containing only admin:DisableUser failed with AccessDenied; adding admin:EnableUser allowed the disable call to succeed, but also granted the account-recovery permission the policy was meant to withhold:
Why Two Actions Must Mean Two Capabilities
Account state transitions are directional. Disabling is frequently delegated to incident responders, fraud controls, or compliance automation, whereas re-enabling restores access and often requires a separate approver.
The authorization contract therefore enforces strict separation:
| Principal policy | Disable target | Enable target |
|---|---|---|
admin:DisableUser only |
allow | deny |
admin:EnableUser only |
deny | allow |
| both actions | allow | allow |
| neither action | deny | deny |
The built-in consoleAdmin policy grants admin:*, so full administrators retain both operations. Only custom restricted policies that relied on admin:EnableUser to disable accounts are affected. The public PBAC reference documents the same contract for admin:EnableUser and admin:DisableUser.
Design Goals and Non-Goals
Goals
- Enforce both existing Admin Actions according to their names.
- Preserve least privilege in both directions.
- Perform one authorization decision and write at most one authorization error per request.
- Preserve the route, query parameters, response format, self-mutation guard, IAM storage call, and site-replication hook.
- Lock in the four-way permission matrix with regression tests.
Non-Goals
- split the endpoint into separate enable and disable routes;
- add a new combined action or change policy syntax;
- change user status persistence or replication;
- redesign Console permissions.
Evaluated and Rejected Alternatives
Keep Checking admin:EnableUser for Both States
This leaves admin:DisableUser non-functional and forces over-privileged policies.
Require Both Actions for Either Transition
This prevents delegating disable-only permissions to incident-response roles.
Try Enable Authorization, Then Retry Disable Authorization
Upstream PR #21482 attempted this approach for disable requests: it called validateAdminReq with EnableUser, and if that returned nil, called validateAdminReq a second time with DisableUser.
However, when validateAdminReq returns a nil object layer, it has already written an HTTP error response. For a Disable-only caller, the first check commits a 403 response before the second check succeeds and mutates account state. Authorization fallback must never mutate state after committing an error response.
Accept Either Enable or Disable for a Disabled Request
validateAdminReq accepts a variadic action list and succeeds if any listed action is allowed. Passing both actions in a single variadic call would allow Disable-only policies to work while letting Enable-only policies continue to disable users.
SILO rejected this option because allowing Enable-only policies to disable accounts violates least privilege. Roles that manage the full account lifecycle should grant both actions explicitly.
Validate the Status Before Authenticating
Rejecting unknown status values before calling validateAdminReq would change error precedence, allowing unauthenticated or unauthorized callers to probe parameter validation before passing the authorization gate.
Unknown values therefore continue to select admin:EnableUser as the default authorization action. Only disabled selects admin:DisableUser; after authorization succeeds, the existing IAM layer rejects invalid status strings.
The Selected Implementation
The repair adds a pure selector function:
The handler reads the route variables, selects the action, and authorizes once:
All logic after the authorization gate is unchanged:
- callers still cannot enable or disable their own account;
globalIAMSys.SetUserStatusvalidates and persists the requested status;- site replication records the status and timestamp;
- response formatting and audit logging follow the existing path.
The selector inspects only the requested target state in the query string. It does not load the target user before authentication or branch on whether the target exists.
Why the Repair Is Safe
Five invariants hold after the change:
- Every valid status maps to exactly one Admin Action.
validateAdminReqis called once, so a failed authorization never proceeds to mutation.- State mutation is reachable only after the selected action is authorized.
- Invalid status values preserve the prior Enable authorization boundary and are rejected afterward by the existing validation path.
- Storage, replication, wire format, and client SDK contracts remain unchanged.
Test Design
Pure Action Mapping
The unit test covers all three selector outcomes:
| Input | Expected action |
|---|---|
enabled |
EnableUser |
disabled |
DisableUser |
| invalid | legacy EnableUser default |
Four-Way IAM Authorization Matrix
The integration test creates dedicated users and policies and exercises the Admin API across all four combinations:
- a Disable-only client disables a target user;
- the same client receives
AccessDeniedwhen attempting to enable that user; - an Enable-only client enables the target user;
- the same client receives
AccessDeniedwhen attempting to disable that user.
The two cross-action denial checks guard against regressions that might accidentally permit both transitions. The test runs inside the IAM server suite — covering request signing, policy attachment, handler authorization, persistence, and Admin-client error decoding — and cleans up all temporary users and policies on completion.
Verification Record
Local verification commands:
Commit 58735ee38 in PR #73 passed all eight remote CI checks (DCO sign-off, format/build/vet, lint and generated files, cmd/ tests, internal/ tests, race detector and S3 Select, cross compilation, and vulnerability analysis) and merged as 2e2377d1c.
Least-Privilege Policy Examples
Disable-Only Operator
This principal can inspect and disable another user, but cannot re-enable it.
Enable-Only Operator
This principal can inspect and enable another user, but cannot disable it. Roles responsible for the full account lifecycle should grant both actions explicitly.
Group-Status Follow-Up
The group status endpoint shares the same structure as the user endpoint:
It exposes admin:EnableGroup and admin:DisableGroup, but the inherited handler likewise checked EnableGroup before inspecting status.
The follow-up commit (229fe2b3c) adds setGroupStatusAdminAction, mirroring setUserStatusAdminAction:
The accompanying integration test creates isolated EnableGroup-only and DisableGroup-only administrators and a target group, verifying that:
- DisableGroup-only can disable;
- DisableGroup-only cannot enable;
- EnableGroup-only can enable;
- EnableGroup-only cannot disable.
Invalid status values still select the default Enable action before hitting the existing validation error, avoiding any pre-authentication validation oracle. The site-replication hook runs only after a permitted mutation succeeds.
Compatibility and Migration
No client or API changes are required; the endpoint, query parameters, status strings, response format, and Admin-client methods are unchanged.
Administrators should review custom restricted policies:
- roles that only disable users require
admin:DisableUser; - roles that only enable users require
admin:EnableUser; - roles that perform both operations require both actions;
consoleAdminand otheradmin:*policies are unaffected;- legacy custom policies granting only
admin:EnableUsercan no longer disable users and must addadmin:DisableUserif both operations are intended.
The same rules apply to group-management roles:
- roles that only disable groups require
admin:DisableGroup; - roles that only enable groups require
admin:EnableGroup; - roles that perform both operations require both actions;
- legacy EnableGroup-only roles can no longer disable groups.
Upstream Disposition and Delivery
Upstream issue #21478 and PR #21482 remain on the archived minio/minio repository as historical context.
Delivery State
| Milestone | User repair | Group follow-up |
|---|---|---|
| Signed-off commit | 58735ee38 (merged as 2e2377d1c) |
229fe2b3c |
| Shipped release | Server 20260903 | Server 20260903 |
Conclusion
Enabling and disabling users or groups carry opposite operational risks. Selecting the required action from the target state and authorizing once before mutation aligns handler enforcement with the IAM policy vocabulary.
This enable/disable repair is independent of the later admin:ChangeMyPassword split; see the password migration guide when upgrading across September releases.
4.26 - IAM Admin API Authorization Boundaries
Server 20261018 ships Server PR #242 (commit 2616d43fb on baseline 00a375c4e, merged as 4e55fc7db) and the bulk self-listing follow-up 995439260 to close SN-2026-017. This design note records the problem scope, handler-level authorization contract, empty-policy claim handling, damaged-key deletion semantics, performance measurements, rejected alternatives, and regression verification.
Problem and scope
Several Admin handlers treated ownership as a fallback after authorization had already denied a request, and listing one’s own service accounts could skip authorization altogether. Those paths could override an explicit user or group deny, an inline session-policy restriction, or an external authorization plugin’s denial.
Credential creation and update exposed a second boundary. A derived credential whose session policy allowed those Admin actions could create a key inheriting broader parent permissions, or remove a service account’s restriction through an update. A non-root administrator with broad Admin permissions could also manage root’s derived credentials. Under built-in authorization, an action grant alone is insufficient for these operations.
The repair covers inspection, cleanup, creation, and update handlers, including bulk key listing and the LDAP creation endpoint. It preserves the policy engine’s global behavior, existing Admin routes and actions, request/response formats, and IAM storage formats without dependency version changes.
Here, built-in authorization means no external authorization plugin is configured; it includes local IAM evaluation for LDAP/OIDC identities. See access management and external authorization for the surrounding policy model.
Authorization contract
The authenticated access key remains the actor supplied to IAM. For ownership
comparisons, its effective user is ParentUser when present, otherwise
AccessKey. Resolving ownership never substitutes the parent for a derived
credential during policy evaluation.
Self inspection and cleanup pass DenyOnly: true into authorization.
Ordinary users need no explicit allow for these established self-service
operations, while an applicable explicit deny still takes effect. Derived
credentials continue through full IAM evaluation, where an active session policy
must explicitly allow the operation. An external plugin’s denial is final, and
handlers no longer turn a denial into success when the target belongs to the
caller.
| Handler | Existing Admin action | Self behavior and additional boundary |
|---|---|---|
InfoServiceAccount |
admin:ListServiceAccounts |
Compare the effective user with the stored service account parent; authorize before returning information. |
ListServiceAccounts |
admin:ListServiceAccounts |
Empty user, caller access key, and caller parent are self aliases. Authorize every request and list the parent for these aliases. |
ListAccessKeysBulk |
admin:ListServiceAccounts |
Normalize self aliases to the parent and use the same self-service helper; cross-user and all=true requests retain their additional authorization requirements. |
InfoAccessKey |
admin:ListServiceAccounts |
Self requires an existing derived target with the same parent. Missing targets do not create a self exception. |
DeleteServiceAccount |
admin:RemoveServiceAccount |
Authorize self cleanup; apply the root boundary; use stored identity metadata even when the target token is damaged. |
RevokeTokens |
admin:RemoveServiceAccount |
Omitted user or the resolved caller parent is self. Authorize before revocation and protect root targets. |
AddServiceAccount, AddServiceAccountLDAP |
admin:CreateServiceAccount |
Retain existing self/other-user IAM authorization; then reject active session policies and unauthorized effective root targets. |
UpdateServiceAccount |
admin:UpdateServiceAccount |
Retain the existing action requirement; then reject active session policies and unauthorized root targets. No new self-update privilege. |
Creation and update are denied to session-limited callers even when their session policy grants the named action. The update restriction also covers metadata-only requests because the existing update implementation can clear the old inline policy when the request omits a new policy.
Three local helpers
Production changes are confined to cmd/admin-handlers-users.go and the claim
copy loop in cmd/admin-handlers-idp-ldap.go. Three small helpers express the
boundaries while leaving IAMSys.IsAllowed unchanged.
hasActiveSessionPolicy reuses the existing session-policy evaluators,
checking temporary credentials before service accounts to match IAM dispatch.
An expiring service account satisfies IsTemp() and is evaluated as STS.
Consequently, a legacy expiring credential marked inherited-policy can still
carry an effective inline restriction; checking the inherited marker first
would permit that credential to create or update broader keys. Non-temporary
service accounts follow the existing service-account evaluator’s inherited
marker and empty-policy rules.
canManageRootCredentials is an additional guard after successful built-in
authorization. It accepts the root key or an unrestricted root-derived
credential (excluding the internal site replication service account). The
signature verifier’s owner flag is cleared whenever the raw session-policy
claim key is present, even when the service-account evaluator treats that
policy as empty. Checking for an active restriction preserves management access
for historical root service accounts created with null, {}, or
{"Statement":null} policies.
isAllowedServiceAccountAction preserves ordinary users’ self inspection and
cleanup after their last policy attachment is removed. For built-in, non-owner,
non-derived callers with DenyOnly, it resolves user and group policies and
evaluates the combined policy (allowing an empty combined policy while honoring
explicit denies). All other callers use IAMSys.IsAllowed, keeping the engine’s
global no-policy behavior unchanged for S3 and other Admin operations.
The creation, update, deletion, and revocation root guards apply only under built-in authorization. When an external authorization plugin is configured, it decides these operations — including root targets — and formerly bypassed self requests now consult the plugin as well.
Empty policies and claim ownership
Code review identified an availability defect in the creation path: copying a
parent service account’s empty inline policy into a new inherited key leaves
stale internal policy claims. If the new key has an expiration, STS evaluation
rejects that empty policy because it has no Version field, leaving the newly
created key unable to perform its intended operations.
The same failure occurs when a creation request explicitly specifies a semantic empty policy together with an expiration. Both cases are resolved at the Admin issuance boundary:
- Both creation handlers exclude
exp,policy.SessionPolicyName,sessionPolicyNameExtracted, andiamPolicyClaimNameSA()when copying the caller’s claims. The new key receives its own expiration and requested policy, while preserving other identity claims including LDAP/OIDC context. commonAddServiceAccountnormalizes a parsed policy to inheritance only whenVersion == "" && Statements == nil && ID == "", matching the empty-policy definition in the service-account evaluator. It clears both the local policy option and the returned request’s policy field so the site replication create message matches the local key.
| Requested child policy | Result |
|---|---|
Omitted, null, {}, {"Statement":null} |
Inherit parent permissions, including when the new key has an expiration. |
| A policy allowing only selected actions | Retain the requested restriction; unrelated actions remain denied. |
{"Version":"2012-10-17"} or {"Statement":[]} |
Do not normalize to inheritance; the expiring-key regression cases remain denied. |
Normalization checks Statements == nil rather than len(Statements) == 0 so
that only the Console empty-policy placeholder forms are normalized to
inheritance.
IAMSys.NewServiceAccount is also called by IAM import and site replication.
Sanitizing claims in that shared constructor could discard an effective
restriction carried only in a historical expiring credential’s claims. The
shared constructor and historical record replay therefore remain unchanged; the
fix applies at the Admin issuance boundary.
Deletion and error behavior
DeleteServiceAccount reads stored identity metadata directly via
globalIAMSys.store.GetUser, checks the service-account kind, and retains the
stored parent without decoding the target session token. The underlying delete
operation uses the same lookup primitive, bypassing both token parsing and the
public user lookup’s disabled/expired credential filter while keeping the
store’s parent-revocation checks.
An authorized administrator can clean up a damaged, disabled, or expired non-root service account, and a non-root administrator cannot use token corruption to conceal a root parent. Preserving the stored parent also ensures that the site replication hook sends deletion messages when appropriate.
Denied deletion of another user’s key, root-guard rejection, and a missing
service account all return ErrAdminServiceAccountNotFound (HTTP 404). A denied
self operation returns ErrAccessDenied (HTTP 403). Other endpoints retain
their existing lookup and error ordering.
Compatibility and limits
The behavioral changes enforce existing denies on self operations, reject credential creation or update by session-limited callers, and protect root credential mutations under built-in authorization. Automation that relied on the old bypasses must use a properly authorized, unrestricted identity. Ordinary users retain their self-inspection and cleanup rights (including after policy detachment), and root’s historical empty-policy service accounts retain their management capabilities.
Newly created empty-policy keys store inherited-policy metadata instead of an
embedded empty policy document, removing the evaluation mismatch for expiring
keys. Reserved internal policy claims are no longer copied from the caller to
the newly issued key and reflect only the new credential’s own attributes.
ListAccessKeysBulk uses the same self-service helper through the October 18 follow-up 995439260. Ordinary users without attached policies can inspect themselves, while explicit denies, derived-credential session restrictions, and cross-user authorization remain enforced. Omitted users, or a single users parameter naming the caller key or parent, resolve to the parent inventory; all=true still requires admin:ListUsers. Regression tests decrypt responses and verify identity contents across ordinary, service-account, and STS aliases.
Existing unusable expiring credentials are not rewritten in place; an authorized parent or administrator should reissue the intended key, verify its access, and remove the old key. Do not bulk-strip policy claims from imported or replicated historical credentials, as that would remove valid session restrictions.
For distributed revocation ordering and concurrent administrative mutation boundaries, see durable IAM revocations.
Cost and rejected alternatives
The #242 production diff adds 59 net lines across two handler files, and the October 18 bulk-list follow-up reuses the same helper. It introduces no new cache, background worker, lock protocol, storage migration, or S3 data-path change. Claim filtering runs inside existing copy loops, and empty-policy normalization reuses the policy already parsed for creation.
Admin creation and update checks may parse the caller’s inline policy (bounded by the 4 KiB service-account policy limit), reusing the existing evaluators rather than maintaining a second definition of an active policy. Self-inspection requests that previously skipped authorization now evaluate policy through the IAM cache, touching storage only when the combined-policy loader encounters a missing document. In external-plugin deployments, newly enforced checks issue a plugin request whose latency depends on that external service.
A local benchmark on Darwin/arm64 (Apple M5 Max, Go 1.27.2, three 200 ms samples) measured:
- Ordinary/inherited session checks: ~11–16 ns with 0 allocations.
- Empty service policy check: ~0.30–0.32 µs.
- ~3 KiB inline policy check: ~10 µs.
- Cached ordinary-user self evaluation: sub-microsecond range matching the existing IAM path (800 B, 15 allocs/op).
These costs require no additional cache or specialized policy representation on the Admin path.
Five alternative approaches were evaluated and rejected:
- Modifying global
IAMSys.IsAllowedbehavior: Rejected because changing global no-policy or self-ownership rules would alter unrelated S3 and Admin operations. - Treating raw session-policy claim presence as a restriction: Rejected because historical root service accounts created with
null,{}, or{"Statement":null}carry the raw claim key while having no effective restriction. - Stripping policy claims centrally in
IAMSys.NewServiceAccount: Rejected becauseNewServiceAccountis shared with IAM import and site replication, where stripping claims could discard an effective restriction carried in a historical expiring credential’s claims. - General policy-containment solver for session-limited delegation: Rejected because session-limited credentials are designed as leaf credentials rather than sub-issuers, and disallowing creation/update from active session policies closes the escape path in 59 lines.
- Unconditional denial of damaged-key cleanup: Rejected because administrators need a way to delete corrupted, disabled, or expired non-root service accounts using stored identity metadata.
Regression evidence
Commit 2616d43fbc3400a02cc1e6f9299e26ae8b2e71a6 and follow-up 9954392605827b213877c45262c741e638598dd8 were verified with the following suites:
| Check | Result |
|---|---|
Complete TestIAM* / TestSTS* selection with real etcd, LDAP, two OIDC providers, and an authorization plugin |
57 top-level tests; 380 passing records including parent tests and subtests; 0 failures, 0 skips. |
TestIAMAdmin* regression tests using signed requests and encrypted Admin payloads |
Passed. |
Focused race run for TestIAMAdmin* and the peer delete/reload test |
Passed. |
| Authentication, identity configuration, and policy configuration packages | Passed. |
make verifiers and make build |
Passed lint, generated-file checks, rebrand/compatibility checks, and binary build. |
Hosted CI checks on PR #242 |
All 8 checks passed (DCO, vulnerability analysis, format/build/vet, lint/generated files, race, cross compilation, internal tests, and cmd tests). |
cmd/admin-handlers-users-authz_test.go covers root empty
policies, policyless self operations, parent/group/session/plugin denials,
legacy expiring inherited credentials, non-root access to root targets,
disabled/expired damaged-key cleanup, deletion replication, self aliases, and
empty-policy child issuance with positive and negative access checks. The LDAP
suite also issues expiring child keys through AddServiceAccountLDAP and
verifies S3 access. Additional boundary probes verified damaged
root-key cleanup boundaries and STS explicit/implicit denials across the five self handlers.
Two existing test fixtures were also tightened: the peer reload test clears stale etcd client state before constructing its object-storage fixture, and the IAM import test derives default policy names from policy.DefaultPolicies instead of a hardcoded list.
Verification commands:
Rollout and upgrade notes
- Shipped in: Server 20261018 (PR
#242, commits2616d43fb,4e55fc7db, and995439260). - Advisory: SN-2026-017.
- Upgrade every server node that accepts Admin traffic so mixed-version routing cannot send a request through an unpatched handler. No new on-disk format or bulk credential rewrite is required.
4.27 - Config Environment Files Are Not Shell Scripts
This note defines the startup contract for MINIO_CONFIG_ENV_FILE and documents the compatibility repair committed as 2aea7fe9c and shipped in Server 20260903. The file is a SILO input format; supporting an optional export prefix for operator convenience does not make it a POSIX shell script.
Overview
SILO can load startup environment variables from a file:
The parser accepts assignments such as:
The last two keys illustrate why shell-identifier validation is too narrow. Multi-target configuration appends the target name verbatim after an underscore. The configuration subsystem does not restrict target names to POSIX shell identifiers: names containing -, ., :, digits, or printable Unicode can be discovered and resolved directly.
An earlier hardening change validated every environment-file key against [A-Za-z_][A-Za-z0-9_]*. That regex rejected my-hook and prevented the server from starting on restart even though the previous loader and the configuration target subsystem accepted it. The repair validates the invariants that SILO and the operating-system environment actually require:
- The key name is non-empty, valid UTF-8, and composed of visible non-whitespace characters.
=and NUL are forbidden in a key name.- NUL is forbidden in a value.
- Invalid input reports the file path and line number without logging the value.
- The entire file is parsed and validated before any environment assignment is applied.
Why the Regression Was Real
The environment-file loader calls os.Setenv after parsing. An operating-system environment block is an array of KEY=VALUE strings, not a POSIX shell variable namespace. Shell assignment syntax is narrower only because a shell must tokenize and expand variable references inside its own command language.
Named SILO configuration targets follow the pattern:
For example:
Target discovery enumerates environment variables by matching the fixed MINIO_<SUBSYSTEM>_<PARAMETER>_ prefix and treats the remaining suffix verbatim as the target ID. Target lookup reconstructs the exact environment variable name without uppercasing or sanitizing that suffix. Rejecting - in the environment-file parser broke a valid discover-to-resolve path without protecting any shell evaluation path, because no shell ever evaluates MINIO_CONFIG_ENV_FILE.
The failure mode is operationally sharp because MINIO_CONFIG_ENV_FILE is read only at process startup. A server could continue running with its existing in-memory environment and then fail on its next restart after a binary upgrade or config reload. Startup must fail fast on malformed input, but the file parser must not enforce a narrower target grammar than the configuration subsystem itself.
File Grammar
Lines and Comments
- Blank lines and whitespace-only lines are ignored.
- Any line whose first non-whitespace character is
#is ignored as a comment. - An optional standalone
exportkeyword followed by whitespace is stripped before key parsing. exportFOO=valuepreserves the keyexportFOO; it is not mistaken for theexportprefix.- The first
=separates the key from the value; any subsequent=characters remain part of the value.
The file is not executed by a shell. It does not perform $VAR expansion, command substitution, backslash escape processing, or inline # comment stripping inside values.
Keys
Leading and trailing whitespace around the key is trimmed. The remaining key must:
- Be non-empty valid UTF-8.
- Contain only Unicode graphic characters (
unicode.IsGraphic). - Contain no whitespace,
=, NUL, control characters, or invisible Unicode format characters.
This grammar preserves OS-compatible variable names and multi-target suffixes while rejecting visually empty or structurally ambiguous keys. Keys beginning with a digit or punctuation character are accepted by the parser; SILO still reads only the exact configuration names used by its subsystems.
Values and Quoting
Unquoted values have leading and trailing whitespace trimmed. To preserve significant leading or trailing spaces, wrap the entire value in matching single or double quotes:
The parser strips at most one matching outer quote pair and does not interpret backslash escapes inside quotes. NUL bytes are always rejected because operating-system environment entries are NUL-terminated C strings.
Failure and Secrecy Contract
Syntax errors abort server startup immediately. Diagnostic messages include the file path, 1-based line number, and the invalid key name or error category, but never print the value—ensuring that a secret or password on a malformed line is never copied into startup logs.
Parsing is atomic (all-or-nothing): a syntax error on any line discards all parsed entries, and os.Setenv calls begin only after the entire file validates. If the operating system rejects a validated os.Setenv call, SILO aborts startup and reports the key name and file path so the process never serves traffic with a partially applied environment.
Because MINIO_CONFIG_ENV_FILE carries root credentials, KMS keys, and webhook tokens, operators must still protect the file with strict filesystem ownership and permissions (0600).
Regression Matrix
The unit and configuration test suites cover:
- Spaces and tabs around
=. - Single- and double-quoted values with significant surrounding spaces.
- Standalone
exportprefixes, including Unicode whitespace afterexport. - Keys beginning with
_, a digit, or punctuation. - Named targets containing
-,.,:, and printable Unicode characters. - End-to-end named-target discovery through the configuration subsystem.
- Empty keys, whitespace inside keys, NUL bytes, and invisible Unicode format characters (such as zero-width spaces).
- NUL bytes inside values.
- Multiple
=characters in URLs, query strings, and base64 tokens. - File-and-line diagnostics that redact secret values.
- All-or-nothing parse failure guarantees.
Compatibility and the Warm-Restart Trap
No configuration migration is required. Existing standard environment variable names behave identically, files using spaces around = parse predictably, and named targets with hyphens or dots work again.
Four visible behaviors are intentional:
- Invalid or invisible key names fail startup immediately instead of being silently ignored.
- Unquoted surrounding value whitespace is trimmed; quote values when leading or trailing spaces are significant.
- Malformed lines stop startup with a redacted, line-numbered diagnostic.
- Valid punctuation-bearing target names are no longer rejected merely because POSIX
shcannot assign them with literalNAME=valuesyntax.
The original #65 investigation also uncovered a subtle warm-restart trap: the legacy parser split on = without trimming key whitespace, storing KEY = new as the trailing-space key KEY . A systemd cold start often masked this bug because systemd parses EnvironmentFile= itself and strips spaces around =. However, when an operator triggered an in-place warm restart (mcli admin service restart), the re-executed SILO process inherited KEY=old from the parent environment, and the newly parsed KEY entry failed to overwrite KEY. The server could therefore restart and continue running with stale root credentials, KMS settings, or identity provider configuration. With key trimming restored in Server 20260903, KEY = new overwrites KEY on both cold start and warm restart; operators who had space-padded assignments in MINIO_CONFIG_ENV_FILE should verify that those values match their intended active configuration before upgrading.
Conclusion
Restoring config.ReadEnvFile in silo-pkg/v3 (1a0d788) and updating the server (a643483be) aligns MINIO_CONFIG_ENV_FILE parsing across cold starts and warm restarts while preserving full backward compatibility with standard shell-style environment files.
4.28 - Two SSE-C Keys, One CopyObject Response
This note documents the CopyObject SSE-C checksum response repair committed as e73436c99 and shipped in Server 20260903. It covers the successful CopyObject XML and HTTP response after the destination object has committed; stored object bytes, checksum metadata, encryption formats, source decryption, federation, and replication remain unchanged.
Overview
An SSE-C CopyObject request can supply two independent customer-provided encryption keys:
| Role | Request Headers | Purpose |
|---|---|---|
| Source | X-Amz-Copy-Source-Server-Side-Encryption-Customer-* |
Decrypt the source object and its stored metadata |
| Destination | X-Amz-Server-Side-Encryption-Customer-* |
Encrypt and later interpret the committed destination object |
SILO encrypted the destination object and sealed its checksum metadata under the destination key. However, after the write committed, both the CopyObjectResult XML builder and the generic PUT-response header helper received the original CopyObject request headers. Because decryptChecksums prioritizes X-Amz-Copy-Source-Server-Side-Encryption-Customer-* headers whenever they are present—which is right while reading the source object—it attempted to unseal the committed destination’s checksum metadata using the source key.
With source key A and destination key B:
The stored destination object and its encrypted checksum metadata were valid, but the one-time CopyObject response omitted the checksum fields. The repair constructs a destination response-header view by stripping the three copy-source SSE-C headers, decrypts the committed destination checksum metadata once under key B, and reuses that map for both the CopyObjectResult XML body and the HTTP response headers.
Observable Failure
Triggering the bug required a checksum-bearing destination object and distinct source and destination SSE-C keys:
Before the repair:
CopyObjectsucceeded with HTTP 200.GetObjecton the destination with keyBreturned the expected plaintext body.HeadObjecton the destination with keyBdecrypted the stored checksum metadata and returnedx-amz-checksum-crc32andx-amz-checksum-type.- The
CopyObjectXML body and HTTP response headers omitted bothChecksumCRC32/x-amz-checksum-crc32andChecksumType/x-amz-checksum-type.
The same mismatch affected in-place SSE-C key rotation (copying an object onto itself with a new SSE-C key). Once the destination metadata was resealed under key B, the request still carried the old key A in the copy-source headers, causing response generation to fail checksum decryption.
Why the Global Decrypter Must Not Change
decryptChecksums prefers copy-source SSE-C headers for a valid reason: earlier in CopyObject, the server inspects the source object’s checksum metadata to decide whether to carry its algorithm forward, recompute a full-object digest from a composite source, or attach a default CRC64NVME checksum. When the source object is SSE-C encrypted, its checksum metadata is sealed under the source key and requires the X-Amz-Copy-Source-Server-Side-Encryption-Customer-* headers.
Inverting the global header priority inside decryptChecksums to prefer destination SSE-C headers would fix the post-commit response while breaking pre-commit source checksum inspection. The boundary is temporal and object-specific:
Implementation
Destination Response View
After the destination write commits, the handler clones the request header map and deletes the three copy-source SSE-C headers:
X-Amz-Copy-Source-Server-Side-Encryption-Customer-AlgorithmX-Amz-Copy-Source-Server-Side-Encryption-Customer-KeyX-Amz-Copy-Source-Server-Side-Encryption-Customer-Key-MD5
Destination X-Amz-Server-Side-Encryption-Customer-* headers remain intact. SSE-S3 and SSE-KMS destination metadata require no customer header and continue through the existing path.
Decrypt Once, Project Twice
Previously, CopyObject called decryptChecksums once while constructing the CopyObjectResult XML struct and a second time inside setPutObjHeaders while writing HTTP response headers. For SSE-S3 or SSE-KMS objects, that duplicated KMS or master-key unseal work.
The repaired flow decrypts the destination checksum map once and projects it into both outputs:
The general setPutObjHeaders helper remains unchanged for PutObject, CompleteMultipartUpload, and DeleteObject, while CopyObject passes the pre-decrypted checksum map directly to the shared header writer.
Regression Matrix
The encryption and API test suites cover:
- Plaintext source to SSE-C destination.
- Compressed and uncompressed SSE-C destinations.
- SSE-C source key
Ato SSE-C destination keyB. - Verification of checksum digest and
ChecksumTypein bothCopyObjectResultXML and HTTP response headers. - Verification that stored destination checksums decrypt with key
Band match the destination body. - In-place same-object SSE-C key rotation from key
Ato keyB. SSE-S3source and destination combinations across erasure backends.
Compatibility and Operational Impact
- Successful
CopyObjectresponses: Checksum digest andChecksumTypefields that were previously omitted when copying into an SSE-C destination now appear in both the XML body and HTTP response headers. - Stored objects: No on-disk rewrite, metadata format change, or encryption format change is required; existing objects were already stored with valid destination checksum metadata.
- Performance: Checksum metadata is decrypted once per
CopyObjectresponse instead of twice, with no extra object read or hashing pass. - Rolling upgrades and rollback: Upgraded nodes return the checksum fields immediately while older nodes may still omit them during a rolling upgrade window; stored objects remain readable across both versions.
Conclusion
Decrypting ObjectInfo before building the CopyObjectResult XML payload (21870fa2e) restores the stored checksum fields on SSE-C and KMS copy responses without altering storage layout or encryption semantics.
4.29 - ListMultipartUploads Behavior and Design Trade-Offs
This note documents the ListMultipartUploads design across its full engineering lifecycle: the current MinIO-compatible contract in Server 20261018, the October 18 partial rollback after 16-drive erasure-set capacity testing, the October 17 on-disk scan experiment (#243, merged as 7e9454456), the September 16 opt-in strict mode (#198), and the original August 30 root-cause analysis.
Current Contract in Server 20261018
Server 20261018 restores the traditional MinIO-compatible public ListMultipartUploads contract on main (commit 4733e1423, followed by empty-upload-parent cleanup repair 0f9199e0c):
- Without
prefix,ListMultipartUploadsreturns uploads currently tracked in the node-local in-memory multipart cache. - With a non-empty
prefix,ListMultipartUploadstreatsprefixas an exact object key (sha256(bucket/object)) and checks the multipart directory for that object on disk. delimiter,key-marker,upload-id-marker, andencoding-typeare parsed by the handler, validated where required (encoding-type=url), and echoed in the XML response, while candidate selection follows the MinIO-compatible cache / exact-key path rather than a full.minio.sys/multipartdisk scan.UploadID,GetMultipartInfo,PutObjectPart,ListObjectParts,CompleteMultipartUpload, andAbortMultipartUploadcontinue to work byBucket + Object + UploadIDacross restarts because active uploads live on disk under.minio.sys/multipart/<sha256(bucket/object)>/<uploadID>.MINIO_API_MULTIPART_LISTINGandapi multipart_listingremain accepted as compatibility no-ops if present in older scripts or stored configs.GET /minio/admin/v3/multipart-preflightremains available and reportsmode: "legacy"with0incompatible uploads.
Why the Default Disk Scan Was Rolled Back on 2026-10-18
Between 2026-10-17 (#243, commit 4ed86dfe6) and 2026-10-18 (commit 4733e1423), main briefly used an unindexed .minio.sys/multipart disk scan for all public ListMultipartUploads calls. Capacity analysis and benchmarks on 16-drive erasure sets showed that making this scan the unconditional default created a worse operational failure mode than the listing limitation it replaced.
Why Restart Never Deleted Multipart Uploads
Active multipart uploads are stored on disk under .minio.sys/multipart/<sha256(bucket/object)>/<uploadID>/xl.meta and part.*. A server restart clears only the in-memory multipartUploadsCache; it does not remove unfinished uploads from disk. Retained UploadID values therefore remain usable across restarts until completed, explicitly aborted, or cleaned up by the background stale-upload sweep (default expiry: 7 days; check interval: 12 hours).
Because unfinished uploads accumulate across restarts and client retries, any operation that walks .minio.sys/multipart on disk scales with the total number of unfinished uploads currently stored in the erasure set, not merely the uploads created since the last process start.
How the 100,000-Entry Guard Fails on a 16-Drive Erasure Set
The October 17 disk scan enforced maxMultipartWalkEntries = 100000 across all drives in an erasure set combined, counting both <sha256(bucket/object)> directories and <uploadID> directories on every drive.
On an N-drive erasure set with U active uploads across H distinct keys, a single bucket-wide ListMultipartUploads call traverses:
When each upload uses a distinct object key ($H = U$):
- On a 4-drive set, $4 \times (10{,}000 + 10{,}000) = 80{,}000$ directory entries—just under the
100,000budget. - On a 16-drive set, $16 \times (3{,}126 + 3{,}126) = 100{,}032$ directory entries—exceeding the
100,000budget at 3,126 active uploads across the entire erasure set (all buckets combined) and failing every public scan with HTTP503 SlowDown. - Even when all uploads share a single object key ($H = 1$), a 16-drive set reaches $16 \times (1 + 6{,}250) = 100{,}016$ entries at 6,250 uploads for that key, so even an exact-key lookup (
prefix=<exact-object-key>) failed with503 SlowDown.
Provenance and Responsibility
| Change | Date / Commit | What It Did | Assessment |
|---|---|---|---|
Initial strict mode (#198) |
2026-09-16 / 4cbb074cc |
Added opt-in disk-backed listing behind MINIO_API_MULTIPART_LISTING=strict, keeping legacy as default |
Safe opt-in boundary, but lacked a hard traversal cap |
Security hardening (SN-2026-009..012) |
2026-09-16 / 143f6970d |
Added checkUploadID, maxMultipartWalkEntries = 100000, 503 SlowDown mapping, and readQuorum filtering |
Essential DoS protection for the disk scan, but counted raw per-drive directory entries across all N drives |
Unconditional unification (#243) |
2026-10-17 / 4ed86dfe6 |
Removed legacy and made the disk scan unconditional for all public ListMultipartUploads calls |
Overreach: turned a 4-drive benchmark result into a mandatory path that fails at 3,126 uploads on 16-drive sets |
Partial Rollback Trade-Off Analysis
| Option | Public S3 Listing Fidelity | 16-Drive Capacity Behavior | Storage Format / Background I/O Impact | Decision |
|---|---|---|---|---|
Keep #243 unconditional disk scan |
Full prefix/delimiter/marker listing up to 3,125 uploads on 16 drives | Fails with 503 SlowDown at 3,126 distinct-key uploads across the set |
16 × (1 + H) ListDir RPCs per uncached public listing |
Rejected |
Raise or remove maxMultipartWalkEntries |
Works past 3,126 uploads | Reopens unbounded O($N \times U$) disk walk and memory allocation on unindexed .minio.sys/multipart |
High latency and FD/RPC amplification under concurrent GET /bucket?uploads |
Rejected |
| Build a persistent per-bucket multipart index | Full S3 listing without sha256(bucket/object) full scan |
Scales with matching prefix | Requires a new distributed metadata index, heal protocol, and crash-recovery contract | Deferred (requires dedicated index design) |
Full git revert of #198, 143f6970d, and #243 |
Restores MinIO-compatible listing | No 100,000-entry cliff |
Would also discard checkUploadID path-traversal protection, abort cache/peer cleanup, and xl.meta identity metadata |
Rejected |
Surgical partial rollback (4733e1423 + 0f9199e0c) |
Restores MinIO-compatible public listing (empty-prefix cache + non-empty exact-key lookup) | No cross-bucket 100,000-entry cliff; exact-key lookup touches only 1 hash directory per drive |
Preserves xl.meta identity fields, checkUploadID, abort cleanup, and empty-hash-parent removal |
Selected |
What Was Retained vs Rolled Back
- Rolled back on the public listing path (
4733e1423):- Public
ListMultipartUploadsno longer walks every<sha256(bucket/object)>directory in.minio.sys/multipart. - Empty
prefixreads the in-memorymultipartUploadsCache; non-emptyprefixhashesbucket/prefixdirectly and inspects only that single object’s<sha256(bucket/object)>directory.
- Public
- Retained hardening and storage repairs (
4733e1423+0f9199e0c):NewMultipartUploadstill writesx-minio-internal-multipart-v1-bucketandx-minio-internal-multipart-v1-objectinxl.meta, keeping on-disk uploads self-describing for diagnostics and future indexed designs.checkUploadIDvalidation remains enforced on multipart operations (SN-2026-009).AbortMultipartUploadstill evicts the local cache entry and notifies peer nodes.cleanupStaleUploadscontinues to remove expired uploads on disk, and commit0f9199e0censures that when the last<uploadID>under.minio.sys/multipart/<sha256(bucket/object)>is removed by expiry sweep, completion, or abort, the empty<sha256(bucket/object)>parent directory is also removed so empty hash directories do not accumulate indefinitely.
Cost of Ordering vs Discovery
In .minio.sys/multipart/<sha256(bucket/object)>/<uploadID>, directory names are SHA-256 hashes of bucket/object, which destroys lexicographical key order and bucket boundaries. Answering an arbitrary prefix query (prefix=trips/) in S3 key order without an index requires reading xl.meta inside every active upload across every bucket in the erasure set before sorting and paginating the first page. By contrast, exact-key lookup (prefix=trips/2026/a.mp4) computes a single SHA-256 hash and lists only that one directory per drive (O(N) ListDir calls instead of O(N × H)).
Validation Benchmarks on Commit 4733e1423
Measured with go test -run=^$ -bench=BenchmarkMultipartPublicListingCapacity -benchmem -benchtime=1x ./cmd:
Scenario (BenchmarkMultipartPublicListingCapacity) |
Before (4ed86dfe6 Disk Scan) |
After (4733e1423 MinIO-Compatible Path) |
Latency (4733e1423) |
Memory / Allocs (4733e1423) |
|---|---|---|---|---|
4_drives_10000_distinct_keys/empty_prefix_cached |
200 OK (walked 80,000 entries on disk) |
200 OK (1,000 uploads returned from cache) |
1.12 ms |
257,104 B / 14 allocs |
16_drives_3126_distinct_keys/empty_prefix_cached |
503 SlowDown (100,032 entries > 100,000 cap) |
200 OK (1,000 uploads returned from cache) |
0.36 ms |
90,208 B / 14 allocs |
16_drives_3126_distinct_keys/exact_key_after_cache_clear |
503 SlowDown (walked all 3,126 hash dirs) |
200 OK (1 upload returned from disk after cache clear) |
0.78 ms |
30,024 B / 389 allocs |
16_drives_10000_uploads_one_key/exact_key_lookup |
503 SlowDown (at 6,250 uploads for one key) |
200 OK (1,000 uploads returned from disk) |
383.85 ms |
13,721,672 B / 236,704 allocs |
Related Records
| Document | Scope |
|---|---|
| S3 API Compatibility — Differences from S3 APIs for Multipart Uploads | Canonical user-facing compatibility reference for ListMultipartUploads in Server 20261018 |
| SILO 20260916 Release Notes | Published legacy default, opt-in strict behavior, UploadID validation, and cancellation boundaries |
Historical Record: The October 17 Unification Attempt (#243)
On 2026-10-17, PR #243 (commit 4ed86dfe6) briefly removed the legacy/strict mode switch and routed all ListMultipartUploads requests through the disk-backed implementation introduced in #198. Although rolled back the next day by 4733e1423, the architectural analysis from #243 is preserved below because it documents the exact storage format, quorum rules, and abort-race edge cases of .minio.sys/multipart.
October 17 Architectural Decision Record
Target Contract of the October 17 Experiment
The October 17 experiment attempted to enforce the following invariants on ListMultipartUploads without introducing a separate metadata database:
- Restart durability:
ListMultipartUploadsreads active uploads from.minio.sys/multiparton disk rather than relying exclusively onmultipartUploadsCache. - S3 filter and pagination semantics:
prefixis matched as a string prefix of the object key,delimitercollapses common prefixes intoCommonPrefixes, and results are ordered by(Key, Initiated, UploadID)withkey-markerandupload-id-markerpagination. - Self-describing upload metadata:
NewMultipartUploadstores the bucket and object name inside each upload’sxl.metaso on-disk uploads can be mapped back to(Bucket, Object)despite the<sha256(bucket/object)>directory layout.
Why Unification Was Attempted
Before #198, callers saw very different results depending on whether prefix was empty, a partial key prefix, or a full object key:
| Request Example | Traditional MinIO / legacy Behavior |
S3 Standard / Disk-Scan Behavior |
|---|---|---|
GET /photos?uploads (immediately after CreateMultipartUpload) |
Returns uploads present in the local node’s in-memory cache | Returns active uploads present at read quorum on disk |
GET /photos?uploads (after server restart) |
Returns empty list (multipartUploadsCache was cleared on restart) |
Still returns active unfinished uploads from .minio.sys/multipart |
GET /photos?uploads&prefix=2026/ (directory prefix) |
Returns empty list unless an object is literally named 2026/ (sha256("photos/2026/")) |
Returns all active uploads whose key starts with 2026/ |
GET /photos?uploads&prefix=2026/trip.mov (exact object key) |
Looks up .minio.sys/multipart/<sha256("photos/2026/trip.mov")> on disk (survives restart) |
Returns uploads whose key starts with 2026/trip.mov |
Issues #198 (2026-09-16) and #213 initially kept both behaviors behind MINIO_API_MULTIPART_LISTING=legacy|strict so upgrades with pre-existing legacy uploads would not break silently. PR #243 (2026-10-17) sought to eliminate the two-mode configuration surface after adding lazy backfill for older uploads.
Alternatives Evaluated During #243
| Candidate | Restart Survival | Prefix & Marker Support | Extra Index Maintenance | Why Chosen / Why It Later Failed |
|---|---|---|---|---|
1. Keep legacy + strict dual mode |
Only in strict |
Only in strict |
None | Rejected in #243 to remove mode split; revisited in 4733e1423 |
| 2. In-memory cache only | No | Yes (while cached) | None | Rejected: loses all visibility on restart |
3. Scan .minio.sys/multipart + xl.meta identity (#243) |
Yes | Yes | None | Chosen in #243; rolled back in 4733e1423 because O($N \times (H+U)$) walk hits 100k cap at 3,126 uploads on 16 drives |
| 4. Sidecar JSON file per upload | Yes | Yes | Extra file create/delete per upload | Rejected: doubles metadata file operations on create/abort |
| 5. Per-bucket on-disk multipart index tree | Yes | Yes (O(matching prefix)) | Requires distributed index heal & cleanup | Deferred: only viable long-term path for large erasure sets, but requires its own crash/heal protocol |
| 6. External KV / SQLite index | Yes | Yes | External state outside erasure coding | Rejected: violates self-contained erasure-set architecture |
On-Disk Storage Layout and xl.meta Identity Fields
An active multipart upload is stored at:
Because <sha256(bucket/object)> is a one-way digest, the directory path alone does not reveal the bucket or object name. Commit 4cbb074cc extended newUploadedObjectMetadata in cmd/erasure-multipart.go to store two internal metadata keys in xl.meta (retained in 20261018):
x-minio-internal-multipart-v1-bucket: plain bucket namex-minio-internal-multipart-v1-object: plain object key
Compatibility check against upstream MessagePack xl.meta readers (verified against commit 3be10fcc1a44f6620ded0bd303461f9d688cca23): MetaSys is a standard map[string][]byte inside xl.meta, so older readers ignore unknown x-minio-internal-* keys without schema errors.
Discovery and Quorum Rules in the Disk-Scan Path
When the disk-scan path (listMultipartUploadsScan) ran in #198 and #243, each erasure set executed five steps:
- List
<sha256(bucket/object)>directories under.minio.sys/multipartacross all online drives, requiring each hash directory to appear on at leastreadQuorum = floor(N/2) + 1drives. - List
<uploadID>subdirectories inside each quorum-present hash directory, validating everyuploadIDwithcheckUploadIDand requiringreadQuorumpresence. - Read
xl.metafrom online drives for each candidateuploadIDand require at leastreadQuorumdrives to agree on(Bucket, Object). - If an older pre-upgrade upload lacked
x-minio-internal-multipart-v1-*keys, resolve(Bucket, Object)when an exact prefix or cache entry supplied the candidate key and lazily backfillxl.meta. - Filter by
Bucket,Prefix,Delimiter,KeyMarker, andUploadIDMarker, sort by(Key, Initiated, UploadID), and merge across pools.
Directory Traversal Cost and 4-Drive Benchmark (#243)
On an N-drive erasure set with H active <sha256(bucket/object)> directories and U active <uploadID> directories across all buckets:
- Total directory entries inspected per uncached scan: $N \times (H + U)$
- Total
ListDiroperations per uncached scan: $N \times (1 + H)$
In #243, BenchmarkMultipartListingCapacity was run only on a 4-drive single-set fixture (N = 4), where 10,000 uploads for a single key ($H = 1, U = 10{,}000$) produced $4 \times (1 + 10{,}000) = 40{,}004$ directory entries:
4-Drive Fixture (BenchmarkMultipartListingCapacity on 4ed86dfe6) |
Active Uploads | Wall Time | Memory Allocated | Allocations |
|---|---|---|---|---|
| Single object key ($H = 1, U = 10{,}000, N = 4$) | 10,000 |
101.6 ms |
3.58 MB |
61,210 |
That single-key, 4-drive benchmark masked the two multiplicative factors that broke on 16-drive sets the following day: $N = 16$ drives ($4\times$ multiplier) and distinct object keys $H = U$ ($2\times$ entry multiplier and $U\times$ ListDir multiplier), which together pushed 3,126 uploads over maxMultipartWalkEntries = 100000.
Abort-vs-Delayed-Create Race on 16-Drive EC:8 Sets
During the #243 review, we also analyzed whether AbortMultipartUpload could leave behind a resurrected upload under a fixed readQuorum = floor(N/2) + 1 (for example, 9 on a 16-drive EC:8 set):
- Suppose
NewMultipartUploadwritesxl.metaon8drives immediately while7drives are stalled in flight and1drive is offline. - A concurrent
AbortMultipartUploaddeletes the8existing copies on the fast drives and returns. - The
7delayedNewMultipartUploadwrites then land on disk, and the1offline drive later heals or comes back with its copy—leaving8readable copies (readQuorumforEC:8object reads) of an upload that the client already aborted.
While AbortMultipartUpload cache eviction and peer notification (retained in 4733e1423) keep the in-memory cache consistent across nodes, eliminating delayed-create resurrection on disk would require tombstones or per-upload generation locks.
Summary of October 17 Code Changes (Later Partially Reverted)
cmd/object-api-interface.go,cmd/server-main.go,cmd/config-current.go: madeapi multipart_listinga compatibility no-op (4ed86dfe6; retained as no-op in4733e1423).cmd/erasure-multipart.go: added lazyxl.metaidentity backfill during exact-key resolution and routed all listings through the disk scanner (4ed86dfe6; public scan routing reverted in4733e1423,xl.metawrite metadata and0f9199e0cempty-parent cleanup retained).
October 17 Test Evidence
go test ./cmd -run 'TestMultipartUploadListing|TestMultipartPreflight|TestMultipartListingConfig'passed on4ed86dfe6.BenchmarkMultipartListingCapacitypassed at101.6 mson a 4-drive single-key fixture, which proved insufficient to catch the 16-drive distinct-key limit.
Follow-Up Lessons from #243
Any future attempt to provide restart-durable arbitrary-prefix ListMultipartUploads must index active uploads by (bucket, object, uploadID) rather than scanning .minio.sys/multipart/<sha256(bucket/object)> directories across all drives in the erasure set, and must benchmark capacity on 16-drive sets with distinct object keys ($H = U$).
Historical Record: September 16 Opt-In strict Mode (#198)
Issue #198 (commit 4cbb074cc, hardened in 143f6970d and 0ff5b37fc) delivered the initial opt-in implementation for Server 20260916:
- Added
MINIO_API_MULTIPART_LISTING=legacy|strict(defaultlegacy) andmcli admin config set/reset ALIAS api multipart_listing. - Added
GET /minio/admin/v3/multipart-preflightso operators could check whether pre-existing.minio.sys/multipartdirectories lackedx-minio-internal-multipart-v1-bucketandx-minio-internal-multipart-v1-objectbefore enablingstrict. - Enforced S3 ordering by
(Key, Initiated, encoded UploadID)andfloor(N/2) + 1drive discovery quorum instrictmode. - Observed during September 16 Docker Desktop testing:
10,000active uploads across4containers took18.7 s(p95 = 2.68 sat concurrency10) due to per-drivexl.metareads over virtualized container filesystems.
Original Root-Cause Analysis (2026-08-30)
Overview
In upstream MinIO’s erasure-coded storage engine (erasureObjects), CreateMultipartUpload stores each upload on disk under .minio.sys/multipart/<sha256(bucket/object)>/<uploadID>, while also recording the upload in an in-memory per-bucket cache (multipartUploadsCache). Because the on-disk directory name hashes bucket/object with SHA-256, the storage layer can jump directly to a known object’s multipart directory when given the exact bucket and object, but cannot list uploads by bucket or by key prefix from disk without opening every active upload directory.
Original August 30 Recommendation
The August 30 investigation recommended fixing ListMultipartUploads in two stages: first writing (bucket, object) identity metadata into xl.meta at NewMultipartUpload time and exposing an opt-in disk-backed mode (#198), and later evaluating whether the disk scan or a dedicated per-bucket index should become the default.
Evidence and Sources
AWS S3 Specification
The AWS S3 ListMultipartUploads specification requires filtering by prefix and delimiter, lexicographical ordering by Key and Initiated time (with UploadId breaking ties), and continuation via key-marker and upload-id-marker.
Repository Issue #198
Issue #198 reported that ListMultipartUploads with a directory prefix (prefix=dir/) returned an empty list even immediately after CreateMultipartUpload, and that unprefixed ListMultipartUploads returned an empty list after a server restart even though UploadPart and CompleteMultipartUpload using the saved UploadId still succeeded.
Upstream MinIO History
- MinIO PR
#5248: Replaced the earlier bucket-mirrored multipart tree with.minio.sys/multipart/<sha256(bucket/object)>/<uploadID>to avoid deep path-length limits and empty parent-directory cleanup bugs on POSIX filesystems. - MinIO PR
#20407: Added the in-memorymultipartUploadsCacheso empty-prefixListMultipartUploadscould return recently created uploads without walking disk, while routing non-emptyprefixdirectly tolistMultipartUploads(bucket, prefix, ...)as an exact object key lookup. - MinIO Issue
#20989: Documented the resulting upstream limitation indocs/minio-limits.md:ListMultipartUploadsis only supported with an empty prefix (from memory) or withprefixset to the full object key.
Code Inspection in pgsty/silo
cmd/bucket-handlers.go(ListMultipartUploadsHandler): extractedprefix,keyMarker,uploadIDMarker,delimiter, andmaxUploads, validatedencoding-type, and passed them toobjectAPI.ListMultipartUploads.cmd/erasure-sets.go(erasureSets.ListMultipartUploads): branched onprefix == ""—readingmultipartUploadsCache.List(bucket)when empty, and hashingpathJoin(bucket, prefix)to callset.listMultipartUploadson a single disk hash directory when non-empty.cmd/erasure-multipart.go(erasureObjects.listMultipartUploads): listed<uploadID>directories under.minio.sys/multipart/<sha256(bucket/object)>, ignoringdelimiter,keyMarker, anduploadIDMarker.cmd/erasure-server-pool.go(erasureServerPools.ListMultipartUploads): iterated across pools and returned early on the first pool that produced a non-empty result whenprefix != "".
Design Review Findings
The initial code review confirmed that the defect was architectural rather than a simple parser typo: the HTTP handler parsed every S3 query parameter, while the storage layer lacked an index mapping bucket + key-prefix to <sha256(bucket/object)>.
Behavior Before #198 (and Restored Default in 20261018)
Empty prefix Path
When prefix == "":
erasureSets.ListMultipartUploadsreads the in-memorymultipartUploadsCacheforbucket.- Uploads created before the last process restart are absent from the cache, even though their
.minio.sys/multipart/<sha256(bucket/object)>/<uploadID>directories remain intact on disk. - Cache synchronization across distributed nodes is best-effort (
updateMultipartUploadInfo).
Non-Empty prefix Path
When prefix != "":
erasureSets.ListMultipartUploadstreatsprefixas a full object name (object = prefix), hashesbucket/objectwith SHA-256 to locate the target erasure set, and reads.minio.sys/multipart/<sha256(bucket/object)>on disk.- If a caller passes a true key prefix such as
prefix=photos/2026/, the server hashes the literal string"bucket/photos/2026/", finds no upload for an object with that exact name, and returns zero uploads. - If a caller passes the exact object key
prefix=photos/2026/video.mp4, the server hashes"bucket/photos/2026/video.mp4", reads the matching<uploadID>directories from disk, and returns all active uploads for that object even after a server restart.
Multi-Pool Early Return
In erasureServerPools.ListMultipartUploads, the non-empty prefix path makes sense for an exact object key because sha256(bucket/object) maps deterministically to one erasure set in one pool, whereas true prefix listing across multiple pools would have needed to merge results from every pool.
Why Legacy Uploads Cannot Be Backfilled Automatically Without Candidate Keys
Because .minio.sys/multipart/<sha256(bucket/object)>/<uploadID>/xl.meta historically stored neither the bucket name nor the object key, an existing pre-20260916 upload directory on disk contains only a SHA-256 digest of bucket/object. Without either knowing the candidate bucket/object string (to verify its SHA-256 digest) or having x-minio-internal-multipart-v1-bucket and x-minio-internal-multipart-v1-object in xl.meta, the server cannot invert sha256(bucket/object) back to (Bucket, Object).
Structural Complexity
| Layer | What It Assumes | Consequence |
|---|---|---|
S3 HTTP API (ListMultipartUploadsHandler) |
prefix is a key prefix; delimiter groups common prefixes; markers paginate sorted keys |
Expects ordered (Key, Initiated, UploadID) candidates per bucket |
On-Disk Layout (.minio.sys/multipart/<sha256(bucket/object)>) |
Callers always supply the exact bucket and object (PutObjectPart, Complete, Abort) |
O(1) lookup by exact key, but no bucket partition and no lexicographical key order |
In-Memory Cache (multipartUploadsCache) |
Tracks uploads observed since process start | Fast O(bucket) listing for empty prefix, but ephemeral across restarts |
Operational Impact
Wire Compatibility Impact
SDKs and backup tools that call ListMultipartUploads with a directory prefix (such as aws s3api list-multipart-uploads --bucket B --prefix dir/) receive an empty list unless prefix equals the full object key.
Storage and Orphan Cleanup Impact
Tools that rely on ListMultipartUploads after a restart or with a directory prefix to discover and abort unfinished uploads cannot see those uploads via listing; however, Silo’s built-in background stale-upload sweep (cleanupStaleUploads) walks .minio.sys/multipart directly by modification time and cleans up expired uploads regardless of ListMultipartUploads behavior.
Operational Trade-Off
As proven on 2026-10-18, forcing ListMultipartUploads to walk .minio.sys/multipart on every uncached public request turns a listing limitation into a 503 SlowDown availability cliff at 3,126 active uploads on 16-drive erasure sets. Keeping the MinIO-compatible path protects cluster-wide ListMultipartUploads availability while preserving exact-key disk lookup and background expiry cleanup.
Severity Classification
- Data integrity risk: None (
PutObjectPart,CompleteMultipartUpload, andAbortMultipartUploadaddress uploads by exactBucket + Object + UploadID). - S3 API compatibility gap: Documented difference in S3 API Compatibility.
Design Options Compared
Option 0: Keep Traditional Behavior Without Documentation
Leaves callers surprised when prefix=dir/ returns empty results. Rejected in favor of explicit documentation and retained xl.meta identity metadata.
Option 1: Preserve MinIO-Compatible Behavior and Document the Contract (20261018 Choice)
Keeps empty-prefix listing backed by the in-memory cache and non-empty prefix backed by exact-key disk lookup, while keeping xl.meta self-describing and fixing empty hash-parent cleanup (0f9199e0c). Selected in 20261018 after the 16-drive scan regression.
Option 2: Scan .minio.sys/multipart on Disk with xl.meta Identity
Implemented as opt-in strict mode in #198 and briefly made default in #243. Works on small upload counts, but scales as $O(N \times (H + U))$ across all buckets in an erasure set and hits the 100,000-entry traversal cap at 3,126 distinct-key uploads on a 16-drive set.
Option 3: Dedicated Per-Bucket Multipart Index
Maintains a bucket-scoped, prefix-ordered metadata index alongside .minio.sys/multipart. Avoids cross-bucket SHA-256 directory scans, at the cost of maintaining and healing a second distributed metadata structure on every CreateMultipartUpload, CompleteMultipartUpload, AbortMultipartUpload, and expiry sweep.
Option 4: Expand In-Memory Cache to Support Prefix/Delimiter Only
Applies full S3 prefix, delimiter, and marker filtering to multipartUploadsCache in memory without disk scanning. Fast and bounded, but still loses visibility of pre-restart uploads until an exact-key operation touches them.
Reference Architecture for Disk-Backed Listing (Implemented in #198, Rolled Back on Public Path in 4733e1423)
1. Contract Definition
Define exact ordering (Key, Initiated, UploadID) and marker continuation rules before touching storage routines.
2. Write Path (xl.meta Identity)
Write x-minio-internal-multipart-v1-bucket and x-minio-internal-multipart-v1-object into xl.meta in NewMultipartUpload (retained in 20261018).
3. Read Path
Require readQuorum agreement across drives when reading multipart metadata from disk.
4. Semantic Filtering Layer
Apply Prefix, Delimiter, KeyMarker, UploadIDMarker, and MaxUploads above the per-set discovery layer.
5. Exact-Key Helper
Keep a direct O(1) exact-key lookup helper (sha256(bucket/object)) for internal callers and exact-key queries.
6. Cache Synchronization
Evict multipartUploadsCache entries on AbortMultipartUpload and CompleteMultipartUpload across peers (retained in 20261018).
7. Migration and Preflight
Provide /minio/admin/v3/multipart-preflight for inspecting on-disk multipart directories (retained as a compatibility endpoint in 20261018).
8. Degraded and Suspended Pools
Skip suspended pools cleanly during multi-pool listing rather than failing the entire request.
Spike Checklist Used for #198
- Verify
MetaSysMessagePack compatibility with existingxl.metareaders. - Measure
ListDirandreadAllamplification acrossNdrives. - Verify
cleanupStaleUploadsremoves empty<sha256(bucket/object)>parent directories (fixed in0f9199e0c).
Verification Gates
Unit Tests
Validate prefix, delimiter, marker pagination, and checkUploadID path-traversal rejection.
Handler and End-to-End Tests
Verify ListMultipartUploadsHandler response XML, URL encoding, and cache vs exact-key behavior across restarts.
Failure and Capacity Benchmarks
Run BenchmarkMultipartPublicListingCapacity across both 4-drive and 16-drive erasure-set topologies with up to 10,000 active uploads.
Delivery Gates
Document the public ListMultipartUploads contract in S3 API Compatibility and keep all historical anchors intact.
Current Engineering Conclusion
Keep the MinIO-compatible public ListMultipartUploads path (4733e1423) and empty-parent directory cleanup (0f9199e0c) as the production default in Server 20261018. Do not re-enable unindexed .minio.sys/multipart disk scans on the public listing path; if full restart-durable prefix listing is required in a future release, design a bucket-scoped multipart index benchmarked on 16-drive erasure sets first.
4.30 - Why CompleteMultipartUpload Must Return ChecksumType: Review of PR #57
PR #57, contributed by Shooks (@Dansyuqri), resolved #47 and shipped in Server 20260903. It restores the ChecksumType element in CompleteMultipartUpload XML responses whenever an object stores an additional S3 checksum.
The Defect and Its Scope
Investigation of #31 (reported by @cbornet) separated two distinct defects. Multipart CRC32 completion calculation was repaired by commits c8590413f and 3e14733f1. After a multipart upload completed, the stored object retained its checksum type and HeadObject or GetObject returned it, but the CompleteMultipartUpload XML response omitted ChecksumType. S3 SDK callers therefore received a valid checksum digest alongside a missing ChecksumType.
Although this omission did not corrupt stored object data, it prevented callers from distinguishing a FULL_OBJECT checksum from a COMPOSITE checksum using the completion response alone. Because both modes use the same Base64 digest length for a given algorithm, the digest string by itself does not indicate which calculation a verifier should reproduce.
Response Contract
| Stored Checksum State | CompleteMultipartUpload Response |
|---|---|
| Full-object checksum | ChecksumType=FULL_OBJECT with the algorithm digest element |
| Composite multipart checksum | ChecksumType=COMPOSITE with the algorithm digest element |
| No additional checksum | No ChecksumType element and no synthesized checksum |
The S3 CompleteMultipartUpload specification defines FULL_OBJECT and COMPOSITE as the two checksum types. The object ETag remains a separate response field and is not a substitute for additional S3 checksum metadata.
Implementation and Evidence
The change adds ChecksumType string with struct tag xml:"ChecksumType,omitempty" to CompleteMultipartUploadResponse and assigns cs[xhttp.AmzChecksumType] after decoding the stored checksum metadata. The response builder reuses the exact stored state already exposed by HeadObject and GetObject; it does not recalculate object bytes or infer a checksum type from the part count.
The merged commit (a96116b12) includes XML response tests covering FULL_OBJECT, COMPOSITE, and checksum-free multipart completions.
Compatibility and Adjacent Work
XML parsers that ignore unknown elements remain compatible, and SDKs that inspect ChecksumType (such as silo-go at bc3b35f97) receive the stored value immediately on completion. No new checksum algorithm, on-disk metadata format, or object migration was introduced.
Two adjacent multipart checksum repairs also shipped in Server 20260903: UploadPart checksum inference and CompleteMultipartUpload checksum error mapping, where CRC64NVME + COMPOSITE is explicitly rejected rather than silently canonicalized.
4.31 - When the Total Is Unknown: Folder Download Progress
SILO Console v2.1.1 displayed NaN% in the Downloads / Uploads drawer while downloading a folder. Although the streamed ZIP archive continued transferring normally and stored objects were intact, the progress bar entered an invalid determinate state—causing operators to assume the transfer had stalled or failed and to click Download repeatedly.
The repair shipped in Console 2.2.0 (16960f7ab, tracking #62) and was embedded in Server 20260903 (edc8be6ed). It enforces a single rule:
A download may enter determinate mode only when it has a finite, positive total measured in bytes applicable to that response. Without such a total, it remains indeterminate until completion, failure, or cancellation.
(Console 2.4.1 later evolved folder and multi-selection downloads to use file-stream writers and native browser download handoff; see Subsequent Evolution below.)
Observed Failure
In SILO Console v2.1.1:
- Upload several objects under a common prefix such as
folder/. - In the parent bucket listing, select
folder/and click Download. - Open Downloads / Uploads while the transfer is in flight.
- The active row displays
NaN%while the ZIP stream continues downloading.
Root Cause
The visible NaN% resulted from a three-layer mismatch between S3 prefix metadata, streamed ZIP responses, and XHR progress arithmetic.
A Prefix Has No Object Size
S3 folders are logical common prefixes rather than stored directory objects. In the bucket listing model, a prefix ends with / and carries logical size = 0 (which Console renders as - in the table). Because the generated API schema marks size as omitempty, zero-valued sizes are omitted from listing JSON. The single-selection download thunk passed object.size directly into the download helper, supplying undefined (or 0 for synthetic prefix records) as fileSize. Neither is a valid denominator.
A Streamed ZIP Has No Known Wire Length
When the server receives a download request for a path ending in /, it lists objects recursively under that prefix and connects a zip.Writer directly to an io.Pipe. Objects are read, compressed with Deflate, and flushed to the HTTP response stream on the fly.
Streaming the ZIP avoids buffering the entire archive in server RAM or temporary disk space and delivers the first response byte immediately. However, because the final compressed archive size does not exist when HTTP response headers are written, the server sends Content-Type: application/zip without a Content-Length header. Summing uncompressed source object sizes cannot substitute for Content-Length because ProgressEvent.loaded measures compressed wire bytes plus ZIP framing headers—two different units.
Progress Events Without a Valid Denominator
The XHR progress listener unconditionally evaluated:
For a folder prefix or zero-byte object, fileSize was undefined or 0, producing NaN (loaded / undefined or 0 / 0) or Infinity (loaded / 0):
Product Contract
The progress UI requires four invariants:
Formulating the contract in terms of denominator validity rather than objectPath.endsWith("/") covers folder prefixes, zero-byte objects, omitted size metadata, and any future chunked response without special-casing object types.
Goals and Non-Goals
Goals
- Never render
NaN%,Infinity%, or a fabricated percentage during folder downloads. - Keep unknown-length transfers in the existing indeterminate progress animation.
- Preserve determinate percentage progress for ordinary non-empty files.
- Ensure completion, failure, and cancellation all exit indeterminate mode.
- Ensure zero-byte files complete cleanly at
100%without intermediateNaN%. - Prevent non-finite or out-of-range numbers from entering Redux state.
Non-Goals
- Buffering complete ZIP archives on the server just to compute
Content-Length. - Using the sum of uncompressed object sizes as a denominator for compressed ZIP wire bytes.
- Redesigning the Redux store schema in this bugfix.
Technical Decision
The repair comprises five focused rules:
D1. Calculate Only from a Valid Total
Extract progress arithmetic into a pure helper (downloadProgress.ts) with no DOM or Redux dependencies:
- A finite positive
objectSizepreserves existing ordinary-file progress calculation. - When
objectSizeis unavailable (0orundefined), a finite positiveevent.totalwithevent.lengthComputable === trueserves as fallback. - Otherwise, the function returns
null, signaling that no truthful percentage exists.
D2. Keep Unknown Totals Indeterminate
Update the XHR progress listener to dispatch progressCallback only when calculateDownloadPercent returns a non-null number:
Because download items initialize with waitingForFile = true—which ObjectHandled already renders as <ProgressBar variant="indeterminate" />—skipping progressCallback when percent === null keeps the row in indeterminate mode until completion, failure, or cancellation.
D3. Make Cancellation Terminal
Completion and failure already cleared waitingForFile, but cancelObjectInList did not. Setting item.waitingForFile = false on cancellation prevents an aborted indeterminate folder download from continuing to animate as if it were still active.
At the XHR boundary, calling req.abort() fires readystatechange (DONE, status = 0) before firing onabort. Guarding the generic DONE branch so status === 0 defers to onabort / onerror ensures user cancellation is recorded as Cancelled rather than Failed.
D4. Normalize Omitted Zero-Byte Sizes
Passing object.size || 0 in the single-selection download thunk restores the omitted logical zero before the response handler checks Blob.size === fileSize, allowing zero-byte objects to transition cleanly to 100% on HTTP 200 completion.
D5. Keep Server ZIP Streaming Unchanged
The Go folder-download handler continues streaming Deflate-compressed ZIP archives over io.Pipe without buffering on disk or in memory.
Progress State Machine
| State | waitingForFile |
percentage |
Terminal Flags | Rendered UI |
|---|---|---|---|---|
| Queued / awaiting first progress | true |
0 |
None | Indeterminate bar |
| Active unknown-total transfer | true |
0 |
None | Indeterminate bar |
| Active known-total transfer | false |
0..100 |
None | Determinate percentage |
| Completed | false |
100 |
done = true |
Completed (100%) |
| Failed | false |
Last valid | failed = true, done = true |
Error state |
| Cancelled | false |
0 |
cancelled = true, done = true |
Cancelled state |
Why This Design Is Sufficient
Ordinary Non-Empty File
objectSize > 0 provides a finite positive denominator, calculateDownloadPercent returns a clamped integer in [0, 100], updateProgress sets waitingForFile = false, and completion sets 100%.
Streamed Folder ZIP
objectSize is 0, lengthComputable is false, and event.total is 0. calculateDownloadPercent returns null, no progress action is dispatched, and the row stays indeterminate until the stream finishes and sets waitingForFile = false, percentage = 100, and done = true.
Future Response with Known Length
If a proxy or future archive endpoint supplies Content-Length (lengthComputable = true and event.total > 0), the fallback branch automatically renders determinate percentages without further code changes.
Zero-Byte File
The omitted size normalizes to 0. Any intermediate progress event returns null rather than evaluating 0 / 0, and upon receiving the empty 200 OK body, Blob.size === 0 matches fileSize === 0 and transitions directly to 100%.
Failure and Cancellation
Both failure and cancellation set waitingForFile = false and done = true, guaranteeing that no terminated transfer continues showing an active indeterminate animation.
Rejected Alternatives
Buffer the Entire ZIP on the Server
Generating the full ZIP in server memory or temporary disk storage just to populate Content-Length would delay time-to-first-byte, increase server memory/disk pressure, and degrade concurrent downloads.
Sum Uncompressed Object Sizes Under the Prefix
Summing logical source object sizes measures uncompressed input bytes, whereas event.loaded measures Deflate-compressed output bytes plus ZIP headers. Because the numerator and denominator have different units, the progress bar would stall below 100% or jump erratically with compression ratio.
Coerce Invalid Progress to 0%
Displaying a determinate 0% bar throughout a multi-gigabyte folder download misleads users into thinking zero bytes have transferred and the connection is dead.
Special-Case Paths Ending in /
Checking objectPath.endsWith("/") would miss zero-byte files, missing metadata fields, and any other chunked response lacking Content-Length. Checking denominator validity handles all of them uniformly.
Route Folders Through Immediate BrowserDownload
In Console 2.2.0, the legacy BrowserDownload helper clicked a hidden <a> tag and immediately marked the download 100% complete before the first byte arrived, sacrificing error detection and cancellation. (A proper native browser handoff was later designed in Console 2.4.1.)
Sanitize NaN Inside ProgressBar Only
Masking NaN only inside the React view component would leave corrupted NaN values in the Redux store and fail to keep waitingForFile in indeterminate mode.
Widen Redux State to percentage: number | null Immediately
Adding null alongside existing waitingForFile, done, failed, and cancelled booleans would multiply state combinations unnecessarily when waitingForFile = true already models and renders indeterminate progress.
Requirements and Acceptance
Functional Requirements
- FR1–FR3: Return only
nullor an integer in[0, 100]fromcalculateDownloadPercent, keep unknown-total folder transfers in indeterminate mode (waitingForFile = true), and preserve determinate percentages for known-total object downloads. - FR4–FR5: Normalize omitted zero-byte sizes so empty objects complete cleanly at
100%, and clearwaitingForFile = falseacross completion, failure, and cancellation.
Non-Functional Requirements
- Keep server-side ZIP streaming (
io.Pipewithout full-archive buffering), S3/Console API routes, and Redux state types unchanged.
Acceptance Criteria
- Zero
NaN%orInfinity%labels appear in the download drawer across folder, ordinary-file, zero-byte, and cancelled transfers.
Verification Plan
Pure Calculation Matrix
| Case | loaded |
objectSize |
lengthComputable |
event.total |
Expected |
|---|---|---|---|---|---|
| Ordinary file, halfway | 50 |
100 |
false |
0 |
50 |
Common prefix (folder/) |
1024 |
0 |
false |
0 |
null |
Initial 0 / 0 |
0 |
0 |
false |
0 |
null |
| Response-total fallback | 50 |
0 |
true |
200 |
25 |
| Zero response total | 0 |
0 |
true |
0 |
null |
| Loaded exceeds total | 150 |
100 |
true |
100 |
100 |
Invalid NaN object size |
10 |
NaN |
false |
0 |
null |
Omitted undefined size |
10 |
undefined |
false |
0 |
null |
Non-finite Infinity total |
10 |
0 |
true |
Infinity |
null |
Negative loaded |
-1 |
100 |
true |
100 |
null |
Reducer State Transitions
Unit tests verify that new downloads start with waitingForFile = true, remain indeterminate when no numeric progress action is dispatched, transition to determinate on valid progress, and clear waitingForFile = false across completion, failure, and cancellation.
End-to-End Browser Regression
Playwright tests against a live SILO + Console instance use Chromium CDP network throttling to observe an active folder download, verify that the drawer shows an indeterminate bar with zero NaN% or Infinity% text, test mid-stream cancellation, and verify full ZIP completion alongside ordinary and zero-byte files.
Implementation Scope
- Add
downloadProgress.tswithcalculateDownloadPercent. - Update
Objects/utils.tsto dispatchprogressCallbackonly for non-null percentages and letstatus === 0XHR completions reachonabort/onerror. - Normalize
object.size || 0in the single-selection download thunk. - Set
item.waitingForFile = falseincancelObjectInList. - Add unit, reducer, and throttled Chromium Playwright regression tests.
Delivery and Rollback
Shipped in Console v2.2.0 (silo-console#29) and embedded in SILO Server 20260903, resolving pgsty/silo#62. No storage or API migration is required.
Definition of Done
-
calculateDownloadPercentreturns onlynullor a finite integer in[0, 100]. - Active unknown-total folder downloads render indeterminate progress; ordinary and zero-byte files render valid determinate completion.
- Completed, failed, and cancelled rows exit indeterminate mode (
waitingForFile = false). - Shipped in Console
2.2.0and Server20260903.
Follow-Up Work
Five follow-up items identified during this review—streaming downloads directly to the browser without in-memory Blob buffering, discriminated progress state, mid-stream ZIP error signaling, component-level non-finite guards, and Blob JSON error decoding—were tracked separately and addressed in subsequent Console releases.
Subsequent Evolution: Browser Streaming Handoff in Console 2.4.1
Alongside this progress repair, Console 2.2.0 also made size an always-present JSON field in object listings and hardened server-side ZIP error propagation so unreadable entries abort the stream rather than being silently omitted.
Subsequently, Console 2.4.1 eliminated in-memory XHR Blob buffering for downloads by handing single-folder and single-object downloads directly to the browser’s native download manager and streaming multi-selection archives through file-system writers where supported. Under that newer architecture, the Console transfer row records the handoff to the browser, while ongoing byte progress and cancellation are managed natively by the browser’s download manager.
4.32 - A Listing Must Not Drop a Null Version That Still Has Quorum
This record covers the listing omission investigated on 2026-09-16: an unversioned bucket under concurrent overwrites returned HTTP 200 listings that lacked keys the same endpoint could still GET. It describes the confirmed mechanism, the targeted repair in commit 8d06424b1 (shipped in Server 20260916), what that repair proves, and the rolling-restart counterexample tracked in #218 that was subsequently closed by PR #239 in Server 20261018 (see the LIST recovery design).
The defect
Listing merges the version streams of the drives that were queried. When drives disagree, mergeXLV2Versions picks a top version and counts the streams that agree with it. A 2022 upstream change (PR #14125) made that count tolerant of signature differences left by healing, which is necessary, but the way it selects and counts is sensitive to input order. With one ordinary null version per drive, a newer minority that sorts last resets the count for the older generation. The older generation that actually has quorum is then discarded, the merge returns nothing, and the resolver reports the key as absent. The request still succeeds, so the client sees a complete-looking listing with a missing key. Reads of the key succeed because the read path resolves quorum independently.
Nine drive orderings on the pre-fix code failed this way with complete signed LIST evidence, while the same content in other drive orderings listed accurately.
The repair
The recount in commit 8d06424b1 runs only where the original selection ends without quorum, and only for inputs where it can be exact: every non-empty drive stream holds a single ordinary null version, none is a free version, and all share the same erasure parameters, judged on the original inputs before any stream is pruned. In that case the versions are regrouped by header and a group is returned only if it reaches the original effective quorum. Mixed histories, explicit version IDs, delete markers, free versions, and mixed erasure layouts keep their previous behavior; the strictness, signature, tie-break, and representative-entry rules are unchanged, and shared callers of the merge see the same result for every previously successful input.
Validation of 8d06424b1:
- All 9 old-order and 7 new-order counterexamples pass, and 5,620 differential inputs keep the contracted result.
- Full signed HTTP listings, object reads, normal and race runs, and the scanner, healing, and migration consumers of the merge passed (6 top-level and 23 nested cases in normal and
-racemode). - A steady-state run of 20,000 successful
LISTcalls under concurrent overwrites on a 4-node, 16-drive cluster omitted zero keys.
Rolling-restart boundary (#218, closed by #239 in 20261018)
On the same 4-node, 16-drive cluster, a rolling restart of the four nodes under concurrent overwrites on the 20260916 build returned 27,966 successful LIST responses, of which 2,624 omitted one to four keys. The omitted keys remained readable: 8,186 of 8,192 same-endpoint GET/HEAD checks returned 200 OK with the last confirmed content. The strongest sample started 17.6 s after the last node’s health check returned and omitted a key whose last successful PUT had been confirmed 11.5 s earlier with no subsequent write. Raw XML, HTTP correlation, and body hashes were verified independently, ruling out pagination or client parsing.
That rolling-restart failure occurred before or around mergeXLV2Versions, where the 20260916 build still had exits that turned transient read or lock failures into silent omissions:
listPathRawread-lock acquisition (NSLock) timing out after 1 second under write contention and skipping or failing the path;- a resolver with fewer valid entries than quorum or no cached metadata when a peer drive disconnected mid-stream;
- the partial-resolution callback dropping an entry when individual readers failed without crossing the set-wide failure threshold.
Issue #218 tracked that rolling-restart investigation, and PR #239 in Server 20261018 resolved it by removing the 1-second listPathRaw read-lock timeout and adding bounded metacache stream recovery. See LIST Recovery Under Node Restart and NSLock Contention for the complete #239 design and benchmarks.
4.33 - A ListObjects Shortcut Must Not Turn a Missing Bucket into an Empty One
Shipped in SILO Server 20260903, PR #37 (commit e9c5340be, merged as 49c8aeac4, closing SILO #32) fixes an S3 compatibility regression where three early-return shortcuts in ListObjects, ListObjectsV2, and ListObjectVersions returned HTTP 200 OK with an empty listing instead of HTTP 404 NoSuchBucket when called against a non-existent bucket.
Too Long; Didn’t Read (TL;DR)
A normal ListObjects, ListObjectsV2, or ListObjectVersions request against a missing bucket reaches the storage layer and receives BucketNotFound. Three shortcut inputs, however, return early before touching storage:
- a
marker(orstart-after) that does not begin withprefix; max-keys=0;- a
prefixbeginning with/, including thePrefix="/"boto3reproduction from #32.
Those three branches returned io.EOF directly. Because the caller treats io.EOF as the normal end of a listing, the client received an empty HTTP 200 response instead of S3’s HTTP 404 NoSuchBucket. The same missing bucket changed from a 404 resource error to a 200 success solely because the query filter parameters differed, breaking S3 compatibility and blocking upgrades from pre-regression releases.
Restoring a bucket-existence check on every listing request would reintroduce cluster-wide RPC fan-out on the normal hot path. Instead, PR #37 replaces only the three bare io.EOF returns with a helper (listPathShortcutEOF) that calls GetBucketInfo once: it returns the underlying error when the bucket is absent or cannot be confirmed, and returns io.EOF when the bucket exists. Normal listings pay zero extra cost; only requests that would otherwise exit before touching storage pay the GetBucketInfo fan-out.
What Is the Problem?
One API Exposes Two Bucket-Existence Semantics
Issue #32 reproduces the defect with a single call against a missing bucket:
AWS S3 raises NoSuchBucket; SILO returned a successful empty listing. The divergence originates in the object-layer listPath control flow:
Prefix="/" is only one of three triggers:
| Shortcut Condition | Why the Result Must Be Empty | Defect Before the Fix |
|---|---|---|
marker does not begin with prefix |
The listing engine does not scan a disjoint range | Returns io.EOF without confirming bucket existence |
max-keys=0 |
The caller explicitly requests zero keys | Equates “zero keys requested” with “valid bucket resource” |
prefix begins with / |
SILO’s flat key space never produces keys with a leading / |
Short-circuits on the filter before checking bucket identity |
For an existing bucket, returning an empty listing from these branches is a valid fast path. For a missing bucket, returning bare io.EOF masks the resource error that must take precedence.
Tracing the Regression to Upstream #18917
The reporter confirmed that RELEASE.2024-01-29T03-56-32Z returned NoSuchBucket and that the regression started in RELEASE.2024-01-31T20-20-33Z. Tracing git history pinpoints upstream minio/minio#18917 (80ca12008), which removed GetBucketInfo from generic handler argument checks and relied on the underlying Put, List, and Multipart storage operations to surface BucketNotFound.
That assumption holds whenever the request reaches storage, but breaks on early-return branches that exit before touching any drive. Fixing #32 therefore requires closing the three control-flow exits that bypass storage, rather than reverting #18917.
Why Fix It?
The S3 Contract Requires NoSuchBucket
Both AWS ListObjects and ListObjectsV2 specify HTTP 404 NoSuchBucket when the target bucket does not exist. Selection parameters (prefix, marker, start-after, and max-keys) filter objects inside an existing bucket; they cannot make a non-existent bucket succeed. Because ListObjectVersions shares the same object-layer listing engine, applying the fix across all three APIs keeps V1, V2, and version listings consistent.
An Empty HTTP 200 Alters Client Control Flow
An empty HTTP 200 and an HTTP 404 drive opposite decisions in client applications:
- HTTP 404 tells provisioning scripts, SDKs, or test harnesses to create the bucket, report a configuration error, or abort;
- an empty HTTP 200 asserts that the bucket exists and currently holds no matching keys;
- integration test suites using SILO as a local S3 target pass against a missing bucket locally and then fail against AWS S3.
The Repair Surface Is Narrow and Verifiable
The defect is confined to three adjacent early returns in cmd/metacache-server-pool.go and does not touch object data, metadata formats, sorting, continuation-token encoding, IAM evaluation, or XML schemas.
Why We Cannot Simply Restore a Global Check
Upstream #18917 removed generic GetBucketInfo checks because calling GetBucketInfo before every Put, List, and Multipart operation fans out across all servers in the cluster; even after request vectorization, that overhead becomes pronounced past 100 nodes.
In SILO, erasureServerPools.GetBucketInfo invokes S3PeerSys.GetBucketInfo, which queries every peer concurrently and reduces quorum across pools while each peer inspects its local bucket state. Neither extreme is acceptable:
- Never check on shortcuts: preserves the bogus HTTP 200 on missing buckets.
- Check before every
List: restores404semantics at the cost of undoing the large-cluster optimization on every normal listing.
Confining the GetBucketInfo call to the three branches that bypass storage solves the compatibility bug without taxing normal ListObjects traffic.
Implementation
Replace Only the Three Bare io.EOF Returns
In cmd/metacache-server-pool.go, each of the three shortcut branches previously executed:
Each branch now calls listPathShortcutEOF:
where listPathShortcutEOF is defined as:
This helper produces three possible outcomes:
- Bucket exists: returns
io.EOF, preserving the empty listing response. - Bucket does not exist: returns
BucketNotFound, which maps to HTTP 404NoSuchBucket. - Cluster state cannot be confirmed: propagates quorum, offline, timeout, or context cancellation errors rather than fabricating an empty success.
The normal listMerged, metacache scan, sorting, pagination, and XML rendering paths remain untouched.
Why the Helper Belongs in listPath
Placing the check directly at the shortcut exits in listPath satisfies three constraints:
- Only
listPathknows that it is about to return early without consulting any storage drive. - Moving the check up into HTTP handler validation would charge every listing request.
- Moving the check down into the scanner cannot work because these three branches never enter the scanner.
Why We Do Not Add an Existence Cache
Caching bucket existence would reduce shortcut fan-out, but immediately introduces cache-invalidation hazards across bucket creation, deletion, site replication, healing, and TTL expiration. Introducing a second source of truth for three low-frequency shortcut paths adds more consistency risk than it saves. If production telemetry ever shows high-volume max-keys=0 or leading-slash probing on large clusters, rate limiting or a dedicated metadata fast path can be evaluated with real traffic data.
Verification Evidence
Object-Layer Contract
Object-layer regression tests run against both single-drive and multi-drive erasure setups across four inputs:
- leading-slash
prefix(Prefix="/"andPrefix="/sub"), max-keys=0,markeroutsideprefix,- a regular
prefixas a control case that naturally receivesBucketNotFoundfrom storage.
Every case exercises ListObjects, ListObjectsV2, and ListObjectVersions and asserts isErrBucketNotFound.
HTTP Wire Contract
The HTTP handler test sends signed S3 requests across all three public listing APIs:
| API | Request Shape | Assertion |
|---|---|---|
ListObjects |
GET /missing-bucket?prefix=/ |
HTTP 404 and XML code NoSuchBucket |
ListObjectsV2 |
GET /missing-bucket?list-type=2&prefix=/ |
HTTP 404 and XML code NoSuchBucket |
ListObjectVersions |
GET /missing-bucket?versions&prefix=/ |
HTTP 404 and XML code NoSuchBucket |
Local and CI Quality Gates
The change passed the full local verification suite (go test ./cmd -count=1 in 116.2 s, the 10 new object-layer and HTTP regression subcases under -race, CGO_ENABLED=0 go build ./..., go vet ./..., and independent design and code review). On GitHub Actions, DCO-signed PR head e9c5340be passed DCO, VulnCheck, and all six Go CI jobs, and merge commit 49c8aeac4 passed post-merge VulnCheck and Go CI.
Derived Risks and Edge Cases
Shortcut Requests Now Fan Out Across the Cluster
On an existing bucket, a shortcut request previously executed a purely local branch; it now calls GetBucketInfo. Local microbenchmarks illustrate the relative shift:
| Path | Observed Local Magnitude |
|---|---|
| Shortcut before the fix | ~0.55 μs, 7 allocs |
| Repaired single-drive shortcut | ~7.8–8.1 μs, 45–47 allocs |
| Repaired 32-drive shortcut | ~70–81 μs, 977 allocs |
| Normal 32-drive listing | ~0.95 ms |
In multi-node deployments, GetBucketInfo includes peer network round-trips and quorum reduction. That cost is why the check is strictly confined to the three storage-bypassing shortcuts. If a misconfigured client polls max-keys=0 or Prefix="/" at high frequency, S3 audit traces and metrics will surface the pattern.
Degraded Clusters Surface Real Quorum Errors
Previously, a shortcut could return an empty HTTP 200 even when peers were offline and bucket read quorum was lost, because it never inspected cluster state. Calling GetBucketInfo means a degraded cluster that cannot establish bucket existence now returns a quorum or timeout error rather than asserting that a valid empty bucket exists.
Bucket Create/Delete Races Remain Non-Linearizable
GetBucketInfo and returning the empty response are two sequential steps: a bucket can be deleted immediately after GetBucketInfo succeeds, or created immediately after BucketNotFound is observed. This matches the concurrency model of all other S3 APIs that validate resource existence before acting.
Clients Relying on the Bug Will Now Receive HTTP 404
Callers that inadvertently depended on Prefix="/" or max-keys=0 returning HTTP 200 against a non-existent bucket will now receive HTTP 404 NoSuchBucket, matching AWS S3 and pre-2024 behavior.
Two Adjacent Edge Cases Left Out of Scope
Design review noted two low-severity boundaries intentionally left outside this change:
- Stale continuation token after bucket deletion: When resuming a metacache continuation, the
c.fileNotFoundbranch still returns bareio.EOF. A stale or crafted continuation token presented after its bucket was deleted could theoretically receive an empty HTTP 200. CallingGetBucketInfoon that branch would penalize normal paginated continuations and requires a separate design. - Handler-level
NotImplementedchecks: Certain V1 and version-listmarker/prefixcombinations returnNotImplementedduring HTTP handler parameter validation before reaching the object layer (whereas V2start-afterreacheslistPath). PR #37 repairs storage shortcuts masking missing buckets without altering HTTP parameter validation precedence.
Rejected Alternatives
Keep Upstream Behavior
Keeping the bare io.EOF returns avoids any shortcut overhead, but preserves a known S3 regression and causes integration tests against SILO to diverge from AWS S3. Rejected.
Restore Generic checkBucketExist
Checking bucket existence before every Put, List, and Multipart call fixes all paths at once, but reintroduces cluster-wide peer fan-out on the normal hot path, undoing upstream #18917. Rejected.
Fix Only Prefix="/"
Patching only the leading-slash branch passes the single boto3 snippet from #32 while leaving the identical bug in max-keys=0 and disjoint marker inputs. Because all three branches sit in listPath and share the same root cause, routing all three through listPathShortcutEOF is cleaner and complete. Rejected.
Add a Bucket-Existence Cache
Adding a TTL cache for bucket existence reduces shortcut latency, but requires invalidation rules across bucket creation, deletion, replication, and healing without any evidence that shortcut traffic warrants a second metadata cache. Rejected.
Complexity and Cost-Benefit Summary
| Dimension | Assessment | Rationale |
|---|---|---|
| Production code complexity | Low | Three call sites and a 7-line helper; no new state, dependency, or on-disk format |
| Test complexity | Low–Medium | Covers V1, V2, version listings, all three shortcuts, a regular-prefix control, and HTTP 404 XML mapping |
| Normal-path risk | None | Zero changes on the listMerged hot path |
| Shortcut runtime cost | Higher on shortcuts only | Replaces a local branch with GetBucketInfo when a shortcut triggers |
| Compatibility value | High | Restores HTTP 404 NoSuchBucket, pre-regression upgrade compatibility, and S3 test fidelity |
| Operational complexity | None | No migration, configuration knob, feature flag, or cache invalidation |
Acceptance Decision
PR #37 was accepted and merged as 49c8aeac4 (retaining Jason Lin as co-author on e9c5340be) with typed error predicates and full V1/V2/version-list object-layer and HTTP-level coverage, and shipped in SILO Server 20260903.
Summary
The listing engine previously overloaded io.EOF to represent both “an existing bucket produced zero matching keys” and “an early filter exit returned before touching storage.” Removing generic bucket-existence checks in #18917 relied on the premise that a subsequent storage operation would surface BucketNotFound. Calling GetBucketInfo at the three storage-bypassing exits in listPath restores that invariant and S3 NoSuchBucket compatibility while keeping the normal listing path unchanged.
4.34 - Read-Only Checksum Audit and Reliable CLI Output
This is the design and implementation record for MCLI’s read-only checksum verification workflow and pgsty/mc#5, the non-TTY output defect discovered during release review. The command merged into pgsty/mc through pull requests #8 and #13 and shipped in mcli 20260903 (bundled in Server 20260903 container images).
Overview
Historical CopyObject implementations could calculate a stored additional checksum over transformed storage bytes (such as compressed or encrypted streams) rather than the logical bytes returned by S3 GET. mcli checksum verify inventories candidate objects and independently streams each logical body through the recorded checksum algorithm. Every candidate is classified into MATCH, MISMATCH, NO_CHECKSUM, WOULD_VERIFY (dry run), one of ten UNKNOWN_* classifications, or one of three SKIPPED_* results.
During release testing, the initial implementation worked in an interactive terminal but printed nothing when stdout was redirected to a file or pipe. MCLI automatically marks non-TTY execution as quiet (globalQuiet) to disable interactive progress bars, and the new command initially treated that internal flag as a user request to suppress audit records. The repair in PR #13 separates semantic command output from progress-bar suppression without altering global quiet behavior or enabling progress bars in CI.
Command and Scope
Version one supports CRC32, CRC32C, CRC64NVME, SHA1, and SHA256 checksums marked as FULL_OBJECT. It can select a single object, an exact VersionID, current objects under a prefix, all object versions, or exact entries from a JSON Lines manifest. It also supports SSE-C key mappings, modification-time and size filters, --dry-run cost estimation, bounded concurrency (--max-workers), global download limits, --json output, and an optional --report JSON Lines file.
The command is strictly read-only: it does not verify COMPOSITE multipart checksums, infer checksum types from an ETag, inspect raw xl.meta on disk, or mutate object metadata. The S3 endpoint must report the checksum type (x-amz-checksum-type) alongside the checksum digest; on an endpoint that omits the type header, every checksummed object is classified as UNKNOWN_CHECKSUM_TYPE rather than guessed.
--rewind, tier restore, server-side batch execution, checkpoint/resume, and repair are outside the current scope.
Read-Only Data Path
For every selected object, mcli checksum verify:
- Sends
HEADwith checksum mode enabled and retains every supportedx-amz-checksum-*digest plusChecksumType. - Classifies unsupported or ambiguous metadata states as
UNKNOWN_*instead of guessing. - Streams
GETlogical bytes through bounded in-memory hashers without writing the object body to local disk. - Uses
VersionIDpinning for immutable versions, orIf-Matchplus a post-readHEADcheck for mutable unversioned andnull-version objects to detect concurrent replacement. - Compares the independently calculated digests against the stored digests.
The S3 transport boundary permits LIST, HEAD, and GET only; automated tests fail if any write method reaches the mock endpoint.
A dedicated S3 helper retains all checksums and their type through StatObject, then reads logical bytes through Core.GetObject without the SDK automatic validator. This avoids changing shared ClientContent or the existing contracts for stat, copy, mirror, and JSON output.
Result and Exit Contract
Every candidate produces one stable classification:
| Result | Meaning |
|---|---|
MATCH |
Every supported stored checksum matches the returned logical object bytes |
MISMATCH |
At least one stored checksum differs from the returned logical bytes |
NO_CHECKSUM |
No additional checksum exists on the object; the body is not read |
WOULD_VERIFY |
--dry-run found a supported FULL_OBJECT checksum |
UNKNOWN_* |
MCLI cannot make a reliable verification statement (for example, COMPOSITE or missing ChecksumType) |
SKIPPED_* |
A filter intentionally excluded the object (SKIPPED_TOO_LARGE, time filter, or delete marker) |
The final summary carries objects, a verified count, the count of every result status, and incomplete. verified equals MATCH plus MISMATCH—the only classifications that actually streamed an object body through a hasher. A run that enumerated many objects and verified none is therefore explicitly visible.
--fail-on accepts mismatch, unknown, no-checksum, any, or none:
- The default
anyreturns exit code1for anyMISMATCH,UNKNOWN_*, or incomplete verification. In particular,SKIPPED_TOO_LARGEmakesanyreturn exit1because the size cap leaves the audit incomplete, whereas time-filter and delete-marker skips do not fail by themselves. no-checksumincludes allanyfailures and also returns1when a checksum is absent orverified == 0.--dry-rundoes not evaluate--fail-on.- CLI argument, authentication, listing, and report-write failures remain command errors rather than per-object classifications.
Output and Automation Contract
Per-object records and the final summary are semantic command output:
- Unless the caller explicitly sets
--quiet,-q, orMC_QUIET=true, stdout receives every object record and the final summary in both TTY and non-TTY execution. - Non-TTY
--jsonemits one compact JSON value per line (JSON Lines); TTY--jsonretains MCLI’s pretty-printed presentation. - Global flags (
--json,--quiet) work when placed at the rootmcli,checksum, orverifycommand level. --reportis independent of stdout: even when explicit--quietsuppresses stdout,--reportstill writes all object records and the final summary as JSON Lines.- Output transport never alters
--fail-onexit decisions.
This separation is required because MCLI’s globalQuiet conflates two inputs: an explicit --quiet flag and an automatic non-TTY check used to disable interactive progress bars. Mutating globalQuiet globally would risk re-enabling progress bar output across cp, get, put, mirror, and other commands in CI logs.
Instead, the repair in PR #13 is command-local. It walks the full CLI context chain to inspect explicit quiet and JSON flags (because the CLI library’s GlobalBool stops at the nearest ancestor flag set), and it restores JSON Lines mode inside the verify action after nested Before hooks run.
Report, Secrets, and Operational Cost
Report files (--report) use mode 0600 on POSIX systems and the account’s filesystem ACLs on Windows. They fail if the target path already exists and record only object metadata and verification results, with no object bodies or SSE-C keys. Input manifests (--manifest) likewise contain only bucket, key, and optional versionId.
Because verification streams the full logical body of every supported object, operators should use --dry-run, --max-size, modification-time filters, --max-workers, and the global download rate limit to bound network and storage I/O. NO_CHECKSUM and UNKNOWN_* counts remain explicit in the summary so unverified objects are never conflated with MATCH.
What a Mismatch Proves
A MISMATCH result establishes that the additional checksum returned by HEAD at verification time does not match the logical bytes returned by GET at verification time. By itself, a mismatch does not identify which historical writer path produced the object (for example, whether a historical compressed CopyObject computed its checksum over compressed storage bytes or whether storage drift occurred), nor does it compare the object against an external source of truth.
Operators should never overwrite checksum metadata in place on disk. Audit and classify affected objects first. For a confirmed mismatch on an active object, write a replacement object under a new key or new version, verify the replacement with mcli checksum verify, and switch downstream consumers to the verified object. Keep UNKNOWN_* objects out of automated remediation scripts.
Verification Record and Schema Contract
The acceptance matrix covers TTY human and JSON output, non-TTY pipes, regular-file redirects, root/parent/leaf --json and --quiet flags, MC_QUIET=true, --report under quiet mode, report-write failures, MISMATCH/UNKNOWN_* exit codes, and real MATCH, MISMATCH, and unsupported COMPOSITE objects against a live SILO server (mc checksum verify reference).
Automated JSON consumers should validate schemaVersion: 1 and distinguish per-object records (type: object) from the final summary (type: summary). --max-workers defaults to 4 and accepts 1–64, bounding concurrent object verification workers.
Tests also cover mismatches among multiple checksums, fixed and null versions, one result per candidate on normal completion, and clean result-stream closure after worker cancellation. InvalidObjectState is classified as a restore condition before generic HTTP 403 handling; SSE-C error classification does not depend on URL or object-name text.
4.35 - Optional Checksums, Mandatory Failure: Repairing UploadPart and UploadPartCopy Compatibility
This is the design and implementation record for SILO #46 (merged as 7fea6d5a5 and shipped in Server 20260903, with follow-up items #63 and #64). It documents how handling an optional S3 part checksum header affected multipart completion semantics, copy responses, compression and encryption pipelines, and compatibility baselines.
Too Long; Didn’t Read (TL;DR)
A multipart upload (MPU) splits a large object into smaller parts. A client may attach a checksum to each part so the server can verify the transfer, but AWS defines that per-part header as optional. SILO previously treated it as mandatory: an ordinary UploadPart failed without one, and UploadPartCopy could never succeed because it has no request body to checksum.
After the fix, SILO still validates a checksum when the client sends one. When the client omits it, SILO computes the checksum in a single pass while reading the logical plaintext bytes and persists the result in part metadata. This happens before compression and encryption, requires no second disk read, and changes no on-disk format.
Decision
When a multipart upload declares a checksum algorithm in CreateMultipartUpload, SILO enforces the following contract:
- If the client supplies a part checksum, the server validates it; a mismatched value or algorithm fails immediately and is never masked by server-side fallback computation.
- If the client omits the part checksum, the server computes it in one pass with the MPU algorithm over the logical plaintext stream, before compression and encryption, and persists the result.
- A normal
UploadPartechoes a checksum response header only when the client supplied the checksum in the request; a server-computed fallback value is not echoed. UploadPartCopyhas no client part-body checksum, so the server computes the value and returns it inCopyPartResult.ListPartsreturns the persisted part checksum.FULL_OBJECTcompletion continues to combine the full-object checksum linearly from stored part checksums.COMPOSITEcompletion continues to require a checksum for every part, which clients can retrieve viaListParts.- Computation happens during the initial read; completion never re-reads the entire object.
How We Found It
The defect surfaced while investigating a separate multipart checksum issue, #31.
#31 concerned CompleteMultipartUpload: for FULL_OBJECT, a client can complete an upload with part numbers, ETags, and an optional full-object checksum without listing every part checksum in the completion XML. Tracing that path backward exposed a stricter check in erasureObjects.PutObjectPart:
Once an MPU declared a checksum algorithm, every UploadPart had to carry a matching x-amz-checksum-* header. Omitting it returned:
API probes reproduced this error on both single-drive and erasure backends.
Inspecting CopyObjectPartHandler elevated the issue to P0. UploadPartCopy has no request body for the caller to checksum: the handler reads the source object, constructs an internal reader, and calls PutObjectPart. No client header or SDK setting could satisfy the check, so every checksum-enabled MPU rejected UploadPartCopy unconditionally.
What AWS Requires
The S3 API specification defines distinct behavior across the three operations:
- The AWS UploadPart API defines each algorithm-specific checksum header as an optional data-integrity check and specifies that the response header is returned only when the checksum was provided in the request.
- The AWS UploadPartCopy API specifies that when an MPU was created with a checksum algorithm,
CopyPartResultincludes the computed checksum for the copied part. - The AWS ListParts API returns the stored checksums for parts in an in-progress upload.
The algorithm and checksum-type support matrix is:
| Algorithm | FULL_OBJECT |
COMPOSITE |
|---|---|---|
| CRC64NVME | Supported | Unsupported |
| CRC32 / CRC32C | Supported | Supported |
| SHA1 / SHA256 | Unsupported | Supported |
FULL_OBJECT applies only to linearizable CRCs, whereas SHA1 and SHA256 require per-part digests to compute the COMPOSITEchecksum at completion.
Although recent AWS SDKs calculate request checksums by default when supported, callers can set request_checksum_calculation = when_required or use low-level APIs that declare an algorithm on CreateMultipartUpload without sending checksum headers on every UploadPart.
Why Removing the Check Is Not a Fix
Simply deleting the check in PutObjectPart would defer the failure to CompleteMultipartUpload.
SILO does not re-read all object bytes during MPU completion. Instead, it reads ObjectPartInfo.Checksums from each part.N.meta:
- a missing entry immediately fails with
InvalidPart; FULL_OBJECTcallsChecksum.AddPartto combine part digests using their part lengths;COMPOSITEconcatenates the raw part digest bytes and hashes them into the object-level digest.
The storage invariant is therefore:
Removing the upload check without populating part metadata would let UploadPart succeed, omit checksums from ListParts and UploadPartCopy, and fail during CompleteMultipartUpload.
Alternatives Considered
| Option | Benefit | Fatal problem | Decision |
|---|---|---|---|
| Delete the strict check | Smallest diff | Part metadata still lacks the checksum; completion fails with InvalidPart |
Rejected |
Relax only FULL_OBJECT |
Unblocks some default CRC clients | Leaves COMPOSITE and SHA1/SHA256 broken; cannot close #46 |
Rejected |
| Re-read every part at completion | Avoids storing a digest during upload | Adds $O(\text{object size})$ second-pass I/O and leaves ListParts and copy responses incomplete |
Rejected |
Always return the server value from normal UploadPart |
Simplifies federation forwarding | Violates the AWS UploadPart response contract |
Rejected |
| Copy the AIStor implementation directly | Commercial precedent | Supports only mergeable CRCs and attaches the hasher where streams may already be compressed or encrypted | Rejected |
| Compute and persist in one pass over logical plaintext | Full protocol compliance, no second I/O, supports both CRC and SHA | Requires an explicit plaintext checksum reader separate from the storage reader | Accepted |
What Commercial AIStor Revealed
We inspected MinIO AIStor RELEASE.2026-08-07T18-34-35Z using Go pclntab symbol recovery and ARM64 disassembly. Static analysis showed that AIStor:
- installs a server hasher when the client checksum is absent;
- persists the result in part metadata;
- includes checksum fields in
CopyPartResult.
However, AIStor enables fallback computation only for CanMerge() algorithms (CRC32, CRC32C, and CRC64NVME), leaving SHA1 and SHA256 COMPOSITE uploads to fail with checksum missing. In addition, AIStor attaches the hasher inside the object layer to r.Reader, which under server-side compression or encryption already carries transformed storage bytes rather than logical S3 plaintext.
How Design Review Overturned the First Design
Our initial design placed the fallback inside erasureObjects.PutObjectPart: inspect the MPU metadata in the object layer and attach a server hasher whenever the incoming reader lacked a client checksum.
Design review caught a critical flaw on the compression path. newS2CompressReader is not a lazy wrapper; its constructor immediately spawns a goroutine:
The S2 writer also prefetches multiple blocks concurrently. After constructing the compressor, the HTTP handler still parses options, sets up encryption, and calls into the object layer. By the time PutObjectPart attached a hasher, the compressor goroutine could already have consumed several MiB from the plaintext reader:
- a large part would produce a checksum missing its leading bytes;
- a small part could hit EOF before the hasher was attached and produce no checksum at all;
- mutating
ServerSideHasherconcurrently withReadwould trigger a data race.
This split the responsibility cleanly across layers:
- The HTTP handler attaches the hasher to the logical plaintext stream before starting any eager transformation (such as concurrent compression or encryption).
- The object layer validates the algorithm, verifies that a checksum result was produced, and persists it atomically in
part.N.meta.
Final Implementation
A Dedicated Logical Checksum Reader
PutObjReader previously carried two readers:
Reader, the stream written to storage (potentially compressed or encrypted);rawReader, used for ETag calculation.
Under compression, rawReader does not always read plaintext bytes directly; it may only forward an ETag through an etag.Tagger wrapper. To keep ETag handling untouched, we added an unexported field to PutObjReader:
checksumReader always wraps the logical S3 plaintext part stream. WithEncryption replaces the storage Reader while preserving checksumReader.
Unexported accessors on PutObjReader:
- report the effective checksum type (client-supplied or server-computed);
- return the client value when present;
- otherwise return the server-computed digest finalized at EOF.
Keeping these helpers unexported avoids expanding the public Go API and provides a shared internal mechanism for #63.
Preparing the Hasher Before Transformations
prepareMultipartChecksumReader inspects the algorithm and checksum type stored in the MPU metadata:
- if the MPU declared no checksum algorithm, it does nothing;
- if the client provided a checksum, it verifies that the base algorithm matches the MPU;
- if the algorithm mismatches, it returns
InvalidArgument; - if the client omitted the checksum, it attaches the matching server-side hasher to the plaintext
hash.Reader.
For normal UploadPart:
- the compressed path prepares
actualReaderafter parsing request checksums (actualReader.AddChecksum) and before callingnewS2CompressReader; - the uncompressed path prepares the request hash reader before wrapping it with encryption.
For UploadPartCopy:
- a checksum-enabled MPU wraps the logical source byte range in an inner
hash.Reader; - a range copy hashes only the requested byte interval;
- compression and destination encryption wrap the stream only after the inner hash reader is installed.
The Object Layer Remains Authoritative
Preparing the hasher in the handler does not relax the object-layer invariant. erasureObjects.PutObjectPart still:
- re-parses the expected MPU algorithm;
- verifies that the effective checksum type matches;
- reads the computed checksum map after erasure encoding finishes;
- logs an internal error and aborts if an enabled algorithm produced no digest;
- writes the checksum alongside the ETag, sizes, and part index into
part.N.metabefore atomically renaming the part.
Any internal caller that bypasses the HTTP handler without preparing a valid checksum is still rejected before committing the part.
CopyPart Response Shape
CopyObjectPartResponse adds the five supported checksum fields:
Each field uses omitempty, preserving the existing XML output for MPUs without checksums. Normal UploadPart continues to echo the checksum header via TransferChecksumHeader only when supplied by the client.
Why It Works
The resulting part pipeline is:
This satisfies all four constraints:
- Protocol compatibility: omitting the optional per-part checksum header succeeds for
UploadPart, andUploadPartCopyworks out of the box. - End-to-end verification when supplied: client-provided checksums are still verified and never masked by server computation.
- Plaintext semantics: checksums always cover logical S3 part bytes rather than compressed or encrypted bytes.
- Single-pass streaming: hashing shares the existing read pass without buffering the full part in memory or re-reading from disk at completion.
hash.Reader finalizes ServerSideChecksumResult upon reaching EOF. On the compressed path, closing the compression pipe synchronizes the compressor goroutine with the storage read, so the digest is guaranteed to be finalized before PutObjectPart reads it after erasure encoding returns. Targeted -race tests verify this synchronization boundary.
The Compatibility-Baseline Blocker
The five new CopyObjectPartResponse fields are exported Go identifiers. SILO’s buildscripts/rebrand-guard scans imports, environment variables, HTTP headers, routes, storage markers, and exported symbols, comparing them bidirectionally against buildscripts/rebrand-guard/compat-baseline.json. Any unrecorded addition fails CI.
After adding the five #46 fields to the baseline, the guard reported two remaining additions:
Those symbols came from an earlier database-notification fix (f1ba68358) on main, which exported LegacyDatabaseTargetError so the cmd startup path could match it via errors.As but omitted the baseline update.
Recording both the two notification symbols and the five #46 response fields produced a clean baseline diff of seven additions and zero deletions:
Verification Evidence
The regression suite runs 76 subtests covering:
- CRC32, CRC32C, and CRC64NVME
FULL_OBJECT; - CRC32, SHA1, and SHA256
COMPOSITE; - valid client checksums, mismatched algorithms, and corrupted values;
- omission of server-computed checksums from normal
UploadPartresponse headers; - presence of server-computed checksums in
UploadPartCopyresponses andListParts; - a 5 MiB + 1 KiB two-part full-object merge;
- zero-length parts and overwriting an existing part number;
- range copies verifying that SHA256 covers only the copied byte range;
- single-drive and 16-drive erasure backends;
- default, versioned, compressed, encrypted, and compressed-plus-encrypted configurations;
- explicit SSE-C and SSE-S3 encryption.
Local validation commands:
Cost, Risk, and Rolling-Upgrade Tradeoffs
When a client omits a part checksum on a checksum-enabled MPU, the server hashes the plaintext stream during upload. CRC hashing adds negligible CPU overhead; SHA1 and SHA256 use more CPU, but both run inline with the existing stream read without buffering parts in memory or adding disk I/O.
During a rolling upgrade, upgraded nodes accept checksum-less UploadPart requests while unupgraded nodes still return 400. Because the on-disk ObjectPartInfo.Checksums format is unchanged, written parts remain compatible across versions, and client-facing behavior converges as soon as all nodes are upgraded.
Why Two Follow-Ups Were Split Out
Design review identified two related issues with separate API and test boundaries:
#63: CopyObject + Compression
Ordinary CopyObject with server-side checksums and compression could also attach its hasher to the compressed stream rather than the plaintext stream. #63 reuses the checksumReader mechanism introduced here to fix CopyObject under its own test matrix.
#64: Legacy Federation
Legacy etcd federation proxies UploadPartCopy as a remote UploadPart request. Because a standard UploadPart response omits server-computed checksums, the proxy cannot read the fallback checksum from the remote response headers to populate CopyPartResult. #64 addresses the federated proxy path without altering external UploadPart response semantics.
Both follow-ups share the same streaming invariant:
Every server-computed S3 checksum binds to the logical plaintext stream, is installed before any eager transform starts, and is validated and persisted by the object layer.
Engineering Takeaways
- Client-optional headers can still back mandatory storage state. When the S3 wire protocol allows callers to omit a per-part checksum, the server must compute it during upload so
CompleteMultipartUploadalways finds complete part metadata. - Internal computation and response headers follow separate rules.
UploadPartcomputes missing part checksums for internal metadata while omitting them from response headers, whereasUploadPartCopyreturns the server-computed digest inCopyPartResult. - Attach stream hashers before concurrent transforms. Because
newS2CompressReaderstarts reading immediately in a background goroutine, any plaintext hasher must be installed before constructing the compressor. - Verify third-party behavior against the full matrix. Disassembling AIStor confirmed the server-side hashing approach, while also showing that SHA1/SHA256
COMPOSITEand compressed streams required broader handling. - Track exported API changes in the compatibility baseline. Updating
compat-baseline.jsonkeeps every intentional change to exported symbols explicit and auditable. - Share core mechanisms across independent fixes. #63 and #64 build on the same
checksumReaderdesign while keeping their own regression suites.
4.36 - BadDigest, InvalidRequest, and the CompleteMultipartUpload Checksum Contract
The repairs for #48 and #50 shipped in Server 20260903, and the streaming-checksum follow-up (PR #143) shipped in Server 20260916. This record supersedes the earlier August proposal to leave CRC64NVME canonicalization unchanged.
20261018: Message Text and Commit Failures
Server 20261018 includes the message fixes from #225, #229, and #238: S3-compatible Content-MD5 capitalization, known algorithm names for PutObject and UploadPart mismatches, and removal of redundant “checksum” wording at completion. Unknown algorithms retain the generic description, and operation-specific error codes and validation requirements remain unchanged. The unchanged UploadPart mapper described below refers to the August repair scope; October updates its message rendering. Clients should match HTTP and S3 error codes rather than parse the full English message text.
Separately, #232 classifies staged xl.meta loss at final commit as 503 SlowDownWrite. That status is distinct from the HTTP 400 client-checksum errors in the table below and does not change genuine incomplete-body handling. See the S3 write-commit reference.
Why Validation Belongs at Completion
CreateMultipartUpload selects the checksum algorithm and object checksum type, and each UploadPart call records its part checksum. CompleteMultipartUpload must validate both the final digest and any explicit type assertion against that stored contract. A completion request cannot change COMPOSITE to FULL_OBJECT merely because both share the same base algorithm, nor can it bypass a mismatched type assertion by omitting the final digest.
The repair distinguishes an omitted checksum type from an explicit type assertion and normalizes only internal representation flags. It introduces operation-specific error types for multipart completion while leaving UploadPart and the global checksum-mismatch error mapper unchanged.
Current Error Contract
All validation failures below return HTTP 400 and do not commit a completed object.
| Request Condition | S3 Error |
|---|---|
| Wrong full-object or composite object digest | BadDigest |
| Explicit supported type differs from the initiated type, including a type-only assertion | BadDigest |
| Unknown or lowercase type token | InvalidArgument |
| Completion algorithm differs from the initiated algorithm | InvalidArgument |
| Missing required composite part checksum | InvalidRequest (naming the algorithm and part number) |
CRC64NVME with COMPOSITE at initiation |
InvalidArgument |
CRC64NVME value with COMPOSITE rejected by checksum parsing at completion |
InvalidArgument |
Bare COMPOSITE type at completion of a CRC64NVME FULL_OBJECT upload |
BadDigest |
Incorrect client checksum during UploadPart |
XAmzContentChecksumMismatch |
For FULL_OBJECT uploads, per-part checksums in the completion XML may be omitted, while any supplied part checksums are still validated against stored part metadata. Matching type-only assertions are accepted. Omitting the optional checksum type header does not assert COMPOSITE. Combining SHA1 or SHA256 with FULL_OBJECT is rejected by the algorithm-and-type parser before reaching the stored-type comparison.
CRC64NVME Decision and Source Evidence
An early review considered deferring #50 pending further SDK compatibility evidence, where CRC64NVME + COMPOSITE had been silently canonicalized to FULL_OBJECT. That silent canonicalization hid invalid client requests instead of failing fast. PR #93 rejects the invalid CRC64NVME + COMPOSITE combination in both headers and trailers, and PR #96 removes the silent canonicalization during CompleteMultipartUpload. Because completion can fail either during header parsing (when both a CRC64NVME digest and COMPOSITE type are supplied) or during stored-type comparison (when only a bare COMPOSITE type header is sent against an initiated CRC64NVME FULL_OBJECT upload), the table above records two distinct error codes (InvalidArgument and BadDigest).
The initial completion error mapping landed in PR #74, with the explicit type-assertion follow-up (7e079ff05) merged through PR #85. The handler test suite in cmd/erasure-multipart-fullobject_test.go exercises full-object and composite digest mismatches, type-only assertions, invalid tokens, and both CRC64NVME rejection stages.
Separate Streaming-Checksum Follow-Up
PR #143, resolving #107 (reported by @cbornet), handles aws-chunked requests that advertise x-amz-trailer but supply the checksum in a standard HTTP header—a pattern used by the AWS Java SDK v2. It also rejects invalid header-delivered checksum values instead of dropping validation. This streaming-checksum repair shipped in Server 20260916 and is independent of multipart completion error mapping.
Compatibility and Remaining Boundaries
Callers that inspect S3 error codes receive BadDigest for completion digest or type mismatches and InvalidRequest for missing composite part checksums. Successful checksum-free multipart uploads and ETag semantics are unchanged, and no on-disk metadata migration is required. Related multipart completion contracts are covered in CompleteMultipartUpload ChecksumType, UploadPart Checksum, and the Duplicate Part Numbers Security Advisory.
Three narrower edge behaviors remain documented boundaries:
- Supplying a final checksum digest at completion when
CreateMultipartUploadrecorded no checksum algorithm is rejected asBadDigest. - Composite part-count mismatches and composite digest mismatches share the same
BadDigesterror description. - A
-Npart-count suffix attached to aFULL_OBJECTchecksum header is ignored rather than validated as a part count.
4.37 - Clearing Stale Fields in Legacy Metadata and Scanner Responses
This article records the design and verification of PR #233 (tested head 86c5a92eef3d, merged as d94b3b3c9251), which shipped in Server 20261018. It fixes stale fields carried over when legacy xl.meta version headers and remote namespace-scanner responses are reused.
Overview
Two independent code paths reused a Go struct without resetting all of its logical state, allowing fields omitted from the new input to retain values from a previous operation. The fix resets those fields directly in the decoder and response builder that know the schema of each message:
| Path | Trigger | Previous state | Repaired state |
|---|---|---|---|
| Legacy V2 version header | Decode a V2 header into a reused struct with nonzero EC fields | EcN becomes zero, EcM retains its old value |
Both absent EC fields are zeroed |
| Legacy V1 version header | Reuse a struct containing a signature or EC fields | Fields absent from V1 retain their old values | Signature, EcN, and EcM are zeroed |
| Scanner update response | Reuse an envelope that previously carried a final cache | Both Update and the old Final are serialized |
Only Update is present |
| Scanner final response | Reuse an envelope that previously carried a progress update | Both the old Update and Final are serialized |
Only Final is present |
Legacy xl.meta Headers
The compact version header inside xl.meta has its own version number, distinct from the outer xl.meta format version and the SILO release number. The current header version is V3 and includes the erasure-coding fields EcN and EcM. Legacy V2 headers have five fields and omit both EC fields; the V1 decoder handles an earlier four-field layout without a signature or EC fields (noted in source comments as never released, so clearing V1 fields is defensive hygiene on a retained decoder).
Both V2 decoding entry points — byte-oriented UnmarshalMsg and stream-oriented DecodeMsg — previously began with:
Assigning z.EcN twice left z.EcM untouched. Decoding into a freshly allocated zero-valued struct masked the typo, but reusing a struct that previously held EcN=4, EcM=8 yielded EcN=0, EcM=8 after decoding a V2 header. The fix changes the assignment to z.EcN, z.EcM = 0, 0 and clears the absent Signature field before decoding V1 headers. Current V3 headers continue to populate EcN and EcM from the wire payload.
In production, loadIndexed reuses the backing slice of version headers, so resetting fields only on fresh allocations would leave that path exposed. The regression suite exercises this slice-reuse path directly. See the updated legacy decoders and indexed-header loading and comparison.
Without this reset, a decoded V2 header varies depending on the struct’s prior contents. Because header equality checks and quorum vote counting compare all struct fields, differing stale EcM values can split otherwise identical headers into separate quorum groups (even though hasEC() requires both EcN and EcM to be nonzero).
Namespace-Scanner Responses
Remote disk scanning streams nsScannerResp envelopes over the grid RPC framework. By contract, each envelope carries either a progress Update (*dataUsageEntry) or a Final summary (*dataUsageCache). The RPC handler pools nsScannerResp structs across requests without zeroing their fields on return.
Previously, NSScannerHandler set only the active pointer for the current message. When a pooled struct was reused in a subsequent scan, both pointers could remain non-nil and serialize together over the wire. Because the client receiver inspects both fields independently — forwarding Update whenever non-nil and recording Final whenever non-nil — a final response could replay a stale progress update into usage aggregation, and a progress response could carry a Final cache pointer from an earlier scan.
Although the legitimate Final message sent at the end of the stream eventually overwrites the client’s cached Final pointer, intermediate envelopes violated mutual exclusion and transmitted stale scan data. The fix explicitly clears the inactive pointer when constructing each response:
This applies to remote disk scanner responses (local disk scans do not serialize nsScannerResp). See the updated server handler, client receiver, and bucket-usage aggregation.
Why grid.Recycler Is Not Called in the Generic Pool
The initial PR draft also called Recycler.Recycle() inside the generic stream handler’s PutResponse before returning the response wrapper to the handler’s sync.Pool. That conflated transferring ownership of inner pooled resources with resetting a struct for reuse.
Specifically, grid.MSS.Recycle() returns its underlying map to a dedicated map pool and sets the struct’s map field to nil. If the outer response wrapper is then placed back into the handler’s response pool, a subsequent NewResponse call can retrieve that wrapper from the pool, skip the constructor, and hand the caller a nil map. A targeted reproduction test passed on the pre-PR main branch and panicked with that generic PutResponse change applied.
The maintainer follow-up therefore dropped the generic PutResponse change and the nsScannerResp.Recycle method, keeping only the explicit field assignments in NSScannerHandler. In the merged commit, internal/grid/handlers.go and cmd/storage-rest-common.go are unchanged relative to pre-PR main, preserving the Recycler ownership contract documented in grid types.
Verification
Deterministic Regression Tests
The merged commit adds TestXLMetaV2HeaderLegacyUnmarshalResetsEc and TestNSScannerHandlerClearsStaleResponses.
The header test covers V2 byte decoding, V2 stream decoding, reused indexed header slices, V1 field clearing, and preservation of V3 EC fields. The scanner test runs the real NSScannerHandler across three consecutive scans and inspects the serialized responses to verify that each message carries only its intended field from the current scan.
To make the scanner test deterministic under sync.Pool, the test pool’s New function pre-populates every newly allocated nsScannerResp with dirty Update and Final pointers. Because the Go runtime can evict sync.Pool entries at any garbage collection cycle, a test that merely returns a dirty object to the pool and hopes to get it back on the next Get can pass vacuously when a fresh zero-valued struct is allocated instead. Pre-populating new allocations guarantees that every Get returns a dirty envelope: the test fails deterministically against pre-fix main (0d3ba5d4cbfd) and passes with the fix.
Distributed Scanner Verification
End-to-end verification ran the merged binary as two server processes managing eight data directories over loopback. The harness created twelve buckets, wrote two differently sized objects into each bucket, and then deleted one object, overwrote the second, and wrote a third object in every bucket.
| Checkpoint | Objects | Total logical bytes | Observed result |
|---|---|---|---|
| Initial writes | 24 | 64,740 | Bucket and cluster totals matched across three consecutive usage updates |
| Delete, overwrite, and add | 24 | 111,960 | Bucket and cluster totals matched across three subsequent usage updates |
| Readback verification | 24 | 111,960 | All 24 object lengths and SHA-256 digests matched |
Both processes logged active bucket scans, with the non-coordinator node executing 119 ScanBucketDrive operations over the remote grid RPC stream.
Remote CI
All eight CI workflows passed on PR head 86c5a92eef3d: DCO, six Go CI jobs, and VulnCheck. Following merge, the six Go CI jobs on main and VulnCheck on main also passed.
Compatibility and Operations
The fix only updates field assignments during legacy header decoding and scanner response construction. Wire protocols, on-disk formats, dependencies, IAM policies, and configuration flags are unchanged, and no object rewrites or cache deletions are needed.
- Upgrade cluster nodes to Server 20261018 or later. Because the wire format is unchanged, nodes can be upgraded in a normal rolling fashion; see the component matrix for release-level compatibility details.
- Allow background scanner cycles to converge. Check scanner status with
mcli admin scanner status ALIAS -n 1and verify bucket object counts and logical sizes after a full scan pass completes. - Do not manually edit metadata or caches. Normal background scanning automatically refreshes derived usage totals without manual changes to
xl.metaor internal cache files.
Delivery and Credit
Julien Laurenceau contributed the initial fixes and tests. The maintainer follow-up narrowed the pooling change to preserve grid.Recycler ownership semantics and added the deterministic dirty-allocation regression suite; the original commits and DCO sign-offs are preserved in the merge history.
4.38 - LIST Recovery: Confirm Only Uncertain Names and Bound the Waiting Cost
Shipped in SILO Server 20261018, PR #239 (merged as fbcb51197, closing #218 and retaining mr javad seydi’s original signed commits from #221) and the October 18 directory/common-prefix follow-ups (3dd53eb81 and 2080b3d03) resolve silent object omissions during concurrent ListObjects and ListObjectsV2 scans. This design builds on the September null-version quorum repair and replaces an earlier recovery proposal that invoked the ordinary GET metadata path directly.
A Successful Directory Scan Can Omit a Live Object
Suppose reports/a.csv can be downloaded with GET, but a LIST request returns HTTP 200 OK without that object name. Backup and inventory tools treat an HTTP 200 listing as authoritative and report the file as missing; a synchronization tool configured to delete destination objects absent from the source listing can delete a valid destination copy. A successful but incomplete listing causes more downstream harm than an explicit error response.
LIST combines directory and metadata records collected from different drives at slightly different times. During concurrent overwrites, deletions, rolling restarts, or pool migrations, some drives return older records, others return newer records, and others temporarily lack a record. Reaching listing quorum requires enough mutually compatible records; unrelated object generations or incompatible erasure layouts cannot simply sum their votes together. Previously, when a directory entry failed to reach quorum, the listing merger could silently drop the entry while returning HTTP 200 for the overall request.
This repair applies to ListObjects and ListObjectsV2 (for example, GET /bucket?list-type=2&prefix=reports/), which is separate from ListMultipartUploads (#79). It introduces no new S3 query parameters, environment variables, or client flags.
Selected Recovery Design
The listing pipeline first runs the existing directory merge so resolved entries stay on the fast path. Recovery admits only an unresolved latest-only entry that has at least one decodable object record and enough decodable records plus failed reader streams to reach the original listing quorum threshold. Failed streams only justify attempting recovery; they never count as proof that an object exists or satisfies read quorum. Inputs below the threshold and full version-history listings (ListObjectVersions) keep their existing behavior.
An eligible unresolved name is carried to the pool merger, where its complete same-name group is confirmed as a unit. The merger acquires an object read lock to exclude concurrent same-name mutations during confirmation, then re-reads current metadata from disk. Confirmation produces one of three outcomes:
- Present: Select current metadata using the object’s actual erasure layout and standard read quorum. Preserve delete markers until the cross-pool merge can discard older live copies in other pools.
- Deleted: Every configured source-set drive slot must return native
FileNotFoundduring the raw metadata read, and the merger must also inspect the name’s current placement target set in every pool before omitting the group. - Unknown: Return the existing error (such as HTTP 503
SlowDownRead) so the caller never mistakes an unconfirmable state for a complete listing. Offline drives, corrupted metadata, missing volumes, and decode errors never count as proof of deletion.
Streaming pagination still returns completed pages before the full bucket scan finishes. Clients must consume all continuation pages; if any page fails, the preceding pages do not form a complete inventory.
Asynchronous cache cleanup also preserves the original request identity: commit 2a4db6a4f captures the cache peer before launching the background cleanup goroutine and passes listing options by value, preventing foreground cache-ID resets from racing with cleanup RPCs. Neither the cache format nor the continuation-token encoding changes.
Why inspect other pools after confirming source-set absence? Consider a two-pool cluster where Pool A’s cached directory scan saw a.csv while Pool B’s earlier scan was empty, and a concurrent migration then moved a.csv to Pool B and removed Pool A’s copy. Omitting the key solely because Pool A is now empty would miss the live object in Pool B. After confirming strict source absence, the merger queries the current target set across pools and replaces the stale same-name group with the live result. Each distinct erasure set is re-read at most once per recovery.
Legitimate DELETE traffic provides a second counterexample: out of 16 directory inputs, suppose 6 hold an older generation, 6 hold a newer generation, and 4 already reflect deletion, so the directory scan cannot form a quorum. Meanwhile, DELETE has already returned HTTP 204, GET returns HTTP 404, and all 16 drives now report FileNotFound. An early iteration of #239 treated that healthy post-delete state as unresolvable and returned HTTP 503. Checking for unanimous native FileNotFound across the source set confirms legitimate deletion while still failing closed on genuine drive or decode errors.
Object Absence Must Not Swallow a Coexisting Directory
Commit 3dd53eb81 handles a same-name group where an object is confirmed absent while a coexisting directory entry remains valid: recovery drops only the absent object and retains the directory / common prefix. Commit 2080b3d03 preserves ordering when pending objects, resolved objects, and directories appear together: duplicate directories collapse into a single common prefix while live objects remain object entries.
In-Memory Metadata Representation
The repair adds no persistent files, xl.meta fields, on-disk indexes, or migrations. It adds a single source-set pointer to the in-memory metacache entry:
Pending entries travel only through internal in-memory channels and are consumed by the pool merger. Both metacache encoders explicitly reject unresolved entries, so listingSet is never serialized into a pagination cache or transmitted over the wire.
The recovery re-read uses a read-only metadata helper with standard version selection and read-quorum rules, avoiding the opportunistic healing and dangling-object cleanup side effects of the full GET path. A narrow in-memory recount applies only after initial quorum selection fails on eligible latest null versions with known erasure layouts: compatible layout groups are counted separately against the original quorum threshold without lowering quorum or mixing votes across layouts.
Code Changes Across Subsystems
| Subsystem | Change and Rationale |
|---|---|
| Directory resolution and pool merge | Carry pending names to the complete same-name group and confirm at the pool merger so per-pool decisions cannot drop live objects or resurrect shadowed copies. |
| Metadata reading | Distinguish unanimous native FileNotFound from read/decode errors; reuse read-quorum and version selection without GET write side effects. |
| Object locking | Acquire the pool-level lock on multi-pool clusters; on single-pool clusters with distinct source and target sets, acquire first-set and target-set read locks in fixed order to cover both DELETE/COPY and PUT lock domains. |
| Pagination cache | Publish terminal scan errors before marking the cache complete so HTTP error responses do not wait on a failing remote cache RPC, and preserve the local cache ID and error if remote update fails. |
| Waiting for missing cache blocks | Check producer state promptly after a block read failure and rotate a single online-disk probe per retry before issuing the full quorum read. |
| Cancellation and lock contention | Make lock-acquisition retry waits context-cancelable, and grant positively identified listing-recovery lock conflicts one short 2 ms wait-and-recheck window. |
| Regression tests and CI | Cover confirmed deletion, cross-pool migration ordering, dual-set lock routing, pagination termination, failed RPCs, context cancellation, and late lock grants under -race. |
The low-level directory-walk callback keeps its existing signature because recovery errors originate in the merger, leaving scanner, healing, rebalance, and decommission callers unchanged.
When single-pool recovery spans two distinct lock domains, holding the first read lock while acquiring the second can briefly delay DELETE or COPY operations routed to the first set. Recovery releases all locks before result encoding, channel sends, and cache RPCs so slow clients cannot extend the lock hold time. If the second lock acquisition fails or the context is canceled, the first lock is released immediately.
Evaluated and Rejected Alternatives
| Option | Benefit | Reason Rejected |
|---|---|---|
| Keep existing behavior | Zero additional runtime cost | HTTP 200 listings can silently omit live objects, breaking backup and sync tools. |
| Return HTTP 503 on every disagreement | Smallest patch; no metadata re-read | Fails requests that a single re-read can resolve and repeatedly fails after healthy concurrent DELETEs. |
| Lock and re-read every listed object | Uniform rule | Imposes per-object lock and disk I/O overhead on all listings, degrading large-bucket scans. |
| Confirm only the source pool or skip locking | Fewer reads and lock acquisitions | Concurrent overwrites, deletions, and cross-pool migrations still produce silent omissions. |
| Shorten retry intervals for all writers globally | Reduces some contention tails | Alters global write scheduling and increases cluster-wide lock RPC traffic. |
| Confirm eligible disagreements with a bounded short wait (selected) | Keeps normal scans on the fast path; returns confirmed state or an explicit error | Adds targeted re-read I/O and lock handling on contended entries. |
Runtime Cost Profile
| Trigger | Additional Cost |
|---|---|
| Ordinary listing entry | On 64-bit builds, the in-memory entry struct grows from 56 to 64 bytes (one 8-byte pointer) plus a branch check; zero extra locks or disk reads. |
| Eligible disagreement | Concurrent per-drive metadata re-read across pending source sets; two read-lock acquisitions on single-pool split domains, or one pool-level lock on multi-pool setups. |
| Confirmed source-set absence | Queries each pool’s current target set (reusing sets already read), bounded by the deduplicated union of source and target sets rather than a bucket scan. |
| High fraction of unresolved names | The ordered merger confirms same-name groups serially, which slows the overall scan when many keys are simultaneously under mutation. |
| Recovery reader vs. writer lock conflict | Each identified recovery lock RPC waits once for a nominal 2 ms before rechecking, capped at 64 concurrent waiters per local locker outside its internal mutex (~264 B and 4 allocs per waiting RPC). |
| Ordinary backoff after failed lock acquisition | Replacing time.Sleep with a cancelable timer/select adds a small allocation only after a lock attempt has already failed. |
For 1,000 simultaneously resident entries, the pointer field adds roughly 8 KiB of heap memory. Each name’s recovery runs under a 5-second context budget, and any partial or late lock grants after cancellation are cleaned up immediately.
Verification and Benchmark Evidence
Local make test passed all 1,773 top-level tests alongside targeted -race suites covering recovery, pagination caches, lock routing, and lock cancellation. On PR commit 87c1c6b3d, all eight CI checks passed (Go CI, DCO, vulnerability analysis, and the snapshot pipeline).
An independent 10-run paired benchmark against main (7e9454456)—two identical-main A/A controls followed by four alternating main/candidate pairs (20 s warmup, 120 s measurement across 8 processes, 2 pools, and 32 logical drive directories on NVMe loopback)—completed 22,410 full scans with zero omissions, duplicates, corruptions, or healthy-state HTTP errors:
| Metric | Paired Candidate / Main p95 Change | 95% Confidence Interval |
|---|---|---|
Full LIST page |
+0.82% | −3.53% to +5.36% |
Same-name DELETE |
−0.38% | −1.80% to +1.05% |
Different-name DELETE |
−0.55% | −4.74% to +3.84% |
Secondary paired p95 deltas were +0.47% for complete paginated scans, +2.82% for PUT, −0.51% for GET, +12.60% for middle continuation pages (max-keys=7), and +18.95% for same-name DELETE p99 (with one isolated 327.248 ms same-name DELETE sample on the candidate vs. 148.186 ms in the A/A control). Aggregate 8-process CPU changed by −0.07% and summed RSS by +0.11%. Because the 95% confidence interval upper bound for full LIST page p95 reached +5.36% (above the strict 5.00% gate) and the two identical-main controls varied by ~12% on different-name DELETE p95, the statistical 5% gate is recorded as inconclusive rather than passed, while point estimates confirm minimal steady-state overhead.
Sequential graceful restarts of all 8 processes completed 5,235 successful full scans (4,754 inside the measurement window) with zero omissions, duplicates, or corruptions; all 139 failed HTTP requests fell within the stopped node’s stop-to-ready window, and all 80 final object states, pagination checks (max-keys of 1, 7, and 1000), 416 present GETs, and 512 absent GETs matched the last acknowledged write or delete.
Compatibility
S3 request parameters, continuation-token encoding, on-disk formats, configuration keys, and client SDK behavior remain unchanged. Entries that enter recovery and cannot be confirmed now fail the LIST request with existing S3 error codes (such as 503 SlowDownRead) rather than returning an incomplete HTTP 200 inventory. Downgrading requires no data migration.
4.39 - Object Lock Retention Bypass and Version-Overwrite Repair Design (SN-2026-015 / SN-2026-016)
This document records the verification, Git provenance, repair design, critical-path evaluation, and candidate trade-offs for two Object Lock (WORM) security vulnerabilities identified through external reports and an internal write-path audit: SN-2026-015 (PutObjectRetention bypass header authorization and condition-key short-circuit) and SN-2026-016 (version-addressed write overwrite bypass and write-path lock gaps). The complete repair (#234, #235, #241) shipped in Server 20261018.
Companion public security advisory:
1. Problem Verification and External Report Cross-Check
1.1 Two Independent External Reports and Internal Audit
This repair addresses two independent external vulnerability reports alongside an internal audit of adjacent write paths:
- Initial Report (F2 primary path): Privately reported by security researcher szopi00 via GitHub Security Advisory (GHSA-67mv-7hc9-wj88), showing that a principal holding
s3:PutObjectRetentionwithouts3:BypassGovernanceRetentioncould shorten unexpiredGOVERNANCEretention by sendingx-amz-bypass-governance-retention: true. - Independent External Report (F1 / F2 / F3): A separate external report independently reproduced three attack paths: unauthorized changes from
GOVERNANCEto multi-yearCOMPLIANCE, bypasses of request-body condition-keyDenypolicies ons3:PutObjectRetention, and shortening unexpiredGOVERNANCEretention without bypass authorization. - Internal Companion Audit (
SN-2026-016/ F4 and directorynull-version overwrites): Auditing all write paths for callers holdings3:PutObjectwithout bypass permission uncovered three related gaps: in-place overwrites via?versionId=without replication trust (putOpts), implicitnull-version overwrites on directory keys (dir/), and completion of persisted multipart uploads (CompleteMultipartUpload).
1.2 Claim-by-Claim Verification of the Independent Report
The following findings and line numbers refer to the pre-fix bcb5fc956 snapshot:
| Reported Claim | Verification | Code Evidence & Impact |
|---|---|---|
Impact 1 (Unauthorized COMPLIANCE retention): Attackers without s3:BypassGovernanceRetention (or subject to a conditional Deny) can turn GOVERNANCE into COMPLIANCE or set COMPLIANCE directly |
Confirmed | cmd/bucket-object-lock.go:205 checks only the raw header !byPassSet; cmd/auth-handler.go:773-781 skips IAM evaluation when the requested mode is COMPLIANCE, leaving byPassSet == true and returning ErrNone at if byPassSet || retSet. Under COMPLIANCE, not even the S3 root account can delete the version through the S3 API before expiry |
Impact 2 (Weakening GOVERNANCE lock): A credential without s3:BypassGovernanceRetention can shorten a GOVERNANCE lock using the raw header |
Confirmed | When the requested mode is GOVERNANCE, isPutRetentionAllowed evaluates byPassSet as false but evaluates retSet (s3:PutObjectRetention) as true; byPassSet || retSet still returns ErrNone |
Blast-radius difference between SILO and upstream MinIO on DeleteObjectVersion: SILO requires explicit s3:DeleteObjectVersion to delete a specific version; upstream MinIO checks s3:DeleteObjectVersion with DenyOnly: true |
Confirmed | SILO commit 75a6734e4 (2026-09-02) introduced deleteObjectAction requiring s3:DeleteObjectVersion, whereas upstream MinIO’s cmd/auth-handler.go still uses DenyOnly: true |
Affected versions and locations: Bug is in server cmd/, not silo-pkg; affects SILO RELEASE.2026-09-03T13-18-01Z (9b11dc94), RELEASE.2026-09-16T00-00-00Z, pre-fix SILO main (bcb5fc956), and the upstream minio/minio snapshot checked at that time |
Confirmed | silo-pkg evaluates policy Deny rules accurately; the flaw is in cmd/object-handlers.go, cmd/bucket-object-lock.go, and cmd/auth-handler.go |
Why unconditional Deny s3:PutObjectRetention cannot be used as a workaround: Unconditionally denying s3:PutObjectRetention breaks uploads using retention headers or default-retention workflows |
Confirmed | Application credentials often require s3:PutObjectRetention with condition keys (s3:object-lock-mode, s3:object-lock-remaining-retention-days) rather than an unconditional Deny |
2. Vulnerability Breakdown and Git Provenance
2.1 SN-2026-015: PutObjectRetention Authorization Short-Circuit (F1 / F2 / F3)
Before the fix, PUT /{bucket}/{object}?retention suffered from two structural defects:
- Signature-only check at handler entry:
PutObjectRetentionHandlercalledauthenticateRequest(ctx, r, policy.PutObjectRetentionAction), which verifies request authentication (signature/token) without evaluating IAM policies; - Conflated bypass check and inverted boolean logic: Inside
enforceRetentionBypassForPutandisPutRetentionAllowed, the server trusted the raw headerx-amz-bypass-governance-retention: trueand combined bypass permission with base retention permission using logical OR (byPassSet || retSet).
This produced three distinct authorization failures (F1–F3) plus a day-count arithmetic bug when clearing retention:
| ID | Attacker’s Granted Permissions | Trigger & Security Impact |
|---|---|---|
| F1 | Authenticated principal with no Object Lock permissions at all (neither s3:PutObjectRetention nor s3:BypassGovernanceRetention) |
Sending x-amz-bypass-governance-retention: true with: (1) an empty <Retention/> body clears unexpired GOVERNANCE retention to zero; or (2) <Mode>COMPLIANCE</Mode> locks the version into COMPLIANCE |
| F2 | Principal with s3:PutObjectRetention but without s3:BypassGovernanceRetention (covered by both external reports) |
Sending x-amz-bypass-governance-retention: true shortens unexpired GOVERNANCE retention or converts it to COMPLIANCE |
| F3 | Principal with s3:PutObjectRetention (subject to conditional Deny) or with s3:BypassGovernanceRetention |
(1) When the requested mode is COMPLIANCE or empty <Retention/>, the raw bypass header skips IAM evaluation in isPutRetentionAllowed, bypassing any conditional Deny on s3:PutObjectRetention; (2) when holding s3:BypassGovernanceRetention, byPassSet || retSet bypasses conditional Deny rules on s3:PutObjectRetention even for GOVERNANCE mode |
| Day-Count Arithmetic Bug | Principal subject to s3:object-lock-remaining-retention-days conditions |
Sending an empty <Retention/> body sets RetainUntilDate to zero time (0001-01-01). math.Abs(objRetention.RetainUntilDate.Sub(t).Hours()) clamped at Go’s 292-year time.Duration limit and evaluated to 106,752 days, bypassing NumericLessThan minimum-retention policies |
Exact Git Commit Provenance
- Correct initial design (Nov 2019 – Mar 2020):
ca96560d5(2019-11-20, Poorna Krishnamoorthy),f931fc7bf(2019-11-25), and #9030 (97cbac604, 2020-03-04):PutObjectRetentionHandlerenforcedcheckRequestAuthTypefors3:PutObjectRetentionat entry and evaluateds3:BypassGovernanceRetentionvia IAM before passing the authorization boolean intoenforceRetentionBypassForPut.
- Defect introduction (Apr 2020 — upstream #9259,
43a3778b4, 2020-04-06, Harshavardhana):- To support the three XML body condition keys (
s3:object-lock-mode,s3:object-lock-retain-until-date,s3:object-lock-remaining-retention-days), this commit removedcheckRequestAuthTypeat handler entry and pushed policy evaluation intoisPutRetentionAllowedinsideenforceRetentionBypassForPut, introducing three bugs at once:enforceRetentionBypassForPutchecked the raw headerif !byPassSet;isPutRetentionAllowedonly evaluated IAM whenretMode == objectlock.RetGovernance && byPassSet, leavingbyPassSet == trueforCOMPLIANCE;isPutRetentionAllowedconcluded withif byPassSet || retSet { return ErrNone }.
- To support the three XML body condition keys (
- Expansion to empty
<Retention/>clear requests (May 2020 — upstream #9677,bede525dc, 2020-05-22, Ashish Kumar Sinha):- Allowed clearing
GOVERNANCEretention via an empty<Retention/>body (Mode == ""), which also took theretMode != RetGovernancebranch and introduced the106,752-day zero-date calculation bug.
- Allowed clearing
- Partial fix in Feb 2025 and mechanical regression in Apr 2025:
- Upstream #20929 (
437dd4e32, 2025-02-12, Ramon de Klein) added an unconditionalcheckRequestAuthType(ctx, r, policy.PutObjectRetentionAction, bucket, object)at handler entry, masking F1 (while breaking condition-keyAllowpolicies and leaving F2/F3 open). - Just 50 days later, upstream #21103 (
8c7097528, 2025-04-03, Harshavardhana) mechanically replacedcheckRequestAuthTypewith signature-onlyauthenticateRequestwhile keeping the stale// Check permissions to perform this object retention operationcomment, reopening F1. SILO forked after this commit and inherited all three flaws.
- Upstream #20929 (
2.2 SN-2026-016: Version-Addressed Write Overwrite Bypass and Write-Path Lock Gaps
In standard S3 semantics, public write APIs (PutObject, CopyObject, CreateMultipartUpload, PostPolicy) never allow callers to specify a target versionId to overwrite an existing version in place. Three related write-path flaws broke this invariant:
| Path ID | Trigger | Security Impact |
|---|---|---|
F4 (Explicit target versionId on write) |
A caller with s3:PutObject passes an existing version’s versionId (UUID or null) to PutObject, CopyObject, CreateMultipartUpload, PostPolicy, or Snowball archive extraction (PAX minio.versionId) |
The storage layer overwrites that version’s payload and metadata in place. For an unexpired GOVERNANCE-locked version, its payload and retention metadata are replaced or cleared |
Companion Gap A (Implicit null version on directory keys dir/ and slow-body TOCTOU race) |
A caller sends a normal PUT /{bucket}/dir/ (without ?versionId=) to an Object Lock bucket |
(1) putOpts maps isDirObject(object) to opts.VersionID = "null", while checkPutObjectLockAllowed sees opts.VersionID == "" via getOpts, skipping the existing-lock check and allowing a normal PUT dir/ to overwrite and erase unexpired GOVERNANCE, COMPLIANCE, and LegalHoldOn locks on the directory object; (2) even with HTTP preflight fixed, preflight runs before reading r.Body, so a writer can pass preflight, pause streaming the HTTP body while an admin sets COMPLIANCE on dir/, and then finish streaming to overwrite the locked null version; (3) checkPutObjectLockAllowed only inspects locks when getObjectInfoFn returns err == nil, failing open on storage read errors |
Companion Gap B (Persisted target version in CompleteMultipartUpload) |
NewMultipartUpload persists fi.VersionID in .minio.sys/multipart/.../uploadID/xl.meta, and CompleteMultipartUpload uses completeMultipartOpts rather than putOpts |
A multipart upload initiated before the upgrade, initiated by a replication credential, or recovered from an older staging snapshot after a surviving replica has already committed and COMPLIANCE-locked that same UUID version has its target fi.VersionID persisted on disk; without an unconditional write-lock existence check during CompleteMultipartUpload, an ordinary s3:PutObject caller can complete that upload and overwrite the COMPLIANCE-locked version |
Exact Git Commit Provenance
- May 2020 —
GOVERNANCEomitted fromcheckPutObjectLockAllowed(upstream #9513,9b3b04ece, 2020-05-04):checkPutObjectLockAllowedonly blocked unexpiredCOMPLIANCEandLegalHoldOninif opts.VersionID != "" && !replicaTrusted, omittingGOVERNANCEand ignoring non-nilerrors fromgetObjectInfoFn. - Sep 2020 — Unauthenticated
?versionId=accepted inputOpts(upstream #10598,56d1b227c, 2020-09-30, Poorna Krishnamoorthy):putOptsaccepted?versionId=from the query string to support replication without checkingtrustedReplication. - May 2023 — Directory objects forced to
nullVersionID(upstream #17132,1d0211d39, 2023-05-04, Harshavardhana):putOptsanddelOptsforcedopts.VersionID = nullVersionIDforisDirObject(object)without updatingcheckPutObjectLockAllowedor the storage commit path.
3. Final Repair Plan: What Code Changes and How
The final repair combines Core Plan A (SN-2026-015) with Entry + Storage Write-Lock Defense-in-Depth + Fail-Closed Preflight (SN-2026-016) across 11 production files under cmd/:
| Issue | File | Summary of Change |
|---|---|---|
| SN-2026-015 (Core Plan A, 3 files) |
cmd/object-handlers.go |
In PutObjectRetentionHandler, evaluate isRetentionActionAllowed(PutObjectRetentionAction) immediately after ParseObjectRetention(r.Body) and before reading any bucket or object state |
cmd/bucket-object-lock.go |
Add isRetentionActionAllowed (evaluating zero/past dates as "0" days and ceiling future days via Unix second/nanosecond arithmetic); rewrite enforceRetentionBypassForPut to require both the bypass header and s3:BypassGovernanceRetention only when weakening an unexpired stored GOVERNANCE lock |
|
cmd/auth-handler.go |
Delete the flawed 45-line isPutRetentionAllowed helper (byPassSet || retSet) |
|
| SN-2026-016 (Entry & Write-Lock Guards, 8 files) |
cmd/object-api-options.go |
In putOpts (the single convergence point for HTTP write options), return errAuthentication (403 AccessDenied) whenever vid != "" && !trustedReplication |
cmd/batch-handlers.go |
In BatchJobReplicateV1.writeAsArchive, set ReplicationRequest: true on outer SnowballOptions so server-side batch replicate archives pass the outer replication marker check |
|
cmd/bucket-object-lock.go |
(1) In preflight checkPutObjectLockAllowed, resolve nullVersionID for isDirObject(object), block unexpired GOVERNANCE in addition to COMPLIANCE and LegalHoldOn, and fail closed immediately on non-NotFound read errors; (2) add fail-closed checkObjectLockedForOverwrite for null-version re-checks under the storage write lock |
|
cmd/bucket-handlers.go |
Call checkPutObjectLockAllowed for directory objects (isDirObject(object)) in PostPolicyBucketHandler |
|
cmd/object-multipart-handlers.go |
Pass trustedReplication to checkPutObjectLockAllowed in CompleteMultipartUploadHandler |
|
cmd/erasure-object.go |
Under the destination object write lock (NewNSLock) in erasureObjects.putObject and erasureObjects.CopyObject, call checkObjectLockedForOverwrite when writing to "" or nullVersionID |
|
cmd/erasure-multipart.go |
In CompleteMultipartUpload under the destination write lock, run checkObjectLockedForOverwrite for null versions and check whether any non-empty fi.VersionID already exists via getObjectInfo (or replicaObjectInfo across pools) |
|
cmd/erasure-server-pool.gocmd/erasure-sets.go |
Supply cross-pool z.replicaObjectInfo for public writes and multipart completions in multi-pool deployments, and forward versioning/replication flags in CopyObject |
3.1 SN-2026-015 Detailed Code Changes (Core Plan A)
(1) cmd/object-handlers.go: Unconditional Base Authorization at PutObjectRetentionHandler Entry
Immediately after authenticateRequest and ParseObjectRetention(r.Body)—and before GetBucketInfo, globalBucketObjectLockSys.Get, or getOpts—enforce s3:PutObjectRetention via isRetentionActionAllowed:
(2) cmd/bucket-object-lock.go: isRetentionActionAllowed and enforceRetentionBypassForPut
isRetentionActionAllowed constructs object-lock-mode, object-lock-retain-until-date, and object-lock-remaining-retention-days from the parsed request body and calls globalIAMSys.IsAllowed:
- When clearing retention (empty
<Retention/>, zeroRetainUntilDate) or for non-future dates,object-lock-remaining-retention-daysis set to"0"(fixing the106,752-day bug caused bytime.Durationclamping andmath.Abs, while keeping the key present soNumericLessThanDenypolicies match clear requests); - For future dates, ceiling days are computed from Unix seconds and nanoseconds to avoid 292-year
time.Durationsaturation.
Inside the object write lock, enforceRetentionBypassForPut evaluates only stored retention state:
(3) cmd/auth-handler.go: Remove isPutRetentionAllowed
Delete the 45-line isPutRetentionAllowed function from cmd/auth-handler.go, eliminating the byPassSet || retSet short-circuit.
3.2 SN-2026-016 Detailed Code Changes (Entry Guards + Write-Lock Re-Checks + Fail-Closed Preflight)
Why must SN-2026-016 retain storage write-lock checks in addition to HTTP entry guards and preflight checks? Four concrete scenarios cannot be prevented by entry or provenance markers alone:
- Slow-body
PUT dir/TOCTOU race (TestPlanReviewSlowDirectoryPut): InPutObjectHandler,checkPutObjectLockAllowedruns before streamingr.Bodyto disk, whereas the namespace write lock (er.NewNSLock) is acquired only afterr.Bodyfinishes streaming to a temporary file. A writer withs3:PutObjectcan initiatePUT /bucket/dir/, pass preflight whiledir/is unlocked, pause streaming the request body while an administrator appliesCOMPLIANCEretention (200 OK), and then finish sending the body. Without re-checking underer.NewNSLock, the pausedPUToverwrites thenullversion and erases the committedCOMPLIANCElock. - Pre-existing or replication-initiated persisted
uploadIdcompletion (TestPlanReviewLegacyMPUCompliance):putOptsruns only duringNewMultipartUpload;CompleteMultipartUploadusescompleteMultipartOptsand takes the targetfi.VersionIDfrom.minio.sys/multipart/.../uploadID/xl.meta. For any upload created before the upgrade (or initiated by a replication principal), completing the upload after the target UUID version is locked underCOMPLIANCErequires no concurrency and onlys3:PutObjectunlessCompleteMultipartUploadvalidates the target under the write lock. - Recovered multipart staging state colliding with a replicated
COMPLIANCEversion (TestRecoveredMultipartUploadAfterReplica): Even when a multipart upload’sfi.VersionIDwas originally generated by the server atNewMultipartUploadtime, if an older multipart staging snapshot is restored after a surviving replica has already re-synced andCOMPLIANCE-locked that completed UUID version, a provenance marker only proves where the UUID came from at initiation—not that the version does not already exist at completion time. Therefore,CompleteMultipartUploadmust unconditionally check under the destination write lock whetherfi.VersionIDalready exists. - Fail-closed read error handling (
TestPlanReviewPreflightReadFailure): Only an explicit non-existence result (ObjectNotFound,VersionNotFound, or delete markerMethodNotAllowed) may be treated as unlocked; bucket lock configuration errors or storage read quorum failures (InsufficientReadQuorum) must reject the write immediately at preflight as well as at commit time.
The four components of the SN-2026-016 repair are:
(1) Reject Untrusted versionId at the putOpts Convergence Point (cmd/object-api-options.go)
All HTTP write option builders (PutObject, CopyObject, NewMultipartUpload, PostPolicy, PutObjectExtract) converge at putOpts:
(2) Set the Outer Replication Marker on Batch Snowball Archives (cmd/batch-handlers.go)
When SILO’s batch replicate streams a Snowball archive via writeAsArchive to another SILO/MinIO target, each tar PAX entry carries minio.versionId. PutObjectExtractHandler requires X-Minio-Source-Replication-Request: true (markerExact) on the outer request before granting entryTrusted:
(3) Enforce Fail-Closed null-Version Lock Checks at Both HTTP Preflight and Storage Write Lock (cmd/bucket-object-lock.go, cmd/erasure-object.go, cmd/erasure-server-pool.go, cmd/erasure-sets.go)
- HTTP preflight (
checkPutObjectLockAllowed): ResolvenullVersionIDfor directory objects (isDirObject(object)), block unexpiredGOVERNANCEalongsideCOMPLIANCEandLegalHoldOnwhen!replicaTrusted, and fail closed immediately on any read error other thanObjectNotFound/VersionNotFound/MethodNotAllowed:
- Storage write-lock commit (
checkObjectLockedForOverwrite): Insideer.putObject,er.CopyObject, ander.CompleteMultipartUploadwhile holdinger.NewNSLock(bucket, object), any non-replication write targeting""ornullVersionIDcallscheckObjectLockedForOverwrite(usingz.replicaObjectInfoacross pools) and fails closed on config or read errors:
(4) Check Persisted fi.VersionID Under the Object Write Lock in CompleteMultipartUpload (cmd/erasure-multipart.go)
Under er.NewNSLock(bucket, object), non-replication multipart completions (!opts.ReplicationRequest && !opts.ReplicaLockReconcile && !opts.DataMovement) check the persisted fi.VersionID from uploadid/xl.meta:
- If
fi.VersionID == "" || fi.VersionID == nullVersionID: callcheckObjectLockedForOverwrite; - If
fi.VersionIDis a non-empty UUID: querygetObjectInfo(orreplicaObjectInfoacross pools) and reject witherrAuthentication(403 AccessDenied) if that version already exists on disk (including delete markers), failing closed on read errors without deleting staged parts.
4. Critical-Path and Performance Impact Analysis
| Operation Path | Execution Path & Overhead After Repair |
|---|---|
GetObject / HeadObject / ListObjects / DeleteObject |
Untouched; read and delete paths do not traverse any modified code |
PutObject / CopyObject in regular buckets (Object Lock disabled) |
putOpts checks vid != ""; under the write lock, checkObjectLockedForOverwrite checks in-memory globalBucketObjectLockSys.Get(bucket) (!retentionCfg.LockEnabled) and returns immediately with no extra disk I/O or cross-pool calls |
Regular PutObject / CopyObject in Object Lock buckets (non-/ keys) |
Object Lock buckets are always versioned; normal single-request writes generate a non-empty UUID fi.VersionID (fi.VersionID != "" && fi.VersionID != nullVersionID), so neither preflight nor write-lock commit performs an extra getObjectInfo read |
Directory object writes (PUT dir/) or null-version overwrites in Object Lock buckets |
Both preflight and the storage write lock read the null version’s metadata and fail closed on read errors, preventing slow-body TOCTOU overwrites and read-failure bypasses |
Multipart upload completions (CompleteMultipartUpload) in versioned buckets |
Performs one fail-closed getObjectInfo (or replicaObjectInfo across pools) lookup under the destination write lock during final multipart commit (NewMultipartUpload and UploadPart are unaffected). Any future optimization to skip this lookup for newly generated uploads must account for snapshot-recovery collisions and be validated separately |
PutObjectRetention (valid extensions and unauthorized requests) |
Base s3:PutObjectRetention authorization runs before reading bucket/object state, rejecting unauthorized requests in memory without disk I/O or lock contention; valid retention extensions return immediately at !weakens inside the write lock without repeating IAM evaluation |
5. Core Security Invariants and Upgrade Compatibility
5.1 Three Core Security Invariants
- Invariant 1: Base permission and lock condition keys must hold before any state is read (closes F1, F3, zero-date clear bypass, and existence probing)
- All three Object Lock condition keys on
PutObjectRetentionderive solely from the XML request body and current time. EvaluatingisRetentionActionAllowed(policy.PutObjectRetentionAction, ...)immediately afterParseObjectRetentionrejects unauthorized or policy-denied requests with403 AccessDeniedbefore any bucket or object lookup.
- All three Object Lock condition keys on
- Invariant 2: Bypass is triggered strictly by weakening an unexpired stored lock and requires both header intent and IAM permission (closes F2)
enforceRetentionBypassForPutruns under the object write lock and inspects storedret.Modeandret.RetainUntilDate. If no lock exists, the lock is expired, or!weakens, no bypass is required. Whenweakens == true, storedCOMPLIANCEunconditionally returns400 ObjectLocked, and storedGOVERNANCErequires bothIsObjectLockGovernanceBypassSet(r.Header)andisRetentionActionAllowed(BypassGovernanceRetentionAction, ...).
- Invariant 3: Public writes cannot overwrite an existing UUID version or a locked
nullversion at either entry or commit, failing closed on read errors (closes F4 and companion gaps)- At entry,
putOptsrejects any untrusted targetversionId, andcheckPutObjectLockAllowedfails closed immediately on locked directorynullversions or read errors. - Under the destination write lock (
NewNSLock),checkObjectLockedForOverwritere-verifiesnull-version writes (closing the slow-body TOCTOU window), andCompleteMultipartUploadverifies that any non-emptyfi.VersionIDdoes not already exist on disk (closing legacy uploads, cross-principal completions, and post-recovery UUID collisions).
- At entry,
PutObjectRetention Decision Matrix
retOK denotes s3:PutObjectRetention allowed with the request’s lock condition keys; bypassOK denotes s3:BypassGovernanceRetention allowed with those same keys:
| Stored Version State | Requested Change | Bypass Header | retOK |
bypassOK |
Previous Behavior | Repaired Behavior |
|---|---|---|---|---|---|---|
| Any | Any | Any | No | Any | Often 200 (F1, F3) or 404/400 | Entry 403 AccessDenied |
Unexpired GOVERNANCE |
Extend retention | Any | Yes | Any | 200 | 200 OK |
Unexpired GOVERNANCE |
Shorten / clear / change mode | No | Yes | Any | 400 InvalidRequest (ObjectLocked) |
400 InvalidRequest (ObjectLocked) |
Unexpired GOVERNANCE |
Shorten / clear / change mode | Yes | Yes | No | 200 (F2 bug) | 403 AccessDenied |
Unexpired GOVERNANCE |
Shorten / clear / change mode | Yes | Yes | Yes | 200 | 200 OK |
Unexpired COMPLIANCE |
Shorten / clear / change mode | Any | Yes | Any | 400 InvalidRequest (ObjectLocked) |
400 InvalidRequest (ObjectLocked) |
| Unlocked or expired | Any | Any | Yes | Any | 200 | 200 OK |
Write-Path Invariant Coverage
| Write Path | Option Convergence & Write-Lock Guard | Post-Repair Guarantee |
|---|---|---|
PutObject (regular PUT / bucket replication / site replication / single-object batch replication) |
evaluateReplicationTrust -> putOpts; null versions run checkObjectLockedForOverwrite under write lock |
Public callers (including root) passing ?versionId= receive 403 at putOpts; trusted replication preserves source version IDs |
CopyObject (including metadata copy) |
Source uses srcOpts, destination uses copyDstOpts -> putOpts; null versions run checkObjectLockedForOverwrite under write lock |
Public callers specifying destination ?versionId= receive 403; trusted replication synchronizes target versions |
PostPolicy (browser form upload) |
putOptsFromReq -> putOpts; directory objects checked at preflight and write lock |
Target versionId receives 403; locked directory objects cannot be overwritten |
PutObjectExtract (Snowball / batch archive replication) |
Outer markerExact + per-entry s3:ReplicateObject -> entryTrusted -> putOpts; writeAsArchive sets outer marker |
Untrusted archive entries with minio.versionId receive 403; server-side batch replicate archives preserve version IDs |
NewMultipartUpload / UploadPart / UploadPartCopy |
Initiation passes through putOpts; UploadPartCopy clears replication trust |
Untrusted callers cannot initiate multipart uploads with ?versionId= (403) |
CompleteMultipartUpload |
completeMultipartOpts + write-lock check keyed on fi.VersionID |
In-flight uploads from before versioning was enabled complete normally; non-replication completions targeting any existing UUID version (including legacy, replication-initiated, or recovered uploads) receive 403; null versions run checkObjectLockedForOverwrite |
Directory object PUT / Copy (/{bucket}/dir/) |
Fail-closed preflight checkPutObjectLockAllowed + write-lock checkObjectLockedForOverwrite |
Overwriting a directory null version with unexpired GOVERNANCE, COMPLIANCE, or LegalHoldOn (including locks applied during slow body transfer) returns 400 InvalidRequest (ObjectLocked); read errors abort the write at both preflight and commit |
5.2 Upgrade and Compatibility Side Effects
Normal new-version writes, object reads, and authorized retention extensions are unchanged. Operators should note the following tightened behaviors:
- Policies granting
s3:BypassGovernanceRetentionwithouts3:PutObjectRetentionno longer allow retention writes (SN-2026-015)- Previous behavior: Due to
byPassSet || retSet, a principal with onlys3:BypassGovernanceRetentioncould callPutObjectRetentionby including the bypass header. - New behavior: Returns
403 AccessDenied. Per AWS S3 semantics,s3:BypassGovernanceRetentionis an additional permission when weakening governance locks;s3:PutObjectRetentionis always required.
- Previous behavior: Due to
- Unauthorized
PutObjectRetentionrequests uniformly return403instead of404or400(SN-2026-015)- Previous behavior: Returned
404 NoSuchBucketor400 InvalidBucketObjectLockConfigurationbefore checking policy. - New behavior: Policy evaluation runs before reading bucket state, returning
403 AccessDenied.
- Previous behavior: Returned
- Clearing retention (empty
<Retention/>) evaluatess3:object-lock-remaining-retention-daysas"0"(SN-2026-015)- Previous behavior: Zero-time duration clamping and
math.Absproduced106,752days, allowing clear requests to bypass minimum-retention-day policies. - New behavior: Evaluates to
"0", enforcing minimum-retention-day policies on clear requests.
- Previous behavior: Zero-time duration clamping and
- Non-replication requests (including
root) can no longer overwrite a version in place via?versionId=(SN-2026-016)- Previous behavior: Any caller with
s3:PutObjectcould pass?versionId=<id>onPUT,CopyObject, orNewMultipartUploadto overwrite that version in place. - New behavior: Returns
403 AccessDeniedatputOptsunless the request is authenticated replication.
- Previous behavior: Any caller with
batch replicateto a SILO target requiress3:ReplicateObjecton the target credentials (SN-2026-016)- Previous behavior: A target credential with only
s3:PutObjecthad internal replication headers stripped, yetputOptsstill accepted?versionId=from the query string. - New behavior: Preserving source version IDs requires target replication trust (
s3:ReplicateObject), matching bucket replication requirements.
- Previous behavior: A target credential with only
- Locked directory objects (
dir/) in Object Lock buckets cannot be overwritten until lock expiry (SN-2026-016)- Previous behavior: Directory objects shared the
nullversion and skipped existing-lock checks, allowing repeatedPUT dir/calls to overwrite and reset retention. - New behavior: Returns
400 InvalidRequest(ObjectLocked) while the directorynullversion holds an unexpiredGOVERNANCE/COMPLIANCEretention orLegalHoldOn.
- Previous behavior: Directory objects shared the
- Separately tracked adjacent item (not bundled in this patch)
PostRestoreObjectHandler(archive restore) reading a snapshot before writing back metadata viaCopyObjectis tracked separately as an archive-restore concurrency item.
6. Candidate Design Evaluation and Trade-Offs
During design review, four candidate write-path designs were evaluated against concurrency and recovery reproducers:
| Dimension | Candidate 1: HTTP Entry & Preflight Only (Rejected) |
Candidate 2: Skipping Write-Lock UUID Check via Multipart Origin Marker (Rejected) |
Candidate 3: fe3935476 Full Write-Lock Checks(Without Fail-Closed Preflight) |
Final Adopted Design: fe3935476 Full Write-Lock Checks + Fail-Closed Preflight |
|---|---|---|---|---|
SN-2026-015 Retention Authz (F1 / F2 / F3 / Zero-Date) |
Passes | Passes | Passes | Passes |
Slow-body PUT dir/ concurrent lock race (TestPlanReviewSlowDirectoryPut) |
Fails (P1): paused body stream overwrites committed COMPLIANCE lock |
Passes | Passes | Passes |
Pre-upgrade or replication-initiated uploadId overwrite (TestPlanReviewLegacyMPUCompliance) |
Fails (P1): CompleteMultipartUpload bypasses putOpts and overwrites COMPLIANCE version |
Passes | Passes | Passes |
Recovered multipart upload colliding with replicated COMPLIANCE UUID (TestRecoveredMultipartUploadAfterReplica) |
Fails (P1) | Fails (P1): origin marker only proves how the UUID was generated at initiation, not that the version is still absent at completion | Passes (unconditional fi.VersionID existence check under write lock) |
Passes (unconditional fi.VersionID existence check under write lock) |
Preflight read quorum failure behavior (TestPlanReviewPreflightReadFailure) |
Fails (P1): preflight ignores read error and overwrites object | Passes | Write lock blocks overwrite, but preflight ignores read error and streams body | Passes (preflight immediately fails closed before streaming body) |
CompleteMultipartUpload lookup behavior |
No extra lookup | Skips lookup on generated marker (introduces recovery collision flaw) | Runs getObjectInfo under write lock |
Retains strict getObjectInfo check under write lock; any future performance optimization is deferred for separate validation |
Conclusion
We adopt the fe3935476 full write-lock defense + fail-closed preflight hardening as the final implementation, and incorporate all four regression reproducers (TestPlanReviewSlowDirectoryPut, TestPlanReviewPreflightReadFailure, TestPlanReviewLegacyMPUCompliance, and TestRecoveredMultipartUploadAfterReplica) into cmd/object-retention-overwrite_test.go.




