SILO Security Chronicle

• 1: October Security Update: Object Lock, Admin Credentials and JWK Parsing
• 2: SN-2026-015 & SN-2026-016: S3 Object Lock Governance Retention Bypass and Version-Addressed Overwrites
• 3: September Security Repairs: Payload Integrity, IAM Revocation and Console Sharing
• 4: SN-2026-011: Fix and Release Status
• 5: CVE-2025-62506: Session-Policy Privilege Escalation
• 6: SILO 20260903 Security Notes: SN-2026-006 through 010
• 7: CVE-2026-32285: The jsonparser Advisory That Required No Patch
• 8: CVE-2026-33322: OIDC JWT Algorithm Confusion
• 9: CVE-2026-33419: LDAP STS Enumeration and the Throttling Chain
• 10: CVE-2026-34204: Replication Metadata Injection
• 11: CVE-2026-39414: Oversized S3 Select Records and a SIMD Bypass
• 12: CVE-2026-40344: Snowball Auto-Extract Authentication Bypass
• 13: CVE-2026-41145: Unsigned-Trailer Query Authentication Bypass
• 14: CVE-2026-42600: ReadMultiple Storage-REST Path Traversal
• 15: SN-2026-002: Internode Path Containment Audit — Paying Off What CVE-2026-42600 Left Owing
• 16: SN-2026-003: s3:versionid Condition Evaluation and Request Condition Hardening
• 17: SN-2026-004: Object-Scoped Policy Grants Reaching Bucket Mutations
• 18: SN-2026-018: Client Source Address Trust, Forwarded Headers, and aws:SourceIp
• 19: SN-2026-019: Notification Keyspace Registration and Validation Redaction
• 20: Duplicate Part Numbers in CompleteMultipartUpload

1 - October Security Update: Object Lock, Admin Credentials and JWK Parsing

SILO Server 20261018 delivers the fixes for SN-2026-015, SN-2026-016, and SN-2026-017, scoped STS token revocation, and Go toolchain and dependency updates across silo, mcli, and silo-console. Server 20260916 and 20260903 are affected (and 20261015 was a superseded pre-release draft).

Object Lock: SN-2026-015 and SN-2026-016

The SN-2026-015 & SN-2026-016 advisory and repair design note document the affected PutObjectRetention and in-place version overwrite paths, reproductions, and mitigations. PR #234 (14042b5aa) and PR #235 (0d3ba5d4c) enforce s3:PutObjectRetention before object lookup, require both x-amz-bypass-governance-retention: true and s3:BypassGovernanceRetention when weakening active GOVERNANCE retention, and reject caller-addressed ?versionId= overwrites while enforcing WORM checks under the object write lock for null-version overwrites.

Every cluster and replication site that accepts S3 writes must be upgraded to 20261018.

SN-2026-017: Admin credential authorization boundaries

SN-2026-017 groups the authenticated Admin API handler authorization defects fixed in PR #242 (commit 2616d43fb) and its bulk access-key listing follow-up 995439260.

The actor passed to IAM policy evaluation is always the authenticated access key (cred.AccessKey), while parent resolution (ParentUser) is used only for comparing ownership. Full handler contracts and benchmark measurements are recorded in IAM Admin API Authorization Boundaries.

Compatibility repairs retained alongside the security fix

• Semantic empty policy normalization on issuance: Creating a service account filters reserved parent policy/expiration claims and normalizes semantic empty policy documents (null, {}, {"Statement":null}) to parent inheritance, while preserving explicit empty statement arrays ({"Version":"2012-10-17","Statement":[]}).
• Damaged non-root key cleanup: Deleting or inspecting a damaged non-root service account falls back to stored identity metadata so operators can clean up corrupted or expired keys without allowing a damaged token to mask a root parent.
• Bulk access-key listing (995439260): Applies the same self-service authorization helper to bulk key listing (ListAccessKeysBulk), resolving omitted users, the caller’s own key, or its parent to the caller’s parent inventory while enforcing explicit denies and cross-user restrictions.
• Error codes: Denied self-service operations return 403 AccessDenied; unauthorized attempts to delete another user’s or root’s service account retain their 404 response.

Operator action

1. Verify administrative automation using your production user, service-account, STS, and external-plugin identities. Any script that relied on a session-restricted credential to create further child service accounts, or a non-root admin to manage root service accounts, must switch to an unrestricted parent or root identity.
2. Audit child service accounts and policy updates created prior to upgrading, revoke unexpected keys, and follow the IAM upgrade and recovery guide.

Follow-up: scoped STS revocation

Release commit 0cd388c66 fixes cache-only RevokeTokens enumeration and incomplete propagation across peer nodes and sites. It persists deletion tombstones for tokens durably known to the cluster—scoped by parent identity, token type, and operation cutoff timestamp—and propagates them through peer notification and deletion healing. See Scoped STS Revocation and the Server 20261018 release notes.

Dependency security and login compatibility

Server 20261018 builds with Go 1.27.2, golang.org/x/net v0.61.0, golang.org/x/crypto v0.58.0, golang.org/x/text v0.43.0, silo-pkg v3.14.3, and go-jose v4.1.6 in go.mod:

• go-jose v4.1.6 (GHSA-mgcc-cww5-fp2q): Bounds CPU consumption when parsing crafted RSA JWKs on the OIDC path and rejects RSA JWKs larger than 4096 bits. Verify OIDC discovery, JWKS endpoints, and login flows before upgrading if your identity provider uses oversized RSA keys.
• Standalone Console: See Console v2.4.2 for standalone Console dependency updates.

Verification and scope

Handler regressions in cmd/ cover explicit self denials, session-restricted delegation, root-credential protection, empty-policy issuance, damaged-key cleanup, decrypted bulk-list identity contents, and Object Lock WORM invariants. See the IAM authorization design note, Object Lock repair design note, and component version matrix.

2 - SN-2026-015 & SN-2026-016: S3 Object Lock Governance Retention Bypass and Version-Addressed Overwrites

SN-2026-015 and SN-2026-016 cover two S3 Object Lock retention and immutability vulnerabilities fixed in SILO Server 20261018 via PR #234 (14042b5aa) and PR #235 (0d3ba5d4c), as detailed in the companion repair design note. While the delete path already enforced s3:BypassGovernanceRetention (CVE-2023-25812), the retention-write path (PutObjectRetention) and four in-place version overwrite paths remained bypassable.

Advisory Scope: SN-2026-015 and SN-2026-016

SN-2026-015: PutObjectRetention Authorization and Governance Bypass (#231)

In PutObjectRetentionHandler (cmd/object-handlers.go), three interacting flaws in enforceRetentionBypassForPut and isPutRetentionAllowed broke S3 Object Lock authorization:

• F1 (Unconditional s3:PutObjectRetention bypass via header): Passing x-amz-bypass-governance-retention: true caused isPutRetentionAllowed to check only s3:BypassGovernanceRetention (!byPassSet), allowing an authenticated principal with zero s3:PutObjectRetention and zero s3:BypassGovernanceRetention permissions to place or extend retention on any unlocked object!
• F2 (Governance retention shortening/clearing without s3:BypassGovernanceRetention): Because isPutRetentionAllowed evaluated byPassSet || retSet, a caller who held s3:PutObjectRetention (retSet == true) but lacked s3:BypassGovernanceRetention (byPassSet == false) could pass x-amz-bypass-governance-retention: true to clear (<Retention/>) or shorten an active GOVERNANCE retention lock, and then delete the object (GHSA-67mv-7hc9-wj88, reported by szopi00).
• F3 (s3:object-lock-remaining-retention-days condition bypass on governance bypass): When x-amz-bypass-governance-retention: true was set on an existing governance-locked object, s3:PutObjectRetention was skipped whenever s3:BypassGovernanceRetention succeeded, bypassing IAM policy conditions such as NumericGreaterThanEquals: {"s3:object-lock-remaining-retention-days": 30} attached to s3:PutObjectRetention.

SN-2026-016: Version-Addressed and In-Place Object Overwrites (#234)

While auditing every write path that can mutate an existing object version in place, we found and closed four write-path bypasses (14042b5aa):

1. Caller-supplied ?versionId= on PutObject and CopyObject: Non-replication callers could pass ?versionId=<existing-uuid> on PutObject or CopyObject (or minio.versionId in Snowball PutObjectExtract PAX headers) to overwrite an existing version in place without checking WORM retention or legal hold on that target version.
2. Directory object (key/) null-version overwrite: Creating or overwriting a directory object (strings.HasSuffix(object, "/")) wrote to nullVersionID even in versioned/Object-Lock buckets without checking existing retention or legal hold on the null version.
3. CompleteMultipartUpload in-place null-version overwrite: In an Object-Lock-enabled bucket with versioning suspended (or targeting an existing null version), CompleteMultipartUpload did not enforce retention/legal-hold checks on an existing null version before replacing it.
4. Replication header trust: Unprivileged callers with s3:PutObject could spoof X-Minio-Source-Replication-Request headers unless gated by s3:ReplicateObject.

Expected S3 Object Lock Semantics

Under the AWS S3 Object Lock specification, PutObjectRetention must enforce two independent orthogonal boundaries:

Reproduction and Verification

The three SN-2026-015 failures can be reproduced directly against pre-20261018 builds (verified against baseline b0a540190):

Reproduction 1: Clearing an Active Governance Lock Without s3:BypassGovernanceRetention (F2)

Assume user nobypass has Allow on s3:PutObjectRetention and s3:DeleteObjectVersion, and no s3:BypassGovernanceRetention grant:

Reproduction 2: Locking an Object with Zero Retention Permissions (F1)

Assume user zeroperm has an empty/unrelated policy with neither s3:PutObjectRetention nor s3:BypassGovernanceRetention:

Reproduction 3: Bypassing s3:object-lock-remaining-retention-days Conditions (F3)

Assume user govadmin holds s3:BypassGovernanceRetention unconditionally, and holds s3:PutObjectRetention conditioned on NumericGreaterThanEquals: {"s3:object-lock-remaining-retention-days": 30}. On an object already under GOVERNANCE lock, sending x-amz-bypass-governance-retention: true with a 5-day retention period previously succeeded (byPassSet == true satisfied byPassSet || retSet even though retSet was false), bypassing the 30-day minimum policy condition.

Root Cause Analysis

In cmd/object-handlers.go, PutObjectRetentionHandler authenticated the caller (authenticateRequest) and delegated all authorization to enforceRetentionBypassForPut, which called isPutRetentionAllowed:

// Pre-20261018 isPutRetentionAllowed (broken)
func isPutRetentionAllowed(ctx context.Context, r *http.Request, bucket, object string,
    retDays int, retDate time.Time, retMode RetentionMode, byPassSet bool,
    cred auth.Credentials, owner bool, claims map[string]interface{}) (retSet, byPass bool) {

    if !byPassSet {
        return isReqAuthenticated(ctx, r, iampolicy.PutObjectRetentionAction, ...), false
    }

    byPass = isReqAuthenticated(ctx, r, iampolicy.BypassGovernanceRetentionAction, ...)
    retSet = isReqAuthenticated(ctx, r, iampolicy.PutObjectRetentionAction, ...)
    return retSet, byPass || retSet
}

And in enforceRetentionBypassForPut:

1. When the existing object had no active retention (opts.Mode == ""), the function checked if !retSet && !govBypass and returned nil whenever byPassSet (x-amz-bypass-governance-retention: true) was true—because on the unlocked path it never even called isPutRetentionAllowed properly, or trusted the header boolean directly (F1)!
2. When the existing object was in GOVERNANCE mode and the caller passed x-amz-bypass-governance-retention: true, isPutRetentionAllowed returned byPass || retSet as the second return value (govBypass). Thus retSet == true (s3:PutObjectRetention alone) made govBypass == true (F2), and conversely byPass == true (s3:BypassGovernanceRetention alone) made govBypass == true even when retSet == false (F3).

Upstream History

Tracing git log shows how three separate changes over five years combined to create F1–F3:

• 2020-04 (43a3778b4, minio/minio#9259): Introduced enforceRetentionBypassForPut and isPutRetentionAllowed with the byPass || retSet boolean conflation.
• 2025-02 (437dd4e32, minio/minio#20929): Refactored retention checks in PutObjectRetentionHandler.
• 2025-04 (8c7097528, minio/minio#21103): Replaced the entry checkRequestAuthType call in PutObjectRetentionHandler with authenticateRequest (which only checks SigV4 signature validity, not s3:PutObjectRetention), leaving enforceRetentionBypassForPut as the sole authorization gate and exposing F1 to any authenticated user with zero permissions.

Repair Design

Five Security Invariants

Our repair enforces five explicit invariants across PutObjectRetention (0d3ba5d4c) and object write paths (14042b5aa):

1. Invariant 1 (Mandatory s3:PutObjectRetention): Every PutObjectRetention request must satisfy s3:PutObjectRetention (including all policy conditions such as s3:object-lock-remaining-retention-days, s3:object-lock-retention-mode, and s3:object-lock-retain-until-date), regardless of whether x-amz-bypass-governance-retention is present.
2. Invariant 2 (Additional s3:BypassGovernanceRetention on governance weakening): If the target version has an active (unexpired) GOVERNANCE lock and the request clears retention (Mode == "") or shortens RetainUntilDate (!RetainUntilDate.After(existing.RetainUntilDate)), the request must additionally carry x-amz-bypass-governance-retention: true and satisfy s3:BypassGovernanceRetention (including its s3:ExistingObjectTag/<key> conditions).
3. Invariant 3 (COMPLIANCE immutability): An active (unexpired) COMPLIANCE lock can never be cleared, shortened, or downgraded to GOVERNANCE by any principal.
4. Invariant 4 (No caller-specified version overwrite): Non-replication S3 clients cannot supply ?versionId= on PutObject, CopyObject, or PutObjectExtract to overwrite an existing version ID.
5. Invariant 5 (In-place null-version write-lock enforcement): Any write path that replaces an existing null version in place (PutObject on a suspended-versioning bucket, directory object key/, or CompleteMultipartUpload) must enforce existing WORM retention and legal-hold checks on that null version.

Where Each Check Lives

In PutObjectRetentionHandler and enforceRetentionBypassForPut:

• s3:PutObjectRetention (retOK) is always evaluated against the proposed retention parameters (retDays, retDate, retMode) and existing object tags.
• s3:BypassGovernanceRetention (bypassOK) is evaluated independently and required whenever an active GOVERNANCE lock is cleared or shortened (or when switching an active GOVERNANCE lock to COMPLIANCE with a shorter date).

Complete Decision Table for PutObjectRetention

Five Rejected Alternatives

During design review we evaluated and rejected five simpler-looking patches:

1. Changing only byPass || retSet to byPass && retSet inside isPutRetentionAllowed: Would fix F2 and F3, but would still leave F1 open on unlocked objects (where isPutRetentionAllowed was bypassed) and would wrongly require s3:BypassGovernanceRetention when a user with s3:PutObjectRetention extends a lock while happening to pass x-amz-bypass-governance-retention: true.
2. Adding a checkRequestAuthType(..., PutObjectRetentionAction) call at the top of PutObjectRetentionHandler: At handler entry, the XML body (<Retention>) has not been parsed yet and the existing object’s tags have not been read from disk, so policy conditions on s3:object-lock-remaining-retention-days, s3:object-lock-retain-until-date, s3:object-lock-retention-mode, and s3:ExistingObjectTag/<key> would all fail!
3. Requiring s3:BypassGovernanceRetention whenever x-amz-bypass-governance-retention: true is present on the request: S3 only requires s3:BypassGovernanceRetention when the operation actually needs governance bypass (clearing or shortening an active GOVERNANCE lock), not when locking an unlocked object or extending an existing lock.
4. Blocking <Retention/> empty XML payloads: Clearing an active GOVERNANCE lock via <Retention/> is a documented MinIO/SILO extension used by mcli retention clear; it must remain supported when the caller holds both s3:PutObjectRetention and s3:BypassGovernanceRetention and sends x-amz-bypass-governance-retention: true.
5. Enforcing SN-2026-016 ?versionId= rejection only on Object-Lock-enabled buckets: Allowing arbitrary clients to overwrite historical versions in place via ?versionId= violates S3 versioning immutability on all versioned buckets, not just Object Lock buckets.

Implementation Notes

• ParseObjectRetention and empty <Retention/> (s3:object-lock-remaining-retention-days): When a caller sends <Retention/> to clear governance retention, retDate is the zero time.Time{}. Previously, getRetentionDays computed a negative integer from time.Until(zeroTime) (~-106751 days due to time.Duration 290-year saturation!). Commit 0d3ba5d4c normalizes zero retDate to 0 days and computes future remaining days using non-saturating calendar/seconds arithmetic so dates beyond year 2316 do not overflow time.Duration.
• s3:ExistingObjectTag/<key> propagation: Both the s3:PutObjectRetention and s3:BypassGovernanceRetention checks receive the target version’s stored tags so tag-based IAM policies evaluate accurately.

Compatibility

• Standard S3 and mcli retention workflows: Fully compatible. Callers that clear or shorten governance retention via mcli retention clear --governance must hold both s3:PutObjectRetention and s3:BypassGovernanceRetention.
• Active-active / site replication: Legitimate replication requests that preserve source version IDs must hold s3:ReplicateObject (included in standard replication policies).

Regression Test Matrix

PR #234 and PR #235 lock down the full matrix in cmd/object-handlers-lock_test.go and cmd/bucket-versioning_test.go:

• All 7 rows of the PutObjectRetention decision table (F1 zero-permission caller, F2 PutObjectRetention-only caller, F3 BypassGovernanceRetention-only caller with s3:object-lock-remaining-retention-days conditions),
• Far-future retention dates (2400-01-01T00:00:00Z) exceeding time.Duration’s 290-year limit,
• PutObject, CopyObject, and PutObjectExtract with caller-supplied ?versionId= / minio.versionId,
• Directory object (key/) and CompleteMultipartUpload in-place overwrites against locked null versions.

Mitigation and Upgrade

Delivery Status

Both SN-2026-015 (0d3ba5d4c, PR #235) and SN-2026-016 (14042b5aa, PR #234) are shipped in SILO Server 20261018. Operators relying on S3 Object Lock WORM governance or compliance retention should upgrade to 20261018 immediately.

3 - September Security Repairs: Payload Integrity, IAM Revocation and Console Sharing

This bulletin covers three security repairs (SN-2026-012 through SN-2026-014) and adjacent dependency updates shipped in Server 20260916 and Console v2.4.1. See the security advisory ledger for the complete index.

Presigned payload integrity — SN-2026-012

The generic authenticated handler path could verify a presigned request using a payload digest supplied via the X-Amz-Content-Sha256 header without verifying the actual request body bytes against that digest. A caller holding a presigned write URL could therefore substitute a different request body (the regression suite reproduces this against presigned PutBucketPolicy).

Commit c4b5e1cb4 in PR #177 binds body digest verification to the effective signed payload hash; any mismatch fails with XAmzContentSHA256Mismatch (400 Bad Request). An explicitly signed UNSIGNED-PAYLOAD token retains its standard meaning. The same PR also ensures IAM policy condition evaluation reads the authenticated signed values (with signed query parameters taking precedence over fallback headers) and removes the internal X-Amz-Signature-Age scratch header, complementing SN-2026-011.

Before upgrading, verify that custom SigV4 presigning clients send body payloads matching any signed X-Amz-Content-Sha256 digest.

Durable IAM revocation — SN-2026-013

Previously, delayed site-replication events, parent account recreation, or missing deletion history could resurrect identities or policy grants that an administrator had already deleted or revoked. PR #191 and PR #192 persist monotonic source revisions, deletion tombstones, and parent-revocation epochs embedded in child credentials so stale replication events and pre-revocation child tokens are rejected.

Because this change introduces persistent IAM revision and tombstone records:

• Upgrade all nodes sharing an IAM backend and all participating replication sites together; mixed-version operation and rolling downgrades are not supported.
• mcli admin cluster iam export exports active records only and omits deletion tombstones; maintain a full backend snapshot and follow the IAM upgrade and recovery guide.
• See the IAM revocation design note for revision ordering, metrics, and group-membership scope.

Anonymous Console share proxy — SN-2026-014

Jiri Pejchal reported that the unauthenticated Console shared-download proxy accepted URLs pointing to the configured Server origin outside the intended presigned object-download surface.

Console PR #56 restricts scheme, host, and port to the configured S3 origin, requires a valid /{bucket}/{object} path, rejects reserved system prefixes, path traversal segments, and operation-changing query parameters, disables upstream HTTP redirects, and forwards caller context cancellation while preserving the exact signed URL bytes for valid object downloads.

Upgrade standalone Console deployments to Console v2.4.1, and upgrade SILO Server deployments (which embed Console via Server PR #209) to Server 20260916.

Adjacent dependency hardening

Server 20260916 upgrades github.com/rabbitmq/amqp091-go to v1.14.0 to address GHSA-6c5v-hqjr-5xxp (fixed upstream in v1.13.0), where a malicious AMQP broker could trigger excessive memory allocation when an AMQP bucket notification target is configured. It also adopts silo-pkg v3.14.1 with the updated JWX JSON encoding dependency.

4 - SN-2026-011: Fix and Release Status

SN-2026-011 was fixed on main starting with 123325430 on 2026-09-13 and shipped in Server 20260916. Server RELEASE.2026-09-03T13-18-01Z and earlier releases are affected (silo-pkg v3.14.0 and mcli 20260913 are client/library releases and do not patch existing Server deployments).

A holder of a signed PUT request could append an unsigned x-amz-copy-source header and turn the authorized upload into a server-side CopyObject of another object readable by the signing key. If the destination permits anonymous reads, the copied object’s contents become readable without holding the signing credentials. Both presigned URLs and Authorization-header requests were affected.

The fix verifies received x-amz-* headers against the request’s signed-header set (retaining only explicit protocol exceptions) before dispatching the operation. Follow-up request-signing and checksum coverage is documented in the signed-header design note.

Remediation

Upgrade SILO Server to Server 20260916 or later. Until all Server nodes are upgraded, scope write-signing credentials to the minimum required object prefixes and avoid granting broad read permissions to principals that issue presigned uploads into publicly readable buckets.

Reported by Oren Yomtov. See the security advisory ledger, the September 16 release hardening summary, and the component version matrix.

5 - CVE-2025-62506: Session-Policy Privilege Escalation

Status: Inherited and released
First SILO community release: RELEASE.2025-12-03T12-00-00Z
Upstream fixed release: RELEASE.2025-10-15T17-29-55Z
GitHub advisory: GHSA-jjjj-jwhf-8rgr
Upstream fix: minio/minio#21642

A service account or STS account constrained by a restricted session policy could invoke an “own account” Admin API operation to create a child service account without the session-policy restriction. Because the policy check set DenyOnly = true for own-account operations, any session policy that lacked an explicit Deny rule allowed the call, and the newly created child service account inherited the broader permissions of the parent user.

SILO did not require a separate backport. The upstream fix and its two regression test groups were already present in the Git history from which the first SILO community release (RELEASE.2025-12-03T12-00-00Z) was cut.

Commit mapping

The identical commit SHA establishes direct source inheritance from upstream history. The Security Advisories ledger indexes the same commit mapping.

Regression evidence

Upstream PR #21642 added two regression test groups, each exercising both root and non-root parent accounts:

• TestServiceAccountPrivilegeEscalationBug2_2025_10_15 (cmd/admin-handlers-users_test.go) verifies that restricted service accounts cannot escalate privileges by creating unrestricted child service accounts.
• TestSTSPrivilegeEscalationBug2_2025_10_15 (cmd/sts-handlers_test.go) verifies the same boundary for restricted STS temporary credentials.

Both test groups run as part of the standard go test ./cmd suite.

Fix and operator boundary

Whenever a session policy (sub-policy) is attached to the caller’s credential, the policy evaluator clears the DenyOnly flag before evaluating the session policy. An own-account operation can no longer pass simply because the session policy lacks an explicit Deny; the restricted session policy must explicitly grant Allow for the action.

Every SILO release contains c1a49490. Operators migrating from an upstream MinIO build older than RELEASE.2025-10-15T17-29-55Z should upgrade to SILO, audit service accounts created by restricted service or STS identities prior to the upgrade, and revoke any unauthorized child accounts.

Related advisory: CVE-2026-32285

Several vulnerability feeds later published CVE-2026-32285 with text describing github.com/buger/jsonparser while pointing to commit c1a49490. For the dependency reachability investigation and why no code change was made for that advisory, see CVE-2026-32285: The jsonparser Advisory That Required No Patch.

6 - SILO 20260903 Security Notes: SN-2026-006 through 010

This bulletin covers five security findings fixed in SILO Server 20260903 (RELEASE.2026-09-03T13-18-01Z), spanning customer-provided encryption keys (SSE-C), client-supplied replication headers, administrative status transitions, and explicit object-version deletion. SN-2026-006 through SN-2026-010 are SILO security-note identifiers grouping the pre-release audit findings inherited from the upstream minio/minio codebase.

Summary

SN-2026-006: zero-byte SSE-C key authentication

Affected operations: GetObject, HeadObject, CopyObject source processing, and GetObjectAttributes for zero-byte objects encrypted with SSE-C.
Tracking: issue #82.
Fixes: b73581b05 and c4fd97d0b.

SSE-C stores a sealed object key and requires the caller to supply the matching customer key on reads. Normal reads authenticate that key while preparing the encrypted payload stream. Because a zero-byte object has no payload block, the legacy fast path returned before unsealing the stored object key.

As a result, an authenticated caller with read access to a zero-byte SSE-C object could receive a successful response while supplying an incorrect customer key, or copy the object under a newly chosen key without proving knowledge of the original key.

The fix authenticates the stored SSE-C key regardless of payload length. Valid keys succeed as before; wrong keys fail with 403 AccessDenied, including null-version and key-rotation paths.

SN-2026-007: SSE-C GetObjectAttributes

Affected operation: GetObjectAttributes on SSE-C objects.
Tracking: issue #84.
Fixes: 474cd5801, 74c97d005, and 21870fa2e.

GetObjectAttributes exposes object size, ETag, checksums, storage class, and multipart part metadata. The inherited handler returned those attributes without authenticating the supplied SSE-C key, and also treated the raw X-Minio-Source-Replication-Request header as permission to skip key verification.

The fix enforces two distinct rules:

• an ordinary S3 caller must supply the valid SSE-C customer key (wrong or missing keys return 403 AccessDenied);
• a replication peer may use the replica metadata path only after passing authentication and s3:ReplicateObject authorization.

SN-2026-008: replication headers are not authority

Affected surface: object reads and writes, multipart completion, object and version deletion, Snowball extraction, Object Lock timestamps, checksum metadata, and bucket events.
Tracking: PR #101.
Primary fix range: 938603458 through 04b097fd9, followed by Snowball and rule-prefix hardening.

While CVE-2026-34204 blocked ordinary PUT and COPY requests from importing a subset of replication SSE metadata, a wider audit showed that several downstream handlers still treated the presence of internal replication headers as proof of a trusted replication request.

Depending on the operation, a caller holding ordinary read or write permissions could:

• trigger an SSE-C no-decryption read path without supplying the customer key;
• preserve a client-supplied source ETag or MTime;
• inject source checksum, actual-size, retention, legal-hold, or REPLICA status metadata;
• invoke replication-only delete semantics;
• suppress bucket notification events;
• carry replication trust across Snowball archive entries.

The fix establishes a single receiver-wide trust calculation:

1. authenticate the incoming request in its original signed form;
2. require a single X-Minio-Source-Replication-Request header with the exact lowercase value true;
3. authorize s3:ReplicateObject or s3:ReplicateDelete on the target resource;
4. grant replicaTrusted only when the authenticated replication status also equals REPLICA;
5. bind the trust decision to a private req.Context() key;
6. strip untrusted internal replication headers on a post-authentication request clone while preserving the trailer map for streaming checksum verification.

See No I/O Before Auth, No Privilege From Headers for the full protocol matrix, CORS interaction, black-box two-site replication verification, and rejected alternatives.

SN-2026-009: user and group status authorization

Affected operations: admin SetUserStatus and SetGroupStatus.
Tracking: PR #73 / PR #85.
Fixes: 58735ee38 and 229fe2b3c.

The inherited handlers checked admin:EnableUser or admin:EnableGroup before inspecting the requested status parameter. A delegated administrator permitted only to enable accounts could therefore disable them as well, while an administrator granted only admin:DisableUser or admin:DisableGroup was rejected.

The requested target state now selects the required admin action:

admin:* and the built-in consoleAdmin policy remain unchanged; only custom least-privilege policies that relied on EnableUser/EnableGroup to perform both transitions require updating.

SN-2026-010: explicit object-version deletion

Affected operations: DeleteObject and each DeleteObjects entry carrying an explicit versionId.
Tracking: issue #58 and PR #104.
Fix range: 75a6734e4 through d2d47a41f.

Previously, the S3 delete handlers authorized explicit-version deletion under s3:DeleteObject and checked s3:DeleteObjectVersion only as a secondary deny filter, allowing callers with s3:DeleteObject alone to permanently delete historical versions.

Server 20260903 requires s3:DeleteObjectVersion whenever a versionId is specified. Unversioned DELETE requests continue to check s3:DeleteObject, and replication targets continue to check s3:ReplicateDelete. Callers granted only s3:DeleteObject can no longer delete explicit versions, and policies that pair Allow s3:* with Deny s3:DeleteObject to prevent permanent deletion must also deny s3:DeleteObjectVersion.

Operator actions

Before upgrading:

1. identify policies and applications that delete explicit object versions and grant or deny s3:DeleteObjectVersion explicitly;
2. inspect custom admin policies for user and group enable/disable operations;
3. confirm that replication service accounts hold s3:ReplicateObject / s3:ReplicateDelete and that ordinary application identities do not;
4. verify that SSE-C clients send the matching customer key on GET, HEAD, GetObjectAttributes, and CopyObject source reads, including zero-byte objects.

After upgrading:

• verify both wrong-key (403 AccessDenied) and valid-key responses on a test SSE-C object;
• test object and delete replication across paired sites;
• test delegated user/group enable and disable permissions independently;
• verify explicit-version deletion with principals holding s3:DeleteObject alone vs. s3:DeleteObjectVersion.

Verification boundary

All five fixes are covered by unit, handler, authorization, streaming-trailer, Snowball, and site-replication regression tests. Full local acceptance on ebac0ca73 passed the cmd race suite and 174 test cases across six deployment topologies (174 PASS / 0 FAIL), followed by the RELEASE.2026-09-03T13-18-01Z release gates documented in the pre-release review.

Residual risks outside this bulletin

• Conditional deletion (If-Match on DeleteObject and per-object ETag on DeleteObjects, #10) and bucket-metadata deletion convergence across site-replication peers (#77) are tracked separately; see the Server 20260903 release notes and Server 20260916 release notes.

7 - CVE-2026-32285: The jsonparser Advisory That Required No Patch

Status: Closed without a code change
GitHub issue: pgsty/silo#26
Referenced upstream commit: c1a49490 (CVE-2025-62506)

The initial assessment of CVE-2026-32285 assumed that the repository might still carry a vulnerable version of github.com/buger/jsonparser, and the team briefly considered replacing the package or maintaining a local fork. Checking the advisory’s referenced upstream commit, the resolved Go module version, and actual symbol reachability changed the conclusion:

1. Public vulnerability feeds mislabeled CVE-2026-32285 as a jsonparser issue while pointing to upstream commit c1a49490, which is the session-policy privilege-escalation fix for CVE-2025-62506 already included in every SILO release since RELEASE.2025-12-03T12-00-00Z.
2. For github.com/buger/jsonparser itself, the repository already resolved v1.1.2, which contains the upstream jsonparser fixes, and govulncheck found no reachable vulnerable symbol.

Instead of manufacturing a cosmetic dependency bump, we recorded the verification evidence and closed pgsty/silo#26.

What was wrong with the initial assumption

The issue was initially framed around the premise that jsonparser had no fixed upstream version in the tree. Acting on that premise without verifying the module graph and commit provenance would have caused real engineering harm:

• altering the dependency graph for zero security benefit;
• risking subtle behavioral regressions in JSON parsing paths for a nonexistent vulnerability;
• taking on long-term maintenance of an unnecessary dependency fork;
• misleading operators into believing that prior SILO releases were exposed when they were not.

Leaving already-remediated code unchanged is a deliberate security decision when backed by version and reachability evidence.

Investigation

The investigation verified four distinct layers:

1. Commit provenance: Checked the commit linked by the CVE feed (c1a49490) and confirmed it is the inherited session-policy fix (CVE-2025-62506).
2. Resolved module version: Verified go.mod and go.sum to confirm that github.com/buger/jsonparser resolves to v1.1.2, which already includes the upstream jsonparser patch.
3. Symbol reachability: Ran govulncheck across the tree and confirmed that no vulnerable symbol is reachable.
4. Feed discrepancy attribution: Traced the remaining alarm to mismatched metadata in external vulnerability feeds rather than unpatched code in the SILO tree.

When evaluating any dependency advisory, four questions must be kept separate: whether a package version was once listed as affected, whether the module is imported, whether a vulnerable symbol is reachable in the compiled call graph, and whether an external entry point can trigger that path with untrusted input.

Why there was no “just in case” upgrade

When the selected version in go.mod already contains the fix, bumping to an arbitrary commit or swapping libraries does not improve security. In a large Go module graph, unmotivated dependency churn expands the regression surface and obscures the cause of later behavioral changes.

We therefore took four concrete steps:

• closed the issue without a synthetic commit;
• recorded the commit provenance, v1.1.2 selection, and govulncheck output in pgsty/silo#26;
• retained automated govulncheck and go.mod version gates in CI to prevent accidental dependency downgrades;
• indexed CVE-2026-32285 alongside CVE-2025-62506 in the Security Advisories ledger so operators auditing scanner reports can find an authoritative answer.

Verification boundary

The investigation on 2026-04-15 established that SILO’s module graph (github.com/buger/jsonparser v1.1.2) and commit history (c1a49490) required no source change for CVE-2026-32285. Any future change to module selection or jsonparser imports remains subject to the standard CI govulncheck gate.

Engineering principle

For dependency CVEs, verify in order: what commit or version the advisory actually points to, which version go.mod selects, whether any vulnerable symbol enters the call graph, and whether a runtime entry point exposes it. Only when those checks reveal an unpatched or reachable flaw should the investigation produce a code diff.

8 - CVE-2026-33322: OIDC JWT Algorithm Confusion

Status: Released
First containing release: RELEASE.2026-04-17T00-00-00Z
Affected entry points: AssumeRoleWithWebIdentity, AssumeRoleWithClientGrants
Fix commit: d24f449
GitHub issue: pgsty/silo#22

The previous OIDC implementation in internal/config/identity/openid placed the relying-party client_secret into the JWT verification keyring while also accepting HMAC signing algorithms (HS256, HS384, HS512). An attacker who obtained that client_secret could therefore sign a forged ID token with HMAC and exchange it through STS (AssumeRoleWithWebIdentity or AssumeRoleWithClientGrants) for temporary S3 credentials.

The final fix (d24f449) enforces asymmetric, JWKS-only verification and deliberately drops HS256/HS384/HS512 token support rather than keeping a configuration toggle that would reintroduce ambiguous trust semantics.

Why the vulnerability was key-role confusion, not missing signature verification

The vulnerable code did verify JWT signatures before issuing STS credentials. The failure lay in which keys the verifier accepted and whether the token’s alg header could select a signing algorithm whose trust semantics did not match the key’s intended role.

In OIDC, the client_secret authenticates the relying party (SILO) to the identity provider’s Token Endpoint. It is a shared credential between the relying party and the IdP—not an asymmetric signing key used by the IdP to assert user identities. By loading client_secret into the local token.Parse keyring alongside JWKS public keys, the verifier allowed a single credential leak to escalate into full identity forgery:

1. The attacker obtained the configured OIDC client_secret.
2. The attacker constructed a JWT claiming arbitrary user or role attributes and signed it with HS256, HS384, or HS512 using the client_secret.
3. The verifier matched the HMAC algorithm against the client_secret in its keyring and accepted the signature.
4. The forged token passed AssumeRoleWithWebIdentity or AssumeRoleWithClientGrants and returned valid temporary STS credentials.

A compatibility path was implemented, then deliberately removed

During development, we first implemented an explicit algorithm-allowlist and allow_hmac-style compatibility prototype (fc47fa0). On the surface, an opt-in flag appeared reasonable: enforce asymmetric verification by default while letting deployments with HMAC-based tokens opt in explicitly.

Design review overturned that approach. Keeping client_secret anywhere inside the local JWT verification keyring meant that enabling the option would silently widen the entire STS trust boundary: anyone holding the relying-party secret could again mint arbitrary user tokens. Furthermore, any future drift in the parser’s algorithm allowlist or JWKS retry path could reopen the vulnerability by default.

Removing the completed compatibility implementation produced a much simpler and more auditable security boundary.

Final invariants

Commit d24f449 establishes four invariants in internal/config/identity/openid:

• Verifier keys are loaded exclusively from the identity provider’s JWKS endpoint (jwks_uri).
• The OIDC client_secret never enters the local JWT verification keyring.
• HS256, HS384, and HS512 tokens are rejected unconditionally.
• Both the initial token verification pass and the JWKS cache-refresh/retry path use the same asymmetric algorithm allowlist (RS256, RS384, RS512, ES256, ES384, ES512).

We also kept the algorithm scope tight rather than expanding JOSE support during a security fix: PS256 and EdDSA remain unsupported.

Verification and release

The test suite in internal/config/identity/openid covers HS256 token rejection, RS256 token acceptance, and JWKS refresh/retry behavior when a key rotation occurs. All temporary compatibility helpers, configuration flags, and HMAC opt-in tests from the initial prototype were stripped from the final commit (d24f449), which shipped in RELEASE.2026-04-17T00-00-00Z.

go test ./internal/config/identity/openid -count=1

Compatibility trade-offs

This fix introduces an intentional breaking change. Identity providers configured to issue HS256, HS384, or HS512 ID tokens must migrate to JWKS-backed RSA or ECDSA signing keys before upgrading SILO. If you suspect the OIDC client_secret was exposed prior to upgrading, rotate the secret in both the identity provider and SILO configuration.

9 - CVE-2026-33419: LDAP STS Enumeration and the Throttling Chain

Status: Released, followed by two rounds of hardening
First containing release: RELEASE.2026-04-17T00-00-00Z
Complete correction: RELEASE.2026-06-18T00-00-00Z
Fix commits: 3b950f8, 18b712d, 9e10f6d, f441108, 5e40665
GitHub issue: pgsty/silo#23

The original defect in AssumeRoleWithLDAPIdentity was straightforward: LDAP STS returned distinct errors for “user does not exist” and “password is wrong,” creating an unauthenticated username oracle. Our first patch unified the external authentication failure response and added rate-limit buckets keyed by both source IP and username.

Subsequent design reviews uncovered a deeper problem: naive success refunds, spoofable proxy headers, reservation accounting, and the shared username bucket itself turned the rate limiter into a targeted denial-of-service primitive. The final June design (5e40665) removed the shared username bucket that enabled cross-origin account lockouts, retained the per-source-IP bucket, and formalized proxy IP attribution as an explicit deployment contract.

Initial threat model

The vulnerable entry point is AssumeRoleWithLDAPIdentity. An attacker requires no existing S3 or LDAP credentials; network access to the STS endpoint is sufficient to compare the HTTP status code and error message returned for an unknown username versus a wrong password, enumerate valid directory accounts, and follow up with password spraying.

At the same time, remediation could not flatten every LDAP error into “invalid credentials.” Directory connection failures, lookup-bind credential errors, and LDAP server timeouts must still surface as HTTP 500 infrastructure errors so operators can detect and diagnose directory outages.

First round: uniform responses and a dual-key limiter

The initial patch on 2026-04-15 (3b950f8) introduced three changes:

• unknown-user lookups and bad-password binds both returned the same external 400 InvalidParameterValue STS authentication error;
• LDAP infrastructure failures continued to return HTTP 500, logging the underlying directory error on the server;
• a new in-memory limiter throttled authentication attempts using buckets for both client source IP and normalized LDAPUsername.

While this closed the response-content oracle and raised the cost of brute-force guessing, reviewing the limiter’s state machine and IP extraction immediately exposed flaws in how tokens were consumed and attributed.

Second round: success refunds, proxy attribution, and token accounting

Follow-up commits on April 16 (18b712d, 9e10f6d, f441108) fixed three defects in the initial limiter:

1. Explicit reserve/commit/cancel/refund lifecycle: Successful authentication and LDAP infrastructure errors must refund their reserved token so legitimate high-frequency logins do not exhaust the failure allowance.
2. Untrusted proxy headers: The limiter must default to the TCP socket peer (RemoteAddr) and ignore client-supplied X-Forwarded-For, X-Real-IP, and Forwarded headers unless the socket peer matches an explicit trusted-proxy allowlist.
3. Bounded token refunds: Cancellation and refund paths must clamp token counts to bucket capacity so concurrent cancellations cannot mint extra capacity.

Third round: why we deleted the username bucket

A second design review in June overturned our initial assumption that “throttling by source IP plus username is strictly stronger than throttling by source IP alone.”

Because the per-username bucket was shared globally across all client origins on a node, an attacker who knew a victim’s LDAPUsername could exhaust that user’s bucket from any IP address (or across rotating IPs). With only a handful of failed requests per minute, an attacker could keep a targeted user’s AssumeRoleWithLDAPIdentity calls permanently throttled before the legitimate user’s request ever reached an LDAP bind.

Commit 5e40665 (shipped in RELEASE.2026-06-18T00-00-00Z) therefore made five adjustments:

• deleted the shared per-username bucket so one origin cannot lock out a specific LDAPUsername for other origins;
• peeled trusted hops from X-Forwarded-For right-to-left and selected the first untrusted client address;
• rejected catch-all trusted-proxy CIDRs (0.0.0.0/0 and ::/0) that would allow any client to spoof X-Forwarded-For;
• dropped RFC 7239 Forwarded from security-sensitive rate-limit bucketing;
• accepted X-Real-IP only when the immediate peer is a configured trusted proxy that overwrites the header.

This proxy trust model was subsequently generalized across all S3 and Admin handlers as SN-2026-018 (MINIO_API_TRUSTED_PROXIES).

Rejected alternatives

Verification and release

The unit test suite verifies the limiter’s reserve/commit/cancel/refund state machine, concurrent access, token refunds on successful binds and LDAP infrastructure errors, external response equivalence between unknown users and bad passwords, and source-IP extraction across direct RemoteAddr, spoofed headers, single- and multi-hop trusted proxies, and rejected catch-all CIDRs (0.0.0.0/0, ::/0).

Note that the end-to-end LDAP security test skips live directory binds when _MINIO_LDAP_TEST_SERVER is unset; full end-to-end verification requires pointing _MINIO_LDAP_TEST_SERVER at a live LDAP instance in addition to running the package unit tests.

Final trade-offs and operational boundaries

• Source-IP-only throttling: Removing the username bucket prevents targeted cross-IP account lockouts, at the cost of no longer enforcing a single-account failure cap across distributed botnet origins.
• Per-node memory state: The limiter is per-node and in-memory rather than a cluster-wide password defense; distributed spraying and LDAP bind timing differences still require directory-side policies or edge rate limiting.
• Proxy configuration: Deployments behind a reverse proxy must configure trusted proxy CIDRs (MINIO_API_TRUSTED_PROXIES) so legitimate clients behind the proxy are bucketed by their client IP rather than the proxy’s socket address.
• Uniform responses are the primary defense: Rate limiting slows down guessing from a single source, while the unified 400 InvalidParameterValue response is what closes the username enumeration oracle.

10 - CVE-2026-34204: Replication Metadata Injection

Status: Released
First containing release: RELEASE.2026-04-17T00-00-00Z
Fix commit: 56fa63b
GitHub issue: pgsty/silo#24

Before 56fa63b, ordinary PutObject and CopyObject requests could smuggle client-supplied X-Minio-Replication-* headers into internal X-Minio-Internal-* SSE and replication metadata. That mismatch between the stored metadata and the actual write path could leave the resulting object unreadable.

The fix stops accepting replication-only metadata in the default extraction path, restores those fields solely inside flows explicitly authorized for ReplicateObjectAction (s3:ReplicateObject), and sanitizes CopyObject request headers before any downstream header consumer runs.

Threat model

Exploiting this flaw required only ordinary object-write permission (s3:PutObject), not internode or replication credentials. Although the input arrived in client-controlled X-Minio-Replication-* HTTP headers, metadata extraction converted those headers into internal replication and SSE state without checking whether the caller was authorized to replicate objects.

When a subsequent GetObject or HeadObject request read the object, the server interpreted the stored object according to that forged internal state and failed to read or decrypt it. Because any authenticated writer could trigger this integrity and availability failure, every server accepting untrusted S3 writes was affected.

The underlying flaw was that client-controlled headers acquired privileged internal semantics before passing a replication authorization check.

Why we chose surgical stripping over rejecting the whole request

The most obvious fix would have been to reject any non-replication request carrying an X-Minio-Replication-* header. However, returning 400 or 403 whenever an extraneous header appears would turn previously ignored headers into hard client failures.

Instead, we designed a strip-by-default, restore-after-auth model:

• the default metadata extraction helper in cmd/handler-utils.go never accepts replication-only metadata;
• ordinary PutObject and CopyObject requests strip those fields before building object metadata;
• only code paths that have passed ReplicateObjectAction authorization explicitly restore replication-only metadata;
• replica-status writes require the same ReplicateObjectAction authorization;
• legitimate multipart and Snowball replication flows explicitly restore the SSE metadata they require after authorization.

This design closes the injection path while keeping wire behavior unchanged for ordinary clients that happen to pass unrecognized headers.

Why CopyObject had to sanitize headers before cleanMetadata

In CopyObject, request headers are not used solely to construct the final destination metadata map. Earlier stages in the handler inspect request headers for precondition evaluation and SSE-C source handling.

If untrusted X-Minio-Replication-* headers were stripped only inside cleanMetadata right before writing the destination object, those earlier consumers would already have acted on the forged headers. Moving header sanitization to the very beginning of CopyObject—before precondition and SSE-C helpers run—enforces a single invariant across the entire handler: untrusted replication headers never enter internal request semantics.

Implementation and verification

Commit 56fa63b updated cmd/handler-utils.go, object handlers, and multipart handlers, backed by regression tests across multiple layers:

• helper-level unit tests for trusted versus untrusted metadata extraction;
• handler-level tests for malicious PutObject and CopyObject requests carrying X-Minio-Replication-* headers;
• CopyObject early-sanitization tests verifying that precondition and SSE-C paths do not observe stripped headers;
• end-to-end verification confirming that forged replication headers no longer produce unreadable objects, while authorized replication, multipart, and Snowball replication flows continue to succeed.

go test ./cmd -run 'Test.*Replication.*Metadata|Test.*CopyObject.*' -count=1

Trade-offs and subsequent hardening

• Default stripping: Internal replication headers sent by ordinary S3 clients are silently stripped rather than honored.
• Fail-safe default for new endpoints: Because replication-only metadata must be explicitly restored inside an authorized branch, forgetting to call the restore helper in a future replication path causes a functional test failure rather than silently reopening an untrusted injection path.
• Follow-up receiver-wide hardening:
  • SN-2026-008 (RELEASE.2026-09-03T13-18-01Z) and the CORS and replication trust design record generalized this pattern into a single post-authentication trust boundary across all internal replication headers (X-Minio-Source-Etag, X-Minio-Source-Mtime, X-Minio-Source-Replication-Request, replication SSE key headers, and X-Amz-Bucket-Replication-Status), covering SSE-C reads, source timestamps, multipart completion, object lock, deletes, and events.
  • Replica Metadata Normalization (RELEASE.2026-09-16T00-00-00Z, PR #194) subsequently ensured that even authorized replication CopyObject requests normalize transport-only aws-chunked Content-Encoding values and strip GHSA-redacted metadata keys rather than re-injecting raw request headers into stored objects.

11 - CVE-2026-39414: Oversized S3 Select Records and a SIMD Bypass

Status: Released; second-round fix completed in June
Initial fix release: RELEASE.2026-04-17T00-00-00Z
Complete fix release: RELEASE.2026-06-18T00-00-00Z
Fix commits: 3252d5b, 5dc705936 (fd69c89)
GitHub issue: pgsty/silo#25

Our initial April patch (3252d5b) reused the existing 1 MiB maxCharsPerRecord limit for both CSV and standard JSON Lines in S3 Select, stopping unbounded buffering while waiting for a record delimiter and returning the S3 OverMaxRecordSize error code to clients.

A follow-up call-chain audit in June uncovered a hardware-dependent gap: on x86_64 CPUs with AVX2/AVX-512 support where simdjson.SupportedCPU() returned true, JSON Lines took a separate simdjson fast path that bypassed the bounded json.PReader completely. The June fix (5dc705936 / fd69c89) routed JSON Lines through the bounded reader on every CPU architecture, while also correcting streaming error codes, parser error wrapping, and pre-error output queue flushing.

Threat model

An authenticated caller with s3:GetObject permission can issue an S3 Select (SelectObjectContent) query against an object containing an arbitrarily long single record without a newline delimiter. If the record reader buffers bytes without an upper bound until it encounters a delimiter, a single query can exhaust server memory and CPU.

Hardware-dependent dispatch makes this class of bug harder to catch: the same S3 Select query can execute different parser implementations depending on the host CPU’s instruction-set flags. Passing a test on a non-SIMD machine does not guarantee that production hosts with AVX2/AVX-512 are protected.

First round: reusing the existing 1 MiB invariant

Rather than introducing a new configuration knob, the April patch (3252d5b) applied the existing maxCharsPerRecord = 1 MiB (1048576 bytes) constant across CSV and standard line-delimited JSON:

• the CSV splitter and line-delimited JSON reader rejected records exceeding 1 MiB before buffering or parsing them further;
• the reader preserved the earliest splitter error instead of letting a subsequent partial decode overwrite it;
• the error propagated to the client as OverMaxRecordSize rather than collapsing into a generic InternalError so clients and monitoring can distinguish an enforced record-size limit from a server fault.

Second round: the hardware-dependent SIMD bypass

Tracing the full S3 Select JSON dispatch chain in June revealed a second path for JSON Lines (DOCUMENT type LINES):

JSON Lines -> simdj.NewReader -> simdjson.ParseNDStream

Whenever simdjson.SupportedCPU() returned true, JSON Lines bypassed the bounded json.PReader and called simdj.NewReader. Inside simdjson.ParseNDStream, the third-party parser continued reading across chunk boundaries until it found a newline. Wrapping the underlying io.Reader in a byte-counting adapter could not solve the problem cleanly: because ParseNDStream reads ahead in chunks, a generic wrapper cannot simultaneously preserve already-completed records in the buffer and enforce an exact 1 MiB cap on the next record.

Rather than layering a brittle byte-counting wrapper around simdjson.ParseNDStream, commits 5dc705936 and fd69c89 disabled the SIMD NDJSON path and routed JSON Lines through the bounded json.PReader on all CPUs. If a SIMD fast path is reintroduced in the future, the SIMD parser itself must enforce the 1 MiB per-record limit and pass the same CPU-independent regression suite.

Stream semantics corrected in the same round

Reviewing the S3 Select streaming pipeline during the June fix also resolved four adjacent protocol issues:

• Interface-based error matching: Used errors.As to pass through any error implementing SelectError, rather than matching only a single concrete struct type.
• Typed JSON parser errors: Wrapped syntax failures in the JSON worker as JSONParsingError instead of returning untyped errors.
• Pre-error batch flushing: Flushed completed records still waiting below the batch threshold in the output queue before emitting a terminal error event, matching S3 Select streaming semantics.
• Input-order error priority: Preserved the first error encountered in input order so a later oversized record cannot overwrite an earlier parse failure on a preceding line.

Deliberately kept out of scope

To keep the security patch focused and avoid speculative behavior changes, three adjacent topics were left out of scope:

• the historical interaction between CSV AllowQuotedRecordDelimiter and the outer physical-newline splitter;
• whether \r in CRLF counts toward the 1 MiB record length limit;
• re-implementing a bounded SIMD NDJSON parser before having a design that enforces record boundaries inside simdjson.

Verification and release

The regression suite in internal/s3select tests oversized CSV and JSON Lines records, OverMaxRecordSize and JSONParsingError propagation, pre-error record flushing, and parser behavior across both SIMD and non-SIMD code paths.

go test ./internal/s3select/... -count=1

The initial fix shipped in RELEASE.2026-04-17T00-00-00Z (3252d5b), and the complete SIMD-independent fix shipped in RELEASE.2026-06-18T00-00-00Z (5dc705936 / fd69c89).

Final trade-offs

• Throughput vs. predictable bounds: Routing JSON Lines through json.PReader gives up SIMD NDJSON acceleration on AVX2/AVX-512 hosts in exchange for a strict 1 MiB per-record memory bound on every CPU.
• 1 MiB record ceiling: S3 Select queries against CSV or JSON Lines objects with individual records larger than 1 MiB are rejected with OverMaxRecordSize; callers must split records exceeding 1 MiB.
• Quoted multiline CSV: Quoted multiline CSV delimiter semantics remain tracked separately from this CVE.

12 - CVE-2026-40344: Snowball Auto-Extract Authentication Bypass

Status: Released
First containing release: RELEASE.2026-04-17T00-00-00Z
Fix commit: efb6e5b00
GitHub advisory: GHSA-9c4q-hq6p-c237

Snowball’s PutObjectExtractHandler (X-Amz-Meta-Snowball-Auto-Extract: true) previously omitted the authTypeStreamingUnsignedTrailer authentication case. A tar archive sent with X-Amz-Content-Sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER and a forged signature could enter untar() before authentication completed, allowing a single unauthenticated request to fan out into multiple object writes.

Commit efb6e5b00 initializes newUnsignedV4ChunkedReader(), validates the decoded content length, and completes SigV4 verification before any tar byte reaches the extractor.

Why the commit subject used a temporary identifier

When the fix was committed on 2026-04-16, the official CVE number had not yet been published by GitHub Security Advisories, so the Git commit subject used the temporary placeholder fake CVE-2026-40028. Once GHSA-9c4q-hq6p-c237 was published, the official identifier was assigned as CVE-2026-40344. We kept the immutable Git commit SHA efb6e5b00 intact and mapped it to CVE-2026-40344 in the advisory ledger.

From one missing authentication branch to bulk object writes

The affected entry point is PutObjectExtractHandler, which handles Snowball batch archive uploads (X-Amz-Meta-Snowball-Auto-Extract: true). When the client selected STREAMING-UNSIGNED-PAYLOAD-TRAILER (authTypeStreamingUnsignedTrailer), PutObjectExtractHandler lacked the switch branch that standard PutObjectHandler used to authenticate the request and wrap the chunked stream.

The impact exceeded a single unauthorized object write: as soon as the unauthenticated tar stream entered untar(), the server extracted each archive entry as a separate object write. A single missing authentication branch was therefore amplified into an arbitrary multi-object write primitive.

Final invariant: the extractor sees zero bytes on authentication failure

Design review established a strict requirement for the fix:

If authentication fails, untar() must have read zero bytes.

That invariant rules out any design that begins streaming or extracting archive entries and attempts to roll back created objects if a later check fails. Because Snowball extraction writes objects across multiple internal paths, proving a clean rollback is far more error-prone than proving that unauthenticated bytes never cross into untar(). Authentication had to close before the extractor was invoked.

Implementation

Commit efb6e5b00 aligns PutObjectExtractHandler with standard PutObjectHandler for authTypeStreamingUnsignedTrailer:

• adds the authTypeStreamingUnsignedTrailer case to the handler’s authentication switch;
• parses and validates X-Amz-Decoded-Content-Length;
• wraps the request body with newUnsignedV4ChunkedReader(), which enforces SigV4 signature verification before yielding payload bytes (sharing the reader boundary hardened in CVE-2026-41145);
• preserves valid signed Snowball auto-extract uploads and trailing CRC32 checksum verification.

Verification

Regression tests in cmd verify the full boundary:

• rejection of a Snowball unsigned-trailer request carrying a forged SigV4 signature with zero objects created;
• rejection of anonymous Snowball auto-extract writes to non-public buckets;
• successful extraction of valid signed Snowball unsigned-trailer archives with trailing CRC32 checksums.

go test ./cmd -run 'Test.*Snowball.*' -count=1

Compatibility and operational boundaries

• Rejected unverified combinations: Clients that sent unsigned-trailer Snowball requests with invalid or unverified query-string signatures will now fail with AccessDenied before extraction begins.
• Shared reader boundary: Because Snowball auto-extract and standard PutObject now share newUnsignedV4ChunkedReader() for unsigned-trailer streams, any future change to chunked trailer parsing must run regression tests for both paths.
• Orthogonal archive surfaces: Closing pre-extraction authentication does not alter how untar() processes archive entry paths, sizes, or object counts once an authenticated caller is inside the extractor.

13 - CVE-2026-41145: Unsigned-Trailer Query Authentication Bypass

Status: Released
First containing release: RELEASE.2026-04-17T00-00-00Z
Fix commit: f444b6f37
GitHub advisory: GHSA-hv4r-mvr4-25vw

Before f444b6f37, query-string SigV4 credentials could enter a STREAMING-UNSIGNED-PAYLOAD-TRAILER upload while the handlers verified the request signature only when an HTTP Authorization header was present. An attacker who knew a valid access-key identifier could therefore complete an object or multipart write using a forged query-string signature.

The fix moves presigned-credential rejection and SigV4 verification directly into newUnsignedV4ChunkedReader() (cmd/streaming-v4-unsigned.go), ensuring every handler that consumes an unsigned-trailer chunked stream shares a single authentication boundary.

Commit identifier note

When the patch was written on 2026-04-16, the official CVE had not yet been assigned, so the commit subject used the temporary placeholder fake CVE-2026-40027. Upon publication of GHSA-hv4r-mvr4-25vw, the vulnerability was assigned CVE-2026-41145. The historical commit SHA f444b6f37 remains unchanged.

Root cause: coupling signature verification to credential transport

The vulnerable entry points included PutObject and PutObjectPart (UploadPart). When a client sent X-Amz-Content-Sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER and supplied its SigV4 credentials (X-Amz-Credential, X-Amz-Signature, etc.) in the URL query string rather than the Authorization header, two assumptions collided:

1. The handler checked whether the Authorization header was non-empty to decide whether to run reqSignatureV4Verify before reading the stream, assuming an empty header meant an anonymous request.
2. Identity resolution and newUnsignedV4ChunkedReader() still consumed the request normally while attributing the write to the access key named in X-Amz-Credential.

As a result, query-authenticated STREAMING-UNSIGNED-PAYLOAD-TRAILER requests bypassed signature verification altogether. An attacker only needed to know a valid access-key ID; the signature value in the query string was never checked.

Why we placed the check inside the shared reader instead of individual handlers

Note that genuinely anonymous STREAMING-UNSIGNED-PAYLOAD-TRAILER uploads were not disabled. When a request carries neither header nor query credentials and the bucket policy explicitly grants anonymous s3:PutObject, the request still follows the anonymous authorization path. What is forbidden is carrying unverified credentials into the stream.

Implementation and verification

Commit f444b6f37 enforces presigned rejection and SigV4 verification at the entry of newUnsignedV4ChunkedReader() in cmd/streaming-v4-unsigned.go and removes the Authorization-header presence gates from PutObject and multipart upload handlers.

Regression tests in cmd verify:

• rejection of forged query-string PutObject and PutObjectPart requests in STREAMING-UNSIGNED-PAYLOAD-TRAILER mode;
• rejection of requests mixing header and query-string SigV4 credentials;
• continued operation of valid header-authenticated unsigned-trailer uploads and anonymous uploads permitted by bucket policy.

go test ./cmd -run 'Test.*UnsignedTrailer.*|Test.*StreamingUnsigned.*' -count=1

Compatibility and operational boundaries

• Presigned unsigned-trailer is unsupported: Combining query-string SigV4 credentials with STREAMING-UNSIGNED-PAYLOAD-TRAILER is rejected unconditionally. Clients using unsigned-trailer chunked uploads must pass SigV4 credentials via the Authorization header.
• Reader-level enforcement: Placing the authentication gate inside newUnsignedV4ChunkedReader() ensures that PutObject, PutObjectPart, and PutObjectExtract (CVE-2026-40344) cannot construct an unsigned-trailer reader without completing SigV4 verification.

14 - CVE-2026-42600: ReadMultiple Storage-REST Path Traversal

Status: Released
First containing release: RELEASE.2026-06-18T00-00-00Z
Fix commit: 73ac524
GitHub advisory: GHSA-xh8f-g2qw-gcm7
Affected scope: Distributed erasure deployments only; cluster-root / internode JWT required

The internode storage-REST route /rmpl (ReadMultiple) decoded a msgpack request body carrying Bucket, Prefix, and Files, and joined those fields directly into filesystem paths without a drive-root containment check.

Our initial fix retained the endpoint and added preflight path validation. A subsequent call-chain audit then revealed that ReadMultiple had had zero production callers since September 2024. We therefore discarded the in-place validation patch and deleted the route, handler, client wrapper, StorageAPI method, and generated msgpack types (73ac524)—eliminating roughly a thousand lines of unused privileged RPC surface.

Threat model

The /rmpl storage-REST route was registered only in distributed erasure mode; single-node deployments do not register storage-REST routes and were unaffected. Exploiting the endpoint required an internode JWT derived from the cluster root credentials, control of a peer node, or interception of unencrypted internode traffic.

Because the attacker-controlled Bucket, Prefix, and Files fields arrived inside the msgpack request body rather than in the URL or query string (r.Form), upper HTTP path-validation middleware never saw them. xlStorage.ReadMultiple joined the raw strings directly onto the drive path and read the resulting files, allowing a caller with internode credentials to escape the drive root and read any file accessible to the SILO server process.

First design: retain the API and validate every path

Our first patch hardened xlStorage.ReadMultiple in place:

• rejected absolute paths, . and .. segments, backslashes, Windows drive prefixes, and NUL bytes;
• verified drive-root containment across drive -> volume -> prefix -> file;
• preserved historical edge cases for an empty Bucket and .minio.sys/multipart;
• returned an error before opening files or streaming responses.

That first draft closed the obvious traversal path, but code review immediately caught a flaw in how the loop interacted with MaxResults.

How MaxResults exposed the flaw of validate-as-you-go

The initial draft validated each entry of Files inside the file-reading loop right before opening it. When a request passed Files = [good, "../../etc/passwd"] together with MaxResults = 1, the loop read the first file, hit MaxResults == 1, and returned early without ever inspecting the second entry.

Although that specific call did not read the second file, it violated a core input-validation rule: a malformed msgpack request containing a path-traversal payload must be rejected before any partial execution or streaming begins. We moved path and length validation into a complete preflight pass over the entire request before touching the filesystem.

Final decision: why we deleted ReadMultiple instead

Before merging the preflight-validation patch, we audited every caller of ReadMultiple across the repository and git history:

• upstream had removed the last production caller in September 2024 when multipart reads migrated to ReadParts;
• the current SILO tree had zero in-tree production callers;
• upstream’s own remediation for GHSA-xh8f-g2qw-gcm7 also deleted ReadMultiple.

Commit 73ac524 removes the /rmpl route, storage-REST server handler, client wrapper, StorageAPI and xlStorage methods, metric definitions, and generated msgpack request/response types.

Verification and release

During the in-place validation phase, we ran unit tests covering xlStorage, the storage-REST client, msgpack encoding, path traversal vectors, and the MaxResults preflight edge case. After switching to full deletion in 73ac524 (shipped in RELEASE.2026-06-18T00-00-00Z), verification confirmed that all routes, interfaces, and generated msgpack surfaces compiled and passed go test ./cmd cleanly.

Compatibility and follow-up audit (SN-2026-002)

• External S3 API: Unchanged.
• Rolling upgrades: Keep all nodes in a distributed cluster on the same SILO release during upgrades.
• Follow-up audit across all internode endpoints: All remaining internode endpoints carrying filesystem paths inside msgpack bodies or Grid RPC frames were audited and hardened in SN-2026-002: Internode Path and Allocation Containment (shipped in RELEASE.2026-08-04T00-00-00Z), which enforced volume and file-path containment across storage-REST, storage Grid RPC, and peer-S3 Grid RPC endpoints.

15 - SN-2026-002: Internode Path Containment Audit — Paying Off What CVE-2026-42600 Left Owing

Status: Released in Server 20260804 (release notes)
Identifier: SN-2026-002 (Security Advisories)
Fix commits: ca7baa670, b6f70ab08, 80e8eaa42, 1af351a70, a36fd8fff
Affected scope: Distributed erasure deployments only; cluster-root / internode JWT required
Prerequisite reading: CVE-2026-42600 · ReadMultiple

Following the removal of /rmpl (ReadMultiple) in CVE-2026-42600, SN-2026-002 audits every remaining internode endpoint carrying filesystem paths inside msgpack request bodies or Grid RPC frames. This advisory records that audit across all four internode protocol surfaces, the twelve inherited defects it uncovered, and the four regressions we introduced and caught while building the fix.

Conclusions first

• Twelve defects across three unprotected protocol surfaces, all inherited unchanged from upstream (verified by per-function checksum comparison across the affected storage files).
• Same architectural root cause as CVE-2026-42600: ReadMultiple was one endpoint on one surface; three protocol surfaces remained exposed to the same gap.
• Four regressions introduced and caught during development: every regression occurred in a rule we invented from scratch (isVolumeRootAlias and guardErasureParams), while reusing the object layer’s existing hasBadPathComponent rule produced zero regressions.

Why counting endpoints was the wrong frame

Earlier spot checks kept counting individual handlers—arriving at 19, then 21, then 22—while missing an entire protocol surface each time. Grouping internode calls by how their arguments reach xlStorage reveals four distinct surfaces:

The first row corrects an earlier misconception: audits that grepped exclusively inside handler functions missed that the 9 query-parameter endpoints are already guarded by the global HTTP middleware (url.ParseQuery(RawQuery) in r.Form). Conversely, the fourth row (peer-S3 Grid RPC) bypasses storageRESTServer.getStorage() completely, meaning no amount of hardening limited to storage-rest-server.go could protect DeleteBucket or peer bucket metadata calls.

Three stacked layers in the storage path

Three independent design choices combined to create the vulnerability class:

1. Path validation ran only on HTTP query strings: The HTTP middleware inspects r.Form, which is populated from url.ParseQuery(RawQuery) and never from a request body (introduced in 2017).
2. Grid RPC bypassed the HTTP middleware chain: /minio/grid/v1 performs a single HTTP upgrade; all subsequent msgpack RPC frames dispatch directly to registered handlers without re-entering HTTP middleware (introduced in 2023).
3. xlStorage assumed callers had already validated paths: getVolDir rejected a volume only when it equaled exact "", ., or .., and pathJoin ran path.Clean against the joined path (settled in 2018).

Each layer relied on another layer to validate paths, and none of them inspected the msgpack or Grid RPC channels in between.

One timeline interaction escalated a DoS bug from a failed request into a full process crash: ShardFileSize has contained an unguarded division by DataBlocks since 2020, but when CheckParts was wrapped inside xioutil.WithDeadline in October 2024 to bound slow disk operations, that division moved onto a bare goroutine where the HTTP server’s recover() middleware could no longer catch a divide-by-zero panic.

Vectors confirmed by execution

Every vector below was reproduced against a live xlStorage instance through the storage-REST and Grid RPC clients using planted sentinel files:

Two vectors deserve specific attention:

• RenameFile with an empty source path (""): An empty srcPath resolves to the volume directory itself. When pointed at .minio.sys, RenameFile(".minio.sys", "", "bucket", "x") moves the entire system metadata volume into a normal bucket, where an ordinary GetObject can read cluster IAM and configuration files—without using .. anywhere in the payload.
• Negative part size (Size = -2): A negative part size causes ShardFileSize to floor both terms to 0. Because checkPart checks only st.Size() < expectedSize, any existing shard file (even a 0-byte truncated file) satisfies 0 < 0 == false and is reported intact. Because FileInfo.IsValid() did not reject negative part sizes, poisoned xl.meta metadata caused background healing to skip corrupted shards silently.

The fix: two chokepoints across 36 endpoints

The security invariant has two parts: every file path in an internode payload must resolve strictly inside its target volume, and every volume must resolve strictly inside the drive root.

Because pathJoin folds leading slashes and backslashes under volumeDir via path.Clean, only .. segments can escape a volume directory, and only volume-root aliases ("", "/", ".", and Windows trailing-space/period components) can target the volume directory itself. Commit ca7baa670 enforces those two rules at two chokepoints:

Two implementation details are critical to preserve during future refactors:

• Checks must run on the raw argument before pathJoin: pathJoin calls path.Clean on the concatenated absolute path (/drive + volume + filePath), which silently collapses /drive/../../etc into /etc. A check placed after pathJoin sees a clean path with no .. and allows the escape.
• NSScanner requires its own volume guard: NSScanner is the only xlStorage method that constructs a volume path without calling getVolDir, making its explicit checkPath guard load-bearing.

We evaluated placing containment checks inside all 33 pathJoin(volumeDir, ...) call sites in xl-storage.go, which would require deciding at each of the 33 sinks whether the volume root itself is a valid target. Enforcing the invariant at getVolDir and the getStorage() decorator—backed by reflection guardrail tests on StorageAPI—covers all remote entry points without touching local erasure hot paths.

Construction log: four regressions we caused ourselves

During development and review, our own drafts introduced four regressions before shipping:

1. Rejecting whitespace object keys (" " and " "): Our first draft of the path guard treated whitespace as an invalid path character, rejecting " " and " "—which are valid S3 object keys. Because PutObject commits writes across drives via RenameData, uploading an object named " " failed on every remote drive and broke write quorum.
2. Rejecting backslash-only keys ("\\") on Unix: The same draft rejected "\\". On Unix, path.Clean treats \ as an ordinary filename character rather than a directory separator. Rejecting "\\" in the internode decorator caused distributed clusters to reject a valid S3 key that single-node deployments accepted.
3. Windows trailing spaces and periods (" " and "..."): The Win32 path normalizer strips trailing spaces and periods from path components, meaning a component composed solely of spaces or periods disappears and resolves to its parent directory. On Windows, both " " and "..." alias the volume root—and "..." was sitting inside our own positive test list of “valid object names” until review caught it.
4. Negative part sizes (Size < 0): Our initial guardErasureParams check rejected zero DataBlocks only when part.Size > 0, equating “non-zero” with “positive” and missing that negative sizes floor ShardFileSize to zero even when DataBlocks is valid.

Two false-green tests and a fuzzer blind spot

Writing the reproduction and regression tests exposed three testing pitfalls:

1. storageRESTClient silently overrode Content-Length: When testing AppendFile with a forged Content-Length: 64 GiB over an empty *bytes.Reader, the client library inspected the *bytes.Reader type and overwrote Content-Length with 0.
2. Go’s http.Client refused mismatched body lengths: Switching to an opaque io.Reader caused Go’s standard HTTP client transport to abort the request locally before sending it because the body closed before 64 GiB. Reproducing the server-side 64 GiB allocation required driving the handler directly via httptest (or a raw socket).
3. An early return in the fuzzer hid the backslash bug: Our first fuzz property test skipped strings composed solely of / and \ via an early return. That early return blinded the fuzzer to "\\" across millions of executions until we rewrote the property test to assert strict inclusion against IsValidObjectName.

Reused semantics vs. invented rules

hasBadPathComponent had already been exercised by production S3 traffic via IsValidObjectPrefix for years and structurally cannot reject any object key creatable through the S3 API. Every regression came from newly invented predicates where edge cases (" ", "\\", "...", Size = -2) diverged from S3 or OS semantics.

Guardrail tests

To ensure future changes to StorageAPI cannot silently reintroduce unguarded paths (avoiding the point-fix regression pattern previously seen with CVE-2026-39414), the test suite enforces three guardrails:

• Falsifiability (mutation verification): Temporarily removing the path and volume guards causes 191 subtests to fail; removing the allocation guards reproduces the 64 GiB and 10.4 GB allocations.
• Legal-traffic fuzz equivalence: Fuzz tests verify that every object key accepted by IsValidObjectName passes guardPaths (1.96M executions) and every valid bucket name passes getVolDir (810K executions).
• StorageAPI reflection guardrail: A reflection test inspects every method on StorageAPI and fails CI by method name if a new method accepting path or volume arguments is added without an explicit entry in the decorator guard table.

go test ./cmd -run 'Test.*Containment.*|Test.*Guard.*|Fuzz.*' -count=1

Independent review rounds

Five successive rounds of independent code review each uncovered one real flaw before release: whitespace keys -> backslash keys -> the AppendFile 64 GiB allocation -> Windows trailing spaces/periods -> negative part sizes. Separating implementation from review caught all four S3 compatibility and integrity regressions before release.

Follow-up fixes in Server 20260804

Five companion commits shipped alongside ca7baa670 in Server 20260804:

• ReadFileHandler read-length cap (b6f70ab08): Commit b6f70ab08 rejects declared read lengths above 5 GiB (the maximum size of a single S3 part on this legacy whole-file bitrot path), preventing arbitrary caller-declared multi-tens-of-GiB allocations.
• Negative part size rejection (80e8eaa42): Commit 80e8eaa42 rejects negative part sizes at the AddVersion write funnel and again in CheckParts, VerifyFile, and internode guardErasureParams, protecting both new writes and already-stored metadata.
• Non-positive erasure block size rejection (80e8eaa42): Commit 80e8eaa42 validates blockSize > 0 in NewErasure and guards rebalance division boundaries.
• ReadParts storage error propagation (1af351a70): Commit 1af351a70 fixes a pre-existing bug where ReadPartsHandler called done(nil) before the storage call completed, which had caused storage errors (errFileNotFound as well as new guard rejections) to be written as plain text into a msgpack keep-alive stream (msgp: attempted to decode type "int" with method for "map").
• Panic recovery in xioutil.WithDeadline (a36fd8fff): Commit a36fd8fff recovers panics inside WithDeadline, logs panic in deadline-bounded work: ... with a stack trace to stderr (rate-limited to 10 stack dumps per process lifetime), and fails only the in-flight request. Operators who monitor node health should add a log alert for panic in deadline-bounded work:.

Known limitations

Two platform boundaries remain:

• Windows test environment: SILO publishes Windows binaries while automated CI runs on Linux; the Windows volume-root alias rule (" " and "...") is derived from documented Win32 path normalization behavior.
• Lexical symlink handling: Path containment is lexical (hasBadPathComponent and isVolumeRootAlias), matching upstream xlStorage semantics, and does not resolve OS symlinks placed inside a drive root.

16 - SN-2026-003: s3:versionid Condition Evaluation and Request Condition Hardening

SN-2026-003 and SN-2026-005 cover a set of S3 IAM policy evaluation fixes shipped in Server 20260804 (80c2c9d03, 2f55347f7, eee05a17c, and silo-pkg v3.12.0 commit 2b087a1; pgsty/silo#40, pgsty/silo#53):

1. SN-2026-003 (80c2c9d03): getConditionValues populated args.ConditionValues["versionid"] = []string{""} even when a request carried no versionId, causing IAM Null conditions on s3:versionid to evaluate as if the key were always present; fixing that alone would have opened a DeleteObjects XML <VersionId> bypass without per-entry rebinding in authorizeRequestWithTags.
2. Condition-source isolation (2f55347f7): Separated query parameters, HTTP headers, internal server keys, and s3:ExistingObjectTag/<key> vs. s3:RequestObjectTag/<key> so untrusted query strings or headers cannot spoof internal condition keys.
3. SN-2026-005 (2b087a1 / eee05a17c): Added ParseConfigStrict to reject degenerate bare resource prefixes ("arn:aws:s3:::" and "*arn:aws:s3:::") when writing IAM and bucket policies.

Conclusions first

1. Unconditional empty slice inverted Null checks: In silo-pkg, nullFunc.evaluate tests whether a condition key is absent by checking len(rvalues) == 0. Because getConditionValues unconditionally assigned args.ConditionValues["versionid"] = []string{vid} (producing []string{""} when vid == ""), len(rvalues) was always 1—making Null: {"s3:versionid": "true"} always false and Null: {"s3:versionid": "false"} always true.
2. Fixing getConditionValues alone breaks DeleteObjects: In multi-object DeleteObjects (POST /?delete), the target <VersionId> is inside each XML <Object> element, not in the URL query string. If getConditionValues simply omits "versionid" when the URL query has no versionId, a policy denying versioned deletes (Deny s3:DeleteObjectVersion or Condition: {"Null": {"s3:versionid": "false"}}) would fail to match DeleteObjects XML entries unless authorizeRequestWithTags rebinds ConditionValues["versionid"] from ReqInfo.VersionID.
3. Whitespace trimming closes a Deny bypass: Version-reading handlers called strings.TrimSpace(r.URL.Query().Get("versionId")), whereas getConditionValues did not trim whitespace. A request with ?versionId=V%20 bypassed Deny StringEquals {"s3:versionid": "V"} while still deleting or reading version V.

Why Null on s3:versionid Was Inverted

In silo-pkg/bucket/policy/condition/nullfunc.go, the IAM Null condition operator checks whether a condition key is missing from the request context:

func (f nullFunc) evaluate(values map[string][]string) bool {
    rvalues, ok := values[http.CanonicalHeaderKey(f.k.Name())]
    if !ok {
        return f.v
    }
    return f.v && len(rvalues) == 0
}

However, in cmd/signature-v4-utils.go, getConditionValues populated "versionid" at the end of the function without checking whether vid was non-empty:

var vid string
if v := r.Form.Get("versionId"); v != "" {
    vid = v
} else if u, err := url.Parse(r.Header.Get(xhttp.AmzCopySource)); err == nil {
    vid = u.Query().Get("versionId")
}
args["versionid"] = []string{vid}

When a caller sent an unversioned request (DELETE /bucket/key with no versionId query parameter), vid was "", and args["versionid"] became []string{""} (len == 1). Consequently:

• A policy allowing only unversioned operations via "Null": {"s3:versionid": "true"} evaluated len(rvalues) == 0 as false and denied valid unversioned requests.
• A policy restricting version-specific operations via "Null": {"s3:versionid": "false"} evaluated !f.v && len(rvalues) > 0 as true and matched even when no versionId was supplied.

The Trap: Why Omitting Empty vid Alone Opens a DeleteObjects Bypass

The obvious one-line fix—only writing args["versionid"] = []string{vid} when vid != ""—turned out to hide a dangerous authorization bypass in DeleteMultipleObjectsHandler (POST /{bucket}?delete).

Unlike single-object DELETE /{bucket}/{object}?versionId=V, a multi-object DeleteObjects request carries no versionId in the URL query string. Instead, the request body contains an XML list of <Object><Key>...</Key><VersionId>V</VersionId></Object> entries. Inside DeleteMultipleObjectsHandler, the server loops over each XML entry, sets r.WithContext (ReqInfo.VersionID = obj.VersionID), and calls authorizeRequestWithTags for DeleteObjectAction.

Consider what happens if an administrator configures a policy that allows normal delete-marker creation (s3:DeleteObject when s3:versionid is null) or denies permanent version deletion via:

{
  "Effect": "Deny",
  "Action": ["s3:DeleteObject", "s3:DeleteObjectVersion"],
  "Resource": ["arn:aws:s3:::mybucket/*"],
  "Condition": {
    "Null": {"s3:versionid": "false"}
  }
}

Because authorizeRequestWithTags calls getConditionValues(r, ...)—which inspects r.Form.Get("versionId") on the HTTP request URL rather than ReqInfo.VersionID—omitting empty vid in getConditionValues without propagating ReqInfo.VersionID means args["versionid"] is absent during every per-object check in DeleteObjects. Worse still, an attacker could pass a decoy query parameter on the batch URL (POST /mybucket?delete&versionId=decoy) while putting the protected <VersionId>target-v1</VersionId> inside the XML body!

The Two-Part Fix (80c2c9d03)

Commit 80c2c9d03 fixes both halves together:

1. Omit "versionid" when vid == "" in getConditionValues:

   if vid != "" {
       args["versionid"] = []string{vid}
   }

2. Rebind "versionid" from ReqInfo.VersionID for DeleteObjectAction in authorizeRequestWithTags (cmd/auth-handler.go):

   if action == iampolicy.DeleteObjectAction {
       reqInfo := logger.GetReqInfo(ctx)
       if vid := strings.TrimSpace(reqInfo.VersionID); vid != "" {
           conditionValues["versionid"] = []string{vid}
       } else {
           delete(conditionValues, "versionid")
       }
   }

Why rebind conditionValues["versionid"] only when action == iampolicy.DeleteObjectAction rather than for every action? Because in CopyObjectHandler, ReqInfo.VersionID holds the destination query parameter (if any), whereas the GetObjectAction authorization check on the copy source must evaluate s3:versionid from x-amz-copy-source (?versionId=... on the source URL). Restricting the ReqInfo.VersionID override to DeleteObjectAction ensures both single-object DeleteObjectHandler and per-entry DeleteMultipleObjectsHandler evaluate the exact version being deleted—and explicitly delete(conditionValues, "versionid") when reqInfo.VersionID is empty so that a URL query decoy (POST /bucket?delete&versionId=query-level-decoy) cannot spoof an unversioned XML entry.

Closing the Whitespace Normalization Gap

While auditing every caller that reads versionId, we noticed that object handlers (GetObjectHandler, HeadObjectHandler, DeleteObjectHandler, PutObjectRetentionHandler, etc.) extract the query parameter with:

versionID := strings.TrimSpace(r.URL.Query().Get(xhttp.VersionID))

In contrast, getConditionValues read r.Form.Get("versionId") (and u.Query().Get("versionId") from x-amz-copy-source) without strings.TrimSpace.

If a policy denied access to a specific version via "StringEquals": {"s3:versionid": "v1"}, a caller requesting ?versionId=v1%20 (with a trailing space) had "versionid": ["v1 "] during policy evaluation—evading the StringEquals deny rule—after which the handler trimmed "v1 " to "v1" and operated on the protected version. Applying strings.TrimSpace to both query versionId and x-amz-copy-source versionId inside getConditionValues closes that mismatch.

Call-Chain Impact and Condition-Source Isolation (2f55347f7)

In companion commit 2f55347f7 (pgsty/silo#53), we also hardened getConditionValues across all other request condition keys:

• Reserved internal keys: Client query parameters and HTTP headers can no longer overwrite server-derived condition keys (username, userid, principaltype, authtype, signatureversion, tlsVersion, cipherSuite).
• s3:signatureAge: Computed directly from the validated SigV4 timestamp rather than trusting client-supplied values.
• Tag condition separation: s3:ExistingObjectTag/<key> is populated exclusively from stored object tags, and s3:RequestObjectTag/<key> exclusively from x-amz-tagging on the incoming request.

SN-2026-005: Bare ARN Prefix Rejection

During the same IAM condition audit, we investigated SN-2026-005 (silo-pkg v3.12.0 commit 2b087a1, wired into Server 20260804 in eee05a17c): what happens when a policy contains a degenerate resource string with an empty bucket and object suffix, such as "arn:aws:s3:::" or "*arn:aws:s3:::"?

Because the legacy parser stripped the arn:aws:s3::: prefix and accepted an empty resource pattern "", that pattern could never match any real bucket or object. In an Allow statement, it was merely a dead rule—but in a Deny statement on Resource or an Allow statement using NotResource, a degenerate arn:aws:s3::: pattern failed open!

To prevent administrators from saving policies that silently fail open without risking startup failures on existing clusters:

• silo-pkg v3.12.0 adds policy.ParseConfigStrict, which rejects degenerate bare ARN prefixes (arn:aws:s3::: and *arn:aws:s3:::).
• Server 20260804 (eee05a17c) calls ParseConfigStrict on all policy write paths (bucket policies, named IAM policies, and service-account/STS inline policies), while keeping stored policy load paths on permissive ParseConfig so existing deployments never fail to boot.

Why SN-2026-003 Is Tracked as an Advisory Rather Than a CVE

Exploiting the s3:versionid evaluation bugs required an authenticated caller operating against a policy that explicitly used s3:versionid conditions (Null, StringEquals, or StringNotEquals). Standard s3:DeleteObjectVersion action checks were unaffected unless conditioned on s3:versionid. We track the condition evaluation and DeleteObjects rebinding fixes under SN-2026-003 and the bare ARN prefix rejection under SN-2026-005.

Scope Boundaries and Related Fixes

1. Multi-Delete Object Lock governance bypass (75a6734e4 / #104): While auditing DeleteMultipleObjectsHandler, we noted how per-entry headers and retention checks interact; Server 20260903 (75a6734e4, SN-2026-008) subsequently tightened DeleteMultipleObjectsHandler so x-amz-bypass-governance-retention is only evaluated against the specific locked entry rather than across the batch.
2. Snowball PutObjectExtract minio.versionId PAX header: Noted during our versionId audit and closed in Server 20261018 (SN-2026-016, 14042b5aa).
3. Other condition keys (username, userid, signatureversion, authType): We intentionally scoped 80c2c9d03 to s3:versionid rather than changing empty-slice semantics for identity keys that are always defined on authenticated requests.

Verification

Regression tests in cmd/signature-v4-utils_test.go, cmd/auth-handler_test.go, and cmd/bucket-policy_test.go verify:

• Null: {"s3:versionid": "true"} and Null: {"s3:versionid": "false"} across versioned and unversioned GetObject, DeleteObject, CopyObject, and DeleteObjects calls,
• ?versionId=v1%20 whitespace trimming against Deny StringEquals {"s3:versionid": "v1"},
• POST /bucket?delete&versionId=query-level-decoy with both versioned and unversioned XML <Object> entries,
• Rejection of arn:aws:s3::: and *arn:aws:s3::: via ParseConfigStrict.

Source References

• 80c2c9d03 — s3:versionid condition omission, whitespace trimming, and DeleteObjects per-entry rebinding (SN-2026-003)
• 2f55347f7 — request condition-source isolation (#53)
• eee05a17c / silo-pkg@2b087a1 — ParseConfigStrict bare ARN prefix rejection (SN-2026-005)

Summary

Shipped in Server 20260804, 80c2c9d03 and eee05a17c (silo-pkg v3.12.0) ensure accurate s3:versionid condition evaluation across single-object and batch DeleteObjects operations while rejecting degenerate bare S3 ARN prefixes on policy writes.

17 - SN-2026-004: Object-Scoped Policy Grants Reaching Bucket Mutations

SN-2026-004 fixes an IAM resource-matching flaw in silo-pkg v3.11.0 (3c24ad1, 1f97549, d8b1fa7) and Server 20260804 (97b7d2804; upstream reference minio/minio#20449): when evaluating bucket-level S3 requests (args.ObjectName == ""), Statement.IsAllowed synthesized "bucket/" as the target resource string, allowing object-scoped wildcard grants such as Allow s3:* on arn:aws:s3:::bucket/* to authorize sensitive bucket-level mutations including s3:PutBucketPolicy and s3:DeleteBucket.

Conclusions first

1. Root cause: When args.ObjectName was empty, Statement.IsAllowed appended '/' to the bucket name ("bucket/"), which matched the glob "arn:aws:s3:::bucket/*".
2. Privilege escalation: A tenant principal given Allow s3:* on arn:aws:s3:::tenant/* (a common shorthand for “all object operations inside this bucket”) could call PutBucketPolicy to grant Principal: "*" on the bucket, or call DeleteBucket, PutReplicationConfiguration, PutBucketLifecycle, PutBucketVersioning, or PutBucketObjectLockConfiguration.
3. Why a targeted fix was required: Simply dropping the trailing '/' for all empty-object checks broke 11 existing STS ListBucket tests and weakened Deny s3:* on arn:aws:s3:::bucket/* policies.
4. Targeted boundary: For the 12 sensitive bucket-mutation actions on Allow statements (!legacyBucketResourceMatch && statement.Effect == Allow && isSensitiveBucketMutation(args.Action)), the resource pattern must match both "bucket" and "bucket/". Setting MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH=on restores pre-20260804 matching if a deployment temporarily relies on legacy policies.

Why arn:aws:s3:::bucket/* Matched Bucket Requests

In silo-pkg/bucket/policy/statement.go (and iam/policy/statement.go), Statement.IsAllowed(args Args) constructed the resource string to test against Statement.Resources as follows:

resource := args.BucketName
if args.ObjectName != "" {
    if !strings.HasPrefix(args.ObjectName, "/") {
        resource += "/"
    }
    resource += args.ObjectName
} else {
    resource += "/"
}

Because the else branch unconditionally appended "/", every bucket-level request (args.ObjectName == "") was evaluated as resource = "mybucket/". When matched against a resource pattern of "arn:aws:s3:::mybucket/*", the glob wildcard * matched the empty suffix after "/" and returned true.

Multi-Tenant Escalation Impact

In AWS IAM and S3 bucket policies, arn:aws:s3:::mybucket identifies the bucket itself, whereas arn:aws:s3:::mybucket/* identifies objects inside the bucket. Administrators routinely write policies that grant a tenant or application team full object access using a wildcard action on the object ARN:

{
  "Effect": "Allow",
  "Action": ["s3:*"],
  "Resource": ["arn:aws:s3:::tenant-a/*"]
}

Because s3:* includes both object actions and bucket actions, and "arn:aws:s3:::tenant-a/*" matched resource = "tenant-a/", a caller holding this object-scoped policy could execute bucket-control mutations on tenant-a:

• s3:PutBucketPolicy / s3:DeleteBucketPolicy: Replace the bucket policy with "Principal": "*" to expose the bucket publicly or grant themselves explicit bucket-level permissions.
• s3:DeleteBucket / s3:ForceDeleteBucket: Delete the bucket outright.
• s3:PutReplicationConfiguration: Point bucket replication at an attacker-controlled destination to exfiltrate objects.
• s3:PutBucketLifecycle: Expire all objects and historical versions in the bucket.
• s3:PutBucketVersioning / s3:PutBucketObjectLockConfiguration: Suspend versioning or alter retention defaults.

Why We Did Not Drop the Trailing Slash Globally

Our first prototype simply removed the else { resource += "/" } branch so that args.ObjectName == "" always evaluated against "mybucket". Running the test suite and code review exposed two major regressions with a blanket change:

1. Historical ListBucket compatibility: Eleven upstream STS and IAM tests in cmd/sts-handlers_test.go intentionally granted s3:ListBucket (or s3:*) on arn:aws:s3:::mybucket/* without listing the bare bucket ARN arn:aws:s3:::mybucket. Dropping "/" for read/list actions would break existing deployments whose application policies only list arn:aws:s3:::mybucket/*.
2. Deny statement weakening: If an administrator wrote Deny s3:* on arn:aws:s3:::mybucket/* expecting it to block all operations on that bucket, evaluating args.ObjectName == "" against "mybucket" instead of "mybucket/" would cause the Deny statement to stop matching bucket-level calls—turning a security tightening patch into a Deny bypass!

The Fix: Action-Scoped Dual Matching

To close the escalation path without breaking ListBucket or weakening Deny statements, Statement.IsAllowed restricts the stricter check to Allow statements on sensitive bucket-level mutations (silo-pkg v3.11.0, wired into the server in 97b7d2804):

resource := args.BucketName
if args.ObjectName != "" {
    if !strings.HasPrefix(args.ObjectName, "/") {
        resource += "/"
    }
    resource += args.ObjectName
} else {
    resource += "/"
}

if args.ObjectName == "" && !legacyBucketResourceMatch &&
    statement.Effect == Allow && isSensitiveBucketMutation(args.Action) {
    return statement.Resources.Match(args.BucketName, args.ConditionValues) &&
        statement.Resources.Match(resource, args.ConditionValues)
}
return statement.Resources.Match(resource, args.ConditionValues)

For those actions on an Allow statement, the resource set must match "mybucket" (so arn:aws:s3:::mybucket/* alone is rejected, while arn:aws:s3:::mybucket or arn:aws:s3:::* is accepted) and match "mybucket/" (see Why Monotonicity Required Matching Both).

Second-Pass Action Audit: Which Bucket Writes Are Covered

Our initial draft only gated PutBucketPolicy and DeleteBucketPolicy, overlooking DeleteBucket and lifecycle/replication mutations. In the second pass, we audited every bucket-level write action (args.ObjectName == "") and divided them into two explicit groups.

The 12 protected sensitive bucket mutations (isSensitiveBucketMutation) are:

1. s3:PutBucketPolicy
2. s3:DeleteBucketPolicy
3. s3:DeleteBucket
4. s3:ForceDeleteBucket
5. s3:PutBucketLifecycle
6. s3:PutReplicationConfiguration
7. s3:PutBucketVersioning
8. s3:PutBucketObjectLockConfiguration
9. s3:PutBucketQuotaConfig
10. s3:PutBucketRetentionConfig
11. s3:PutBucketCors
12. s3:DeleteBucketCors

(Note: when Server 20260804 shipped, per-bucket CORS handlers were not yet active; when Server 20260903 implemented per-bucket CORS in #71 / #80, s3:PutBucketCors and s3:DeleteBucketCors were already protected in isSensitiveBucketMutation.)

Four lower-risk or provisioning bucket actions were intentionally kept on historical matching to avoid breaking existing automation pipelines that scope credentials to arn:aws:s3:::app-bucket/*:

• s3:CreateBucket (creating the bucket itself when provisioned with a prefix-scoped policy),
• s3:PutBucketTagging / s3:DeleteBucketTagging,
• s3:PutBucketEncryption,
• s3:PutBucketNotification.

Why Monotonicity Required Matching Both "bucket" and "bucket/"

We wanted a strict monotonicity guarantee: when MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH=off (the default), no Allow statement may authorize a request that would have been denied under legacy matching.

Proving that invariant took two iterations during code review:

1. Attempt 1 (NotResource failure): When we simply tested statement.Resources.Match(args.BucketName, ...) instead of "mybucket/", a policy using NotResource: ["arn:aws:s3:::mybucket/*"] flipped from denied (because "mybucket/" was inside the excluded NotResource set) to allowed (because "mybucket" did not match the excluded pattern).
2. Attempt 2 (single-character ? wildcard failure): Even without NotResource, testing only args.BucketName failed monotonicity on fixed-width wildcards! Consider Allow s3:PutBucketPolicy on Resource: ["arn:aws:s3:::mybucke?"]. Under legacy matching, "mybucket/" is 9 characters long, so mybucke? (8 characters) did not match and the request was denied. If the new check tested only "mybucket" (8 characters), mybucke? would suddenly match and allow a request that legacy mode denied (1f97549).

Requiring statement.Resources.Match(args.BucketName, ...) and statement.Resources.Match(resource, ...) ("mybucket" and "mybucket/") guarantees mathematical monotonicity for all Resource and NotResource patterns: the new check is a pure subset (NewAllow = LegacyAllow ∧ MatchesBareBucket) of legacy Allow evaluation (d8b1fa7). Note that this means a synthetic pattern like arn:aws:s3:::mybucke? (which matches "mybucket" but not "mybucket/") will not authorize sensitive bucket mutations; administrators should write arn:aws:s3:::mybucket or arn:aws:s3:::mybucket*.

Operator Impact and Compatibility Escape Hatch

• Standard policies need no changes: Any policy that grants bucket administration on arn:aws:s3:::mybucket, arn:aws:s3:::mybucket*, or arn:aws:s3:::* continues to work unchanged.
• Object-only policies (arn:aws:s3:::mybucket/*): Can still read/write objects and call ListBucket, CreateBucket, PutBucketEncryption, PutBucketTagging, and PutBucketNotification, but will now receive AccessDenied on the 12 sensitive bucket mutations unless arn:aws:s3:::mybucket is explicitly added to Resource.
• Escape hatch: If an existing automation workflow requires arn:aws:s3:::mybucket/* to authorize one of the 12 sensitive bucket mutations during migration, set MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH=on until the policy is updated.

Follow-Ups Left Open

• Startup policy audit: SILO does not rewrite stored IAM or bucket policies on upgrade; operators can inspect policies with mcli admin policy info to check whether any administrative role relied on arn:aws:s3:::bucket/* for bucket mutations.
• Self-explaining denial diagnostics: S3 AccessDenied responses intentionally do not disclose which statement or ARN shape failed to match.

Summary

Shipped in Server 20260804 (5ff00073a and silo-pkg d8b1fa7 / v3.12.0), the 12 sensitive bucket-mutation actions require the policy Resource to match the bucket ARN itself unless MINIO_API_LEGACY_BUCKET_RESOURCE_MATCH=on is explicitly set.

18 - SN-2026-018: Client Source Address Trust, Forwarded Headers, and aws:SourceIp

SN-2026-018 addresses client source-IP spoofing across aws:SourceIp IAM conditions, audit logs, bucket notification events, and admin HTTP traces in silo-pkg v3.12.0 (62f326d) and Server 20260804 (00a8171cb, pgsty/silo#41): handlers.GetSourceIP previously trusted X-Forwarded-For, X-Real-IP, and RFC 7239 Forwarded from any TCP peer by default, and setting _MINIO_API_XFF_HEADER=off only disabled X-Forwarded-For while continuing to trust client-supplied X-Real-IP and Forwarded headers.

Conclusions first

1. _MINIO_API_XFF_HEADER=off left two headers wide open: Setting _MINIO_API_XFF_HEADER=off skipped X-Forwarded-For, but handlers.GetSourceIP immediately fell through to X-Real-IP and Forwarded. Any direct client could forge X-Real-IP: 10.0.0.1 to satisfy aws:SourceIp policies.
2. “Put a reverse proxy in front” is often insufficient: Even behind an ingress proxy, if the proxy appends to X-Forwarded-For (proxy_add_x_forwarded_for) while the server reads the left-most IP, or if internal pods can reach the server’s ClusterIP directly, an attacker can still spoof aws:SourceIp.
3. MINIO_API_TRUSTED_PROXIES provides explicit right-to-left chain verification: Server 20260804 introduces MINIO_API_TRUSTED_PROXIES (off / none, CIDR/IP list, or all / * for legacy behavior) and walks X-Forwarded-For / Forwarded from right to left, stopping at the first hop not in the trusted proxy set.

Every Consumer of handlers.GetSourceIP

In SILO, handlers.GetSourceIP(r) (silo-pkg/handlers/proxy.go) is the single function responsible for extracting the client IP address from an incoming *http.Request. Four security- and observability-sensitive subsystems depend on its return value:

Note that handlers.GetSourceIPRaw returned the first matched header token without validating whether it was even a syntactically valid IP address (a unit test in upstream silo-pkg literally asserted that X-Forwarded-For: _gazonk returned "_gazonk"!). While IpAddress condition evaluation parses the string with net.ParseIP, audit logs and notification payloads received the raw string.

Why _MINIO_API_XFF_HEADER=off Did Not Stop Spoofing

In silo-pkg/handlers/proxy.go, GetSourceIPRaw inspected three groups of headers in order:

var xffHeaders = []string{"X-Forwarded-For", "x-forwarded-for"}
var realIPHeaders = []string{"X-Real-IP", "x-real-ip"}
var forwardedHeaders = []string{"Forwarded", "forwarded"}

When PR minio/minio#20977 (and minio/pkg#306) added _MINIO_API_XFF_HEADER=off in February 2025, it gated only the xffHeaders loop:

• With _MINIO_API_XFF_HEADER=off, GetSourceIPRaw skipped X-Forwarded-For,
• Then immediately checked X-Real-IP (realIPHeaders) and Forwarded (forwardedHeaders) unconditionally!

Thus, an operator who set _MINIO_API_XFF_HEADER=off specifically to prevent aws:SourceIp spoofing on a directly exposed SILO endpoint remained completely vulnerable to curl -H "X-Real-IP: 192.0.2.10" or curl -H "Forwarded: for=192.0.2.10".

Upstream History since 2017

This area had a long back-and-forth history:

• 2017-07 (minio/minio#4736): Upstream briefly removed X-Forwarded-For from GetSourceIP after recognizing that clients could spoof aws:SourceIp.
• Subsequent refactors: Reverse-proxy deployments complained that aws:SourceIp and audit logs only saw the load balancer’s IP, so X-Forwarded-For, X-Real-IP, and Forwarded were re-admitted—taking the first (left-most) comma-separated entry in X-Forwarded-For.
• 2023-08 (minio/minio#17878) & 2025-02 (minio/minio#20977): Operators requested a way to disable X-Forwarded-For trust, resulting in _MINIO_API_XFF_HEADER, which left X-Real-IP and Forwarded active.

Why “Just Put a Reverse Proxy in Front” Is Not Enough

A common misconception is that forwarded-header spoofing only matters when SILO is exposed directly to the internet without a reverse proxy. In practice, two standard production topologies remain vulnerable even when a reverse proxy is present:

1. Left-most X-Forwarded-For vs. proxy append (proxy_add_x_forwarded_for): Standard nginx configurations use proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;, which appends the client’s real IP to whatever X-Forwarded-For header the client sent. If the client sends X-Forwarded-For: 10.0.0.5, nginx forwards X-Forwarded-For: 10.0.0.5, 203.0.113.99. Because legacy GetSourceIP unconditionally took the first (left-most) address (10.0.0.5), the attacker’s spoofed IP won even behind nginx!
2. Coexisting ingress and direct paths (Kubernetes ClusterIP): In Kubernetes, external traffic arrives through an Ingress controller, while in-cluster workloads connect to the silo Service (ClusterIP) directly. If GetSourceIP blindly trusts forwarded headers whenever an Ingress exists, any pod inside the cluster can connect to the ClusterIP directly and forge X-Forwarded-For or X-Real-IP to impersonate a trusted subnet.

Design of MINIO_API_TRUSTED_PROXIES

To solve both problems cleanly, silo-pkg v3.12.0 (62f326d) and Server 20260804 (00a8171cb) introduce MINIO_API_TRUSTED_PROXIES:

Note that handlers.GetSourceScheme (X-Forwarded-Proto) is intentionally left untouched so TLS redirect and presigned URL scheme detection behind reverse proxies continue to work regardless of MINIO_API_TRUSTED_PROXIES.

Seven Engineering Trade-Offs

1. Default Compatibility vs. Secure-by-Configuration

Changing the unset default to off in a minor release would immediately break every reverse-proxy deployment that uses aws:SourceIp policies without having set MINIO_API_TRUSTED_PROXIES. We kept the unset default compatible and made MINIO_API_TRUSTED_PROXIES=off and CIDR lists first-class, documented controls.

2. Why We Reversed Widening _MINIO_API_XFF_HEADER=off

In our first draft, we widened _MINIO_API_XFF_HEADER=off so that it also disabled X-Real-IP and Forwarded. Running the server test suite immediately failed TestXFFDisabled in cmd/generic-handlers_test.go!

Inspecting TestXFFDisabled showed why: the test specifically modelled an nginx reverse proxy that overwrites X-Real-IP: $remote_addr (trusted single IP) while appending to X-Forwarded-For: $proxy_add_x_forwarded_for (untrusted left-most IP). Operators who set _MINIO_API_XFF_HEADER=off in that exact topology relied on SILO ignoring X-Forwarded-For so that it would fall through to X-Real-IP. We therefore preserved _MINIO_API_XFF_HEADER=off semantics for TestXFFDisabled when MINIO_API_TRUSTED_PROXIES is unset, and provided MINIO_API_TRUSTED_PROXIES=off to disable all forwarded IP headers (X-Forwarded-For, X-Real-IP, and Forwarded).

3. Why MINIO_API_TRUSTED_PROXIES Is Env-Only (No mc admin config Key)

In SILO’s startup sequence, the HTTP listener and initial request routing come up before the distributed api subsystem configuration is loaded from erasure storage. If trusted proxies were configured via mc admin config set api, there would be a startup window before config load where GetSourceIP either failed open or failed closed, and updating the setting at runtime would require atomic reloads across silo-pkg. Making MINIO_API_TRUSTED_PROXIES an environment variable configured before listener startup eliminates any race window.

4. Loopback Exemption: Peer vs. Chain Hop

SILO’s built-in FTP and SFTP servers (cmd/ftp-server.go, cmd/sftp-server.go) and local inter-node forwarding connect to the S3 layer over loopback (127.0.0.1 / ::1) and pass the external client’s IP in X-Forwarded-For. Therefore, when the immediate TCP peer (r.RemoteAddr) is loopback, SILO treats that local peer hop as trusted so FTP/SFTP audit logs and aws:SourceIp continue to see the real FTP/SFTP client IP.

Crucially, however, loopback is not implicitly trusted as an intermediate hop inside the X-Forwarded-For header chain! Otherwise, an external client behind one trusted proxy could send X-Forwarded-For: 10.0.0.1, 127.0.0.1, causing the right-to-left walker to skip the proxy (r.RemoteAddr), skip 127.0.0.1 in the header, and trust the attacker’s 10.0.0.1.

5. Header Precedence: X-Forwarded-For Before X-Real-IP

When MINIO_API_TRUSTED_PROXIES is configured with a CIDR/IP list, GetSourceIP evaluates X-Forwarded-For (right-to-left), then X-Real-IP (validated with parseIP), then Forwarded (right-to-left). Note that this differs intentionally from getSTSLDAPTrustedProxySourceIP (cmd/sts-handlers.go), which checks X-Real-IP first for single-hop LDAP STS proxies; in general multi-hop S3 proxy chains, X-Forwarded-For carries the hop-by-hop chain needed for right-to-left verification while X-Real-IP may remain frozen at an outer hop.

6. Broad CIDR Warning

If an operator sets MINIO_API_TRUSTED_PROXIES=10.0.0.0/8 when ordinary application pods also run in 10.0.0.0/8, any pod in that /8 is treated as a trusted proxy and can forge X-Forwarded-For. Operators should list only the dedicated ingress/load-balancer IPs or subnet.

7. Multi-Node Distributed Clusters

In a multi-node SILO cluster (silo server http://node{1..4}/data), a load balancer may route a client request to Node 1, which internally proxies or forwards sub-requests to Nodes 2–4:

Compatibility Details

• No non-IP garbage returned in trusted-proxy mode: When MINIO_API_TRUSTED_PROXIES is active (off or CIDR list), every candidate header token is validated as a real IP address; malformed tokens like _gazonk are rejected rather than returned to callers.
• Removal of unreachable IPv4-mapped prefix string hack: Legacy proxy.go contained a string check for "::ffff:" that was unreachable or mishandled hex forms; netip.Addr.Unmap() now normalizes IPv4-mapped IPv6 addresses (::ffff:192.0.2.1 -> 192.0.2.1) cleanly.

Bugs Caught During Multi-Pass Code Review

Iterative code review of the patch across silo-pkg and silo caught nine subtle edge cases before release:

Second-Pass Review Findings

1. init() vs. MINIO_CONFIG_ENV_FILE startup ordering: Reading MINIO_API_TRUSTED_PROXIES and _MINIO_API_XFF_HEADER only inside a package init() function meant values loaded later from MINIO_CONFIG_ENV_FILE in serverHandleEnvVars were ignored! Server 20260804 explicitly re-applies both settings in serverHandleEnvVars.
2. Loopback in-chain spoofing (X-Forwarded-For: spoofed, 127.0.0.1): Caught the distinction between trusting a loopback r.RemoteAddr peer vs. trusting 127.0.0.1 inside an X-Forwarded-For header list (fixed as described in Trade-Off 4).
3. IPv6 zone identifiers (fe80::1%eth0): Link-local IPv6 peers include a %zone suffix in r.RemoteAddr. Using netip.ParseAddr and stripping the zone before CIDR matching prevents link-local peers from failing IP parsing.
4. Multiple X-Forwarded-For header lines (r.Header.Values): HTTP proxies such as HAProxy or Envoy may append a second X-Forwarded-For header line rather than comma-joining onto the first line. Calling r.Header.Get("X-Forwarded-For") read only the first (client-controlled!) header line. We switched to iterating r.Header.Values("X-Forwarded-For") from last header line to first.
5. Internal forwarder.go header hygiene: When MINIO_API_TRUSTED_PROXIES rejected an untrusted peer’s X-Real-IP or Forwarded header, internal HTTP forwarding needed to avoid re-injecting untrusted headers alongside a trusted peer hop.
6. ::ffff:0:0/96 CIDR unmapping: Unmapping the prefix address of ::ffff:0:0/96 without checking prefix length would turn a /96 IPv6 prefix into 0.0.0.0 and match all IPv4 addresses; we preserve CIDR prefix semantics accurately.

Third-Pass Review Findings

7. Fail-closed on empty CIDR list (MINIO_API_TRUSTED_PROXIES=","): If an operator sets MINIO_API_TRUSTED_PROXIES="," (or whitespace-only), silently treating the empty slice as “unset (trust all)” would fail open! Startup now returns an explicit configuration error if MINIO_API_TRUSTED_PROXIES is set to an empty or unparseable value.
8. os.LookupEnv vs. env.Get(""): Using env.Get(EnvAPITrustedProxies, "") could not distinguish between MINIO_API_TRUSTED_PROXIES being unset and being set to ""; we check explicit presence so empty settings fail closed.
9. Zero-allocation backwards scan and 100-hop cap: Allocating strings.Split(header, ",") on every request allowed a client sending a 1 MB X-Forwarded-For header full of commas to trigger a ~33 MB slice allocation per request. We replaced strings.Split with an in-place right-to-left scanner and capped inspection at 100 hops.

Operator Recommendations

• Direct-attached SILO (no reverse proxy): Set MINIO_API_TRUSTED_PROXIES=off.
• SILO behind nginx / HAProxy / Envoy / Ingress: Set MINIO_API_TRUSTED_PROXIES to the CIDR or IP list of the reverse proxies (and SILO node IPs in a multi-node cluster).

Out of Scope

• PROXY protocol v1/v2: TCP-layer PROXY protocol support at the listener is separate from HTTP X-Forwarded-For / Forwarded parsing.
• GetSourceScheme: Left unchanged so HTTPS scheme detection behind TLS-terminating proxies continues to work independently of MINIO_API_TRUSTED_PROXIES.
• ExistingObjectTag/* condition isolation: Fixed separately in 2f55347f7 (also shipped in 20260804) so only stored object tags—never client request headers—populate ExistingObjectTag/* conditions.

Severity Classification

Tracked as SN-2026-018 in the SILO advisory ledger (as with CVE-2026-42600, minio/minio is archived; severity is Moderate-to-High depending on whether policies rely on aws:SourceIp). Upgrading to Server 20260804 and setting MINIO_API_TRUSTED_PROXIES closes the spoofing vector across IAM, audit logs, notifications, and traces.

19 - SN-2026-019: Notification Keyspace Registration and Validation Redaction

SN-2026-019 covers a class of notification configuration defects fixed in Server 20260804 (162ded343 and 672c07a0f, pgsty/silo#39): notification parser and legacy-migration paths read or wrote KVS configuration keys that were missing from Default*KVS, while CheckValidKeys echoed raw key values in validation errors and FetchEnabledTargets aborted all active notification targets on the first validation failure. In Server 20260903 (f1ba68358), the discrete PostgreSQL and MySQL host/credential keys uncovered by our AST audit were retired in favor of connection_string and dsn_string.

Conclusions first

1. Four uncoordinated surfaces drifted apart: In the notification configuration subsystem, Default*KVS (valid key schema), Help* (help text and secret/sensitive classification), GetNotify* (parser), and SetNotify* (legacy config migration) were maintained by hand without a cross-check.
2. One invalid target disabled all notification targets: FetchEnabledTargets validates every notification subsystem sequentially and returns on the first error, leaving the global notification target list nil across the server.
3. Validation errors echoed secret values: When CheckValidKeys rejected an unknown or mistyped key, it formatted the error using kv.String(), printing key=value—including credentials or passwords—into logs and admin CLI responses.
4. AST-enforced invariant: Server 20260804 registers the missing NATS and AMQP keys, redacts values from CheckValidKeys errors, expands HasSensitiveKeys, and enforces a compile-time AST test ensuring every key read or written by cmd/config/notify is registered in Default*KVS.

A Puzzling Bug Report

The investigation started from a bizarre symptom reported by kuldeep-link11: an operator ran mc admin config set us notify_nats:FITCHECK address="..." subject="..." and received an error complaining about a completely different target (notify_nats:ONE) and two keys (nkey_seed= tls_handshake_first=off) that the operator had never passed on the command line:

Error: Invalid key/value pair 'nkey_seed= tls_handshake_first=off' found in subsystem 'notify_nats:ONE'

Two separate mechanisms combined to produce that confusing error:

1. Tokenizer absorption on mc admin config set: kvFields in internal/config/parser.go splits input tokens by matching only keys registered in DefaultKVS. When a user passes an unregistered key on the command line, kvFields does not recognize it as a key delimiter and silently absorbs "nkey_seed=..." as part of the preceding key’s value (such as subject).
2. Legacy migration poisoning on notify_nats:ONE: Earlier in the deployment’s life, the legacy configuration migrator (SetNotifyNATS in cmd/config/notify/legacy.go) had serialized notify_nats:ONE directly into the KVS struct—bypassing kvFields—and written nkey_seed and tls_handshake_first as discrete KVS keys. When the operator later edited notify_nats:FITCHECK, validateConfig re-validated all stored targets and tripped over notify_nats:ONE.

One Constant Used for Two Different Namespaces

Inspecting internal/config/notify/target/nats.go immediately revealed a second bug right next to nkey_seed:

const (
    NATSAddress = "address"
    NATSSubject = "subject"
    ...
    EnvNATSAddress         = "MINIO_NOTIFY_NATS_ADDRESS"
    EnvNATSSubject         = "MINIO_NOTIFY_NATS_SUBJECT"
    NATSUserCredentials    = "MINIO_NOTIFY_NATS_USER_CREDENTIALS"
)

When PR minio/minio#19139 added NATS user credentials support in February 2024, NATSUserCredentials was placed inside the Env* constant block with the value "MINIO_NOTIFY_NATS_USER_CREDENTIALS", and then used as both the environment variable name and the KVS key name in DefaultNATSKVS. As a result, setting user_credentials=/path/to/creds via mc admin config set notify_nats was rejected as an unknown key, whereas the literal string MINIO_NOTIFY_NATS_USER_CREDENTIALS=/path/to/creds was what DefaultNATSKVS expected on disk.

Four Uncoordinated Surfaces

Auditing all notification targets in cmd/config/notify/ showed that every target defines four separate surfaces:

• Default*KVS (cmd/config/notify/config.go): the canonical list of valid KVS keys enforced by CheckValidKeys,
• Help* (cmd/config/notify/help.go): user-facing help metadata and Type (Sensitive / Secret),
• GetNotify* (cmd/config/notify/parse.go): reads KVS keys and MINIO_NOTIFY_* environment variables,
• SetNotify* (cmd/config/notify/legacy.go): migrates older JSON config structs into KVS pairs.

Across NATS and AMQP, four keys had drifted across those four surfaces:

Notice the AMQP bug in the last row: SetNotifyAMQP in cmd/config/notify/legacy.go had a copy-paste typo that wrote strconv.FormatBool(cfg.Immediate) under the key target.AmqpInternal and never serialized cfg.Internal at all.

Blast-Radius Amplifier: All-or-Nothing Target Loading

Why did a broken notify_nats:ONE target matter so much? In cmd/config/notify/parse.go, FetchEnabledTargets loads all ten notification subsystems (amqp, elasticsearch, kafka, mqtt, mysql, nats, nsq, postgres, redis, webhook) in sequence. If any subsystem returns a validation error from CheckValidKeys, FetchEnabledTargets returns early before populating the active target map.

Consequently, a single invalid key in an unused notify_nats target causes startup or config reload to leave the global notification target list empty, silently disabling all bucket notifications (such as active Kafka or Webhook targets) until an administrator removes the offending config entry.

We considered changing FetchEnabledTargets to isolate errors per subsystem and keep other targets running, but declined that change: across SILO, configuration validation is intentionally all-or-nothing so that mc admin config set rejects broken configurations rather than partially applying them. Fixing key registration removes the false-positive failures without weakening config transaction semantics.

Fixing Key Registration and Classification

Commit 162ded343 fixes the NATS and AMQP keyspace defects:

1. Split key constant from env constant: target.NATSUserCredentials is now "user_credentials", and target.EnvNATSUserCredentials is "MINIO_NOTIFY_NATS_USER_CREDENTIALS".
2. Register all four keys in DefaultNATSKVS and DefaultAMQPKVS: user_credentials, nkey_seed, and tls_handshake_first are registered in DefaultNATSKVS, and immediate is registered in DefaultAMQPKVS.
3. Complete HelpNATS and HelpAMQP: Added help entries for user_credentials, nkey_seed, and immediate. Both user_credentials and nkey_seed take local file paths on the server filesystem (just like client_key), so we classified them as config.Sensitive rather than config.Secret.
4. Fix SetNotifyAMQP: SetNotifyAMQP now writes cfg.Internal to target.AmqpInternal and cfg.Immediate to target.AmqpImmediate.

Narrow Backward Tolerance for Stored NATS Configs

What if a deployment prior to 20260804 had worked around #19139 by storing the literal key MINIO_NOTIFY_NATS_USER_CREDENTIALS=/path/to/creds in its persisted cluster configuration?

Simply renaming NATSUserCredentials to "user_credentials" would cause CheckValidKeys to reject that persisted configuration on upgrade—triggering the exact FetchEnabledTargets outage described above.

While designing backward tolerance, we uncovered a subtle trap between two identically named functions in internal/config:

• The free function config.CheckValidKeys(subSys, kvs, defaultKVS, deprecatedKeys...) accepts an optional variadic list of deprecated keys and permits them during load validation.
• The method kvs.CheckValidKeys(subSys, defaultKVS) calls CheckValidKeys(subSys, kvs, defaultKVS) without passing any deprecated keys.

We updated the NATS load path (GetNotifyNATS) to tolerate "MINIO_NOTIFY_NATS_USER_CREDENTIALS" via the deprecated-key list on config.CheckValidKeys, and implemented deterministic precedence when populating UserCredentials:

1. MINIO_NOTIFY_NATS_USER_CREDENTIALS environment variable (highest precedence),
2. user_credentials KVS key,
3. Legacy MINIO_NOTIFY_NATS_USER_CREDENTIALS KVS key (fallback when user_credentials is empty).

Redacting Values in Validation Errors (SN-2026-019)

Before 20260804, both config.CheckValidKeys and KVS.CheckValidKeys built their error string by appending kv.String() (key=value) for every unrecognized key. If an administrator mistyped a secret key name—for example, pasword=Hunter2 or auth_tokn=eyJ...—or if a legacy migrator wrote an unregistered credential key, the error message echoed the cleartext secret into the console response and server error logs.

Commit 162ded343 changes both CheckValidKeys functions to report only the offending key names (kv.Key), never kv.Value.

In commit 672c07a0f, we also audited config.HasSensitiveKeys (internal/config/config.go), which masks secrets when exporting configurations via mc admin config get or generating health-diagnostics bundles. We expanded HasSensitiveKeys to cover all credential key names used across notification, identity, and KMS subsystems (password, auth_token, secret_key, client_secret, nkey_seed, user_credentials, sasl_password, connection_string, and dsn_string).

Enforcing the Invariant with an AST Audit Test

To make sure Default*KVS, parse.go, and legacy.go can never drift apart again, we wrote an AST-based verification test in cmd/config/notify/config_audit_test.go. Using go/ast, the test parses parse.go and legacy.go, resolves every kvs.Get(...) and kvs.Set(...) call back to its string constant definition in internal/config/notify/target, and verifies that every referenced key exists in the corresponding Default*KVS.

While reviewing the audit test itself, we found and closed two blind spots in our initial test implementation:

1. Reverse floor assertion: Checking only that parse.go ⊆ Default*KVS would pass even if the AST walker broke and extracted zero keys! We added the reverse check requiring every key registered in Default*KVS (except documented universal keys like enable and comment) to be read in parse.go and asserting a minimum key-count floor per subsystem.
2. Untyped composite literals: In legacy.go, helper tables constructed config.KVS{...} using untyped {Key, Value} composite literals rather than calling kvs.Set(). Extending the AST visitor to inspect config.KVS composite literals immediately surfaced a second bug in the codebase.

What the AST Audit Caught: PostgreSQL and MySQL Discrete Keys

As soon as the AST visitor inspected config.KVS composite literals in legacy.go, it flagged SetNotifyPostgres and SetNotifyMySQL: both functions wrote five discrete keys—host, port, username, password, and database—that were read by GetNotifyPostgres / GetNotifyMySQL (parse.go) when connection_string / dsn_string was empty, yet were completely absent from DefaultPostgresKVS and DefaultMySQLKVS!

In addition, because password was not registered in DefaultPostgresKVS or DefaultMySQLKVS, passing password=... or migrating a legacy PostgreSQL/MySQL config triggered CheckValidKeys and printed the database password in the error message prior to 162ded343.

We handled this in two steps across two releases:

• Short-term fix in Server 20260804 (672c07a0f): Registered host, port, username, password, and database in DefaultPostgresKVS and DefaultMySQLKVS (and marked password as Secret in HelpPostgres / HelpMySQL) so that existing configurations and the new AST invariant were immediately consistent.
• Architectural cleanup in Server 20260903 (f1ba68358): Having two parallel ways to configure database connections (connection_string vs. discrete host/port/username/password/database keys) caused SetNotifyPostgres and SetNotifyMySQL to overwrite connection_string with empty discrete fields during config round-trips. Server 20260903 retired the five discrete keys across Default*KVS, Help*, parse.go, and legacy.go, standardizing exclusively on connection_string (PostgreSQL) and dsn_string (MySQL).

Declined Changes

• Do not change kvFields tokenizer framing in this patch: Changing kvFields to split on arbitrary key=value tokens rather than known DefaultKVS keys would alter how unquoted spaces inside values (such as SQL queries or subjects) are parsed. With Default*KVS now verified by the AST audit, valid keys are always recognized by kvFields.
• Do not weaken FetchEnabledTargets fail-fast validation: As explained in Blast-Radius Amplifier, configuration validation remains atomic across all subsystems.

Review Record

Code review verified the AST audit’s reverse floor assertions, untyped config.KVS composite literal inspection, CheckValidKeys value redaction (162ded343), and HasSensitiveKeys credential coverage (672c07a0f).

Summary

Shipped in Server 20260804 (162ded343 and 672c07a0f) with follow-up cleanup in Server 20260903 (f1ba68358), all notification KVS keys are verified against Default*KVS by AST tests and unrecognized-key errors redact values.

20 - Duplicate Part Numbers in CompleteMultipartUpload

In CompleteMultipartUpload, the S3 API requires part numbers in the completion XML to be strictly ascending. Up through Server 20260725, SILO enforced that invariant with sort.SliceIsSorted and a strict < comparator—which accepts non-decreasing sequences such as [1, 1] and duplicates the referenced part in the assembled object. Server 20260804 (pgsty/silo#49) replaces that check with an explicit adjacent-pair comparison (Parts[i-1].PartNumber >= Parts[i].PartNumber) and fixes an adjacent empty-part panic in ReadParts (2602177ef).

Conclusions first

1. The bug is real and small: CompleteMultipartUpload used sort.SliceIsSorted with <, which tests non-decreasing order (a[0] <= a[1] <= ...) rather than strictly increasing order (a[0] < a[1] < ...). Passing duplicate part numbers such as [1, 1] succeeded and duplicated the underlying part in the final object.
2. This is a data-integrity defect, not a CVE: Triggering it requires a caller who already holds s3:PutObject permission on the target key and a valid uploadId for an in-progress multipart upload. It does not cross tenant, bucket, or credential boundaries.
3. The fix stays at the HTTP handler boundary: cmd/object-handlers.go rejects Parts[i-1].PartNumber >= Parts[i].PartNumber with InvalidPartOrder (HTTP 400). Sparse part numbers ([1, 3], [5, 9], [3]) remain valid.

Why sort.SliceIsSorted Accepted Duplicates

In cmd/object-handlers.go, CompleteMultipartUploadHandler validated the parsed <CompleteMultipartUpload> part list with:

if !sort.SliceIsSorted(complMultipartUpload.Parts, func(i, j int) bool {
    return complMultipartUpload.Parts[i].PartNumber < complMultipartUpload.Parts[j].PartNumber
}) {
    writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrInvalidPartOrder), r.URL)
    return
}

In Go’s standard library, sort.SliceIsSorted iterates i from n-1 down to 1 and calls less(i, i-1). If less(i, i-1) returns true, the slice is out of order. When two adjacent elements are equal (Parts[i].PartNumber == Parts[i-1].PartNumber), Parts[i].PartNumber < Parts[i-1].PartNumber evaluates to false, so SliceIsSorted treats them as sorted.

Replacing sort.SliceIsSorted with a direct adjacent check closes the gap:

for i := 1; i < len(complMultipartUpload.Parts); i++ {
    if complMultipartUpload.Parts[i-1].PartNumber >= complMultipartUpload.Parts[i].PartNumber {
        writeErrorResponse(ctx, w, errorCodes.ToAPIErr(ErrInvalidPartOrder), r.URL)
        return
    }
}

Two Refactors Preserved the Defect

Tracing git log -S across the handler history shows that the non-decreasing check survived two separate cleanups over nine years:

Every unit and mint test for ErrInvalidPartOrder tested inversions such as [2, 1], never adjacent duplicates such as [1, 1].

Observed Impact

Before the fix, uploading a single 5 MiB part (partNumber=1) and completing the upload with two <Part> entries both referencing PartNumber=1 and its ETag produced the following behavior:

Because CompleteMultipartUpload consumes the upload state on success, a buggy client that accidentally duplicated an entry in its completion XML (for example, when retrying an UploadPart call and appending to an in-memory slice instead of deduplicating by part number) silently wrote a corrupted object of the wrong size and could not retry completion without re-uploading every part.

Why This Is Not a CVE

We evaluated whether a malicious caller could weaponize duplicate part numbers for storage amplification:

1. No physical disk amplification: In erasure-coded storage, CompleteMultipartUpload records part metadata in xl.meta referencing the existing part.1 data files on disk rather than copying part bytes per entry. Reading the object streams part.1 twice, and background healing or copying materializes the logical size, but the caller could achieve the same result legitimately by uploading two parts or calling UploadPartCopy.
2. Same-caller scope: Executing CompleteMultipartUpload requires write access to the object and knowledge of the uploadId. No unauthenticated or read-only principal can trigger it.

The issue is therefore tracked as a correctness and S3 protocol compliance bug (#49) rather than a security advisory (SN-2026-*).

Handler vs. Object-Layer Boundary

During investigation we checked whether erasureObjects.CompleteMultipartUpload (cmd/erasure-multipart.go) also validates part ordering. It does not: the object layer iterates for i, part := range parts and matches each entry against uploadID part metadata.

We kept the check in CompleteMultipartUploadHandler (cmd/object-handlers.go) rather than duplicating it inside every ObjectLayer implementation for two reasons:

1. Single untrusted entry point: Client-supplied completion lists enter only through CompleteMultipartUploadHandler. Internal callers (batch-replicate, restore, decommission, rebalance, and PutObjectPart pathways) construct part lists directly from existing ObjectInfo.Parts or sequential 1..n loops.
2. Fail-fast before lock acquisition: Rejecting malformed XML in the HTTP handler avoids acquiring the object’s namespace lock or reading upload metadata from disk.

Preserving Sparse Part Numbers

The S3 specification requires part numbers in CompleteMultipartUpload to be strictly increasing (1 <= PartNumber <= 10,000), not contiguous and not starting at 1. A client that uploads parts 1, 2, 3 and completes using only [1, 3], or completes using [5, 9] or [3], is conforming to S3.

To make sure our fix did not over-constrain valid clients, we added explicit regression cases for:

• [1, 3] (sparse gap),
• [3] (single part not starting at 1),
• [1, 1] and [1, 2, 2, 3] (adjacent duplicates, rejected with ErrInvalidPartOrder),
• [1, 3, 2] and [2, 1] (decreasing order, rejected with ErrInvalidPartOrder),
• [1, 2, 1] (non-adjacent duplicate, rejected with ErrInvalidPartOrder).

Error Code Precedence for Out-of-Range Duplicates

One subtle edge case is an input that is both out of range (PartNumber < 1) and duplicated, such as [0, 0] or [-1, -1].

Because CompleteMultipartUploadHandler validates part ordering before iterating into the object layer (where individual part numbers are looked up and ErrInvalidPart is returned), inputs like [0, 0] previously passed sort.SliceIsSorted and failed later in the object layer with InvalidPart (HTTP 400). With strict ordering enforced up front, [0, 0] and [-1, -1] now fail immediately at the handler with InvalidPartOrder (HTTP 400). Both error codes map to HTTP 400 Bad Request and neither is retriable; a single out-of-range part ([0] or [-1]) still reaches the part lookup and returns InvalidPart.

Mutation Testing and a Misleading Comment

While verifying the unit tests in cmd/object-handlers-multipart_test.go, we ran two manual mutations against the handler:

1. Mutating the check to require Parts[0].PartNumber == 1: Immediately caught by the sparse-part regression tests ([3] and [5, 9]).
2. Deleting the len(complMultipartUpload.Parts) == 0 guard: Our draft test comment originally claimed that removing the empty-parts guard in the handler would allow an empty multipart completion to succeed. Running that mutation revealed something much more interesting: an empty Parts slice bypassed the handler check, entered the object layer, and triggered an index-out-of-range panic in ReadParts (cmd/storage-rest-client.go) inside an errgroup goroutine.

That second mutation served two purposes: we corrected the misleading comment in cmd/object-handlers-multipart_test.go, and we fixed ReadParts in commit 2602177ef (also shipped in Server 20260804) so that an empty partNumbers slice returns early instead of indexing partNumbers[0].

Declined Changes and Follow-Ups

• No retrospective repair tool: Existing objects assembled from duplicate part lists before 20260804 have valid erasure metadata for their logical size; automatic truncation or deduplication would risk corrupting objects intentionally assembled from identical chunks.
• No duplicate object-layer check: As noted in Handler vs. Object-Layer Boundary, internal callers construct part lists from validated metadata, and 2602177ef closes the ReadParts empty-slice boundary found during mutation testing.

Review Record

Three independent review passes verified the handler predicate, error-code precedence (InvalidPartOrder vs. InvalidPart), sparse part numbering ([3] and [5, 9]), and the ReadParts empty-slice guard (2602177ef).

Summary

Shipped in Server 20260804 (5df7b5466 and 2602177ef), CompleteMultipartUploadHandler enforces strict part-number monotonicity (PartNumber[i-1] < PartNumber[i]) and ReadParts safely rejects empty part-number slices.