SN-2026-022: Active-Active Tagging Proxy Authentication and Remote Tag-Policy Bypass
Status: Fixed on main (14ecb222e, 67a90dfef, 293ab78b4); scheduled for the next post-20261109 Server release
Identifier: SN-2026-022 (Security Advisory Ledger, #337)
Severity: Critical
Affected scope: SILO Server (RELEASE.2026-11-09T00-00-00Z and earlier) and inherited upstream MinIO releases since 2024-01-12 (b2b26d9c9)
Unaffected components: silo-console, mcli, silo-pkg, and kes
When active-active bucket or site replication is enabled, an S3 object version may temporarily or permanently reside on a remote replication target while absent from the local cluster receiving a client request. To keep tagging operations transparent across peers, GetObjectTaggingHandler, PutObjectTaggingHandler, and DeleteObjectTaggingHandler (cmd/object-handlers.go) proxy requests for locally missing objects to active-active replication targets via proxyTaggingToRepTarget (cmd/bucket-replication.go), signing the outbound request with the server’s privileged replication credentials.
Inherited from upstream MinIO, this proxy path contained two distinct security defects: DeleteObjectTaggingHandler looked up the local object and proxied to remote targets before authenticating or authorizing the caller, and all three tagging handlers proxied locally missing objects without fetching or evaluating the remote replicas’ stored tags against IAM s3:ExistingObjectTag/<key> conditions.
Summary
- What went wrong:
- Unauthenticated remote tag deletion and key-existence oracle:
DeleteObjectTaggingHandlerexecutedgetObjectInfoandproxyTaggingToRepTargetahead ofcheckRequestAuthType. A completely unauthenticatedDELETE /<bucket>/<key>?taggingrequest for an object present only on an active-active peer wiped the remote object’s tags (204 No Content) using the server’s internal replication target credentials, and distinguished missing keys (404 NoSuchKey) from present local keys (403 AccessDenied) on private buckets withouts3:ListBucket. - Tag-conditioned policy bypass across replication peers: When an object was absent locally,
GetObjectTagging,PutObjectTagging, andDeleteObjectTaggingevaluated IAM policies against an empty tag string ("") and forwarded the operation to active-active targets without inspecting the tags actually stored on those replicas—allowing callers blocked byDenyrules ons3:ExistingObjectTag/*(such asclassification=restricted) to read, overwrite, or erase remote tags, while falsely rejecting callers whoseAllowrule required a matching tag.
- Unauthenticated remote tag deletion and key-existence oracle:
- Upstream provenance: Both defects were introduced in upstream MinIO commit
b2b26d9c9(2024-01-12, minio/minio#18649, “support proxying of tagging requests in replication”). Every upstream MinIO release after January 2024 and SILO Server20261109and earlier are affected. - Fix: Commits
14ecb222e,67a90dfef, and293ab78b4authenticate every tagging request upfront, gate remote probing onauthorizeRequestWithUnreadTagsso unauthorized callers cannot trigger cross-site reads, inspect and authorize stored tags on every selected present replica before returning or mutating tags, tolerate missing replicas caused by replication lag, and fail closed when any selected target cannot be read.
1. Upstream Provenance and Root Cause
Unauthenticated Execution in DeleteObjectTaggingHandler
In PutObjectTaggingHandler and GetObjectTaggingHandler, upstream code called checkRequestAuthType (or authenticateRequest) before touching storage. In DeleteObjectTaggingHandler, however, b2b26d9c9 inserted the local lookup and active-active proxy block directly above authentication:
Two consequences followed immediately:
- Remote tag erasure without credentials: Whenever
(bucket, object)was absent on the receiving cluster (ObjectNotFoundorVersionNotFound) and a matching replication rule selected a proxy-enabled target holding the object,proxyTaggingToRepTargetcalledclnt.RemoveObjectTagging(ctx, bucket, object, ...)using the replication target’s own admin/replication credentials and returned204 No Content—never checking whether the incoming HTTP request was even signed. - Private-bucket key-existence oracle: Even on non-replicated buckets, an unauthenticated
DELETE /bucket/key?taggingreturned404 NoSuchKeywhengetObjectInfofailed withObjectNotFound, versus403 AccessDeniedoncegetObjectInfosucceeded and reachedcheckRequestAuthType.
Blind Proxying Across Tag-Conditioned IAM Policies
AWS S3 and SILO support attribute-based access control (ABAC) on object tags via s3:ExistingObjectTag/<key>:
GetObjectTagging/GetObjectVersionTaggingPutObjectTagging/PutObjectVersionTagging(which evaluates boths3:ExistingObjectTag/<key>on current tags ands3:RequestObjectTag/<key>on replacement tags)DeleteObjectTagging/DeleteObjectVersionTagging
When an object exists locally, the handler reads oi.UserTags from xl.meta and calls authorizeRequestWithExistingTags(ctx, r, action, oi.UserTags). When the object was missing locally, however, b2b26d9c9 proxied GET, PUT, and DELETE directly to active-active peers without reading remote tags:
- Callers subject to an explicit
Denywhens3:ExistingObjectTag/tier == "prod"bypassed theDenywhenever their request landed on a site that had not yet received the object, becauseExistingObjectTagevaluated as empty locally and the peer executed the proxied request under the replication service account. - Conversely, callers granted
Allowonly whens3:ExistingObjectTag/team == "analytics"received a false403 AccessDeniedwhenever their request hit a site missing the local copy. - In topologies with two or more active-active replication targets, tags on Target A and Target B can temporarily diverge while an earlier tag update is still propagating. Authorizing only the first responding target—or letting a mutation proceed on Target A before discovering that Target B carries a
Deny-triggering tag—would violate atomicity of authorization across replicas.
2. Impact Scope
| Deployment Topology | Exposed Surface |
|---|---|
| Active-active bucket or site replication | Anonymous callers can delete remote object tags (DELETE ?tagging) for objects absent on the ingress site; authenticated callers can bypass s3:ExistingObjectTag Deny policies on GET/PUT/DELETE ?tagging, and conditional Allow policies falsely fail with 403. |
| One-way or asynchronous replication | replicateSync=false does not itself disable proxying. Matching replication rules and proxy-target state determine selection; one-way or asynchronous mode is not an unaffected boundary. |
| Standalone / non-replicated buckets | Unauthenticated or unauthorized DELETE ?tagging leaks whether (bucket, key) exists (404 NoSuchKey vs 403 AccessDenied) even when the caller lacks s3:ListBucket; local tag authorization on existing objects is unaffected. |
3. Solution Design: Why It Is Necessary and Sufficient
Commits 14ecb222e, 67a90dfef, and 293ab78b4 unify GetObjectTaggingHandler, PutObjectTaggingHandler, and DeleteObjectTaggingHandler around a single local-and-remote authorization pipeline:
- Unconditional upfront authentication: All three handlers run
authenticateRequestbefore reading local metadata or considering replication targets. Anonymous requests on private buckets stop immediately unless bucket policy grants anonymous access. - Pre-probe gate (
authorizeRequestWithUnreadTags): WhengetObjectInforeturnsObjectNotFoundorVersionNotFound, the server does not blindly dial remote peers for every authenticated user. Instead,authorizeTaggingProxyevaluates whether the caller could possibly be allowed under some tag assignment (authorizeRequestWithUnreadTags). Callers with no matchingAllowstatement (or an unconditionalDenyindependent of tags) are rejected locally without generating cross-site RPC traffic or leaking remote existence; missing-object error masking (404vs403) honorss3:ListBucketconsistently acrossGET,PUT, andDELETE, while transient local storage errors (503 SlowDownRead) are preserved for authorized callers rather than collapsed intoAccessDenied. - Two-phase remote inspection across selected targets (
readTaggingProxyTargets):- Phase 1 (Read & Authorize Every Selected Present Replica):
getProxyTargetsselects matching replication targets.readTaggingProxyTargetsfirst skips targets without a client, already marked offline, or disabled for proxying, then callsGetObjectTagging. The following error handling applies to the selected targets:- If the peer returns
NoSuchKeyorNoSuchVersion(for example, because replication to a third site is lagging), that absent replica is skipped rather than aborting the request—unless zero peers hold the object, in which case standard404/403masking applies. - If any peer fails with a network, authentication, or
5xxstorage error, authorization fails closed immediately and aborts before any mutation runs, because that unreadable replica could hold a tag that triggers an explicitDeny. - For every peer where the object version is present, the server runs
authorizeRequestWithExistingTags(ctx, r, action, targetTags.String())(including incomingRequestObjectTagheaders onPutObjectTagging). If any present replica fails authorization, the entire request is denied (403 AccessDenied).
- If the peer returns
- Phase 2 (Mutate Retained Verified Targets): PUT and DELETE reuse only the client handles authorized in Phase 1. At least one attempted write must succeed; an error is returned when all attempted writes fail. Skipped targets cannot count as successful writes. This is not an atomic transaction across targets.
- Phase 1 (Read & Authorize Every Selected Present Replica):
Sufficiency and Necessity
- Necessary: Every component of the two-phase flow closes a distinct failure mode verified by adversarial tests (
cmd/tagging-proxy-authz_test.go):- Without upfront
authenticateRequest, anonymous callers reach storage and proxy paths. - Without
authorizeRequestWithUnreadTags, any authenticated caller with zero permissions on the bucket could force cross-siteGetObjectTaggingfan-out on arbitrary keys. - Authorizing only one peer fails in multi-target topologies when Peer 1 has stale tags and Peer 2 already carries a
Denytag; conversely, requiring all peers to hold the object breaks active-active availability during normal replication lag. - Failing open on an unreadable peer would allow an attacker or transient outage to bypass tag-based
Denyrules.
- Without upfront
- Sufficient: All three S3 tagging endpoints (
GET,PUT,DELETE) sharereadTaggingProxyTargets(cmd/bucket-replication.go), ensuring identical authentication,ExistingObjectTag/RequestObjectTagevaluation, missing-key masking (s3:ListBucket), and multi-target failure semantics.
4. Trade-Offs, Caveats, and Operational Costs
| Dimension | Engineering Trade-Off |
|---|---|
Extra remote GET before proxied PUT/DELETE |
Locally present objects do not need these proxy reads. When an object is absent locally, the proxy reads tags from each selected eligible target before returning or mutating remote tags. |
| Fail-closed availability under peer outage | A selected target that fails during the tag read aborts the proxy request before mutation. Targets already marked offline or disabled for proxying are skipped during selection; this does not guarantee inspection of every configured peer. |
| TOCTOU window across independent sites | Active-active S3 replication does not hold distributed cross-site locks across WAN links. Phase 1 authorizes the tags observed immediately prior to Phase 2’s remote write; concurrent independent tag writes on two sites continue to reconcile via replication tagging timestamps (TaggingTimestamp). |
Upgrade and Mitigation Guidance
- Upgrade every SILO Server node across all replicated sites to the post-
20261109release containing14ecb222e,67a90dfef, and293ab78b4. Because any unpatched peer accepting client S3 traffic can still proxy unauthenticatedDELETE ?taggingcalls using its replication credentials, every site in an active-active mesh must be upgraded. - Temporary mitigation prior to upgrade: If untrusted network clients can reach S3 endpoints on an active-active cluster before upgrade, block unauthenticated
?taggingDELETErequests at the reverse proxy / load balancer (for example, rejectingDELETErequests carrying query parametertaggingwhen theAuthorizationheader andX-Amz-Signaturequery parameter are both absent). - Post-upgrade audit: Deployments relying on object tags for ABAC isolation (
s3:ExistingObjectTag/*) on active-active replicated buckets should audit recent audit logs for unexpectedDeleteObjectTaggingorPutObjectTaggingoperations and verify tag integrity across peers.
See the Security Advisory Ledger, Candidate Server Compatibility Notes, and SN-2026-024 (GetObjectAttributes & ZIP Tag Authorization).